This one is missing 2 features also missing in the doc (INDEX BY for associations, and arbitrary joins) and adds the support of ON which is not implemented.

What is the reason to have this ON constant in the query builder ? It is confusing to get a DQL parse exception when using it if it is there.

On a side note, what is the canonical source for the EBNF ? There is 2 different locations in the code (the phpdoc of parser methods and the phpdoc of AST nodes created by the parser), plus the doc. Shouldn't we try to limit the duplication and have a way to check the consistency of the doc ?

The following "multi-dimensional index" should be perfectly possible, with a default hydration mode:
SELECT b as business, p as product FROM Businesses b INDEX BY b.id JOIN Products p WITH b.id = p.businessid INDEX BY p.id

However, b.id is completely ignored (it is a numeric primary key).

I tried to go further, giving 2 products a matching barcode and indexing by barcode and then a (unique, numeric) productid. Only the barcode worked as a key and only one of the products with a matching barcode was selected. I used this query to test:
SELECT p FROM Products p INDEX BY p.barcode JOIN p.businessid b INDEX BY p.id

I also flagged the docs, because I don't think a userid should/could be starting from 0.

WITH seems to allow to only "partially" load a collection, so the collection in memory does not fully represent the associations available in the database.

The resulting collection is marked as "initialized" and it seems there is no way to tell later on whether/how (with which expression) the collection has been initialized.

When using the collection filtering API, the "initialized" flag on the collection will lead to in-memory processing. If a collection has been loaded WITH a restricting clause and another filter is applied later, results may not be what one might expect.

I assume this is by design (no idea how the collection could be "partially" loaded and behave correctly under all conditions), so filing it as a documentation issue.

If you eager-load a collection using WITH, for the resulting entities that collection is marked as initialized as described above.

Should you happen to come across the same entity during hydration in another (later) context where you explicitly eager load the same association without the WITH restriction (or with another one), the collection on that (existing) entity won't be re-initialized and still contains the associated objects found during the first query.

I marked this as "Major" because this change represents a BC break. Because EBNF was not updated, I initially believed this to be a bug in ORM and wasted a lot of time debugging Doctrine code before I discovered this change was intentional.

When I follow bugtracker tutorial I think that there is an error when working with Annotations, see these examples:

Start a new bugtracker project as described in tutorial from scratch, create folders and files as tutorial expose, then do following changes:

1) put Product, Bug, User class files at root level, same level as bootstraps files and create_xxxxx files
2) Create 'entities' folder, but leave it empty.
3) Create 'yaml' and 'xml' at root level too and add related files.
4) Open bootstrap and edit paths to Product, Bug and User class files to read from root files as in 1) so they can be read from scripts.

1) YAML
a) When using YAML as mapping driver you need a path to Setup::createYAMLMetadataConfiguration( ) method, I use this
Setup::createYAMLMetadataConfiguration(array(_DIR_."/yaml"), $isDevMode);
where "yaml" directory is same level as bootstraps and create_xxxxx files are. When executing script to create a product
Doctrine work as expected, creating a row inside table correctly.
b) If you comment line where class Product is included, the object can't be found at runtime and will throw an exception as expected.
c) With uncommented require Product line, change yaml name folder to anything, and Doctrine throw an exception (MappingException) as expected

so Setup method path argument is considered correctly, Doctrine engine must know where yaml files for classes are.

2) XML
a) When using XML as mapping driver you need a path to Setup::createXMLMetadataConfiguration( ) method, I use this
Setup::createXMLMetadataConfiguration(array(_DIR_."/xml"), $isDevMode);
where "xml" directory is same level as bootstraps and create_xxxxx files are. When executing script to create a product
Doctrine work as expected, creating a row inside table correctly.
b) If you comment line where class Product is included, the object can't be found at runtime and will throw an exception as expected.
c) With uncommented require Product line, change xml name folder to anything, and Doctrine throw an exception (MappingException) as expected

so Setup method path argument is considered correctly again, Doctrine engine must know where xml files for classes are.

3) Annotations
a) When using Annotations as mapping driver you need a path to Setup::createAnnotationsMetadataConfiguration( ) method, I use this
Setup::createAnnotationsMetadataConfiguration(array(_DIR_."/entities"), $isDevMode);
where "entities" directory is same level as bootstraps and create_xxxxx files are (but remember THEY ARE EMPTY). When executing script to create a product
Doctrine WORK AS EXPECTED, creating a row inside table correctly and still I don't know how if THERE IS NO Annotations files.
b) If you comment line where class Product is included, the object can't be found at runtime and will throw an exception as expected.
c) With uncommented require Product line, change name folder to anything, and Doctrine WILL NOT throw an exception, continue with execution.
d) Copy /Product.php to /entities/Product.php, then comment docblocks from /Product class (using // or delete them). When running script Doctrine throw a MappingException with message: "Class Product is not a valid entity or mapped super class", when as follow concepts from 1) (yaml) and 2) (xml) it should search docblocks from /entities/Product.php file (path argument from Setup), right?

so Setup method path argument IS NOT CONSIDERED, Doctrine engine use already defined classes to get Annotations docblocks using php reflexion classes, methods and functions.

How to deal with this? I mean...

a) Erase path argument from Setup::createAnnotationMetadataConfiguration methos (and similar functions for Annotations) because is not needed, classes and annotations must be defined before.
b) Add support to find docblocks from path argument when no valid dockblock is found from class definition, so entities classes can live without docblocks because they are found inside Setup path function argument, as YAML and XML do.

I know that is easy to follow tutorial guidelines to develop applications in Annotations point of view, load them before Doctrine script start (with require/include or autoloaders, etc) and will work, but I think that is wrong how tutorial and functional logic are given, so a) and b) are my proposed solutions. I think b) should be right, get dockblocks from a class already defined and if are not defined it follow XML and YAML logic: read metadata from other files.

When you are looking for a config example for the bidirectional mapping of an one-to-many association you will just find an example with XML, but not with YAML or PHP. It would be nice if somebody could add an example or a link to the bidirectional one-to-one association, because it should be the same, right?

it does not include class declarations although the collections associated are both mentioned. It should be clear to which target entity they belong and therefore their classes should be declared.

from the context it seems that the associated classes should probably be User and Group, and the owning side is User. So the association should probably be inversed by 'users', although the example mentions 'features'.

the mapping for the inverse side maps a collection called $features, although this should probably be $users. Also the class declaration for the Group class is missing.

Some other code fragments in chapter 19 have similar issues. I think they could easily be replaced by the examples from the earlier chapters, like for the bidirectional man-to-many association the example from chapter 5:

The documentation recommends using private variables in entities. This can be problematic on entities with relations when using caching drivers as the proxy objects cannot access private variables and so the caching driver can throw notices like

...apc_store(): "_id" returned as member variable from __sleep() but
does not exist in ...

Making member variables protected resolves this issue when caching is enabled.

This information would be helpful on the documentation so others can be made aware of this issue. We spent a few days trying to debug the issue before understanding exactly what was going on.

Make sure that both calls equal to the same generator type. You can now modify the @Id fields in your entities. Additionally, make sure that you set the IdGenerator after you created the database using e.g. SchemaTool->create().

Hi, this doesn't seem to work for me. I have written a small database export / import utility. As long as I use the automatic ID generation, everything works flawlessly, but I'm trying to preserve the existing IDs. I do exactly what you've suggested in your post. It works for @OneToOne relations, but I get the following error messages when persisting entities that are parts of @ManyToOne relations:
Notice: Undefined index: [....] in [...]Doctrine/ORM/UnitOfWork.php on line 2655
I'm using version 2.2.2
Am I doing something wrong?

In the second "use" statement it references a "EntityManagerHelper" from the "Doctrine\DBAL\Tools\Console\Helper\" package. However, it does not exist there. It does exist in the "Doctrine\ORM\Tools\Console\Helper\" package though, and replacing it seems to work.

Let's say you want to process a bulk of 25 objects and have a batchsize of 20.
With the code provided the last 5 would not be saved as far as I understand unless you do another flush after the for-loop.

This is probably very clear to any experienced Doctrine developer but maybe it is also confusing for beginners like me (some internet sources say, that flush is executed automatically at the end of the request, but obviously it is not). Maybe this could be mentioned somewhere?

In SQL semantics, all four statements would be false (or more precisely, null). In PHP semantics, both null == null and null == 0 would be true, while null != null and null != 0 would be false.

It would be helpful to have the semantics of the comparison operators defined. While comparisons with non-null values behave in a common-sense way, it is hard to guess how queries involving comparison operators on fields allowing null values or null query arguments will filter the results, without knowing the exact semantics of the comparison operators with regard to null values.

The Documentation for @JoinColumn annotation states:
"This annotation is not required. If its not specified the attributes name and referencedColumnName are inferred from the table and primary key names."

However, this seems not to be correct. If you have non-standard name for the @Id columns for a @OneToMany/@ManyToMany the name and referencedColumnName are not correctly inferred.

Please make it more clear that prePersist and postPersist events are called only when creating new entity (that is, prior and after a database insert).

IRC log:
helmer
Hi. I have a question regarding (pre|post)persist events. Why are the events ignored for second persist (should fire prePersist) + flush (should fire postPersist) in the following pastie: http://pastebin.com/V8CrPWkM Is it a bug or am I missing sth?
Stof
helmer: there is no second persist. persist() means saying Doctrine to manage the entity. Once it is managed, you are not persisting it anymore but updating it
helmer
Stof: so basically one could define these two events for themselves as (pre|post)Insert?
beberlei
yes
helmer
thanks beberlei&stof! though a suggestion to docteam .. perhaps make it more clear to people like me, current doc can be kind of misleading ie: "There are two ways for the prePersist event to be triggered. One is obviously when you call EntityManager#persist()" http://www.doctrine-project.org/docs/orm/2.0/en/reference/events.html#prepersist

The XML Mapping documentation indicates that "orphan-removal" is an attribute on One-to-One or Many-to-One elements, but in DoctrineORM version 2.0 that does not work. It seems I have to make "orphan-removal" a child attribute of those elements to turn it on.

Just to be clear:

Documentation says to do this, but it does not work:
<one-to-many field="..." orphan-removal="true"> ... </one-to-many>

An mapped superclass is an abstract or concrete class that provides persistent entity state and mapping information for its subclasses, but which is not itself an entity. This annotation is specified on the Class docblock and has no additional attributes.

This doesn't adequately communicate how to use it. It took me several minutes of failing before I downloaded the PDF and did a search for @MappedSuperclass to find an example of how it's used.

Specifically the following were unclear:

Is this defined on the superclass or on the children classes?

If it's defined on the child classes, does it take parameters? The name of the super class?

It was not at all apparent to me that it was mutually exclusive with the @Entity tag

A mapped superclass has not many restrictions and these are mentioned in the docs (i.e. only unidirectional associations), what David mentions above should work, if it doesnt its a bug, I think DDC-511 looks like that same issue.

It doesnt matter. A @MappedSuperclass can be anywhere in an inheritance hierarchy and it always does the same thing, inherit its mapping information to subclasses (but its not itself an entity). The docs say:

Mapped superclasses, just as regular, non-mapped classes, can appear in the middle of an otherwise mapped inheritance hierarchy (through Single Table Inheritance or Class Table Inheritance).

as well as

Entities support inheritance, polymorphic associations, and polymorphic queries. Both abstract and concrete classes can be entities. Entities may extend non-entity classes as well as entity classes, and non-entity classes may extend entity classes.

So entities, mapped superclasses and plain non-mapped classes can appear mixed in an inheritance hierarchy. Nevertheless all the classes in a hierarchy that are entities must use 1 inheritance strategy, you can not mix inheritance mapping strategies in a single class hierarchy.

@"If it's defined on the child classes, does it take parameters? The name of the super class?"

No, it doesnt. The docs dont mention any parameters either which is correct.

@"It was not at all apparent to me that it was mutually exclusive with the @Entity tag"

Here is the list of reserved keywords for MySQL: http://dev.mysql.com/doc/refman/5.0/en/reserved-words.html
The documentation does not clearly mention that the backticks for a column with a reserved keyword in the ORM mapping file. This file could be in xml, yaml or php.
So,the right way to apply backticks is: <field name="group" type="string" column="`group`" length="32" nullable="false"/>

the DQL Parser don't parse properly functions when a ScalarExpression is needed like of all case functions.

In fact first function token is interpreted as a T_IDENTIFIER and enter on line 1663 of Doctrine\ORM\Query\Parser class. in search of math operator, when not found this case considere that the token is a row element with no considération of the functions procession treated after.

fix of this bug consist to enclose the line 1672 by a if (!$this->_isFunction()).

I install the Symfony v2.0.18. and made small TestBundle.
I made schema database, by CLI "console doctrine:schema:update --force"
Result: Database schema updated successfully!
But I saw that I lost a field 'user_id' in a table 'AttachTree' (see Attach)

What version of 2.1 are you using? We don't actually support 2.1 anymore. Inheritance has always worked as used in hundrets of unit-tests, this changes look quite major a bug to have been missed before. I can't really explain whats happening here.

Update and delete executors for Class Table Inheritance (JOINED) are extremely slow on MySQL platform. It is most probably due to use of subselect on the temporary table.
The slowdown is really significant as the table size increases. As an example, lets have a root entity with one subclass:

As you can see, MySQL is drastically slower on even relatively small tables. This currently makes Doctrine unusable for this type of inheritance on MySQL. The solution probably would be to avoid subselect in WHERE clause in Doctrine\ORM\Query\Exec\MultiTableUpdateExecutor and Doctrine\ORM\Query\Exec\MultiTableDeleteExecutor.

Changing this would be an improvement where we would hint if databases prefer subselects or joins for different operations. This would increase complexity of the SQL generation since now we are getting along with just one SQL generation strategy.

This would be especially handy when the difference between certain classes is only "implementational" (i.e. a subclass only functions differently/implements abstract methods and does not specify any additional fields). Using class table inheritance would result in tables only containing an id column.

Just wondering if that kind of implementation isn't changing expectations because it looks like IN(NULL) will select all NULL rows then, even though this is not explicitly requested by the user. I don't care really TBH but this behaviour is not very transparent to the user. Personally I would like to see an error instead... Then at least I know what's going on and can fix that by additional checks instead.

The question is if we can change this behaviour without breaking applications that rely on the current one. Because changing the code to throw an error breaks applications that insist on returning 0 rows for an empty array. I don't know what rules apply here concerning BC. What do you think Marco Pivetta?

Steve Müller feel free to patch DBAL.
It's a bug, not a supported feature. If we have 2 tickets:
1- If I pass an empty array, I get nullable rows and I can't fix this
2- If I pass an empty array, it used to return nullable rows, now it returns nothing
Which one would you fix?

will fail if MyEntity is part of a join table inheritance hierarchy and hence the update is handled by the MultiTableUpdateExecutor. The insert statement for the temporary ID table causes the following PDOException:

SQLSTATE[HY093]: Invalid parameter number: number of bound variables does not match number of tokens

The problem is in counting the number of input parameters in the update part of the statement. Lines 127-131 of the MultiTableUpdateExecutor:

And when I try to execute it, leaving the first paramter null, I got this error:

An exception occurred while executing 'SELECT n0_.numero AS numero0, n0_.anno AS anno1, n0_.id AS id2, n0_.nombre AS nombre3, n0_.activo AS activo4, n0_.comite_tecnico_id AS comite_tecnico_id5 FROM nomencladores.norma n0_ WHERE n0_.anno LIKE '%34%' OR n0_.nombre LIKE '%sad%'':
SQLSTATE[42883]: Undefined function: 7 ERROR: operator does not exist: integer ~~ unknown
LINE 1: ...o_id5 FROM nomencladores.norma n0_ WHERE n0_.anno LIKE '%34%...
^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.

I think in somewhere Doctrine is failing or not, not so sure, if not what will be the solution around this issue I'm having?

This request is in followup to my doctrine-user message "Doctrine 2.0: Nested joins'.
I am a bit surprised by the responses in that defining precedences in joins by placing parenthesis around join expressions is not well-known. Although not in the original SQL92 specification it is a major and important feature offered by all the RDBMS's that Doctrine 2 supports, and oftenly performs better than using subselects or alike. Doctrine 1 did not support it, but imho Doctrine 2 should support it to be a mature allround ORM.

As a short example the following is a SQL statement with a nested join, where the nesting is absolutely necessary to return only a's together with either both b's and c's or no b's and c's at all:

What further needs to be done is that the DQL parser loosly couples the ConditionalExpression to any of the previously parsed JoinAssociationPathExpression's instead of tieing it explicitely to the JoinAssociationPathExpression that preceedes it according to the old BNF notation. The new BNF should however not require any changes to the hydrator. Therefore I have the feeling that improving the DQL parser for nested joins does not require extensive work, while the benefit of running these kind of queries is considerable.

I surely hope you will consider implementing this improvement because it would save me and others from the hassle of writing raw SQL queries or executing multiple (thus slow) queries in DQL for doing the same. Thanks anyway for the great product so far!

Yes, this is a possible solution for DDC-512 but on the SQL level. I still don't see this as appropriate for DQL, it just doesnt make sense to me, DQL joins object associations, there is no precedence.

On a side note I would still like to know/see the following for this issue:

Some realisitic DQL examples where this feature would be essential, i.e. there is no other way to do it.
This also means explaining what the impact on the resulting object graph is and why it makes sense.

Which other ORMs support this on the OQL/Criteria level?

So far, my stance on this issue is:

1) It doesnt make sense (semantically) in DQL
2) Its rarely needed
3) When you really need it you can use a NativeQuery anyway and use this nesting in SQL, where it probably belongs and makes more sense
4) It would (unnecessarily) complicate DQL

Hi Roman. I understand your doubts, and I have been breaking my head over
creating a realistic example the last few hours that would hopefully convince
you for implementing this feature. But actually I cannot find one that you wouldn't
consider to be trivial. I do have a number of very complex optimized queries written
for sportskickoff dot com (using Doctrine 1.2) but they are probably hard to understand
because they may not be selfdescribing. Below is one example literally ripped from
the application. Still they often can be broken down to my example query in this
ticket's description, but applied grouping, additional other joins on the root component
and/or other criteria made them impossible to rewrite using subselects or choosing
another root component. Most often they just performed way best using the nested
syntax and saved me a number of additional queries.

But let's put it another way. I would also like this feature to be supported in DQL
because I just do not want to use native queries. Why would I want to use native
queries if it can be done using DQL? In DQL I work with class names and field
names, and they may differ from the underlying table and column names. Doctrine
takes care of that mapping based on my schema/annotations and I do not
have to "know" these mappings. In native queries I suddenly do have to "know"
these mappings. I use Doctrine because it makes my application portable and
enables me to work with my database in an OOP way like I do in my model,
abstracting things. The need for native queries partly reverts the benefits Doctrine
offers in the first place.

Btw, I recall to have successfully used the nested join syntax in HQL (.NET Hibernate)
but I cannot find examples on the web or a BNF notation.

Furthermore, in reply to your stances:
1) It indeed doesnt make sense (semantically) in DQL, it only makes the result
set different, but not the way data is hydrated into objects;
2) Its indeed rarely needed for inserting, updating and populating basic lists but
it allows you to better select what combinations of associated rows are joined
and which not in more optimized queries without having to use native queries,
or because they perform better than using subseletcs and alike.
3) Not having to use native queries is just an extra reason for using Doctrine and
maintains the abstraction the ORM provides througout on'es whole application
4) Why would it complicate DQL, if people do not know about or understand
the feature it wouldn't matter because not using parenthesises is the default
way to specify joins?

Well, this is it, can't find any more words to promote and make you enthusiastic.... lol.

Imagine a book store that sells books of various authors and keeps track of those sales.
Let's say you would have an admin page that lists all authors, and for each author
its also shows the books and their sales dates since january 1st, but only for those
books that were actually sold and contain an A in its name. An optimized SQL query
to fetch all the information at once would be something like:

If the database would contain thousands of books, but sales for just a
few books, this will definitely perform better than using subselects.
Off course one would like to fetch array graphs instead of objects for
further optimization, but this hopefully shows my point.

I have attached a test casefor a similar query, though without the additional
join constraints for clarity. I surely hope you can consider it.

One last note, you shouldn't be afraid that nesting joins is not in the
ansi SQL spec. Select queries are about record sets and products
between these sets, tables are just the basic means of providing record
sets to the query. This is an important terminological difference to think about.
Specifying precedence with parenthesis around joins is a logical and
natural evolution of the ansi sql standard. For example views are a good
proof of this concept, I could define book B INNER JOIN sale S as a view
and LEFT JOIN that to authors to get effectively the same result
set as the above example. The database server would internally perform the
same query (though may additionally take indexes on the view into account).
That said, rdbm's that support this syntax would certainly never drop the
feature, as its not a feature but just plain logical and smart querying!

P.S. I had a hard time finding out how to run the test cases, I could not find
it in the Doctrine 2 documentation, development wiki, cookbook or any other
place, while finally it was as easy as running phpunit Doctrine_Tests_AllTests
from within the tests/ directory, or just phpunit Doctrine_Tests_ORM_Functional_Ticket_DDC349Test
for my test. Could you please add some info about this somewhere, it might
save others some googling.

@"The need for native queries partly reverts the benefits Doctrine offers in the first place."

That is something I hugely disagree with. Neither SQL abstraction, nor database vendor independence is the main purpose of an ORM like Doctrine 2.
It is the state management of your objects, the transparent change tracking, lazy-loading and synchronization of the object state with the database state and nothing of this gets lost when using native queries.

We could rip out DQL and any other querying mechanism except a basic find() (and lazy-loading, of course), only providing the native query facility and even only supporting MySQL and would still retain all the core ORM functionality.

NativeQuery is one of the best and core "features" of the project. It is even the foundation for DQL. A DQL query is nothing more than an additional (beautiful) abstraction but what comes out is a native query + a ResultSetMapping, the same thing you can build yourself in the first place, even using the mapping metadata to construct the query. Nothing forces you to hardcode table and column names in native queries if you don't want that. Just use the mapping metadata, DQL does the same.

SQL abstraction and database vendor independence is icing on the cake, not the heart of the ORM.

Comparisons that use floating-point numbers (or values that are converted to floating-point numbers) are approximate because such numbers are inexact. This might lead to results that appear inconsistent:

I was affected by exact the same issue. Even when running git master at the beginning of last week, it was still broken.
Today I retestet. Luckily your latest commits (from beginning with: 445798ed46291f2639b3657142bd2f934d1be8a6) to the BasicEntityPersister seemed fixed it.

@revrev:
Could you please retest your example? This bug might be fixed. Thx.

[Doctrine\DBAL\DBALException]
An exception occurred while executing 'ALTER TABLE Unit CHANGE decimal decimal_numbers SMALLINT NOT NULL':
SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'deci
mal decimal_numbers SMALLINT NOT NULL' at line 1

The DATE_ADD built-in function of Doctrine supports only the "hour", "month" and "day" intervals. I don't see the point of not having all the intervals available. These should be "minute", "second" and "year". I didn't check compatibility with other database vendors rather than MySQL but I don't think it should be a problem

Now to the question: how can I remove the WHERE clause from this query. On more complex queries this part of the WHERE clause makes it not possible to use some indexes that are defined. This can be resolved by adding type to the indexes, but this makes my indexes larger and I feel this is not necessary.

The AbstractPage is the root in the inheritance tree. Thus we are interested in ALL records in the table. Omiting the WHERE part does precisely that.

So the question is: how can I make Doctrine remove this WHERE part where it is not necessary.

also joins Employee even though I am not interested in Employee properties.

Person is the base class in this use case and it should be possible to only retrieve those information. Thinking of OOP, if I instanciate Person I do not have any reference to its child either.
This is especially useful for large base tables where you want to only retrieve base information without inferring the childs.

IMO the DQL should be modified to allow selecting root/base information only, similar to the INSTANCE OF operator, that allows selecting specific childs only.

When executing partial DQL queries it may be important to keep given order of columns e.g. for "pairs" hydrator when first column of a pair is used as a key and second - as value. For example query like this:

select partial u.{id,name} from my:User u

will expect "id" to be first in resulted set and "name" to be second and not vice versa.

However Doctrine parses this part of statement via iterating over fields mapping from entity's class metadata (as can be seen in Doctrine\ORM\Query\SqlWalker::walkSelectExpression()):

Currently the only practical reason for it that I found is "pairs" hydrator. However it is, of course, possible to implement it without such change too.

Generally speaking this behavior (getting result set with same order of columns that was given in a query) is something that is feeling "natural" for operations with database since it is how you normally get results from SQL queries.

Maybe it will be enough to mention in documentation for Doctrine that given columns order is not guaranteed to be kept.

in a big project if you have queries spread out in different
repositories,
when you have to modify a cache lifetime, you have to search the query and
modify the code, than test it.
Is not so easy also to answer to 'how much is the cache for the query XYZ?'

the idea:
Each group of repository (bundle) should have in a single point maybe into its config file a place where you could set the lifetime of the various queries.

When using a LIKE condition in a WHERE clause, I would like to be able to use the foreign key of a single value association. For example:

Imagine Product and Department are both entities. Department has the field Name with a unique index. Product is has a ManyToOne association called Department with the Department entity, referencing the field Name.

I would like to write:
SELECT FROM Product p WHERE p.Department LIKE '% Tools'

However, at present I need to write:
SELECT FROM Product p
JOIN p.Department d
WHERE d.name LIKE '% Tools'

The issue is one of performance. On large record sets the first query runs several magnatudes more quickly than the second, particularly when four or five joins are involved.

Looking at the DQL grammar from the online docs, the relveant lines are:

This would improve consistancy with other aspects of DQL. For example, AggregateExpression COUNT, NullComparisonExpression , GroupByItem, and ArithmeticPrimary all allow a SingleValuedPathExpression rather than the more strict StateFieldPathExpression.

Bascially my frustration is that as DQL currently stands, foregin keys which are already existant in a db table cannot be used in a LIKE expression without doing an unnessessary JOIN. Foreign keys can already be used in DQL for BETWEEN, IS NULL and comparison expressions, so why not LIKE expressions also? It appears the only thing holding this back is an unrequired restriction in the DQL grammar.

Hi! Let me start by saying you guys did a great job with Doctrine 1 and that I can't wait to start using Doctrine2

I will explain this feature request with an example. I have a User entity wich relates one to many to a Picture entity. Picture has a " is main picture" boolean field. Not all users have a main picture. I would like to be able to select all Users, each with their main picutre, if that exists, or some Null value, if it does not exists, in one query, using join. I would also like for the result collection to contain Picture entities on the first level, with the User beinng accessible as an aggregate of Picture.

The way I can think doing this is by using a RIGHT or LEFT join (not INNER) as to also select Users that don't have a main picture. I can do this by selecting

SELECT Picture p, p.User u FROM p RIGHT JOIN u WITH p.main=1

but right joins afik are not available atm in either version of Doctrine, or by selecting

SELECT User u, u.Picture p FROM u LEFT JOIN p WITH p.main=1

and somehow instructing the hydrator to consider Picture as the root object for the generated object tree and User as a "child" of Picture.

For users without a picture, the Picture object would somehow indicate it is NULL, while still holding a refference to the User.

Makes sense? If there is an alternate way to achieve this, please enlighten me, tough I think it would still add felxibility if we could hint the hydrator for the root object in a tree.

Thanks for the suggestion. However, this was just an example to demonstrate some lack of flexibility, I am not strictly looking for a solution to this example, but to the concept behind it.

Also, how would I get the result with Picture on the top level and User aggregated to Picture with the model you suggested? Unless I am missing something, wouldn't I end up in the same situation?

I can post-process the results myself and create a new collection easily, ofc, but it would be better (and more optimal) if I could tell the hydrator to do this, similar to how INDEXBY is passed as an option to the hydrator.

Setting such a query hint to TRUE should result in all entities being retrieved by that query to be read-only for the purposes of change-tracking. Note that the entities themselves need not necessarily be read-only in general.

This feature is a flush performance tweak that can be used to query for objects but not let the returned objects run through change-tracking on flush. Any other managed objects are tracked as usual so you can do a read-only query for 100 entities and persist a new entity in the same unit of work with optimal flushing performance.

Internally, DQL2 could simply be transformed to the equivalent DQL1 by replacing the condition with conditions for each internal property. The advantage is that the one writing the query does not have to refer to the internal fields; the transformation is hidden.

A complicating factor is that Value Objects are Embeddables, but not every Embeddable is a Value Object. So there is always the question if objects need to be compared by reference or by their properties.

So, perhaps it's an idea to introduce a special operator ~ for comparing objects by their value to make the distinction explicit? Like so:DQL3:

I created a pull request that contains an idea how the same concept (the ~ operator) might be applied to criterias on in-memory collections.

Just some thoughts and ideas. I'd love to hear some discussion on this as I think it would make Doctrine really powerful in supporting rich, expressive domain models. It would be great if both in-memory collections and DQL supported this!

This would require to change the DQL to SQL conversion based on the fact that u.address is the path to an embeddable. It might impact performances
Using a separate operator would at least allow to know that it needs a special handling, without having to do complex changes to all places using the = operator.

A complicating factor is that Value Objects are Embeddables, but not every Embeddable is a Value Object. So there is always the question if objects need to be compared by reference or by their properties.

Embeddables cannot be compared by reference. They don't have an identity in the database. The only thing we have to compare them are their properties.

True, if we look at the database level, we can only compare by reference. However, if you look beyond ORM and also to the Collections package, then if you want to do a matching on a collection of entities that have an Embeddable by using a Criteria on that Embeddable, then you do have both options (see my referenced pull request). Then two operators might come in handy, so that the additional operator can be introduced to both ORM as well as Collections.

If we would go for one operator (=), then I think the Criteria in Collection also needs to be changed so that it performs a loose comparison on objects and a strict comparison only on scalars. Perhaps that is already out of the scope of the current issue, but either way it would be preferrable to have a consistent solution.

No, saying that we can only compare by reference is wrong. We *cannot* compare by reference (there is no way to reference them).

And talking about the needs of the criteria here is irrelevant, as this discussion is about building the DQL language, not about building the Criteria API. The criteria API can still have a separate operator to deal with value object even if the DQL uses = to compare embeddables. (btw, changing the Criteria comparison to loose on objects would break the comparison of relations, so it is totally impossible as it would be a BC break)

Christophe Coevoet not a performance impact since DQL => SQL is cached.
Adding a new operator resolution requires bigger efforts and I'm pretty sure it'll be slower than converting u.address to multiple clauses (we do it already with composite identifiers).

As http://sqlfiddle.com/#!7/6169b/1 shows it is not a big deal to transform such a query into an equal query which prevents the usage of "WHERE ... IN" constraints at all. I guess the case "pair(a,b) is not unique" can be safely ignored if redundant pairs are skipped on temp table insertion.

Currently ID hash is generated using implode function with space character. I would like to point out that problems may raise if the ID column values are allowed to contain the space themselves - different objects could return equal ID hash values.

The trivial one-line solution would be to serialize the array instead. Cheaper solution from performance perspective would be escaping the space character for ID values.

The current ClassMetadata API is based on a lot of array juggling (for performance reasons).

While that was understandable with PHP 5.3, all the array access operations are currently:

slowing things down

making the code very hard to read/understand

I suggest re-coding the ClassMetadata internals (public properties and such) so that well-described properties are defined.

Additionally, as a bonus, we'd get a performance boost by just moving all the class-alias and type resolution logic from the runtime into the ClassMetadataFactory (or similar) API, saving tons of performance at every run.

This would make metadata manipulation from events a bit messier (user needs to know which value to change during which event), but would allow using better constrained metadata structures in future, and that would disallow mistakes during event listeners execution as well (internal validation).

The changes in my sugestion code don't consider the prefix '_', indepedent of the characteres in alias column name, the function will remove the last characteres from alias... see an example with column name DT_VERIFICACAO_SITUACAO_ACESSO.

The current code create a sql:
DT_VERIFICACAO_SITUACAO_ACESSO AS _VERIFICACAO_SITUACAO_ACESSO23,

Fabio B. Silva Can you comment on Rudi Uhrig Neto solution? Its easily changed, but I am wondering if this is backwards compatible. I would want to merge the fix back to 2.3 and 2.4. Branch is ready to be committed for me locally.

At some point in the Doctrine releases (I don't remember which version it was), the default naming of the join table of a ManyToMany relation changed from using the property name to using the name of the target entity.

This makes it impossible to use multiple ManyToMany relations to the same target entity in a class without naming join tables explicitly. A SchemaException is thrown by the SchemaTool when trying to update the schema.

Note that this issue is not caught by the SchemaValidator. It will display us that the mapping files are correct (before throwing the above exception in the second step when trying to compare it to the existing database)

Using annotations in my model mapped to "mapped superclass" I can use one to many relations et every bidirectional relation I want.
Using yaml or xml it suddenly doesn't work and throw me an error. Annotations should work the same way and should not authorize one to many or any bidirectional relation.

I have a situation where I wanted a single table to use INSERT DELAYED. Its an audit log table where I expect each http request to generate many inserts for. In an effort to not over tax the system I implemented a custom Entity Persister so that it would work. This obviously doesn't work with all mapping drivers. However if this is a feature that you think is worth integrating I will fork it on github and complete the implementation alongside any changes/improvements requested...

When you try use custom annotation in mappedsuperclass like here http://pastebin.com/YMxKvcLk and then i try get metadata for class i get this error
Undefined index: fieldName
ClassMetadataInfo.php function addInheritedFieldMapping
Problem is that custom annotation doesnt have fieldName.
Quick fix is add condition to test if fieldName isset.

1. The information isTransient() has to be moved to the ClassMetadataFactory and cached there.
2. The information getAllClassMetadataNames() can be cached
3. A debug/development mode should be introduced, leading to filemtime caching and checks so that you can use ApcCache and such in development.

If you subscribe to loadClassMetadata you will usually modify the metadata for some classes.
The problem is, that that data has to be loaded from somewhere. But later down the chain you can't get to it. Now data specific to what you need in your loadClassMetadata would ideally reside in the same location.
If we take for example a file, than all data for a specific entity is in the same file.

My proposal would be to add function get(Original|Raw)MappingData into interface Doctrine\ORM\Mapping\Driver\Driver which would either return raw data or data in a object specific for that Driver or null if it doesn't make sense for that driver. Please note, that when loading from e.g XmlDriver we should return simplexmlnode or dom node as loadClassMetadata should be in its own namespace and not pollute the Doctrine one.

This is caused by taking the join column name as the identifier while generating a property name for annotation. The mapping driver detects that the same property is already defined and ends the convert process. A little bit smarter approach for me was to take the local table name. But this assumes a specific style of join table naming convention.

indexBy implementation on containsKey (and possibly other EXTRA LAZY implemented methods) of PersistentCollection in EXTRA LAZY mode uses property name as column name instead of column name for OneToMany collections

For example property = fieldName, column name field_name.
Set indexBy on OneToMany annotation to fieldName (as it should be ) and it tries to use fieldName in backend instead of field_name

It should create a table "user_role" with 2 columns which are CHAR(36).

But it ignores the Column-Attributes and creates a table "user_role" with 2 CHAR(255) columns.

This has various downsides:

It's unusable when using MyISAM, because of limited index size. (CREATE TABLE fails, see DBAL-423)

If using GUID-Type (see DBAL-423 with the changes from the linked ull request) and specify "length=36" and "fixed=true" on the Column-Annotation, no changes for the entity-tables itself are generated when running orm:schema-tool:update. However, there are still changes for the many-to-many-table generated (because internal "fixed" is false and length is unset) which represent the current state of the columns. These changes are always generated.

It would be nice to be able to specify which collation to use on a field basis.

This would for example be useful when you have case-sensitive (utf8_bin), and case-insensitive (utf8_general_ci) values. Right now, this needs to be manually added to migration files (which is ok for projects, but it is not so nice for distributable libraries).

In Doctrine 2.3, the mapping works correctly, and you end up with a 3-part primary key, with a user property mapped to the User entity, and a datasourceCode property mapped to the DatasourceCode entity. All good.

In 2.4, the following error is given: Single id is not allowed on composite primary key in entity UserEmail.

Removing one of the foreign keys in the table (either to User or DatasourceCode) but keeping the primary key set to all 3 columns allows the mapping to work. But, if you then remove one of the columns from the primary key (say, email_datasource) it fails again.

The problem is that Doctrine seems to detect that you only have on id field on this entity and then a new function of 2.4 throws this error. I will try to reproduce this with your table. Until then could you show me the var_dump() of the $class variable in Doctrine\ORM\ClassMetadataFactory#completeIdGeneratorMapping()? This would help very much already.

I tested this again using 2.3.4 (the version which contains this fix) and it is still occurring. Attempting to import mapping for a table with 2 foreign keys in the primary key results in the error "Database does not have any mapping information." Adding a third column on the primary key "fixes" the issue.

Currently our developers are being asked to add a fake third part to the key to work around the issue, then delete that key once they get into the entity class. This is a bit tedious and I'd love to see a fix!

When executing the command doctrine: mapping: import I get the following error

[Doctrine\ORM\Mapping\MappingException]It is not possible to map entity 'EstablecimientoSec' with a composite primary key as part of theprimary key of another entity 'EstablecimientoSecPlano#idEstablecimiento'.

While Trying to Embed an object into a mapped superclass an exception is thrown:

PHP Fatal error: Uncaught exception 'Doctrine\ORM\Mapping\MappingException' with message 'Duplicate definition of column 'embed_someField' on entity 'Entity' in a field or discriminator column mapping.' in /home/dev/Workspace/symfony/perto/vendor/doctrine/orm/lib/Doctrine/ORM/Mapping/MappingException.php:555

So, the problem is when the one needs to use association key in composite identifier; they are added in the end of the identifier array, which is clearly not always suitable in regards to performance.
For example, following mapping:

In version 2.5 added support sequence with the strategy of identity.
But I cannot get the correct name of the sequence with the strategy of identity. With the strategy sequence produces the correct name. It is taken from the configuration sequenceName. Strategy identity name sequence will not be taken from the configuration, and is always generated is automatically (which is not working properly in my case).

My situation:
I have realty entities (flat, house, ...) they have hundreds parameters, wich title places in REFERENCE table. For example House has a few floor materials (stone, wood, etc.) and I want to get them. But each parameter type defined by type_id field in REFERENCE table, for example for floor material is's 115. My raw SQL:
SELECT h.id,... FROM HOUSE h
LEFT JOIN object_detail mf /material_floor/ ON mf.realty_id = h.id AND mf.data_class = h.data_class AND mf.type_id = 115
LEFT JOIN Reference mfr ON mfr.city_id = h.city_id AND mfr.id = mf.detail_id

XmlDriver and AnnotationDriver does not allow for "options" on the "id" element, which the YamlDriver does.

This would help, since specifying "unsigned integer" using column-definition will make the doctrine orm:schema-tool:update think that the unsigned column in the database table doesn't match the signed integer in the mapping.

I noticed some problems with
the doctrine-mapping.xsd (which apparently is not used to validate the
mapping, but only provided to support code completion in your xml-
editor)

The order in which you define the elements like cascade and join-
column(s) for the relation-entities is currently important since it is
a sequence, but is this really desired? Why force this order? defining
cascade before join-column would make perfect sense as well, doesn't
it.

I looked in the repository and it seems xsd is not valid for all branches 2.1.x, 2.2, 2.3.4, 2.4.0-RC2 and master

If you use in your MySQL database default values, indexes or string primary key, you get incorrect mapping by mapping generator. For get it - just use in database one or more from listed abilities, generate mapping for that and then try to dump-sql with schema-tool:update.

In using a database-first approach utilizing orm:convert-mapping to generate xml, the validation and schema-tool reports that my composite primary key (ex. Columns A, C, B) be dropped and added in the order in which the mapping appears in the xml (ex. Columns A, B, C).

These columns are not auto-increment and are simply a mixture of int and varchar.

I'm working with Symfony 2.1, and I need to know if an association is nullable for a given entity (to know if a form field should be marked as 'required'). So I'd like to have a isAssociationNullable() method in the ClassMetadataInfo class, that should do the same thing that the isNullable() method does for fields.

Hi there,
AbsractFileDriver is using the filename to know the managed class.

It's a cool feature because it's allow loading on-demand.
The problem is, that the filename must be the name of the Class.

It should be great to be able to manually map XML/YAML File description to a Class, like :
$drivers->addMappingFile ( array ( "filename" => "class", "filename2" => "class2") );

This feature is simple to implement, just add a new array inside AbsractFileDriver to know the mapping.
When using the current method with addPaths, parse the folder to get traditional XML/YAML file where filename corresponding to classname and add it to the mapping array.

If an IDE would like to support annotations, it's currently only possible to display all resolvable classes in a code-hint menu when autocompleting annotations, as
basically any class can be used for annotations.

To make it possible for IDEs to detect classes which are explicitly meant to be used as annotations, it would be nice to agree on some common
way of documenting annotations in PHPDocBlocks.

For databases that don't support those persistent comments, you could use normal SQL comments in the table definition. Of course they wouldn't be stored by the database, but at least they would appear in the generated schema, that's better than nothing.

1. I setup the owning side of a relationship (let's use ManyToMany, the hardest one for this)

2. Later, I decide to setup the inverse side of the relationship. When I do this, I of course add the `mappedBy` attribute so that it points to the exact property/relationship we're dealing with

3. Then, I also need to go back to the owning side and add `inversedBy`.

Why is step 3 (inversedBy) needed? Isn't this redundant information, since the `mappedBy` fully maps out that these 2 associations form different sides of the same relationship? I would love to remove this, I hate explaining to people starting with relationships to now go back to the main entity to add this key. It feels like duplication, which I think people sense.

I'm not sure why JPA introduced this sort of bidirectional mapping, but for practical purposes, it makes it possible for us to load metadata for associations on both sides of the associations. Without having both mappedBy and inversedBy we'd be forced to scan through all existing mappings to find which pieces of the jigsaw fit together.

I don't think we have a good solution for this except for a "build mappings" step that warms up a cache, and that's a very radical architectural change that we can only introduce in 3.x

Hi, please excuse me if what I say is wrong and do not hesitate to correct me, it's my first immertion in Doctrine code .

So I checked a little bit the situation in the code. I don't see the problem dropping inversedBy. It's easier understandable for the final user.
In facts right now the ManyToMany work very well with only `mappedBy` options so the `invertedBy` is clearly a duplicated information.

The problem for the implementation of this feature is that the ownerSide is decided by a single method for every mappings, so she's quite complexe. So the method decide only on using `mappedBy` and `invertedBy`. This behavior should be modifiable without too much troubles.

Mapping driver probably generates entities in wrong order, so entities defined in discriminator map does not exist when entity Person is being generated.

Note: The easiest workaround is to temporarily remove discriminatorMap definition from entity, generate entities, then put discriminatorMap back and then generate entities again. It would be great if it would be possible to generate all entities in only one step.

When mapping a many-to-one relationship (in XML format), I made the mistake of having both sides be inversed-by (instead of one using mapped-by). This resulted in the following error message on doctrine:schema:update

Doctrine\ORM\ORMException: This behaviour is (currently) not supported by Doctrine 2 (uncaught exception) at /Volumes/User Data/Users/Tom/Sites/BM2/vendor/doctrine/orm/lib/Doctrine/ORM/ORMException.php line 217 while running console command `doctrine:schema:update`

This message could be more helpful. It doesn't tell me where the error is nor what the error is.