Exploratory testing:

This improvement makes a lot of changes throughout the site.
Admin settings:
[Site administration ► Users ► Permissions ► User policies]
fullnamedisplay
This setting is what most students will see.
Update the setting to include some of the following name fields:
firstnamephonetic
lastnamephonetic
middlename
alternatename
Basic testing
Change the admin settings above.
Test that the following areas display as expected:
The following places have been updated to show display names with the new settings:
Navigation block
"Logged in as" area in the top right hand corner.
Participants page
All of the core modules
Admin settings
Profile pages
It is recommended that exploratory testing be done to try to eliminate as many bugs and regressions as possible.
Exploratory testing:
Things to test:
Installing Moodle
Upgrading Moodle
Participants page
Navigation
Activities
Admin settngs
Gradebook
Webservices
Backup

Description

Alternate name fields

This is a proposal on behalf of The Moodle Association of Japan (http://moodlejapan.org). We are willing to work closely with Moodle HQ on this improvement and help by providing some funding for its development.

The standard Moodle has only two fields for names "firstname" and "lastname". In countries that use a different writing system, this means that two sets of names need to be displayed, their name in the local writing system and their name in Romanization. Moodle needs a way to input both types of names, and for both the individual and the administrator (and perhaps at the course level) to determine which names appear in the various Moodle contexts.

For standard English and European usage, these would be ignored, but for Japanese, Chinese, Korean and perhaps other writing systems such as Thai, the various Indian languages and Arabic. Possible entries would look like this:

lastname

firstname

lastphon

firstphon

Alternate

Japanese

佐藤

宏

サトウ

ヒロシ

Satou Hiroshi

Korean

李

奎報

이

규보

Yi Gyubo

Chinese

陳

美和

ㄔㄣ

ㄇㄟㄏㄨㄚ

Chen MeiHua

The site administrator would select how the $FULLNAME variable would appear. For Japanese, for example, the administrator might select lastname + firstname + alternate so that the students' names would appear in Kanji, but with the Romanization of it following the Kanji. This would allow everyone to know how the name should be pronounced. (Even Japanese need this since there are alternate pronunciations for Kanji. "神部" for example, could be read as "Kamibe" or "Kambe" depending on the preference of the user's family.)

Any combination of the above could also be selected as the default for user names that appear in various tables.

If the selected fields for tables is one of the Asian phonetic fields, then rather than the standard alphabet display for filtering the list by first name and last name, a similar selection display would appear for the selected alphabetic system.

For Chinese, a selection would also need to be made for the phonetic system to use. Taiwan uses the "BoPoMoFo" (Zhuyin) system, while mainland China uses the Pinyin system.

Student ID Display

For many Asian languages, another problem not encountered so much in the West is people with identical names. This, plus the fact that some of the languages (namely Japanese and Chinese) have no convenient method for ordering names in Chinese characters, means that most schools rely on the student ID for ordering and identifying students rather than (or in addition to) their names. Thus, there should also be an option in the site set-up selecting the StudentID field to be displayed to the left of names on various listings such as the grade book, assignment evaluation, etc.

Complexity & coding time:
This problem is quite complex since it involves modifying core code in perhaps 50-100 files that do database I/O, table displays, preference selections, etc. and would need to be done in close coordination with Moodle Headquarters.

Who would benefit from this improvement?:
Probably every institution in Asia that has staff who do not read the local language need to be able to display names in both the local language and Roman letters. Various, rather unsatisfactory ad hoc solutions are currently used but a permanent, more elegant solution is urgently needed.

Activity

Hideto, Peter, and I discussed this issue with Michael de Raadt at the MoodleMoot Japan conference. He raised two other possible reasons for alternate name fields that have been requested in the past:

avatars/nicknames in certain contexts

complete anonymity in certain contexts

These other situations would be related to the issue thus requiring a flexible way to display and order class name lists, or allow varying levels of privacy in some contexts. Michael went through several strategic approaches that I think he will suggest as soon as he is back at the office. As the Stable Branch lead developer, he will be the first to address this issue.

At our discussion, I also pointed out that student number needs to part of the display with these name displays, as most class rosters are arranged by student number in ascending order for grading, attendance-taking, assignments, etc. Presently, at my university Moodle in Japan, we use five data types within two profile fields in order to handle class rosters.

This is not really enough because many teachers also need a roman character lastname and firstname. Our university student management system (by Fujitsu) has all three alphabets available to all teachers in printing out class rosters. All sets of numbers and names are available in reports and displays.

Don Hinkelman
added a comment - 26/Feb/12 6:53 PM - edited Hideto, Peter, and I discussed this issue with Michael de Raadt at the MoodleMoot Japan conference. He raised two other possible reasons for alternate name fields that have been requested in the past:
avatars/nicknames in certain contexts
complete anonymity in certain contexts
These other situations would be related to the issue thus requiring a flexible way to display and order class name lists, or allow varying levels of privacy in some contexts. Michael went through several strategic approaches that I think he will suggest as soon as he is back at the office. As the Stable Branch lead developer, he will be the first to address this issue.
At our discussion, I also pointed out that student number needs to part of the display with these name displays, as most class rosters are arranged by student number in ascending order for grading, attendance-taking, assignments, etc. Presently, at my university Moodle in Japan, we use five data types within two profile fields in order to handle class rosters.
profile lastname field: student number + lastname Chinese character + firstname Chinese character
profile firstname field: lastname phonetic (kana) alphabet + firstname phonetic alphabet
This is not really enough because many teachers also need a roman character lastname and firstname. Our university student management system (by Fujitsu) has all three alphabets available to all teachers in printing out class rosters. All sets of numbers and names are available in reports and displays.

Gordon Bateson
added a comment - 26/Feb/12 7:00 PM I would also like to add my support for dividing the proposed "alternate" field into "altfirst" and "altlast", as this would allow for sorting of the student lists on English first or last name.

Aaron Batty
added a comment - 27/Feb/12 8:02 AM I second both Mr. Hinkelman's and Mr. Bateson's further suggestions. It would be great if I had 7 fields to work with: 2 for kanji, 2 for phonetic, 2 for roman, 1 for student number.

Eric Hagley
added a comment - 27/Feb/12 9:51 AM As mentioned, I'm sure that every institution in Asia would benefit from this addition. I look forward to its implementation and greatly appreciate all endeavors to that end.

+1 on the main gist. If you want Moodle used on any scale in Japan and probably much of the rest of Asia, you need a clean way to handle your students' names.

At the risk of complicating things, a couple of things that occur to me having built a (non-Moodle) system for a school with a mixture of Japanese, Chinese and US students, and Japanese-speaking and English-speaking staff and teachers. (Maybe MAJ have already discussed all this and figured this is the optimal proposal, in which case feel free to ignore me):

Having one big "alternate" field looks to condemn you to never being able to display or sort by Roman last name. (Or first name if you input it in the order "last first".) Wouldn't it be better to do "lastroman" and "firstroman"?

Knowing what to display where is a bit fiddly if you want to do it right. (Maybe this is why the MAJ proposal punts the problem to the end user with a big old "alternate" field, that you can stuff whatever you like in?) We normally want to show the Japanese name to Japanese speakers, and the English name to English speakers. That makes your name settings language-dependent, rather than having one site-wide setting for what $FULLNAME means.

This goes beyond Asian languages and we probably shouldn't go there, but if you're customizing $FULLNAME you may even want to do it by role. We have some external graders who don't even get to see the students' names - all they get to see is their student IDs.

Edmund Edgar
added a comment - 27/Feb/12 11:47 AM +1 on the main gist. If you want Moodle used on any scale in Japan and probably much of the rest of Asia, you need a clean way to handle your students' names.
At the risk of complicating things, a couple of things that occur to me having built a (non-Moodle) system for a school with a mixture of Japanese, Chinese and US students, and Japanese-speaking and English-speaking staff and teachers. (Maybe MAJ have already discussed all this and figured this is the optimal proposal, in which case feel free to ignore me):
Having one big "alternate" field looks to condemn you to never being able to display or sort by Roman last name. (Or first name if you input it in the order "last first".) Wouldn't it be better to do "lastroman" and "firstroman"?
Knowing what to display where is a bit fiddly if you want to do it right. (Maybe this is why the MAJ proposal punts the problem to the end user with a big old "alternate" field, that you can stuff whatever you like in?) We normally want to show the Japanese name to Japanese speakers, and the English name to English speakers. That makes your name settings language-dependent, rather than having one site-wide setting for what $FULLNAME means.
This goes beyond Asian languages and we probably shouldn't go there, but if you're customizing $FULLNAME you may even want to do it by role . We have some external graders who don't even get to see the students' names - all they get to see is their student IDs.

I have been adding profile fields for each of my profiles for years now. The problem is that I cannot define what data to display in my userlist. Often the information I want to see (or sort users by) does not appear on the screen. If I could easily define the data to be displayed and use extra profile fields, this problem would be solved for me.

Adam Jenkins
added a comment - 27/Feb/12 12:59 PM I have been adding profile fields for each of my profiles for years now. The problem is that I cannot define what data to display in my userlist. Often the information I want to see (or sort users by) does not appear on the screen. If I could easily define the data to be displayed and use extra profile fields, this problem would be solved for me.
Sorry if this is solved in 2.2, I'm still stuck on 1.9.

Thanks for putting this forward on the tracker. I know that this issue has been brewing up for some time, so noe it can get some attention from us at Moodle HQ as well as from the community.

This issue has 28 votes already and I suspect that will grow. I believe this improvement would benefit more than Asian language speakers.

I believe that displaying student ID numbers is already possible, although how this is done could be improved and made more obvious and universal. Even though it is related to the display of names, I think it should be handled separately from this issue, and if more work needs to be done, then further issues should be launched.

Storing name data

In relation to allowing alternate name fields to be associated with a user, there are two possibilities I have thought of.

'''Adding more fields to the user profile directly.''' This would be inflexible, but simple. However, we already have a number of fields in user profiles that we are considering removing and turning into custom profile fields.

'''Adding a new type of custom profile field for names.''' This would mean that administrators would need to establish the fields they want, but they would be free to create as few or as many fields as they wish. It might also be possible that a set of default custom profile fields are added with a new installation. We wouldn't need to try to create a solution to suit everyone now and into the future. This type of custom profile field, once added could be treated like other profile fields (for importing data, authentication, etc.), but could be recognised as special when defining how names would be displayed.

Displaying names

At the moment we have a focal point in the code where we decide how names are displayed. The full name display can be controlled by an admin. There are a few options for name order and whether first and/or last names are displayed. The name display can also be configured in the language file, so arbitrary patterns are possible. There are capabilities in place to override restrictions if partial names are displayed. This system could become more flexible so that it could be configured into more formats than what can be chosen from a list and without manipulating a language file.

Context

At the moment, display of full names is controlled at the site level. This is a good starting point for development, however there is the potential for names to be displayed in different ways at lower context levels, such as in a course or a module. This would allow instructors with different needs to define how names appear within a site.

Anonymity

An extension to this work could be to allow people to appear anonymously. If the context control is added, this could allow for names to be hidden within a course or specific module. However there is more to anonymity than simply hiding a name, and achieving anonymity could be more complex and adding alternate name fields. In my opinion I would leave anonymity out of this project, at least initially.

Some further considerations:

How should alternate names appear in tables, such as the gradebook, reports and in activity result tables (assignments, quizzes, etc.)?

How should alternate names be sorted? Is Unicode order of characters appropriate? I think we may need a more universal means of comparing names.

I invite all interested parties to vote, comment, propose, design and of course code. Let's make Moodle better for everyone.

Michael de Raadt
added a comment - 27/Feb/12 1:20 PM Thanks for putting this forward on the tracker. I know that this issue has been brewing up for some time, so noe it can get some attention from us at Moodle HQ as well as from the community.
This issue has 28 votes already and I suspect that will grow. I believe this improvement would benefit more than Asian language speakers.
I believe that displaying student ID numbers is already possible, although how this is done could be improved and made more obvious and universal. Even though it is related to the display of names, I think it should be handled separately from this issue, and if more work needs to be done, then further issues should be launched.
Storing name data
In relation to allowing alternate name fields to be associated with a user, there are two possibilities I have thought of.
'''Adding more fields to the user profile directly.''' This would be inflexible, but simple. However, we already have a number of fields in user profiles that we are considering removing and turning into custom profile fields.
'''Adding a new type of custom profile field for names.''' This would mean that administrators would need to establish the fields they want, but they would be free to create as few or as many fields as they wish. It might also be possible that a set of default custom profile fields are added with a new installation. We wouldn't need to try to create a solution to suit everyone now and into the future. This type of custom profile field, once added could be treated like other profile fields (for importing data, authentication, etc.), but could be recognised as special when defining how names would be displayed.
Displaying names
At the moment we have a focal point in the code where we decide how names are displayed. The full name display can be controlled by an admin. There are a few options for name order and whether first and/or last names are displayed. The name display can also be configured in the language file, so arbitrary patterns are possible. There are capabilities in place to override restrictions if partial names are displayed. This system could become more flexible so that it could be configured into more formats than what can be chosen from a list and without manipulating a language file.
Context
At the moment, display of full names is controlled at the site level. This is a good starting point for development, however there is the potential for names to be displayed in different ways at lower context levels, such as in a course or a module. This would allow instructors with different needs to define how names appear within a site.
Anonymity
An extension to this work could be to allow people to appear anonymously. If the context control is added, this could allow for names to be hidden within a course or specific module. However there is more to anonymity than simply hiding a name, and achieving anonymity could be more complex and adding alternate name fields. In my opinion I would leave anonymity out of this project, at least initially.
Some further considerations:
How should alternate names appear in tables, such as the gradebook, reports and in activity result tables (assignments, quizzes, etc.)?
How should alternate names be sorted? Is Unicode order of characters appropriate? I think we may need a more universal means of comparing names.
I invite all interested parties to vote, comment, propose, design and of course code. Let's make Moodle better for everyone.

"How should alternate names be sorted? Is Unicode order of characters appropriate? I think we may need a more universal means of comparing names."

Ideally you'd want to sort lists of Japanese names by their phonetic reading ("lastphon, firstphon" above), which bears no resemblance to what you'd get if you sorted the Chinese characters in the lastname / firstname fields by any method (unicode code-point order, utf8_general_ci, utf8_japanese_ci):

木内 [Kiuchi]
菊地 [Kikuchi] ("ku" comes after "u" in the Japanese phonetic sort order)
木倉 [Kikura] ("ra" comes after "chi" in the Japanese phonetic sort order)

I think if you're sorting phonetic data you can probably rely on the database's default collation (utf8_general_ci or whatever) at least for Japanese. I'm not sure if there are any languages or cases where that assumption breaks.

(The catch is that you may not actually have that phonetic data for everybody, in which I case I suppose falling back on whatever order utf8_general_ci thinks is best for sorting the last/first data is the bet you can do.)

If you're displaying a list headed by Roman-alphabet names rather than Japanese ones, you'd want to sort by those and ignore the Japanese phonetic order.

Edmund Edgar
added a comment - 27/Feb/12 3:42 PM "How should alternate names be sorted? Is Unicode order of characters appropriate? I think we may need a more universal means of comparing names."
Ideally you'd want to sort lists of Japanese names by their phonetic reading ("lastphon, firstphon" above), which bears no resemblance to what you'd get if you sorted the Chinese characters in the lastname / firstname fields by any method (unicode code-point order, utf8_general_ci, utf8_japanese_ci):
木内 [Kiuchi]
菊地 [Kikuchi] ("ku" comes after "u" in the Japanese phonetic sort order)
木倉 [Kikura] ("ra" comes after "chi" in the Japanese phonetic sort order)
I think if you're sorting phonetic data you can probably rely on the database's default collation (utf8_general_ci or whatever) at least for Japanese. I'm not sure if there are any languages or cases where that assumption breaks.
(The catch is that you may not actually have that phonetic data for everybody, in which I case I suppose falling back on whatever order utf8_general_ci thinks is best for sorting the last/first data is the bet you can do.)
If you're displaying a list headed by Roman-alphabet names rather than Japanese ones, you'd want to sort by those and ignore the Japanese phonetic order.

Geordie McGarty
added a comment - 27/Feb/12 6:49 PM This would be a MAJOR improvement for asian based Moodle users. I agree with Don Hinkelman's comments as well. I really hope this can be implemented in a near future build.

Ronald Notestine
added a comment - 01/Mar/12 8:13 PM I agree this would be a big improvement. I also would like to see seven fields, including altfirst, altlast, and studentnumber. My thanks to Tom Robb for initiating this.

In response to Edmund Edgar's comments about sort order, for Kanji there is no order that is useful and rememberable, so other means are used. List ordering is normally done by the assigned student numbers, which in turn have probably be assigned in some sort of natural order. In Japan, the student numbers are normally assigned in kana order, thus, sorting the students by their ID numbers is equivalent to sorting them by their kana order, as long as the students are all in the same faculty and entered in the same year.

In the system that I initially proposed, entries in the "lastphon" and "firstphon" categories can be used to order the various reports. There would have to be a selection in the site set-up (if not at the course-level, too) which would allow admins to specify a sort order combination (first only, last only, first-last, last-first) which would perhaps not appear in the listing itself, but would be referred to when the list is sorted by name.

Thomas Robb
added a comment - 15/Mar/12 11:23 AM In response to Edmund Edgar's comments about sort order, for Kanji there is no order that is useful and rememberable, so other means are used. List ordering is normally done by the assigned student numbers, which in turn have probably be assigned in some sort of natural order. In Japan, the student numbers are normally assigned in kana order, thus, sorting the students by their ID numbers is equivalent to sorting them by their kana order, as long as the students are all in the same faculty and entered in the same year.
In the system that I initially proposed, entries in the "lastphon" and "firstphon" categories can be used to order the various reports. There would have to be a selection in the site set-up (if not at the course-level, too) which would allow admins to specify a sort order combination (first only, last only, first-last, last-first) which would perhaps not appear in the listing itself, but would be referred to when the list is sorted by name.

I'm noting that at this point, with 45 votes, this issue is the 29th most voted for issue in the dev backlog. I suspect, though, that the impact of this change would be very beneficial in many parts of the world.

Shane: I've added you as a watcher on this issue as you said you were interested in working in this area at MootAu12.

Michael de Raadt
added a comment - 12/Jul/12 9:43 AM I'm noting that at this point, with 45 votes, this issue is the 29th most voted for issue in the dev backlog. I suspect, though, that the impact of this change would be very beneficial in many parts of the world.
Shane: I've added you as a watcher on this issue as you said you were interested in working in this area at MootAu12.

I have been working on one possible solution to this improvement. It follows Michael's idea of using user profile fields. This still needs a lot of work, but you're quite welcome to load the patch and try it out for yourself. Instructions on how to use this patch are located in the comments of issue MDL-34754 (a sub issue of this one).

Concerns that I have about this solution are that it does a lot more calls on the database, which could potentially slow down your system. If I continue to progress down this path then I will need to possible alter different pages that provide lists of students on a single page.

Adrian Greeve
added a comment - 10/Aug/12 3:52 PM I have been working on one possible solution to this improvement. It follows Michael's idea of using user profile fields. This still needs a lot of work, but you're quite welcome to load the patch and try it out for yourself. Instructions on how to use this patch are located in the comments of issue MDL-34754 (a sub issue of this one).
Concerns that I have about this solution are that it does a lot more calls on the database, which could potentially slow down your system. If I continue to progress down this path then I will need to possible alter different pages that provide lists of students on a single page.
I welcome your suggestions and recommendations on this issue.

Hello Adrian. Actually, we are not interested in a "patch" since that would only be a stop-gap solution. We need a solution that quickly allows the sys admin to determine which, of many name fields – including the idnumber – to appear in all relevant display with full capability to sort on them, as can be done with the standard two name fields. Also politically, it renders non-Western alphabets as second-class citizens. Since Moodle would not work 'out of the box' without numerous adjustments. We are looking for a total solution, well-integrated in to all aspects of the base code, but which would not impact on the performance of those who do not require these special features.

MoodleJapan (MAJ) is about to send out a request for bids for a feasibility study. Once the study is completed, it will be submitted to Moodle HQ for comments and approval, after which MAJ and other interested parties will attempt to fund the coding.

Thomas Robb
added a comment - 03/Nov/12 4:55 PM Hello Adrian. Actually, we are not interested in a "patch" since that would only be a stop-gap solution. We need a solution that quickly allows the sys admin to determine which, of many name fields – including the idnumber – to appear in all relevant display with full capability to sort on them, as can be done with the standard two name fields. Also politically, it renders non-Western alphabets as second-class citizens. Since Moodle would not work 'out of the box' without numerous adjustments. We are looking for a total solution, well-integrated in to all aspects of the base code, but which would not impact on the performance of those who do not require these special features.
MoodleJapan (MAJ) is about to send out a request for bids for a feasibility study. Once the study is completed, it will be submitted to Moodle HQ for comments and approval, after which MAJ and other interested parties will attempt to fund the coding.

I have now created a new version with suggestions made by various people. I've created a couple of screen shots and provided the code.
As this is a proof of concept model, the extra name fields only show up on the participants page. If we decided to go ahead with this solution then changes will have to be made in all areas of code where we want to add these fields.

(In my haste of posting up the screen shots I accidentally added the first and last name the wrong way around.)

Adrian Greeve
added a comment - 07/Dec/12 11:24 AM - edited I have now created a new version with suggestions made by various people. I've created a couple of screen shots and provided the code.
As this is a proof of concept model, the extra name fields only show up on the participants page. If we decided to go ahead with this solution then changes will have to be made in all areas of code where we want to add these fields.
(In my haste of posting up the screen shots I accidentally added the first and last name the wrong way around.)

Hi Adrian,
Thanks for the screen shots. It is exciting to see some movement on this issue. One important field missing from the first screenshot(#873) is the Student ID column. In Japan, and likely other countries, 90% of our reports, name lists, assignments, etc. need to be in order of student ID. All attendance is taken in this order, and grades are listed by this student ID. As Tom Robb mentioned in the issue description above, could you add a student ID column to the left of the firstname/surname column? It actually is the most important of all user identifier fields.

Don Hinkelman
added a comment - 07/Dec/12 2:17 PM Hi Adrian,
Thanks for the screen shots. It is exciting to see some movement on this issue. One important field missing from the first screenshot(#873) is the Student ID column. In Japan, and likely other countries, 90% of our reports, name lists, assignments, etc. need to be in order of student ID. All attendance is taken in this order, and grades are listed by this student ID. As Tom Robb mentioned in the issue description above, could you add a student ID column to the left of the firstname/surname column? It actually is the most important of all user identifier fields.

MoodleJapan requested bids for a preliminary study of how "alternate names" could be added to the Moodle core code in the most efficient manner. "Version 2" in Sapporo was then charged with the study and they have come through with the following report.

We hope that Moodle HQ will carefully consider their plan, and if it looks feasible, we are prepared to discuss the coding process – particularly "who" and "how much"! MoodleJapan does not have sufficient funds to pay for the coding on its own, but it might be feasible if Moodle HQ were willing for forgo a sufficient part of the amount that we normally donate from the proceeds of the annual Moot to this cause. We estimate the that cost will come to somewhere between $10,000 and $16,000 USD.

I have attached their report in three separate uploads. They also provided an estimate for the total cost of the development work which comes to approximately $16,000USD.

Thomas Robb
added a comment - 10/Feb/13 1:41 PM MoodleJapan requested bids for a preliminary study of how "alternate names" could be added to the Moodle core code in the most efficient manner. "Version 2" in Sapporo was then charged with the study and they have come through with the following report.
We hope that Moodle HQ will carefully consider their plan, and if it looks feasible, we are prepared to discuss the coding process – particularly "who" and "how much"! MoodleJapan does not have sufficient funds to pay for the coding on its own, but it might be feasible if Moodle HQ were willing for forgo a sufficient part of the amount that we normally donate from the proceeds of the annual Moot to this cause. We estimate the that cost will come to somewhere between $10,000 and $16,000 USD.
I have attached their report in three separate uploads. They also provided an estimate for the total cost of the development work which comes to approximately $16,000USD.

I have studied the plan that you have submitted. And I have a good understanding of what you are trying to achieve. In general the concept is plausible and would work, but the addition of new functions and sql queries makes the current system more complex and difficult to extend. Ideally, the user retrieval code needs to be re-written and centralized so that there is no need to alter code in so many areas. An MDL has already been raised for this improvement (MDL-37047).

Some of the code examples look to implement the use of the student id number. This has already been implemented with MDL-26647.

Martin is basically the gate-keeper for what does and does not get included in core code. I have added him as a watcher so that he might make a comment on this issue. I think that Martin would be the best person to consult with about getting this project completed.

Adrian Greeve
added a comment - 12/Feb/13 3:24 AM Hello Thomas,
I have studied the plan that you have submitted. And I have a good understanding of what you are trying to achieve. In general the concept is plausible and would work, but the addition of new functions and sql queries makes the current system more complex and difficult to extend. Ideally, the user retrieval code needs to be re-written and centralized so that there is no need to alter code in so many areas. An MDL has already been raised for this improvement ( MDL-37047 ).
Some of the code examples look to implement the use of the student id number. This has already been implemented with MDL-26647 .
Martin is basically the gate-keeper for what does and does not get included in core code. I have added him as a watcher so that he might make a comment on this issue. I think that Martin would be the best person to consult with about getting this project completed.

Thank you for your response. We also believe that the plan "can work" but we are not tied to the suggested approach IF there is another, more feasible method that achieves the same result.

We, however, are not looking at the issue from only the programming standpoint, but from two other viewpoints, 1) the matter of functionality of the program in Asian contexts and 2) a subtle ethical issue of the current version being discriminatory against the Non-Western world and culture.

Let me elaborate on the second issue first. When Moodle was in its infancy people were happy to use it as long as it worked. Now, however, it is a mature product that should be marketable in any major country in the world. We are past the point where, in good conscience, we can say to the majority of the population of the world, "Here it is, we've made it for our own needs, but if you want to use it, you've got to either use it 'as is' or jump through your own set of hoops to make it work for you" – which could perhaps be termed "computer code imperialism". We need to face the fact that most people in the world do not have a unique "first name" and "last name" in the English sense of the terms.

Now, on the first issue, we in MoodleJapan are willing to work with MoodleHQ to add this additional functionality. We are willing to sending someone to Perth to talk over the nitty gritty face-to-face or simply to help fund the development elsewhere. As stated in my previous message, one source of funding coud be part of the funds that we annually donate to MoodleHQ from the Moot proceeds.

Concerning the matter of the idnumber field, I have posted a separate tracker issue concerning this (MDL-38061) since it does not actually work in the new Assignment module. Furthermore, the idnumber should be one of the fields that can be selected to appear in the left column of reports as the primary sortable field.

Basically, it is no longer a technical issue so much as one of priorities. We hope that you can consider pushing this issue to the top of the list.

Thomas Robb
added a comment - 20/Feb/13 2:38 AM - edited Hello Martin, Adrian, Michael and all
Thank you for your response. We also believe that the plan "can work" but we are not tied to the suggested approach IF there is another, more feasible method that achieves the same result.
We, however, are not looking at the issue from only the programming standpoint, but from two other viewpoints, 1) the matter of functionality of the program in Asian contexts and 2) a subtle ethical issue of the current version being discriminatory against the Non-Western world and culture.
Let me elaborate on the second issue first. When Moodle was in its infancy people were happy to use it as long as it worked. Now, however, it is a mature product that should be marketable in any major country in the world. We are past the point where, in good conscience, we can say to the majority of the population of the world, "Here it is, we've made it for our own needs, but if you want to use it, you've got to either use it 'as is' or jump through your own set of hoops to make it work for you" – which could perhaps be termed "computer code imperialism". We need to face the fact that most people in the world do not have a unique "first name" and "last name" in the English sense of the terms.
Now, on the first issue, we in MoodleJapan are willing to work with MoodleHQ to add this additional functionality. We are willing to sending someone to Perth to talk over the nitty gritty face-to-face or simply to help fund the development elsewhere. As stated in my previous message, one source of funding coud be part of the funds that we annually donate to MoodleHQ from the Moot proceeds.
Concerning the matter of the idnumber field, I have posted a separate tracker issue concerning this ( MDL-38061 ) since it does not actually work in the new Assignment module. Furthermore, the idnumber should be one of the fields that can be selected to appear in the left column of reports as the primary sortable field.
Basically, it is no longer a technical issue so much as one of priorities. We hope that you can consider pushing this issue to the top of the list.
Cheers,
Tom

Thanks for continuing to support this issue. It's good to have vocal supporters behind such initiatives to push them along. If we've caused any frustration through our lack of progress, I apologise.

We recognise the importance of this functionality. It is usually our goal to aim for an ideal solution, rather than a less favourable one, but perhaps an ideal solution is not immediately feasible and a compromise may be acceptable for the present time. This is not so much a matter of resources or money, but of fitting in with the constraints of the current Moodle architecture.

We will have a meeting here next week, when Martin returns, to discuss the various options that we can pursue now and into the future.

Michael de Raadt
added a comment - 20/Feb/13 3:10 AM Hi, Thom.
Thanks for continuing to support this issue. It's good to have vocal supporters behind such initiatives to push them along. If we've caused any frustration through our lack of progress, I apologise.
We recognise the importance of this functionality. It is usually our goal to aim for an ideal solution, rather than a less favourable one, but perhaps an ideal solution is not immediately feasible and a compromise may be acceptable for the present time. This is not so much a matter of resources or money, but of fitting in with the constraints of the current Moodle architecture.
We will have a meeting here next week, when Martin returns, to discuss the various options that we can pursue now and into the future.

We met today and hashed out the issues. I guess it's been complicated over the years with a variety of approaches and also the fact that there are some other very similar requirements around that are not quite covered by the Asian languages use cases. I hope we can resolve it soon.

In particular we need to agree on exactly what new fields we will add to the user table. We need to make sure they truly covers all the use cases, because it's not going to be as flexible as using custom name fields, so I'd like to then get some feedback from those in various countries to make sure it's OK before we start.

My starting guess is:

firstnamephon

lastnamephon

alternatename (useful for three-name countries too?)

aliasname (for handles and anonymity)

Assuming that is all OK, I think the implementation of the main stuff would be pretty quick and safe, Adrian can easily do it. After that there will be a lot of "mopping up" to improve the display of names in various smaller pages around Moodle or to improve the efficiency with better queries - it would be nice to get help with that from the community, since they have a better idea of exactly what they want to see there.

I have one main question ... with the phonetic fields, will sorting these by UTF actually work? We rely on the database to do sorting and I'm not sure it's enough for what you want. If so then awesome, but if not then please let's think this through, because Asian language sorting is a pretty big kettle of fish and perhaps could be considered separately to this.

Martin Dougiamas
added a comment - 27/Feb/13 7:12 AM Hi guys,
We met today and hashed out the issues. I guess it's been complicated over the years with a variety of approaches and also the fact that there are some other very similar requirements around that are not quite covered by the Asian languages use cases. I hope we can resolve it soon.
I've jotted a summary of a solution at http://docs.moodle.org/dev/Alternate_name_fields and Adrian is going to flesh that out with more details.
In particular we need to agree on exactly what new fields we will add to the user table. We need to make sure they truly covers all the use cases, because it's not going to be as flexible as using custom name fields, so I'd like to then get some feedback from those in various countries to make sure it's OK before we start.
My starting guess is:
firstnamephon
lastnamephon
alternatename (useful for three-name countries too?)
aliasname (for handles and anonymity)
Assuming that is all OK, I think the implementation of the main stuff would be pretty quick and safe, Adrian can easily do it. After that there will be a lot of "mopping up" to improve the display of names in various smaller pages around Moodle or to improve the efficiency with better queries - it would be nice to get help with that from the community, since they have a better idea of exactly what they want to see there.
I have one main question ... with the phonetic fields, will sorting these by UTF actually work? We rely on the database to do sorting and I'm not sure it's enough for what you want. If so then awesome, but if not then please let's think this through, because Asian language sorting is a pretty big kettle of fish and perhaps could be considered separately to this.
Cheers,
Martin

"I have one main question ... with the phonetic fields, will sorting these by UTF actually work? We rely on the database to do sorting and I'm not sure it's enough for what you want. If so then awesome, but if not then please let's think this through, because Asian language sorting is a pretty big kettle of fish and perhaps could be considered separately to this."

Response:

There is a problem with the sortability of the various alphabets. Many alphabetical scripts do have a sorting order that is different from UTF-8. As Martin hints, this requires a separate algorithm to be implemented in order to get them into the standard order. In Thai and Indic languages, for example, a short "i" precedes that consonant that it phonetically follows so without and sorting rules, it would come out in the wrong order. For example a word pronounced "adi" is actually written as "aid". Also all of the sounds/glyphs are in the same UTF-8 order regardless of the Indic language, but the various languages have some differences in alphabetization.

In Korean, North and South Korea have different alphabetization standards. and the common UTF-8 encoding doesn't sort properly for either.

Nevertheless, even without an encoding to sort into the correct order, the result of a sort on the UTF-8 codes will come out in a predictable order, just not the standard one and would be usable until a sorting algorithm is implemented for that language.

In the ideal world, there would be appropriate algorithms to create a code string in a separate field which would be referred to when sorting is requested. The sort field would be updated each time the relevant phonetic field is updated. (Naturally, if someone manually modifies the phonetic field in the database, the sort field would remain unaffected.)

Actually, even in English we sometimes have a need for such a field. Book titles, for example, often start with "The" which many applications wish to be ignored when displaying a list of book titles. This is accomplished by having a separate sorting field. This the title field has "The Adventures of Tom Sawyer" but the alpha field has ""Adventures of Tom Sawyer, The".

I would suggest that a new function sortlang() be built that would take 1) a character string and 2) a variable representing the desired sorting algorithm and then return the desired sort string, which can then be input into the designated sort field paired with the original character string.

The actual rules for how to create a sort string for any given language are not so difficult to code. I've personally constructed scripts to sort Japanese kana and Khmer into their respective standard alphabetical orders. Surely such algorithms for many languages already exist on the Internet and can be compiled into a workable set of sort algorithms. The individuals in charge of the various Moodle language pack maintainers should be the first people to be queried concerning their respective sorting needs.

In addition to applying the sortlang() function to the alternate name fields, we might also consider implementing them in the database module, where the user can designate a specific field to hold the sort field and designate the original field as a trigger: Anytime the original field is modified, the sort field is also populated with the corresponding data.

Thomas Robb
added a comment - 28/Feb/13 12:49 AM Martin asks:
"I have one main question ... with the phonetic fields, will sorting these by UTF actually work? We rely on the database to do sorting and I'm not sure it's enough for what you want. If so then awesome, but if not then please let's think this through, because Asian language sorting is a pretty big kettle of fish and perhaps could be considered separately to this."
Response:
There is a problem with the sortability of the various alphabets. Many alphabetical scripts do have a sorting order that is different from UTF-8. As Martin hints, this requires a separate algorithm to be implemented in order to get them into the standard order. In Thai and Indic languages, for example, a short "i" precedes that consonant that it phonetically follows so without and sorting rules, it would come out in the wrong order. For example a word pronounced "adi" is actually written as "aid". Also all of the sounds/glyphs are in the same UTF-8 order regardless of the Indic language, but the various languages have some differences in alphabetization.
In Korean, North and South Korea have different alphabetization standards. and the common UTF-8 encoding doesn't sort properly for either.
Nevertheless, even without an encoding to sort into the correct order, the result of a sort on the UTF-8 codes will come out in a predictable order, just not the standard one and would be usable until a sorting algorithm is implemented for that language.
In the ideal world, there would be appropriate algorithms to create a code string in a separate field which would be referred to when sorting is requested. The sort field would be updated each time the relevant phonetic field is updated. (Naturally, if someone manually modifies the phonetic field in the database, the sort field would remain unaffected.)
Actually, even in English we sometimes have a need for such a field. Book titles, for example, often start with "The" which many applications wish to be ignored when displaying a list of book titles. This is accomplished by having a separate sorting field. This the title field has "The Adventures of Tom Sawyer" but the alpha field has ""Adventures of Tom Sawyer, The".
I would suggest that a new function sortlang() be built that would take 1) a character string and 2) a variable representing the desired sorting algorithm and then return the desired sort string, which can then be input into the designated sort field paired with the original character string.
The actual rules for how to create a sort string for any given language are not so difficult to code. I've personally constructed scripts to sort Japanese kana and Khmer into their respective standard alphabetical orders. Surely such algorithms for many languages already exist on the Internet and can be compiled into a workable set of sort algorithms. The individuals in charge of the various Moodle language pack maintainers should be the first people to be queried concerning their respective sorting needs.
In addition to applying the sortlang() function to the alternate name fields, we might also consider implementing them in the database module, where the user can designate a specific field to hold the sort field and designate the original field as a trigger: Anytime the original field is modified, the sort field is also populated with the corresponding data.

Presumably even if you have a custom solution on the PHP side you're still going to need the database to be able to sort the right way for when you need to display things in pages. And since this stuff can be a bit complicated, it feels like a good problem to punt off to a database vendor...

PS. I guess you guys have already discussed this and have good reasons for doing it this way, but it does strike me that the long-term sort problem on this scheme is going to be sorting by the names in the "alternate" field, which is only hard because we've squidged two bits of data (first and last name) into one box...

Edmund Edgar
added a comment - 28/Feb/13 6:57 AM Thomas, have you looked at the various Unicode database sort collations which you can tell your database to use when sorting? If so, would you say they're not up to the task?
http://collation-charts.org/mysql60/by-charset.shtml#utf8
Presumably even if you have a custom solution on the PHP side you're still going to need the database to be able to sort the right way for when you need to display things in pages. And since this stuff can be a bit complicated, it feels like a good problem to punt off to a database vendor...
PS. I guess you guys have already discussed this and have good reasons for doing it this way, but it does strike me that the long-term sort problem on this scheme is going to be sorting by the names in the "alternate" field, which is only hard because we've squidged two bits of data (first and last name) into one box...

Thomas, sorting in PHP isn't hard but it can be painfully inefficient when listing, say, 100,000 users in a paged format and someone wants to show page 23. This means you actually need to pull everything from the database, sort it in PHP with the custom sort and then throw most of it away.

In SQL of course this is a trivial and quick operation using LIMIT.

Obviously a database way of doing this would be preferred, but if it's not possible then it's just not possible and those languages that won't work with UTF need to identified as such and done using the slower PHP way.

Edmund, yes the collations were what I was driving at ... though not sure how well they work, and how they would work on a user table, say, with mixed language data.

Martin Dougiamas
added a comment - 28/Feb/13 7:11 AM - edited Thomas, sorting in PHP isn't hard but it can be painfully inefficient when listing, say, 100,000 users in a paged format and someone wants to show page 23. This means you actually need to pull everything from the database, sort it in PHP with the custom sort and then throw most of it away.
In SQL of course this is a trivial and quick operation using LIMIT.
Obviously a database way of doing this would be preferred, but if it's not possible then it's just not possible and those languages that won't work with UTF need to identified as such and done using the slower PHP way.
Edmund, yes the collations were what I was driving at ... though not sure how well they work, and how they would work on a user table, say, with mixed language data.

First to respond to Edmund, the collation sequences will not do the job. All they do is to order the characters and indicate which ones should be considered as equivalent for sorting purposes. If characters are in reverse order from how they should be sorted, as in my earlier "aid/adi" example, then the string has to be pre-processed. Another example is the long vowel sign used in Japanese katakana. There is a single symbol that is used, but the value it takes for pronunciation and sorting, is the value of the preceding vowel. Thus, you can have words like "ka-" and "ki-" that need to be alphabetised as "kaa" and "kii", respectively.

As Martin points out, sorting like this can be very cpu intensive, thus the best solution would be to have a sorting key (aka collation key) in the database which is updated anytime the corresponding display field is updated. When a sorting algorithm isn't available, then the only choice is to use the appropriate collation sequence for alphabetisation.

Thomas Robb
added a comment - 01/Mar/13 1:13 AM First to respond to Edmund, the collation sequences will not do the job. All they do is to order the characters and indicate which ones should be considered as equivalent for sorting purposes. If characters are in reverse order from how they should be sorted, as in my earlier "aid/adi" example, then the string has to be pre-processed. Another example is the long vowel sign used in Japanese katakana. There is a single symbol that is used, but the value it takes for pronunciation and sorting, is the value of the preceding vowel. Thus, you can have words like "ka-" and "ki-" that need to be alphabetised as "kaa" and "kii", respectively.
As Martin points out, sorting like this can be very cpu intensive, thus the best solution would be to have a sorting key (aka collation key) in the database which is updated anytime the corresponding display field is updated. When a sorting algorithm isn't available, then the only choice is to use the appropriate collation sequence for alphabetisation.

Would everyone be OK if we kept sorting out of scope for this issue (and made a new issue for it), in order to help this one have a better chance for 2.5? The more I think about it the more my head hurts. eg multiple tables, multiple fields per table, multiple languages per field, database update madness when inserting new entries, clumping display into "alphabet" groups like A-Z for euro languages etc ...

Martin Dougiamas
added a comment - 01/Mar/13 2:43 AM - edited Would everyone be OK if we kept sorting out of scope for this issue (and made a new issue for it), in order to help this one have a better chance for 2.5? The more I think about it the more my head hurts. eg multiple tables, multiple fields per table, multiple languages per field, database update madness when inserting new entries, clumping display into "alphabet" groups like A-Z for euro languages etc ...

I don't want to speak for Tom, but I think there is a stronger desire to get the alternate name fields than the sorting. Tom did suggest a strategy for allowing sorting in a second language that sounded good, but it will be quite complex and will need a bit more threshing out before it could even start to be considered for implementation. I suggest Tom starts a second issue, linking it to this issue, to cover the sorting aspect.

We also discussed the number of fields needed. It could be argued that the alternate name should have first and last name components. It would be good to get some further discussion on that before we finalise changes. We don't want to have to do this again.

I also raised the default for the capability to allow teachers to edit name formats. I had one vote for allowing teachers to edit name formats in courses and activities by default and two votes against (not including my vote against). I think the security considerations associated with allowing teachers to potentially override a site-wide setting, potentially revealing personal information, should cause us to set a default of not allowed with an admin able to allow a teacher to have this capability. But there may be some people with strong reasons for leaving this more open.

Michael de Raadt
added a comment - 01/Mar/13 2:04 PM We had a good discussion about this over dinner tonight.
I don't want to speak for Tom, but I think there is a stronger desire to get the alternate name fields than the sorting. Tom did suggest a strategy for allowing sorting in a second language that sounded good, but it will be quite complex and will need a bit more threshing out before it could even start to be considered for implementation. I suggest Tom starts a second issue, linking it to this issue, to cover the sorting aspect.
We also discussed the number of fields needed. It could be argued that the alternate name should have first and last name components. It would be good to get some further discussion on that before we finalise changes. We don't want to have to do this again.
I also raised the default for the capability to allow teachers to edit name formats. I had one vote for allowing teachers to edit name formats in courses and activities by default and two votes against (not including my vote against). I think the security considerations associated with allowing teachers to potentially override a site-wide setting, potentially revealing personal information, should cause us to set a default of not allowed with an admin able to allow a teacher to have this capability. But there may be some people with strong reasons for leaving this more open.

IMO, the really important consideration is to support languages with more than two names. Spanish and Arabic being obviously huge ones. I'll direct some Spanish and Arabic people here to look at the spec and comment on suitability for them.

Martin Dougiamas
added a comment - 06/Mar/13 6:46 AM IMO, the really important consideration is to support languages with more than two names. Spanish and Arabic being obviously huge ones. I'll direct some Spanish and Arabic people here to look at the spec and comment on suitability for them.

David Monllao (one of our Spanish developers) is of the opinion that in Spanish they are quite used to putting both surnames in the one field. So no problem there probably, but it would be good to more confirmation of this.

Also still need some feedback from an Arabic perspective. I'll ask Muhammad ibn Saeed ibn Abd al-Aziz al-Filasteeni about it.

Martin Dougiamas
added a comment - 06/Mar/13 7:04 AM David Monllao (one of our Spanish developers) is of the opinion that in Spanish they are quite used to putting both surnames in the one field. So no problem there probably, but it would be good to more confirmation of this.
Also still need some feedback from an Arabic perspective. I'll ask Muhammad ibn Saeed ibn Abd al-Aziz al-Filasteeni about it.

and that can become more complex if we put the second surname in action:

Eloy la Fuente de la Plaza

for sorting, could be:

Eloy Fuente, la Plaza, de la

But, in practice... not many software allows that elaborated processing of particles/articles and peoples use to manually:

1) Ignore the particles thing completely.
2) If it's required for some legal cause, then the surnames are introduced in the order they need to be sorted.

So I'll conclude that one unique field for any pair of surnames (current way) is 95% ok. Although if available, it would be correct to have a place to those articles/particles and/or to separate surnames.. but surely as something to define as optional settings. Current behavior is perfectly good as default.

Eloy Lafuente (stronk7)
added a comment - 06/Mar/13 9:06 AM - edited Hi,
in Spanish we certainly have 2^n surnames (accumulated from all our ancestors), although only the 1st pair is used legally.
And they have an official order (dad 1st and then mum, by default, or the opposite, each person can decide about that legally ). But once decided, the combination becomes de official one.
So for all meanings, to have both together in an unique surname field is enough, agree with David.
BUT, there is one little details where I see Moodle lacking other software and it's that, often, we ignore particles/articles for sorting (and displaying!) so for example:
Michael de Raadt
should be sorted as:
Michael Raadt, de
Or more complex, you can find people named:
Eloy Lafuente (my case, my 1st surname is "Lafuente" all-together)
Eloy la Fuente
Eloy de Lafuente
Eloy de la Fuente
that, for sorting purposes (and sometimes display too), should be:
Eloy Lafuente
Eloy Fuente, la
Eloy Lafuente, de
Eloy Fuente, de la
and that can become more complex if we put the second surname in action:
Eloy la Fuente de la Plaza
for sorting, could be:
Eloy Fuente, la Plaza, de la
But, in practice... not many software allows that elaborated processing of particles/articles and peoples use to manually:
1) Ignore the particles thing completely.
2) If it's required for some legal cause, then the surnames are introduced in the order they need to be sorted.
So I'll conclude that one unique field for any pair of surnames (current way) is 95% ok. Although if available, it would be correct to have a place to those articles/particles and/or to separate surnames.. but surely as something to define as optional settings. Current behavior is perfectly good as default.
Ciao

One more point... note that, also, that the whole articles thing applies, generally, only to sorting in lists (and optionally displaying there), I mean, when one name is shown individually (post author...), then the "real" name is the preferred name to be displayed ("Michael de Raadt"). Displaying it as "Michael Raadt, de" is somehow ugly.

Eloy Lafuente (stronk7)
added a comment - 06/Mar/13 9:15 AM One more point... note that, also, that the whole articles thing applies, generally, only to sorting in lists (and optionally displaying there), I mean, when one name is shown individually (post author...), then the "real" name is the preferred name to be displayed ("Michael de Raadt"). Displaying it as "Michael Raadt, de" is somehow ugly.

About sorting again... I found out that the Indic languages DO sort properly. Even though, as I said, the short vowel "i" precedes the consonant that it should follow, it is actually coded in the proper position for pronunciation, but displayed in front of the consonant when typed or printed. Thus there is no sorting problem to worry about. In Dutch, however, one Moodle site that I am familiar with places all of those surname prefixes after the main part of the name: firstname: Michael lastname: Raadt de; firstname; Vincent lastname: Gogh van

For these, an expanded set of name fields would allow them to place the prefixes in a separate field so that they could be designated to appear before the main surname on screen displays but allowing the main surname to be used for sorting purposes. One kludge that can be used with two field names is to place the prefix at the end of the firstname, thus: firstname: Michael de lastname: Raadt; ; firstname; Vincent van lastname: Gogh

Thomas Robb
added a comment - 07/Mar/13 5:44 AM About sorting again... I found out that the Indic languages DO sort properly. Even though, as I said, the short vowel "i" precedes the consonant that it should follow, it is actually coded in the proper position for pronunciation, but displayed in front of the consonant when typed or printed. Thus there is no sorting problem to worry about. In Dutch, however, one Moodle site that I am familiar with places all of those surname prefixes after the main part of the name: firstname: Michael lastname: Raadt de; firstname; Vincent lastname: Gogh van
For these, an expanded set of name fields would allow them to place the prefixes in a separate field so that they could be designated to appear before the main surname on screen displays but allowing the main surname to be used for sorting purposes. One kludge that can be used with two field names is to place the prefix at the end of the firstname, thus: firstname: Michael de lastname: Raadt; ; firstname; Vincent van lastname: Gogh

My feeling is that perhaps sorting should still be left to a follow-up issue.

Regarding Arabic, I got one response supporting middlenames.

In fact the existing Alternate name field could be used to support this without too much trouble. You'd just need to define the token string to be "firstname alternatename lastname" and probably also update the language pack to make it display nice in the user profile etc. it's not totally pretty but it would work. (XA)

Otherwise we could just add a "middlename" field explicitly and make it simple. (XB)

Martin Dougiamas
added a comment - 08/Mar/13 4:40 AM My feeling is that perhaps sorting should still be left to a follow-up issue.
Regarding Arabic, I got one response supporting middlenames.
In fact the existing Alternate name field could be used to support this without too much trouble. You'd just need to define the token string to be "firstname alternatename lastname" and probably also update the language pack to make it display nice in the user profile etc. it's not totally pretty but it would work. (XA)
Otherwise we could just add a "middlename" field explicitly and make it simple. (XB)
XA or XB?

Sorry, but (XB) is not acceptable. Japanese, Chinese and Koreans do NOT have middle names. Please name the field as something that is not so ethnocentric. If there are three fields that can be "mixed and matched" in various displays, it would be an improvement on the current system. Note that there are two somewhat different issues here. One is how $fullname is formed and displayed, and the other is how each of these fields can be displayed in a selectable order (by the site admin, at least) in various reports. To take Japan as the example here, Japanese may want to sort on the IDnumber since these are typically assigned in the order of the Kana syllabary. Expats, on the other hand, would wan to sort either by Surname-Given name or Given-Surname, depending on what name they use when they call on the students in class. Will this be do-able with either the (XA) or (XB) proposal?

Thomas Robb
added a comment - 18/Mar/13 9:13 PM Sorry, but (XB) is not acceptable. Japanese, Chinese and Koreans do NOT have middle names. Please name the field as something that is not so ethnocentric. If there are three fields that can be "mixed and matched" in various displays, it would be an improvement on the current system. Note that there are two somewhat different issues here. One is how $fullname is formed and displayed, and the other is how each of these fields can be displayed in a selectable order (by the site admin, at least) in various reports. To take Japan as the example here, Japanese may want to sort on the IDnumber since these are typically assigned in the order of the Kana syllabary. Expats, on the other hand, would wan to sort either by Surname-Given name or Given-Surname, depending on what name they use when they call on the students in class. Will this be do-able with either the (XA) or (XB) proposal?

I don't think that having the field there mandates that people use it; in the same way we don't require people to use the phonetic or alternate name fields. The point is we're trying not to be ethnocentric by including all these fields. The only further step we could take would be to give generic titles to all the fields like "Alternate name 1", but I'm not sure that is wise.

People can already sort by ID number if that field is displayed and data is filled in it. I don't think we need to mix that with the name fields.

Michael de Raadt
added a comment - 19/Mar/13 1:58 AM Hi, Tom.
I don't think that having the field there mandates that people use it; in the same way we don't require people to use the phonetic or alternate name fields. The point is we're trying not to be ethnocentric by including all these fields. The only further step we could take would be to give generic titles to all the fields like "Alternate name 1", but I'm not sure that is wise.
People can already sort by ID number if that field is displayed and data is filled in it. I don't think we need to mix that with the name fields.

I agree with Michael. The names that we give the fields is simply only a suggestion as to how it could be used. If anyone feels strongly about the field name then maybe they can ask who ever does the translation for their specific language to label it something different. Naming the fields with generic terms I also think is not a good idea. If we take making the fields non 'ethnocentric' then where do we stop. Most Indonesian people do not have a last name, first name no longer seems like an appropriate term. We would have to rename all name fields to something generic.

Adrian Greeve
added a comment - 21/Mar/13 12:42 AM - edited I agree with Michael. The names that we give the fields is simply only a suggestion as to how it could be used. If anyone feels strongly about the field name then maybe they can ask who ever does the translation for their specific language to label it something different. Naming the fields with generic terms I also think is not a good idea. If we take making the fields non 'ethnocentric' then where do we stop. Most Indonesian people do not have a last name, first name no longer seems like an appropriate term. We would have to rename all name fields to something generic.
+1 for XB

I hope all this is going to be something only available if the admin opts-in.

And it won't be shown/used/listed at all by default, so people will continue using the current & simpler way (first+second and done), correct?

(crossing fingers...)

My POV is that 1% of sites requiring such phonetic/romanization/sorting detail (and I think I'm exaggerating largely) cannot cause the other 99% to get complex-er/slower display/sorting criteria by default. And it's going to be complex-er/slower (I bet).

Eloy Lafuente (stronk7)
added a comment - 25/Mar/13 5:29 PM I hope all this is going to be something only available if the admin opts-in.
And it won't be shown/used/listed at all by default, so people will continue using the current & simpler way (first+second and done), correct?
(crossing fingers...)
My POV is that 1% of sites requiring such phonetic/romanization/sorting detail (and I think I'm exaggerating largely) cannot cause the other 99% to get complex-er/slower display/sorting criteria by default. And it's going to be complex-er/slower (I bet).
Just my thoughts... ciao

Ahem, Eloy. Check the world population figures. Asia composes slightly more than 1% of the population. Right now Moodle is a fringe LMS in Asia mostly relegated to use by Western foreigners (me, for example). If we want to continue as a Western/Anglo/Roman LMS then the 1% figure may be correct. If Moodle is to be adopted by Asian institutions (China, India, etc.), then 60% will be a closer figure. 100% of the sites in Japan will use this feature.

Don Hinkelman
added a comment - 25/Mar/13 11:38 PM Ahem, Eloy. Check the world population figures. Asia composes slightly more than 1% of the population. Right now Moodle is a fringe LMS in Asia mostly relegated to use by Western foreigners (me, for example). If we want to continue as a Western/Anglo/Roman LMS then the 1% figure may be correct. If Moodle is to be adopted by Asian institutions (China, India, etc.), then 60% will be a closer figure. 100% of the sites in Japan will use this feature.

Hmm, translators may decide to name the user fields in whatever way they want. In my computer it is me who decides what is the appropriate order. I would expect Moodle to allow me to configure it with some reasonable defaults based on my language or global admin settings.

1/ add some more general name fields - display name, middle, nickname, name3, name4 or whatever else (all of these would have to be included in all user related query results - we already have API for required avatar fields, we could use that)
2/ let translators decide how these fields are called in each language (admins may override it)
3/ let translators decide what is the expected default name format in each language (new string)
4/ let administrators decide what is the default and what are other options how to show names
5/ let users select own name format preference if they do not like default
6/ collation support in databases is an inconsistent mess, we need to add support in DML layer first, for now there is only one global collation

Performance is the main problem here, because names are printed very often, it should imo go into the user table directly, text fields are definitely not acceptable.

Petr Skoda
added a comment - 26/Mar/13 12:22 AM Hmm, translators may decide to name the user fields in whatever way they want. In my computer it is me who decides what is the appropriate order. I would expect Moodle to allow me to configure it with some reasonable defaults based on my language or global admin settings.
1/ add some more general name fields - display name, middle, nickname, name3, name4 or whatever else (all of these would have to be included in all user related query results - we already have API for required avatar fields, we could use that)
2/ let translators decide how these fields are called in each language (admins may override it)
3/ let translators decide what is the expected default name format in each language (new string)
4/ let administrators decide what is the default and what are other options how to show names
5/ let users select own name format preference if they do not like default
6/ collation support in databases is an inconsistent mess, we need to add support in DML layer first, for now there is only one global collation
Performance is the main problem here, because names are printed very often, it should imo go into the user table directly, text fields are definitely not acceptable.

The current default is to use the format specified in the language pack. That will remain the default unless it is overridden. It will be possible for different language packs to use the new fields, keeping in mind the the fields may have no information.

Michael de Raadt
added a comment - 26/Mar/13 1:19 AM Hi, Eloy.
The current default is to use the format specified in the language pack. That will remain the default unless it is overridden. It will be possible for different language packs to use the new fields, keeping in mind the the fields may have no information.

I'm just saying, make it opt-in, defined by lang and override-able by admin. Just that.

Finally, pay some attention to Petr's wise words above, it will be super-cool to have all those name schemas supported.... but right now we are not able to provide properly collated sorting from database, so this is only going to work for display but not for listings (in an efficient way). And IMO the main point about this is to be used for (better) listings. Display can be achieved with just 1 field (wholename).

Eloy Lafuente (stronk7)
added a comment - 26/Mar/13 8:54 AM - edited Don et all,
I'm not saying about not to implement this.
I'm just saying, make it opt-in, defined by lang and override-able by admin. Just that.
Finally, pay some attention to Petr's wise words above, it will be super-cool to have all those name schemas supported.... but right now we are not able to provide properly collated sorting from database, so this is only going to work for display but not for listings (in an efficient way). And IMO the main point about this is to be used for (better) listings. Display can be achieved with just 1 field (wholename).
Ciao

Martin Dougiamas
added a comment - 26/Mar/13 9:48 AM Note above that sorting and collation was removed from the scope of this.
This is just about alternate name display and I think most of what Petr mentioned has already been covered (implemented).

The linked patch would cause major performance problems, solution would require changes all over the place when fetching user data.

At present we have perf problems in filters caused by tree based settings, this feels like the same problem. In any case course level name formats sound wrong because we base everything on contexts now.

Display name link makes me worried, again new perf problems there - we did not solve linking of avatars properly yet.

I personally do not like the proposed new field names much, it should be imho something more language neutral at the database level. My vote is: middlename, aliasname, altname1 and altname2.

Petr Skoda
added a comment - 26/Mar/13 10:12 AM The linked patch would cause major performance problems, solution would require changes all over the place when fetching user data.
At present we have perf problems in filters caused by tree based settings, this feels like the same problem. In any case course level name formats sound wrong because we base everything on contexts now.
Display name link makes me worried, again new perf problems there - we did not solve linking of avatars properly yet.
I personally do not like the proposed new field names much, it should be imho something more language neutral at the database level. My vote is: middlename, aliasname, altname1 and altname2.

Martin Dougiamas
added a comment - 05/Apr/13 9:06 AM Petr please be specific about the performance problems you see, and help this issue proceed rather than just stopping it.
I don't think the database field names are much of an issue.

Potential performance problems:
1/ all user info must be fetched in original queries, we can not fetch it later in renderers - hardcoding it in queries is not very flexible
2/ adding course/module level settings to user display is a problem similar to filters, luckily usernames are not displayed in navigation; in any case caching still has some costs

These are the showstoppers for me in the current patch:
a/ please no more course/cm level only settings, we have context tree for this since 2.0 - the first bug report would be for block settings
b/ this has to define which fields are required, there is already a bug for languages with one name only
c/ missing $user->displayname which would allow free style user name entry - many other systems have it, it helps with sorting too, it is completely language neutral; this alone could probably resolve 80% of problems; this could be also generated automatically from first+last+other names
d/ I do not like the proposed new fields - see above, there are too many
e/ PARAM_TEXT is not a general text cleaning type - multilang is not allowed in display name format, is it?
f/ some commit messages do not include MDL, others waste space with unnecessary " usability / language : " prefix, invalid structure of git commit messages in general
g/ we did not even finish adding of context to format_(text|string) and we have another incomplete API change, I believe that fullname() should be fully deprecated if we decide this is the way; why is it even implemented in moodlelib as functions? why not only in new renderer?
h/ if the resulting displayname is empty it should fall back to something, we do not want empty ghost names, right?

Petr Skoda
added a comment - 05/Apr/13 10:05 AM - edited Potential performance problems:
1/ all user info must be fetched in original queries, we can not fetch it later in renderers - hardcoding it in queries is not very flexible
2/ adding course/module level settings to user display is a problem similar to filters, luckily usernames are not displayed in navigation; in any case caching still has some costs
These are the showstoppers for me in the current patch:
a/ please no more course/cm level only settings, we have context tree for this since 2.0 - the first bug report would be for block settings
b/ this has to define which fields are required, there is already a bug for languages with one name only
c/ missing $user->displayname which would allow free style user name entry - many other systems have it, it helps with sorting too, it is completely language neutral; this alone could probably resolve 80% of problems; this could be also generated automatically from first+last+other names
d/ I do not like the proposed new fields - see above, there are too many
e/ PARAM_TEXT is not a general text cleaning type - multilang is not allowed in display name format, is it?
f/ some commit messages do not include MDL, others waste space with unnecessary " usability / language : " prefix, invalid structure of git commit messages in general
g/ we did not even finish adding of context to format_(text|string) and we have another incomplete API change, I believe that fullname() should be fully deprecated if we decide this is the way; why is it even implemented in moodlelib as functions? why not only in new renderer?
h/ if the resulting displayname is empty it should fall back to something, we do not want empty ghost names, right?

The main purpose of this is to help people to read and write names of foreigners. Quite often this is the expected form of first+last names on sites with users that are speaking different languages. On community servers where everybody is supposed to understand English it is part of the etiquette to use ISO-8859-1 characters only - Cyrillic, Greek or Chinese alphabets would be very confusing for the majority of people there.

In theory you might romanise to any language based on latin which is probably more similar to phonetic names in different languages. Examples: Russian -> Czech.

==Phonetic names==

Some names are written in alphabets that do not have fixed phonetic forms, even native speakers need to know how to pronounce them because there might be multiple options. Examples: Japanese in Kanji -> Furigana see http://en.wikipedia.org/wiki/Japanese_name

If I understand this correctly the real name is just one and the phonetic form is used on business cards or when meeting new people. This seems to match the intended uses of our custom profile fields, or you can type it in profile description. People that need to know how to read names may easily visit users profile.

-------

The way I understand this patch it tries to resolve mostly phonetic name display which can be easily done via custom profile fields, I believe there is little need to show phonetic names everywhere in Moodle, or sort by phonetic names, sorry. If really necessary it could be worked around by using phonetic forms in standard user->displayname field. So far I did not see any computer system that stores phonetic names of users separately, why do we need to be special here?

Petr Skoda
added a comment - 05/Apr/13 11:17 AM oh! let's not forget user upload and auth systems, we must map external things to our internal storage. Most systems use following fields:
firstname - aka given name
lastname - aka family name
middlenames - there can be one or more separated by space, or just initials
displayname - the full name of user
aliasname
I would like to summarise problems with non-latin languages:
==Romanisation==
Various transcriptions of alphabets, the result is usually ASCII. examples: Russian -> English, Chinese -> English, Japanese -> English, Czech -> English, Greek -> English.
The main purpose of this is to help people to read and write names of foreigners. Quite often this is the expected form of first+last names on sites with users that are speaking different languages. On community servers where everybody is supposed to understand English it is part of the etiquette to use ISO-8859-1 characters only - Cyrillic, Greek or Chinese alphabets would be very confusing for the majority of people there.
In theory you might romanise to any language based on latin which is probably more similar to phonetic names in different languages. Examples: Russian -> Czech.
==Phonetic names==
Some names are written in alphabets that do not have fixed phonetic forms, even native speakers need to know how to pronounce them because there might be multiple options. Examples: Japanese in Kanji -> Furigana see http://en.wikipedia.org/wiki/Japanese_name
If I understand this correctly the real name is just one and the phonetic form is used on business cards or when meeting new people. This seems to match the intended uses of our custom profile fields, or you can type it in profile description. People that need to know how to read names may easily visit users profile.
-------
The way I understand this patch it tries to resolve mostly phonetic name display which can be easily done via custom profile fields, I believe there is little need to show phonetic names everywhere in Moodle, or sort by phonetic names, sorry. If really necessary it could be worked around by using phonetic forms in standard user->displayname field. So far I did not see any computer system that stores phonetic names of users separately, why do we need to be special here?

The issue fetching additional name information was known from the outset. It will come at a cost, particularly in lists of names. We are working to make this more efficient and easier for developers to provide this information, but we will always have to allow for (plugin and core) code that does not provide the additional name fields. So for sites that choose to use additional name fields when displaying names, there will be a possible performance hit.

The issue of mixing contexts is a hairy one. Where a user's name appears on a page, the parts of the page may be generated using different contexts (like navigation, blocks and the header). Adrian was looking at this, but I'm not sure where it was up to.

I think the people who reported this issue and are watching it have a better understanding of the need for names in various languages. I don't think we should do half a job on this.

Michael de Raadt
added a comment - 08/Apr/13 1:56 AM The issue fetching additional name information was known from the outset. It will come at a cost, particularly in lists of names. We are working to make this more efficient and easier for developers to provide this information, but we will always have to allow for (plugin and core) code that does not provide the additional name fields. So for sites that choose to use additional name fields when displaying names, there will be a possible performance hit.
The issue of mixing contexts is a hairy one. Where a user's name appears on a page, the parts of the page may be generated using different contexts (like navigation, blocks and the header). Adrian was looking at this, but I'm not sure where it was up to.
I think the people who reported this issue and are watching it have a better understanding of the need for names in various languages. I don't think we should do half a job on this.

I was thinking about the phonetic names, I think it should be possible to allow extra names via user profiles fields, it would be slower but that would be imho acceptable cost because only sites that want to use custom names would be affected. The name format could include standard {$a->firstname, {$a->lastname, {$a->middlenames} and {$a->aliasname} and any "profile_" prefixed names such as {$a->profile_firstnamephonetic} and {$a->profile_lastnamephonetic}. This way you could even define multiple phonetic systems as requested in the description.

Petr Skoda
added a comment - 12/Apr/13 8:41 AM My -5 for the patch for the reasons above.
I was thinking about the phonetic names, I think it should be possible to allow extra names via user profiles fields, it would be slower but that would be imho acceptable cost because only sites that want to use custom names would be affected. The name format could include standard {$a->firstname, {$a->lastname, {$a->middlenames} and {$a->aliasname} and any "profile_" prefixed names such as {$a->profile_firstnamephonetic} and {$a->profile_lastnamephonetic}. This way you could even define multiple phonetic systems as requested in the description.

Rajesh Taneja
added a comment - 15/Apr/13 1:54 AM Hello Petr,
We have changed the design to make it more efficient.
Advantages of having this in:
Extra user fields are part of user profile field set and if set, will check if user has filled them.
User name display can be controlled at course/ Module level, giving more flexibility.
As format is coming from course/Module, it will not go though old fullname function and will improve performance for current sites.
Anonymous username problem can be sorted with this patch.
Will only affect site who enable alternate user name fields (for non-optimised calls to display_name())

Martin Dougiamas
added a comment - 15/Apr/13 3:57 AM Yes, the main reasons we rejected user profile fields way back in this issue was because:
doing it that way is much slower and more complicated
the use cases in this bug already cover 99% of the possible variations people need
the cost outweighs the benefit

Helen Foster
added a comment - 15/Apr/13 6:28 AM I'm echoing Eloy's comment:
I hope all this is going to be something only available if the admin opts-in.
And it won't be shown/used/listed at all by default, so people will continue using the current & simpler way (first+second and done), correct?
I don't think 'User name display can be controlled at course/ Module level, giving more flexibility' can be considered an advantage unless it is only displayed for sites that want it.

Rajesh Taneja
added a comment - 15/Apr/13 6:35 AM Thanks Helen,
Alternate User fields will only be displayed/checked if they are enabled at site level by admin.
Also, only those fields will be displayed which admin wants.
I am not sure if we took care of show/hide option at course/module level, but sure we can take care of it.

This improvement will rely of a lot of changes through out moodle. If the code goes through integration, I'll be happy to create additional issues to update from the fullname function.
Attached is a grep dump of where the fullname function is used (400+ lines).

Adrian Greeve
added a comment - 17/Apr/13 6:06 AM This improvement will rely of a lot of changes through out moodle. If the code goes through integration, I'll be happy to create additional issues to update from the fullname function.
Attached is a grep dump of where the fullname function is used (400+ lines).

PARAM_TEXT is not correct for displaynameformat (it's not a multilang field)

Need get_string for 'Allow link to the profile page'

displaynameformat missing from backup/restore

new user fields missing from backup/restore

new user fields missing from webservices

typo in $addtionalfieldempty

Not sure if it has been raised - but why would you not want to link the username to the profile?

commented code in lib/navigationlib.php

new displayname function is not checking 'moodle/site:viewfullnames'

difference between username and user_name is a bit hard to read in code

ID number shows up twice (need to edit both)

There are lots of places in the code still calling "fullname()" (grep tells me 434 uses) I think it is important that this is applied consistently in core. (Not looking back at the long list of comments right now - but I am not sure why we haven't made this backwards compatible with the fullname function).

You have changed the tablelib class - but not the get_extra_user_fields - IMO both are required.

We still have $CFG->showuseridentity - IMO it doesn't make sense to have both

Damyon Wiese
added a comment - 17/Apr/13 8:50 AM Hi Adrian,
This is looking good - I took a look at the branch and here is a braindump of big and little things I spotted:
The admin settings are in the wrong section (IMO) they should be in the "User policies" section not the "Site policies" section.
The name of the function "displayname" is too generic - it could be the displayname of a course.
You are not calling displayname correctly - the second parameter is the format but you have:
e.g.
$OUTPUT->displayname($user, $sitecontext);
in lots of places
You need to pass size as an array:
$mform->addElement('text', 'displaynameformat', get_string('displaynameformatcourse'), 'size = 50');
PARAM_TEXT is not correct for displaynameformat (it's not a multilang field)
Need get_string for 'Allow link to the profile page'
displaynameformat missing from backup/restore
new user fields missing from backup/restore
new user fields missing from webservices
typo in $addtionalfieldempty
Not sure if it has been raised - but why would you not want to link the username to the profile?
commented code in lib/navigationlib.php
new displayname function is not checking 'moodle/site:viewfullnames'
difference between username and user_name is a bit hard to read in code
ID number shows up twice (need to edit both)
There are lots of places in the code still calling "fullname()" (grep tells me 434 uses) I think it is important that this is applied consistently in core. (Not looking back at the long list of comments right now - but I am not sure why we haven't made this backwards compatible with the fullname function).
You have changed the tablelib class - but not the get_extra_user_fields - IMO both are required.
We still have $CFG->showuseridentity - IMO it doesn't make sense to have both
Cheers - Damyon

This has got really complex. Adrian, Raj and Damyon have been spending a lot of time on this but it seems to be spawning new problems.

There was talk today in HQ about dropping aliasname and any context support, because this would simplify everything a lot.

The Anonymous feature was my particular wheelbarrow because roleplaying and anonymity were long-standing requests, will touch similar areas, and didn't look too complicated to add, however, even if we did it this way it looks like the interface would still be quite imperfect (one alias per user per site, no teacher control).

A better interface might be some page at course level where the teacher (and optionally students) can decide their aliases and assign people to them. With this method we will still need to modify activities to support this feature but we could probably call a different function than fullname() to print the alias. So it would be quite separate and manageable as a separate project.

So on reflection I'm OK to drop this requirement and go back to just simple site settings that modify how fullname() works for the whole site, no contexts. Basically I see that as:

Martin Dougiamas
added a comment - 23/Apr/13 7:23 AM This has got really complex. Adrian, Raj and Damyon have been spending a lot of time on this but it seems to be spawning new problems.
There was talk today in HQ about dropping aliasname and any context support, because this would simplify everything a lot.
The Anonymous feature was my particular wheelbarrow because roleplaying and anonymity were long-standing requests, will touch similar areas, and didn't look too complicated to add, however, even if we did it this way it looks like the interface would still be quite imperfect (one alias per user per site, no teacher control).
A better interface might be some page at course level where the teacher (and optionally students) can decide their aliases and assign people to them. With this method we will still need to modify activities to support this feature but we could probably call a different function than fullname() to print the alias. So it would be quite separate and manageable as a separate project.
So on reflection I'm OK to drop this requirement and go back to just simple site settings that modify how fullname() works for the whole site, no contexts. Basically I see that as:
extending $CFG->fullnamedisplay to support more user field tokens and intervening filler text (such as brackets). eg "fullname middlename lastname (firstnamephon lastnamephon)"
making it handle missing data intelligently (eg removing empty brackets or excess whitepace from the above example)
checking calls to fullname() to make sure they are efficient and preload extra fields as much as possible
I feel sorry for all the work Adrian especially has done but better for part to land than none ...

get_additional_name_fields (#L15R3665), seems to be redundant, as it's just using hard-coded values. Seems, unnecessary processing. IMHO, it should either return all name related fields or should be removed.

#L15R3679 you can use implode to join array. No need of substring.

Rather then passing two parameters to get_additional_name_fields, can't it be achieved with one ($joinwith)?

#L6R93 In this case you are using get_additional_name_fields and later (#L6R108) you have hard-coded these fields. It might be nice to have some consistency with use of alternate field names.

Similar hard-coding of alternate fields has been done in other places.

#L15R3632 can be cache this, as this is site based and looping though displayname for every displayed name seems redundant and process hungry.

Not sure if string clean-up is required (#L15R3644), as CFG is set by admin, do we bother about cleaning this ?

Typo in debugging message #L15R3623

$mainuserfields (#L29R384), should include username. email is not required as that is returned by user_picture::fields

Rajesh Taneja
added a comment - 17/Jun/13 7:56 AM Patch seems nice Adrian, few things which you might want to consider:
get_additional_name_fields (#L15R3665), seems to be redundant, as it's just using hard-coded values. Seems, unnecessary processing. IMHO, it should either return all name related fields or should be removed.
#L15R3679 you can use implode to join array. No need of substring.
Rather then passing two parameters to get_additional_name_fields, can't it be achieved with one ($joinwith)?
#L6R93 In this case you are using get_additional_name_fields and later (#L6R108) you have hard-coded these fields. It might be nice to have some consistency with use of alternate field names.
Similar hard-coding of alternate fields has been done in other places.
#L15R3632 can be cache this, as this is site based and looping though displayname for every displayed name seems redundant and process hungry.
Not sure if string clean-up is required (#L15R3644), as CFG is set by admin, do we bother about cleaning this ?
Typo in debugging message #L15R3623
$mainuserfields (#L29R384), should include username. email is not required as that is returned by user_picture::fields
$mainuserfields doesn't include alternate name fields.

Thanks for the review.
I have considered the things that you have pointed out and:

I created this function as a central location for the additional name fields. If in the future we wanted to add more, all we would have to do is add them to the array in that function. I definitely don't think that it should be removed all together. Hardcoding the additional name fields everywhere doesn't make sense at all. I can see that having all name fields in the same location would be useful. I'll add firstname and lastname to the array.

I'm not sure by what you mean by joining the array on line 3679. This line is a string and I'm removing the last two characters so that there is no trailing comma.

Get additional name fields has a dual purpose of returning an array and providing an sql snippet. The first parameter is set to "true" if you want the sql snippet and then you can add an additional alias if required. Removing one of the parameters would mean that it couldn't be used in both situations.

You're right this can be improved and has now been fixed.

Same as above.

I don't think that we can cache this. We have to loop through each name and replace it with the users details for each user because each user has a unique name. Perhaps I'm missing what you mean.

This isn't a screen clean up for the admin, this is so we don't display something like "Adrian () Greeve" when the middle name has not been entered. I've added a comment with a similar explanation.

This has been fixed.

This has now been included.

This does actually include the alternate name fields as user_picture::fields() includes alternate name fields.

Adrian Greeve
added a comment - 18/Jun/13 3:34 AM Hello Raj,
Thanks for the review.
I have considered the things that you have pointed out and:
I created this function as a central location for the additional name fields. If in the future we wanted to add more, all we would have to do is add them to the array in that function. I definitely don't think that it should be removed all together. Hardcoding the additional name fields everywhere doesn't make sense at all. I can see that having all name fields in the same location would be useful. I'll add firstname and lastname to the array.
I'm not sure by what you mean by joining the array on line 3679. This line is a string and I'm removing the last two characters so that there is no trailing comma.
Get additional name fields has a dual purpose of returning an array and providing an sql snippet. The first parameter is set to "true" if you want the sql snippet and then you can add an additional alias if required. Removing one of the parameters would mean that it couldn't be used in both situations.
You're right this can be improved and has now been fixed.
Same as above.
I don't think that we can cache this. We have to loop through each name and replace it with the users details for each user because each user has a unique name. Perhaps I'm missing what you mean.
This isn't a screen clean up for the admin, this is so we don't display something like "Adrian () Greeve" when the middle name has not been entered. I've added a comment with a similar explanation.
This has been fixed.
This has now been included.
This does actually include the alternate name fields as user_picture::fields() includes alternate name fields.

I've made some further alterations to the code and done a comparison of a few of the pages around moodle for DB calls.

I have a plan to include caching (MDL-38606) to try and take away more of the processing that is currently done in the fullname function.

comparison

Database calls

file

before

after

home page

60

60

admin/user.php

45

45

admin/user/user_bulk.php

44

44

course/view.php

47

47

enrol/users.php

109

109

grade/report/grader/index.php

66

66

user/index.php

57

57

mod/assign/view.php

186

186

mod/choice/view.php

69

69

As you can see, there is no impact on the database calls made on each of these pages. This is because the sql statements have been updated to add the new fields where necessary. The fullname function itself does not make any database calls.

Plugin developers will need to alter their own sql statements to include these new files. Developers that have user the user_picture::fields function to create their user statement will not have to make any improvements at all.

Rajesh, if you could please have another look at the code and tell me what you think.

Adrian Greeve
added a comment - 21/Jun/13 6:43 AM I've made some further alterations to the code and done a comparison of a few of the pages around moodle for DB calls.
I have a plan to include caching ( MDL-38606 ) to try and take away more of the processing that is currently done in the fullname function.
comparison
Database calls
file
before
after
home page
60
60
admin/user.php
45
45
admin/user/user_bulk.php
44
44
course/view.php
47
47
enrol/users.php
109
109
grade/report/grader/index.php
66
66
user/index.php
57
57
mod/assign/view.php
186
186
mod/choice/view.php
69
69
As you can see, there is no impact on the database calls made on each of these pages. This is because the sql statements have been updated to add the new fields where necessary. The fullname function itself does not make any database calls.
Plugin developers will need to alter their own sql statements to include these new files. Developers that have user the user_picture::fields function to create their user statement will not have to make any improvements at all.
Rajesh, if you could please have another look at the code and tell me what you think.

Patch looks great, just a minor issue.
As per description of fullnamedisplayprivate, it should only be visible to users with moodle/site:viewuseridentity capability, but as per #L15R3596, it is only valid if $override is true. Also, should we consider having forcealternatename, similar to forcefirstname/forcelastname CFG? (probably another issue)

Rajesh Taneja
added a comment - 25/Jun/13 8:14 AM Thanks for fixing this Adrian,
Patch looks great, just a minor issue.
As per description of fullnamedisplayprivate, it should only be visible to users with moodle/site:viewuseridentity capability, but as per #L15R3596, it is only valid if $override is true. Also, should we consider having forcealternatename, similar to forcefirstname/forcelastname CFG? (probably another issue)
Also, please update testing instructions replacing
fullnameformat with fullnamedisplay
fullnameformatprivate with fullnamedisplayprivate

Thank Rajesh for having another look for me.
I've removed the second fullnamedisplayprivate setting and updated the testing instructions. I created another issue to look at putting this setting back as I think that it would be very useful.

Adrian Greeve
added a comment - 28/Jun/13 1:47 AM Thank Rajesh for having another look for me.
I've removed the second fullnamedisplayprivate setting and updated the testing instructions. I created another issue to look at putting this setting back as I think that it would be very useful.
Sending to integration review.

moodlelib.php - fullname
The regex added for tidying up misc characters sticks out to me.
After mulling it over (hot cross buns + coffee) I'm just not sure that this should be there.
To summarise the code there we are trying to clean up strings removing characters associated with the display of field is found to be empty.
It deals with one specific situation, where a pair of characters are encompasing a field.
There are a couple of things I don't like about this:

The regular expression doesn't match pairs, it matches a start character and an end character that have no relation to the other. Another words the characters '-(' may be stripped if they appear next to each others.

There is separation of what was a field and what was not. Even if the template only contained fields we'd still execute the regex. User content (aka field content) will still be searched and acted upon.

We're dealing with just one possible case of unwanted characters. Other cases such as placing hyphens or commas between fields is not dealt with. While this could be considered extending functionality I think this helps to lead onto what I think may be a better solution to this issue.

As an example of where this would fall over consider the template "last (first, middle)".

So what should be done. Well first - do we want to implement this magic stripping of redunant characters?
I could imagine people wanting it to be done so perhaps yes we want it but I'd like to be sure the question has been asked.
Truthfully I don't feel we should be doing this. I think we should be applying the template and leaving it at that. If the template envolves braces, hyphens or any character other character and they end up being redundant we don't try to fix it.
Any other solution is going to lead to either involve unmet cases or an element of guessing.

So really there are three options I see:

Don't do it.

Change it so that it only removes "template" content. This was it will only remove matching "paired" characters that appear redundant in the template.

Leave it as it is, breakable magic. (I've made lots of noise for no reason )

If options 2 or 3 are selected I think we also need many more unit tests to cover what is done.

other things

1. outputcompontents.php user_picture There are a couple of things here, so sub numbers it is:

Having to array_merge in get_all_name_fields when using self::$fields is really unfortunate. $fields is a static property that should contain all required fields. Not only do you add the calls to array_merge but in the process you have likely caused a backwards compatability issue for anyone who may have overridden the user_picture class. IMHO $fields needs to include ALL required fields.

You've broken user_picture::fields() if it is called without a tableprefix, extrafields, or idalias. This would be fixed by above and while it would be unlikely perhaps someone out there is calling it in this way, best no break it aye.

3. moodlelib.php get_all_name_fields This could be named better I feel. We seem to have several _user_field functions already. A more consistent name may be get_all_user_name_fields.

4. moodlelib.php get_all_name_fields Arg 1 $fields could certainly be named better. What about "$returnsql" or something that better describes what that argument does.

5. moodlelib.php get_enabled_additional_names Naming again although looking at this function I would really question whether it is needed. Its only used in user/editlib.php and the uses there look like they could easily be factored out.

After looking at get_all_name_fields and get_enabled_additional_names did you have in mind some plan to allow those functions to be overridden or something? There seems to be a bit of code around working with those two functions and any differences. However it appears to me that if they can't be overridden that we would always know the differences?

Anyway, at this point I've found several reasons to send this back without getting to far into the patch and I've spent most of the morning on it.
I'm going to reopen this now and proceed with integration reviews. Once I've got more time I'll come back to it and continue my review.

Sam Hemelryk
added a comment - 30/Jun/13 11:37 PM First up this change appears to be causing a few unit tests to fail.
moodlelib.php - fullname
The regex added for tidying up misc characters sticks out to me.
After mulling it over (hot cross buns + coffee) I'm just not sure that this should be there.
To summarise the code there we are trying to clean up strings removing characters associated with the display of field is found to be empty.
It deals with one specific situation, where a pair of characters are encompasing a field.
There are a couple of things I don't like about this:
The regular expression doesn't match pairs, it matches a start character and an end character that have no relation to the other. Another words the characters '-(' may be stripped if they appear next to each others.
There is separation of what was a field and what was not. Even if the template only contained fields we'd still execute the regex. User content (aka field content) will still be searched and acted upon.
We're dealing with just one possible case of unwanted characters. Other cases such as placing hyphens or commas between fields is not dealt with. While this could be considered extending functionality I think this helps to lead onto what I think may be a better solution to this issue.
As an example of where this would fall over consider the template "last (first, middle)".
So what should be done. Well first - do we want to implement this magic stripping of redunant characters?
I could imagine people wanting it to be done so perhaps yes we want it but I'd like to be sure the question has been asked.
Truthfully I don't feel we should be doing this. I think we should be applying the template and leaving it at that. If the template envolves braces, hyphens or any character other character and they end up being redundant we don't try to fix it.
Any other solution is going to lead to either involve unmet cases or an element of guessing.
So really there are three options I see:
Don't do it.
Change it so that it only removes "template" content. This was it will only remove matching "paired" characters that appear redundant in the template.
Leave it as it is, breakable magic. (I've made lots of noise for no reason )
If options 2 or 3 are selected I think we also need many more unit tests to cover what is done.
other things
1. outputcompontents.php user_picture There are a couple of things here, so sub numbers it is:
Having to array_merge in get_all_name_fields when using self::$fields is really unfortunate. $fields is a static property that should contain all required fields. Not only do you add the calls to array_merge but in the process you have likely caused a backwards compatability issue for anyone who may have overridden the user_picture class. IMHO $fields needs to include ALL required fields.
You've broken user_picture::fields() if it is called without a tableprefix, extrafields, or idalias. This would be fixed by above and while it would be unlikely perhaps someone out there is calling it in this way, best no break it aye.
unalias will be fixed by the above as well.
2. moodlelib.php get_enabled_additional_names
More thorough unit tests here would reveal a bug:
$valuearray = array('second', 'firsthalf', 'first');
$formatstring = 'firsthalf first second';
$expectedarray = array('0' => 'firsthalf', '10' => 'firsthalf', '16' => 'second');
$this->assertEquals($expectedarray, order_in_string($valuearray, $formatstring));
You already commented about this bug in the code
3. moodlelib.php get_all_name_fields This could be named better I feel. We seem to have several _user_field functions already. A more consistent name may be get_all_user_name_fields.
4. moodlelib.php get_all_name_fields Arg 1 $fields could certainly be named better. What about "$returnsql" or something that better describes what that argument does.
5. moodlelib.php get_enabled_additional_names Naming again although looking at this function I would really question whether it is needed. Its only used in user/editlib.php and the uses there look like they could easily be factored out.
After looking at get_all_name_fields and get_enabled_additional_names did you have in mind some plan to allow those functions to be overridden or something? There seems to be a bit of code around working with those two functions and any differences. However it appears to me that if they can't be overridden that we would always know the differences?
Anyway, at this point I've found several reasons to send this back without getting to far into the patch and I've spent most of the morning on it.
I'm going to reopen this now and proceed with integration reviews. Once I've got more time I'll come back to it and continue my review.
Thanks for the work and sorry for the delay,
Sam

Adrian Greeve
added a comment - 02/Jul/13 3:37 AM Thanks Sam for the review.
I've gone ahead and made changes as per your comments.
I've fixed up the places that were causing unit test problems and run all unit tests to make sure that I haven't missed something.
The addition of the regex line is per the instructions of Martin as commented here https://tracker.moodle.org/browse/MDL-31776?focusedCommentId=217808&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-217808
We had a quick discussion about this. What I have done is greatly simplified the expression. I had some characters in there that I don't think we really needed to check. I'm only looking at different types of brackets now. I've added about nine additional tests to show how this works.
I added the additional fields to the user_picture class. This has solved the other issues that you mentioned about that.
I fixed up the mistake in the code for get_enabled_additional_names and added a couple more tests for this.
I re-named get_all_name_fields to get_all_user_name_fields.
The first argument for get_all_user_name_fields is now called $returnsql.
get_enabled_additional_names has been removed and the logic for this is now only located in user/editlib.php
If you or some other integrator would be so kind as to please have another look at this issue, I would be most grateful.
Thanks.

Damyon Wiese
added a comment - 08/Jul/13 3:10 AM Hi Adrian,
Keep going you are almost there (but not quite).
Here are the issues I spotted in this patch:
Editing a user profile gives debugging warnings:
line 3613 of /lib/moodlelib.php: call to debugging()
line 3130 of /repository/lib.php: call to fullname()
line 319 of /lib/form/editor.php: call to initialise_filepicker()
Saving a user profile gives "undefined property"
Undefined property: stdClass::$fullnamedisplayprivate in /home/damyonw/Documents/Moodle/integration/master/moodle/user/editlib.php on line 132 Call Stack: 0.0008 833248
The issue with templates needs to be solved IMO. Sam's example is the first one I came up with too - ie:
firstname, lastname (firstnamephonetic, lastnamephonetic)
Which gives e.g. Sandy, Raynor (, )
for most users without phonetic names.
Maybe - when a field is empty - remove all punctuation surrounding the field (until you hit whitespace)
In lib/badgeslib.php - can you also remove this line:
$userfrom->firstname = !empty($CFG->badges_defaultissuername) ? $CFG->badges_defaultissuername : $admin->firstname;
Changes in get_users_listing (lib/datalib.php) seems to be missing a comma in the SQL.
In flexible table you have added support for sorting by your new name fields - but the columns in the DB do not have indexes (bad).
These seem pretty small (except maybe the regex one) - ping me when you update the branch and I'll take another look.
Thanks!

I've gone back through the code and I have sorted out the various problems.

The debuggging warnings that you were receiving, I am pretty sure, are due to the fact that the code wasn't rebased since the last release and the update code didn't run on your machine. This will solve:

user profile debugging warnings

Saving a user profile displaying "undefined property"

Undefined property: stdClass::$fullnamedisplay etc. etc.

I've re-written the regular expression to handle the example that you have given and increased the unit tests to handle this problem.

Is there a specific reason that this line should be removed?
The current line is :

Adrian Greeve
added a comment - 08/Jul/13 7:21 AM Thanks Damyon,
I've gone back through the code and I have sorted out the various problems.
The debuggging warnings that you were receiving, I am pretty sure, are due to the fact that the code wasn't rebased since the last release and the update code didn't run on your machine. This will solve:
user profile debugging warnings
Saving a user profile displaying "undefined property"
Undefined property: stdClass::$fullnamedisplay etc. etc.
I've re-written the regular expression to handle the example that you have given and increased the unit tests to handle this problem.
Is there a specific reason that this line should be removed?
The current line is :
$userfrom->firstname = !empty($CFG->badges_defaultissuername) ? $CFG->badges_defaultissuername : $admin->firstname;
And you want me to change it to:
$userfrom->$addname = !empty($CFG->badges_defaultissuername) ? '' : $admin->$addname;
I was a bit worried with this if statement not being the same as the others, so I decided to leave the logic as it was. I'm happy to remove it if it won't break anything.
This SQL doesn't have a comma as $extrafields already has a comma at the start of that string.
I've added indexes for the additional name fields.
Thanks.

Damyon Wiese
added a comment - 09/Jul/13 1:53 AM - edited Just documenting our conversation on the regex. It is better to have a simple fast rule that works for 90% of cases that a specific complex rule that is hard to explain. So preferred algorithm is:
step 1 - replace empty fields with "EMPTY"
step 2 - run a regex for [:punct:]*EMPTY[:punct:]* and replace with ''
step 3 - profit :)

Adrian Greeve
added a comment - 10/Jul/13 2:00 AM - edited I found a notice being displayed in the forums. I quickly made a fix for it. It's based on the current integration branch.
wip- MDL-31776 -master-alternatenamesfix
I also found a bug in the question area. I have also included a commit for this as well.

Marina Glancy
added a comment - 10/Jul/13 7:57 AM I was not integrating it and this was your luck Adrian (because I'm an evil).
There is a lot of code duplication in this issue. This could easily be avoided by
1) adding the 3rd parameter to the function get_all_user_name_fields() which specifies prefix for the retrieved sql fields
2) adding a new function that does a reverse mapping, i.e. to substitute repeating pieces of code like
$allnames = get_all_user_name_fields();
foreach ($allnames as $allname) {
$tempname = 'creator' . $allname;
if (isset($question->$tempname)) {
$u->$allname = $question->$tempname;
}
}
or the same without prefix
$allnames = get_all_user_name_fields();
foreach ($allnames as $addname) {
$userinfo[$reviewer->reviewerid]->$addname = $reviewer->$addname;
}
I can see lots of similar chunks of code.
Well, second one is probably too much now, maybe as a suggestion for improvement. But I would very recommend to add the $prefix argument to get_all_user_name_fields() now. What do you think?

Adrian Greeve
added a comment - 10/Jul/13 9:31 AM Thanks for the comments Marina.
I completely agree. I think these ideas are excellent.
I've created MDL-40612 to make these improvements as they will take some time to change code through out core.
Thanks again.

After having a discussion with Fred about how user names are now displayed. I think that he raises a valid point and that we should re-evaluate how names are displayed though out moodle. I've created an issue for this. MDL-40616.

Adrian Greeve
added a comment - 11/Jul/13 3:11 AM After having a discussion with Fred about how user names are now displayed. I think that he raises a valid point and that we should re-evaluate how names are displayed though out moodle. I've created an issue for this. MDL-40616 .

Marina Glancy
added a comment - 11/Jul/13 3:21 AM This issue requires documentation (at least for the settings pages that are changed) and developers documentation. Also it needs to be included in release notes

Eloy Lafuente (stronk7)
added a comment - 16/Jul/13 4:33 PM Please don't forget both user and developer documentation for this.
A mini-guide about how to move to use/support the new fields would be awesome (we are already receiving code showing users here and there and it's hard to explain the required changes/available API).
Ciao

Adrian, please don't forget about dev docs. There are more places where this is not implemented, function fullname() displays a debug information and there is no clue in the function itself on how to change the code.

Marina Glancy
added a comment - 25/Jul/13 4:46 AM Adrian, please don't forget about dev docs. There are more places where this is not implemented, function fullname() displays a debug information and there is no clue in the function itself on how to change the code.

Dan Poltawski
added a comment - 30/Jul/13 1:47 AM Hi Adrian,
We've noticed a sporadic unit test failure related to this a few times on the integration server. It concerns us a little so have created MDL-40929
love,
Integrators
xxx

Mary Cooch
added a comment - 06/Nov/13 2:51 PM Removing docs_required link with thanks to Adrian for this http://docs.moodle.org/26/en/Additional_name_fields and also qa_test_required as there is a test here MDLQA-6673 (passed)