1. In Moodle 1.9, create a course with the following resources having both Summary and Full text filled.
1.1 Web page (via "Compose a web page")
1.2 Text page (via "Compose a text page") with the format field set to "Moodle auto-format"
1.2 Text page with the format field set to "HTML format"
1.3 Text page with the format field set to "Plain text format"
1.4 Text page with the format field set to "Markdown format"
2. Make a ZIP backup of the course
3. Restore the backup in 2.1 or 2.2 or 2.3dev
4. Make sure you have TinyMCE enabled in your profile
5. TEST: Go and try to re-edit all four resources. Make sure that
4.1 TinyMCE is used to edit the page content of the legacy Web page resource
4.2 TinyMCE is used to edit the page content of the legacy Text page resource with the format set to "HTML format"
4.3 Good old textarea is used to edit the content of all other pages, with the format field displaying the correct value.

Testing difficulty level: Easy, requires Moodle 1.9 available.
1. In Moodle 1.9, create a course with the following resources having both Summary and Full text filled.
1.1 Web page (via "Compose a web page")
1.2 Text page (via "Compose a text page") with the format field set to "Moodle auto-format"
1.2 Text page with the format field set to "HTML format"
1.3 Text page with the format field set to "Plain text format"
1.4 Text page with the format field set to "Markdown format"
2. Make a ZIP backup of the course
3. Restore the backup in 2.1 or 2.2 or 2.3dev
4. Make sure you have TinyMCE enabled in your profile
5. TEST: Go and try to re-edit all four resources. Make sure that
4.1 TinyMCE is used to edit the page content of the legacy Web page resource
4.2 TinyMCE is used to edit the page content of the legacy Text page resource with the format set to "HTML format"
4.3 Good old textarea is used to edit the content of all other pages, with the format field displaying the correct value.

Michael de Raadt
added a comment - 07/Sep/11 3:26 PM Hi, George.
Could you please be more specific about the circumstances surrounding this issue?
Perhaps you try to restore the backup on demo.moodle.net to see if the problem is with the system or with the backup itself.

George Chen
added a comment - 07/Sep/11 11:37 PM Hello Michael,
How to restore the backup on the demo.moodle.net site? I have registered as username with georchen. Please allow me to restore the course.

Hi Michael,
I have get my client approved and created an account for your at http://sandbox.cteched.net/course/modedit.php?update=48&return=0
here is your login information
username:moodletest
password:moodletest
my client complained about the WYSIWYG editor not working on this.Please let me know if you have any questions

George Chen
added a comment - 09/Sep/11 10:47 PM Hi Michael,
I have get my client approved and created an account for your at
http://sandbox.cteched.net/course/modedit.php?update=48&return=0
here is your login information
username:moodletest
password:moodletest
my client complained about the WYSIWYG editor not working on this.Please let me know if you have any questions

I think that the cause for this is that, in 1.9 (and its backups) we didn't have all those xxxformat fields that are everywhere in 2.0.

So, surely, in the conversion we are defaulting to FORMAT_MOODLE or to "nothing" that also leads to FORMAT_MOODLE.

Perhaps we could try some sort of format detection in the conversion process, I've to check that with David.

Surely the detectors won't be 100% accurate but I think that looking for some html tags we could get a good number of FORMAT_HTML detected. The ones not detected will need to be solved by hand (with the workaround commented above).

Eloy Lafuente (stronk7)
added a comment - 22/Sep/11 3:52 AM I think that the cause for this is that, in 1.9 (and its backups) we didn't have all those xxxformat fields that are everywhere in 2.0.
So, surely, in the conversion we are defaulting to FORMAT_MOODLE or to "nothing" that also leads to FORMAT_MOODLE.
Perhaps we could try some sort of format detection in the conversion process, I've to check that with David.
Surely the detectors won't be 100% accurate but I think that looking for some html tags we could get a good number of FORMAT_HTML detected. The ones not detected will need to be solved by hand (with the workaround commented above).
Ciao

Firstly, regarding "The Moodle auto-format isn't supported in 2.x." - that is not true. Of course the "Moodle auto-format" is supported in 2.x, as well as "Plain text" and "Markdown" are. They just do not have wysiwyg intentionally as their users prefer plain text areas.

Eloy is right, this sounds like a result of the format detection during the backup conversion. The 1.9->2.1 backup converter follows the same pattern as the upgrade code does: usually using something like

if ($CFG->texteditors !== 'textarea') {

$data['intro'] = text_to_html($data['intro'], false, false, true);

$data['introformat'] = FORMAT_HTML;

}

So most of the places from 1.9 are supposed to be migrated into FORMAT_HTML if the TinyMCE was enabled during the migration / upgrade.

George: is it that the HTML editor disappeared from all fields in the restored course? Or just from some of them?

David Mudrak
added a comment - 22/Sep/11 10:01 AM Firstly, regarding "The Moodle auto-format isn't supported in 2.x." - that is not true. Of course the "Moodle auto-format" is supported in 2.x, as well as "Plain text" and "Markdown" are. They just do not have wysiwyg intentionally as their users prefer plain text areas.
Eloy is right, this sounds like a result of the format detection during the backup conversion. The 1.9->2.1 backup converter follows the same pattern as the upgrade code does: usually using something like
if ($CFG->texteditors !== 'textarea') {
$data['intro'] = text_to_html($data['intro'], false, false, true);
$data['introformat'] = FORMAT_HTML;
}
So most of the places from 1.9 are supposed to be migrated into FORMAT_HTML if the TinyMCE was enabled during the migration / upgrade.
George: is it that the HTML editor disappeared from all fields in the restored course? Or just from some of them?

George, can you please send me that ZIP backup to david@moodle.com so that I can try to reproduce the problem and analyze the contents of the backup file. Of course I won't share the backup nor will provide it to 3rd parties.

David Mudrak
added a comment - 13/Jan/12 9:55 PM George, can you please send me that ZIP backup to david@moodle.com so that I can try to reproduce the problem and analyze the contents of the backup file. Of course I won't share the backup nor will provide it to 3rd parties.

Craig Drayton
added a comment - 09/Feb/12 4:44 AM Hi all, any progress on this issue?
Is there a way to change all "Moodle Auto-Format" to "HTML Format"? We have migrated hundreds of M1.9 courses and wish to avoid tutors having to manually change every page to HTML format.

We are in the process of migrating instructors over to Moodle 2.2 and this issue still persists (restoring from 1.9). I've found that this is regularly happening with Pages (like Craig mentioned). With hundreds of courses being moved over, we are going to have some big trouble with our faculty if they have to use this work around to get this squared away.

David or Eloy, is there anything I can do to help move this forward? I'm not a developer but I can sure help with testing and providing more details on this.

Justin Litalien
added a comment - 16/Feb/12 11:35 PM We are in the process of migrating instructors over to Moodle 2.2 and this issue still persists (restoring from 1.9). I've found that this is regularly happening with Pages (like Craig mentioned). With hundreds of courses being moved over, we are going to have some big trouble with our faculty if they have to use this work around to get this squared away.
David or Eloy, is there anything I can do to help move this forward? I'm not a developer but I can sure help with testing and providing more details on this.
Thanks!

Can you send me some 1.9 course backup where this can be reproduced? If you don't want or can't share the course, send it to my email david@moodle.com (I won't provide it to the 3rd parties). When sending the source, please append the instructions to reproduce - ie what module (Page etc) does not work after the restore for you.

David Mudrak
added a comment - 17/Feb/12 11:07 AM I just tried to look at http://sandbox.cteched.net/course/modedit.php?update=48&return=0 (the link that George Chen added above on 09/Sep/11 4:47 PM) and I can see the TinyMCE editor displayed there normally.
Can you send me some 1.9 course backup where this can be reproduced? If you don't want or can't share the course, send it to my email david@moodle.com (I won't provide it to the 3rd parties). When sending the source, please append the instructions to reproduce - ie what module (Page etc) does not work after the restore for you.

Many thanks to Justin Litalien for the excellent steps to reproduce provided in email. I was finally able to reproduce the problem in restored 1.9 Resources converted to 2.x Page modules. And also I was able to track the problem pretty easily.

It turns out that this is the buggy code in moodle1_mod_page_handler::process_legacy_resource() method:

$page['contentformat'] = (int)$data['reference'];

This code was copied from the page_20_migrate() function but applied incorrectly for all resource types. But it must be applied only for 1.9 resources of the type "Plain text page" (that stores the used format in the "reference" field). The resources of type "Web page" must force FORMAT_HTML but the casting code above assigns the format to a value 0, that is FORMAT_MOODLE.

I'm working on the patch for this bug in resources. I'm wondering how this could be missed during the testing as it is pretty obvious problem...

Please note I was not (and still am not) sure if this report was originally meant for resources only or for HTML editors everywhere. I believe the most of places that are trying to calculate the format field are correct. So if you experience missing HTML editors EVERYWHERE in the restored course, surely the problem will be elsewhere.

Increasing severenity to Critical as this bug leads to data loss with no easy way of auto-repair.

David Mudrak
added a comment - 21/Feb/12 5:45 PM Many thanks to Justin Litalien for the excellent steps to reproduce provided in email. I was finally able to reproduce the problem in restored 1.9 Resources converted to 2.x Page modules. And also I was able to track the problem pretty easily.
It turns out that this is the buggy code in moodle1_mod_page_handler::process_legacy_resource() method:
$page['contentformat'] = (int)$data['reference'];
This code was copied from the page_20_migrate() function but applied incorrectly for all resource types. But it must be applied only for 1.9 resources of the type "Plain text page" (that stores the used format in the "reference" field). The resources of type "Web page" must force FORMAT_HTML but the casting code above assigns the format to a value 0, that is FORMAT_MOODLE.
I'm working on the patch for this bug in resources. I'm wondering how this could be missed during the testing as it is pretty obvious problem...
Please note I was not (and still am not) sure if this report was originally meant for resources only or for HTML editors everywhere. I believe the most of places that are trying to calculate the format field are correct. So if you experience missing HTML editors EVERYWHERE in the restored course, surely the problem will be elsewhere.
Increasing severenity to Critical as this bug leads to data loss with no easy way of auto-repair.

I just submitted patches for 2.1, 2.2 and 2.3dev to be reviewed and hopefully integrated into this weekly builds.

Unfortunately there is no way how we could fix already restored resources in a safe way via the upgrade step. However, those who have hundreds of resources with this issue, may want to consider the following SQL:

Beware! You should use a set of SELECT statements to explore the data to be modified first. This SQL statements sets the format of all Page modules to HTML if the page content field contains characters used for HTML tags. If there is a page written in different format (like Plain text or Markdown), this statement will ruin its formatting. Even worse, it can introduce a security problem (imagine there was a Plain text page explaining how JavaScript and HTML can be used in some nasty way - by setting the format field to 1 you effectively execute the examples in the page visitors' browsers). There is no reliable way how to get to the original type and text format of the resource... Good luck.

David Mudrak
added a comment - 21/Feb/12 6:55 PM I just submitted patches for 2.1, 2.2 and 2.3dev to be reviewed and hopefully integrated into this weekly builds.
Unfortunately there is no way how we could fix already restored resources in a safe way via the upgrade step. However, those who have hundreds of resources with this issue, may want to consider the following SQL:
UPDATE mdl_page SET contentformat = 1 WHERE content LIKE '%<%>%' AND contentformat = 1;
Beware! You should use a set of SELECT statements to explore the data to be modified first. This SQL statements sets the format of all Page modules to HTML if the page content field contains characters used for HTML tags. If there is a page written in different format (like Plain text or Markdown), this statement will ruin its formatting. Even worse, it can introduce a security problem (imagine there was a Plain text page explaining how JavaScript and HTML can be used in some nasty way - by setting the format field to 1 you effectively execute the examples in the page visitors' browsers). There is no reliable way how to get to the original type and text format of the resource... Good luck.

BTW I think I know how this was missed during the testing now. Testers just made sure that the restored pages are displayed correctly (and they are as "Moodle auto-format" displays HTML correctly) but they did not try to re-edit them.

David Mudrak
added a comment - 21/Feb/12 6:58 PM BTW I think I know how this was missed during the testing now. Testers just made sure that the restored pages are displayed correctly (and they are as "Moodle auto-format" displays HTML correctly) but they did not try to re-edit them.

The main moodle.git repository has just been updated with latest weekly modifications. You may wish to rebase your PULL branches to simplify history and avoid any possible merge conflicts. This would also make integrator's life easier next week.

Eloy Lafuente (stronk7)
added a comment - 23/Feb/12 7:35 PM The main moodle.git repository has just been updated with latest weekly modifications. You may wish to rebase your PULL branches to simplify history and avoid any possible merge conflicts. This would also make integrator's life easier next week.
TIA and ciao

Eloy Lafuente (stronk7)
added a comment - 01/Mar/12 7:43 PM Well,
I wish I said it every time
you do the things you do.
You always lend a helping hand,
and I'm filled with gratitude.
You are strong and generous
for each and everyone one of us.
I am eternally grateful,
I cannot say thanks enough.
Sorry for the (un)cool bit above, lol. Closing this as fixed. Ciao

Susan Mangan
added a comment - 01/May/12 6:12 AM We are just in the process of backing up 1.9 courses and restoring into 2.2.2 and have come across this problem in the Summary section of Assignments. Can this be reopened to address this?

Rather than reopening this issue, please create a new issue for the summary section of assignments. If you could provide a sample backup file, this would help developers to reproduce the problem easily.

Helen Foster
added a comment - 01/May/12 1:01 PM Hi Susan,
Rather than reopening this issue, please create a new issue for the summary section of assignments. If you could provide a sample backup file, this would help developers to reproduce the problem easily.