Details

1. Try to import a Moodle XML file. (Create one by exporting from another Moodle). Make sure it still works.

2. Try to import a GIFT file. Make sure it works. You can find one in question/format/gift/simpletest/fixtures/questions.gift.txt

3. Try to get hold of a blackboard_six format file, and if you can, try importing it.
(See the comments be Chris Follin who can probably send you a test file, even thought it cannot be attached here.) Alternatively use the 2 attached files from Jean-Michel Vedrine on March 4th. He explains the expected results in a comment below.

1. Try to import a Moodle XML file. (Create one by exporting from another Moodle). Make sure it still works.
2. Try to import a GIFT file. Make sure it works. You can find one in question/format/gift/simpletest/fixtures/questions.gift.txt
3. Try to get hold of a blackboard_six format file, and if you can, try importing it.
(See the comments be Chris Follin who can probably send you a test file, even thought it cannot be attached here.) Alternatively use the 2 attached files from Jean-Michel Vedrine on March 4th. He explains the expected results in a comment below.

Description

Hello Tim,
In some import formats there is a missing function :
public function mime_type()
As a consequence the import is broken
For instance the blackboard_six format is missing
public function mime_type()

{
return 'application/zip';
}

And the learnwise format is missing
public function mime_type()

{
return 'application/xml';
}

I will do a quick review and will report in comments the others that I found.
Unfortunately I don't have a test file for each format and all these formats don't have tests included so I only speak from looking at the code and for people posts in the forum (this is how I discovered the problem.

Activity

Sorry I was wrong blackboard_six expect a zip file but can also accept a dat file
blackboard expect an xml file
qti_two seems to expect a zip file but does it still works ?
webct ?? no idea ?? maybe just a text file maybe a text file with a strange extension ? In fact I know nothing about webct

Jean-Michel Vedrine
added a comment - 16/Jul/11 5:13 PM Sorry I was wrong blackboard_six expect a zip file but can also accept a dat file
blackboard expect an xml file
qti_two seems to expect a zip file but does it still works ?
webct ?? no idea ?? maybe just a text file maybe a text file with a strange extension ? In fact I know nothing about webct

So here is the final summary
1) blackboard_six format is not fixable with the current API because it can import not one but 2 types of files (zip and dat)
2) learnwise format is easy to fix just adding
public function mime_type()

{ return 'application/xml'; }

3) qti_two is only an export format (no import) so it doesn't need a mime_type function

The only real problem is the blackbord_six. Possible workarounds :
1) create 2 differents formats one for zip files and one for dat files with possible inheritance of code between them ?
2) change API so that mime_type can return an array of accepted types ?
3) any better solution you can think of ?

Jean-Michel Vedrine
added a comment - 03/Aug/11 4:49 AM So here is the final summary
1) blackboard_six format is not fixable with the current API because it can import not one but 2 types of files (zip and dat)
2) learnwise format is easy to fix just adding
public function mime_type()
{ return 'application/xml'; }
3) qti_two is only an export format (no import) so it doesn't need a mime_type function
The only real problem is the blackbord_six. Possible workarounds :
1) create 2 differents formats one for zip files and one for dat files with possible inheritance of code between them ?
2) change API so that mime_type can return an array of accepted types ?
3) any better solution you can think of ?

This is a bigger bug than people seem to think. I can't get ANY imports to work other than GIFT imports, and even then I have to rename the file to .txt to get past this error. Currently, I can't get any Examview exports to work in Moodle 2.1 I've tried Blackboard 6 & 7. they are zip files. WebCT (also a zip file).

Matthew Davidson
added a comment - 30/Aug/11 9:35 PM This is a bigger bug than people seem to think. I can't get ANY imports to work other than GIFT imports, and even then I have to rename the file to .txt to get past this error. Currently, I can't get any Examview exports to work in Moodle 2.1 I've tried Blackboard 6 & 7. they are zip files. WebCT (also a zip file).

This is a serious issue. I too can not import questions because the requested file type is not correct. The error message I receive is "The type of the file you selected (application/zip) does not match the type expected by this import format (text/plain)" when trying to import from an xml file, and I have tried several other types with no success.

Richard Arzillo
added a comment - 12/Sep/11 11:40 PM This is a serious issue. I too can not import questions because the requested file type is not correct. The error message I receive is "The type of the file you selected (application/zip) does not match the type expected by this import format (text/plain)" when trying to import from an xml file, and I have tried several other types with no success.

Tim,
I seem at least 2 ways to solve the problem for the blackboard_six
1) make 2 differents formats plugins for Blackboard V6, one accepting zip files and the other dat files

2) modify the code allowing the mime_type function to return either a string as currently (for BC) or an array. blackboard_six would return array('application/zip', 'text/plain')for instance
the only change needed is to modify question/import_form.php validate_uploaded_file function replacing the code
if ($file->get_mimetype() != $qformat->mime_type())

Jean-Michel Vedrine
added a comment - 27/Sep/11 8:22 PM Tim,
I seem at least 2 ways to solve the problem for the blackboard_six
1) make 2 differents formats plugins for Blackboard V6, one accepting zip files and the other dat files
2) modify the code allowing the mime_type function to return either a string as currently (for BC) or an array. blackboard_six would return array('application/zip', 'text/plain')for instance
the only change needed is to modify question/import_form.php validate_uploaded_file function replacing the code
if ($file->get_mimetype() != $qformat->mime_type())
{
$a = new stdClass();
$a->actualtype = $file->get_mimetype();
$a->expectedtype = $qformat->mime_type();
$errors['newfile'] = get_string('importwrongfiletype', 'question', $a);
}
by a code that first checks if $qformat->mime_type() is an array or nor and acts accodingly.
The advantage of the second solution is that it requires no change to the formats.

I cannot believe this hasn't become a red flag in M2? The ability to import massive question banks is one of the greatest features. I know that the work around in 1.9 was to upload the file, then browse from file manager. It would error when attemting to import into the question bank, but w/ the new file structure of 2.1 that work around is not available.

Ray Waller
added a comment - 11/Jan/12 7:02 AM I cannot believe this hasn't become a red flag in M2? The ability to import massive question banks is one of the greatest features. I know that the work around in 1.9 was to upload the file, then browse from file manager. It would error when attemting to import into the question bank, but w/ the new file structure of 2.1 that work around is not available.

Chris Follin
added a comment - 21/Feb/12 7:40 AM We encountered this specific to BB6 zip files. This is a little bit more than a minor issue and there doesn't seem to be a workaround, so I'm bumping priority one level.
Jean-Michel, have you used the code you included in your last comment? Would it be possible for you to attach a patch or git branch? Thanks!

Cleverer plugins could peek inside the file to run a quick sanity check on the internal format. For example, BB6 .DAT files are XML internally, but since .DAT is a fairly generic and opaque file extension it could easily contain raw binary data.

Jonathon Fowler
added a comment - 27/Feb/12 1:28 PM Might it be worthwhile implementing a method on the qformat_default class that accepts a file object and returns true/false if the plugin supports it or not, and then have the import_form.php code call that new function. The importer could override it if necessary to be more fancy. For example, see https://github.com/jonof/moodle/commit/7adec66fcec2855b16fc20fdd7ff58ce12ab726f
Cleverer plugins could peek inside the file to run a quick sanity check on the internal format. For example, BB6 .DAT files are XML internally, but since .DAT is a fairly generic and opaque file extension it could easily contain raw binary data.

1. I think handles_files would be better called can_import, or can_import_file, so, for example, the code reads if (!$qformat->can_import($file)) {

2. The PHPdoc comment for the function in the base class needs to be done properly. You need to add @param $file, and it would be worth adding a sentence or two about what is expected - that is that it should be a quick check that the file can plausibly be imported. This method will not go and parse the entire file and report all errors.

If you can fix those two up, then this can be submitted for integration. Thanks

Tim Hunt
added a comment - 27/Feb/12 5:30 PM Yes, that is a much better API.
Can I just ask for two changes to your patch?
1. I think handles_files would be better called can_import, or can_import_file, so, for example, the code reads if (!$qformat->can_import($file)) {
2. The PHPdoc comment for the function in the base class needs to be done properly. You need to add @param $file, and it would be worth adding a sentence or two about what is expected - that is that it should be a quick check that the file can plausibly be imported. This method will not go and parse the entire file and report all errors.
If you can fix those two up, then this can be submitted for integration. Thanks

Hello,
This is definitely the good api,
But please note that this will not be enought to "repair" the blackboard 6 import format because of other changes in the way Moodle now manage images. We need to also commit other changes from MDL-25492. But now that I am finaly able to somewhat use github (thanks to your help Tim !) I will clone Tim's commit to MDL-25492 and combine with this change to have a workable solution.

Jean-Michel Vedrine
added a comment - 27/Feb/12 6:22 PM Hello,
This is definitely the good api,
But please note that this will not be enought to "repair" the blackboard 6 import format because of other changes in the way Moodle now manage images. We need to also commit other changes from MDL-25492 . But now that I am finaly able to somewhat use github (thanks to your help Tim !) I will clone Tim's commit to MDL-25492 and combine with this change to have a workable solution.

Jonathon Fowler
added a comment - 27/Feb/12 7:27 PM @Tim: Hopefully this is better. You may want to squash together the previous commit I posted and this one: https://github.com/jonof/moodle/commit/f759050e7363a73376f55aa833867b1b8a5749a5

Yes, that is great. Can you squash the two commits together and then make commits cherry-picked to the 2.1 and 2.2 stable branches too? Basically, we want this bug report to look like this: http://tracker.moodle.org/browse/MDL-31501, with the Pull-* fields completed. Thanks.

Tim Hunt
added a comment - 27/Feb/12 8:54 PM Yes, that is great. Can you squash the two commits together and then make commits cherry-picked to the 2.1 and 2.2 stable branches too? Basically, we want this bug report to look like this: http://tracker.moodle.org/browse/MDL-31501 , with the Pull-* fields completed. Thanks.

I think I have this sorted out now. I usually use Git shackled to Subversion so I rarely get the chance to play with its really nice branching and merging functionality. As such, some of my branches were a bit weird.

Chris Follin
added a comment - 28/Feb/12 10:35 PM I have a BB6 file from a client. I need to verify with that client that it's okay to post the file publicly. If it's not okay, perhaps I could email it directly to whomever is doing the testing.

Tim, the client replied that the file contains copyrighted information from a textbook publisher and thus we cannot share it publicly. They also don't have any way to create an innocuous file. However, we believe it would be okay to send the file directly to a developer for testing as long as it is used for testing only and is not shared. I will be happy to email it if someone can let me know to whom it should be sent.

Chris Follin
added a comment - 29/Feb/12 12:31 AM Tim, the client replied that the file contains copyrighted information from a textbook publisher and thus we cannot share it publicly. They also don't have any way to create an innocuous file. However, we believe it would be okay to send the file directly to a developer for testing as long as it is used for testing only and is not shared. I will be happy to email it if someone can let me know to whom it should be sent.

Hello,
Please be carefull not to complicate anymore an already complicated situation !
Files called "blackboard zip" or "blackboard xml" are not all of the same type !!! Se my post http://moodle.org/mod/forum/discuss.php?d=148877&parent=831418 if you want to understand the problem !!
The file "domain and range test.zip" uploaded in this issue today by Don is not a good example for testing ! This is a file made by Examview and such files were never imported correctly by any version of the core import/export formats included in Moodle (be it 1.9,x, 2.0.x 2.1.x, or 2.2.x).
There are third party plugins import formats able to import this file (for instance the one I submitted under the name "examview_bbsixplus" to issue MDL-25492).
But in this issue we are only dealing with the "blackboard_six" import format.
I will submit a proper file to test the fix after verifying that it import correctly with Moodle 1.9 blackboard_six import format. I only need a few hours to do so.

Jean-Michel Vedrine
added a comment - 29/Feb/12 3:26 AM Hello,
Please be carefull not to complicate anymore an already complicated situation !
Files called "blackboard zip" or "blackboard xml" are not all of the same type !!! Se my post http://moodle.org/mod/forum/discuss.php?d=148877&parent=831418 if you want to understand the problem !!
The file "domain and range test.zip" uploaded in this issue today by Don is not a good example for testing ! This is a file made by Examview and such files were never imported correctly by any version of the core import/export formats included in Moodle (be it 1.9,x, 2.0.x 2.1.x, or 2.2.x).
There are third party plugins import formats able to import this file (for instance the one I submitted under the name "examview_bbsixplus" to issue MDL-25492 ).
But in this issue we are only dealing with the "blackboard_six" import format.
I will submit a proper file to test the fix after verifying that it import correctly with Moodle 1.9 blackboard_six import format. I only need a few hours to do so.

Doesn't look like I can attach files so here's a an exported Blackboard assessment containing multiple choice questions (two questions contain images). I thought you might find this useful for testing:

I think the siutation with blackboard exported assessments and import into the moodle question bank is complicated further by different 'strands' of the blackboard product. We have Blackboard 8.04(a WebCT upgraded strand). Unfortunately this quite vanilla quiz won't import into Moodle using either of the following Moodle import methods:

Blackboard V6+
WebCT format

This has meant we are using Respondus to extract from Blackboard, convert to Moodle and publish into a Moodle course.

Hope you find the file useful, if you need any more test files just let me know.

Mike Wilson
added a comment - 29/Feb/12 5:21 PM - edited Doesn't look like I can attach files so here's a an exported Blackboard assessment containing multiple choice questions (two questions contain images). I thought you might find this useful for testing:
https://docs.google.com/open?id=0BwvY95sOa16DLW4xcC1fajVRM2lzR05ya0x1T2xYQQ
I think the siutation with blackboard exported assessments and import into the moodle question bank is complicated further by different 'strands' of the blackboard product. We have Blackboard 8.04(a WebCT upgraded strand). Unfortunately this quite vanilla quiz won't import into Moodle using either of the following Moodle import methods:
Blackboard V6+
WebCT format
This has meant we are using Respondus to extract from Blackboard, convert to Moodle and publish into a Moodle course.
Hope you find the file useful, if you need any more test files just let me know.

The main moodle.git repository has just been updated with latest weekly modifications. You may wish to rebase your PULL branches to simplify history and avoid any possible merge conflicts. This would also make integrator's life easier next week.

Eloy Lafuente (stronk7)
added a comment - 01/Mar/12 7:45 PM The main moodle.git repository has just been updated with latest weekly modifications. You may wish to rebase your PULL branches to simplify history and avoid any possible merge conflicts. This would also make integrator's life easier next week.
TIA and ciao

Hello, I have attached 2 files sample_bb6.dat and sample_bb6.zip. these 2 files contain the exact same questions.
I have verified that these 2 files import correctly in Moodle 1.9. (see below the expected result)
Here is the expected result when using the "Blackboard V6+" standard format in Moodle 1.9 :
You should see a warning saying that the matching question cant be imported because rendered matching question type is not installed THIS IS EXPECTED (unless of course rendered matching qtype is installed) because blackboard matching questions can't be imported as standard match Moodle questions. So this is part of the test !
After import 5 questions should have been created : 3 essays questions, 1 multiple choice question and a true false question.
All these questions will have "strange" names made of a long alphanumeric string. This is also the "expected" result !!

Jean-Michel Vedrine
added a comment - 05/Mar/12 2:10 AM Hello, I have attached 2 files sample_bb6.dat and sample_bb6.zip. these 2 files contain the exact same questions.
I have verified that these 2 files import correctly in Moodle 1.9. (see below the expected result)
Here is the expected result when using the "Blackboard V6+" standard format in Moodle 1.9 :
You should see a warning saying that the matching question cant be imported because rendered matching question type is not installed THIS IS EXPECTED (unless of course rendered matching qtype is installed) because blackboard matching questions can't be imported as standard match Moodle questions. So this is part of the test !
After import 5 questions should have been created : 3 essays questions, 1 multiple choice question and a true false question.
All these questions will have "strange" names made of a long alphanumeric string. This is also the "expected" result !!

line 397 of \lib\dml\moodle_database.php: dml_write_exception thrown
line 999 of \lib\dml\mysqli_native_moodle_database.php: call to moodle_database->query_end()
line 1031 of \lib\dml\mysqli_native_moodle_database.php: call to mysqli_native_moodle_database->update_record_raw()
line 69 of \question\type\essay\questiontype.php: call to mysqli_native_moodle_database->update_record()
line 432 of \question\format.php: call to qtype_essay->save_question_options()
line 119 of \question\import.php: call to qformat_default->importprocess()

Good news is that this show this issue is solved : both .zip and .dat files are now accepted !
Bad news is that this show there are other problems in the blackboard_6 format in Moodle 2.x (this format has not been updated to take into account the fact that now all html fields have an xxxxformat field associated and also images management has not been updated)
But there is already another issue open in the tracker to fix these other problems (see MDL-25492 assigned to me)
So I suggest that after testing you get the same results as me you close this bug as solved (because now the .zip and.dat files are accepted) and I concentrate on polishing my fix to MDL-25492.
Note : I have verified that the fix I submitted to MDL-25492 is working and that in combining the fixes of MDL-25492 and MDL-28364 I am able to import all the Blackboard 6 examples files I have.

Jean-Michel Vedrine
added a comment - 07/Mar/12 2:20 PM Hello Eloy,
I tested these files in Moodle 2.x and you will get several notices and a fatal error :
Debug info: Column 'responseformat' cannot be null
UPDATE mdl_qtype_essay_options SET questionid = ?,responseformat = ?,responsefieldlines = ?,attachments = ?,graderinfo = ?,graderinfoformat = ? WHERE id=?
[array (
0 => 44,
1 => NULL,
2 => NULL,
3 => NULL,
4 => '',
5 => NULL,
6 => 11,
)]
Stack trace:
line 397 of \lib\dml\moodle_database.php: dml_write_exception thrown
line 999 of \lib\dml\mysqli_native_moodle_database.php: call to moodle_database->query_end()
line 1031 of \lib\dml\mysqli_native_moodle_database.php: call to mysqli_native_moodle_database->update_record_raw()
line 69 of \question\type\essay\questiontype.php: call to mysqli_native_moodle_database->update_record()
line 432 of \question\format.php: call to qtype_essay->save_question_options()
line 119 of \question\import.php: call to qformat_default->importprocess()
Good news is that this show this issue is solved : both .zip and .dat files are now accepted !
Bad news is that this show there are other problems in the blackboard_6 format in Moodle 2.x (this format has not been updated to take into account the fact that now all html fields have an xxxxformat field associated and also images management has not been updated)
But there is already another issue open in the tracker to fix these other problems (see MDL-25492 assigned to me)
So I suggest that after testing you get the same results as me you close this bug as solved (because now the .zip and.dat files are accepted) and I concentrate on polishing my fix to MDL-25492 .
Note : I have verified that the fix I submitted to MDL-25492 is working and that in combining the fixes of MDL-25492 and MDL-28364 I am able to import all the Blackboard 6 examples files I have.

I agree with Jean-Michel. Let us consider this bug test passed, because the part of the problem that we set out to address here is done, and let us use MDL-25492 for tracking the rest of the work required.

Tim Hunt
added a comment - 07/Mar/12 3:43 PM I agree with Jean-Michel. Let us consider this bug test passed, because the part of the problem that we set out to address here is done, and let us use MDL-25492 for tracking the rest of the work required.

Eloy Lafuente (stronk7)
added a comment - 10/Mar/12 5:28 AM Your changes are now upstream and will be included in the next minor released scheduled for March 13th (next Monday!).
icao_reverse('arreis olik rebemevon afla letoh ognat');
Closing, ciao