We are sad to announce that we have decided to retire the QRTEngine and support. The continuing and often unpredictable developments in the Qualtrics environment make it increasingly difficult to provide a stable QRTE platform.

In September last year we first noticed compatibility issues between Qualtrics’ newest Survey Engine (Jiffy) and the QRTEngine. Over several calls with Qualtrics’ engineering team we were unable to come to a satisfying solution in fixing the QRTEngine for the Jiffy engine, and were left with recommending users to revert to the old Qualtrics Survey Engine, for which we implemented a feature in the QRTEngine to automatically re-direct to the non-jiffy engine. However, it seems likely that the old Qualtrics Survey Engine will be retired at some point, which is a major threat to the method.

Unfortunately, several recent attempts to contact the Qualtrics team were not successful. So we do not know when the old Qualtrics Survey Engine will stop being supported. However, we got the impression that Qualtrics is not willing to support anything QRTEngine-related anymore. Repeated reports of sudden bugs in their engine were typically also not taken seriously, although they sometimes “spontaneously” disappeared, only then to resurface in new forms once again later on.

We also recently learned from users that the Qualtrics data handling platform will be moved to the so-called “Qualtrics Insight Platform”. Data output by this new platform seems not to be compatible with the QRTEParser, although – as one of you suggested at the forum – you can try to remove the 3rd row of your output csv file and see whether in that case the data is still parsed correctly.

Considering the minimal support we can expect from Qualtrics and the fact that our team is running this project in spare time without funding, we think that we are unable to continue the project in a sustainable way.

So our advice would be to finish up your projects running QRTEngine and start to look for alternatives – although we have to admit that we are not aware of any other free online experiment software yet that is easy to use without requiring extensive programming skills. Anyway, if you are a coder and like to integrate our code in another open survey environment, let’s try to team up and let us know at support@qrtengine.com

In case you have support questions, please use the forum and ask the community for help. We will keep our site and the online parser alive at least until the end of 2016.

We still believe in online research and that this the way to go for future psychological research. We hope that the QRTEngine helped in getting your online research done. We thank you for your interest in our method and apologize for the QRTEngine coming to its end-of-life so suddenly.

We noticed that most of the issues related to the parser are due to problems installing the required dependencies, and as such have decided to release this webservice, to increase the ease-of-use of the QRTEngine method.

We have taken a few security measures, aimed at minimizing the risk associated with file parsing, enumerated below.

We generate a highly probably unique link for each file you upload, by generating a “Universally Unique Identifier” (https://en.wikipedia.org/wiki/Universally_unique_identifier) for each of them. This download link is only communicated to you through the e-mail you provide. This should prevent anyone from being able to successfully, randomly, guess the correct download url of a file.

We delete any upload files, log files and parsed results that have been on our server for more than 24 hours. Both to prevent our hard disk from filling up, and to prevent as much as possible the accidental leakage of sensitive data on our end.

DISCLAIMER: we do not guarantee the safety of your data. This solution is aimed at data sets for which the leakage of it would not reveal any sensitive information regarding persons. For sensitive data we recommend reviewing the source code of the command line parser, and using that instead.

Offline (CLI) parser

As a result of releasing the new version of the webparser, we have also slightly updated the offline parser. We have made it clearer that it only runs on Python 2.7 (we never got the Python3 parser working properly on all data files) and have included instructions on how to run/install Python2.7 on your system, if it’s not already installed, in the README.md.

Also, older versions of the parser had the following error happening consistently:

JavaScript

1

ImportError:No module named builtins

builtins was required for Python2/3 hybrid support, but since the parser has been made Python-2 ONLY, builtins is no longer required, and as such this error should have been fixed.

Next to these changes, we have introduced two new variables in the output data file called “Parser[Version]” and “Parser[Type]”. Parser Version denotes the version of the parser that was ran, currently this is version 2. Parser Type denotes the method which was used to parse the file. “CLI” for the offline version, “WEB” for the webparser.

As a result of the Qualtrics JFE (Jiffy) update in September, the QRTEngine has ceased to work properly, not just v16, but also, as we discovered recently, v15. For example we found that at the start of the trial a flash occurs where all objects can sometimes be shown briefly, Qualtrics server delays are accompanied by a Qualtrics swirl and stimuli are no longer vertically centered. Several attempts at fixing this have proven unsuccessful, because of unknown processes happening during the switch from one trial to the next.

Recently, one of you notified us of a way to turn off this new JFE Engine so that we can still work with version 16 (basically by adding ?Q_JFE=0 to the URL). We therefore launch a new version of the QRTEngine (v18) which is based on version 16, but it also automatically re-directs to a JFE disabled version of the survey by adding Q_JFE=0 as a GET parameter to the URL. We also made a few bug fixes (embedded data was not being stored in non-qrte questions, and we corrected pre-caching of images which seemed to work only intermittently before).

The problem with disabling the jiffy engine is that we’re back to our current situation, as soon as Qualtrics decides to cut support for their old survey engine (and the method with which we disable the jiffy engine). We will ask Qualtrics how long they are going to support the JFE disable option, but let’s hope it will be there for at least a couple of months.

We’ve come to the conclusion that fixing v15 (which would become v19) for this new JFE engine will be very difficult, also due to a major lack of time on Erwin’s part. In addition, Qualtrics needs to provide currently unimplemented javascript APIs to take care of the between-trial stuff that’s happening, and the timeline when these APIs will be released is currently unknown. In the past, support people from Qualtrics actually promised us that the introduction of the jiffy engine would solve issues of between-trial delays. However, the sad reality is that their update actually seems not have improved the timing between loops (trials) in a Loop & Merge block at all, and that it only improved the timing within a loop (which was handled by the QRTEngine already in the first place).

As such, there is no telling when v19 would be released, and whether this will be done at all (if you are interested in joining our team as a developer, please let us know). However v18 will work for the time being, and we recommend to stay tuned for news about v19. We are not planning any changes to the QRTEngine API, so surveys built for v18 should continue to work in v19 (barring changes to survey structure induced by Qualtrics).

In Qualtrics’ most recent update the Survey Engine now caches Loop & Merge questions (uses the new Javascript Form Engine;JFE or Jiffy for insiders). Although in the long run this will cause significantly better performance of surveys (much lower ITI delays), in the short run this means that the QRTEngine is temporarily no longer compatible with Qualtrics’ Survey Engine. Unfortunately, Qualtrics didn’t offer us a chance to test the QRTEngine against this new version of Qualtrics’ Survey Engine and there seems to be no way to temporarily load the old Qualtrics’ Survey Engine, so we now have to play catch-up. We will keep you updated on our progress, and hopefully have the QRTEngine working again by next week.

If there are programmers that want to be involved in getting the new QRTEngine running again, please let us know at support@qrtengine.com

Thanks for your understanding.

Best regards,

The QRTEngine team

Update 13 Oct 2015 – Jonathan:

Erwin has completed a first work-around that brings back the fundamental QRTE functionality:

We realise that a lot of issues arising in the QRTE method happen due to the workings of the java parser. Either the java parser doesn’t work at all on your system, it is buggy, or the input file is generated from a faulty survey and the parser doesn’t give a clear reason why it fails. For these reasons, we are in the process of creating a parser webservice, which can be found here: https://github.com/ErwinHaasnoot/qrteparser-webservice.

Although the webservice is not yet finished, the parser we wrote for it is in a much better working condition than the java parser, and as such we will be retiring the java parser in favour of the new one written in Python, which seems to be de-facto language for much scientific software being written nowadays.