As some of you have tried, and reported back to us, v15 of the QRTEngine seems to run fine on the new Qualtrics JFE engine. However, embedded data wasn’t stored when using this version. This issue has now been taken care of, and we suggest everyone to switch from v16/v17_jfe_fix to v15_jfe_fix. The js file can be found here:

As you may have noticed, fixing the QRTEngine has taken longer than originally planned, due to complexities involved in the recent the Qualtrics update.

There seem to be a myriad of different problems, and it’s hard to keep track of individual cases through comments/forums/posts. Therefore, we suggest any issues with the current version of the QRTEngine to be posted to our issue tracker on github, https://github.com/ErwinHaasnoot/QRTEngine/issues. Make sure to include a description of the problem in the post and a link to related exported surveys (.qsf files). Or if uploading the .qsf is not an option, you can share the survey with me at ehaasnoot1[at]sheffield.ac.uk (make sure to mention the name of the survey in the issue you’ve created). Please note, this is the e-mail my Qualtrics account is registered to, I no longer have access to the mailbox.

The same goes for issues with the parser. It’s possible that due to recent updates in the Qualtrics Survey Engine, the file output structures changed, which would affect the parser as well.

I’m hoping to spend more time tonight, tomorrow night and over the weekend to tackle these issues, so hopefully we can get a broad range of issues in the tracker, which would allow me to test possible fixes as thoroughly as possible.

We are very sorry for any inconvenience caused by this and would like to thank you for your understanding.

A lot of people have recently (for about a week) been having issues with the QRTEngine, regarding false positives in one of the survey validation functions. Specifically, an alert box popping up right at the start of a trial indicating ‘You need to define two answer form fields in the Exit questions!’ even though there are two answer form fields included in that question. We have investigated this issue and found that this is due to a bug recently introduced by Qualtrics in one of its JavaScript APIs called getChoiceValue. We have contacted Qualtrics to notify them of this issue, so hopefully it will be fixed soon.

The only work-around we can currently suggest is completely disabling the alert for now, by commenting it out. This implies that the engine is no longer checking whether the appropriate exit fields are defined, so if you do not do this correctly you may risk not acquiring the data correctly. So please be very careful and check your parsed data.

We have uploaded this temporary, version of the QRTEngine, that does this for you. So if you’re having issues with alerts being falsely thrown, please download the following QRTEngine source code and fully overwrite the QRTEngine version that’s currently included in your surveys.

We have just released v16 of the QRTEngine, bringing some major enhancements to the methods. Most prominent of which is the arbitrarily low Inter trial interval (ITI) fix. We have managed to off-load the request of the next trial to be done during the current trial, rather than at the end. If the trial is long enough (longer than the average time it takes to request a new trial), this should result in a significant decrease in the lower bound of the ITI. We have found that ITIs of 50 ms are within the realm of possibilities now, the exact value depending on the specs of the respondent’s computer, rather than on the connection speed of the respondent and its geographical location.

This fix removes one of the major limitations to the QRTEngine that we mentioned in the paper and in future versions of the QRTEngine we plan to bring the ITI lower bound down even more. However, the solution is not entirely risk-free. Our solution involves juggling HTML input elements across different pages, so that storage of the current trial is separated from the loading of the next trial. Although this is possible in the current version of Qualtrics, Qualtrics support hasn’t confirmed us yet whether they are continuing to support this feature in future releases. In order to protect the QRTEngine against this, we have turned off the option by default. In order to turn it on, add the following parameter to the Init question code:

JavaScript

1

loadDuringTrial:true,

As part of the ITI enhancement, we fixed the ‘fixation flash issue’ (causing the fixation to disappear for a frame or two after a page load, which is on by default) and as a result we also support background colors other than white now. This fixation flash fix works regardless of whether you turn the loadDuringTrial toggle on or off.

We have released the source of our Parser on Github, and it can be found here: QRTEParser. Feel free to clone and if necessary fork the project. As we’re looking to build a new interface in JavaFX 2.0, rather than SWT, only the command line version of the parser is available right now. JavaFX should make it easier to make a GUI that’s actually cross-platform, and would allow us to offer a Linux version of the parser as well. We will post more information about the inner workings of the Parser in a tutorial soon.

Website:

We have received a lot of support questions, which has allowed us to fine-tune the guides currently present on the website. Next to that, we have updated our FAQ with several extra items (most importantly, how to unhide standard Qualtrics questions) and are currently working on compiling the most common debugging strategies you will need for successfully building your task in the QRTEngine, which can be found here.

In a comment on our blog post, Robin van Emden made us aware of jsPsych, an alternative framework for online experimentation (see his comment here), and asked us how it compared to the QRTEngine. Please note that my findings are based on a necessarily limited, though hopefully representative, experience of working with jsPsych. I have tried to be as honest as possible about both methods, because I feel the jsPsych method in some cases is a serious alternative to the QRTEngine for online experimentation. Prior to publication, we gave Josh de Leeuw, author of jsPsych, the opportunity to review the post and have revised the document based on his comments.

Both our method, the QRTEngine, and the jsPysch attempt to fill up a gap that currently exists in online experimentation. There is a need for a method that provides a flexible framework for building experiments, that is highly precise in each individual run of the experiment (ie, approaches the researcher’s intended experiment with high accuracy on individual machines, not a trivial feat!), is easy to set-up, which makes the data storage and analysis easy and runs natively in browsers (no plug-ins required, preferably not even Flash). Based on my experience of building an experiment, I think that the QRTEngine and jsPsych both have different strengths and weaknesses in the aforementioned needs for a new framework. The following table quickly summarizes those strengths, and weaknesses. Strengths are denoted with a +, weaknesses with a -.

* = Will be implemented in jsPsych v4.0, to be released end of october.

** = Will be implemented in QRTEngine v17.

Please read on for a more in-depth look into why those strengths and weaknesses arise.

Flexibility

First off, the QRTEngine is heavily integrated with Qualtrics (to the point where it would lose a lot of functionality if taken out of this system). jsPsych is more a stand-alone client-side framework that, granted, is a whole lot more flexible than the QRTEngine, due to its independence of any specific (survey) platform. Personally, I like the modularity of the jsPsych framework, where plug-ins are treated as ‘templates’ for trials and trial blocks, and the ‘experimental flow’ is dictated by the order in which these templates are requested. In the QRTEngine, this would be similar to using Qualtrics’ library function to import blocks that implement certain paradigms, at will. However, Qualtrics doesn’t make this convenient at the moment, as you can’t ‘subscribe’ to any library at will, it is either account (which is limited), company (again, limited) or Qualtrics-wide (not everyone who uses Qualtrics is interested in the QRTEngine). On the other hand, as the jsPsych author himself notes, it is currently inconvenient to perform conditional and looping structures (come jsPsych v4.0, this will no longer be true) in your experiment and although QRTEngine doesn’t support loops at the moment (although plans have been made to implement support for those, through forward-skip and back-track logic), the QRTEngine does support a wide array of types of conditional structures, both through Qualtrics’ own interface and the within-trial QRTEngine interface. One major constraint in the QRTEngine still hasn’t been resolved, which is the impossibility of arbitrarily low ITIs. Between-trial page reloads take time, a problem which jsPsych doesn’t suffer from, as full trial blocks can be ran within one page-load. So, both methods have their ups and downs regarding flexibility of the framework, however jsPsych seems to win out on flexibility.

Accuracy

The accuracy of both methods is something where the QRTEngine, at the moment, seems to have an advantage. The architecture of the QRTEngine provides a central loop that not only goes through stimuli in trials (and trials) in sequence, but also directly provides highly accurate timings, through the use of the requestAnimationFrame (rAF) method (and less accurate timings through setTimeout where rAF is not available). rAF is a relatively new JavaScript feature that allows callbacks to be synchronized with the screen refreshes, making sure that the function is called no more than once each screen refresh. Next to that, rAF callbacks have a higher priority than setTimeout callbacks, which means that it is less likely that callbacks are delayed due to other ‘things’ happening in the webpage (keyboard/mouse responses, garbage collection). This all happens within tens of milliseconds, which makes setTimeout infeasible for experiments such as Masked Priming (which we reported on in our paper), but it should be fine for experiments that require accuracies of around 200 ms (or higher).

Currently, jsPsych does not use rAF, which immediately results in less accurate timings (only using setTimeout, which is similar to the method used by Crump, 2013), however an issue ticket has been created in the jsPsych repository by the original author to address this issue, which is set to be resolved by the coming version 4.0. So, as of this writing, if highly accurate timing within a trial is your main concern and you accept bad timing of the inter-trial interval, the QRTEngine seems to be your best choice. After the release of jsPsych v4.0, there should be no more major differences in the accuracy of timing.

Ease-of-use

Ease-of-use is usually a bit at odds with flexibility. On the one hand when creating an experiment, you want as few actions to give you an experiment as powerful as possible, however this usually means abstracting many low-level details away, reducing the flexibility of the framework in the process. I feel like the QRTEngine is slightly easier in use as less of the snippets actually has to be understood for the QRTEngine to set-up the flow of an experiment than for jsPsych, which requires you to string plug-ins together and to set up your own plug-in if the current plug-ins don’t provide a template for the paradigm you need (granted – it should, as the framework becomes more popular, happen less and less). As for creating your stimuli, I feel like the Qualtrics wysiwyg editor has a definite leg-up on jsPsych here. Being able to use Qualtrics’ wysiwyg editor allows you to 1) overview your current trial as you set it up and 2) reduces the technical expertise needed in writing stimulus content in HTML and styling it with CSS. Next to that, the QRTEngine integrates very well with every other aspect of Qualtrics, the resulting data file (after parsing) is in long format with the data gathered through the QRTEngine mixed in seamlessly with data gathered through conventional Qualtrics survey aspects. Thus, what the QRTEngine lacks in flexibility, in a sense, it makes up for in ease-of-use.

Data storage

There are two main parts of the data storage that I’d like to discuss. First off, there’s the within-experiment data storage and its API. jsPsych automatically stores all data related to previous trials and allows you to fairly easily recall all this data (even from several trials back). The QRTEngine’s data storage & API is modeled after E-Prime’s, where one is expected toexplicitly store any relevant information about previous trials in order to be able to use them in following trials. Frankly, I feel like the jsPsych data API and the scope of it is superior to the QRTEngine’s, so one thing I consider working on is extending the scope of the QRTEngine data API, hopefully making it easier in use (without breaking backwards compatibility). On the other hand, the QRTEngine does provides more extensive data reporting out of the box, such as onset and offset times of stimuli, durations (both intended and measured), onset times of responses and more.

Secondly, there’s the issue of aggregating the data of all individual participants into one big data file and then analysing that data. This, apparently, is outside of the scope of jsPsych; one needs a third party platform, as no native method is provided to submit the gathered data back to the server. The author of jsPsych has published a paper integrating jsPsych with psiTurk, which allows for easy data collection on mechanical turk. Robin van Emden has supplied a method to do this in Qualtrics by stuffing the data into an embedded data field. I can’t comment on how easy-to-use psiTurk is, as I haven’t tried it out, but the method Robin van Emden uses makes the jsPsych part of the survey fairly inconsistent ( the jsPsych data is stored in ‘long format’, whereas qualtrics data is stored in ‘wide format’, which would make building the final data file harder by hand) with the remaining part of the Survey. On the other hand, although the QRTEngine’s final data file nicely integrates QRTEngine data with conventional Qualtrics data, the parsing process necessary to get such a data file is one of the major weaknesses in the experimental flow of the QRTEngine. The parsing tool I feel, needs a lot of polish to make it, for example, useable on Linux and to make it more robust against failures (more transparent error logs are in the making). And the process itself (of parsing) is necessarily fairly complex, building the parser is definitely non-trivial. Here it is sort of a toss-up, do I set-up my own easier to use platform for collecting data with jsPsych, or do I use a method that relies on a complex parsing process that is difficult to troubleshoot if it fails.

Other points

A last point that has to be specifically mentioned is that the QRTEngine relies on Qualtrics for it to work. Although free Qualtrics trials are available, a full-blown license can be fairly expensive. Setting up your own web-server and running a jsPsych experiment is probably much cheaper nowadays, but it requires more technical expertise. As Qualtrics is in use by many universities for normal Surveys anyway, the issues become a bit less prominent.

Conclusion

If we summarize the previously mentioned points, we can conclude that each method has its own strengths and weakness. The jsPsych framework currently is better in terms of flexibility and within-experiment data storage, whereas I feel the QRTEngine is easier to use, both in designing your experiments (within what is allowed by the framework) and extracting a useable data file from the platform the experiment is ran on (even though it makes use of a complex parsing process to do so). Also, due to the QRTEngine using requestAnimationFrame to present stimuli, it is currently more accurate in terms of timings, and has superior reporting on these timings. The most accurate version of the QRTEngine is less well supported by older browsers, but it rolls back to an equally accurate version to jsPsych, if rAF is not supported by the browser.

Let’s conclude by mentioning that both jsPsych and the QRTEngine have a few steps to make when it comes to their development. However, currently, the QRTEngine requires less of a technical expertise from the researcher than jsPsych does. jsPsych, on the other hand, is more flexible in terms of which types of paradigms you can build, if you know a bit about programming (in JavaScript). Furthermore, the QRTEngine seems to provide higher precision timing than jsPsych does and more extensive reporting on those timings. If the highest accuracy is required, the QRTEngine would currently be the prefered method for running those experiments, but only if arbitrarily low ITIs are not necessary. If the highest accuracy is not a requirement, users interested in online experimentation should consider the remaining requirements of their current experimental paradigm, and choose whichever method fits best with those requirements, as both jsPsych and the QRTEngine have their strengths and weaknesses.

Note: jsPsych v4.0 is set to be released soon (end of October) and brings significant fixes to issues listed in this blog-post. I will update the post once I have a chance to test out this new version.

We are currently working out the kinks in the website behind the scenes, but have decided to open up our forums for general use. So please find your way to the forums! We will be there, providing support, but hope to see you share whatever you feel like sharing. See you there!

This website is intended to be the web “homebase” of the Qualtrics Reaction Time Engine (QRTE for short). We hope this website will be your one-stop shop for everything QRTE-related. On this website, we hope to keep track of all publications made with the Engine and post tutorials and examples of both beginner and advanced techniques alike. Next to that, of course, we will provide the necessary components (the .js file and the parser) for you to successfully run your own experiments online.

The goal we had in mind when creating the Engine was to provide a highly reliable and precise method and framework for creating and running online behavioural research experiments. We combined the ease of use of the online survey platform Qualtrics (which allows free trial accounts) with new HTML5 techniques that allow for very precise timing of stimuli in JavaScript. And all that’s required to unlock these features in Qualtrics is to copy and paste a bunch of JavaScript/CSS code in a window and stick to a certain design pattern!

Our paper has been accepted for publication in the journal Behavior Research Methods and reports three experiments that, in combination with Amazon’s Mechanical Turk, show the possibilities of online experimentation. With the last of those three experiments showing that the engine can even be used to study subliminal priming!