"I have a 4x2x3 design and want to include both the STROOP and the
WCST with autistic and typically developing participants. I have just
seen your website. Would this work with PsyToolkit?" → Please try to
find out the answer yourself first. Also, I am not a replacement
supervisor, so do not ask complex questions about how to set up your
survey (unless you are willing to make a donation and the question is
clear and you are willing to read up yourself first).

"Can you please send me more information about a survey on your survey list?" → Everything I know about the surveys comes from the sources I list. Please check those yourself (download the paper, etc). I cannot send original papers, unless I am the author on them.

Psy is short for Psychology and Toolkit refers to the various
software tools PsyToolkit comes with.

What can one do with PsyToolkit?

A lot. Apart from this website with lessons and documentation, you can
setup your own online or laboratory-based studies. In short, you can
program and run cognitive psychological experiments, you can run
surveys, and you can do this all offline or online, including online
data collection.

Can PsyToolkit measure response times (RT)?

Yes.

Can I use PsyToolkit pre-programmed experiments and surveys for my university research?

Yes.

I want to XYZ (replace XYZ with the description of your study), can I do this in PsyToolkit?

Can I run a questionnaire and include an experiment as shown on the PsyToolkit website?

Yes, and it is reasonably easy to do. This tutorial explains best how to do it step by step.

Does PsyToolkit work with MTurk?

Yes, that is very easy. In MTurk you set up a "survey link" project,
and you use the survey URL from PsyToolkit. Make sure that at the end
of the survey, PsyToolkit participants get an end code (or
alternatively, design your own random code). It is trivially easy. There is a complete PsyToolkit lesson about this topic:
MTurk lesson

Can PsyToolkit exchange variables with other websites, such as Qualtrix/SONA (etc)?

Yes. PsyToolkit can communicate with Qualtrix or Sona using so-called
CGI variables — if you want this, please
contact the PsyToolkit developer for
details, you will get an answer within one day. There is a fee to use
this feature to support PsyToolkit (after all, you can perfectly well
run online studies without these external websites, it is only for
those who can already pay for expensive systems like Qualtrix/SONA,
etc).

Are you on the social media?

All important PsyToolkit information is on its own page, so there is
not that much need. But PsyToolkit is on various social media for
documentation (YouTube) and news (twitter), and likes (Facebook). Here
are the links:

There is a lot of free software available, and there is a profound
philosophy behind
it. Read more about
this here. Free software makes it more accessible to use, especially
for students.

Where is PsyToolkit based?

The PsyToolkit webserver is based in a professional data center located in Strasbourg (mainland Europe).

Who pays for it?

The project depends on donations, for example to pay the data
center. You can support by making a
donation here.

Who uses PsyToolkit?

PsyToolkit is being used around the world by researchers and students. Many psychology
students use it for their research projects.

What is the difference between Eprime, Superlab, and PsyToolkit?

There are many similarities, but also differences. PsyToolkit is on
par with those packages. But unlike purely commercial packages,
PsyToolkit aims to be educational and accessible, that is, there are
many demos and surveys on the websites psychology students can learn
from. You can run your whole non-commercial project without paying
(but donations are welcome).

Does PsyToolkit work on tablets and mobile phones?

Yes, surveys can easily be run on mobile phones. Experiments too, but
because experiments often use a keyboard, you need to decide whether
you want to use key strokes in your experiment. Response times from
touch screens are less precise than key presses.

Is the response measurement timing reliable?

It is different for the web based version and the Linux offline
version. The latter is super precise. For many experiments, you
probably do not necessarily need such high levels of precision. Yet,
it is good enough to reliably measure relatively small effects, such
as Inhibition Of Return or the Simon Effect (i.e., effects of less
than 50 ms effects).

How can we support PsyToolkit?

You can do one of the following things:

Write about the software on the social media

Send me an email about what you like about the software, I am always
interested to hear positive stories about people who benefit from
the software, and it motivates me to expand the software.

Write a more formal letter of support to me explaining how you use
the software and why you think it is good. Ideally, this letter
would be sent as PDF. Such letters help me to get the support within
my organisation.

Help with translating the survey interface into your language, if
your language is not yet available.

Can you implement feature X, please?

I am always open to suggestions of new features. Just send me an
email.

I have sent you a question, why did I not receive an answer yet?

I no longer reply to questions about permissions to use (see here why), because they are all answered in the FAQ. I do not answer to questions that are answered here in general. Sometimes, I am too busy and it may take time. I generally try to answer, but sometimes they get burried under the many newer emails I get. If you still have not found an answer and it has not been addressed in the FAQ, please try again.

How can I delete my account?

This is very simple. There are 2 completely different ways to do this:

Just do not login for one year. If you do not login for more than
one year, you will be reminded by email to login. If you do still not
login, your account will be deleted.

Go to settings, and select "delete your account". You then receive
an email with a code you need to enter here to finish the process.

All legal questions, including permissions for surveys, etc)

There are many questionnaires on PsyToolkit’s website. If you have a
question about them, please make sure you read this first
carefully. This because I get many of these questions, I will often not be able to answer them — that is exactly why I created this FAQ.

Replace the XYZ below with whatever scale you are asking about.

I want to use one of your questionnaires. Can I get your permission to use it?

I get this question quite a lot. Here are the rules:

Whenever you use PsyToolkit, you need to cite its two publications (see above)

Nobody is allowed to use PsyToolkit for commercial purposes or as part of a business, unless explicit permission has been given by me (this is rare).

Students can use PsyToolkit for educational or research purposes without further permission

The surveys in the survey library are almost all based on work by resarchers who have absolutely nothing to do with PsyToolkit. As PsyToolkit developer, I have personally checked that it appears (to the best of my knowledge, which is unfortunately not infallable) they can be used without violating copyright. In every online survey, there is a section about legal stuff. That is to the best of my knowledge. If you don’t trust it, please check yourself with the original study and its authors.

My supervisor wants that you write me a formal letter to tell me that I can use the questionnaires on your website. Can you please do so?

As said, the rules of permission are on the website. Tell your supervisor to have a look there. Also, please understand that I simply do not have the time to write letters of permission to the thousands of people who are using PsyToolkit. If you insist, please make a donation of at least £50, and you will get your letter.

I want a hard copy of XYZ’s scale, can you please send it because I need it for my dissertation?

Because I do not own the scales, I cannot redistribute papers about them. You can typically find them in your library or sometimes even on the Internet.

I want to use the XYZ scale with Google Forms/SurveyMonkey (etc), can you please tell me how I best do this?

I can only answer questions about PsyToolkit.

I do not get a response from XYZ via email, can you please help me instead?

Some of the authors of the scales will either be retired or disseased. I am not in a position to answer on their behalve.

If I create my own survey (or experiment), whom does it belong to? Do I loose my copyright by putting it on the PsyToolkit website?

It still belongs to you in every respect. You do not loose your copyright. If you made it, it belongs to you.

You show the means, but not the standard deviations, why not, and can you send them to me?

When I write about surveys and questionnaires, I like to show some averages. I keep it simple (no standard deviations). The details are typically in the papers I cite. You can find them at the bottom of each page in the survey and experiment libraries (under references). If you want the details, please get the papers yourself via your library (I will not send them to you due to copyright issues).

About online experiments

After running my experiment, I can see the data, but how can I save it to my harddisk?

It depends what exactly you want. There are different scenario:

When you just want to program and test your experiment, you cannot directly "save" your data to harddisk. You need to copy and paste it. This is due to security mechanism of the browser.

If you want to run an online experiment as part of online data collection, you can embed your experiment in an online survey. That way, the data will be saved on the PsyToolkit server, and you can download them from there. Check the following question.

I have my experiment. But how do I get my experiment "online" so that people can "click" it?

This is very simple. It takes a few steps:

First make sure your experiment on its own works well.

In addition to your experiment, you also must create an online questionnaire.

The online questionnaire needs to embed your experiment, that is
very simple to set up (see example below).

Now compile your survey and then you get a URL which you can
distribute to people.

Imagine you have an experiment called "stroop". Then you can embed in your survey as follows:

l: myexperiment
t: experiment
- stroop

You can read about this in detail about how to get your experiment online here.

Is the scripting language for online experiments different from the Linux version?

No, it is the same, although some features are designed only for the
Linux version (such as external keyboards) and some features only for
online experiments

How can I collect data from multiple participants?

This is very easy. You need to run the experiment within an online
survey. You need to set up an online survey and embed your experiment
in it. Then you get an online web link (URL) and you can give that to
your participants. All the data will be collected online and can be
analyzed within the PsyToolkit website. A complete lesson on how to do
this can be found here.

Which browser should I use?

The online PsyToolkit version works fine with all modern web
browsers. The PsyToolkit developer primarily works with
Firefox, which is probably
the best available browser.

At the beginning of online experiments, there are onscreen messages such as Click to start and Wait a second. Can I change these?

Somehow, only the first keypress response during the experiment is processed, and then the browser does not respond anymore?

This is a very unusual error that can happen under one very specific circumstance when using Firefox. If users select in the preferences the option to start searching in the web page "Search for text when you start typing" as soon as they type (which people rarely use), then the keyboard responses are not properly processed.

Online questionnaires

How long can surveys run?

Currently, they can be active for as long as you want, but there
are no guarantees.

Can you support my language?

Currently, PsyToolkit supports many foreign languages. Contact me if
it does not support your language. It is relatively easy to add new
languages, as long as someone is willing to translate the basic
phrases used.

Will surveys stay valid if the software is updated?

Yes, the old versions will stay active. If you compile a survey
with a specific PsyToolkit-for-the-web version, this will remain
active, even though the latest version of PsyToolkit might have new
or changed features. This makes the life of researchers easier.

Which browsers are supported?

All modern browsers, including Internet Explorer version 10 and
higher. Embedding experiments with keyboard input requires a real
keyboard (touchscreens won’t work very well or not at all).

Is there a welcome screen for surveys?

If you create a survey, you can put welcome-information in the various
text boxes under survey intro screen.

Can I upload multiple files at once?

Yes.

Where can I add information for participants (such as plain language statement, etc)?

If you create a survey, you can put this information in the various
text boxes under survey intro screen.

How do I put my university logo on the welcome screen?

Here are the two steps how to do this:

Find the weblink to your logo, such as http://my_university.edu/logo.jpg

In the "About this survey" section, enter <img src='http://my_university.edu/logo.jpg'>

Save and compile and it should show up

What sort of survey items are available?

There are many different items available, including Likert scales,
choice questions, text boxes, embedded experiments and videos,
etc. Check the online documentation under
Question Types. You can
also copy and paste existing scales from the
survey library.

At the top of the web page, the percentage completed is
indicated. This might not always be accurate if you include
conditional elements (i.e., you cannot predict how long the survey
will last if certain questions might be skipped depending on how some
questions are being answered).

I have a "check" question, but I would like that people can enter some text if they want to choose something that is not on my list. Can I do that, and if so, how?

There are a number of cognitive tasks in the online library. You will
still need to understand data analysis techniques to extract the data
you want. If there are special tasks you are interested in, you can
program them yourselves, or make a suggestion to me to implement it.

How can I invited participants to participate in the survey I have created?

When you create a survey and select it to go online, a URL link is provided. You need to send this to your participants yourselves or advertise somewhere on a website, the PsyToolkit website will not do that for you.

What sort of format will the data be stored in?

The data will be available in a raw format (text file) and in the common spreadsheet format CSV, which can be opened with all spreadsheet software. Excel export is also supported.

Can surveys be programmed so that participants get feedback about their own answers at the end?

Each survey you create has one survey link. This means, in essence,
that participants are unidentifyable. You can, of course, ask people
to identify them with a question about their email. Also, you can give
people an identifier code at the end of the survey, which they can
then use for course credits,
etc. Check here
how that works.

Can you run surveys in different languages?

Yes, the text on the buttons and so on can be set to quite many different languages.

Under download, you can specify that you want to add Excel files, which can be easily and directly read into SPSS.

I have a survey with an embedded experiment. How do I download the mean response times and error rates of participants?

You need to make sure that you specify the parameters of the embedded experiment and then download again. You can do this after data collection. Click the blue questionmark button at "Download participant data" for detailed instructions.

Can I have multiple questions on "one" screen?

No, that is currently not possible. It is rarely an issue, and the current system has advantages, because you can get the exact response time for each question answered. Even so, having multiple question items together is high on the todo list.

Is there a way of knowing the response time of each question in the surveys?

Yes. When you download your survey data, you will find a file named data_times. This file contains the response time for each question (in milliseconds). That is, the time from when the question is being displayed until the participant clicks the continue to next question button.

Which external website can I use to upload large sound or image files?

You can use dropbox (create a dropbox account first). Basically, you
want to do this as follows:

Upload your file.

Select "Share the file."

In the share dialog, select "Copy the link"

Now you can copy and paste the link to your PsyToolkit script

You are not yet done yet! The link as given is not entirely correct, you need to make a small change:
You must make absolutely sure that at the end of the link, you look for the question mark. Remove anything that follows in the link after the questionmark and make sure the URL ends with ?raw=1

When running the survey, people get the following message (example below): "Error: The survey has been created with a different version of PsyToolkit that seems to be referred to. Please click the following link to continue this survey"?

This rarely happens, but it sometimes happens to people. First of all, in principle you can ignore it, because the link in the error message leads to the correct link. But you can easily get rid of the message altogether. Let me first explain why it happens. Imagine you compile your survey with version 2.5.1 and you send participants the survey. Then later, you update your survey and recompile it in a newer PsyToolkit version, for example 2.5.2. If people still have the old link, they will get this message. If people click the correct link in this error message, then everything will be okay.

The solution is simple: Once you have sent a specific survey link that belongs to a specific version of PsyToolkit (for example 2.5.1), you need to stick with that version. You can still login to older PsyToolkit versions, they will stay active for a long time. In order to make sure people do not get the message, go back to the older version and recompile.

Data storage

Where are online data stored?

Data are stored on the PsyToolkit webserver. This server is based in a
Central European professional data center (Strasbourg).

Who manages the server?

The server is managed by Professor Gijsbert Stoet (PhD, Chartered Psychologist based in the UK).

In your experience, have IRBs/Ethics Committees generally approved of PsyToolkit’s use, the security of the server more specifically?

Yes, this has never been an issue.

Can stored data be lost?

This is very unlikely, because data are being backed up daily. If you
delete a file on the server though, the backup will also delete it
within 24 hours (so if you want it back, please contact me as soon as
possible). PsyToolkit does not give any guarantees, though (see next
point about what you can do to always be on the safe side).

Can I store the data myself?

Yes, you can download all your surveys, experiments, and data via the
web interface. If you are concerned, just download your data
yourself. As said, it is unlikely to get lost, but if your data is
important to you, I recommend you to make sure that you make regular
downloads. It is easy to do.

Who has access to the data collected in online surveys?

Only PsyToolkit developer Professor Dr. Gijsbert Stoet has access to
all data, because he manages the PsyToolkit server.

Is there any "no data loss" guarantee?

Although data loss is extremely unlikely because of the distributed
backup system (see above), you are responsible for your data. There
are no guarantees given by PsyToolkit. Please note that you can
download all collected data anytime to make a backup copy on your own
computer at home. This is recommended.

Are online survey participants informed about data storage?

Online survey participants are informed about data storage at the
bottom of the welcome survey of each online survey. It is possible for researchers not to show the welcome screen. The researchers need to choose how important this information is to them (there may be situations where the information is provided elsewhere or differently).

Is PsyToolkit GDPR compliant?

The GDPR is about personal data storage. By default, data are not linkable to real persons, unless you ask people to give their name or other personal identifiers in the survey. If you do not ask for personal identifiers, GDPR does not apply anyway. But if you do ask for personal identifiers, please note that you are the only person who actually uses the data. The data will be stored on the system until you delete it. It is up to you to delete the data when needed (you can do so).

Is the IP address or country information of participants stored?

I sometimes get asked this because ethics review boards migth not like
it if you store the IP
address of a participant. Therefore, PsyToolkit gives you the option
to save or not save the IP address of the participant. Please note
the following as well: Some people overestimate the capacity to
identify someone using an IP. In reality, IPs are not as useful as
some people think they are. Even though IP addresses are linked to
individual computers, it is typically practically impossible to
identify who uses an IP. In theory, there will be law enforcement
agencies who can, with time-consuming detective work, find out where a
participant would be located, but the reality is that in practical
terms, the IP address is still pretty anonymous and of little use
except for finding out in which country the participant is. You can
read more about this
here.

Does the server use secure transmission?

Yes, the server uses the secure version of HTTP, the web protocol used
for communication between computer browsers and servers. This means
that all communications are encrypted. In many browsers, this is
indicated with a "padlock" symbol in the addressline.

I did an onlineline experiment, but I cannot download the data. Where are the data stored for later download?

If you program your own experiment, you can compile and run it from
within the experiment editor screen. At the end of the experiment, you
will see your data, but these will not be stored on the PsyToolkit
server. You will, however, see your data at the end of the experiment
in a table below the experiment.

If you want that your experiment data are being stored on the
PsyToolkit server, you need to embed it in a survey. Then every time
you do the experiment (as part of the survey), your data will be
stored on the server for later analysis. For details how this works,
please
see
this lesson.

Writing experiments

Some Q&A about how to set up your own experiments.

What is the output data format of experiments?

The output data format is ascii text. Further, it is entirely up to
the designer of the script which variables are being save (with the
save statement).

In designing an experiment on your platform, I noticed that the blocks that call the table replace each trial that is called (i.e in one block it would be theoretically possible to get the same stimuli for all trials). Is there anyway to set it to where the block calls stimuli without replacement?

Linux version

PsyToolkit experiments can be compiled on Linux computers. This
is only of interest for some people. If you do not know what Linux is,
please just skip this section altogether.

Is there a recommended Linux distribution?

No, PsyToolkit for Linux should run on any modern distribution. Mint Linux is one of the most popular. Linux is tested mostly on Fedora, Mint, and Ubuntu. The PsyToolkit developer works with all these 3 Linux distributions.

Where can I download the PsyToolkit software package for Linux?

See the Linux page of PsyToolkit for details how to get the install
files. Click
here.

Will you implement this on other platforms, such as Mac or Windows?

No, but there is other free software available for those platforms. Of
course, you can also just run the web-based version on Windows or Mac.

Which external keyboards does PsyToolkit support?

Cedrus, BlackBox and IOlab (although these latter ones are probably no
longer being made or sold). You can also build your own keyboard and
use the parallelport ().

Does PsyToolkit work with the new Wayland graphics stack?

Yes. PsyToolkit uses the SDL (version 2) libraries for graphics and
sound. SDL supports Wayland. Of course, Wayland is the default
graphics stack on Fedora Linux, which the PsyToolkit developer works
on most.