In this post, I collect and log my experience in developing an online experiment using jsPsych. I am implementing a series of experiments related to the novel-word learning paradigm described in Ripollés et al. (2018), but this will capture the work I needed to put in to get the first one up and running the data collection.

My Background in all things “web”

Since your mileage might vary depending what you know, let me briefly sketch what I knew before starting with jsPsych for this project.

Back in 2010, I ran an online experiment. We recruited participants via MTurk and the participant-facing part was programmed with Flex/Actionscript as part of Adobe’s Flash (don’t judge me.. even youtube was still using flash back then). The backend was running off a LAMP webserver hosted by my uni. Prior to that, I had been writing a blog (also on the same uni webserver) where I mostly had edited the HTML/CSS files by hand.

Afterward, I tried to learn a bit of Ruby on Rails and create my own To-Do list manager with Evernote as its back-end (one version of the code is still on github..), but I didn’t get very far.

In short, I would say that

  • I was familiar with HTML/CSS/Javascript, but I didn’t actually write any for a few years
  • I can copy/paste and understand some basic php and can interact with SQL-Databases via phpMyAdmin
  • I understand basic aspects of htaccess/htpassword as means of password-protecting parts of a website
  • I understand potential issues related to experimental design/random assignment to conditions in the online-context
  • I have a decent programming experience in other programming languages (Matlab, R, python, etc.)
  • I have programmed several offline (i.e. lab) experiments using Psychtoolbox in Matlab and understand issues related to timing accuracy, randomization, etc.

Reading through the documentation of jsPsych and watching the tutorial videos, I would say my knowledge is about right: I knew the basics of web-stuff, but I was missing the tools to implement a working experiment in a browser. I have at least superficial knowledge of more server-side stuff (e.g. I’m hosting this site with nginx, not apache, and I am running some dynamic parts off docker-images). But they are not relevant to the development of online experiments, so far at least..

Why jsPsych?

There are several tools out there to conduct online experiments. Even though my immediate needs are probably met by most tools, I have a list of desiderata for future experiments:

  • accurate timing (ideally including relative timing of visual and audio inputs)
  • microphone input
  • eye-tracking via webcam

jsPsych is starting to support both eye-tracking and mic-input. Further, there was a recent meta-study suggesting that the timing accuracy of jsPsych is relatively good. In the end, I chose jsPsych because it can be completely self-hosted and is pure Javascript. The last point was mostly so that I do not need to bother with writing e.g. the experiment in python, and then getting translated to Javascript (I’m looking at you, psychopy - it has both vendor-locking for the host and translation..).

What do I need to run an experiment with jsPsych?

jsPsych’s requirements are as simple as any online experiment’s can be:

  1. webserver which hosts the (static) HTML, CSS, and Javascript files which contain the experiment
  2. a “dynamic” server where you will store the data from the participants

Note that jsPsych is a tool for writing the Javascript files which actually contain the experimental code and run it on a participant’s computer. In a sense, the tutorials/documentation of jsPsych “only” describes the files which will be loaded by the participant’s browser.

How the data is sent back to the server for storage is only briefly hinted at (but with handy code-examples in php for how to handle it on the server). That is, how to store the resulting data in a database needs to be written “outside” of jsPsych. Same is true for how to download that data from the database onto a machine for analysis.

How precisely you need to do that will heavily depend on the webserver you will be using for putting the files online. In my case, my current uni provides a simple linux webserver with php and MariaDB. For that, I was able to adapt the php-examples from the jsPsych website.

Learning jsPsych

The documentation of jsPsych is great and the youtube videos by its main developer Josh de Leeuw were easy to follow. I would recommend to first watch the videos and then go through the written documentation. The videos gave me a good intuition of jsPsych terminology and key concepts.

There were two terms with somewhat surprising meanings in jsPsych:

  1. a trial
  2. a plugin

What jsPsych considers a trial did not match with my notion of trial. Let’s think this through with an example of a simple Stroop task: I would describe a trial in the Stroop task as a sequence of several screens (like powerpoint slides), namely a screen with a fixation cross, then a screen with the stimulus, and finally a response screen. They all are conceptually part of a single trial for me. Then might come an blank inter-trial pause, and a new trial would start with a new stimulus screen and a new response screen. For jsPsych, a trial is any such “screen”. So, the above Stroop task trial would be 3 jsPsych-trials: a fixation-trial, a stimulus-trial, and a response-trial (and potentially a blank pause trial). Those can be grouped and handled as a unit, but such a grouping is implemented via a timeline - an array of trials.

Similarly, the terminology of “plugin” was a bit surprising. For jsPsych, any such jsPsych-trial is implemented by a plugin - a dedicated Javascript file describing a Javascript class which must implement a certain set of functions and properties (cf. plugin development). Different types of “screens” are implemented by different plugins. For example, a simple “show stimulus and wait for a single button response” type of screen is implementd by one plugin. You want participants to type a word into a text-input field? Use a different plugin. In short, every jsPsych-trial has the property type and it has the value of a plugin-name.

But even though writing your own plugins might sound intimidating, it is not really. For example, I wrote a custom plugin by taking an already existing one and extending its functionality with code I found in another (specifically, I needed an survey-like input field which could be set to have a maximal response time-window).

Once you have a clear experiment in mind, I suggest to simply start adapting what you learn in the videos to your own experiment. And look through the documentation as needed.

NB: I started developing my experiment with jsPsych version 6.3, but updated to jsPsych 7.0 after it came out. Not sure whether the videos were updated for the new version of jsPsych. But even if not, the switch from 6.3 to jsPsych 7.0 was not difficult and is described on a dedicated page of what needs to be changed. So I would still recommend to watch the old videos first, but maybe always double-check with the written docs so it matches your version of jsPsych.

NB2: I just updated to jsPsych 7.1 and noticed that some parts of the documentation are not matching the actual code (jsPsych.data.getInteractionData() returns an array with event, but the docs mention type). Hopefully, more people switching to the new version and report such errors on github will solve itself in the coming weeks.

The non-jsPsych stuff

This section is potentially highly idiosyncratic since it depends on my uni’s webserver settings. Yours will most likely quite be different..

php to database communication

At the latest at the end of the experiment, the participant’s data needs to be sent back to “you”. The most common way of doing this is sending it from javascript to the same webserver from where the experiment files were provided to the participants. There, a script, typically a php script, is able to receive the upload and somehow processes it. One way would be to simply create a new file and store that there. The more usual way is to write it into a database. Both ways are briefly described on the jsPsych website.

In my case a php-script (mostly copy/pasted from jsPsych’s documentation) receives the upload from Javascript and stored that info to a MariaDB table I created in advance. The code provided by jsPsych silently ignores any data which it could not write to the database. On the one hand, this is good, because malicious people cannot start putting easily data into your database, but to avoid loosing data due to some weird bug, I also log any such attempts on the server - without reporting it back to the participant.

Tip: During development, a simple way to assert that all data is being stored is to use the function from the tutorial to display the full data in the browser in csv-format (jsPsych.data.displayData('csv');) - in addition to uploading it to the database. Then, save it manually to disk, and compare the content with what was stored in the database. I wrote myself a simple R-notebook for this.

Error reporting

TL;DR: do log all kinds of errors/warnings/notices, but display only very generic “sorry, something went wrong” to the participants.

While googling about how to properly connect to the database, I found the blog phpdelusions. Apart from an excellent discussion on why you should use prepared statements when interacting with your SQL-Database, it also has a post on error reporting. I have implemented the recommended workflow as good as I could.

Limiting the number of participants

Depending how you plan on recruiting participants, you might need to program a “study is complete” yourself. The core issue is that per default, anyone who knows the link to your experiment can do it. At my instituition, I wold need to pay anyone who finishes the experiment, no matter whether I have already reached the desired sample size or not. In addition, I am planning a series of experiments but I want any given person to participate at most in one of them. So, I try to avoid recruiting too many participants for the any given experiment (and spending too much money).

To this end, I created a simple “landing page” in php, which checks in the database how many people have finished the experiment and let’s you proceed with the experiment only if there are participants missing. Actually, it also checks how many started “recently” (i.e. within the typical duration of the task) and counts them temporarily into the “already participated” counter. Depending on whether the number of “actually done” or “including those working right now” people surpases the desired sample size, new people would be shown a page stating that they might want to come back later, or that the study finished collecting data.

In either case, I told participants that I will have more experiments soon, and that they can come back later.

Next steps

Currently, I’m analyzing the first batch of data. I will write a dedicated post on my lessons learned in the coming days and some potentially generalizable observations.

References

Bridges, D., Pitiot, A., MacAskill, M. R., & Peirce, J. W. (2020). The timing mega-study: Comparing a range of experiment generators, both lab-based and online. PeerJ, 8, e9414. https://doi.org/10.7717/peerj.9414

Ripollés, P., Ferreri, L., Mas-Herrero, E., Alicart, H., Gómez-Andrés, A., Marco-Pallares, J., Antonijoan, R. M., Noesselt, T., Valle, M., Riba, J., & Rodriguez-Fornells, A. (2018). Intrinsically regulated learning is modulated by synaptic dopamine signaling. ELife, 7, e38113. https://doi.org/10.7554/eLife.38113