Does it matter if we use student participants in our software engineering experiments?

Recently Davide Falessi and co-workers published a paper entitled “Empirical software engineering experts on the use of students and professionals in experiments” [1] in which they make some quite strong claims about the benefits of using students as opposed professional participants in empirical software engineering experiments.  This has provoked considerable discussion in the research community and consequently there will be a series of commentaries on the original paper:
  • Dag Sjøberg and Gunnar Bergersen – “Comments on ‘Empirical Software Engineering Experts on the Use of Students and 31 Professionals in Experiments’ by Falessi et al., EMSE 2018
  • Per Runeson – “Realism in Software Engineering Experiments — A Search in Vain?”
  • myself (Martin Shepperd) – “Inferencing into the Void: Problems with Implicit Populations
followed by a response from some of the original authors.  The complete article will appear in the December issue of the journal Empirical Software Engineering [2].
In the meantime, I will try to summarise my arguments and link to an early version of my comments on arXiv.
  • It’s important to make considered decisions about the types of participant we use and appreciate the potential threats to validity.
  • We should avoid dichotomising participants as professionals/students since a student might be a part-time or former professional and in any case their experience may be more or less relevant.
  • It’s important to be explicit about the type of population being investigated.
  • We also need to consider how we sample tasks, artefacts and settings (as well as participants).  Are these representative? Are there potential interactions?
  • Sometimes pragmatism wins the argument: using students may be the only option, but if that’s the case let’s be honest and say it’s a matter of expediency rather than claim it’s actually better than using professionals (if that is indeed the case).
  • In terms of advocacy, using professionals is more likely to be persuasive for getting new software engineering techniques adopted in practice.
So whilst I appreciate Falessi et al. [1] initiating a discussion around the choice of participant in our experiments, my fear is some researchers may attempt to use this paper as a blanket justification for taking the easier path of using students when they are not representative of the population of interest.  Presently (18.10.2018), they have 14 citations.  After excluding one duplicate and one in Portuguese, 8/12 argue that it’s ok to use students because Falessi et al. say so.  A typical example is “Falessi et al. state that controlled experiments with students are as valid as experiments with experts”. Obviously there are occasions when this may be so, but to use this paper as blanket permission worries me.  In fact it worries a good deal.
[1] Falessi, D., Juristo, N., Wohlin, C.,Turhan, B., Münch, J., Jedlitschka, A., Oivo, M., “Empirical software engineering experts on the use of students and professionals in experiments”, Empirical Software Engineering 23(1), pp452–489 (2018).
[2] Feldt, R., Zimmermann, T., Bergersen, G., Falessi, D., Jedlitschka, A., Juristo, N., Münch, J., Oivo, M., Runeson, P., Shepperd, M., Sjøberg, D., Turhan, B., “Four commentaries on the use of students and professionals in empirical software engineering experiments”, Empirical Software Engineering 23(6), pp3801-3820 (2018).


This blog was updated (25.10.2018) to reflect the new title and authorship of [2] which has been changed at the request of the journal editors Robert Feldt and Tom Zimmerman to better reflect the content.
It was further updated (28.11.2018) to provide a link and full publication details for reference [2].

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s