Online testing and computerised test reports continue to grow in popularity as easy, cost-effective psychometric assessment options. They allow for quick processing of any number of individuals, yet can also be used for just a few candidates. The need for office space for testing sessions can be eliminated; so can the need for staff to administer and supervise testing.
There are benefits to computerised test reports delivered online. Processed reports are easier to manage and more profitable for test publishers compared with bulk hard copies of test questionnaires and answer sheets.
But there are a few things to consider when you’re working with online test reports and some cautions that should be taken into account.
First, can you adjust a report? They aren’t always amenable. If you’re a consultant sharing computerised test reports with clients, consider whether you can add any information you consider relevant. This might include information that can give a more complete and accurate assessment of the candidate’s performance–information from feedback interviews with the candidate, other test scores, other sources of data.
When presented with a report that summarizes likely behaviours of another, we tend to consider it accurate. Prepared test result reports usually look very professional. They are likely to include impressive graphs and charts to indicate candidate scores, and to be written in language that conveys accuracy. Consider that instilling confidence in readers helps to sell tests.
But have the technical details and psychometric qualities of the test been demonstrated sufficiently to backup conclusions? These include questions of validity, reliability, fairness and availability of matched norm or comparison groups.
When we’re presented with a report summarising abilities, characteristics or likely behaviours of ourselves or of others, we have a tendency to consider it accurate, and to be able to readily point to reasons supporting its accuracy. One reason is the Barnum, or Foyer effect. PT Barnum is credited with the phrase “something for everyone”. We want to believe anything that is written about ourselves. We consider even vague descriptions of our character to be accurate assessments. A classic example is horoscopes.
We are likely to believe we can draw accurate conclusions about others, even when information is limited or inaccurate. The actor-observer bias influences our evaluations of others, leading us to attribute another person’s behaviour to an underlying, stable personality characteristic. For example, observing that one of my co-workers occasionally walks out of the job 20-30 minutes early leads me to conclude, “What a chancer! She doesn’t much care about keeping her job.” But actually, my co-worker has a legitimate arrangement in place to leave early on occasion to collect her children.
Our cognitive systems are susceptible to these types of evaluative biases, and even becoming aware of them does not prevent anyone from relying on them as a first response. These biases make us more likely to believe general information about others, and give us confidence in our ability to predict their future behaviour.
Descriptions of behaviour such as those included in computerised test reports are likely to be seen as both accurate, and as justification for conclusions about what the test-takers are really like. Always consider: How accurate is the report? Does it convey the extent to which it may be inaccurate?
Accuracy questions are related to how much error affects scores. The term “measurement error” relates to the likely accuracy or truth of a candidate’s test score. Is this actually his level of verbal reasoning? Is she really that optimistic?
Tests do not measure abilities or characteristics perfectly. Does the test report indicate the potential of being not exactly correct? Are terms like “she/he is likely to” or “may be…” used, rather than absolutes like “they are”? More definitive phrases imply that a test is error free, which none of today’s measures are.
The level of confidence in score accuracy should be carefully considered when comparing job candidates. This may be presented as information on measurement error or confidence intervals surrounding test scores. Is this information available based on a test report? If not, there is a risk of making the wrong hiring decision–one that is unfair to candidates and may result in bad hiring decisions.
In evaluating whether a report’s scores are accurate, there’s another issue to consider. How certain can you be that the report matches the candidate asked to take the test? If tests are completed remotely, could persons have colluded, or relied on assistance from other sources when completing the test?
To help ensure that the intended candidate completes a test, you may have test-takers sign an honesty contract specifying removal from further consideration if cheating is detected. Consider providing alternative test versions to different candidates. Follow unsupervised testing with re-sitting the test in a supervised setting, something typically done at a later stage of the selection process.
Finally, there is the question of whether a test and subsequent report was a good choice to assess candidates for the intended job. Decisions about which online test to use involve considerations that are identical to those required for choosing paper and pencil tests for selection. These include the availability of evidence from a job analysis to determine which skills, abilities and traits are required for a particular position. What other forms of evaluation will be used during assessment, and what role does testing play in light of these? These questions are an essential part of the selection process that must precede a search for suitable tests.
Issues surrounding online testing, computerised test reports and test selection are covered more extensively in Selection by Design’s Test User courses. Our June, 2016 Test User training dates are fast approaching. Visit our registration page for more information.
If you have questions about computerised test reports, interpreting the accuracy of an online test, or any other queries surrounding online testing, feel free to ask in the comments section below or contact us directly.
