By this time next year, most of you will have graduated from the master’s programme and made yourselves available on the job market. In this blog I want to shed some light on a gamified assessment called pymetrics games that you may possibly encounter during the assessment stage of a job application process.
For those of you who are not familiar with assessments or applying for jobs in general, the traditional job application process (as I have experienced it) generally looks something like Figure 1.
The traditional (online) assessment stage will generally consist of:
- ability tests measuring your performance when it comes deductive, numerical and logical reasoning;
- personality test (questionnaire)
This is where Pymetrics comes in, a company that specialises in developing gamified assessments for recruitment purposes. Companies have opted to fully replace their aforementioned assessment stage with Pymetrics’ patented pymetric games. Some of the most notable companies being: Boston Consulting Group (BCG), JP Morgan, Accenture and Unilever.
What do these pymetric games entail?
The pymetric games are Pymetrics’ core product. It is an online gamified assessment in which candidates have to play through a series of 12 minigames that take two to three minutes each. The assessment uses neuroscience and AI in order to assess a broad range of 91 different cognitive traits. An example of one of the minigames is depicted in Figure 2.
How does it work (in a nutshell)?
Pymetrics creates a custom algorithm for a company by having at least 50 top performers of said company play the pymetric games. Subsequently, this model is used as a benchmark when evaluating applicants’ results. Pymetrics markets its algorithm as entirely bias free, having succesfully subjected the algorithm to extensive AI audits in order to prove their claim.
So what’s the catch?
As my fellow peers Andrew Tan and Tamas Vincze have already explained in great detail: algorithms are inherently biased. In addition, an independent AI audit of Pymetrics’ algorithm found that although it passed the formal checks, the audit itself did not prove that the tool is bias free nor that it actually picks the most qualified candidates for a job.
This brings me to my question: how do you feel about an AI-based hiring assessment being put into practice? Would you much rather prefer the traditional online assessment? Having personally experienced both types of assessment, I am curious to see where my fellow peers stand, especially as you prepare yourselves for your job search.
References
Did this blog pique your interest? Please refer to the used sources below for more in-depth coverage on this topic:
- https://www.technologyreview.com/2021/02/11/1017955/auditors-testing-ai-hiring-algorithms-bias-big-questions-remain/
- https://digital.hbs.edu/platform-digit/submission/pymetrics-using-neuroscience-ai-to-change-the-age-old-hiring-process/
- https://www.graduatesfirst.com/pymetrics
- https://hackingthecaseinterview.thinkific.com/pages/bcg-pymetrics-test
- https://www.jobtestprep.com/pymetrics-games#balloon-game
- https://sierrasoln.com/hiring-process/
Hey Kevin, nice article! I was not aware of Pymetrics, and I think you did a good job explaining the algorithm and its obvious benefits, but also implications.
How would I feel about an AI-hiring assessment? As you have already mentioned, AI can be biased in ways we, as humans, might not even understand (Brynjolfsson & Mcafee, 2017). This could result in an unethical bias, which organisations often try to avoid by implementing an algorithm in the hiring process (Raghavan et al., 2020). On the other hand, it is also demonstrated that bias in an organisation is not by default increased or decreased with the implementation of AI (van der Broek, Sergeeva & Huysman, 2019).
Would I rather have an online assessment? Difficult and interesting dilemma. I assume the online assessment would have a survey-like structure, and a person would review scores and/or answers. As aforementioned, humans are also burdened by bias, thus bias cannot be considered as an advantage nor disadvantage. However, van der Broek, Sergeeva and Huysman (2019) argue that the use of a hiring algorithm shape what is considered ethical, which would push me towards preferring an algorithmic hiring process. Can we properly understand what bias we would be subjected to?
Sources:
van den Broek, E., Sergeeva, A. and Huysman, M., 2019. Hiring algorithms: An ethnography of fairness in practice.
Brynjolfsson, E., and Mcafee, A. 2017. The business of artificial intelligence: what it can and cannot do for your organization. Harvard Business Review
Raghavan, M., Barocas, S., Kleinberg, J. and Levy, K., 2020, January. Mitigating bias in algorithmic hiring: Evaluating claims and practices. In Proceedings of the 2020 conference on fairness, accountability, and transparency (pp. 469-481).