In this bonus conversation, Nic and Caroline pick up the questions that didn’t make it into their previous episode and uncover some of the most interesting overlaps between psychology and product design.
They discuss how AI and Work Samples are reshaping talent assessment, what psychologists need to safeguard as technology becomes more embedded in hiring, and how keeping the human in the loop remains critical. They also share where they find creative inspiration and end, as always, with a laugh.
00:00 – Introduction
Nicola and Caroline revisit their earlier chat and introduce the bonus questions that sparked this discussion.
01:00 – Work samples and AI
Are simulations really the gold standard of validity? Nicola explains the evidence and the trade-offs, and how technology is making them scalable and cost-effective.
03:00 – Safeguarding psychology in an AI world
Fairness, explainability, reliability: Caroline and Nicola explore how to balance innovation with ethical practice and why human oversight still matters.
06:30 – Collaboration and the human in the loop
Why psychologists, engineers, and product teams now work side by side — and why that makes for better assessments.
07:30 – Building trust with transparency
How clear communication and responsible rollout help clients adopt new technology confidently.
10:00 – Starting small and scaling smart
Nicola’s “eat the elephant one bite at a time” approach to introducing new features and assessments.
12:00 – Inspiration everywhere
Caroline shares how product ideas can come from anywhere, from streaming apps to restaurant menus, and why curiosity is the real driver of innovation.
14:40 – Wrap-up
A light-hearted close on the joys (and perils) of watching TV or shopping with a product manager.
Caroline Fry (00:00)
In a previous episode, Nic and I interviewed each other. It was a great conversation, but a few questions ended up on the cutting room floor. Here are some of the outtakes we couldn't leave behind.
Caroline Fry (00:00)
I had another question about the specifics of assessments themselves, particularly work samples, which are a big part of our working life at the moment. There’s a growing focus on simulations or work samples as the gold standard of validity. From a psych perspective, how valid are they really? And what are the trade-offs compared to more traditional assessments like cognitive or personality measures?
Nicola Tatham (00:33)
Great question. It’s something I think about a lot. In terms of the “gold standard,” they do have really strong validity. Across meta-analyses looking at multiple validity studies, they tend to fare very well—there’s no dispute about that. As for face validity, meaning whether candidates buy into them and find them relevant, they also score highly.
Nicola Tatham (01:02)
They perform well because candidates are undertaking a task that reflects the role they’re applying for. That makes it compelling for both candidates and assessment users. So why haven’t we been doing this for years? Because they’re not always an easy solution.
Nicola Tatham (01:30)
Some of the trade-offs include cost, time, scalability, scoring, and reliability. They’ve traditionally been harder to develop because they’re more abstract, and scoring often requires input from a human rater, which can introduce inconsistency. Traditional measures like cognitive and personality tests tend to be cheaper to develop, shorter, and still strong predictors of job performance. So for us, it’s about thinking: why are we now
talking about work samples? I think it's because with the advent of AI and technology, makes it possible for us to create realistic work samples that are far more scalable, far more cost effective. So we can now build an automated scoring, adaptive branching, we can use natural language processing, all of that great stuff so that we can now simulate job tasks digitally. I guess there's probably a sliding scale of how far you go with that, balancing cost, predictive validity, face validity, and so on and so forth. But yeah, to answer your question.
They're up there when it comes to validity and over time, think we're now finding ways to make them more accessible to more of our clients and partners.
Caroline Fry (03:08)
Yeah, I think I've been thinking a lot about, obviously the tech and how that supports the development of psychometric assessments and, the scalability that this tech, and as a whole tech, it launches and then, it gets commoditized and those benefits we can feel for real, like as we implement those kinds of solutions. So it just, it opens up a lot of opportunity, doesn't it? But one thing, obviously, there's, all know there's many conversations around AI at the moment. And I do see the opportunity in development of products and features of AI, but it's specifically related to psychometrics as, it becomes more embedded in psychometrics or in hiring solutions generally like HR tech. What do you think psychologists need to safeguard most fairness, explainability, or something else entirely?
Nicola Tatham (04:09)
I want to safeguard all of that, all of the above. I don't want them to become mutually exclusive. I think that's quite mean asking me to choose between them. I guess as psychologists, first and foremost, we've always been used to safeguarding fairness and validity and reliability. And I bang this drum on every podcast that we have where we've got the opportunity to, that is not going to go away. But obviously, in the world of AI, then they've got to sit alongside things like explainability, transparency, candidate privacy, all of those things do equally matter. I think explainability matters for the trust and the legal defensibility piece, but a model that's explainable, but not fair or not valid is still a problematic model. So I'm going to sit on the fence on that one and say, I want all of the above and I'm not going to choose.
Caroline Fry (05:01)
Anything else that you think, you mentioned additional reliability, you your still want reliability and validity as alongside fairness.
Nicola Tatham (05:09)
Yes, I think also that's keeping the human in the loop as well. So when we're making nuanced decisions, we don't just rely on the robots to do that for us. And it's recognizing, keeping on top of all of the guidelines and the legislation around AI as well, which is why we are seeing.
people like me working far more closely with people like you and with engineers and with data scientists and with AI experts. No one person can do this alone. It used to be the psychologist would develop a test and then get somebody to print the test for them. And it was job done, but there is just no way that we can operate in our silos anymore. And it makes for a better overall product if we're all working and pushing in the same direction to achieve that and sharing our expertise to do that.
Caroline Fry (05:37)
Yeah, fully agree the benefits of that multidisciplinary approach and everyone bringing their different skills. It's much more — I mean, selfishly for all of us — I think it's much more enjoyable as well to learn and work together and understand more of each other's expertise. This is blowing it out more broadly... Where do you think we've done a really good job outside the world of assessments at applying psychological principles in a way that helps people? Whether that's technology and education or day-to-day life.
Nicola Tatham (06:35)
I don't know if it's very ethical, but I'm going to share it anyway. I think I use the concept of operant conditioning — the whole sort of rewarding the behavior that you want to see more of — regularly with my kids. So they're now 18 and 20. I'll tend to offer to pay for dinner if they're willing to come out for me. And the more I do it, the more likely they are to join me again. These principles can work at home as well as in the lab, and I'm pretty sure my kids aren't going to watch this, so I'm probably safe to share that.
Caroline Fry (07:09)
Okay, I mean, are we talking Pavlov's dog here? Like, what have you done? Oh yeah, the rats! I remember I did a psychology AS level. I think I remember that case study. Oh, brilliant. Yes. Yeah. Yeah.
Nicola Tatham (07:12)
I think it was rats, but I promise I'm not comparing my children to rats. I think it was the pressing the thing for food, wasn't it? So yeah, not too many steps removed from that.
Nicola Tatham (07:26)
What have you learned about introducing new assessment formats or AI features, for example, in organisations that are either very risk averse or perhaps highly regulated?
Caroline Fry (07:40)
Well, I'm hardly the first person to say this and we talked before about explainability, but to me, transparency is critical. Like in the case of AI, there are way too many horror stories out there, particularly poor experiences on one side and biased or unfair decision-making on the other. So as an HR tech supplier, I think we have to make sure we have explainability, transparency, and that we're aligned to any necessary regulatory frameworks. And as we know, in the case of AI, that's really quickly evolving. We need to be clear and transparent about the benefits and limitations of these kinds of features or technology. In TA especially, the tech is only as good as the process it sits within, I think. So making sure we have full knowledge of the client's use cases and goals is critical as well, to help them understand whether a solution or a particular feature is the right fit.
Like we talked about before, not one size fits all. It doesn't benefit us to force some functionality, some jazzy new functionality that we've built onto a customer when it isn't the right thing for their process, and it won't deliver the outcomes they need. Not because the technology, the feature that we built, or the assessment is substandard, just because it's a square peg, round hole, and doesn't complement the process that it's working with them.
Nicola Tatham (08:58)
They're just not ready at that point.
Caroline Fry (09:02)
And I think timing too is very important. As we know, TA works in cycles — often many cycles within one organisation. So rollouts of new features and assessments need to be considered as part of the client's wider recruitment ecosystem, if I can call it that. I think this actually helps. The industry for us actually helps in some ways because of the cyclical nature — usually within TA, you can find some time when something needs to be refreshed or reconsidered. Like, we were just talking at the top of this podcast about the skills landscape changing so quickly. So we actually end up having quite a few opportunities working with our clients to say, okay, well, you're reformatting what this role is doing or looking for this recruitment cycle. So at that point, we can look at the new features, we can look at potentially new assessments to start enhancing their process.
And I've actually found that more often than not, the TA leads that we work with in organisations are very keen to continuously improve because they recognise that. And the challenges that they face in attracting top talent make that a priority for them anyway. So if there are new things that enhance the candidate experience or a new assessment that’s got better face validity or better predictive validity, they're going to be open to implementing it.
Caroline Fry (10:25)
And I think we've said in the podcast before — for those that are really risk averse and potentially a bit, shall I say, stuck in their ways, or have a lot of regulatory issues that they've got to work around — like we said before, starting small is usually the best approach. I know you are heavily involved in piloting with our clients, piloting a new feature or an assessment first to make sure it's the right fit. Be intentional about what you're expecting.
And we had Craig, our Customer Success Director, on recently, and he said, make sure you've got a baseline against which you can measure to make sure you know what the improvements are that you're looking for. And I think all of these things would build confidence for a risk-averse or regulated organisation to be able to make a positive change and adopt new technology or features.
Nicola Tatham (11:10)
Yeah. Eat the elephant one bite at a time, or something along those lines. Just start small. Start small. And I guess for us, it's to make it as easy for our customers as we possibly can to take on new features, to understand new assessments, and make them easy to access, use, and understand — and understand the benefits of them, which I think is what you started your answer with.
Caroline Fry (11:17)
Something like that.
That's the product thing. What problem are you solving for them with this feature, with this assessment? What does it bring to them?
Nicola Tatham (11:48)
Yeah, and it's not just going, “Here, we've got this new feature.” It's explaining why that new feature is going to be useful to them and why not to be fearful of it. Okay, so one last question. I'm sort of getting my revenge on you really, because you just asked me about how I use psychological principles outside of work. Where do you look for ideas or inspiration outside the world of assessments — whether that's from other tech products, general design practices, or even completely different industries?
Caroline Fry (12:23)
Okay, well, this is a really product answer, but everything’s a product. So everywhere, in everything — obviously with tech, our platform is tech. We're all using tech every day. I do notice if I use a particularly nice intuitive interface of anything — whether it's your streamers on TV or a menu ordering experience when you go to a restaurant and browse through. There are small things you can pick up from pretty much any daily experience you're having.
Packaging too — in and out of tech, there are so many things. I think I can be a bit boring about product and notice if something's been packaged or positioned really well. I think a key attribute of product people is that they’re naturally curious. You don’t take anything at face value, and you look beyond. I would say I look everywhere for inspiration, but mostly, especially as we're building candidate experiences, it's that tech experience.
So many marketing sites do so well in terms of design and things like that. I think we could really learn a lot. I think we've spoken in previous podcasts as well that maybe sometimes the science part of the psychometrics — like you said, back in a previous life, people did hours of batteries, certainly batteries of tests — they certainly weren't thinking about the user experience at that point in time. And I think the industry has definitely caught up now, but for a long time, they didn’t.
And I think it's just that user-centric mentality — thinking about experiences. Anything you experience, anywhere you go. I think we've had team socials at places like Flight Club — there are some really nice things where you can really appreciate when something has been built around the user experience. But yeah, there's no limit to it. It's annoying that you can't really switch it off sometimes actually. That's the worst thing about it.
Nicola Tatham (14:41)
I was going to say, bet you're a joy to watch TV with.
Caroline Fry (14:44)
Yeah, it is a little, yeah. But everywhere is the simple and probably obvious.
Nicola Tatham (14:52)
A day out shopping with Caroline Fry is quite different to a day out shopping with anyone else.
Caroline Fry (14:53)
Yes, don't do it.
Thanks for hanging out with us on The Score. If you enjoyed this conversation on all things psychology and product, don't miss what's coming next. New episodes drop every two weeks on YouTube, Spotify, or wherever you get your talent acquisition insights.
In this bonus episode of The Score, Caroline Fry, Sova’s Head of Product, and Nicola Tatham, Chief IO Psychologist, dive into the questions that didn’t make it into their previous conversation, from the validity of work samples to the ethics of AI and the art of balancing innovation with fairness.
At the heart of their discussion is a question many organisations are asking right now: how do we create assessments that are both scientifically robust and still feelhuman?
Nicola explains that work samples and simulations have long been considered one of the most valid ways to predict job performance. They’re realistic, engaging for candidates, and closely mirror the tasks someone would actually do on the job.
But until recently, they came with trade-offs: they were time-consuming, costly, and difficult to score reliably. That’s changing fast: with AI, adaptive branching, and automated scoring, organisations can now use digital simulations that combine realism with scalability, and bring the gold standard of assessment to far more candidates.
When it comes to AI, we shouldn't have to pick between fairness, explainability, and validity, all are essential. A model that’s explainable but unfair is still a bad model. Instead, we need to foster a collaborative approach, where psychologists, engineers, and data scientists work together to design tools that are ethical, transparent, and effective.
From a product perspective, transparency is the foundation of trust. In a space crowded with hype and horror stories about AI bias, honesty about what a product can and can’t do is vital. As Caroline puts it, “Technology is only as good as the process it sits within.” AI can make assessments smarter, but it doesn’t replace the need for clear communication, human oversight, and sound psychological principles.
Not every company is ready to adopt new AI-driven features straight away. The best way to build confidence is to pilot first, measure outcomes, and scale gradually. That way, organisations can see the benefits before rolling out widely.
Sova is a talent assessment platform that provides the right tools to evaluate candidates faster, fairer and more accurately than ever.
.webp)
.webp)
.webp)
