How to test a candidate’s AI skills.
Give them a real task from the job and let them use their own AI tool. Then read their conversation with the AI. Someone good with AI is not the one who uses it most, but the one who notices when it is wrong.
What strong AI skills look like in a candidate
Two-thirds of leaders (66%) say they would not hire someone without AI skills, in Microsoft and LinkedIn’s 2024 Work Trend Index, a survey of 31,000 people (Microsoft). Few say what the skills are. In practice, someone good with AI manages it like a fast, confident junior colleague: they decide what to hand it, brief it, and check what comes back.
Signs a candidate has strong AI skills
- Understand Reads the material first. Breaks the problem down. Spots when the brief is wrong.
- Brief Gives context, limits, examples and what done looks like. Keeps the judgment calls.
- Check Tests against the source. Catches the made-up fact and the wrong number.
- Correct Says what is wrong and why. Changes approach early.
- Sign off Ready for the reader. Says what was checked and what was not. Keeps confidential data out.
- Reuse Saved prompts, skills, workflows and automations. The next run is faster and the same.
Not a sign of AI skills either way
- How many prompts they write, or how long they are
- How fast they finish
- Which AI tool they use, or how many they can name
- A certificate or a course
- Polished writing or perfect English
What doesn’t tell you if a candidate has AI skills
Most of the quick ways to check AI skills measure something nearby: use, knowledge or results.
| Method | What it tells you | Why it is not enough |
|---|---|---|
| Asking “Do you use AI?” | Everyone says yes. | It tells you they use it, not whether they check what it gives them. |
| A quiz about AI | What they know about AI. | Knowing that AI makes things up is not the same as catching it when it does. |
| A certificate or course | That they took the course. | Most courses test knowledge with questions that have a right answer, not work. |
| Finished work only | What they handed in. | AI makes most finished work look good. You cannot see who did the thinking. |
| A test where AI is banned | How they work without AI. | That is not how they will work in the job, and the ban is hard to enforce. |
How to assess AI skills at each stage of hiring
You do not need a test at every stage. A better question early, a real task, and one follow-up question at the end cover most of it.
- 1In the applicationAsk one specific question instead of “Do you use AI?”: “Tell us about a time AI gave you something wrong. How did you notice?” A real answer has a detail in it.
- 2On the first callAsk them to walk you through the last piece of work they did with AI: what they gave it, what came back, what they changed. Listen for checking, not for tools.
- 3In a taskGive a real task from the job, in their own AI tool, with AI expected. Build in one place where AI tends to go wrong. Collect the conversation with the AI, not only the result.
- 4In the final interviewPick one moment from their conversation with the AI and ask about it. Why they trusted that number; why they rewrote the brief. It is hard to answer for work you did not do.
AI skills questions for the first call
- 1Tell me about the last time AI gave you something wrong. How did you notice?
- 2What do you tell the AI before you start a task like this one?
- 3What do you never let AI decide for you?
- 4What did you check before you sent your last piece of AI-assisted work?
Build in one place where AI tends to go wrong
On a clean task, almost everyone’s AI does fine, and you learn little. Real work has places where AI goes wrong. Put one in, and you see who checks. Five kinds that work in most roles:
| Kind | What it shows |
|---|---|
| A number that has to match a source | The AI’s total looks right and is not. Who checks it against the file? |
| A definition that changes the answer | A word like “revenue” or “active user” means something specific here. Who asks before the AI guesses? |
| A detail buried in a long document | One line in a contract or a transcript changes the result. Who reads it, and who lets the AI skim? |
| A source that does not say what the AI claims | The AI cites a page or a study. Who opens it? |
| Something that should not leave the room | Bank details, a salary, a customer name. Who takes it out before handing over? |
Read the conversation with the AI, not only the result
The finished work shows what was handed in. The conversation shows how they got there. Six questions to answer from it, the same in every role:
| Skill | The question | Weak looks like |
|---|---|---|
| Understand | Do they understand the task before handing it to AI? | Asks for the answer first. Takes the brief as given. |
| Brief | Do they give AI what it needs? | Pastes everything and says “do this”. |
| Check | Do they catch it when AI is wrong? | Ships what looks finished. |
| Correct | Do they fix it when it isn’t working? | “Try again”, “make it better”, on repeat. |
| Sign off | Is what they hand over ready to send? | Filler, too long, claims nobody checked. |
| Reuse | Do they make it repeatable? | Redoes the work by hand each time. |
In Lemonsignal, candidates do the task in Claude Code, Codex, Claude Cowork or ChatGPT, and you get the whole conversation as a session replay, with the six skills scored and every score linked to the moment it came from.
Questions about testing AI skills
How do you test whether a candidate is good with AI?
Give them a real task from the job, let them use the AI tool they already use, and say that AI is expected. Build in one place where AI tends to go wrong. Then read their conversation with the AI, not only what they hand in, and ask about one moment in it in the interview.
What does “good with AI” mean for a job candidate?
That they get good, correct work done with it. They understand the task before handing it to AI, give it what it needs, catch it when it is wrong, fix it, and hand over work that is ready to send. Using AI a lot, or fast, is not the same thing.
Can you tell from a CV whether someone is good with AI?
Not reliably. A CV can list tools and courses, but not whether the person checks what AI gives them. Ask one specific question in the application, and test it with a task.
What questions show whether a candidate is good with AI?
Questions about a specific moment: when AI was wrong and how they noticed, what they told it before starting, what they checked before sending. A vague answer is the signal.
Should candidates be allowed to use AI in the test?
If they will use AI in the job, yes. A test where AI is banned measures how someone works without it, which is not how they will work once hired.
How long should a test of AI skills take?
One to three hours of the candidate’s time for a task close to the real job, with a deadline of a few days. Shorter tests can work for a first screen, but they show less of how someone checks and fixes work.