Imagine your company is evaluating a new AI assistant.
The vendor says:
“Our AI is 95% accurate, enterprise-ready, secure, autonomous, and trained on your business.”
It sounds impressive.
But what does each part actually mean?
95% accurate on what?
Secure against which risks?
Autonomous enough to do what?
Does “trained on your business” mean the model was fine-tuned, connected to company documents, or simply given access to information through retrieval?
And perhaps the most important question:
What evidence supports the claim?
In Part 1 of this series, we learned the basic language of AI.
In Part 2, we followed what can happen between asking an AI a question and receiving an answer or action.
Now we reach the final part of AI fluency:
judgment.
The goal is not to become suspicious of every AI claim.
It is to know what to ask before accepting one.
Start with the claim, not the marketing language
Consider this statement again:
“95% accurate.”
It gives us a number.
But the number means very little until we know how it was produced.
Was the AI tested on:
- 100 questions or 100,000?
- simple questions or difficult ones?
- public information or the company’s real documents?
- one language or several?
- carefully prepared prompts or ordinary user questions?
- the same kind of work your organisation actually plans to use it for?
A number without context can look precise while telling us very little.
So when you encounter an AI claim, a useful first question is:
What exactly was measured?
Accuracy and reliability are not the same thing
The words accuracy and reliability are often used as if they mean the same thing.
They do not.
Accuracy
Accuracy asks whether an answer or prediction is correct according to some expected result.
Suppose an AI assistant answers 95 out of 100 test questions correctly.
On that test, we might describe its accuracy as 95%.
But that says nothing yet about the remaining five failures.
Were they minor mistakes?
Or did the system give dangerously wrong answers with complete confidence?
Reliability
Reliability asks whether the system behaves consistently enough for the intended use.
Imagine two systems.
System A
Gets 95% of questions right but occasionally invents a confident answer when it does not know.
System B
Gets 92% right but reliably says:
“I do not have enough information to answer this.”
for most uncertain cases.
Which one is better?
There is no universal answer.
It depends on the task.
For drafting a casual summary, System A may be acceptable.
For legal, financial, medical, or operational decisions, the behaviour of the failures may matter much more than the average score.
Remember it
Accuracy asks: Was the answer correct?
Reliability asks: Can I depend on the system to behave acceptably over time and across situations?
That is why a single accuracy percentage should rarely end the discussion.
A benchmark is a test, not the real world
AI announcements frequently mention benchmarks.
A benchmark is a standardised test or dataset used to compare systems on particular tasks.
Benchmarks can be useful.
They can help researchers and developers compare:
- language understanding;
- coding;
- mathematics;
- reasoning tasks;
- image recognition;
- retrieval;
- safety behaviour;
- other capabilities.
But a benchmark result does not automatically tell you how well a system will perform in your organisation.
Suppose an AI model scores highly on a document-question benchmark.
Your real environment may contain:
- poorly formatted PDFs;
- outdated policies;
- scanned documents;
- tables;
- conflicting versions;
- internal abbreviations;
- multiple languages;
- incomplete data.
Those conditions may not resemble the benchmark at all.
A useful way to think about it is:
A benchmark tells you how the system performed on that benchmark.
Anything beyond that needs evidence.
Visual 1 — Benchmark versus real use
Benchmark
Controlled test Known dataset Defined scoring method
Result
95%
Then compare with:
Real workplace
Messy documents Different users Unexpected questions Changing information Different consequences
Key idea: Benchmark performance is evidence, but not proof of performance in every real-world setting.
What is an evaluation?
An evaluation, often shortened to eval, is a structured way of testing how an AI system behaves.
A benchmark can be part of an evaluation.
But evaluations can be much broader.
For our fictional company assistant, we might test:
- Does it answer common policy questions correctly?
- Does it retrieve the right documents?
- Does it avoid inventing policy exceptions?
- Does it say when information is missing?
- Does it cite the correct source?
- Does it behave consistently when the same question is phrased differently?
- Does it refuse actions it is not authorised to perform?
- Does it use the right tool?
- Does it stop and ask for approval when required?
That is much closer to the real problem than simply asking whether the underlying model scored well on a public benchmark.
Current evaluation frameworks increasingly separate different aspects of performance. For example, retrieval, groundedness, relevance, response completeness, tool use, task completion, and safety can be measured separately rather than collapsed into one number.
Remember it
Benchmark = a standard test.
Evaluation = a broader process for measuring whether the system behaves well for the intended task.
Evaluate the task you actually care about
Return to our fictional AI assistant.
Suppose the vendor says:
“95% accurate.”
A better evaluation might include real questions such as:
“How many leave days can I carry into next year?”
“What happens if I join halfway through the year?”
“Does this policy apply to contractors?”
“The policy does not answer my case. What should I do?”
The final question is especially important.
A useful AI system should not only perform well when the answer exists.
It should also behave sensibly when the answer does not exist.
That is where a small test set can reveal more than a broad marketing number.
Visual 2 — Evaluate the real task
Question 1
Clear answer exists.
Expected behaviour: answer correctly.
Question 2
Answer requires an exception.
Expected behaviour: find and apply the exception.
Question 3
Policy does not contain the answer.
Expected behaviour: say that the information is unavailable rather than inventing it.
Key idea: Evaluation should test normal cases, difficult cases, and situations where the system should not answer.
One good result does not prove general capability
This matters whenever you see an AI demonstration.
A system may perform very well on one example.
That does not mean it will perform equally well on:
- different users;
- different languages;
- longer documents;
- unusual inputs;
- changing data;
- other industries;
- higher-risk decisions.
This is sometimes called a generalisation problem.
The system may work well under the conditions in which it was tested but less well outside them.
NIST’s AI Risk Management Framework explicitly calls for documenting limitations in how far results can be generalised beyond the conditions under which a system was developed and evaluated.
A good question is:
Where has this been tested, and where has it not?
Bias and fairness need context
Another common AI discussion involves bias.
Bias does not simply mean:
“The AI has an opinion.”
In AI systems, bias can arise from many sources:
- training data;
- labels;
- historical patterns;
- sampling;
- how a problem is defined;
- how outputs are interpreted;
- how the system is deployed.
Consider a hiring system trained on historical decisions.
If past hiring practices disadvantaged a group, the system may learn patterns connected to those decisions.
Even if a protected characteristic is removed explicitly, other variables may still correlate with it.
That is why fairness cannot be judged only by asking whether the model intentionally discriminates.
The more useful question is:
Do the system’s outcomes create unfair differences for relevant groups in this context?
Fairness is also not one universal mathematical rule.
Different situations may require different definitions and trade-offs.
NIST treats fairness with harmful bias managed as one of several trustworthiness characteristics rather than something that can be reduced to a single universal score.
Privacy starts before the model answers
Suppose your company assistant can access:
- employee files;
- customer records;
- contracts;
- emails;
- financial data.
A system can produce perfectly accurate answers and still be unsafe if it exposes information to the wrong person.
This is why privacy is not simply:
“Does the model remember my prompt?”
Useful questions include:
- What information can the system access?
- Does it access only what the user is entitled to see?
- Where is conversation data stored?
- How long is it retained?
- Is sensitive information sent to another service?
- Can administrators review usage?
- Is data used for model improvement?
- Can access be revoked?
For systems that use tools and company data, access control matters as much as answer quality.
A powerful AI system with excessive permissions can create more risk, not less.
“Secure” is too broad by itself
Now return to the vendor statement:
“Enterprise-ready and secure.”
Secure against what?
Security can include very different concerns:
- unauthorised access;
- credential theft;
- prompt injection;
- data leakage;
- malicious files;
- unsafe tool calls;
- privilege escalation;
- compromised integrations;
- model or service abuse;
- denial of service.
No product is simply “secure” in every possible sense.
A better claim would explain:
- what threats were considered;
- what controls exist;
- what testing was performed;
- what risks remain.
NIST similarly treats security and resilience as distinct trustworthiness characteristics that require evaluation and documentation.
Remember it
When you hear:
“Secure AI”
ask:
Secure against which threat, under which conditions?
Guardrails: controls around behaviour
You will also hear the term guardrails.
Guardrails are controls intended to constrain or guide AI behaviour.
They might include:
- content filters;
- allowed-tool lists;
- permission checks;
- validation rules;
- rate limits;
- human approvals;
- policy checks;
- output restrictions.
For example, an AI assistant might be allowed to draft a payment request but not submit it.
Or it might be allowed to search employee information but only within the requesting user’s permissions.
Guardrails are useful.
But the existence of a guardrail does not mean the system can never fail.
A better question is:
What behaviour does this guardrail control, and how has that control been tested?
Human oversight means more than adding an approval button
Many AI products mention human-in-the-loop or human oversight.
That sounds reassuring.
But the quality of the oversight depends on how it is designed.
Imagine an AI system prepares 500 decisions per day and asks one employee to approve each one.
Technically, a human is involved.
Practically, the employee may start clicking Approve without meaningful review.
Good oversight should answer questions such as:
- What decisions require review?
- What information does the reviewer see?
- Can the reviewer understand why the system made the recommendation?
- Can the reviewer reject or correct it?
- Are high-risk cases escalated differently?
- Is the workload realistic?
Human oversight should be a meaningful control, not a label.
Visual 3 — Guardrail versus human oversight
AI proposes action
Automated guardrail
Is this action permitted?
If allowed:
Human review for consequential action
Approve / Reject / Modify
External action
Key idea: Automated controls and human oversight solve different problems and can be used together.
“Trained on your business” can mean several things
This is one of the easiest AI phrases to misunderstand.
A vendor says:
“The system is trained on your company data.”
What happened technically?
There are several possibilities.
The information is supplied in the prompt or context
Your company material is given to the model during the task.
The underlying model has not necessarily been changed.
The system uses RAG
Relevant company information is retrieved and supplied to the model when needed.
Again, the base model may remain unchanged.
The model was fine-tuned
Additional training changed the model’s parameters to adapt its behaviour or performance.
These approaches are not the same.
One may be more appropriate than another depending on the task.
Visual 4 — Three different approaches
Context
Give information during the interaction.
RAG
Retrieve company information when needed.
Fine-tuning
Further train the model.
Key idea: All three can make an AI application more useful for a business, but only one of them necessarily means additional model training.
Whenever someone says:
“Our AI is trained on your data,”
ask:
Do you mean context, retrieval, or actual model training?
“Reasoning” is useful language, but be careful with what it implies
Modern AI systems are often described as reasoning models or as having stronger reasoning capabilities.
In practical product discussions, this usually means the system performs better on tasks requiring several connected steps, such as:
- mathematics;
- coding;
- planning;
- logical problems;
- combining information.
That can be a useful description of capability.
But words such as:
- thinking;
- understanding;
- reasoning;
- knowing;
are also ordinary human words.
Using them for AI can make it easy to assume more about the system than has actually been demonstrated.
The useful question is not:
“Does this AI really think?”
For most product decisions, a better question is:
What tasks can it perform, how was that measured, and where does it still fail?
That keeps the discussion connected to evidence.
Autonomy is not one switch
Another popular word is autonomous.
A vendor may say:
“Our AI agent works autonomously.”
That could mean very different things.
The system might:
- choose which document to search;
- decide which tool to call;
- make several intermediate decisions;
- send an email without approval;
- modify customer data;
- execute financial transactions.
Those are very different levels of autonomy.
Instead of asking:
“Is it autonomous?”
ask:
What can it decide by itself?
and:
What can it do without human approval?
Those questions reveal much more.
Evaluate the path, not only the final answer
This becomes especially important with agents.
Suppose an agent successfully books the correct meeting.
The final result looks perfect.
But perhaps it:
- queried unnecessary systems;
- accessed information it did not need;
- called the wrong tool several times;
- exposed sensitive information along the way;
- succeeded only after repeated failures.
If we look only at the final meeting booking, we miss important behaviour.
Modern agent evaluations increasingly distinguish between the final result and the trajectory or sequence of actions the agent took to reach it. Google, for example, documents final-response evaluation separately from trajectory evaluation, while Microsoft exposes metrics around task completion and tool selection/use.
Remember it
For an agent:
Did it finish the task?
is one question.
How did it finish the task?
is another.
Visual 5 — Evaluate both outcome and path
Goal
“Prepare the monthly report.”
Agent path
Search data → Call tools → Analyse → Generate report
Evaluate:
- Were the right tools used?
- Were permissions respected?
- Were unnecessary actions avoided?
Final result
Was the report correct and complete?
Key idea: Agent quality includes both the result and the path taken to produce it.
A demo is not an evaluation
AI demonstrations can be extremely useful.
They help us see what technology can do.
But a demonstration usually shows:
This worked once under these conditions.
An evaluation asks:
How often does it work, under which conditions, and how does it fail?
That distinction becomes especially important when an AI system is moving from:
interesting prototype
to:
something people depend on.
If a vendor shows a polished demo, useful follow-up questions include:
- How many similar cases were tested?
- What was the success rate?
- What were the failure cases?
- Were ordinary users involved?
- Was the test data representative?
- What happens when required information is missing?
- What happens when the system encounters an unexpected input?
A failed evaluation is not always bad news.
Finding failures before deployment is one of the purposes of evaluation.
AI quality can change after launch
Even a strong pre-release evaluation is not the end.
An AI application operates in an environment that can change.
For example:
- the underlying model may change;
- company documents may change;
- retrieval indexes may be updated;
- users may ask new kinds of questions;
- external tools may change;
- policies may change;
- attackers may discover new ways to manipulate the system.
That means AI systems may also need monitoring after deployment.
Google’s current agent-evaluation guidance distinguishes frequent development evaluation, scheduled regression testing, and continuous production monitoring.
The lesson is simple:
Evaluation is not only something you do before launch.
One AI score cannot answer every question
Return to our fictional vendor.
Imagine its presentation says:
AI Trust Score: 92/100
That may be useful internally if the scoring method is well defined.
But it should not hide the underlying questions.
A system could be:
- highly accurate but weak on privacy;
- secure but unreliable;
- reliable for English but poor in another language;
- excellent at answering questions but unsafe when given tool access;
- fair in one tested population but not evaluated in another.
This is why AI trustworthiness is better understood as several connected dimensions rather than one universal score. NIST’s framework explicitly separates characteristics such as validity and reliability, safety, security and resilience, accountability and transparency, explainability, privacy, and fairness.
Try it yourself: interrogate an AI claim
Take a statement like:
“Our AI assistant is 97% accurate.”
Do not immediately decide whether the claim is good or bad.
Instead, ask:
- 97% on what task?
- What test data was used?
- How many examples were tested?
- Were difficult and missing-information cases included?
- Was performance measured on users like ours?
- What kinds of failures occurred in the remaining 3%?
- Does the number measure only answer accuracy, or also safety and reliability?
- Has performance been tested after deployment?
Now take another claim:
“Our agent is fully autonomous.”
Ask:
- What decisions can it make?
- Which tools can it use?
- Which actions require approval?
- What prevents it from exceeding its permissions?
- How is task success measured?
- Is the sequence of tool calls evaluated?
- What happens when the system becomes uncertain?
You are no longer reacting to the wording.
You are asking for evidence.
That is the skill this article is trying to develop.
A simple framework for evaluating AI claims
When you encounter an AI claim, use five questions.
| Question | What it helps reveal |
|---|---|
| What exactly is being claimed? | Removes vague language |
| How was it measured? | Reveals the evaluation method |
| What was it tested on? | Shows whether the evidence matches your use case |
| How does it fail? | Exposes limitations hidden by average scores |
| What controls exist when it fails? | Connects capability with safety and governance |
These questions work whether the claim concerns:
- accuracy;
- reliability;
- security;
- autonomy;
- reasoning;
- fairness;
- enterprise readiness.
Return to the vendor statement
We started with:
“Our AI is 95% accurate, enterprise-ready, secure, autonomous, and trained on your business.”
Now we can translate that into better questions.
“95% accurate”
On which tasks?
Against what expected answers?
Under what conditions?
What happened in the other 5%?
“Enterprise-ready”
What scale has it operated at?
What access controls, monitoring, governance, and support exist?
“Secure”
Against which threats?
What testing supports the claim?
“Autonomous”
Which decisions and actions can it perform without approval?
“Trained on your business”
Was the model actually fine-tuned?
Or is the application using context or RAG?
The marketing sentence has not necessarily become false.
It has become testable.
That is the difference.
AI fluency is not knowing every AI term
Across these three articles, we have moved through three levels.
Part 1 — Learn the language
We learned terms such as:
- AI;
- machine learning;
- generative AI;
- models;
- LLMs;
- training;
- inference;
- prompts;
- tokens;
- context windows.
Part 2 — Understand the experience
We followed what happens when AI works with:
- grounding;
- retrieval;
- RAG;
- memory;
- multimodal input;
- tools;
- agents.
Part 3 — Develop judgment
We learned to question:
- accuracy;
- reliability;
- benchmarks;
- evaluations;
- bias;
- privacy;
- security;
- guardrails;
- human oversight;
- fine-tuning claims;
- reasoning;
- autonomy.
Knowing the words is useful.
Understanding how the system works is better.
But neither is enough on its own.
The final skill is knowing what evidence to ask for.
The habit worth keeping
You do not need to challenge every AI claim with a technical audit.
For many everyday tasks, that would be unnecessary.
But when the answer matters, the decision has consequences, or the system can take action, one question is worth remembering:
What evidence would make me comfortable relying on this?
Sometimes the answer will be simple.
A source.
A small test.
A human review.
A permissions check.
In other situations, it may require a serious evaluation.
The level of evidence should match the importance of the decision.
That is a more useful form of AI fluency than memorising every new term the industry creates.
AI Fluency series
Part 1 — Learn the language The Language of AI: Essential Terms Everyone Should Understand
Part 1 explains AI, machine learning, generative AI, models, LLMs, training, inference, prompts, tokens, and context windows.
Part 2 — Understand the experience How AI Answers Your Questions: Understanding the Terms Behind the Experience
Part 2 looks at grounding, retrieval, RAG, hallucinations, memory, multimodal AI, tools, and agents.
Part 3 — Develop judgment Understanding AI Claims: The Terms That Help You Make Better Decisions
You are here.
References and further reading
- NIST — AI Risk Management Framework
A framework for understanding and managing AI risks and trustworthiness. - NIST — Trustworthy and Responsible AI
Covers validity and reliability, safety, security and resilience, accountability and transparency, explainability and interpretability, privacy, and fairness. - NIST AI RMF — Risks and Trustworthiness
Further explanation of trustworthiness characteristics and their relationship to context. - Microsoft Learn — RAG and AI evaluation guidance
Covers groundedness, relevance, retrieval quality, response completeness, and evaluation of AI applications. - Microsoft Agent Framework — Evaluation
Covers agent task completion, tool use, groundedness, relevance, and safety evaluation. - Google Cloud — Agent Evaluation
Covers task-success evaluation, final-response evaluation, trajectory evaluation, regression testing, and production monitoring.
Technology, evaluation methods, and product terminology continue to change. Treat vendor terminology as a starting point. For consequential decisions, check the underlying evidence and the specific system documentation.