Lesson 4 of 5 · Free to read
Check AI output without reading every word
Check AI output without reading every word
Audience: Final-year students and engineers in their first three years
Prerequisites: Lessons 1–3. Python 3 for the checker script.
Time: about 25 minutes
What you will keep: a small checker script and a way to compare two prompts fairly
This lesson does not cover: formal evaluation frameworks for AI products.
Why this matters
"It looked fine" is how bad AI output reaches production. But you cannot read every line of every answer, either. The middle path is lightweight evaluation: a few cheap, repeatable checks that catch most problems, and a fair way to decide whether a new prompt is actually better than the old one.
Three cheap checks
1 · Run it. For code, the best evaluation is execution: tests, a type checker, a linter, or running the program on a real input. Ask for code you can run, then run it.
2 · Check the shape. If you asked for JSON with certain keys, a table with certain columns, or exactly five items, check that mechanically. Shape problems are the easiest to catch and the most common when a prompt is reused.
3 · Compare against a known answer. Keep a few inputs whose correct output you already know: your golden examples. When you change a prompt, run it on those first. If it gets a known case wrong, you have learned something without reading anything else.
A checker for structured replies
Suppose your test-case prompt from lesson 1 asks for JSON like {"tests": [{"name": ..., "input": ..., "expected": ...}]}. Save each raw reply in a text file, then run this script on them. It uses only the Python standard library.
"""Check saved AI replies against the shape you asked for.
Usage: python check_replies.py replies/*.txt
Each file holds one raw reply that should be JSON like:
{"tests": [{"name": "...", "input": "...", "expected": "..."}]}
"""
import json
import sys
REQUIRED_KEYS = {"name", "input", "expected"}
def check(text):
problems = []
try:
data = json.loads(text)
except json.JSONDecodeError as err:
return [f"not valid JSON ({err.msg} at line {err.lineno})"]
tests = data.get("tests") if isinstance(data, dict) else None
if not isinstance(tests, list) or not tests:
return ['no non-empty "tests" list']
names = set()
for i, t in enumerate(tests, start=1):
if not isinstance(t, dict):
problems.append(f"test {i} is not an object")
continue
missing = REQUIRED_KEYS - t.keys()
if missing:
problems.append(f"test {i} missing {sorted(missing)}")
name = t.get("name")
if name in names:
problems.append(f"test {i} repeats the name {name!r}")
names.add(name)
return problems
if __name__ == "__main__":
failed = 0
for path in sys.argv[1:]:
with open(path, encoding="utf-8") as f:
problems = check(f.read())
status = "OK " if not problems else "FAIL"
print(f"{status} {path}" + ("" if not problems else ": " + "; ".join(problems)))
failed += bool(problems)
print(f"{len(sys.argv) - 1 - failed} of {len(sys.argv) - 1} replies usable")
Run it:
python check_replies.py replies/*.txt
Example output on three saved replies:
OK replies/a.txt
FAIL replies/b.txt: not valid JSON (Expecting value at line 1)
FAIL replies/c.txt: test 1 missing ['expected']; test 2 repeats the name 'x'
1 of 3 replies usable
Reply b started with "Sure! Here are the tests:", which is a common reason JSON fails to parse. That tells you to add "Reply with JSON only, no other text" to the prompt. Change REQUIRED_KEYS to match whatever shape you ask for.
Comparing two prompts fairly
When you improve a prompt, prove it on the same inputs:
- Pick five real inputs, including at least one awkward case.
- Run the old prompt and the new prompt on all five, in fresh conversations.
- Score each output with the same short rubric, for example: correct (0–2), right format (0–1), usable without edits (0–1).
- Write the scores in a small table and keep the prompt that wins on the cases you care about.
| Input | Old prompt | New prompt |
|---|---|---|
| Empty file | 2 | 3 |
| Normal ticket | 4 | 4 |
| Very long ticket | 1 | 3 |
Five inputs will not give you statistics, but they stop you keeping a "better" prompt that only looked better on the one example you tried. AI answers vary from run to run, so for an important prompt run each input twice.
Exercise: check and compare
- Take the structured-output prompt you wrote in lesson 1 or 2. Save its replies for five inputs as text files.
- Adapt
check_replies.pyto your shape and run it. Fix the prompt until all five replies pass. - Write one improvement to the prompt, then compare old against new on the same five inputs with the rubric above.
- Save the scores table in your prompt library.
Key points
- Run code, check shape mechanically, compare against known answers.
- Ask for "JSON only" when a script will read the reply.
- Compare prompts on the same five inputs, not one lucky example.