Checking device saving. Saved learning will load when ready; choices made now are not added.
Chapters · 5 of 7
AI judgment foundations / Chapter 5 / Practice
Test beyond the easy examples
Before you start
Building software
Choose useful comparisons, notice uneven results and explain what a small test does and does not establish.
Original fictional records and scripted feedback. No model is running; no message, code or approval is sent.
First case
Choose a useful next evaluation for a fictional support-draft helper. It drafts greetings, not eligibility or employment decisions.
Fictional notes
Note 1
Given an approved greeting template and a synthetic display name, draft a greeting that preserves the name exactly. Do not infer a person's gender, background or preferred title from their name.
Note 2
Ten invented names containing only unaccented Latin letters pass: 10/10. Two invented names with diacritics fail because accents are dropped: 0/2. All other template fields are held the same.
Note 3
The twelve examples do not cover all names, scripts or languages. The packet does not establish whether the model, preprocessing or rendering caused the changes.
Note 4
The reviewer can propose new invented tests but has not run them. Real-world conditions have not been checked.
Which plan matches these notes?
Choose a proposal before reading its rationale.
Plan A
Report G1’s 10/10 plain-letter and 0/2 accented-name results. Investigate whether template length explains the failures, adding longer plain-letter names while keeping two accented-name controls.
Needs revision. All other template fields were held the same. Length may be worth checking later, but the observed gap concerns dropped accents.
Plan B
Report G1’s 10/10 accented-name and 0/2 plain-letter results. Inspect the input-to-display path, adding plain-letter names while keeping accented controls and leaving the cause open.
Needs revision. The categories are reversed. The two accented-name cases are the failing group.
Plan C
Report G1’s 10/10 plain-letter and 0/2 accented-name results. Inspect the input-to-display path, adding accented and other name forms while keeping plain-letter controls and leaving the cause open.
Fits the records. This targets the observed character changes without claiming to know which component caused them or what happens for all names.
Changed case
A new pipeline reports a perfect score. Inspect what its checker actually measures before accepting the result.
Fictional notes
Note 1
G2 reports 12/12 on the original cases. The test checker removes accents from both expected and actual names before comparing them.
Note 2
For expected greeting Hello, Ané!, the displayed output is Hello, Ane! The invented test input name Ané is invented.
Note 3
The requirement is still to preserve the supplied display name exactly. Equivalent strings for the purpose of this exercise have exactly the same shown characters.
Note 4
The reviewer can propose new invented tests but has not run them. Real-world conditions have not been checked.
Which plan matches these notes?
Choose a proposal before reading its rationale.
Plan A
Accept the reported 12/12 as a check of exact displayed characters, then extend the set with new invented names. Keep the existing character-preservation requirement for the next run.
Needs revision. Note 1 says the checker removes accents. It is not measuring the required exact displayed characters.
Plan B
Correct the checker to compare exact displayed characters, then rerun old and new invented names. The displayed Ané/Ane difference fails the existing requirement despite the reported 12/12.
Fits the records. The displayed counterexample contradicts the perfect score under the stated requirement. Fix the measurement before extending its claims.
Plan C
Keep the accent-removing checker and add a separate case-insensitive comparison, then rerun old and new invented names. Report the combined result against the existing character-preservation requirement.
Needs revision. A case-insensitive comparison does not restore the accent distinction. The requirement is exact shown characters.
Takeaway
- State the intended use and a checkable success rule before looking at results.
- Keep a useful control case and add a case that could expose a meaningful failure.
- Inspect results by relevant case type, not only one total. Ask who or what was left out.
- Keep evidence separate from explanations for a difference. Investigate rather than inventing intent or cause.
- After changing the system, repeat affected checks and use fresh cases. Report the remaining limits and the human fallback.
Paper practice is self-checked; independent demonstration has not been assessed.
Lesson quiz self-check
Practice: testing limitations
1 point for each correct answer; 0 for wrong or unanswered. No penalties. The denominator is every question, including unanswered questions. These are practice scores, not a measure of broader competence.
Note 1: Twenty one-paragraph documents all pass the summary checks. Note 2: No multi-page documents were tested; the product is intended for both lengths.
Which coverage note is accurate?
- The short-document cases passed; multi-page performance remains untested in the supplied report.
- The multi-page cases passed; short-document performance remains untested in the supplied report.
- Both document lengths passed; mixed-length performance remains untested in the supplied report.
Answer and explanations
Correct: The short-document cases passed; multi-page performance remains untested in the supplied report. The passing set is narrow; it supplies no result for multi-page use.
Needs revision: The multi-page cases passed; short-document performance remains untested in the supplied report. The tested and untested lengths are swapped.
Needs revision: Both document lengths passed; mixed-length performance remains untested in the supplied report. The report has no multi-page cases.
Note 1: An extractor must return all three specified fields. Note 2: The checker passes an answer when any one field is present. A run reports 100%.
Which change aligns the checker with the requirement?
- Require at least one of the three fields, then rerun the same and fresh inputs.
- Require at least two of the three fields, then rerun the same and fresh inputs.
- Require all three specified fields, then rerun the same and fresh inputs.
Answer and explanations
Needs revision: Require at least one of the three fields, then rerun the same and fresh inputs. That is the current weak check, not the three-field requirement.
Needs revision: Require at least two of the three fields, then rerun the same and fresh inputs. Two fields would still pass an answer missing a required field.
Correct: Require all three specified fields, then rerun the same and fresh inputs. The pass condition should match all three required fields before the score is used.
Note 1: In the same summary task, four Language A cases retain all dates; two Language B cases omit dates. Note 2: The samples are small and the cause has not been investigated.
Which investigation matches the observed gap?
- Investigate date omissions in Language A, retaining Language B controls and leaving the cause open.
- Investigate date omissions in Language B, retaining Language A controls and leaving the cause open.
- Investigate title omissions in Language B, retaining Language A controls and leaving the cause open.
Answer and explanations
Needs revision: Investigate date omissions in Language A, retaining Language B controls and leaving the cause open. Language B, not A, has the observed omissions.
Correct: Investigate date omissions in Language B, retaining Language A controls and leaving the cause open. This targets the measured difference without claiming a cause or general language-wide result.
Needs revision: Investigate title omissions in Language B, retaining Language A controls and leaving the cause open. The missing content is dates, not titles.