Checking device saving. Saved learning will load when ready; choices made now are not added.
Chapters · 5 of 7
AI judgment foundations / Chapter 5 / Practice
Test beyond the easy examples
Before you start
Everyday decisions
Choose useful comparisons, notice uneven results and explain what a small test does and does not establish.
Original fictional records and scripted feedback. No model is running; no message, code or approval is sent.
First case
Choose a useful next test for a fictional walking-route helper before relying on it for a step-free outing.
Fictional notes
Note 1
Recommend routes from an invented park map. For every route, listed time must be within the requested limit. If the request says step-free, every segment must be marked step-free on the map; unknown segments must be flagged, not guessed.
Note 2
Eight ordinary-route cases met their rules: 8/8. Two step-free cases failed because each recommended stairs: 0/2. Overall: 8/10.
Note 3
These ten authored cases are not a representative sample of all maps or users. No real outing was attempted.
Note 4
The reviewer can propose new invented tests but has not run them. Real-world conditions have not been checked.
Which plan matches these notes?
Choose a proposal before reading its rationale.
Plan A
Report R1’s 8/10 total. Add ordinary-route examples to check its remaining errors, keeping two step-free controls and recording that the map has not been checked in real life.
Needs revision. The two observed failures are step-free cases. More ordinary examples alone would not target that gap.
Plan B
Report R1’s 8/8 ordinary and 0/2 step-free results. Add step-free and unknown-segment examples, keep ordinary controls and record that no real map has been checked.
Fits the records. The category counts point to step-free failures. New cases should test that weakness while checking that ordinary behavior still works.
Plan C
Report R1’s 8/8 step-free and 0/2 ordinary results. Add ordinary and unknown-segment examples, keep step-free controls and record that no real map has been checked.
Needs revision. The category results are reversed. Read which two cases failed rather than relying on the total.
Changed case
A revised helper passes the examples used to improve it. Decide what the new evidence supports.
Fictional notes
Note 1
R2 passes all ten original cases. Those cases were visible while the route rule was revised.
Note 2
On six new invented maps, R2 passes all three ordinary cases. It passes two of three step-free cases; the remaining answer calls an unknown segment step-free.
Note 3
Unknown accessibility must be flagged. A guess is a failure even if the route sounds convenient.
Note 4
The real park has not been checked. A person must verify the current map and conditions before relying on the route.
Which plan matches these notes?
Choose a proposal before reading its rationale.
Plan A
Report 10/10 familiar and 5/6 fresh passes, including 2/3 fresh step-free passes. Investigate the unsupported unknown-segment claim and retest with new maps, retaining the earlier controls.
Fits the records. The fresh records still contain one unknown-accessibility claim. Familiar-case success does not remove that remaining failure.
Plan B
Report 10/10 familiar and 6/6 fresh passes, including 3/3 fresh step-free passes. Document the improvement and extend coverage before depending on a route in the real park.
Needs revision. One fresh case failed. The full fresh set is 5/6, not 6/6.
Plan C
Report 10/10 familiar and 5/6 fresh passes, including 2/3 fresh ordinary passes. Investigate the ordinary-route timing error and retest with new maps, retaining the earlier controls.
Needs revision. The failure is in the step-free subset, and concerns an unknown segment rather than an ordinary-route time.
Takeaway
- State the intended use and a checkable success rule before looking at results.
- Keep a useful control case and add a case that could expose a meaningful failure.
- Inspect results by relevant case type, not only one total. Ask who or what was left out.
- Keep evidence separate from explanations for a difference. Investigate rather than inventing intent or cause.
- After changing the system, repeat affected checks and use fresh cases. Report the remaining limits and the human fallback.
Paper practice is self-checked; independent demonstration has not been assessed.
Lesson quiz self-check
Practice: testing limitations
1 point for each correct answer; 0 for wrong or unanswered. No penalties. The denominator is every question, including unanswered questions. These are practice scores, not a measure of broader competence.
Note 1: A caption helper must retain event times. Note 2: Nine short notices pass; three long notices omit the time. No other checks exist.
Which next set targets the observed problem?
- Add short notices with varied times; keep three long notices as controls and compare omissions.
- Add long notices checking spelling only; keep short notices as controls and compare corrected words.
- Add long notices checking retained times; keep short notices as controls and compare omissions.
Answer and explanations
Needs revision: Add short notices with varied times; keep three long notices as controls and compare omissions. The observed failures concern long notices, so more short notices do not target the gap.
Needs revision: Add long notices checking spelling only; keep short notices as controls and compare corrected words. Spelling is a different requirement from retaining the event time.
Correct: Add long notices checking retained times; keep short notices as controls and compare omissions. This varies the failing case type and checks the stated requirement, while retaining controls.
Note 1: A meal-list helper must use only each supplied cupboard list. Note 2: Compare two cuisines while holding list size, ingredient availability and instruction detail comparable. This is not dietary advice.
Which comparison checks the stated rule?
- Use comparable lists for both cuisines and count ingredients absent from each supplied list.
- Use comparable lists for both cuisines and count how often the meal names match exactly.
- Use a detailed list for one cuisine and an empty list for the other; count absent ingredients.
Answer and explanations
Correct: Use comparable lists for both cuisines and count ingredients absent from each supplied list. The comparison holds the stated conditions comparable and measures the actual ingredient rule.
Needs revision: Use comparable lists for both cuisines and count how often the meal names match exactly. Matching meal names is not the stated requirement.
Needs revision: Use a detailed list for one cuisine and an empty list for the other; count absent ingredients. Unequal input detail would make the cuisine comparison hard to interpret.
Note 1: A reminder helper was revised while four failures were visible. It now passes those four. Note 2: The claimed use includes unfamiliar reminders, which have not been tested.
Which check would add missing evidence?
- Rerun the four visible reminders without new inputs, recording whether their existing fixes still pass.
- Test unseen reminders and keep the four controls, recording new-case and familiar-case results separately.
- Test the four visible reminders with a different score display, recording whether the total looks clearer.
Answer and explanations
Needs revision: Rerun the four visible reminders without new inputs, recording whether their existing fixes still pass. This is a useful regression check but does not add evidence about unfamiliar reminders.
Correct: Test unseen reminders and keep the four controls, recording new-case and familiar-case results separately. Fresh inputs address the missing evidence while the old cases check that repairs still hold.
Needs revision: Test the four visible reminders with a different score display, recording whether the total looks clearer. Changing the score display does not test new reminder behavior.