AIYING ANALYSIS · PRACTICE CARD 002
The Script That Passed Once
Ravindra Dastikop with ChatGPT
We were taught to get our code running first. Now that AI can help with that, what should we learn to do?
Imagine an AI-generated script that combines weekly sales spreadsheets. It finishes without errors and produces a plausible total. This week, however, the same sale appears in two files. If each sale ID should count once, can you predict the correct total before you run the script?
Unlearn this. Treating a successful run as proof of correctness. Start learning this. Turn a business rule into a test by choosing inputs and working out the expected result independently.
What the research supports
Broader tests catch errors that initial checks miss. Jiawei Liu and colleagues expanded the HumanEval coding benchmark’s test cases roughly 80-fold. In their evaluation of 26 models, the expanded tests caught incorrect generated solutions that had passed the original tests. That finding concerns a benchmark of programming tasks. Card 002 applies the same testing idea at spreadsheet scale by choosing an input that distinguishes counting rows from counting sales. Liu et al., NeurIPS, 2023.
Start with the requirement. NIST’s developer verification guidance recommends black-box tests derived from specifications or requirements. In this exercise, the requirement is that each sale ID counts once. You can check the result without inspecting the implementation. If two rows share an ID but show different amounts, agree on what the script should do before writing that test. Flagging the conflict for review is one possible rule. Black, Guttman and Okun, NIST IR 8397, 2021.
Check what learners can do on their own. In a high-school mathematics field experiment, students with unrestricted AI access did better during practice but performed worse on a later unaided assessment than students who had practised without AI. A tutoring version designed to provide hints largely mitigated that downside. The finding gives Card 003 a reason to include an unaided check and informs the fresh shipment case in Card 002. A workshop pilot would help establish whether Card 002 improves independent test design. Bastani et al., PNAS, 2025.
Try it in ten minutes
- On paper, calculate the expected sales total. File A contains A101 for ₹1,000 and A102 for ₹500. File B repeats A101 for ₹1,000. Count each sale ID once.
- Ask AI to review your reasoning and pose one question before giving the answer. Revise your test by recording its inputs, rule and expected total.
- Without AI, try the shipment case. File X lists S1 for 8 units. File Y repeats S1 and adds S2 for 5 units. Each shipment ID counts once. The script reports 21 units. Calculate the expected total and explain the discrepancy.
No coding setup is needed. In a workshop, ask learners to compare which rule they used after they finish the independent attempt. Share the card with someone learning to review generated code or spreadsheet automations.
How will I know I am learning this?
I can predict the correct result for a new test and explain why a successful run can still be wrong.

