AIYINGFrom AI Awareness to Accomplishment
Showing posts with label AI-Generated Code. Show all posts
Showing posts with label AI-Generated Code. Show all posts

Thursday, November 19, 2026

The Script That Passed Once — How to Test AI-Generated Code | AIYING Practice Card 002

AIYING ANALYSIS · PRACTICE CARD 002


The Script That Passed Once

Ravindra Dastikop with ChatGPT

We were taught to get our code running first. Now that AI can help with that, what should we learn to do?

Imagine an AI-generated script that combines weekly sales spreadsheets. It finishes without errors and produces a plausible total. This week, however, the same sale appears in two files. If each sale ID should count once, can you predict the correct total before you run the script?

Unlearn this. Treating a successful run as proof of correctness. Start learning this. Turn a business rule into a test by choosing inputs and working out the expected result independently.

What the research supports

Broader tests catch errors that initial checks miss. Jiawei Liu and colleagues expanded the HumanEval coding benchmark’s test cases roughly 80-fold. In their evaluation of 26 models, the expanded tests caught incorrect generated solutions that had passed the original tests. That finding concerns a benchmark of programming tasks. Card 002 applies the same testing idea at spreadsheet scale by choosing an input that distinguishes counting rows from counting sales. Liu et al., NeurIPS, 2023.

Start with the requirement. NIST’s developer verification guidance recommends black-box tests derived from specifications or requirements. In this exercise, the requirement is that each sale ID counts once. You can check the result without inspecting the implementation. If two rows share an ID but show different amounts, agree on what the script should do before writing that test. Flagging the conflict for review is one possible rule. Black, Guttman and Okun, NIST IR 8397, 2021.

Check what learners can do on their own. In a high-school mathematics field experiment, students with unrestricted AI access did better during practice but performed worse on a later unaided assessment than students who had practised without AI. A tutoring version designed to provide hints largely mitigated that downside. The finding gives Card 003 a reason to include an unaided check and informs the fresh shipment case in Card 002. A workshop pilot would help establish whether Card 002 improves independent test design. Bastani et al., PNAS, 2025.

Try it in ten minutes

  1. On paper, calculate the expected sales total. File A contains A101 for ₹1,000 and A102 for ₹500. File B repeats A101 for ₹1,000. Count each sale ID once.
  2. Ask AI to review your reasoning and pose one question before giving the answer. Revise your test by recording its inputs, rule and expected total.
  3. Without AI, try the shipment case. File X lists S1 for 8 units. File Y repeats S1 and adds S2 for 5 units. Each shipment ID counts once. The script reports 21 units. Calculate the expected total and explain the discrepancy.

No coding setup is needed. In a workshop, ask learners to compare which rule they used after they finish the independent attempt. Share the card with someone learning to review generated code or spreadsheet automations.

How will I know I am learning this?

I can predict the correct result for a new test and explain why a successful run can still be wrong.

Accomplish with AI

Powered by Blogger.

Featured post

The Explanation That Felt Like Understanding — How to Test What You Learn with AI | AIYING Practice Card 003

Ravindra Dastikop with ChatGPT We were taught to study worked examples until each step made sense. Now that AI can help with that...

Search This Blog

List of blogs

Search This Blog

About

Translate

Instant Stories

Read in Your Language

Test

share this

ShareThis

Popular Posts