AIYINGFrom AI Awareness to Accomplishment
Showing posts with label Spreadsheet Automation. Show all posts
Showing posts with label Spreadsheet Automation. Show all posts

Thursday, November 19, 2026

The Script That Passed Once — How to Test AI-Generated Code | AIYING Practice Card 002

AIYING ANALYSIS · PRACTICE CARD 002


The Script That Passed Once

Ravindra Dastikop with ChatGPT

We were taught to get our code running first. Now that AI can help with that, what should we learn to do?

Imagine an AI-generated script that combines weekly sales spreadsheets. It finishes without errors and produces a plausible total. This week, however, the same sale appears in two files. If each sale ID should count once, can you predict the correct total before you run the script?

Unlearn this. Treating a successful run as proof of correctness. Start learning this. Turn a business rule into a test by choosing inputs and working out the expected result independently.

What the research supports

Broader tests catch errors that initial checks miss. Jiawei Liu and colleagues expanded the HumanEval coding benchmark’s test cases roughly 80-fold. In their evaluation of 26 models, the expanded tests caught incorrect generated solutions that had passed the original tests. That finding concerns a benchmark of programming tasks. Card 002 applies the same testing idea at spreadsheet scale by choosing an input that distinguishes counting rows from counting sales. Liu et al., NeurIPS, 2023.

Start with the requirement. NIST’s developer verification guidance recommends black-box tests derived from specifications or requirements. In this exercise, the requirement is that each sale ID counts once. You can check the result without inspecting the implementation. If two rows share an ID but show different amounts, agree on what the script should do before writing that test. Flagging the conflict for review is one possible rule. Black, Guttman and Okun, NIST IR 8397, 2021.

Check what learners can do on their own. In a high-school mathematics field experiment, students with unrestricted AI access did better during practice but performed worse on a later unaided assessment than students who had practised without AI. A tutoring version designed to provide hints largely mitigated that downside. The finding gives Card 003 a reason to include an unaided check and informs the fresh shipment case in Card 002. A workshop pilot would help establish whether Card 002 improves independent test design. Bastani et al., PNAS, 2025.

Try it in ten minutes

  1. On paper, calculate the expected sales total. File A contains A101 for ₹1,000 and A102 for ₹500. File B repeats A101 for ₹1,000. Count each sale ID once.
  2. Ask AI to review your reasoning and pose one question before giving the answer. Revise your test by recording its inputs, rule and expected total.
  3. Without AI, try the shipment case. File X lists S1 for 8 units. File Y repeats S1 and adds S2 for 5 units. Each shipment ID counts once. The script reports 21 units. Calculate the expected total and explain the discrepancy.

No coding setup is needed. In a workshop, ask learners to compare which rule they used after they finish the independent attempt. Share the card with someone learning to review generated code or spreadsheet automations.

How will I know I am learning this?

I can predict the correct result for a new test and explain why a successful run can still be wrong.

The Script That Passed Once — How to Test AI-Generated Code | Unlearn This. Start Learning This. #002



Ravindra Dastikop with ChatGPT

The situation

Every Friday, you combine sales figures from several spreadsheets. You ask AI to write a script that does it for you. You run it on last week’s files, and the total looks right. Would you trust it next Friday?

The old habit

When we learned to code, getting a program to run was a milestone. It’s easy to carry that habit into everyday work and treat the first successful run as proof that the job is done.

Unlearn this

Stop writing every line of routine code yourself by default. Describe the output you need, give AI a small sample, and let it draft the script.

Start learning this

Practise defining what correct means before you run the code. If the same sale ID appears in two files, should the script count it once, count it twice, or flag it for review? Find out what a repeated ID means in your workflow. Then turn that decision into a test.

Try it once

  1. In your first sample spreadsheet, enter sale A101 for ₹1,000 and sale A102 for ₹500.
  2. In the second spreadsheet, enter the same A101 for ₹1,000.
  3. For this exercise, assume a repeated ID represents the same sale. Before running the script, write down the expected total of ₹1,500.
  4. Run the script on both files. If it reports ₹2,500, it counted A101 twice. The script ran successfully, and your test caught the wrong result.

How will I know I am learning this?

I can design a small test, predict the correct result, and explain why a plausible-looking output is wrong. On the next script, try writing the test case and expected result before asking AI to draft any code.

AIYING · Accomplishing with AI

Accomplish with AI

Powered by Blogger.

Featured post

The Explanation That Felt Like Understanding — How to Test What You Learn with AI | AIYING Practice Card 003

Ravindra Dastikop with ChatGPT We were taught to study worked examples until each step made sense. Now that AI can help with that...

Search This Blog

List of blogs

Search This Blog

About

Translate

Instant Stories

Read in Your Language

Test

share this

ShareThis

Popular Posts