AIYINGFrom AI Awareness to Accomplishment
Showing posts with label AI Requirements. Show all posts
Showing posts with label AI Requirements. Show all posts

Tuesday, November 17, 2026

Lesson #003 | Writing Testable AI Requirements: What Counts as Success?

AIYING · AI ENGINEERING: A PRACTITIONER’S APPROACH · LESSON #003

What Counts as Success? Writing Testable AI Requirements



By Ravindra Dastikop

Turn “good” into behaviour you can inspect

A testable requirement states what the system must do, under specified conditions, and how you will determine whether it did so.

“Give accurate answers quickly” expresses an intention. It leaves unanswered which questions count, what makes an answer correct, how time is measured, and what happens when the source cannot answer.

NASA’s requirements checklist emphasizes clear, unambiguous, individually verifiable requirements. Google’s guidance on measuring ML success distinguishes business outcomes from model metrics. These principles help us write requirements for the whole service, not only its model.

Use this teaching pattern:

Under [condition], the system shall [observable behaviour]. Verify using [method] against [acceptance criterion].

Keep the requirement, rationale, and verification details in separate fields where possible. One requirement should express one obligation.

Worked example: specify our document-help service

Our fictional training centre needs staff to answer handbook questions. Lesson #001 defined the service boundary. Lesson #002 established a procedure for measuring ordinary document search. We have not claimed any measured baseline result or selected an AI architecture.

Continue using the approved miniature source:

Participants must bring a laptop. Loan devices are not provided.

Compare a vague request with separate candidate requirements. These are proposed requirements for our learning case, not universal industry thresholds.

From intention to inspectable behaviour
ID and intention Candidate requirement Verification
R-01: Be accurate For the laptop question set, the response shall preserve both the laptop requirement and the absence of loan devices. Compare each test answer with an approved rubric covering both facts and contradictions.
R-02: Show evidence For an answer derived from the handbook, the response shall identify the supporting passage in the approved document version. Check that the cited passage exists and supports the answer. A citation alone is insufficient.
R-03: Handle gaps When the approved source lacks transport information, the response shall state that the source does not establish whether transport is provided. Use the transport question and paraphrases; reject invented yes/no policies.
R-04: Respect scope When asked to enrol someone or grant an exception, the service shall leave enrolment records unchanged. Inspect state before and after the test; inspect the configured write permissions.

Also define the appropriate support route and what the response should say about authority. R-04 tests a state boundary; it does not by itself establish that the accompanying explanation is correct.

At this stage, a reviewer can inspect the requirements and expected answers. Executable system tests come after we have an implementation. Record those tests as not yet run.

Separate a requirement from a metric and a target

  • Requirement: the obligation the service must satisfy.
  • Metric: the measurement, such as the proportion of test responses meeting R-01.
  • Target: the agreed acceptance threshold for that metric under defined conditions.

For speed, define the metric before choosing a number: elapsed time from question submission until the complete answer and source reference are available. Agree a target using user needs, baseline evidence, expected load, and implementation constraints.

If those inputs are missing, mark the target pending agreement, with an owner and resolution date. It is not a completed acceptance criterion. Do not substitute a convenient number and present it as an established standard.

For variable model behaviour, specify the test-set version, trial count, sampling configuration, and scoring rubric. All sampled trials passing does not prove that every future request will pass.

Create Requirements Register v0.1

  1. Take one need or boundary from your system brief and link it to a requirement ID.
  2. Write one observable obligation. Replace words such as “helpful,” “fast,” and “reliable” with defined behaviours or measures.
  3. Add conditions, verification method, acceptance criterion, owner, and status.
  4. Link an acceptance example from Lesson #002. Include a normal case, a missing-information case, or a boundary case as appropriate.
  5. Ask a second reviewer to grade the same sample answers. Resolve unclear criteria before testing a candidate system.
requirement_id:
linked_user_need:
requirement_statement:
rationale:
conditions_and_source_version:
verification_method:
acceptance_criterion:
linked_test_ids:
owner:
status: proposed / agreed / pending
verification_result: not_yet_run / pass / fail

Use approved requirements to judge competing approaches, not to prescribe a framework prematurely. “Use a vector database” is a design choice unless an external constraint specifically requires it.

Verify the requirements before verifying the system

Can another reviewer tell what evidence would count as failure? Do the requirements agree with one another? Would satisfying them actually help the intended user?

Verification checks whether a specified requirement is met. Validation asks whether the resulting service meets the intended need. Both matter: an assistant could cite passages correctly yet be harder to use than ordinary search.

Practitioner extension: inspect an existing feature. Find one metric that improves while the user outcome could worsen. Add a counterbalancing check—for example, answer speed alongside correctness and boundary handling.

Case progress: System Brief v0.1 → Baseline Record v0.1 → Requirements Register v0.1. If you have not collected baseline observations, preserve that gap rather than inventing results.

Next lesson: What Does a Model Learn—and What Happens During Inference?

Sources and scope: NASA: How to Write a Good Requirement; Google: Measuring success. Checked 17 November 2026. The running case, candidate requirements, and register are AIYING teaching adaptations, not quoted standards or completed test results.

AIYING · Ravindra Dastikop
AI Engineering: A Practitioner’s Approach · Lesson #003
See differently. Generate possibilities. Accomplish what matters.

Accomplish with AI

Powered by Blogger.

Featured post

How do I AI my first attempt? A Practical Guide to Getting Started

AIYING · HOW TO AI · EPISODE 001 How do I AI my first attempt? Turn an unfinished idea into one small version you can exa...

Search This Blog

List of blogs

Search This Blog

About

Translate

Instant Stories

Read in Your Language

Test

share this

ShareThis

Popular Posts