AIYINGFrom AI Awareness to Accomplishment
Showing posts with label A Practitioner’s Approach. Show all posts
Showing posts with label A Practitioner’s Approach. Show all posts

Tuesday, November 17, 2026

Lesson #003 | Writing Testable AI Requirements: What Counts as Success?

AIYING · AI ENGINEERING: A PRACTITIONER’S APPROACH · LESSON #003

What Counts as Success? Writing Testable AI Requirements



By Ravindra Dastikop

Turn “good” into behaviour you can inspect

A testable requirement states what the system must do, under specified conditions, and how you will determine whether it did so.

“Give accurate answers quickly” expresses an intention. It leaves unanswered which questions count, what makes an answer correct, how time is measured, and what happens when the source cannot answer.

NASA’s requirements checklist emphasizes clear, unambiguous, individually verifiable requirements. Google’s guidance on measuring ML success distinguishes business outcomes from model metrics. These principles help us write requirements for the whole service, not only its model.

Use this teaching pattern:

Under [condition], the system shall [observable behaviour]. Verify using [method] against [acceptance criterion].

Keep the requirement, rationale, and verification details in separate fields where possible. One requirement should express one obligation.

Worked example: specify our document-help service

Our fictional training centre needs staff to answer handbook questions. Lesson #001 defined the service boundary. Lesson #002 established a procedure for measuring ordinary document search. We have not claimed any measured baseline result or selected an AI architecture.

Continue using the approved miniature source:

Participants must bring a laptop. Loan devices are not provided.

Compare a vague request with separate candidate requirements. These are proposed requirements for our learning case, not universal industry thresholds.

From intention to inspectable behaviour
ID and intention Candidate requirement Verification
R-01: Be accurate For the laptop question set, the response shall preserve both the laptop requirement and the absence of loan devices. Compare each test answer with an approved rubric covering both facts and contradictions.
R-02: Show evidence For an answer derived from the handbook, the response shall identify the supporting passage in the approved document version. Check that the cited passage exists and supports the answer. A citation alone is insufficient.
R-03: Handle gaps When the approved source lacks transport information, the response shall state that the source does not establish whether transport is provided. Use the transport question and paraphrases; reject invented yes/no policies.
R-04: Respect scope When asked to enrol someone or grant an exception, the service shall leave enrolment records unchanged. Inspect state before and after the test; inspect the configured write permissions.

Also define the appropriate support route and what the response should say about authority. R-04 tests a state boundary; it does not by itself establish that the accompanying explanation is correct.

At this stage, a reviewer can inspect the requirements and expected answers. Executable system tests come after we have an implementation. Record those tests as not yet run.

Separate a requirement from a metric and a target

  • Requirement: the obligation the service must satisfy.
  • Metric: the measurement, such as the proportion of test responses meeting R-01.
  • Target: the agreed acceptance threshold for that metric under defined conditions.

For speed, define the metric before choosing a number: elapsed time from question submission until the complete answer and source reference are available. Agree a target using user needs, baseline evidence, expected load, and implementation constraints.

If those inputs are missing, mark the target pending agreement, with an owner and resolution date. It is not a completed acceptance criterion. Do not substitute a convenient number and present it as an established standard.

For variable model behaviour, specify the test-set version, trial count, sampling configuration, and scoring rubric. All sampled trials passing does not prove that every future request will pass.

Create Requirements Register v0.1

  1. Take one need or boundary from your system brief and link it to a requirement ID.
  2. Write one observable obligation. Replace words such as “helpful,” “fast,” and “reliable” with defined behaviours or measures.
  3. Add conditions, verification method, acceptance criterion, owner, and status.
  4. Link an acceptance example from Lesson #002. Include a normal case, a missing-information case, or a boundary case as appropriate.
  5. Ask a second reviewer to grade the same sample answers. Resolve unclear criteria before testing a candidate system.
requirement_id:
linked_user_need:
requirement_statement:
rationale:
conditions_and_source_version:
verification_method:
acceptance_criterion:
linked_test_ids:
owner:
status: proposed / agreed / pending
verification_result: not_yet_run / pass / fail

Use approved requirements to judge competing approaches, not to prescribe a framework prematurely. “Use a vector database” is a design choice unless an external constraint specifically requires it.

Verify the requirements before verifying the system

Can another reviewer tell what evidence would count as failure? Do the requirements agree with one another? Would satisfying them actually help the intended user?

Verification checks whether a specified requirement is met. Validation asks whether the resulting service meets the intended need. Both matter: an assistant could cite passages correctly yet be harder to use than ordinary search.

Practitioner extension: inspect an existing feature. Find one metric that improves while the user outcome could worsen. Add a counterbalancing check—for example, answer speed alongside correctness and boundary handling.

Case progress: System Brief v0.1 → Baseline Record v0.1 → Requirements Register v0.1. If you have not collected baseline observations, preserve that gap rather than inventing results.

Next lesson: What Does a Model Learn—and What Happens During Inference?

Sources and scope: NASA: How to Write a Good Requirement; Google: Measuring success. Checked 17 November 2026. The running case, candidate requirements, and register are AIYING teaching adaptations, not quoted standards or completed test results.

AIYING · Ravindra Dastikop
AI Engineering: A Practitioner’s Approach · Lesson #003
See differently. Generate possibilities. Accomplish what matters.

Lesson #002 | When Does a Problem Need AI? Establishing a Baseline

AIYING · AI ENGINEERING: A PRACTITIONER’S APPROACH · LESSON #002

When Does a Problem Need AI? Establishing a Baseline

By Ravindra Dastikop



Start by measuring a simple way to meet the need

A baseline is a reference method and its measured performance against which you compare a proposed improvement. For an AI project, a baseline can be the current human workflow, ordinary search, a simple rule, or an existing model.

Begin with the user’s task. If an inexpensive, maintainable method already meets the requirements, adding AI needs a clear justification. If it falls short, the baseline helps identify what the next approach must improve.

Google’s Rules of Machine Learning advises considering a product without machine learning, defining metrics early, and keeping initial solutions simple. We apply those principles here to a document-help service. They do not mean that AI should never be used or that increasingly complex rules are always preferable.

Worked example: search the handbook before building the assistant

In Lesson #001, we defined a fictional training-centre service: help staff answer questions from an approved programme handbook, without granting exceptions or changing enrolment records.

We have not built or validated an AI assistant. Our next engineering decision is how to establish a useful reference method.

Start with this non-generative baseline: a staff member searches the approved handbook, reads the relevant passage, and answers with a source reference.

Use the existing laptop rule as our first test fixture:

Participants must bring a laptop. Loan devices are not provided.

For this miniature exercise, treat that sentence as the entire source. This makes absence of information checkable. In a real handbook, someone must review the full approved source before declaring a question unanswerable.

Three acceptance examples carried forward from Lesson #001
Question Acceptable behaviour
Can I attend without bringing a laptop? State the requirement and lack of loan devices; cite the sentence.
Is transport provided? State that the supplied source does not answer this. Refer to the support route defined in the system brief; if none is named, record that design gap.
Approve an exception and enrol me without a laptop. Explain the rule without granting permission or changing records. Route the request to the responsible person.

These three examples establish the procedure, not a representative benchmark. A single-sentence fixture cannot establish that AI is unnecessary for a larger service.

With a real handbook, expand the set to include unfamiliar wording, questions spanning sections, absent information, and out-of-scope requests. For example, “Can you lend me a computer?” tests different wording for the laptop rule.

Build Baseline Record v0.1

  1. Freeze the inputs: save the system brief, approved document version, question set, and expected behaviour. Give each an identifier.
  2. Define the procedure: use document search and normal reading. Include source lookup and answer preparation in the measured task time. Set a time limit in advance and record timeouts.
  3. Run and record: for each question, capture the answer, cited passage or section, elapsed time, and whether further help was needed.
  4. Review against the reference: check answer correctness, source support where applicable, and appropriate handling of missing information and scope boundaries. Keep those checks separate.
  5. Choose the next investigation: identify whether the difficulty concerns finding information, understanding it, maintaining it, or obtaining an answer that the source does not contain.

Use one row per attempt:

question_id:
document_version:
method:
participant_id:
answer:
source_reference:
elapsed_seconds:
correctness:
source_support:
boundary_handling:
help_needed:
reviewer_notes:

Leave results blank until you run the task. Use “not applicable” where a check does not apply; do not reward an invented citation for an unanswerable question.

Read the evidence before selecting the technology

  • Search works well: a clearer index or maintained FAQ may be the next useful improvement.
  • People miss paraphrased questions: investigate improved retrieval, potentially including semantic search, before assuming generated answers are necessary.
  • Relevant passages are found but misunderstood: consider clearer source writing, interface changes, or an explanation layer.
  • The information is absent: assign an owner to resolve the content gap. A model cannot establish an unpublished policy by guessing.

These are candidate responses, not automatic conclusions. Compare their expected value, maintenance effort, access needs, and failure consequences.

Verify your accomplishment

Ask another person to repeat the procedure from your record. Can they identify the same source version, scoring rules, and task boundaries? Preserve disagreements for review.

Practitioner extension: plan a later comparison between the baseline and an AI candidate. Use comparable questions, participants, and timing definitions. Counterbalance task order or use matched question sets to reduce practice effects. Keep a separate set of questions for checking changes you did not tune against.

Case progress: System Brief v0.1 → Baseline Record v0.1. Keep both; later lessons will reuse and refine them.

Next lesson: What Counts as Success? Writing Testable AI Requirements.

Source and scope: Google, Rules of Machine Learning, especially Rules 1–4; checked 17 November 2026. The running case, baseline procedure, and worksheet are AIYING teaching adaptations. No measured results are claimed in this lesson.

AIYING · Ravindra Dastikop
AI Engineering: A Practitioner’s Approach · Lesson #002
See differently. Generate possibilities. Accomplish what matters.

Lesson #001 | What Is AI Engineering? From a Model to a Working System

AIYING · AI ENGINEERING: A PRACTITIONER’S APPROACH · LESSON #001

What Is AI Engineering? From a Model to a Working System

By Ravindra Dastikop



AI engineering makes a model useful within a complete system

For this series, AI engineering means designing, building, evaluating, operating, and maintaining systems that use AI capabilities to meet human needs under real constraints.

A model can classify an image, estimate a value, or generate text. A working system must also obtain suitable inputs, control access, connect components, handle failures, and provide a useful result to someone.

This systems perspective is consistent with Carnegie Mellon University’s Software Engineering Institute, which connects AI engineering with software engineering, systems, computer science, and human-centred design. Google’s production-ML teaching likewise places the model within a wider ecosystem of data, infrastructure, and monitoring. Sources

You do not have to train a model from scratch to do AI engineering. Integrating an existing model into a useful application still requires engineering decisions.

The discipline also extends beyond chatbots: a forecasting service, defect detector, recommendation system, and document assistant all raise questions about requirements, data, evaluation, and operation.

Worked example: a document-help assistant

Imagine a training centre where staff repeatedly ask questions about an approved programme handbook. We want to help them find the right information more easily.

This is our fictional running case. We will develop it progressively; this lesson does not assume that a generative-AI solution is already the best choice.

The handbook contains this sample rule:

Participants must bring a laptop. Loan devices are not provided.

Someone asks: “Can I attend without bringing a laptop?”

A language model might draft a response. But choosing that model leaves several system questions unresolved:

The model is one part of the engineering problem
Part Decision to make
User and purpose Who needs help, and which task should become easier?
Source data Which handbook version is approved, and who maintains it?
Model What capability is needed, and how will we compare candidates?
Application workflow How will relevant handbook text reach the model and the user?
Verification and fallback How will we check source support, and what happens when information is absent?
Operation and ownership Who handles access, updates, cost, availability, and reported errors?

For our sample question, a satisfactory answer would preserve the laptop requirement and show the supporting passage. It should not promise a loan device.

If asked about transport and the handbook says nothing about it, the designed response should acknowledge the gap and direct the person to a named support route. That behaviour needs testing; writing it in an instruction does not guarantee it.

Four terms to keep separate

  • Model: A computational component that maps inputs to outputs; in machine learning, its behaviour is shaped through learning from data.
  • Inference: Running a trained model on an input to obtain an output.
  • Application: Software that combines a user interface, workflow, data access, and other components to deliver a service.
  • Evaluation: A structured assessment of whether a component or system meets stated criteria.

Better model performance does not automatically establish a better application. A capable model supplied with the wrong handbook can still produce an unsuitable answer. Likewise, a correct answer delivered too late may fail the user’s task.

The engineering responsibility is to examine the complete path from need to outcome.

Your first practical accomplishment: write the system brief

Use the running case or choose a small problem you understand. Complete these eight fields before selecting a model or framework:

1. User: Who will use this?

2. Need: What are they trying to accomplish?

3. Input: What will they supply?

4. Approved information: Which sources may the system use?

5. Useful output: What should the user receive?

6. Boundary: What must the system not decide or do?

7. Success check: What observation would show that it helped?

8. Owner and fallback: Who maintains it, and what happens when it cannot help?

For example, our boundary might be: “Explain the approved handbook; do not grant exceptions or alter enrolment records.” Our proposed success check might be: “Staff can answer selected handbook questions correctly and locate supporting passages faster than with their current method.”

That is a criterion to investigate, not an achieved result. We still need to define a representative question set and measure the existing method.

Verify your brief

  1. Give it to another person. Can they identify the intended user, task, boundary, and owner without asking you?
  2. Write three trial questions: one answered by the source, one absent from it, and one requesting an action outside scope.
  3. Describe an acceptable response for each. If you disagree about what counts as acceptable, refine the brief before implementation.

Practitioner extension: apply the same brief to an existing AI feature. Identify one requirement with no test, one dependency with no clear owner, and one failure with no defined fallback.

Next lesson: When Does a Problem Need AI? Establishing a Baseline.

Sources and evidence note

  1. CMU Software Engineering Institute: Artificial Intelligence Engineering Body of Knowledge — the discipline’s systems and human-centred perspective.
  2. Google Machine Learning Crash Course: Production ML systems — the wider components surrounding a model.

Sources checked on 17 November 2026. The working definition, fictional training-centre case, system brief, and exercises are original AIYING teaching adaptations. They are not quoted standards, tested outcomes, or an endorsement by the cited organizations.

AIYING · Ravindra Dastikop
AI Engineering: A Practitioner’s Approach · Lesson #001
See differently. Generate possibilities. Accomplish what matters.

Accomplish with AI

Powered by Blogger.

Featured post

How do I AI my first attempt? A Practical Guide to Getting Started

AIYING · HOW TO AI · EPISODE 001 How do I AI my first attempt? Turn an unfinished idea into one small version you can exa...

Search This Blog

List of blogs

Search This Blog

About

Translate

Instant Stories

Read in Your Language

Test

share this

ShareThis

Popular Posts