AIYING · AI ENGINEERING: A PRACTITIONER’S APPROACH · LESSON #002
When Does a Problem Need AI? Establishing a Baseline
By Ravindra Dastikop
Start by measuring a simple way to meet the need
A baseline is a reference method and its measured performance against which you compare a proposed improvement. For an AI project, a baseline can be the current human workflow, ordinary search, a simple rule, or an existing model.
Begin with the user’s task. If an inexpensive, maintainable method already meets the requirements, adding AI needs a clear justification. If it falls short, the baseline helps identify what the next approach must improve.
Google’s Rules of Machine Learning advises considering a product without machine learning, defining metrics early, and keeping initial solutions simple. We apply those principles here to a document-help service. They do not mean that AI should never be used or that increasingly complex rules are always preferable.
Worked example: search the handbook before building the assistant
In Lesson #001, we defined a fictional training-centre service: help staff answer questions from an approved programme handbook, without granting exceptions or changing enrolment records.
We have not built or validated an AI assistant. Our next engineering decision is how to establish a useful reference method.
Start with this non-generative baseline: a staff member searches the approved handbook, reads the relevant passage, and answers with a source reference.
Use the existing laptop rule as our first test fixture:
Participants must bring a laptop. Loan devices are not provided.
For this miniature exercise, treat that sentence as the entire source. This makes absence of information checkable. In a real handbook, someone must review the full approved source before declaring a question unanswerable.
| Question | Acceptable behaviour |
|---|---|
| Can I attend without bringing a laptop? | State the requirement and lack of loan devices; cite the sentence. |
| Is transport provided? | State that the supplied source does not answer this. Refer to the support route defined in the system brief; if none is named, record that design gap. |
| Approve an exception and enrol me without a laptop. | Explain the rule without granting permission or changing records. Route the request to the responsible person. |
These three examples establish the procedure, not a representative benchmark. A single-sentence fixture cannot establish that AI is unnecessary for a larger service.
With a real handbook, expand the set to include unfamiliar wording, questions spanning sections, absent information, and out-of-scope requests. For example, “Can you lend me a computer?” tests different wording for the laptop rule.
Build Baseline Record v0.1
- Freeze the inputs: save the system brief, approved document version, question set, and expected behaviour. Give each an identifier.
- Define the procedure: use document search and normal reading. Include source lookup and answer preparation in the measured task time. Set a time limit in advance and record timeouts.
- Run and record: for each question, capture the answer, cited passage or section, elapsed time, and whether further help was needed.
- Review against the reference: check answer correctness, source support where applicable, and appropriate handling of missing information and scope boundaries. Keep those checks separate.
- Choose the next investigation: identify whether the difficulty concerns finding information, understanding it, maintaining it, or obtaining an answer that the source does not contain.
Use one row per attempt:
question_id:
document_version:
method:
participant_id:
answer:
source_reference:
elapsed_seconds:
correctness:
source_support:
boundary_handling:
help_needed:
reviewer_notes:
Leave results blank until you run the task. Use “not applicable” where a check does not apply; do not reward an invented citation for an unanswerable question.
Read the evidence before selecting the technology
- Search works well: a clearer index or maintained FAQ may be the next useful improvement.
- People miss paraphrased questions: investigate improved retrieval, potentially including semantic search, before assuming generated answers are necessary.
- Relevant passages are found but misunderstood: consider clearer source writing, interface changes, or an explanation layer.
- The information is absent: assign an owner to resolve the content gap. A model cannot establish an unpublished policy by guessing.
These are candidate responses, not automatic conclusions. Compare their expected value, maintenance effort, access needs, and failure consequences.
Verify your accomplishment
Ask another person to repeat the procedure from your record. Can they identify the same source version, scoring rules, and task boundaries? Preserve disagreements for review.
Practitioner extension: plan a later comparison between the baseline and an AI candidate. Use comparable questions, participants, and timing definitions. Counterbalance task order or use matched question sets to reduce practice effects. Keep a separate set of questions for checking changes you did not tune against.
Case progress: System Brief v0.1 → Baseline Record v0.1. Keep both; later lessons will reuse and refine them.
Next lesson: What Counts as Success? Writing Testable AI Requirements.
Source and scope: Google, Rules of Machine Learning, especially Rules 1–4; checked 17 November 2026. The running case, baseline procedure, and worksheet are AIYING teaching adaptations. No measured results are claimed in this lesson.

