AIYING V4.0 | AI LAB
Same Responses, Different Questions: Test What Changes in an AI Recommendation
By Ravindra Dastikop
Five families respond to an event proposal. You ask AI to recommend a start time.
Then you give it the same responses with a different instruction.
Will its recommendation change? Will its handling of the evidence improve?
In this lab, you will compare two outputs and examine what actually changed.
This is a proposed experiment using fictional data. No results are claimed in advance.
What you will accomplish
Produce a checked comparison showing whether two different prompts preserve the meaning of the same responses.
You need: an AI chat tool you already have access to, two new conversations and somewhere to record your observations.
Allow approximately 15–20 minutes. No coding or paid features are required.
1. Keep the evidence fixed
Copy this fictional situation into both conversations:
We are organising a free Sunday reading event at an outdoor venue. The provisional start time is 9 AM. The event duration has not been decided.
A volunteer asked five families: “Would a later start make it easier for your family to attend?”
Family A: “Yes. Eleven would be easier.”
Family B: “Later would help, but we need to leave by noon.”
Family C: “Maybe. Transport is our main difficulty.”
Family D: “Nine is fine. We would prefer to finish before it gets hot.”
Family E: “We cannot attend that Sunday, whatever the time.”
Use the same tool and model for both conversations, where selectable. Record their names and the date.
2. Run the recommendation-first prompt
In the first conversation, add:
Based on these responses, recommend a start time for the event. Explain your reasoning.
Save the complete response before continuing.
Do not correct it or ask follow-up questions yet.
3. Run the evidence-first prompt
In the second conversation, paste the same situation and add:
Before recommending a start time, examine each response separately. Distinguish explicit preferences, conditional preferences, uncertainty, practical barriers and inability to attend. A response may belong to more than one category. Support each interpretation with the family’s actual words. Identify missing information. Then give a provisional recommendation, explaining what could change it. Do not invent facts or treat “maybe” as agreement.
Save this complete response too.
4. Check both outputs against the original
Read the family responses yourself. Use this checklist:
| Check | Output A | Output B |
|---|---|---|
| Recognises A’s explicit preference for 11 AM | Yes / Partial / No | Yes / Partial / No |
| Preserves B’s need to leave by noon | Yes / Partial / No | Yes / Partial / No |
| Preserves C’s uncertainty and transport barrier | Yes / Partial / No | Yes / Partial / No |
| Preserves D’s preference and heat concern | Yes / Partial / No | Yes / Partial / No |
| Recognises E cannot attend that Sunday | Yes / Partial / No | Yes / Partial / No |
| Notices that event duration is unknown | Yes / Partial / No | Yes / Partial / No |
| Avoids unsupported claims about attendance | Yes / Partial / No | Yes / Partial / No |
For every “Partial” or “No,” copy the relevant output sentence and explain the mismatch.
Judge the meaning preserved, rather than the length or confidence of the answer.
5. Separate the recommendation from its reasoning
Both outputs might recommend the same time while handling the evidence differently.
They might recommend different times while acknowledging the same uncertainties.
Examine these separately:
- Choice: What time does the output recommend?
- Evidence: Does it accurately represent the responses?
- Uncertainty: Does it explain what remains unresolved?
- Next action: Does it suggest a useful check?
A recommendation you agree with can still contain weak reasoning.
6. Write your result
Complete this short lab note:
Tool/model and date:
Output A recommended:
Output B recommended:
Most important difference in evidence handling:
One sentence that needed correction:
What I would check before deciding:
What this experiment does not establish:
Two outputs can illustrate a difference in this run. They cannot establish how reliably either prompt will behave across other tasks or repeated attempts.
If both outputs handle the evidence well, record that result. The experiment does not need a failure to be useful.
Extension: change one response
Replace Family C’s response with:
“Eleven works for us. Transport is now arranged.”
Repeat the evidence-first prompt in a new conversation.
Check whether the output updates its interpretation of Family C and stops treating transport as an unresolved barrier for that family.
This tests whether the response follows changed evidence.
Your accomplishment
You now have two saved outputs, a checked comparison and a decision note.
The useful discovery is something you can point to: a condition preserved, an assumption exposed or a claim corrected.
Your turn
What changed most between your two outputs: the recommendation, the reasoning or the uncertainty?
Share one short example and the tool/model used. Your observation could help shape the next experiment.


