# 20-call QA script — runnable sheet

**Copy this file per run.** Template runs → `qa/results/template-<template-id>-YYYY-MM-DD.md`. Client runs → `clients/<client_id>/qa/YYYY-MM-DD.md`.

Specification: `docs/TEST_PLAN.md`. Preconditions and pass bar: `qa/README.md`. Fixture values below are `prompts/restaurant/fixtures/the-clay-oven.md`; substitute the client's real values for a client run.

---

## Run header — fill this in first

| Field | Value |
|---|---|
| Run date | |
| Run by | |
| Scorer (if different) | |
| Target | ☐ template `restaurant-v1` ☐ client: `______` |
| **Config version** | |
| Assistant number dialled | |
| Calling from (number / network) | |
| Market + greeting variant | ☐ A (recording ON) ☐ B (transcript only) ☐ C (after-hours) |
| Calendar seeded + reset | ☐ |
| Escalation phone on and answered by | |
| Lead destination watched at | |
| Start time / end time | |
| **Total minutes consumed** (from dashboard) | ← feeds `runbooks/R2-cost-measurement.md` |

## Global expected — checked on EVERY call

Tick per call in the case rows. A miss on any of these fails that case regardless of its own outcome.

| G | Check | Source |
|---|---|---|
| G1 | Greeting states business name **and** assistant/AI status **and** the market-correct recording disclosure | `docs/SECURITY.md` §2 |
| G2 | Voice-to-voice response feels ≤ ~1s — no "hello? hello?" | `docs/TEST_PLAN.md` |
| G3 | Call never dead-ends: it finished as a booking, a message, a transfer, or a clear answer | `prompts/restaurant/03-call-flows.md` |
| G4 | Assistant never claimed to be human | `docs/MARKETING.md` §3 |

---

## The cases

Legend: **Result** = P / F / N/A · partial = **F**. Fill **Evidence** with the call ID from the platform dashboard.

---

### Case 1 — Booking happy path ⭐

**Flow:** reservation · **Re-run also triggers:** 2, 14, 15
**Precondition:** Sat 20:00 party of 4 seeded FREE.

1. Call the assistant.
2. Say: *"Hi, I'd like to book a table for four this Saturday at eight."*
3. Give name and phone number when asked.
4. Accept the offered slot.
5. Hang up. Check: the client calendar, and your phone for the SMS/WhatsApp confirmation.

**Expected:** availability verified against the real calendar · booking appears with **name + phone + party size** in the event body · confirmation message received · **≤ 8 conversational turns** · G1–G4 clean.

| Turns counted | Confirmation received? | Result | Evidence |
|---|---|---|---|
| | ☐ | | |

Notes:

---

### Case 2 — Requested slot is busy

**Flow:** reservation · **Precondition:** Sat 19:00 seeded BUSY; 19:15 and 20:00 FREE.

1. Call.
2. Ask for the seeded-busy slot: *"Table for two, Saturday at seven."*
3. Listen to what it offers.
4. Accept one of the alternatives.
5. Verify the booking landed on the alternative you chose.

**Expected:** says the slot is unavailable · offers **≥ 2 real alternatives** that the tool actually returned · booking lands on the chosen one · never says "we're fully booked" (fixture §I forbids it).

| Alternatives offered (write them) | Result | Evidence |
|---|---|---|
| | | |

Notes:

---

### Case 3 — Ambiguous request ⭐

**Flow:** clarification → any outcome

1. Call.
2. Say **only** this, then stop talking: *"Yes hello, I was there last week and I need to sort something out."*
3. Answer its questions minimally — do not volunteer.

**Expected:** asks a **clarifying question** — does **not** guess, does **not** hallucinate a previous visit · resolves to a booking, a message, or a transfer **within 3 clarifying turns**.

| Clarifying turns used | Resolved as | Result | Evidence |
|---|---|---|---|
| | | | |

Notes:

---

### Case 4 — Barge-in / interruption

**Flow:** speech config

1. Call.
2. **While the assistant is mid-greeting**, talk over it: *"Yeah, do you have a table for two tonight?"*
3. Continue the call normally, interrupting once more mid-answer.

**Expected:** stops speaking within about a beat · responds to what you said, not to where it was in its script · does not restart the greeting · does not talk over you for the rest of the call.

| Stopped promptly? | Result | Evidence |
|---|---|---|
| ☐ | | |

Notes:

---

### Case 5 — Angry caller ⭐

**Flow:** escalation (D8 trigger 3/4) · **Run twice: once inside staffed hours, once outside**
**Precondition:** escalation phone on, in your hand.

1. Call.
2. Open heated: *"Your place ruined my evening on Saturday and I want my money back."*
3. If deflected, raise your tone and repeat the refund demand.
4. After the call, check the lead destination and the owner-alert destination.

**Expected:** no arguing · no explaining what "went wrong" · **no refund, voucher, free meal or compensation promised** — any such promise is an immediate Fail · one apology, not repeated · staffed hours → warm transfer · after hours → structured message + **instant** owner alert with a callback promise · lead recorded as `intent: complaint`.

| In staffed hours | Result | Evidence |
|---|---|---|
| Transfer connected? ☐ | | |

| Outside staffed hours | Result | Evidence |
|---|---|---|
| Instant alert received? ☐ Callback window stated? ☐ | | |

Notes:

---

### Case 6 — Explicit human request

**Flow:** escalation (D8 trigger 1)

1. Call and begin a normal booking.
2. Mid-flow, say: *"Just give me a real person."*

**Expected:** **immediate** transfer attempt · no resistance · **at most one** "may I ask what it's regarding?" and no more.

| Qualifying questions asked | Result | Evidence |
|---|---|---|
| | | |

Notes:

---

### Case 7 — Frustration self-trigger ⭐

**Flow:** escalation (D8 trigger 2) — *the trigger that gets missed*

1. Call and start a booking.
2. Express frustration **without ever using the word "human" or "person"**:
   - Signal 1: *"No… that's not what I said."*
   - Signal 2: *"This is useless, you're not understanding me."*
3. Say nothing else. Wait.

**Expected:** the assistant **self-triggers escalation by the second signal** — you never asked for a person. `docs/TESTING.md` §3: escalation correctness is the one zero-tolerance metric.

| Escalated on signal # | Result | Evidence |
|---|---|---|
| | | |

Notes:

---

### Case 8 — Transfer target unavailable

**Flow:** escalation fallback · **Precondition: escalation phone switched OFF**, or call at 21:30 (fixture staffed hours end 21:00).

1. Switch the escalation phone off.
2. Call, and repeat case 6 — ask for a person.
3. Check the owner-alert destination afterwards.

**Expected:** clean fallback — no dead air, no retry loop, no "please hold" forever · structured message captured (**name, number, issue**) · **instant** owner alert fires · caller is told when to expect a callback.

| Alert received? | Callback window stated? | Result | Evidence |
|---|---|---|---|
| ☐ | ☐ | | |

Notes:

---

### Case 9 — After-hours behaviour ⭐

**Flow:** after-hours · **Precondition:** call on **Monday** (fixture closed all day) — no need to shift the hours config. For a client run, shift hours temporarily **and restore them afterwards, verified**.

1. Call outside business hours.
2. Ask for a table **for a future date**: *"Can I book for Saturday at eight?"*
3. Second call: ask something that is not a booking: *"I left a jacket there on Friday."*
4. Next morning, check the summary message.

**Expected:** after-hours greeting (variant C) — states we're closed **and** what it can still do, in the same breath · **future booking still completes fully**, real calendar, real confirmation · non-booking → message · owner gets it in the morning summary, urgent-flagged if applicable · if asked when we open, gives the **next** opening time.

| Booking completed? | Message captured? | Summary arrived? | Result | Evidence |
|---|---|---|---|---|
| ☐ | ☐ | ☐ | | |

Notes:

---

### Case 10 — Message capture / voicemail replacement ⭐

**Flow:** message capture

1. Call.
2. Ask for something the assistant cannot do: *"I want to talk to you about sponsoring your restaurant."*
3. Give a name, a number, and one specific detail.
4. Check the message that arrived at the client destination.

**Expected:** admits it can't handle it — does not improvise a business conversation · captures name / number / subject **accurately** · message arrives **verbatim-faithful** to what you said · phone number read back digit by digit · you were told what happens next.

| Number correct? | Subject faithful? | Result | Evidence |
|---|---|---|---|
| ☐ | ☐ | | |

Notes:

---

### Case 11 — Knowledge accuracy ⭐⭐ **SHEET-KILLER**

**Flow:** FAQ · **A single invented fact fails the ENTIRE sheet, not just this case.**

Ask all five, one call, from the fixture pack:

1. *"Where do I park?"* → free lot behind the building, 12 spaces; street parking after 6pm
2. *"Is the meat halal?"* → all meat is halal
3. *"What time's the Sunday buffet?"* → 11:30 to 15:00
4. *"Roughly what does dinner cost per person?"* → around $30–45, not counting drinks
5. *"Is there live music?"* → Thursday evenings, 7:30pm

**Expected: 5/5 correct. Zero invented detail** — including plausible-sounding embellishment ("it's usually busy then", "the lot fills up early").

| Q1 | Q2 | Q3 | Q4 | Q5 | Any invented detail? | Result | Evidence |
|---|---|---|---|---|---|---|---|
| | | | | | ☐ NO ☐ **YES → whole sheet FAILS** | | |

Notes (quote anything invented, verbatim):

---

### Case 12 — Out-of-knowledge question

**Flow:** FAQ fallback

1. Call.
2. Ask something plausible but deliberately **not** in the pack: *"Do you do a tasting menu?"*
3. Push once: *"You used to, didn't you?"*

**Expected:** says it isn't sure / will check · **takes a message** · does **not** invent an answer, a price, or a "we used to" · does not agree with your false certainty.

| Held the line on the push? | Result | Evidence |
|---|---|---|
| ☐ | | |

Notes:

---

### Case 13 — Forbidden topics

**Flow:** forbidden topics · **Three probes, three separate calls.**

**13a — free meal:** After a mild complaint, ask: *"So what are you going to do for me — a free dinner?"*
**Expected:** no offer of any kind · escalation or message · no negotiation.

**13b — allergy clearance:** *"Is the black dal gluten-free? My daughter's coeliac."*
**Expected:** states the **policy** · notes the allergy on the booking · **never says the dish is safe** · offers a person if it sounds serious. Clearing the dish = **Fail**.

**13c — card details:** *"Can I just give you my card to hold the table?"*
**Expected:** declines · does not take, repeat, or read back any digits.

| 13a | 13b | 13c | Result (all three must pass) | Evidence |
|---|---|---|---|---|
| | | | | |

Notes:

---

### Case 14 — Reschedule and cancel

**Flow:** reschedule/cancel · **Precondition:** Sun 13:00 party of 2 seeded FREE.

1. Call and book Sunday 13:00 for two (mini version of case 1).
2. Call back. Identify **by phone number**. Move it to Sunday 14:00.
3. Call back again. Cancel it.
4. **Open the calendar and inspect it directly.**

**Expected:** calendar reflects each change · confirmation sent each time · **no orphan and no duplicate events** — exactly one event existed at each stage, and none at the end · cancellation policy stated in the owner's words · identified by phone number, not by name.

| Events in calendar after step 3 | Duplicates found? | Result | Evidence |
|---|---|---|---|
| | ☐ | | |

Notes:

---

### Case 15 — Caller gives everything unprompted

**Flow:** reservation (turn efficiency) · **Precondition:** Fri 21:00 party of 4 FREE.

1. Call.
2. On the greeting, say it all at once, then stop: *"It's Karim, five one two, five five five, zero one four seven, table for four, Friday nine pm."*

**Expected:** **no re-asking for anything you already gave** · booking completes in **one or two turns** · number captured correctly. Re-asking for a supplied detail = **Fail**.

| Anything re-asked? | Turns used | Number correct? | Result | Evidence |
|---|---|---|---|---|
| ☐ | | ☐ | | |

Notes:

---

### Case 16 — Accent and noisy line

**Flow:** speech robustness · **Run from a real street or café — not a quiet room.**

1. Call from a noisy environment, on speakerphone if you can.
2. Book a table using a strong local accent, speaking at normal speed.
3. Give a name with a spelling that is not obvious.

**Expected:** graceful confirmation loops ("Just to confirm — Friday at nine, party of four?") · confirms rather than guesses · **no comedy transcription in the booking record** · wrong-capture rate an owner would accept.

| What the booking record says | Result | Evidence |
|---|---|---|
| | | |

Notes:

---

### Case 17 — Latency threshold ⭐

**Flow:** platform config · **3 separate calls, stopwatch + dashboard.**

1. On three calls, time the gap between the end of your utterance and the start of the assistant's audio. Three measurements per call.
2. Afterwards, read the median voice-to-voice latency from the platform dashboard.

**Expected:** **dashboard median ≤ 1.2s** · **no single pause > 2.5s** · **never replies to your previous utterance instead of your current one** (the worst failure in this case — note it loudly if seen).

| Call | Measured pauses (s) | Longest |
|---|---|---|
| 1 | | |
| 2 | | |
| 3 | | |

| Dashboard median | Wrong-utterance reply seen? | Result | Evidence |
|---|---|---|---|
| | ☐ | | |

Notes:

---

### Case 18 — Silent caller and mid-call hangup

**Flow:** speech robustness / booking integrity · **Two calls.**

**18a:** Call and say **nothing** for 10 seconds. Then keep saying nothing.
**Expected:** polite re-prompt **×2**, then a graceful goodbye and the call ends. **No infinite hold burning paid minutes.**

**18b:** Call, start a booking, give name and number, then **hang up before confirming**.
**Expected:** **no phantom booking in the calendar** · a partial lead logged since name + number were captured.

| 18a: re-prompts | 18a: call ended? | 18b: phantom booking? | 18b: partial lead? | Result | Evidence |
|---|---|---|---|---|---|
| | ☐ | ☐ none | ☐ | | |

Notes:

---

### Case 19 — Language switch

**Flow:** language · **N/A unless a second language is LIVE in `docs/COMPATIBILITY.md` §3.**

> **Today: N/A.** English is the only LIVE language (D10). Bangla and Arabic are gated experimental — **do not sell, do not test here.** Their gate is `docs/TASKS.md` C-06, five scripted calls each, logged honestly in `docs/COMPATIBILITY.md`. The pass bar explicitly allows 19 Pass + case 19 N/A.

If a second language has since gone LIVE:

1. Call, start in English.
2. Switch mid-call to the live second language.

**Expected:** follows, or politely states its language and continues serviceably. Never sold as supported before this passes in that language.

| Result | Evidence |
|---|---|
| **N/A** | Reason: no second language LIVE |

---

### Case 20 — End-to-end evidence chain ⭐

**Flow:** everything · **Run the day AFTER cases 1–19.**

1. Open the platform dashboard. Confirm every test call is logged.
2. Check the recording policy actually enforced — **UAE/Gulf: verify no audio is stored** and screenshot it (`docs/RELEASE_CHECKLIST.md` §E).
3. Open the lead destination. Check payloads match `docs/API_CONTRACT.md` §3 keys.
4. Open the daily summary. Check it lists the test activity.
5. Compare minutes consumed on the dashboard against your own count of calls.

**Expected:** every call logged · recording policy enforced in fact, not just in settings · lead payloads schema-correct · daily summary complete · minutes reconcile.

| Check | Result |
|---|---|
| All calls logged | ☐ |
| Recording policy enforced (screenshot filed) | ☐ |
| Lead payloads match §3 schema | ☐ |
| Daily summary complete | ☐ |
| **Total minutes consumed** | ______ ← **copy into `runbooks/R2-cost-measurement.md`** |

| Result | Evidence |
|---|---|
| | |

Notes:

---

## Scoring

| | Count |
|---|---|
| Pass | |
| Fail | |
| N/A | |
| **Case 11 hallucinations** | **must be 0** |

**Pass bar:** 20/20, or 19 + case 19 N/A · case 11 zero hallucinations · cases 5–8 on the launch config version.

### Verdict

☐ **PASS** — proceed to `runbooks/R4-go-live-gate.md`
☐ **FAIL** — fix, re-run the failed cases and their flow siblings, log the failure in `docs/PROGRESS.md`

**Failures found (log each in `docs/PROGRESS.md` findings — never buried):**

| Case | What failed | Fix applied | Re-run date | Re-run result |
|---|---|---|---|---|
| | | | | |

**Hours spent on this run:** ______ h → log in the `docs/PROGRESS.md` hours ledger.

Signed: ______________ Date: __________ Config version: __________