Compare the chatbot answers
Five answer flows. The same 24 questions and answer instructions. Inspect the answers, source passages, costs, and review reasons behind the comparison.
| Flow | Supported answers | Median API time | Model cost / 24 answers |
|---|
Each version answered the same 24 customer questions. An AI evaluator scored the answers against rules covering policy accuracy, calculations, evidence, and missing information. The three-passage result is the first draft of the reconsideration flow; its calls and cost are included in that flow’s two-call total.
What the answer needs
What the controls tell you
Nine passages: selected before evaluation using eight development questions. One resembles the gift-return evaluation case and requires more than six passages. Nine and twelve covered all required source phrases; three covered five questions and six covered seven.
Reconsider: another model call with its first draft and exactly the same three passages. It cannot retrieve new information. Compare it with “Three passages” to see whether checking the draft helped.
All documents: all 24 complete source articles, about 29 KB, go directly to the model. This is a useful alternative for this small collection.
Adaptive search: three initial passages and up to two focused searches, each returning three. When the allowance is exhausted, the final answer uses a fresh prompt with accumulated evidence and the common instructions, without planning text or the search conversation. This happened in 9 of 24 cases. Each answer labels its final prompt mode. Retrieved passages include rank and similarity score; complete documents do not.
Time is the sum of model and retrieval API calls, excluding account-verification overhead. Costs use reported generation tokens at USD 1/M input and USD 3.20/M output before credits. Embeddings, vector operations and storage are separate. Required-phrase highlights help locate evidence; they are not a quality score.