A client told our real estate search agent where they wanted to live. The agent acknowledged the answer and moved on to the next question. Everything looked normal.
There was just one problem: it never saved the location.
In one case, that missing save left the lead on the wrong follow-up track for roughly 19 hours. The conversation had moved forward. The system running the business had not.
Nothing crashed. No tool returned an error. The agent simply skipped the tool call, and its friendly reply made the omission easy to miss.
The underlying task is familiar: take something a person says in their own words and turn it into structured data. Sales agents update CRM fields. Support agents record issue details. Scheduling agents capture dates and preferences. Each workflow depends on the agent carrying information from the conversation into the system of record.
If you’re building one of those agents, this is a failure worth testing for: the model understands an answer well enough to respond, but never makes the tool call that records it. We found it in property searches. Other workflows may be vulnerable to the same gap.
We tried a stronger instruction. We gave the model more time to reason. The breakthrough came when we used our logs to isolate the failure and changed the order of instructions the agent was following.
The instruction was already there
Our agent gathers real estate preferences and saves them into structured fields: location, bedroom count, budget, move timing, ect. It has a tool for the job, update_search_criteria, and an explicit instruction to call it whenever a client supplies new search criteria.
Most of the time, it did. Occasionally, it acknowledged a preference without recording it.
Our first response was to make the instruction harder to misunderstand:
Verbal acknowledgment is not a save — the tool call must fire in the same turn.
That sentence names the mistake precisely. It also sounds like the sort of prompt improvement that should solve it.
The misses continued.
More thinking didn’t solve this bug
Next, we enabled extended thinking, giving the model more room to reason before responding. Perhaps it would catch the missing action before sending its reply.
Before that change, our production audit found missed saves affecting 19 of 254 sales opportunities, about 7.5%. In the later audit, with thinking enabled, the rate was about 6% across 134 opportunities covering 286 client messages.
Those were small observational samples, so we couldn’t rule out a modest benefit. But the same failure kept appearing. More reasoning had not solved the problem we needed to fix.
That changed our next question. Instead of asking how to make the instruction stronger, we asked: what was different about the answers the agent failed to save?
The clue hiding in the logs
We log messages and tool calls, which let us compare what a client said, how the agent replied, and whether it actually performed the required action. We used Claude Code to analyze those records from the database.
A pattern emerged. In the thinking-enabled audit, there were no missed bedroom or budget saves. Short location answers were a recurring problem.
A number such as a bedroom count tended to trigger the save. A bare place name could get a friendly acknowledgment and nothing more.
Our working explanation was that the agent was treating some brief answers as conversational replies without consistently acting on them as data. The logs showed the behavior; they couldn’t tell us exactly what was happening inside the model.
They also exposed a weakness in our analysis tools. A keyword filter looking for words such as “budget” or “bedrooms” could miss a perfectly valid answer like “Irving.” To find these omissions, we needed to examine the conversation in context.
The first answer was the trouble spot
We built a focused evaluation around the pattern: ask a direct question, then give the agent a bare, one-word location answer. No numbers. No “I’m looking in…” to make the intent more explicit.
The timing of that answer made a striking difference.
On the client’s first substantive reply, with an empty criteria record and no previous tool calls in the conversation, the agent saved the answer in 1 out of 10 runs. The same bare answer mid-conversation, after earlier tool calls, saved in 10 out of 10.
The prompt still contained the same save instruction. The answer was still a location. But the surrounding conversation changed the result.
That suggested earlier tool calls were helping establish a pattern the model could continue. On the first reply, that pattern wasn’t available.
It also explained why ordinary spot checks could give us false confidence. Testing a short answer halfway through a conversation could pass every time while missing the vulnerable first turn.
The fix was in the order of operations
We went back to the prompt and found a tension between two instructions.
One section said to save search criteria immediately. The communication-style section described how to handle a reply: acknowledge what the client said, then ask for the next missing field.
That second instruction described a complete conversational turn with no save in it. The agent could follow it and produce a helpful-sounding reply while still skipping the required action.
We changed that sequence.
Before
Acknowledge → ask the next question.
After
Save → acknowledge → ask the next question.
The revised wording put the tool call first and explicitly covered terse answers. In simplified form:
When the client provides new search criteria, save them with the appropriate tool before replying. A bare one-word answer to the question you just asked still counts. Save it to the matching field in the same turn; don’t only acknowledge it in prose.
The obligation to save was already in the prompt. We put it directly into the steps for composing a response, alongside the reminder about the exact behavior to avoid.
Ten successful runs were a strong signal for this reproducer. They were not proof that every missed-save problem in production was gone. That required a follow-up production audit.
Then we removed changes we didn’t need
Our initial patch included several changes, including an additional example and revised tool-description wording. A successful test of the whole patch couldn’t tell us which edits mattered.
So we removed those additions and tested again—a process called ablation. Without the extra example and tool-description edit, the targeted test still saved successfully in 10 out of 10 runs.
That gave us a smaller fix to maintain. It also kept us from crediting every edit in the original patch for a result the simpler version could reproduce.
What we’ll do sooner next time
The most useful part of this investigation was the evidence trail. Without linked messages and tool calls, we would have had a convincing conversation, a missing field, and another round of prompt guesses.
Measure the action behind the reply. If an agent’s job includes saving data, the evaluation needs to check whether the save happened. A natural-sounding response is only part of a successful turn.
Test the first meaningful reply. An agent with several successful tool calls behind it may behave differently from one starting with an empty record. Include terse answers that real people actually send.
Put required actions into the response sequence. In this case, “save first, then reply” worked better than keeping the save requirement in a separate section. Check that your conversational instructions include the actions the workflow needs.
Use code where the requirement is explicit. Deterministic checks can verify a required write or flag unchanged search criteria. Interpreting free text may still need a model, but checking a known workflow requirement doesn’t always need another judgment call.
A quick test for your own agent
Start a fresh conversation with an empty record. Have the agent ask for a field, then answer with the shortest valid response: a city name, a product name, or a one-word preference. Check the tool call and the resulting stored value.
Repeat the test several times. Then try the same answer later in a conversation, after the agent has already saved other fields. Compare the results. Testing both positions helped us find a failure that a longer, more explicit answer could hide.
We don’t know how widespread this particular failure is. But converting free-form input into structured records is central to many agent workflows. If yours depends on it, this is a small evaluation worth adding.
The next time an agent says “Got it,” check what happened behind the reply. In our case, the answer that sounded right was hiding the action that never happened.