Using myWiki chat, the first thing you feel is not the bill. It is the few seconds waiting for an answer. Send one line, sit still, and “shouldn’t this come back already?” runs first. On the ledger, chat unit cost is small next to organize, but the feel was the opposite. The center of “slow” was wait time, not dollars.
One branch of that frustration: even simple turns tried to search. “응,” a one-line follow-up, continuing the last thread, and the pipeline still seemed to go find the wiki. So we added a branch. First decide whether this turn truly needs wiki ground, and only then select documents. In code it is close to a needSearch intent gate.
What decides “search needed”
Mid-July, reading the code, the question was natural. “When I send a message, do we always put the space document map into the prompt?” Answer: no. Reading the map from disk, attaching the map to the answer prompt, and putting the map into selection are three different things.
The next question was sharper. “Search-needed is a model judgment. Shouldn’t it get the map?” Instinct says you need the shelf list to know whether to look. Design deliberately did not. Stage 1 only asks: does this utterance need new wiki ground? Stage 2 asks which docs, against the map. Stage 1 materials are rule prompt, recent chat slice, this user message. What the wiki holds is not in that layer.
Like listening without the library catalog: is this a find-a-book question, or can we keep talking from the last few lines? Put the map into the gate and “search needed?” tilts into “is there a similar title on the list?” Tokens burn on small talk. So intent and list matching were split. Holes exist. Misjudge “history is enough” and you miss wiki detail. Still, then we chose over-search and wait time first.
The weight of putting the whole document map into the answer prompt
Branching reduces “no search” turns. Search turns still carried weight. When related docs hit, the answer prompt got the whole space document-map summary plus bodies. Selection had already seen the map and picked doc ids. Re-injecting the full map into chat prompt is showing the list again to the answer model. More docs in a space, more weight.
Organize had already run that lesson once. Hot-path models need unit price, frequency, volume, and injection size tightened together. Chat inject is the same version. The change was simple. When candidate docs exist, drop the full map phrase from the answer prompt, keep selected doc meta (title, doc id, one-line summary) and bodies only. Body caps: 4,000 chars per doc, 10,000 total. Before: about 14k/28k. Selection already saw the map, so re-injecting it into the answer prompt had weak reason.
Smoke matched that. Fixture with related docs: answer prompt must not contain a full “wiki document map” phrase, and injected bodies stay under cap. As a cut the token width job, that was one clean cut.
And chat still did not get much better
Written this far it can sound like a success story. Real feel is not there yet.
In plain talk: the problem was time first, simple turns still tried to search, so we branched. After that, chat still did not improve much. Answers are too short. Search was never “done right,” also because of cost. Wiki text often feels pasted. The layer that should hold context and derive something new barely got touched.
Tone prompts did move. Allow small talk, worry, comfort. On wiki hit, ban long source paste; ask for summary, synthesis, opinion. Stop pushing wiki-centric identity lines like “this chat will auto-organize into spaces and docs.” Quality is still unfinished. One prompt line does not erase the paste feel. Short answers stay many days.
“Proper” search is still homework. The search-needed branch reduces over-search. Real search quality (what should hit, when to open wiki detail on follow-ups, how deep vs cost) is another lever. More calls mean more wait and more money again. The sense built while squeezing organize cost hangs on chat search too. So some of it was left unattached. Not shameful unfinished work. A square still empty on the same bottleneck map.
Where this sits in the series
Across the build posts in this series, we cut injection that was needlessly large on the hot path. Full document-map re-inject gone, body 4k/10k caps, search-intent branch. Organize-cost lessons moved onto chat inject.
Open homework is whether chat is a conversation worth using. Short answers as default, search kept shallow for cost, reading the wiki but not writing new sentences. Tone prompts alone do not close it. When the product goes real-use, those three squares will likely come back to the front.
Numbers on the cost table and the seconds I hate are not the same axis. This post cut wait and injection width. Conversation quality is still on the homework shelf.