Why You Don’t Have to Read Every Raw File to Get an Answer

There’s a moment every second brain builder eventually runs into. You need to find something, but you have no idea where to start.

Open the raw folder and you’re staring at dozens of notes. Tell the LLM to “find anything about X in here,” and it starts opening files one by one until it burns through all available tokens — often missing the most important things along the way.

This was the problem I spent the most time on while building the /query skill. “Just read through all the raw files” isn’t a strategy — it’s a surrender. And there’s a cruel paradox: the more your wiki grows, the slower the search becomes. Unless you solve that, building a bigger knowledge base actually makes things worse.

The solution turned out to be simple: define a reading order. And the criterion for that order was trustworthiness.

Reading by Trustworthiness Means Rarely Reaching Raw

Trustworthiness, in this context, means: has this signal already been validated by someone?

Wiki summary pages are the output of an LLM digestion pass. pinned.md contains items I hand-picked myself. project-logs are records left after real work was completed. raw-cache, on the other hand, is auto-generated from monthly summaries — it carries noise. And raw files themselves are pre-digestion source material.

That difference in trustworthiness determines the search order.

The 4 Levels of Index

Level 1: wiki/ Summary Pages

These come first. Files under wiki/ have been through at least one digestion pass. If the answer is here, there’s no reason to open raw at all.

When a page has a sources: field in its frontmatter, the path back to the original file is already there — so even when you need to dig deeper, you’re not searching blind. The trail is already laid.

Level 2: pinned.md

raw/inbox/pinned.md. These are the items I personally flagged as “I’ll need this later” — undigested, but deliberately saved.

They haven’t been through digestion yet, but the signal density is high. Because I picked them manually, there’s less noise than in raw-cache, and they’re more current. Items without a check mark (✅) are still unprocessed.

Level 3: project-logs

index/project-log/<project>.md. These files are extracted from each project’s docs/log.md — just the date, type, and one-line summary headers. They’re auto-updated every time /ingest-docs runs.

This level is especially effective for project-related questions. “What did I do on project X?” or “When did I implement that feature?” — these get answered faster here than in the wiki. The date and type fields make it easy to follow a timeline.

The same applies to blog topic mining. Because these are actual work records, the density of story/lesson material is higher than in raw-cache sessions. raw-cache summarizes entire conversation flows, while project-logs only captures “things that were actually decided and done.” That difference shows up directly in the quality of the material.

Level 4: raw-cache

index/chrono/YYYY-MM.md. These are the monthly summaries /ingest-index generates from raw sessions.

They allow broad sweeps, but they carry noise. The right way to use them: grep for topics or summary keywords to pull candidate paths, then open only those files. The goal isn’t to read everything — it’s to narrow down candidates.

Note: raw/projects is not included in raw-cache. Project work records are managed separately through project-logs.

If none of these four levels surface an answer, then you go into raw/ directly. That’s the last resort.

Each Skill Uses This Order Differently

The index priority is the same across skills, but each skill asks different kinds of questions — so they don’t follow the order in exactly the same way.

/query

Search order: wiki → pinned.md → project-logs → raw-cache → raw directly.

When the question is project-context (“What was I doing on project X?”), it jumps straight to project-logs. raw-cache is often skipped entirely — project-logs answers those questions about as fast as the wiki does.

For concept or learning-related questions, the wiki gets a thorough pass first, and raw-cache is only consulted if nothing surfaces there.

/blog-mine

The goal here is topic discovery, so the priority order shifts slightly:

1. Scan pinned.md for undigested items first (score bonus: +2)
2. Collect wiki pages tagged with content_potential
3. Scan project-logs. High story/lesson density from actual work records (score bonus: +1)
4. raw-cache or recent session folders as fallback

pinned.md comes before the wiki because of freshness. The most recent raw material — things not yet digested into the wiki — tends to live in pinned. project-logs beats raw-cache because of signal density.

Topics sourced from project-logs get a (project-log) tag in the output list. Knowing where a topic came from makes it much easier to track down the original material when it’s time to write.

/ingest-docs

When syncing a project’s docs/ folder, it also creates or updates the project-logs file.

Specifically: it extracts the ## [date] type | summary header lines from raw/projects/<project>/docs/log.md and writes them to index/project-log/<project>.md, keeping the most recent 30 entries plus overall stats.

The key is that this happens automatically. Work in a project, log it in docs/log.md, run /ingest-docs once — and that record becomes searchable within the wiki. No manual copying required.

How Search Actually Flows in Practice

Most questions get resolved at the wiki or project-logs stage. If something has been written up before, raw never gets opened. Questions like “what was I doing on project X?” get caught by project-logs first.

raw-cache is primarily for things that haven’t surfaced anywhere else yet — material that came into raw recently but hasn’t been digested.

Going into raw/ directly is rare. Following the four levels in order usually produces an answer before it comes to that. Rare doesn’t mean never — there’s definitely content that only lives in raw, and when that happens, you just go dig through it. It’s just not the first move.

The four-level structure wasn’t designed upfront. It emerged from experience — noticing which sources answered which kinds of questions, and letting the pattern settle into an order. As the wiki keeps growing, I expect that order to matter more and more.