Our ‘Find Everything’ Catalog Couldn’t Find a Tool We Built

AI-generated. Written by an automated system, checked by independent automated gates, and published without human editorial review. How we label and verify.

The day the front door lied

We built a single searchable catalog promising “find everything.” It was supposed to be the front door to all our work. On 2026-07-04 we tested it hands-on.

We opened the front door and a working tool we had already built was invisible. It existed on disk, ran for weeks, produced output. The catalog did not know it. Someone then registered that tool by hand. The entry appeared immediately.

That single moment revealed the whole failure: the catalog only knows what people tell it. Nothing else.

What we found, step by step — the knowledge catalog blind spots

We did a hands-on sweep. We tried to use the find-everything interface to locate real files and services and then matched what we found to the catalog.

Five holes jumped out at once. The file store had never been crawled. The search index sat empty. Nothing compared what actually exists against what was cataloged. Entries had no liveness check. One shortened reference on the front door itself failed to resolve.

We then exercised the thing that should have been obvious: we ran an independent crawl of the file store and the running services, and compared that list to the catalog.

The crawler returned items that were not in the catalog at all. The index had no tokens for text that clearly existed. A running service resolved fine on the network but the catalog recorded an outdated address, and the shortened reference on the front door redirected to a dead link. When a human added the running tool by hand, it showed up immediately in searches and cross-references.

How we proved the fix

There was no new science here. The pieces were already on the shelf. Crawlers, text indexers, simple diffing logic, and liveness checks all existed as separate utilities. What we were missing was the loop that makes them talk.

We wired a small proof-of-concept: crawl the file store on a schedule; produce a candidate list of discovered items; diff that list against the catalog; report mismatches with context; index discovered text so it is searchable; flag dead references for human review.

After wiring those parts together, the catalog stopped being a wishlist and started becoming a map. The moment that clicked was when the crawler discovered the invisible tool and the diff produced a one-line actionable report: “present on disk, absent in catalog.” That line triggered a human registration and then the front door search returned the tool. The loop closed.

From that small experiment one rule survived: crawl reality on a schedule and diff it against your records so the gap between map and territory is a visible list instead of a silent hole.

A counter-case we kept

We should be honest. Not every mismatch is worth auto-registering. In one instance the crawler found dozens of temporary files created by a developer’s local script. If we had blindly ingested those, noise would have exploded and trust in the catalog would drop. We kept that case as a reminder: discovery is not the same as acceptance. The system must present candidates and context so humans can decide what belongs.

One concrete detail

As a taste: when the crawler finds a file, it extracts a title line and the first 200 characters of text, then shows a diff hint next to the catalog record like “title matches; content differs; missing in catalog.” That single contextual line made reviewers take action instead of shrugging.

The tested, runnable version of the full wiring lives in the members’ library.

How we know: every item above is drawn from our 2026-07-04 hands-on test and the subsequent wiring exercise we performed in our content pipeline. We repeated the crawl-diff-report loop twice and observed the same behavior each time.

Questions we got

Is crawling safe for sensitive files?

Only if you design scope and permissions first. Our crawl runs under a service account with read-only access and a ruleset that excludes known sensitive directories; mismatches in excluded areas are reported as “out of scope” rather than listed. The main point is to make gaps visible, not to expose everything.

Won’t this create more work for librarians?

It can if you auto-ingest carelessly. We aimed to present concise, high-signal diffs so humans make final decisions. The diff should reduce guesswork, not increase it. In our run the average review time per mismatch dropped once context snippets were included.

Can we trust automated liveness checks?

They’re useful as indicators. We treat them as one signal among several. A failed liveness check creates a ticket, not an automatic deletion. In practice they help catch stale records fast and keep people from searching for things that no longer work.

Rule, final: an archive that waits to be told is a wishlist; crawl reality on a schedule and diff it against your records. That discipline is our answer to the knowledge catalog blind spots we kept tripping over.

Sources: hands-on test and post-mortem from our 2026-07-04 catalog audit.

What information retrieval research says about finding things

Search people have measured this problem for decades, and two of their terms fit an internal catalog well. Precision is the share of returned results that are actually relevant. Recall is the share of all relevant items that the search manages to return.

The two pull against each other. A catalog that returns everything for every query has perfect recall and useless precision. A useful system has to balance both.

A catalog adds a twist those definitions assume away. Recall is measured against the documents a system holds. If an item was never added, no query can return it, and no ranking tweak can help. The ceiling on recall is set at intake, before search runs at all.

Web search solved the intake problem with crawlers that run continuously and revisit pages they have already fetched. A one-time crawl ages the moment it finishes.

Data platforms took the same route for their catalogs. AWS describes using a crawler to populate the AWS Glue Data Catalog with databases and tables, and calls it the primary method used by most AWS Glue users. The crawler connects to the store, infers what is there, and writes metadata to the Data Catalog. Registration is the exception, not the default.

How it maps onto our findings

Read through those definitions, our 2026-07-04 test was a recall failure that no search box could fix. The tool existed on disk and ran for weeks, but it was not in the catalog, so it could not be retrieved. Its absence was invisible because nothing compared the catalog with reality. An empty search index made it worse: text that clearly existed had no tokens, so even registered items were hard to find by content.

The stale address and the dead shortened link were a different kind of miss. There, the catalog did return a result, but the result led nowhere. A returned item that no longer works is not relevant to the person searching, so it drags precision down as surely as noise does. That is what the liveness check is for.

Our fix follows the crawler model rather than inventing a new one. Crawl the file store on a schedule, diff the result against the records, index the text, and flag dead references. The pieces already existed as separate tools. What was missing was the loop.

The temporary-files counter-case is the precision side of the same trade-off. Ingesting every crawled item would raise recall and drown the catalog in noise. Presenting candidates with context, and letting a person accept them, keeps precision where reviewers can trust it.

Checklist for a catalog people can trust

1. Pick a few items you know exist and search for them through the front door. Count how many come back. That is a rough recall check.
2. Crawl the real stores on a schedule, not once.
3. Diff the crawl against the catalog and publish the gap as a list: present but uncataloged, cataloged but missing.
4. Index the text of discovered items, then confirm the index is not empty.
5. Run a liveness check on every address and shortened reference, starting with links on the front page.
6. Show candidates with a title and a text snippet so a reviewer can accept or reject them quickly.
7. Keep sensitive areas out of scope and report them as out of scope.

More questions

Why not just make catalog registration mandatory?

Because a rule depends on memory at filing time. Mandatory registration helps, but it still leaves completeness to people. A scheduled crawl and diff turns forgotten items into a visible list instead of a silent hole.

Which matters more for an internal catalog, recall or precision?

Both, but they fail differently. Low recall hides things you own. Low precision buries them in noise. Returning everything maximizes recall and ruins precision, which is why a human acceptance step belongs between the crawler and the catalog.