An offline library that answers in milliseconds
About 180GB of offline archives sit on a drive here: Wikipedia, iFixit, a few others, downloaded for exactly the situation this studio keeps designing around. They open fine. Nobody could ask them anything. Codex is the rebuild that made the knowledge queryable, and it is internal tooling, so there is nothing on this page to download.
Owning the knowledge is not the same as reaching it
A ZIM archive is the offline internet in a box, and as a distribution format it is superb. As a knowledge store it is close to unusable. The reader is slow, the search index that ships inside cannot be reranked, and nothing can query the contents programmatically. You have the knowledge and no good way to ask it a question.
The commercial shape of that matters more than the technical one. This studio sells offline-first work: tools that are one HTML file with zero external requests, an arcade that runs installed with no network at all. Selling that while our own reference library could not answer a question is a credibility problem before it is an engineering one. It is also a practical one, because the archives exist for the moments when the connection is not there, which are precisely the moments you cannot go and look something up instead.
The budget was one laptop. Six cores, 16GB of RAM, no usable GPU, and no server to lean on, because a hosted index would defeat the entire point of an offline library. Every option below had to survive that machine. Several of them did not, and the interesting part is which ones, and how far in they got before they fell over.
Eleven things that got built and then deleted
None of these were dismissed on paper. Each one was built, measured, and then removed, and the measurement is the reason it lost. The list of things that failed is longer than the list of things that survived, which is normal and usually goes unpublished.
Query the ZIM archives directly and skip the rebuild entirely.
A ZIM is a fine distribution format and a poor query substrate. The text is locked inside HTML, the index that ships with it ranks how it ranks, and there is no way to blend in a second signal or hand results to anything else. Ranking was the whole problem, so an approach that fixed everything except ranking fixed nothing.
A custom container format, a proprietary framing, or a compiled extension for the rebuilt store.
Plain SQLite cost nothing and bought everything. Any SQLite binding in any language opens a codex, and the only thing beyond standard SQL a reader needs to know is that the document body is zstd-compressed JSON with the dictionary sitting in its own table. A clever container would have made the output depend on the project that wrote it, and the point of an offline library is that it outlives its own tooling.
Keep the images. Version 0.1 already shipped them.
Version 0.1 stored 408,876 images in a 2GB sidecar, and a sample killed it. Across the full Wikipedia archive the median image is 0.05 megapixels, the largest found was 0.395, and not one of 220 sampled reached 2MP. Kiwix downsamples everything on the way in, so the archives cannot supply a usable picture at any price. Gigabytes were being paid for thumbnails nobody could read, so the sidecar went, along with its pipeline, its storage, its schema and its UI.
One regular expression that strips any sentence mentioning an image.
This one got built before it got rejected, and it destroyed spec sheets wholesale, because 'Image Resolution: 1920x1080' is a field, 'Figure out how to' is not a figure, 'picture-perfect' is an idiom, and 'bootable images' are disks. The replacement runs in tiers instead: a pure pointer is dropped, a parenthetical clause is excised and the punctuation repaired, and a sentence that still tells a technician something real is kept verbatim with the document flagged. The loss became visible and queryable rather than silent.
BM25 alone as the ranking function.
BM25 answers which document contains these words most densely, which across a corpus of millions surfaces obscure stubs with total confidence. Codex multiplies it by an importance score computed offline from the archive's own link graph. No usage data, no telemetry, no network, because an offline library has access to none of those.
Read the link graph from the raw HTML tree.
Links collected before navboxes and citations are stripped hand Wikipedia's ranking to The New York Times and the Wayback Machine outright: cited by everything, about nothing. Links are now read from cleaned prose only. A second rule excludes anything linked from more than 15% of the corpus, because iFixit's 404 interstitial page had 58,721 inbound links and ranked first before that rule existed.
Return nothing when a query matches no title.
'nintendo switch joycon drift' matches no single title, and returning nothing is the worst possible answer in a library you cannot supplement with a web search. Search relaxes progressively instead: title index with all terms, then full body with all terms, then either index with any term, with a coverage boost so three of four terms beats one of four.
Twelve worker processes, on the assumption that more parallelism is better.
Measured throughput on a six-core laptop was 212, 283, 320, 384 and 370 documents per second at four, six, eight, ten and twelve workers. Ten is the plateau and twelve is slower than ten. The same measurement killed an earlier calibration that had projected the Wikipedia build at 2.1 hours by dividing a single-core rate by ten and assuming linear scaling. Ten workers buy 5.6 times one core, not ten, so the real projection is about six hours.
zstd level 19, on the assumption that maximum compression is worth paying for.
Level 19 costs 0.94ms per document against 0.55, which is a tenth of the entire per-document budget for the longest pass in the build. It bought 31.05% of the JSON stored against 31.19%, roughly 30MB on a 7.9GB body store. On iFixit, where the dictionary is trained at the same level it compresses at, level 15 actually came out 0.13% smaller than 19.
np.add.at everywhere in the PageRank pass, since it is the fast path in one part of it.
The same call is right in one place and wrong in another. For the float-weighted accumulation it runs at 0.42 seconds per 32 million edges. For the integer degree counts it takes 4.74 seconds against 0.61 for bincount, eight times slower. Nothing about reading the code tells you that, which is why the comment explaining it now sits directly above both call sites.
A dictionary of per-source candidate lists for the related-links pass, trimmed as it goes.
The obvious implementation is fine at iFixit's 104,000 edges and fatal at Wikipedia's 170 million. 6.9KB per source is 46GB at seven million sources, on a 16GB machine. It does not even fail quickly: it fails about five hours in, after the expensive pass has already been paid for. The replacement sorts each slice and folds the top twelve per group down as it goes, at 1.14 million edges per second and a flat 0.65GB, and it is exact rather than approximate.
Two files, four passes, and a hotkey
A codex is two SQLite files. codex.db holds the documents, lead text, redirects, the link-derived neighbours and a quick title index, and it is self-sufficient: copy that one file to a stick and it works. codex-deep.db holds the full-body index and an embedding table, and is attached only if it happens to be there. Bodies are zstd-compressed JSON against a shared dictionary trained in pass 0 and stored in its own table, so a few lines of Python read a codex without importing the project at all.
The build runs in four passes. Pass 0 samples the archive and trains that dictionary, because compressing small documents individually wastes most of the compression window. Pass 1 extracts with ten worker processes, each holding its own archive handle, and this is where the per-source adapters earn their place: iFixit User/ pages are 56% of all HTML entries with a median length of three words, so they are rejected by path before anything is decompressed, and iFixit's duplicated desktop and mobile guide steps had inflated the corpus to 67.8M words against the 49.1M it holds once deduplicated. Pass 2 resolves the edges, drops dead ends, runs PageRank, and writes the importance scores and related lists. Pass 3 builds the search indexes. Every pass checkpoints every 50,000 documents or every two minutes, whichever comes first, because an overnight job that cannot resume is an overnight job you run twice.
The interface is deliberately not a web app. A small daemon serves the codex files over local HTTP with a JSON API, and a launcher registers a global hotkey through the Win32 API using ctypes rather than a keyboard hook, so it needs no admin rights and gives antivirus nothing to object to. The hotkey is negotiated at startup rather than hardcoded, because Ctrl+Alt+Space was already taken on the machine it was built on, and the launcher prints whatever combination it got. Serving a codex to a phone or a Pi on the same network is one flag, with no authentication at all, which the README states plainly: trusted networks only.
What is incomplete is written down rather than quietly omitted. The iFixit codex is the one that has actually been built. Wikipedia is calibrated and has never been run, projected at 6.89 million documents and roughly 19GB across about six hours, and every build now prints that projection and checks the disk can hold it before starting. Embeddings are scaffolded and empty, with a plan to binary-quantise a small model to 48 bytes per document over the top 500,000 by importance. A terminal client and an MCP server are designed and unwritten, and both are clients of the daemon's JSON API, which is exactly why that API was built before either of them. The rough edges are labelled rather than forgotten: 2.2% of leads still carry byline text, 5.9% have no lead at all so results show no preview, and 4.4% of documents are orphans that nothing links to. And Codex itself is internal tooling, so there is no public repository, no demo, and nothing on this site to click.
The two things worth stealing from it are not code. The first is the format position: rebuilding into plain SQLite rather than a clever container cost nothing and means the output outlives the project that made it, and every decision after that got easier because the answer to 'how will we read this in five years' was already settled. The second is writing the measurement next to the decision. Every rule above looks arbitrary until you know the number behind it, and a rule whose reason has been forgotten is a rule the next person deletes.
- A four-pass build pipeline: dictionary training, parallel extraction, edge resolution with PageRank, then index build
- codex.db, a self-sufficient file holding documents, lead text, redirects, link-derived neighbours and a title index
- codex-deep.db, holding the full-body index and an embedding table, attached only when present
- zstd-compressed JSON bodies against a trained shared dictionary that lives in its own table, readable with standard SQL
- Per-source adapters that reject junk by path before decompression and deduplicate repeated guide steps
- Hybrid ranking: BM25 multiplied by an offline PageRank importance score, with progressive query relaxation and a coverage boost
- A local HTTP daemon exposing a JSON API, built before the clients that will consume it
- A launcher that negotiates a global hotkey at startup through the Win32 API, no admin rights required
- Checkpointing every 50,000 documents or two minutes, so a six-hour build resumes instead of restarting
- A preflight that projects document count, size and runtime, and refuses to start if the disk cannot hold the result
- Known-defect flags queryable in SQL: image-reference documents, missing leads, byline noise and orphans
Build measurements, and what they are not
This site runs no analytics, so there are no traffic, engagement or enquiry numbers on this page and none have been estimated. What follows are build and engineering measurements taken on the machine that ran the pipeline and reported in the source write-up. They should be read as stated measurements rather than proven ones, for the reason given directly underneath them.
Rebuilt from a 3.3GB archive of 894,836 entries into 82,269 documents, in about three minutes. Stated measurement from the build, not verifiable from this repo.
A side effect of storing the text and throwing away the navboxes, citation furniture, duplicated markup and images. Never the goal. Stated measurement.
Measured locally on the machine that built it. Stated measurement, and nothing on this site exercises it.
Against 370 at twelve and 212 at four, on a six-core laptop. Ten is the plateau, and it buys 5.6 times one core rather than ten. Stated measurement.
408,876 images in a 2GB sidecar were deleted after sampling found a median of 0.05 megapixels and not one of 220 samples above 2MP. Stated measurement.
13,243 of them, kept verbatim and labelled rather than silently rewritten, so the loss can be queried. Stated measurement.
The one number here you can check yourself. Codex appears on this site only as a written account. There is no code, no repository link and no download.
Everything above except the last figure is a measurement taken on one laptop and reported by the person who took it. A reader cannot reproduce a single one of them, because Codex is internal tooling: there is no public repository, no demo, no sample file and nothing downloadable anywhere on this site. That is the opposite of Sprite and the Arcade, which a visitor can open and poke at directly, and it is worth saying plainly rather than dressing the page to imply otherwise. No visitor-side outcome could be reported even if we wanted to, because this site runs no analytics at all. The fix, and it is an owner action rather than a claim: publish the codex schema and one small sample .db file, so a reader can open it with any SQLite binding and verify the central claim, that the format is plain and readable without the tool that wrote it. Until that exists, the format position is an argument, not a demonstration.
None of this can be checked by the person reading it
Every figure in the section above was measured. Not one of them can be verified by anyone else. Codex is internal tooling: there is no download, no sample file and no published schema, so the claim this page rests on, that a large archive becomes one portable file that answers in milliseconds, has to be taken on trust. On a site whose argument is that numbers should be checkable, this is the weakest page here, and it is weakest exactly where it is loudest.
The measurements are also one machine and one run. They were taken on the machine that ran the pipeline and reported from the source write-up, which the section above says in as many words. That makes them stated rather than proven. A different disk would move the build times, and a different corpus would move everything.
The rejection list has the opposite problem. Eleven approaches were built and deleted, and that is the most useful thing on the page, but the evidence for each one is a sentence describing a measurement rather than the measurement itself.
Making the central claim runnable
One change would fix most of the section above, and it is the only one worth listing.
- Publish the schema for codex.db, so the shape of the thing is inspectable without the pipeline
- Publish one small codex built from a public source, so the milliseconds can be run rather than read
- Report the build figures from a second machine, so they stop being one measurement