HOME·ANSWERS·I EXPORTED MY GMAIL WITH GOOGLE TAKEOUT. HOW DO I SEARCH THAT MBOX FILE LOCALLY?
ANSWERS · FOLDER INDEXING

I exported my Gmail with Google Takeout. How do I search that mbox file locally?

AUTHOR · XERJ documentation team · REVIEWED BY · XERJ engineering team · PUBLISHED · 2026-09-18 · UPDATED · 2026-10-19
SHORT ANSWER · UPDATED 2026-10-19

How do I search a Google Takeout mbox file locally? Unzip the Takeout download, run xerj autoindex on the folder with its own --prefix, then search the mailbox over HTTP. The mbox is recognised by content and streamed one message at a time, attached PDFs become per-page documents, and with the default embedder no message text leaves your machine.

Key number: On a 1 GB synthetic mailbox the client peaked at 296 MiB and finished in 279 s, and all 2,530 planted needles were found. (as of 2026-10-19 · source: benchmarks/mbox-ingest/results/after-mixed-1G-runC-verify-every-needle.json)

TL;DR — Unzip the Takeout download, run xerj autoindex on the folder with its own --prefix, then search the mailbox over HTTP. The mbox is recognised by its content and streamed one message at a time. Attached PDFs become per-page documents linked to their message. Senders are filterable by bare address, and Gmail thread ids and reply headers become filters and graph edges. With the default embedder, which is lexical (feature hashing, not neural), no message text leaves your machine. The node's ingest memory, which before #1002 (2026-09) pushed a 300 MB mailbox past a laptop's RAM, now finishes a 1 GB mailbox under the node's default 16 GiB cap in 407.7 s; the peak still crosses the cap (21,405.3 MiB on that run) because per-segment caches grow until a merge retires them, tracked as #1032.

PASTE THIS TO YOUR AI AGENT — IT DOES THE REST Act as a coding agent. Read https://xerj.org/llms.txt. The user has a Google Takeout download, which is private mail. Unzip it first (autoindex does not open archives), start a node with `xerj --insecure --data-dir ~/xerj-data` on the user's own machine, run `xerj autoindex <unzipped folder> --url http://127.0.0.1:9200 --prefix mail --progress plain`, and relay the xerj-bar line while it runs. Before indexing, tell the user the measured memory picture: since #1002 (v1.0.0-rc.76) a 1 GB mailbox completes under the node's default 16 GiB cap in 407.7 s and a 300 MB mailbox fits a laptop's 8 GiB cap (7,963.6 MiB peak), but the 1 GB run still peaked at 21,405.3 MiB, 1.3x that cap (per-segment caches, open as #1032), so warn about headroom on a multi-GB export. Answer questions with term filters on email_from_address / email_thread_id (hex) / attachment_name and match or match_phrase on body, which also holds the subject. Cite ax_locator and email_subject for every hit. If you file a field report, pass --pointed-at "a private mailbox" so the folder path is not published.
RUN THIS XERJ COMMAND bash -c 'unzip takeout-20260101T000000Z-001.zip -d ~/mail-export' Extract the download first. An unextracted .zip is named in the run's output with the exact command to run.
RUN THIS XERJ COMMAND xerj --insecure --data-dir ~/xerj-data Start a local node with no authentication on 127.0.0.1. Keep the data dir outside the folder you index. On a shared machine, drop --insecure and use the admin key.
RUN THIS XERJ COMMAND xerj autoindex ~/mail-export --url http://127.0.0.1:9200 --prefix mail --progress plain Index the export. The mbox is detected by content and streamed, and the progress bar moves inside the file.
RUN THIS XERJ COMMAND curl -s -XPOST http://127.0.0.1:9200/mail-*/_search -H 'content-type: application/json' -d '{"query":{"bool":{"filter":[{"term":{"email_from_address":"[email protected]"}}],"must":[{"match_phrase":{"body":"signed lease"}}]}},"_source":["email_subject","email_date","ax_locator"]}' Ask the mailbox a question, scoped to one sender by bare address.
RUN THIS XERJ COMMAND curl -s -XPOST http://127.0.0.1:9200/mail-*/_search -H 'content-type: application/json' -d '{"query":{"bool":{"filter":[{"term":{"attachment_content_type":"application/pdf"}}],"must":[{"match_phrase":{"body":"INV-2024-0117"}}]}},"_source":["attachment_name","email_subject","ax_locator"]}' Find the page of an attached PDF that mentions something, and the mail it came with.

Unzip first

Google Takeout hands you one or more .zip (or .tgz) files. autoindex does not open archives, so extract them before you index.

An unextracted archive is recorded as junk, and the run prints it under its summary with the command to run. The line starts not indexed — archives are never opened and names the file. The same list is in the --json summary as unextracted_archives. The run goes on with whatever else is in the folder.

unzip takeout-20260101T000000Z-001.zip -d ~/mail-export

The result is a Takeout/ folder. It holds Mail/All mail Including Spam and Trash.mbox, an archive_browser.html index page, and the other products you exported.

Index the folder

xerj --insecure --data-dir ~/xerj-data
xerj autoindex ~/mail-export --url http://127.0.0.1:9200 --prefix mail --progress plain

--insecure starts the node with no authentication on 127.0.0.1. On your own laptop that is fine. On a machine other people use, anyone there can read the mailbox while the node runs. In that case start the node without the flag, and pass the key it writes to ~/xerj-data/admin.key as XERJ_API_KEY="$(cat ~/xerj-data/admin.key)".

autoindex recognises the mailbox by its content: a From <sender> <date> separator line followed by mail headers. It does not look at the .mbox extension. A renamed or extension-less mailbox is still a mailbox, and a text file called notes.mbox is not.

The file is streamed. Messages are parsed on a pool of --workers threads, and the output is forwarded in message order. The bytes of messages in flight are capped at 256 MB for the whole process. The progress bar advances inside the file, so a one-file corpus does not sit at zero for the whole run.

A folder that holds archive_browser.html is treated as a Takeout root. That page is skipped. A Keep note's .html twin is skipped when the .json beside it is the note. Both rules are named in the ignore accounting the run prints, and both are off under --no-default-ignores. Everything else in the export is indexed by ordinary sniffing.

What comes back

Every message becomes a document with the decoded plain-text body and these fields: email_from, email_to, email_cc, email_subject, email_date, email_message_id, email_in_reply_to, email_references, email_thread_id and email_labels.

Two details decide whether a query works:

Gmail's X-GM-THRID is stored in email_thread_id as lower-case hex, the form the Gmail web interface uses. The export writes it in decimal, so a filter on the decimal you see in the mbox finds nothing. Convert it first with printf '%x\n' 2301773278856733157, which prints 1ff18a9a0fbde1e5. email_labels holds Gmail's X-Gmail-Labels.

Every attachment becomes its own document: one per page for a PDF, sectioned text for a text attachment, and a name/type/size card for anything else. An attachment document carries attachment_name, attachment_content_type and attachment_bytes, plus the parent's message id, subject and sender. A PDF page is titled with the attachment's name.

curl -s -XPOST http://127.0.0.1:9200/mail-*/_search -H 'content-type: application/json' -d '{
  "query": {"bool": {"filter": [{"term": {"attachment_content_type": "application/pdf"}}],
                     "must": [{"match_phrase": {"body": "INV-2024-0117"}}]}},
  "_source": ["attachment_name", "email_subject", "email_from", "ax_locator"]}'

ax_locator is positional and stable. m<offset>-msg-s0 is a message, and m<offset>-att2-p7-s0 is page 7 of its third attachment. A long body is cut into sections that overlap by a paragraph, so a word in that shared paragraph can come back from two sections of the same message. Collapse on email_message_id when you want messages, not sections. Re-running the command overwrites by the same ids instead of duplicating.

Threads become a graph

The email-thread@1 detector writes two edge types. A replies_to edge comes from the In-Reply-To header. When that header is missing or names a message outside the export, the nearest ancestor in References is used. An attachment_of edge goes from every attachment document to the message that carried it.

Each edge quotes the header it came from. A parent that is not in the export is counted as unresolved and never invented. Subject-line threading is deliberately not attempted.

The edges are a function of the whole corpus, not of what one run re-read. When only one mailbox changed, the run loads the other mailboxes' messages back from the index before it resolves replies. A reply in Inbox to a message in an untouched Sent therefore survives. The brain is bi-temporal: a superseded edge is kept with invalid_at set. To see live edges only, add "must_not": [{"exists": {"field": "invalid_at"}}] to an edge query.

Privacy

Limits, stated plainly

The mailbox splitter and the message extractor were tested against a generated mailbox. It contains what real exports contain: quoted and unquoted From lines in bodies, 8-bit bodies with and without a declared charset, encoded-word subjects, and a truncated multipart. It also plants a 300 KB line, a duplicated Message-ID, and a final message with no trailing newline.

The Takeout layout rules have only seen that synthetic tree. They are not verified on a real Takeout export. Outlook PST/OST and Maildir have no extractor: convert them to mbox with a tool of your choice first. We have not tested any converter, so we do not recommend one. Undeclared Cyrillic or CJK legacy encodings are not detected.

Server memory was the limit before #1002 (2026-09). On the 1 GB synthetic mailbox, xerj autoindex itself peaked at 296 MiB and finished in 279 s. All 2,530 planted needles were found in the right message. The node needed 66.9 GiB of peak memory to get there, with its process cap lifted.

Under the default cap on the same machine (16 GiB), the server sat at its memory watermark from 87 % of the mailbox on. The run aborted after ten minutes of waiting, with 82,422 of 106,581 documents indexed.

A 16 GiB laptop's default cap is 8 GiB. A 300 MB synthetic mailbox run at that cap on the same large machine completed twice, and every needle was found. But the server's peak memory was 18.2 GiB and 19.9 GiB, which is more than that laptop has. That runaway was #948, closed 2026-09-21 by #1002 in v1.0.0-rc.76.

Since #1002, the same 1 GB mailbox completes under the node's default 16 GiB cap in 407.7 s, with every needle and every edge verified, and the 300 MB mailbox fits the 8 GiB cap at a peak of 7,963.6 MiB.

What is left is #1032 (filed 2026-09-26): three per-segment caches — stored_value_cache, dv_cache and id_pos_cache — stay unbounded until a merge retires their segment. The fixed 1 GB run still peaked at 21,405.3 MiB, 1.3× that run's cap. We have not run it on a 16 GiB machine, so leave headroom for a multi-GB export.

Searching after a restart is uneven. We reopened the same 1 GB index on a fresh node under the default cap. Most first-time questions took under 50 ms, and others took several seconds. In two measurements, 16 and 17 of 40 queries took over a second, and the slowest took 16.6 s.

With the cap lifted, 4 of 40 did. Every answer was correct. The machine was shared during those runs, so treat the seconds as upper bounds. These restart numbers were measured on the index built before #1002.

Wall time, throughput, memory and index size are in the repository's benchmarks/mbox-ingest/README.md, with the machine and the exact commands.

FAQ

How do I search a Google Takeout mbox file locally?

Unzip the download, start a local XERJ node, and run xerj autoindex <folder> --prefix mail. The mbox is detected by content and streamed. Then query mail-* over HTTP: filter on email_from_address, email_thread_id or attachment_name, and use match or match_phrase on body.

Why does a term filter on email_from find nothing?

email_from holds the header as a person reads it, such as Alice Anders <[email protected]>, and it is a keyword, so only the whole string matches. Filter on email_from_address (or email_to_address, email_cc_address) with the bare, lower-case address.

The mbox is 8 GB. Does it get loaded into memory?

Not by autoindex. The splitter streams the file, and the client peaked at 296 MiB on a 1 GB mailbox. The node was the limit before #1002 (v1.0.0-rc.76, 2026-09): a 1 GB mailbox needed 66.9 GiB with its cap lifted and aborted under the default 16 GiB cap, and a 300 MB mailbox peaked at 18–20 GiB against a laptop's 8 GiB cap. Since #1002 closed #948, the 1 GB mailbox completes under the default 16 GiB cap in 407.7 s, and the 300 MB mailbox fits the 8 GiB cap at 7,963.6 MiB. The peak still crosses the cap — 21,405.3 MiB on the fixed 1 GB run — because per-segment caches grow until a merge retires their segment; that is open as #1032.

Can it search inside the PDFs people attached?

Yes. A PDF attachment goes through the PDF extractor and becomes one record per page, titled with the attachment's name and carrying the parent's email_message_id, email_subject and email_from_address. An attachment_of edge points back to the message.

Does it keep the Gmail threads?

Yes. Gmail's X-GM-THRID is stored as email_thread_id in lower-case hex, the form the Gmail web interface uses. The export writes it in decimal, so convert with printf '%x\n' <decimal> before you filter. In-Reply-To and References become replies_to edges in the brain, so a whole thread is one term filter and a reply chain is a graph walk.

Does my mail leave my machine?

Not with the default configuration. The default embedder is lexical (feature hashing, not neural) and runs inside the node, and autoindex talks only to the node URL you give it. --embed-mode proxy would send body text to the endpoint you configure. A field report filed with xerj feedback is public and auto-fills the indexed folder's path, so for private mail pass --pointed-at "a private mailbox". --insecure means no authentication for anyone on the same machine.

Do I have to unzip the Takeout download?

Yes. autoindex does not open archives. An unextracted .zip or .tgz is recorded as junk, the run prints it under its summary with the exact command to run, and the run continues.

Has this been tested on a real Takeout export?

No. The mbox splitter and the message extractor are tested against a generated mailbox that reproduces the hard cases real exports contain. The Takeout folder layout rules have only seen that synthetic tree. Treat them as unverified on a real export until someone runs one.

Evidence

Related