Message chunks

A message is often too big to be the unit a search answers with. A forwarded thread can carry twenty exchanges, and the one paragraph that answers a question is somewhere inside it. MailFathom therefore cuts a message's extracted text into overlapping passages — chunks — and stores each of them with the span it came from and a hash that identifies it. Every message is cut unless its folder is mapped with GenerateEmbeddings: false.

Nothing retrieves chunks yet. They exist because the passage, not the message, is what a vector is produced for, and because deriving them is free: chunking reaches no provider, opens no connection, and costs an instance with no embedding provider configured nothing but the rows. Body text and the lexical index describes the text they are cut from, and Stored email schema describes the table they are stored in.

When chunking runs

A message is cut after it has been stored and never in the transaction that stored it. Two stages stand between the two, and both can change what the cut should produce: spam classification can decide the message is not derived from at all, and the owner's rules can file it into a folder mapped differently from the one it arrived in. Passages are not undone by a message moving afterwards, so the account's synchronization run cuts as its last local step — after the classification pass and after the rule pass — and offers each message it cut for embedding. The arrival pipeline draws the whole order and states what each stage waits for.

Both paths that extract text obey it. The backfill that re-derives text for mail stored before extraction existed cuts through the same writer, and the embedding backfill sweep cuts whatever the account run's own batch budget did not reach. A message classification was holding needs neither of them: the account's next run re-evaluates it and cuts it in the same run, once a verdict admits it or its wait runs out.

A message classification files as junk is never cut on any path, and one already cut when a junk verdict arrives has its passages removed. Junk is kept out of what a deployment derives from mail holds all of it, including what a message waiting on a verdict costs.

A message that yielded no text is cut into nothing. That covers a body that carried no words and a body that arrived encrypted: both keep the search document that makes them findable on their subject and participants, and neither gains a chunk.

A message of a folder mapped with GenerateEmbeddings: false is cut into nothing either, on every path that would cut one, which is what keeps its content from reaching an embedding provider at all. Everything else about the folder is unchanged — it is mirrored, listed, searched lexically, and read like any other. Passages cut before the switch was set stay where they are; nothing removes them, and the embedding backfill stops selecting the folder rather than sweeping for the vectors it now never produces. A message of a folder no mapping names is cut into nothing for a different reason: cutting is offered to the folders configuration maps and leaves embedded, so a folder outside that list is outside the offer rather than switched off inside it. Mail stored under an alias whose mapping was later removed therefore gains no further passages, and an account mapping no folder at all gains none anywhere. What a mapping decides beyond where the folder is states the switch beside the other two and what an unmapped folder is instead.

A run that could not read a message's body at all — because the raw MIME was never stored, or because nothing could parse it — leaves the passages alone rather than removing them. A remote message is immutable, so a run that failed this time is no reason to forget what a run that succeeded already cut, which is the same rule the search document follows.

The boundary rules

Rule	Value	What it decides
Source form	The trimmed text	Which of extraction's two readings is cut
Target	1000 characters	The most a chunk may hold
Minimum	250 characters	How far a chunk must reach before a separator may end it
Overlap	200 characters	How far back into the previous chunk the next one starts
Separators	\n\n, then \n, then a space	Where a chunk is preferably ended
Rule-set version	1	The number the rules above are recorded under

The cut walks the text once. Each window reaches at most the target; inside it, the last occurrence of the strongest separator that still leaves a chunk of at least the minimum is where the chunk ends, and a window offering no qualifying separator is cut at its own end. The separator belongs to the chunk it ends, so the offsets name a contiguous span: read the extracted text from a chunk's start offset for the length of its text and you get that chunk back, character for character.

Two of those numbers are choices worth stating. The source form is the trimmed text — the reading extraction produced by removing quoted history and signatures — because that is what somebody actually wrote, and cutting the untrimmed form would fill a mailbox with passages of repeated thread history. The untrimmed reading is kept beside it and is reachable by a rule set that names it, so an over-aggressive trim is recoverable rather than permanent. The minimum exists because a blank line early in a long paragraph would otherwise end a chunk after a handful of words; it shortens no chunk, it only refuses a break that would.

Every boundary falls between text elements, so no chunk begins or ends inside a surrogate pair or a combining sequence. A single text element longer than the whole window is the one case the target cannot bound: it is taken whole, overrunning by a few characters, because refusing it would leave the walk with nowhere to go.

The cut consults no clock, no random source, and no culture, and every comparison is ordinal. The same text under the same rules produces the same passages, with the same offsets and the same hashes, on any machine and in any order.

The per-message ceiling

A message is cut only as far as Embeddings:MaxCharactersPerEmail, which is 200 000 characters unless a deployment says otherwise. That is a long report with its quoted history and far beyond what ordinary correspondence reaches; what it excludes is the log dump, the exported table, and the machine-generated transcript — none of which anybody asks a question about, and any of which would otherwise become hundreds of passages and hundreds of paid vectors. Raw MIME is bounded in megabytes, so without a ceiling here one message's cost would be whatever a sender attached.

The message is bounded rather than refused. Its opening is cut, embedded, and retrievable, which is the part the rest of a message elaborates, and the cut lands on the nearest text-element boundary at or below the ceiling like every other boundary here. What the ceiling left out is recorded on the message itself — the length its text had when the cut stopped short — so the question "could the answer have been in the part nobody embedded" is answered from a stored fact rather than guessed at from a chunk count. Two metrics carry the same thing across a deployment: how many messages the ceiling reached, and how much text it left out.

The ceiling is deliberately not one of the boundary rules above, and therefore not part of the hash. The rules decide what a passage says; the ceiling decides only how many passages there are, and a passage the ceiling leaves in place keeps exactly the digest it had. So moving the ceiling changes nothing about a message it does not reach, and leaves every stored vector as comparable as it was. What it does cost is the messages it does reach: those are re-cut, and a re-cut replaces a message's passages whole for the reason below, so an oversized message a moved ceiling reaches is embedded again from its first passage. Embedding generation states the ceiling beside the two that bound a deployment rather than a message.

What the hash covers

Each chunk carries a SHA-256 digest, written as sixty-four lowercase hexadecimal characters. It is computed over:

• the chunk's own text;
• every rule in the table above — the source form, the target, the minimum, the overlap, the separator ladder in order, and the rule-set version;
• whether the text was derived from HTML rather than read from a plain-text part.

Covering the rules as well as the text is the whole point. A digest over the text alone would report a message's chunks as unchanged after the boundaries were tuned, and would leave whatever hangs on them attached to boundaries they no longer describe. Because the rules are covered, a boundary change produces different chunks even where the words are identical, and it is a local re-cut rather than anything a provider is paid for.

Fields are length-prefixed and numbers are written big-endian, so the encoding is one-to-one and the digest depends on the values rather than on the machine that computed them.

Re-chunking an unchanged message

Re-deriving a message compares what the chunker just produced against what the message already has — the ordinals and the digests, never the stored text. Identical means nothing is written at all: no update, no delete, no insert. That is what keeps a restart, a content repair, or a backfill from re-doing work already paid for, and what keeps whatever later hangs on a passage hanging on the same row.

Anything else replaces the message's passages whole rather than reconciling them one by one, because a boundary change shifts every ordinal after the first difference and a row-by-row merge would only make that look survivable.

One thing is compared outside that decision: what the per-message ceiling left out. Text that grew past the ceiling while everything up to it stayed identical yields exactly the same passages and a different truncation, so the record is written from the current derivation rather than from whether any passage moved. An unchanged message still writes nothing — the value it would be given is the value it already has.

The rule-set version

Every chunk records the version of the rules it was cut to, in a column beside its hash. Nothing reads it to decide correctness — the hash already covers the rules, so a change to them is a changed digest whatever the column says. It exists so that a backfill can be asked how much of a mailbox is still cut to the previous rules, which is the one question a hash has no answer to.

The version is not what makes a rule change take effect, and it is not checked against anything. It is a label, and it is accurate only because changing any rule and incrementing it are one edit.

Privacy

A chunk's text is mail content and personal data by default, and it inherits the source message's classification, retention, access, export, and erasure obligations whole. Nothing about being derived makes it a lesser copy.

• Deleting a message deletes its passages. The chunk rows cascade from the stored email, so the deletion path that reaches a message reaches everything cut from it without a rule anybody has to remember.
• No chunk text reaches a log, a metric, a trace, or an error message. The ordinal, the offsets, and the length are the only things about a chunk that are safe to report; the digest identifies a passage and is treated the same way.
• The chunk copies none of the message's other coordinates. The account, the folder, the sender, the recipients, the date, and the subject are reached through the message a chunk belongs to rather than duplicated onto it, so this table never becomes a second searchable copy of who somebody corresponds with.
• Deriving a chunk sends nothing anywhere. It is a local computation over text the instance already holds.

What a lossy reading carries forward

A message that offered only HTML has text that was inferred from markup, and extraction marks that reading as lossy. Every chunk cut from such a text carries the marker forward, so a later ranking change can weigh a lossy passage differently without walking every message's extraction again. The marker is part of the hash as well: the same words read from markup and read from a plain-text part are worth different amounts, so they are not one passage under two names.

What is not here

Anything about what happens to a chunk once it exists. A vector is keyed on the chunk it was produced for — Stored vectors describes the table it lands in, Embedding generation the boundary that produces it, and Automatic embedding what decides that a newly synchronized message's chunks should be embedded at all. What is still missing beyond those is the backfill for mail stored before a profile existed, and the index built over the vector column. Ranking of any kind, and any change to what search_emails returns. Chunking attachment payloads, which extraction never opens in the first place.