mailbox-smith: Deterministic Parsing Before AI

Date Created: 2026-09-08
Last Updated: 2026-09-08

1. Introduction

mailbox-smith is an idea for handling email data before it reaches analytical systems such as BigQuery or RAG pipelines.

The basic data flow is:

MBOX
  ↓
bronze
  ↓
mailbox-smith
  ↓
JSONL
  ↓
silver
  ↓
BigQuery / RAG / downstream analysis

The original MBOX file acts as the bronze layer: the source mailbox data, close to what the mail client stores locally.

JSONL acts as the silver layer: normalized, structured data suitable for querying, indexing, and downstream processing.

The main concern is not merely converting MBOX to JSONL. It is preserving a path through which a human can inspect and verify how the silver representation was produced.

A useful analogy is a highly autonomous car that still has a steering wheel.

The automated system may be capable of doing most of the work, but the human-operable interface remains valuable because it makes the system inspectable and understandable.

For mailbox-smith, that “steering wheel” is direct access to the mailbox-level source data.

This matters particularly for RAG systems. A RAG pipeline operating on derived JSONL cannot necessarily determine whether the original email was parsed correctly, whether MIME decoding was accurate, whether messages were omitted, or whether a field was interpreted incorrectly.

Fortunately, an MBOX file is often small enough that human investigation remains practical.

The design principle is therefore:

Automate the pipeline, but preserve a human-drivable path back to the source.

2. Data Schema

A. Data Layers

bronze: Original MBOX mailbox data. This remains the source representation against which extracted data can be verified.
silver: Normalized JSONL records generated deterministically from the mailbox.
downstream: Systems such as BigQuery or RAG that consume the silver data for search, analysis, aggregation, or retrieval.

For structured emails, useful silver fields may include both the extracted value and information about how it was obtained.

For example:

raw_value
parsed_value
parser_name
parser_version
source_message_id

Keeping this information allows a downstream value to be traced back to the corresponding source email and parsing rule.

B. Design Policy

Deterministic first: If an email has a known structure, use deterministic parsing rather than AI.
Interpret near the source: Extract and normalize meaning during the bronze-to-silver transformation instead of asking downstream systems to infer it later.
Human inspectability: Preserve the ability to inspect the original MBOX message whenever an extracted value looks suspicious.
Traceability: Silver records should retain enough provenance to identify their originating mailbox message.
AI as fallback: Use AI only where structure, token patterns, and deterministic contextual rules are insufficient.

A concise version of the policy is:

Use deterministic parsing whenever structure exists; use AI only where ambiguity begins.

This also reduces the risk of strange downstream interpretations.

Rather than passing an entire email body to BigQuery and later trying to infer fields such as user IDs, dates, amounts, or event types, mailbox-smith can extract those values explicitly before loading them into the warehouse.

In this sense, the parser layer behaves as a schema firewall.

BigQuery then stores and queries already-interpreted fields instead of becoming part of the interpretation process.

3. Parser Types

Parser Type	Method	Suitable Use
Type A: Structural Parser	Parse HTML structure with tools such as BeautifulSoup	Stable HTML templates, tables, labelled fields, known DOM structures
Type B: Token / Keyword Picker	Extract recognizable values using regex or fixed token patterns	Fields whose syntax is stable even when layout varies
Type C: Contextual Rule Parser	Find a label or keyword and deterministically parse nearby content	Semi-structured messages where field position varies but local context remains stable
AI-assisted Parser	LLM/RAG-based interpretation	Irregular free-form text where deterministic rules are insufficient

Type A: Structural Parser

Many automatically generated emails contain stable HTML structures.

For example:

<table>
  <tr>
    <th>User ID</th>
    <td>id_123456</td>
  </tr>
</table>

A structural parser can use BeautifulSoup to identify the relevant element and extract the value directly.

This approach is especially suitable for:

form-generated emails
order confirmations
monitoring alerts
system notifications
reports
transactional emails

When the HTML structure is stable, there is little benefit in asking an AI system to interpret it.

Type B: Token / Keyword Picker

Sometimes the layout varies, but the value itself follows a stable pattern.

For example, a user ID may always have the form:

\bid_\d{6}\b

This pattern can extract values such as:

id_123456
id_654321

The surrounding HTML or text layout can change without affecting the parser.

This type of parser is useful for identifiers such as:

user IDs
order numbers
ticket numbers
reference codes
fixed-format timestamps
other well-defined tokens

Type C: Contextual Rule Parser

A third deterministic strategy sits between strict structural parsing and AI.

For example:

User ID: id_123456
Amount: 12000
Date: 2026-09-08

The parser does not need to know the exact HTML structure. Instead, it can look for a label such as User ID: and parse the following token.

Conceptually:

"User ID:" → next matching ID token
"Amount:"  → next numeric/currency value
"Date:"    → next parseable timestamp

This remains deterministic and explainable while tolerating more layout variation than a DOM-specific parser.

The preferred parsing progression is therefore:

structure
    ↓
token pattern
    ↓
contextual rule
    ↓
AI fallback

4. Maintenance Notes

Extensibility: Additional deterministic parser types can be added as recurring email formats are discovered. Parser-specific modules can remain independent of Thunderbird or other mail user agents (MUAs).
Generation: JSONL should be generated programmatically from MBOX so that the bronze-to-silver transformation is reproducible.
Interpretation: JSONL is a derived representation, not the ultimate source of truth. When an anomaly appears downstream, investigation should be able to return to the corresponding MBOX message.
Verification: Useful mailbox-level tools may include message counts, header summaries, MIME inspection, duplicate detection, extraction diagnostics, and bronze-to-silver consistency checks.
Parser versioning: If parsing rules change, storing the parser name and version with extracted records can make differences between old and new silver data explainable.
Downstream policy: BigQuery and RAG should primarily consume explicit structured values rather than infer structure from raw email bodies whenever deterministic extraction is available.

The overall role of mailbox-smith is therefore broader than an MBOX-to-JSONL converter.

It provides an inspectable boundary between original mailbox data and downstream analytical systems:

MBOX
  │
  │  human-inspectable source
  ▼
mailbox-smith
  │
  │  deterministic parsing
  │  validation
  │  provenance
  ▼
JSONL
  │
  ▼
BigQuery / RAG

The underlying philosophy is simple:

Interpret once, deterministically, as close to bronze as possible. Store the interpretation explicitly in silver.

And when deterministic interpretation is no longer sufficient:

Use AI only where ambiguity begins.