Generated from docs/softmap.toml by docs/tools/softmap.py, which refuses to build if the data breaks the layering rules or disagrees with the repository. Edit the data in the same change that builds or alters a component.
2
Map
Click a component to draw its relationships. Each separate piece of software is its own framed stack, with its layers top to bottom. The border shows status; a red dot marks components that handle untrusted input.
calls (selected component calls this)called bynotifies (event or callback, upward)notified by
Select a component on the map.
3
Conventions
Calls go down. A component may call its own layer or a lower one, never a higher one, and the call graph has no cycles.
Events go up. Lower layers reach higher ones only through events or a handler the higher layer registered.
Plugins meet core at a contract. Core never imports provider code, and a plugin never touches the databases.
Mail is data. Anything that handles message text is marked untrusted, and its text reaches a model only as UntrustedText.
Mutations are gated in code. The operation registry's confirm gate, not a prompt, decides whether mail may change. Sending is never exposed over MCP.
index.db is disposable. Everything in it can be rebuilt from raw.db.
Thin command-line client of the local API: daemon control, mail commands, the generic operation fallback and the event stream. Commands are lazy-loaded; most are one registry operation each.
Forwards one JSON-RPC message to /mcp (or /mcp/agent) and returns core's answer, or a structured MCP error.
External none
Notes
Does not start core and never opens the databases; it imports no core code beyond discovery.
Core down: a tool call is an isError result {code: core_not_running} with the start hint, other requests a JSON-RPC error, and initialize is answered locally so a client that starts first still connects. It finds core again on every message.
--agent speaks to the swarm door with OAKEN_AGENT_TOKEN.
Browser client in its own app (oaken-webui), parallel to oaken-cli: it serves the static UI and reverse-proxies /v1/* to the running core over loopback — the general API bridge, never MCP. Core stays headless (ADR 0007).
Holds the API token server-side and adds X-Oaken-Client: ui; the token never reaches the browser, which sees one same-origin server.
Serves the oakenmail-templates UI (demo marker stripped so it calls the API); the published demo under docs/ keeps the marker and its dummy fixtures.
Partial: the server, the proxy and the static serving are built and tested; the UI pages themselves are only partly wired, and core's vestigial static_dir mount is not yet removed.
Loopback only; does not start core. Core down -> 503 core_unavailable, like the CLI.
General local API for the CLI and web UI: HTTP and server-sent events on 127.0.0.1 with token auth. Routes are generated from the operation registry; a guard checks Host, Origin, content type, token, rate and size before any route runs.
The AI-enabled surface of core's API and the heart of the product: a local, data-only MCP server for sensitive mail. Tools are generated from the operation registry; sending is never exposed.
The SDK's stateless streamable-HTTP server over a CoreClient, plus the Surface that guards it. One class for the person's door and the swarm door.
Functions
Signature
What it does
McpServer._call_tool
Calls the registry as the caller class mcp; a mutation returns a proposal and needs the token; mail-derived results are fenced as untrusted; results are compact and bounded.
External none
Notes
Runs whether or not the web UI does. Its own token (mcp_token), caller class and limits (ADR 0004).
Mounted at /mcp on the loopback listener; [api] mcp = false removes it. A door is exactly its own path.
With the swarm on, a second door at /mcp/agent takes agent tokens and offers claim_job, heartbeat_job, complete_job and fail_job from an agent-only registry; the registry audits each claim, completion and failure with the agent named.
Hybrid keyword and vector search merged by rank fusion, a token-budgeted context, and question answering whose every claim must quote the message it cites.
Built means: tested with the fake embedder and model, including a vector outage, hostile queries, fabricated quotes and the budget as a property test. Offline numbers use stand-ins on a synthetic corpus built so each retriever has a weakness; a live run is manual.
ask refuses mail from more than one account in a single prompt, and is off over MCP until the user turns it on.
Nothing attaches Services.retrieval or a model service to a running process until the daemon (prompt 16).
Marks mail read only when the user opens it or says to (a summary alone never does). Local first: sets the state and a dirty marker, audits, then writes back through a durable job. A change nothing can write stays local and says so.
mark_read, mark_unread, message_presented, message_dismissed, undo, resume_pending and apply_server_flags (a SyncRunner flag sink).
ReadResult
dataclass
What changed, what was unchanged, missing or local-only, and the audit ids.
External none
Notes
The write job is readstate.write, one per message. A system actor is refused outright; reasons that are not the user's need confirm=True, which the operation registry passes after its gate.
Providers are keyed by account id, so write-back reaches the right mailbox of several.
Per-account and per-folder sync schedules in the database: interval or cron with a timezone, quiet hours, a floor, a concurrency cap, catch-up, backoff, an authentication pause and an audit trail for every edit.
Staged, idempotent, resumable: store, then Sieve filter and index (keyword-searchable from here), then embed and enrich asynchronously. Every stage skips work whose content hash it has already done.
Registers the ingest.parse, .embed, .enrich and .reprocess jobs; embed_message, enrich_message, reprocess, backfill_paused and pipeline_status.
SyncRunner
class
Pulls from a provider, stores each message, queues parse, saves the cursor only after the batch. A backfill pauses at the queue cap; new mail never does.
Backends
dataclass
Callables for the embedder, LLM, enrichers and vector store, so plugins can come and go while the pipeline runs.
Vectors
class
The one seam between core and a vector store. Optional store capabilities (stored_hashes, begin_model, activate_model, drop_model, prune_orphans) are detected, not required.
UsageStore
class
ai_usage in raw.db: calls, cached calls, tokens in and out, and tokens saved by caching.
Quarantine
class
Writes flagged attachments to an owner-only folder. Never reads, parses or runs them.
PipelineStatus
frozen dataclass
Per-stage counts, embedding speed, backfill state and the parked (failed) jobs with reasons.
Functions
Signature
What it does
Pipeline.reprocess(stage) -> job id
One low-priority job per stage and account; parse queues index.rebuild, embed swaps models without removing the old vectors first.
SyncRunner.sync(account, provider) -> SyncResult
Stored, unchanged, deleted, flag changes, paused and errors. Provider errors are reported, not raised.
UsageStore.usage_summary(since) -> list[UsageRow]
Per stage and model; enrichment tokens are estimates.
PluginContext.subscribe(event_type, handler)
Plugins listen to core and plugin events, read-only; the host ends them when the plugin stops.
External none
Notes
Order is filter first, so discarded or skip_enrich mail never reaches the model stages. Embed runs at priority-1 and enrich at priority-2 of the parse job.
Failures retry with backoff up to ingest.stage_max_attempts, then are parked and shown in pipeline_status. Keyword search works throughout an outage.
A changed embedder model leaves embed jobs 'stale' until reprocess('embed'); new mail during a swap is embedded to the building model only.
Enrichments live in raw.db (they cost model time); index.db stays disposable.
Several vector stores: only the Vectors seam exists; split and caps are planned (see 08 notes).
docs.cleaned_text is duplicated with chunks.text for now; pointers are a later prompt.
Flagged attachments are written, never read back, to a 0700 folder as 0600 .quarantined files named so they cannot escape or execute. Over-size attachments stay in the message.
RFC 822 to structured fields and cleaned text (quotes, signatures and boilerplate removed); thread keys; token counts. Pure and deterministic: no database, no network, no model.
The user's Sieve (RFC 5228) scripts organize mail as it lands: a disposition, labels, flags and an analysis marker, stored as attributes in raw.db. A small engine of our own; sending actions are recorded and never executed.
Entry-point discovery, validation, lifecycle, capabilities and health for the five plugin contracts. Fakes and reusable contract suites live in oaken.testing.
The provider contract: pull, flag write-back, auth and send, each declared by overriding its method. First-party providers are Gmail, SMTP (send only) and IMAP. pull takes an optional folder; one instance runs per account. Nothing routes a send yet.
The LLM backend contract: structured outputs, with a locality flag so a secure workspace can refuse a remote model. The first-party backend uses a local Ollama.
Summary, triage, actions and the header/MIME inspector: first-party enrichers whose results are stored once at ingest. The three that use a model reach it only through ctx.llm and generate_cached.
The one way a plugin reaches a model; core picks the account's workspace.
python -m oaken.evals enrich
Scores the eval fixtures against the configured model; manual, never CI.
External none
Notes
Built means: the four enrichers run in the real pipeline against FakeLLM, with tests for every rule, the injection fixtures and the cache. Not yet run against a live model: do that with oaken.evals.
summary and actions skip bulk mail by default (skip_bulk).
Core does not yet attach an LLM service to the host (no daemon until 16): host.llm_access = WorkspaceLLM(service, raw).
The first-party embedder and LLM backend for a local Ollama. Loopback only unless ai.allow_remote is set; never pulls a model; structured output through Ollama's format schema.
The first-party Gmail provider: full and incremental sync, flags, labels, trash, archive, star and send, over Google's REST API with a bring-your-own OAuth client. The refresh token lives only in the OS keyring.
Calls
none
Called by
none
Notifies (event or callback, upward)
none
Notified by
none
Types
Name
Kind
Meaning
GmailProvider
class
The oaken.providers plugin: authenticate, pull, set_flags, send, plus labels, trash, archive and star.
TokenManager
class
Keeps an access token valid: proactive renewal, one shared refresh, 401 retry, rotation, invalid_grant becomes a health state.
KeyringStore
class
The refresh token in the OS keyring (service oaken, user gmail:ACCOUNT); no keyring means a clear error, never a file.
GmailApi
class
REST client: timeouts, a concurrency cap, backoff on 429, rate-limit 403 and 5xx, Retry-After, one 401 retry.
Cursor
dataclass
Sync position: a full sync's page (with the history id read before listing) or an incremental history id.
GmailConfig
pydantic model
[plugins.gmail]: client_secret_file, account, include and exclude labels, page_size, concurrency, timeout, retries.
Functions
Signature
What it does
GmailProvider.authorize(show_url)
PKCE loopback OAuth in the browser; stores the refresh token in the keyring.
GmailProvider.set_flags(ids, add, remove)
Seen is the UNREAD label's absence, Flagged is STARRED; one batchModify, per-id fallback with PartialFailure.
GmailProvider.send(raw)
messages.send; never retried after a timeout or 5xx, since the mail may have gone.
External Gmail API, OS keyring
Notes
Scope is gmail.modify only; no permanent delete and no drafts.
invalid_grant sets needs re-authorization, publishes an event and stops asking until the token changes. A Testing-mode Google app expires tokens in 7 days.
Spam and Trash are never synced; moving a message there reports it deleted.
Test live later: the fake server proves the requests, not that Google accepts them.
A provider that only sends: it submits a message through your own SMTP server over STARTTLS or implicit TLS, never in the clear. It declares only auth and send. The password lives only in the OS keyring.
Calls
none
Called by
none
Notifies (event or callback, upward)
none
Notified by
none
Types
Name
Kind
Meaning
SmtpProvider
class
The oaken.providers plugin: authenticate (log in once, send nothing) and send. One connection per send.
SendFailed
exception
What a refused or failed send raises: accepted and refused recipients with the server's codes, and transient (4xx, network) versus permanent (5xx, TLS).
AuthRequired
exception
The password is missing or was rejected; storing a new one fixes it, retrying does not.
KeyringStore
class
The password in the OS keyring (service oaken, user smtp:ACCOUNT); no keyring means a clear error, never a file.
The envelope and bytes to submit: Bcc goes to the envelope only and is removed from the headers, a Message-ID is added if missing, nothing else changes.
SmtpProvider.send(raw)
Returns the Message-ID. A partial refusal raises SendFailed listing who got it, so a retry can skip them.
External SMTP server, OS keyring
Notes
There is no plaintext option. A server that does not offer STARTTLS is refused before the password is sent, and certificates are always checked.
A wrong password (530, 534, 535, 538) makes the plugin unhealthy and publishes an event; no attempt is made again until a different password is stored.
Test live later: the in-process fake server proves the protocol handling, not that your provider accepts it.
A generic IMAP provider for any server: it mirrors every folder except Trash and Junk, finds new, deleted and changed mail per folder, and writes back flags, Trash, Archive and moves. TLS is always on and the password lives only in the OS keyring. It cannot send.
Calls
none
Called by
none
Notifies (event or callback, upward)
none
Notified by
none
Types
Name
Kind
Meaning
ImapProvider
class
The oaken.providers plugin: authenticate, pull, set_flags, plus trash, archive and move. One connection per pull or write.
ImapConfig
pydantic model
[plugins.imap]: host, port, security (tls or starttls only), username, account, folder include and exclude lists, trash and archive folders, page_size, flag_scan_limit, timeout.
Cursor
dataclass
Per folder: UIDVALIDITY, the highest mod-sequence read, and the UIDs delivered as ranges.
Session
class
One TLS-only connection: login, LIST, SELECT, UID SEARCH, FETCH, STORE, COPY, MOVE and EXPUNGE, with replies parsed to plain values.
AuthRequired
exception
The password is missing or was rejected; storing a new one fixes it, retrying does not.
PartialFailure
exception
Some messages changed and some did not; failed maps each message id to why.
KeyringStore
class
The password in the OS keyring (service oaken, user imap:ACCOUNT); no keyring means a clear error, never a file.
Functions
Signature
What it does
ImapProvider.pull(cursor)
New mail is the server's UIDs minus the delivered ones, deletions the reverse; flag changes come from CONDSTORE or a capped re-read; a changed UIDVALIDITY re-reads the folder.
folders.to_sync(all, config)
The folders to mirror: selectable, not Trash or Junk (server attributes first, name guesses only when the server gives none), within the include and exclude lists.
ImapProvider.set_flags(ids, add, remove)
UID STORE per folder; never sets \Deleted; an id from a renumbered folder fails instead of acting on the wrong message.
External IMAP server, OS keyring, imapclient (BSD)
Notes
No plaintext option. A server that does not offer STARTTLS is refused before the password is sent, and certificates are always checked.
Messages are fetched with BODY.PEEK, so reading never marks them read.
Without UIDPLUS or MOVE nothing is moved, because a plain EXPUNGE could destroy other mail marked deleted.
Special-use folder markings come from the plain LIST reply; otherwise name guesses apply. imapclient is synchronous, so every call runs in a worker thread, one at a time.
No IDLE and no QRESYNC yet: the scheduler polls, and deletions are found by comparing UID sets.
Test live later: the in-process fake server proves the protocol handling, not that your provider accepts it.
Source of truth: zstd RFC 822 blobs (content-addressed), accounts, provider ids, sync cursors and read-state intents, with an LFS folder for large messages. The job queue shares the file.
Everything derived and rebuildable from raw.db: cleaned text split into chunks and an FTS5 keyword index with ranked, highlighted search. Disposable: it is bound to one raw.db and can always be rebuilt.
Parse, clean, chunk and index one message; a deleted message is removed instead.
rebuild_index(raw, index, ...) -> RebuildResult
Walk raw.db oldest first and index again. Checkpointed, resumable, prunes deleted messages and publishes IndexRebuildProgress.
register_index_jobs(runner, indexer)
Handle the index.rebuild job (one at a time, backfill priority, one per account).
External none
Notes
External-content FTS5 (not contentless) so snippet() works; triggers keep it in step with the chunks table.
index.db records the raw.db store_id and refuses a different store; a lower pipeline_version marks it stale for rebuild.
Message text never reaches a log, and attachment bytes are never indexed.
Planned, later prompt (with the UI): chunk text as offsets into docs.cleaned_text instead of a second copy. Deferred: it saves about one copy of the text and costs a join on every FTS rebuild.
Mail reaches a model only as UntrustedText in a fenced, per-call-nonce block; instructions are constants. Answers are validated against a pydantic schema, with one repair retry. Results are cached per workspace.
Run every time the docs are built; the build fails if any of these stops being true.
Result
Check
Detail
pass
Components
35 components, 82 documented functions
pass
Relationships
64 calls and 6 events or callbacks
pass
Layering
Every call goes to the same or a lower layer; every callback goes strictly upward
pass
Call graph
Acyclic
pass
Call flows
3 flows, 26 steps, each following a declared relationship
pass
Repository
Built and partial components point at paths that exist; planned ones have none yet
Entry points, which nothing else calls because the user or an event drives them: CLI, MCP stdio bridge, Web UI, Core daemon, Attachment quarantine, Ollama plugin, Gmail plugin, SMTP send plugin, IMAP provider, Docs site.
8
Open points
Topic
Note
OAuth refresh tokens expiring
Having to re-authenticate every day or so is untenable. Prompt 11 must keep the refresh token alive: publish the OAuth app out of Testing mode (Testing-mode refresh tokens expire after 7 days), refresh access tokens silently, and fail clearly on invalid_grant.
SQLite driver
Standard sqlite3 in a dedicated executor, or aiosqlite. Prompt 02 decides and records an ADR.
sqlite-vec
Prompt 08 starts with a spike to confirm it loads on Linux, macOS and Windows before the VectorStore plugin is built on it.
Embedding throughput on CPU
Not yet measured. Tuning is deliberately out of scope until there is real mail to ingest.
Web UI framework
The web UI is its own app (oaken-webui) that serves the UI and proxies to core; core stays headless (ADR 0007). The client-side framework for the pages themselves is still deferred — they are plain HTML/CSS/JS for now.
Plugin sandboxing
Plugins are in-process and first-party only in v1. Isolation is a later decision.
OAKENMAIL