Sieve filtering

A mailbox needs structure: this is a newsletter, that is from the list, this one can be hidden. Sieve is the standard language for exactly that, so rules are portable, readable and already known to people who run mail servers. Oaken has its own small Sieve engine (oaken.mail.sieve), pure like the parser: no database, no network, no model.

raw.db message ──parse──▶ ParsedMessage ──sieve scripts──▶ disposition + labels + flags
                                                              │
                                                              └─▶ message_attrs in raw.db

Filtering runs after the message is stored and before it is indexed. Because the raw message is kept, changing a script re-derives the attributes with no new download.

Scripts are *.sieve files in the sieve folder next to config.toml (mail.sieve.dir changes it). They run in name order (10-lists.sieve, 20-work.sieve), and stop ends them all.

require ["fileinto", "imap4flags", "vnd.oaken.analysis"];

# Mailing lists get a label and are marked read.
if exists "list-id" {
    fileinto "Lists";
    addflag "\\Seen";
}

# Newsletters never go to a model.
if header :contains "list-unsubscribe" "" {
    skip_enrich;
}

# Mail from people I work with stays in the inbox.
if address :domain :is "from" "example.com" { keep; }

A script with an error is reported and skipped, so one typo does not switch off the rest. A script that fails while running (a bad regular expression built from a variable, a limit reached) loses its own actions only, and the message stays in the inbox.

Trying a script

You edit the files; nothing writes them for you. oaken sieve check FILE compiles one and names the line of the first error. oaken sieve test FILE --message ID (or --eml PATH) runs it on one message and shows each effect with the script:line that caused it, storing nothing. oaken sieve list shows what core would load and any script it would skip, and why. oaken sieve refilter (the filter stage of reprocess) re-reads the files and re-runs them over stored mail.

Oaken reads and organizes mail. It does not deliver it, so Sieve actions map onto a mailbox view:

The disposition is one of three values. Another action beats discard, and an explicit keep beats everything. A message is one object, so unlike a delivery agent, flags are the flags in effect when the scripts finish (a discarded message carries none).

Everything in RFC 5228 (header, address, envelope, exists, size, allof, anyof, not, if/elsif/else, stop, keep, discard, fileinto, redirect) plus these extensions: fileinto, copy, imap4flags, envelope, body (:text), relational, regex, variables (including match variables), date, subaddress, reject, ereject, vacation, the comparators i;ascii-casemap (default), i;octet and i;ascii-numeric, and vnd.oaken.analysis.

A require of anything below is an error with the reason:

body looks at the message text only: the text parts, or the HTML turned into text when there is no plain part. An attachment's contents are never visible to a script.

The SMTP envelope is not available from a provider, so envelope is derived from Return-Path (sender) and X-Original-To or Delivered-To (recipient), falling back to From and the first To. The date tests use UTC unless you pass :zone.

Attributes live in raw.db (message_attrs), because organization is what the user sees and a backup of raw.db should keep it.

A re-run replaces only the sieve rows of that message, in one transaction. message_filters records the hash of the script set that last ran on each message. Edit a script and the hash changes, every message becomes stale, and FilterEngine.refilter() walks them oldest first, resumably. A MessageFiltered event is published for each message.

Mail is hostile input, and a script is code, so both are bounded:

• A step limit (max_steps) counts every command, test and key comparison, and an action limit (max_actions) caps what one script can produce.
• A :matches pattern is matched by dynamic programming, so it cannot backtrack, and the text is cut to max_match_chars. A :regex is checked when the script is compiled and refused if it repeats a repetition ((a+)+), and the text is cut to max_regex_chars. The check is best effort.
• Scripts are size-limited, nesting is limited, and numbers are bounded.
• The package imports no network, process or mail-sending code, and a test checks that.