awk has no /* */ comments, only #. The C-style comments in the awk program made it fail to compile, so the normalisation produced nothing, both sides compared empty and every case was reported as agreeing: the harness said 0 differences whatever the servers did.

Two cases are added: a body part (BODY.PEEK[1]) and a partial body (BODY.PEEK[]<0.100>), the latter being what Thunderbird for Android asks for. leaf check-in: d857e8572b user: bapt tags: trunk

APPEND refused anything past a hidden 100 MB cap when no quota was configured, while the QUOTA responses reported the same account as unlimited. A client saving a sent message was therefore told quota exceeded by a server telling it there was no limit. The check now applies only to a configured quota. check-in: 293d2a1040 user: bapt tags: trunk

A part that is empty was skipped before the counter advanced, so the parts after it took the number of the one before. The part the client asked for, from the BODYSTRUCTURE it was given, then looked absent and was answered NIL. The numbering is positional, so every part counts; an empty one yields an empty body instead. check-in: 89b196f599 user: bapt tags: trunk

A delivery replaces a mailbox index by a rename (.index.tmp -> index), so a kqueue watch on the index file itself ends up on an inode that no longer exists: NOTE_WRITE never fires on it. Only the 60-second wakeup did the work, and new mail was announced late by up to a minute.

The watch is now on the mailbox directory, where a rename into it does fire. The descriptor belongs to the store, so the watch no longer closes it either. check-in: 361987d99f user: bapt tags: trunk

An imported message keeps the UID it had in Dovecot, and that UID is set by assigning store->uidnext just before the append. The storage files are read in their own order, which says nothing about the UID order: the uidnext left behind when the last message of a mailbox is placed is the UID of that message, not the highest one. The deliveries that follow can then be handed a UID that is already taken, which is how a mailbox ends up with two messages under the same UID and with a numbering its clients cannot make sense of.

uidnext is now left just past the highest UID the mailbox holds, at the two points that close a store after an import. A message whose UID is already taken gets a fresh one instead of a second message under the same UID.

Test: import_dovecot_mdbox_uidnext, with three messages whose storage order and UID order disagree (50, 10, 30). check-in: e7ccc6b006 user: bapt tags: trunk

Thunderbird, told that the server takes COMPRESS, turns it on and then sends nothing: every session ends at the COMPRESS acknowledgement, no command follows, and every folder shows empty -- 381 of them in one afternoon of logs. What the client sees is indistinguishable from an empty mailbox.

The compression itself still works when a client asks for it (the two tests that use it pass), so only the advertisement goes, until the stream is understood. The capability test now checks that it is not offered. check-in: d1d2a5e954 user: bapt tags: trunk

A directory that holds mailboxes but was never a mailbox itself has no index. Two paths made one: the STATUS part of LIST-EXTENDED opened a store for every listed directory, and SELECT opened one for whatever it was given. The empty index that resulted turned a container into a selectable, empty folder -- a client opening it showed nothing at all, and it stopped being reported as \Noselect.

Neither path touches such a directory now: LIST-STATUS has nothing to report for a container, and SELECT answers NO, as the protocol asks for a name that cannot be opened. check-in: 9a7138eec3 user: bapt tags: trunk

The section asked for may name a part of the part, as in BODY[2.MIME], BODY[1.TEXT] or BODY[1.HEADER]: the whole part by default, its MIME headers, or its body. SnappyMail uses those forms, and a request for one was dropped in silence, which is exactly the sort of answer a client cannot tell from an empty message. A part name that is not of that form is still refused. check-in: 1a95b0b3c6 user: bapt tags: trunk

SnappyMail asks for the body of a message as BODY.PEEK[1], a part of a multipart message, and the item was dropped silently: the reply held no body at all and the message showed up empty in the webmail.

The part is now located by walking the structure the BODYSTRUCTURE walker already knows, and answered with the section the client asked for. A part that does not exist is NIL rather than nothing at all.

The bytes of the message are fetched for that item too: without it the lookup ran on nothing and answered NIL for every part, which stayed hidden as long as the same FETCH also asked for something else that pulled the data in -- exactly what the first test did. check-in: 07f8c3fcf5 user: bapt tags: trunk

LIST-EXTENDED defines the \Subscribed attribute for a listing asked with RETURN (SUBSCRIBED): the server is to say, for each mailbox, whether it is subscribed. The option was accepted and then ignored, so every mailbox came back without it. SnappyMail asks for exactly that and shows only the subscribed mailboxes, which is why its folder tree stayed empty even once the containers and the CHILDREN flags were right.

The attribute is now set, from the same subscription list LSUB uses. check-in: be45d506fa user: bapt tags: trunk

A directory that holds other mailboxes was left out of LIST unless a message had been delivered to it directly, because scan_dirs only reported a directory that has an index of its own. A client builds its folder tree out of the names it is given, so every folder under such a directory hung from nothing: SnappyMail asked for its folders, was answered correctly for each of them, and still showed no tree -- the containers it needed were simply absent.

They are now listed, in order, and marked \Noselect with \HasChildren, which is what a directory that cannot be opened is. That is what Dovecot answers, and Dovecot is what the clients were used to. check-in: 555f64daf6 user: bapt tags: trunk

The server announces CHILDREN, but the \HasChildren and \HasNoChildren attributes were only worked out when a client asked for them with RETURN (CHILDREN); every other listing answered \HasNoChildren for every mailbox. A client is entitled to expect the flags from a server that announces the extension: SnappyMail builds the folder tree out of them and saw no folder with children at all.

Nothing is added to the answer, only the attribute that was already there is now right. check-in: a06449009e user: bapt tags: trunk

The importer sets uidnext to the UIDs of the Dovecot dump, which may arrive out of order: the last one written then leaves uidnext below a UID the store already holds, and every delivery after that takes a UID that is taken. Two different messages end up under one UID, and no client can tell them apart -- it lists the flags of one and fetches the body of the other, which is what opening a message showed.

The invariant is now enforced where every save goes through: uidnext is raised past the highest UID before the header is written. check-in: 270e4065d3 user: bapt tags: trunk

A change the client was told OK about could be dropped without a trace: mdbox_save() returns an error when the lock fails or when the index cannot be written, and nobody looked at it. The three steps that can fail -- locking, creating .index.tmp, replacing the index -- now log the reason. check-in: bd715c6dd5 user: bapt tags: trunk

The mailbox was read once, when it was selected, and kept as it was: a message delivered afterwards -- another process writes it -- or one expunged by another session stayed unknown to that session. A STORE or an EXPUNGE naming such a message answered OK and did nothing at all, and a client cannot tell that from success: the messages it had just marked deleted stayed in the mailbox, and pressing the sync key over and over changed nothing.

Every command that works on a mailbox now checks the index first, and tells the client what moved, the way RFC 3501 asks: EXPUNGE for what another session removed, EXISTS and RECENT for what arrived. NOOP is included, it is the polling command; the commands that replace the mailbox, and IDLE which watches it on its own, are left out.

Only the file size of the index is compared, so a command whose mailbox did not move pays a stat. check-in: 3ef015802f user: bapt tags: trunk

RFC 6609: a script may run another one in place, from the user own scripts or from the site directory, with :once to include it at most once per execution and :optional for a script that may be missing. A script that includes itself, directly or through another script, is refused, and the nesting is bounded at the three levels the RFC asks for.

Every script keeps its own local variables and its own capabilities, as the RFC requires, and the names it declares with "global" live in a namespace the whole execution shares, which the global prefix also names. A return stops the immediate script only, where stop halts every one of them.

The site script directory is opened before the sandbox is entered and travels with the delivery, as the user directory already did. check-in: c3966fa245 user: bapt tags: trunk

string (RFC 5229 5) compares two lists that come from the script rather than from the message, with the same match types as the other tests.

mailboxexists (RFC 5490) looks the mailbox up in the user directory, which the sieve environment now carries (sieve_env.userfd, filled by the LMTP delivery and by the IMAP imapsieve path). The metadata tests of that RFC need annotations this server has no concept of, so the mailbox capability is not announced: the test works, but no client is told to generate the other half of it. check-in: a8190a1a75 user: bapt tags: trunk

RFC 5260: the date test extracts an RFC 2822 date-time from a header (the value after a semicolon is accepted, as in Received), shifts it to the zone the script asks for -- :zone, :originalzone, or the local one -- and compares the requested part: year, month, day, date, julian, hour, minute, second, time, iso8601, std11, zone and weekday. currentdate is the same on the current time and has no header name. The tags are accepted in any order, as the RFC own example shows, the key is a string-list, and a part that does not exist is refused while parsing rather than silently.

subaddress (RFC 5233) adds the :user and :detail address parts to the address and envelope tests, and relational (RFC 5231) adds the :value match type with its operator to the size test. check-in: 22efed94ba user: bapt tags: trunk

variables: a successful :matches hands the wildcards to the script as ${1}..., ${0} being the whole matched value, each wildcard taking as little as possible and the whole list being replaced by every new match. The set action takes the six modifiers of the RFC, applied in the precedence order it gives, and refuses a match variable as a name. A test that the short-circuit rule leaves unevaluated no longer sets them: that is what the no_side_effects counter is for, raised around a branch that is not taken and around a term whose result no longer matters.

encoded-character: ${hex:..} and ${unicode:..} are decoded before the variable substitution, so what they produce is read again as a string, as the RFC own example shows. Both behaviours -- expanding variables and decoding sequences -- wait for the matching require, since without it "${...}" is plain text.

:regex (RFC 6608) compares with an extended regular expression. The system regex does not report the parenthesized subexpressions, so only the whole value is recorded in the match variables. check-in: 79fabb70f6 user: bapt tags: trunk

SORT and THREAD matched their search criteria by hand, for a handful of flag criteria, and took anything else for a match: a client asking for "SORT (REVERSE DATE) UTF-8 FROM contabo" got the whole mailbox sorted, the filter quietly ignored. Both now go through the evaluator SEARCH uses, which also answers BAD for a criterion that cannot be parsed.

They also answered with UIDs where RFC 5256 asks for sequence numbers, and the client read those as sequence numbers: as soon as the mailbox had a hole, SORT named the wrong messages (SEARCH already answered correctly). The UID forms keep answering with UIDs. check-in: 05f0fc1f61 user: bapt tags: trunk

SORT=DISPLAY, LIST-EXTENDED, CHILDREN, MULTIAPPEND, SAVEDATE and STATUS=SIZE were missing, and each needs its behaviour before it can be announced: a name that is advertised but not served is worse than none, because clients then rely on it.

The capability list now lives in one place. It was written out in the greeting and in the CAPABILITY reply, and the two could drift.

STATUS SIZE (RFC 8438) reports the sum of the message sizes. The STATUS items are built by a single function: the plain and the LIST-STATUS forms had drifted, only the second answering HIGHESTMODSEQ. That function serves LIST-EXTENDED RETURN (STATUS (...)) as well.

MULTIAPPEND (RFC 3502) appends each [flags] [date] literal group, the groups after the first being read from the stream, and the APPENDUID names the last message. A command following the last literal is told apart by its first character: a tag can never start with the "(", "{" or quote that opens a group.

SORT DISPLAY (RFC 5255) sorts by the display name of the From address, and by the address when there is no display name.

LIST-EXTENDED (RFC 5258) parses the selection options that precede the reference -- without that, (SUBSCRIBED) was taken for the reference -- and RETURN (CHILDREN) adds the CHILDREN attribute (RFC 3348), which used to be sent as \HasNoChildren for every mailbox.

SAVEDATE (RFC 8514) needed no code: COPY and MOVE already keep the internal date.

Left out: CATENATE, BINARY, NOTIFY, ESORT, CONTEXT=SEARCH, URL-PARTIAL, I18NLEVEL, LOGIN-REFERRALS and SNIPPET, whose behaviour is not there and which these clients do not use. check-in: a1a8c48406 user: bapt tags: trunk

It compared the raw header value, so :domain and :localpart were ignored, a display name could match as if it were an address, and a header carrying several addresses was only ever tested as a whole.

The header is now split into addresses -- quoted strings, angle brackets and comments are respected -- and each one, with the requested part, is tested: the criterion is true as soon as one matches (RFC 5228 5.1). The test covers a second address of a list, a display name and the address parts. check-in: b4b230da1c user: bapt tags: trunk

Four defects found while comparing with the Dovecot setup.

SEARCH handed the whole criteria string to a matcher that understood a single criterion, so every combination a client sends -- "UNSEEN SINCE 1-Jan-2020", "NOT DELETED", "OR ..." -- matched nothing, silently. The criteria are now parsed and evaluated as RFC 3501 defines them: a term list is an AND, NOT and OR take sub-expressions, parentheses group them. The same parser runs once with no message first, so an unknown or malformed criterion is answered with BAD instead of matching nothing. HEADER unquotes its field name (a quoted one was looked up with its quotes and never matched), CHARSET is absorbed rather than taken for a criterion (a text search prefixed with it returned nothing, and an unsupported charset is now BADCHARSET), and SENTBEFORE, SENTON and SENTSINCE compare the Date header, which was not implemented.

A message copied or moved into a mailbox was not added to the full-text index, and the search only scans the messages when the index has no hit: a query that had other hits silently omitted it. Both now index the new message, and MOVE no longer drops the keywords COPY kept.

The envelope test was not implemented: it compares the SMTP envelope (from and to) with the address parts :all, :domain and :localpart. An IMAP session has no envelope, and the test is then false, as in Dovecot.

ManageSieve did not list editheader and duplicate, both implemented, so clients never offered them; envelope is listed too. check-in: 523a04bb5f user: bapt tags: trunk

The debug log wrote the client command line verbatim, so a LOGIN put the password in clear and an AUTHENTICATE its base64 initial response, into files read by the daemon group. The argument list of those two commands is now hidden; every other command keeps its arguments, which are what identify a client bug in the wild.

The ManageSieve service logs its lines the same way and is fixed too. LMTP has no authentication, so its raw line logging is unchanged. check-in: 44dbdba54b user: bapt tags: trunk

PRAGMA user_version rewrites the database header, with a journal file and an fsync (31 ms measured here), and fts_open runs once per connection. Writing it unconditionally made every login pay for it: 49 ms per LOGIN, 0.84 ms once the write is skipped. Creating the table stays unconditional, it is idempotent and costs nothing. check-in: 62761b9d65 user: bapt tags: trunk

A command reply is written line by line (the output stream is line buffered), so a multi-line reply leaves unacknowledged data on the socket. Nagle holds every segment after the first until the client acknowledges it, and the client delays that acknowledgement: each such command cost an extra round trip, 40 to 70 ms measured over the loopback, while a single-line reply was immediate. A session polling 65 mailboxes with STATUS took 3.3 s instead of 5 ms.

The options were applied only on the TLS branch of handle_client, so plaintext connections had Nagle enabled; they are now set on both paths. check-in: 9ceba26d6a user: bapt tags: trunk

rspamd refuses a learn, including an explicit one, while the message already reads as spam with a high probability. In 4.1 that guard (lua_bayes_learn.can_learn) keeps its own default threshold and is not reached by the classifier autolearn.options block, so spam_min, ham_max and enabled=false there do nothing, and check_balance only governs scan-time autolearn. Record the supported bypass, a Learn-Type: bulk request header passed by rspamc, and note the learn cache. check-in: 48f6c9ce60 user: bapt tags: trunk

Two defects kept the spam and ham learning scripts from ever running.

The configuration parser looked for "imapsieve" at the top level only, while the other server settings and the deployed configuration put it inside server{}. The rules were silently dropped. The block is now read from server{}, with the top level kept as a fallback, and the loaded rule count is logged at startup so a configuration that loads none is visible.

vnd.dovecot.pipe also takes a list of arguments. The parser rejected it but still executed the program, so report-spam.sieve ran "rspam.sh" ["learn_spam"] as a bare rspam.sh, and rspamc learned nothing. pipe now parses the list and the helper passes it as argv after the program name, bounded to 16 arguments of 1024 bytes. A script that fails is logged instead of being ignored. check-in: 7da5d15f4d user: bapt tags: trunk

The harness subscribes to its mailbox, to cover SUBSCRIBE and LSUB, but never unsubscribed -- and deleting a mailbox leaves the subscription behind. A client that follows the subscription list, such as neomutt with imap_check_subscribed, then tried to open a mailbox that no longer exists and reported

NO Mailbox doesn't exist: ImapDiffTest

The setup now unsubscribes before deleting, which also clears leftovers from an earlier run, and a final cleanup unsubscribes and deletes both test mailboxes plus the one the UTF-7 case creates.

Measured on the live subscription files: Dovecot's list and bmaild's now hold the same 65 entries (Dovecot stores them tab-separated behind a version header, the importer converts them), and neither mentions a test mailbox. check-in: 08a65cea7f user: bapt tags: trunk

daemon -r restarts the supervised program forever with no backoff, so a permanent failure becomes an endless loop, and it makes the service impossible to stop as well: killing the program only makes the supervisor start it again. The daemon's rc.d already dropped the flag; the fanout's now does too.

Measured: with the flag gone, killing the supervisor and then the pid recorded in the pidfile really stops the fanout, which is what a final synchronisation needs before it wipes and rebuilds a store. check-in: 33307d00f4 user: bapt tags: trunk

The harness now starts from empty mailboxes, so a leftover from an earlier run no longer shifts the UIDs: that alone accounted for five differences, the three THREAD cases and the two SORT cases, which are all UID based. THREAD and SORT were already answering correctly.

The normalisation is now one awk pass over the replies rather than a chain of sed rules. It splits an item list into name/value pairs, keeping a parenthesised group as a single token, sorts the items by name, and sorts the flags inside a group. The protocol gives no meaning to either order, and the two servers use that freedom differently -- the reference echoes the order the client asked for in FETCH and uses one of its own in STATUS, bmaild does the opposite -- so the order is not a difference worth reporting. The values are still compared; only their order is dropped. Replies carrying an ENVELOPE, a BODYSTRUCTURE or a body section are left untouched.

With this, the harness reports 80 cases matching the reference and no difference at all. check-in: 4b182a26ce user: bapt tags: trunk

Messages that share a base subject without any references form one thread in the reference, written with each message in its own group -- ((1)(2)) -- because such a thread has no root: RFC 5256 2.1 calls it a dummy thread. bmaild wrote one group per message, so they came back as separate threads. Two refinements the measurements pinned down: when one of the messages carries a Re: prefix it is a reply, so the first message becomes the root and the thread is written beside it ((1 2)), and a lone message whose children come from real references keeps the ordinary form as well ((3 4)).

The separator was the other difference: a thread ended and the next began with ") (" where the reference writes ")(", so UID THREAD ORDEREDSUBJECT differed too even though its grouping was already right.

Measured against the reference on two scenarios, one with plain subjects and one carrying a Re: reply: THREAD REFERENCES and UID THREAD ORDEREDSUBJECT now answer exactly the same, and so do SORT (SUBJECT) and SORT (DATE).

thread_references_dummy_subject covers the plain case; thread_references and uid_thread_references kept their expectations with the separator corrected, as they were asserting the old form. check-in: 8257aaa8e8 user: bapt tags: trunk

SORT only handled DATE/ARRIVAL and SIZE: any other key fell through to the date comparator, so SORT (SUBJECT) came back in arrival order while the reference orders by base subject, and REVERSE SUBJECT was wrong the same way.

sort_entry now carries the base subject. It is read with mdbox_fetch_headers and base_subject, which the THREAD code already used, so only the headers are read. Two comparators use it, comparing case-insensitively and falling back to the UID so the order is stable.

imap_sort_subject delivers Zebra, Alpha and Mango and expects 2 3 1, which is what the reference answers.

Note for the next step: cmd_sort answers with UIDs whatever the form used, while a plain SORT must answer with message sequence numbers and only UID SORT with UIDs -- the same distinction SEARCH and UID SEARCH already make. The diff harness still reports its "SORT by subject" case for that reason, and its "UID SORT by date" case probably for another: the date comparator does not break ties on the UID, so with messages delivered in the same second the order is whatever qsort leaves. check-in: b62f176b72 user: bapt tags: trunk

Recent is a session flag. The reference counts it in SELECT and STATUS, but never returns it among the flags of a message, and it forgets it once the session that saw the messages has ended. bmaild did the opposite on both points: it put Recent in every FETCH flag list -- which the diff harness reported on three cases -- and it kept the flag for good, so a mailbox always looked newly delivered.

The flag table that emits flags no longer contains it, and mdbox_clear_recent clears it on the way out of imap_serve for a session that had the mailbox selected read-write. The index is written by the existing save, so a mailbox with nothing to clear costs no write.

recent_cleared_across_sessions checks that a second session sees none and fetch_flags_without_recent that a message with no flags of its own comes back with an empty list. The harness cannot see the first case: its own sessions append the messages they then count.

Measured: the suite passes, and the STATUS case of the harness now reports RECENT 0 like the reference. check-in: 15e21a8261 user: bapt tags: trunk

The cases of bmaild_test.sh share one maildir, so a case that counts messages saw what the cases before it had delivered: imap_search_unseen started failing as soon as the search cases appended to it. setup_env removes the directory first now, so every case starts from a known state.

tests/imap-diff.sh also carries what was fixed while using it as the reference comparison: the messages go to files instead of shell variables (command substitution strips the trailing newline, which left a bare CR and made the two servers disagree on a message size), raw() feeds openssl from a file so the session is not interpreted twice by printf, and the normalisation covers the greeting, BYE and the timezone forms.

Measured with it: 69 cases match the reference, 11 differ, none of them a capability that the server advertises but does not serve. check-in: 523d088f57 user: bapt tags: trunk

Three behaviours were fixed without a regression test: HEADER <field> <value> which answered nothing, the search hits that must be dropped when the mailbox no longer holds the message (a sequence number of 0 used to reach the client), and the LARGER/SMALLER criteria which did not exist at all.

search_header_field checks that the named header is the one read and that a value found in another header does not match; search_skips_expunged expunges a message and checks that no zero is answered -- only that, because the maildir is shared with the other cases of the file, so the exact result set is not predictable; search_larger_smaller uses thresholds well away from the real sizes so it does not depend on them. check-in: f20cd6c6c4 user: bapt tags: trunk

The criterion took the whole "field value" text as the query and looked for a header literally named "__hdr", so SEARCH HEADER SUBJECT imap-diff matched nothing where the reference returns the three messages.

The value is now the query and the field names the header. The index has no column per header, so this criterion skips the full-text search and is answered by the brute-force scan, which extracts that header and matches it, giving the same answer as the reference.

Measured with tests/imap-diff.sh: SEARCH HEADER and UID SEARCH HEADER now match; 69 cases pass against 11 differences, all of them either the \Recent flag or SORT/THREAD. check-in: f1a87dd6ef user: bapt tags: trunk

The full-text index can still hold a row for a message that was expunged, and SEARCH then answered with a sequence number of 0 for it (SEARCH BODY alpha returned "0 2" where the reference returns "2"), and UID SEARCH answered with a UID the mailbox does not have ("1 3" where the reference returns "3").

Translate UIDs to sequence numbers and validate in the same pass, keeping only the hits that resolve, so neither a zero nor a stale UID can reach the client.

Measured with tests/imap-diff.sh: SEARCH BODY and UID SEARCH BODY now match the reference. check-in: baa114c5d3 user: bapt tags: trunk

A client that sends PLAIN without an initial response is answered with an empty challenge and returns the SASL data in a CONT. That path did not remember the request id or a step, so the CONT was refused with "Unexpected continuation" and the client received a 535. SnappyMail authenticates that way: the Postfix log showed "SASL PLAIN authentication failed: Unexpected continuation" on every attempt, with sasl_username=(unavailable), while the inline form of PLAIN succeeded on the same account.

The pending request is now recorded (step 3) and the continuation runs the same PLAIN decode as the inline form. Both forms also clear the state on completion, so a late or repeated CONT is not matched against the old request.

Verified on the wire: AUTH .. PLAIN followed by CONT .. now answers OK where it answered FAIL reason=Unexpected continuation before. Covered by auth_service_plain_continuation. check-in: d8f5e5adfe user: bapt tags: trunk

The search index was never told when a mailbox went away. A mailbox deleted and created again under the same name reused the rows of its predecessor, and since the importer now preserves the UIDs of the server it copies, importing the same mailbox twice left rows carrying the old UIDs beside the new ones: searches answered with duplicated sequence numbers and with zeros for messages that no longer exist. Measured on a two message mailbox after several imports, SEARCH SUBJECT answered "1 1 1 1 1 1 0 0 0".

reindex-fts was unaffected, as it deletes the whole database first; the shared index is what makes this necessary for the other callers.

fts_remove_mailbox existed and was called nowhere. It is now called when a mailbox is deleted, and once per mailbox by the importer before it rewrites it.

The differential suite went from 57 to 68 cases passing out of 80; the text searches, LARGER and SMALLER are now identical to the reference. check-in: 46d5940972 user: bapt tags: trunk

The item was not recognised, so a client asking for the headers other than a given list received no section at all, where the reference returns every header except those named.

The extraction already keeps the named headers, so the negative form inverts that choice, and the reply names the section as the client wrote it.

Verified against Dovecot: HEADER.FIELDS.NOT (RECEIVED) and HEADER.FIELDS (SUBJECT DATE) both return sections of the same length, 160 and 65 octets. check-in: 072a80e101 user: bapt tags: trunk

Both were missing, so SEARCH LARGER <n> and SEARCH SMALLER <n> answered an empty set where the reference returns the messages whose size matches. The size is already recorded for each message, so the criteria are a comparison on it.

Verified against Dovecot on a 200 octet message: LARGER 10 and SMALLER 100000 both return it and LARGER 100000 returns nothing, on both servers. check-in: 0ee7a8c08c user: bapt tags: trunk

UID SEARCH had a criteria chain of its own, so a text criterion (SUBJECT, TEXT, BODY, HEADER) matched nothing there while SEARCH found it: the full-text search lives in cmd_search only. Both paths also pushed msg->uid into the result vector, so SEARCH answered with UIDs instead of message sequence numbers. The two only differ once a mailbox has gaps in its UIDs, which any store that has seen expunges will have.

do_search() now takes a flag telling it which numbering to return, UID SEARCH delegates to it, and the results are translated from UIDs to sequence numbers when the client asked for SEARCH. The translation is a binary search over the message vector, which is kept sorted by UID (mdbox_seq_of_uid).

Verified on a mailbox holding three messages with the second expunged: SEARCH ALL answers "1 2" and UID SEARCH ALL "1 3", and UID SEARCH SUBJECT <text> finds the message where it used to answer an empty set. check-in: 5d938e2694 user: bapt tags: trunk

A text part with no Content-Type is text/plain; charset=us-ascii (RFC 2045) and the reference reports that charset, where bmaild reported no parameter at all: a message without a Content-Type header, which is common, was described differently.

BODY[HEADER.FIELDS (..)] and BODY[HEADER.FIELDS.NOT (..)] now end the returned section with a blank line, as the reference does, so the section has the same length (105 octets for a two header selection, measured).

Both verified against Dovecot on a message carrying no Content-Type header: the BODYSTRUCTURE and the section length are now identical. check-in: 4e0a7b5e99 user: bapt tags: trunk

do_fetch_msg reported a single hardcoded part: always TEXT/PLAIN (or HTML), charset UTF-8, with the size and line count of the whole message. Clients read BODYSTRUCTURE to find attachments, so a multipart message was described as one text part.

Parse it instead: split a part into headers and body, read Content-Type, Content-Transfer-Encoding, the charset, the boundary and the name parameter, recurse into multipart bodies, and report each leaf with the octet count and line count of its own body. The line count is emitted for text parts only, the name and charset go in the parameter list, the disposition is NIL unless the part carries a Content-Disposition header, and the parts are concatenated without a separator before the subtype, so the reply has the shape the reference produces.

Verified against Dovecot on a multipart/mixed message with a pdf attachment: both answer the same BODYSTRUCTURE. check-in: b3d4351d6b user: bapt tags: trunk

bmailctl keeps the UIDs of the server it imports from, and gives each imported mailbox a fresh UIDVALIDITY. Appending in storage-walk order made mdbox_append_kw hand out UIDs arbitrarily: in a 4384 message INBOX the highest UIDs carried mail from 2022, so clients that order or synchronise by UID (SnappyMail, Thunderbird for Android) showed a scrambled mailbox. UIDs are only meaningful together with the UIDVALIDITY they belong to, so it is renewed too; without that, a client keeps a cache built against the previous numbering and reports an inconsistent mailbox.

mdbox_append_kw no longer sets FLAG_RECENT in bulk mode: an import is not new mail, and clients reported "N new messages" on every open. Both changes need a re-import to affect an existing store.

tests/imap-diff.sh compares a reference server with a candidate: the same mailbox and messages on both, the same catalogue of commands, normalised replies diffed. It covers every search criterion in plain and UID form, every body section, SORT and THREAD and their UID forms, flags and keywords, COPY and MOVE, modified UTF-7 names, and that each advertised capability is answered. The unit tests pass while the server is broken, since they check what the server does rather than what a client expects; the six bugs fixed today were found by this comparison. norm() uses sed -E, because FreeBSD sed matches nothing with BRE "\|" or "\[". Its first run also reports that BODYSTRUCTURE is a naive stub and that RFC822.SIZE is one byte too large. check-in: 167b672679 user: bapt tags: trunk

The importer appended messages in storage-walk order, and mdbox_append_kw hands out uidnext++, so UIDs were assigned arbitrarily: in a 4384 message INBOX the highest UIDs carried the oldest mail (the last ones were dated 2022, 2023 and 2026-01, measured). Clients that order or synchronise by UID were therefore scrambled -- SnappyMail showed an unreadable order, and Thunderbird for Android showed nothing between the date of its last successful sync and now -- while clients that sort by the Date header were unaffected, which is exactly the split that was observed.

mdbox_append_kw assigns uidnext++, so setting uidnext from the dump entry before appending gives the message the same UID it had on the server it came from. Both import paths do that now.

The msgs vector can then be filled out of order, while mdbox_get_uid() binary-searches it and merge_disk_index() merges two sorted sequences, so mdbox_save() sorts it before anything reads or writes the index.

Verified with a synthetic store: a dump announcing uid 17 and 18 now yields an index listing 17 and 18, where it used to list 1 and 2. check-in: 7e15dd1890 user: bapt tags: trunk

import-dovecot walked storage/m.* and dropped any message whose metadata carried no GUID, straight away and without counting it:

if (msg.guid[0] == '\0' || guid_imported(userfd, msg.guid)) continue;

Modern Dovecot keeps the GUID in its index rather than in the file, so a dbox file alone cannot always identify a message. On a store of 99662 messages, 41 were dropped this way: their GUID is in the doveadm dump, which reads the index, but absent from the file. They were then invisible to every client, and reproducible: the same 41 were missing after each import, in the same mailboxes.

Place them by the original mailbox ('B' metadata) and the UID recorded in the dbox message header, both of which the dump provides once "uid" is added to the fetch item list (the manual page now asks for it). The lookup is keyed on mailbox and UID because a UID alone repeats across mailboxes, and it still refuses to place a message the dump does not know, so expunged mail is not resurrected.

A message that neither the GUID nor the (mailbox, UID) pair can place is now counted in *skipped, so an import can no longer lose mail in silence.

Verified with a synthetic store: two messages, one with a GUID and one without but with "BWork", a dump identifying both, and both end up in the Work mailbox; before the change only one did. check-in: 2dc90085ba user: bapt tags: trunk

UID SEARCH had a criteria chain of its own, handling ALL, UNSEEN, SEEN, FLAGGED, DELETED, UNDELETED and a sequence set. Every other criterion -- SINCE, BEFORE, ON, KEYWORD, UNKEYWORD, YOUNGER, OLDER, $ -- matched nothing, and the command still answered OK with an empty set. A client that synchronises with "UID SEARCH SINCE <date>", which is what Thunderbird does, therefore never saw anything newer than its cached state: it displayed nothing after the date of its last successful sync.

Route that branch through search_match(), which already implements every flag it handled plus the date criteria. A new uid_set argument tells it that inside UID SEARCH a sequence set is a set of UIDs, as RFC 3501 6.4.8 requires.

The date criteria were missing altogether, so SEARCH SINCE, BEFORE and ON answered with an empty set too: measured with the witness "SEARCH SINCE 1-Jan-1990", which returned nothing on a mailbox that had two messages. They are implemented against the message INTERNALDATE, comparing calendar days rather than adding 86400 seconds, which would be wrong on a changeover day.

uid_search_criteria covers SINCE, YOUNGER, UNDELETED and a UID set. check-in: 60f6fb0db2 user: bapt tags: trunk

Mailbox names on the wire are modified UTF-7 unless the client enabled UTF8=ACCEPT. bmaild only handled raw UTF-8: it announced UTF8=ACCEPT but implemented nothing behind it, so a client that did not enable the extension -- which is what every client coming from Dovecot does, since Dovecot does not announce it -- sent "Perso/sant&AOk-" and was answered "Mailbox not found". Folders with accented names were unreachable for those clients.

Decode incoming names and encode outgoing ones, but only when the session did not enable UTF8=ACCEPT, as the RFC requires. The LIST, LSUB, GETACL, LISTRIGHTS and MYRIGHTS replies were also printing the name by hand with hardcoded quotes, which skipped the encoding and would have broken any name containing a quote; they now go through mailbox_astring() too.

utf7_mailbox_names creates, lists, STATUSes and SELECTs "Caf&AOk-" (Cafe with an accent) without enabling the extension. The ten assertions that demanded quoted names in LIST, LSUB, GETACL and MYRIGHTS replies were updated: Dovecot answers them unquoted, and quoting is now decided by whether the name needs it. check-in: f28b8e2e18 user: bapt tags: trunk