Spiced Up

When the first version of the Internet Message Access Protocol 4 (IMAP 4) appeared in 1994, mobile phones looked more like bones you would give your dog than phones – and they couldn't do much more than bones either. Digital data trickled off the Internet through modems in homeopathic doses. The Universal Mobile Telecommunications System (UMTS) and mobile email clients did not even exist, and email was only tentatively gaining ground.

Growing Demand

To adapt IMAP 4 to modern demands, and especially mobile clients, the Network Working Group of the Internet Engineering Task Force enriched the IMAP protocol in a whole series of RFCs with extensions that give IMAP new features, but without disturbing clients and servers that use older versions.

Many of the protocol extensions are still at the proposal stage; the unofficial IMAP wiki [1] has a fairly comprehensive list (see Figure 1) if you are interested in finding out more. Some of these extensions are also part of the Lemonade (License to enhance message-oriented network access for diverse environments) profile. Under this umbrella, RFC 4550 [2] gathered more than 20 IMAP protocol extensions explicitly to improve communication with mobile clients.

Although it's possible that no server has implemented the more exotic extensions, administrators are amazed that they need to enable other extensions on the server because they are not an integral part of the protocol (Table 1).

In this article, we present a selection of commonly used extensions and briefly explain their purpose. More details are available in the related RFCs, which are often dozens of pages long.

Discoverers: CAPABILITY

The CAPABILITY command [3] is not an extension but an important function of IMAP 4. The client uses it to discover which extensions the server supports. The server returns a long line that lists its capabilities as key words. If the ID command [4] is enabled on the server side, the client is allowed to send information about itself to the server, such as its version number and name. However, the RFC says that the server and client should not use this information to try and work around bugs; instead, they are meant to help the postmaster identify bugs and notify the software vendor.

Silent Observer: IDLE

According to the IMAP 4 protocol, a client must actively ask the server whether new email is available or whether someone has deleted existing messages, although it makes more sense for the server to notify the client when new email arrives because such on-demand requests save resources. Although the server might occasionally respond with EXISTS (e.g., if the size of the mailbox changes), the client cannot rely on this behavior and has to ask anyway.

If the server returns IDLE [5] as a capability, the client can allow it to send unsolicited messages about new email while the client is in IDLE mode. To enable this, the client sends an IDLE command to the server, which replies with a continuation request (+). As long as the IDLE status exists, the server can send messages without sequence numbers (untagged), such as EXISTS or EXPUNGE. The client itself cannot send any commands of its own at this time.

The master/slave relationship ends as soon as the client sends a DONE. In this case, the server sends any outstanding untagged lines and responds with a sequence number to the DONE. If the client fails to terminate the IDLE command by sending DONE, the server can throw it out, assuming a timeout is defined. Thus, the RFC recommends that clients send another IDLE after 29 minutes to avoid this fate.

Pre-Filtered: NOTIFY

IDLE has the disadvantage that it neither controls nor limits what commands the server sends and how it responds to certain events. Because idle also only works for a single mailbox, the server and client set up another TCP connection for each additional mailbox query. NOTIFY [6], however, puts the client in charge. It extends IDLE, letting the client determine the mailboxes from which it wants to receive messages. This makes NOTIFY a multi-mailbox IDLE while also streamlining communication.

The client uses NOTIFY SET to specify both the mailboxes and the kind of information it wants to receive. The server then sends a set of attributes  – along with the FETCH response – to the client that now becomes a passive observer. The NOTIFY effect continues until the client sends a new NOTIFY command or one of the entities closes the IMAP connection.

Because some mobile clients only want updates for messages that match a certain search pattern, the NOTIFY RFC also defines additional attributes for the UPDATE option, which take effect in this case.

Fully Synchronized: QRESYNC

The QRESYNC [7] extension quickly resynchronizes a user's mailbox. It is an extension of the CONDSTORE command. CONDSTORE in turn checks a mailbox for state changes that occur when a user is using different email clients or multiple users share a mailbox. If a user on client A changes the status of an email to "unread," you also want client B to notice this; the client needs to sift through the mailbox again (resync) to do this.

Although CONDSTORE identifies these changes and resolves conflicts caused by simultaneous access by multiple clients, Thunderbird and other mail clients need to send the UID FETCH and UID SEARCH commands in this case. In contrast, QRESYNC allows for a resync, including tracking of deleted files at one fell swoop. The VANISHED response introduced for this (and replacing EXPUNGED) discovers deleted email efficiently.

Use of the SELECT command on the client side removes the need for concurrent connections to the server that only exist to prevent resyncs. QRESYNC especially benefits mobile devices because they often have to retrieve their data more often because of poor network coverage. Little wonder that this extension is one of the Lemonade extensions mentioned earlier.

To use QRESYNC, the server needs to return ENABLE [8] as a capability. The ENABLE extension lets clients enable a particular capability explicitly. In the case of QRESYNC, the RFC requires the clients to have previously sent a targeted ENABLE QRESYNC to the server.

On the Move: MOVE

Before the IETF issued a helpful RFC [9] with two statements – MOVE and UID MOVE, making it possible to move email messages between mailboxes  – the only option was to combine several independent commands (COPY, STORE, and EXPUNGE) to achieve the same effect.

This implementation was found to be suboptimal for several reasons: If communications break down between the three steps, the move processes become stuck. This effect also confuses users because their clients still show the messages in this intermediate state. Additionally, in shared mailboxes, the third step might move not only the selected messages but also, inadvertently, messages marked for deletion by third parties.

MOVE solves these problems, whereas UID MOVE moves messages on the basis of their unique identifiers. For a client to discover whether a server supports this feature, the server needs to respond to the CAPABILITY request with MOVE.

Quick Search: ESEARCH

The IMAP standard already includes two search functions in the form of SEARCH, which searches email messages on the basis of their Message Sequence Numbers (MSNs), and UID SEARCH, which uses the unique identifier. One disadvantage, however, is that you cannot limit the number of returned results.

ESEARCH [10], on the other hand, gives the seeker various result options. Thus, you can limit the search results to a minimum and maximum value or return only a certain number of results. Even finding all results is quicker because the search results are returned as a sequence set, which is compact and saves bandwidth. What's more, this set can be used in a subsequent command.

New: SORT and THREAD

The next two extensions, SORT and THREAD, share a common RFC [11]. SORT surprisingly is not a fixed part of the IMAP protocol. Representing email in threads is not really new either, and ultimately only a sort algorithm. SORT and THREAD take place on the server side, thus saving client resources (Figure 2). Both extensions have to process the terms to be sorted in line with the requirements of I18NLEVEL=1, which is part of an RFC [12] that regulates the handling of international character sets.

The server advertises its sorting capability by returning SORT as a response, and it also outputs the threading algorithm for the THREAD capability. A server response thus might be THREAD=ORDEREDSUBJECT, which is the simpler of the two available threading algorithms the RFC introduces as "the poor man's threading." The algorithm first sorts email by subject then, given the same subject line, sorts by delivery date, where ORDEREDSUBJECT then converts these messages into threads.

Of course, this can cause problems if someone replies to a thread but changes the subject line. In this case, the REFERENCES algorithm chips in. It is much more sophisticated than ORDEREDSUBJECT and needs six main steps to create a thread (explained in detail by RFC 5256 [11] on pages 8-11).

More Accessible: UIDPLUS

According to the RFC, UIDPLUS [13], part of the Lemonade specification, is an extension that handles particularly hard-to-reach clients. The extension is designed to reduce time and resource consumption. To do so, it defines an additional command: UID EXPUNGE deletes messages that have both the \Delete flag set and a unique identifier (UID) that appears in the sequence set of the currently selected mailbox. It proves to be particularly suitable when a client that is only temporarily connected re-synchronizes with the server.

If the UID EXPUNGE extension is used instead of EXPUNGE, the client no longer accidentally deletes messages from other users that have been marked for deletion while it was offline. In this case, the second criterion does not apply because the UIDs of the deleted messages do not belong to the mailbox. This behavior becomes relevant when users share mailboxes or for users of both mobile and fixed clients.

Tricky: LIST-EXTENDED

Traditionally, the commands LIST and LSUB show the contents of mailboxes. But new IMAP extensions have given rise to a need for specific types of lists. Because LIST and LSUB are not innately extensible, clients were forced to run a series of increasingly long commands to achieve the effect required by the extensions.

The LIST-EXTENDED extension [14] solves the problem by modifying the existing LIST command, such that it no longer requires any special commands but can be supplemented with a variety of options and search patterns. The new syntax is downwardly compatible and is used only if the first or second word after the command starts with a bracket, or if the LIST command has more than two parameters. Otherwise, the traditional LIST command serves as a fallback.

Real Names: NAMESPACE

Because IMAP 4 does not define a default namespace, two generic namespace conventions have emerged: In the "Personal Mailbox" model, the default namespace only represents the personal mailboxes of the user. In contrast, the "Complete Hierarchy" model includes all other digital mailboxes that the user can access – in addition to the user's own mailbox. The problem with these models relates to the configuration overhead associated with them.

The NAMESPACE command [15] allows clients to discover automatically the server-defined prefixes for mailboxes (Figure 3) and so distinguish between personal mailboxes, mailboxes of other users, and shared folders. This allows admins to assign public inboxes (e.g., the mailing list) a prefix of public, to distinguish shared mailboxes with the shared prefix, and to tag the per-user mailboxes as private. This not only saves manual labor, but the mailboxes on a namespace can thus also reside in different places on the network and use different formats, for example Maildir and Mbox.

Everything Goes: ACL

Mailboxes assigned with the use of NAMESPACE typically also have different access rights. Access Control Lists for mailboxes and the ACL [16] extension allow the administrator to view and modify rights using IMAP commands. The associated RFC from 1997 turned out to be lacking; thus, RFC 4314 [17] followed up by defining new access rights and adding clarity to the rights situation.

Besides the ACL string, the server needs to use RIGHTS = to indicate the rights it supports when responding to CAPABILITY: currently 11 standard rights and two virtual rights (d and c), which exist for reasons of compatibility with RFC 2086. Possible commands in combination with the RFC include: SETACL, GETACL, DELETEACL, LISTRIGHTS, and MYRIGHTS.

ACL entries consist of two components (access identifier and rights) and can be assigned to specific mailboxes. The access identifier usually consists of the username (a UTF8 string) and is also accepted by LOG and AUTHENTICATE. However, more than one identifier can exist for each user. Additionally, the RFC defines an anyone identifier, which references a generic identity that includes anonymous logins. If the identifier is preceded by -, the associated user is denied the assigned rights.

Whereas lowercase letters show the rights of the user, the RFC uses numbers for rights assignments in the context of server implementations: Postmasters can use these to assign or deny certain rights across the board, thus ensuring, for example, that users can only manage their own mailboxes.