Our New Email Setup

In the previous post of this little email series, I have been bragging about configurability, and the associated learning curve. A prime example are Emacs and its Org mode, both of which I use. At the outset, when I start using a new tool, I am tempted to search the Interwebs for insight how power users make use of it. If you search for Emacs and Org mode setups, you will be swamped. Github bursts with “my emacs config” repositories, and Org mode is second to that by a small margin only. Hence, this description takes a “guided tour” approach, leaving room for your own considerations and choices of how to use email.

If you start trying out the setups you find when searching the Interwebs, you will notice a major drawback fairly quickly: most just don’t work. And that’s because they are either part of a bigger setup, and you are missing parts of that, or the UX they establish is so alien to your mindset that you are incapable of making any meaningful use of them whatsoever. Or all of the above. The important lesson that this teaches is twofold; for one, everybody’s mind works differently in subtle, and sometimes not so subtle ways; secondly, what’s really useful to you heavily depends on what is useful to what you do, and how you typically go about it. You will have to sit down and find out what works well for your brain, and what works well for the tasks at hand. Nobody can do this research for you, and yes, it will take time. Looking back, it seems hard to believe I could have achieved anything a year ago, two years ago, etc. given the darn simple (compared to what I have now) setups of Emacs and Org mode.

Am I finished with my setup? No, and don’t expect to be any time. It’s a continuous process. A temptation to withstand, is trying to advance too fast by pulling in configuration snippets from other people and seeing how you can make them work for you. It should rather be the other way around: think about what would be a super time-saver or work facilitating improvement. Dream it up; the sky’s the limit. Only then go out and see what config snippets you can find on the web to achieve what you imagined. The advantage of this approach is that it gives you time to go through a learning curve. Each new feature forces you to later your workflow. Your brain will need time to “work it in”, i.e. to remember the new sequence of steps to do something that you used to do differently (or maybe not at all) before. When you use the new feature like it had always been there, only then move on to adding the next improvement. This process of making gradual improvements, and at the same time learning how to use them is the only way, IMHO.

My requirements were:

With these words of warning to take it slowly, build up step-by-step, and to make sure you can revert back to the previous step and live with that until you have fixed it, here is my new mail setup.

The first thing to understand is that in each of your email accounts, there is a sending, and a receiving side. The two are technically independent services offered by you Email service provider. The sending, and receiving service use different protocols, and it has hence become customary to name them after the protocols they use: SMTP (for sending), and IMAP (for receiving).

Anatomy of an Email account with a sending, and a receiving side.

When you use any mail client software with a graphical user-interface, it will connect to both services, and use them for sending and receiving mail. This happens “under the hood” without you noticing. Technically, there is no reason for bundling the two together, however. You could use a separate software for sending and receiving; and this is exactly what we are doing.

Overview of our new mail setup.

As can be seen in the diagram above, the tools I chose are msmtp for the sending side, and mbsync (formerly known as isync) for the receiving side. I didn’t conduct any performance testing, or other practical evaluation for choosing between these two, and a couple of other options that exist. The choice is based on reading reviews, and gauging popularity from these, as well as skimming through the feature lists. Also, both can be installed via homebrew. I haven’t had to regret the choices yet, so can’t say anything negative.

msmtp offers a nice treat for macOS users in that it is able to retrieve the authentication passwords, for connecting to your sending servers, from the macOS keychain. Setting this up is fully described in the authentication section of the msmtp manual. Similarly, TLS certificates presented by remote servers can be verified against the roots of trust used (set tls_trust_file to system; see here). msmtp also comes with a two helper scripts to transparently (i.e. invisible to the MUA) queue up outgoing mail when there is no Internet connection: msmtpq, and msmtp-queue. msmtpq is meant to be used by an email client in “sendmail mode”; it is invoked by the email client directly as msmtpq. msmtp-queue is for queue management from the command line. Both scripts get installed into <install-root>/share/msmtp/scripts/msmtpq, and you will have to put them into your PATH one way or the other. There are a couple of descriptions out there on the Interwebs on how to automatically send queued messages once a network connection becomes available. I felt that this was not desirable, since it means whenever my laptop joins a network, for instance a public one at the train station, a network observer would immediately be able to know where I have email accounts. Instead, I have decided to make the flushing of the outgoing queue one step of the overall email polling, which I trigger manually. That way, I can control if and when that happens.

mbsync offers a similar macOS keychain treat via its PassCmd configuration option. To use the macOS keychain, you’ll have to set this to security find-internet-password -s <smtp-server> -a <user-name> -w (replacing <smtp-server>, and <user-name> with appropriate values, of course). Overall, the mbsync configuration is somewhat wordy, and setting things up may seem daunting at first. This is owed to the fact, that mbsync is a very powerful tool. mbsync‘s own documentation leaves some room for improvement here, so I would suggest that you search the Interwebs for examples of mbsync configuration files (like e.g. this one) to inspire yourself from.

With these two, you should be able to basically send and receive mail. I recommend, before moving on, that you experiment with the two tools a bit on the command line first, to verify that everything works as expected. Once you’re confident with that, it’s time to start thinking about what email workflow you want to implement.

Conceptual email workflow

At the very top level, it will be a carousel type process, that keeps repeating round after round:

  1. clean up the local email store (move messages between directories, physically delete stuff marked for deletion, etc.);
  2. download new email to a local mail store and sync local changes (e.g. deletions) back to the server;
  3. let me act on the email messages in my local store (read, delete, respond, etc.).

Before going into further details of the setup, let’s take a step back, and look at the tools on the far right of the overview diagram: Emacs and notmuch. About the former, I have ranted enough already, so I’ll focus on the latter. Email tagging is known by several names. Outlook calls it “categories”, Gmail calls it “labels”, and Apple Mail calls it “flags”, to name a few. The one true advantage of tags is that a message can only ever be in a single folder only, whereas you can give it as many tags as you like. Once properly marked up with tags, just throw it into the big black hole called “the archive”. Pull it back out by one of its tags, or by entering a search. The search facility is the key notion when it comes to successfully finding the information you need; so no prisoners are to be made here. To my experience, the search facilities of most all-in-one email applications work somewhat well for the casual user. But when you try to be a bit more sophisticated, more often than not, you’re hitting the ceiling of what search means the app offers you pretty quickly. I guess in terms of UX there is probably a trade-off between having a more elaborate search scheme, and deterring new users by its complexity. I’m power user, and I want a powerful tool. notmuch‘s search interface sets the gold standard for email searching, IMHO. Another pro about notmuch is that it comes with an Emacs elisp package, which provides you a UI for notmuch inside Emacs. Using PGP or S/MIME? No worries, Emacs and notmuch can handle that.

Coming back to the workflow carousel mentioned above, notmuch also plays nicely with you email setup for interfacing with your email accounts. notmuch‘s command for scanning for new mail, and adding the new messages o its database is notmuch new. Luckily, notmuch invokes a couple of hooks both, before and after this step. These hooks are executed, and can hence be executable programs, shell scripts, or symbolic links to these. With this flexibility, all you have to do is invoke notmuch new to make your workflow carousel spin a full round.

notmuch pre- and post-new hook scripts

This is for example my pre-new hook file:

#!/usr/bin/env -S zsh # -*- mode:sh; -*- # cleanup jobs # # (tag:deleted OR tag:spam) AND (age > 7d) --> DEL # folder:sent --> archive # folder:imap*/sent* --> archive # folder:imap*/archive* --> archive # (folder:imap/inbox AND NOT tag:inbox) --> archive # ~/Projects/bfew/bfew # synchronize to/from IMAP servers # mbsync -a # send queued outgoing messages # msmtp-queue -r
Code language: Bash (bash)

bfew is a perl script I wrote for myself, which does the things listed in the comment above its invocation. Since notmuch indexes and tags your emails, but it does never alter your emails in any way. Hence, when you mark a messages as “deleted” in notmuch, it is tagged as “deteled”, and not shown to you, but it physically still is on your archive. So you will have to do the the actual, physical deletion of the message yourself. Of course you can use notmuch‘s search capabilities to figure out which files to delete. After moving stuff around, or deleting files, be sure to run notmuch new --no-hooks to make notmuch aware of the changes you made, before issuing the next notmuch search command.

mbsync -a syncs the contents of my local mail folder with the IMAP servers behind my email accounts. New mail is downloaded, and messages I have moved away or deleted locally, are deleted from the server.

Finally, msmtp-queue -r sends out any queued, outgoing messages I have sent while not being connected to the Internet.

Now, after the pre-new hook is finished, I’m left with new email messages having been downloaded (mbsync -a), but not yet seen by notmuch. As the hook name suggests, the next step is that notmuch executes the actual notmuch new command, in which it discovers newly appeared messages (which have been downloaded in the course fo the IMAP sync), scans and indexes them, and tags them as new arrivals.

Just having new messages tagged as “new”, and “unread” is not much of help, however. The real productivity boost comes with clever filtering and tagging of the new arrivals, so as t give you an immediate, clear overview of what the new messages relate to. This is done in my post-new hook file:

#!/usr/bin/env -S zsh # -*- mode:sh; -*- # tag new mail using afew # afew -t -n # compact the notmuch db once a week # sched_file=${MAILDIR}/.notmuch/compact.next if [[ -f $sched_file ]]; then next_due=$(cat ${sched_file}) fi today=$(date +"%Y-%m-%d") if [[ ("${next_due:-}" = '') || ("$next_due" = "$today") || ("$next_due" < "$today") ]]; then next_due=$(date -v "+7d" +"%Y-%m-%d") echo $next_due > $sched_file notmuch compact fi
Code language: Bash (bash)

afew (home page, and documentation) is a python script to help you with the tagging. By default, notmuch tags all new messages with the “inbox” tag. afew puts “a foot in the door” by instructing you to configure notmuch such as to instead of “inbox”, use the “new” tag for new messages. afew then acts on al messages tagged with “new”, and at the very end of the filtering and tagging process, replaces the “new” tag on all messages that still have it with the “inbox” tag. The advantage of this approach is that afew can act only those messages that are actually new, and not just left in your inbox by you, and that you can choose to remove the “new” tag in the filtering process (for instance to archive messages from certain mailing lists directly, without presenting them in your inbox).

In the afew configuration file, the filter definitions must be numbered. If you start out with 1, 2, 3, etc. and later want to insert a new filter between – say – 2 and 3, you would have to renumber all subsequent filter definitions. To make this less error prone, to reduce the amount of work, and since the numbering need not be contiguous, I recommend to partition the number-space into blocks, for example starting at 100, 200, 300, etc., and simply keep appending filters to these blocks. Below is a redacted copy of my afew configuration to show this. Note how the numbering is scoped by the filter, i.e. it restarts with every new filter.

### # spam # [SpamFilter] # default filter [Spam(HeaderMatchingFilter)] [Spam.100] message = Spam (minimum level) header = X-Spam-Level pattern = ^\*{1,} tags = +spam [Spam.200] message = Spam (SPF) header = X-Received-SPF pattern = ^(fail|softfail|none|neutral|temperror|permerror) tags = +spam [Spam.300] message = Spam (x-w3c-hub-spam-status) header = X-W3C-Hub-Spam-Status pattern = ^([Tt][Rr][Uu][Ee]|[Yy][Ee][Ss])\b tags = +spam [Spam.301] message = Spamn (x-clx-spam) header = X-CLX-Spam pattern = ^([Tt][Rr][Uu][Ee]|[Yy][Ee][Ss])\b tags = +spam [Spam.302] message = Spam (x-suspected-spam) header = X-Suspected-Spam pattern = ^([Tt][Rr][Uu][Ee]|[Yy][Ee][Ss])\b tags = +spam [Spam.400] ... [Spam.401] ... [Spam.402] ... [Spam.403] ... [Spam.404] ... [Spam.405] ... [Spam.406] ... [Spam.407] ... [Spam.408] ... [Spam.409] ... ### # kill threads # [KillThreadsFilter] # default filter ### # mailing lists # [MailingLists(HeaderMatchingFilter)] # ignore stuff [MailingLists.100] message = Tagging mailing lists (<listname>) header = List-Id pattern = <...> tags = +foobar/devel;+deleted;-new [MailingLists.101] ... [MailingLists.102] ... [MailingLists.103] ... # more lists [MailingLists.200] message = Tagging ... header = Subject pattern = Flobber(-Info| Push Service) tags = +flobber-logs;-new # even more lists [MailingLists.300] ... ### # sent # [ArchiveSentMailsFilter] # default filter ### # inbox # [InboxFilter] # default filter
Code language: plaintext (plaintext)

The decisive part of each filter will apparently be the tags line. Compare line 71, and line 84 for instance. In line 71, I add the “deleted” tag, and remove the “new” tag. The former will cause it to be physically deleted after seven days (by my bfew script), and the latter causes it to not be picked up by the InboxFilter at the end (which will replace all “new” tags by “inbox” tags). This is, because this is for a mailing list which I want to have a look to eventually, but don’t want to swamp my inbox with, and don’t mind deleting older messages I haven’t looked at yet. On line 84, on the other hand, I am just removing the “new” tag, so the messages for this list are not shown in my inbox, but they are not deleted, because I want to archive them for possible later reference.

Suggested Best Practices

Tagging

Of course, the flexibility and unlimited nature of tags can be dangerous. It’s easy to spend fifteen extra minutes adding a ton of tags every time you read a new message. Assign tags to messages for tings only that can not be inferred from the message itself. There is for instance little merit in assigning a tag for all messages from a given person, because that information is of the From: header, and notmuch offers you a from:/<regex>/ search. Also, notmuch adds a few useful tags like “attachment”, “encrypted”, etc. automagically for you. My recommend approach is to try to keep the number of tags you assign to any given message as small as possible; ideally one, sometimes perhaps two, hardly ever three or more.

And it’s also easy to create so many different tags that you completely forget which ones you’ve used. Luckily, the notmuch UI inside Emacs helps you with this by giving you a master list of all tags you have attached to messages. This helps you keep an overview of all your tags, and spot typos in tag names.

notmuch also allows you to search for tags by regular expression. This allows you to establish a naming convention for your tags, and then search for a common prefix (tag:/prefix.*/). For my mailing lists, I for example use a naming convention of organisation/listname. So think about what incoming “email streams” you have, and what their semantics for you are. Maybe mailing lists are less of a concern to you, but clients and suppliers are; or departments and projects; or, or, or. Try to think about how you would be searching for these messages in the future.

Mail store folder structure

The diagram below shows the mail store directory layout that I have ended up with.

Suggested mail store folder structure

The drafts and sent folders at the top level are for what it says on the package. Postponed messages go into drafts until you finally sent them off. The sent folder is so that you can simply set the Fcc: header to sent, and a copy of the outgoing message will go there. Of course you will have to take care of moving messages from sent to the archive yourself (cf. what I said about my bfew script above).

Under imap-accounts I have one subfolder per account, and let mbsync store the contents of each IMAP server in the corresponding account folder. Once a message no longer has the “inbox” tag, my bfew script moves it from the imap-accounts tree over to the archive. That way, all messages tagged as “inbox” remain in the imap-accounts tree, and therefore also on the respective IMAP server, and are hence also visible to other email clients, such as e.g. the one on my smartphone.

Under archive I have one folder per month. Here, my bfew script analyses each message for in which month of which year it was sent, and files it in the corresponding folder, creating that folder in case it doesn’t exist. I chose this approach, since I wanted to exploit notmuch‘s capability of scanning only folders where something has changed since the last scan. With this folder layout, when invoking notmuch new, notmuch will not rescan any archive folders, except the one for the current month, because that is the only one where something would have changed (e.g. because messages have been moved from the imap-accounts tree). This should speed up new mail scanning considerably.

Do you need all features?

notmuch-mode in Emacs offers you a great many features, because it tries to expose as much as possible from notmuch‘s features, and allow you to cater things the way you want. But which way should you want it? I have no concrete recommendation for you here, but to think well what would work for you, and why. Imagine to receive an email, how will you be handling it?

I have, for example, turned off all UI markup of the “unread” status of messages. For my workflow, it is sufficient that they are in my inbox. Having unread messages show in bold for instance, does not add any additional benefit for how I deal with them. Hence, I have turned off the visual distractions for “unread”. Similarly, I don’t want to be lured away from what I am currently doing by “you have N new messages”. I have hence turned off the regular new mail checking; new mail will be fetched when I manually trigger it only. That way, I will see new messages not when someone else thinks I should see them, but when I am ready to deal with new mail.

Bonus tip: check out Tips and Tricks for using Notmuch with Emacs. If you use Org, I suggest you take a look at ol-notmuch, which can assist you to create todos with a link to an email message. With my setup, while viewing a message, I can type C-c c r, and end up in an Org capture buffer with a new todo item that has a headline of “Email from <name>: <subject>”, and the headline is a hyperlink back to the message in notmuch. To set this up, you will need an entry in org-capture-templates similar to this:

("r" "Reply to an email" entry (file+headline "" "Tasks") "* TODO %a SCHEDULED: %t :LOGBOOK: - State \"TODO\" from \"\" %U :END: %(if (string= \"\" \"%i\") \"\" \"#+BEGIN_QUOTE %i #+END_QUOTE \")%?")
Code language: Lisp (lisp)

To complement this, and to extract the information from notmuch, you will need to define a corresponding function to be invoked when you call up the r capture template:

(defun org-notmuch-capture () "Capture notmuch mail to org mode." (interactive) (org-store-link nil) (org-capture nil "r"))
Code language: Lisp (lisp)

If you also (require 'ol-notmuch), then org-store-link will create an appropriate link, and Org will know how to call into notmuch-mode when you follow that link. Creating a todo for an email message with four keystrokes (C-c c r); that’s what I call productivity!