A brief history of email

Am I alone in thinking that open-source email technologies have completely stagnated over the last few years?  It seems like 10-year old technology is still state of the art when it comes to building your own email servers these days, and I’d love to do something  about it.

When Bytemark started back in 2002, we got a lot of geeks setting up virtual machines with us because they wanted control of their own email.  What did a geek do if (s)he didn’t self-host?  The popular free-of-charge alternatives were your employer, school, dial-up ISP, Hotmail, fastmail, Yahoo – either you got stingy mailbox allowances measured in megabytes, an uncool-looking address, lack of access through IMAP or POP, lack of access outside of your network – there was always a down side to a free mailbox.

When we were first selling VMs, there was an aspirational aspect to our marketing.  Run your own server!  Stop using crappy shared hosting!  Set your email up the way you want!  And if you were willing to put in a little effort, or simply knew what you were doing, it was quite easy to set up email on lots of domains, for lots of people, in exactly the way you wanted, using exactly the same technology that Microsoft, or Yahoo, or your ISP were using.  exim, qmail, postfix, cyrus, Squirrelmail, horde … these were the tools of the trade, and there was a lot of overlap in what was free, and what the big boys used.

Then in 2004-5 GMail crept out, and started to push people’s expectations of a free email service – a gigabyte of storage was a big deal!  Microsoft and Yahoo had to react quickly to keep their users by upping their quotas, adding better searching and slicker webmail interfaces.

But I think that’s when open source started to fall behind.  Mozilla Thunderbird – which is pretty much the pinnacle of the open-source desktop email client (along with nearly-identical Evolution & KMail) hit 1.0, but I’m struggling to think of anything it does now that it didn’t do then.  While GMail was showing that email ought to be organised around conversations, fast and arbitrary sorting, and the assurance of ever-increasing storage, free software still wants you to push your email into folders and files, and fast indexing isn’t at all easy to implement.

“Email is boring, I just want it to work”

I hear this a lot – from experienced sysadmins who know how to configure their own email systems, but don’t want to bother any more.  Now that GMail do a stonkingly good offering for free, the bar is raised and there is simply l.ess motivation for a geek to work on their own system “just because”.  It used to be that free software and a cheap host would give you a better email system than a free webmail service.  But with the web interface being more important than it used to be, this ain’t true any more.

Fashion is scarily relevant when wondering why free software developers work on the things they do.  And there’s nothing more fashionable than web development right now; it’s moving fast enough for new coders to make a dramatic mark with the right idea, the right backing, or a really smart bit of technology.  Plus giving away great applications for free isn’t as popular it was; it’s far more usual  for a developer to host it himself and sell subscriptions.

Finally, email technologies are mature, and somewhat resistant to change.  You might be able to think of a brilliant extension to SMTP or IMAP “if only everyone else would agree to it” but there’s no chance.  However good your idea, the last 10-15 years have shown that email technologies are too mature and widespread for anything new to become essential to everyone – the relatively minor additions of DomainKeys and friends from 4-5 years ago are still a long way from being universally useful.

So because of these roadblocks to innovation in free software, it’s now easier to innovate if you’re a commercial developer being paid to work on a black box – either Google on GMail, or Microsoft on Exchange.  These products have the benefits of communicating with the whole world through open standards, but being able to innovate internally, and gain competitive advantage from thsoe innovations.  The developers who might be interested in bringing free software up to scratch are too busy with their day jobs, and the groups of users who might motivate such development are happy enough with GMail, Exchange, or less functional free software.

When it comes to big email providers, I’m often reminded of f a joke about Royal Mail cutbacks from a 40-year old radio comedy.  A spokesman announces that if you’ve posted a letter with a second-class stamp, it will receive a series of humiliating frankings (“second class is lower class!” or “who’s a pauper then?”), and may not be delivered at all.  Therefore you should write and tell the receipient to go and pick it up from the sorting office.

That’s the ridiculous conclusion you’re left with when a customer complains that their outgoing  email ends up in a Hotmail spam folder, or btinternet.com drop it silently.  There’s little you can do other than point them at the over-protective recipient’s mail provider, and tell them to make a phone call instead if it was important.  Go through this enough times with the same client, and they start to think it’s your problem, not the recipient’s.  It adds up to a less-than-ideal situation for smaller mail hosts like us.

Well, you would say that

As the MD of hosting company, I want people to host their own stuff on (our) rented servers.  Free software is the largest asset any ISP can wield in getting people to rent its servers, so if that base of free software falls behind what users want, all ISP services become less valuable.

All hosting companies should be falling over ourselves to sponsor free software development so we can continue compete with the mega-hosted applications like GMail – not  stating they’re sick of handling support for their email service, and people should use GMail (in a terribly nice way of course).

(aside: Uh, Dreamhost dudes, once Google work out a way of automating away the customer support side of web hosting, as they’ve done with email, they will try their hand at it.  Right now Google only have their app engine that they know only organised developers will look at, and it’s a bit nichey.  But they’re in this for the long haul, and I’m sure somebody at the chocolate factory has done the sums on hosting millions of PHP/MySQL applications for pennies each.  Once those customers can host at Google too, the circle is complete: punters will use their Google web browser, on their Google phones, to search Google, to find sites hosted at Google, all publishing adverts by Google.  Lack of customer support won’t matter if it works)

I’m not picking on DH as hosts (or indeed free software sponsors, I’m sure they’re that too), just pointing out the danger in their “outsourcing” advice, whether it be to Google or any other large potential competitor.

What’s new

So email is something hosting companies need to wrest back before anything else gets pulled away from us, and to do that  we need free software.  Here’s the best of the new:

  • Lamson SMTP is an application development framework for email, intended to break apart the muddle of MTA configuration languages, alias files, dot files, procmail and other mess  of building an email-based application.  It may take a few years to get mail administrators to learn new tricks, but this is definitely the direction they should be going in.
  • sup is a console email client incorporating all the good ideas from GMail.  Slightly worrying in its dependence on giant local indexes and occasional crashes.  Unfortunately no integration with IMAP other than gobbling messages from an inbound store, which meant I couldn’t re-file messages in shared mailboxes, so had to leave it aside for my own use.  But the author is keen to make it over as a network service and separate client, which is absolutely the right way – good luck Mr. Morgan.
  • Archiveopteryx is a database-backed mail store which breaks incoming messages into bits for fast searching.  The design and code quality looks really high (it uses C++, PostgreSQL and huge numbers of internal tests).  Unfortunately the promise of this approach over boring old Maildirs has yet to be fulfilled, as the webmail interface is just a proof-of-concept, and stodgy folder-based IMAP is its main user interface.  However the data store is a sound  base to build something more innovative, and the company is funding itself from paying clients – the best of old and new business models I hope.
  • Roundcube is an opensource web/IMAP client with a simple design reminiscent of Apple Mail – very nicely distilled.
  • Not forgetting good ol’ Dovecot – not new, but the author did a super job of making IMAP and POP3 mail stores simpler to build, and is leading the way in supporting new IMAP features with a recent 1.2 release.

That’s it for now – when I have some time later this year I’ll try to rework our company’s own internal mail to take advantage of the best of the new, and I’d be happy to hear of any projects I might have missed.

As I’ve said previously, we’re still very happy to offer free and (if necessary) very high-powered hosting to interesting free software projects, now with the qualification especially if it involves email!  Contact me directly if you contribute to a project that would fit the bill.