A brief history of email

Am I alone in thinking that open-source email technologies have completely stagnated over the last few years?  It seems like 10-year old technology is still state of the art when it comes to building your own email servers these days, and I’d love to do something  about it.

When Bytemark started back in 2002, we got a lot of geeks setting up virtual machines with us because they wanted control of their own email.  What did a geek do if (s)he didn’t self-host?  The popular free-of-charge alternatives were your employer, school, dial-up ISP, Hotmail, fastmail, Yahoo – either you got stingy mailbox allowances measured in megabytes, an uncool-looking address, lack of access through IMAP or POP, lack of access outside of your network – there was always a down side to a free mailbox.

When we were first selling VMs, there was an aspirational aspect to our marketing.  Run your own server!  Stop using crappy shared hosting!  Set your email up the way you want!  And if you were willing to put in a little effort, or simply knew what you were doing, it was quite easy to set up email on lots of domains, for lots of people, in exactly the way you wanted, using exactly the same technology that Microsoft, or Yahoo, or your ISP were using.  exim, qmail, postfix, cyrus, Squirrelmail, horde … these were the tools of the trade, and there was a lot of overlap in what was free, and what the big boys used.

Then in 2004-5 GMail crept out, and started to push people’s expectations of a free email service – a gigabyte of storage was a big deal!  Microsoft and Yahoo had to react quickly to keep their users by upping their quotas, adding better searching and slicker webmail interfaces.

But I think that’s when open source started to fall behind.  Mozilla Thunderbird – which is pretty much the pinnacle of the open-source desktop email client (along with nearly-identical Evolution & KMail) hit 1.0, but I’m struggling to think of anything it does now that it didn’t do then.  While GMail was showing that email ought to be organised around conversations, fast and arbitrary sorting, and the assurance of ever-increasing storage, free software still wants you to push your email into folders and files, and fast indexing isn’t at all easy to implement.

“Email is boring, I just want it to work”

I hear this a lot – from experienced sysadmins who know how to configure their own email systems, but don’t want to bother any more.  Now that GMail do a stonkingly good offering for free, the bar is raised and there is simply l.ess motivation for a geek to work on their own system “just because”.  It used to be that free software and a cheap host would give you a better email system than a free webmail service.  But with the web interface being more important than it used to be, this ain’t true any more.

Fashion is scarily relevant when wondering why free software developers work on the things they do.  And there’s nothing more fashionable than web development right now; it’s moving fast enough for new coders to make a dramatic mark with the right idea, the right backing, or a really smart bit of technology.  Plus giving away great applications for free isn’t as popular it was; it’s far more usual  for a developer to host it himself and sell subscriptions.

Finally, email technologies are mature, and somewhat resistant to change.  You might be able to think of a brilliant extension to SMTP or IMAP “if only everyone else would agree to it” but there’s no chance.  However good your idea, the last 10-15 years have shown that email technologies are too mature and widespread for anything new to become essential to everyone – the relatively minor additions of DomainKeys and friends from 4-5 years ago are still a long way from being universally useful.

So because of these roadblocks to innovation in free software, it’s now easier to innovate if you’re a commercial developer being paid to work on a black box – either Google on GMail, or Microsoft on Exchange.  These products have the benefits of communicating with the whole world through open standards, but being able to innovate internally, and gain competitive advantage from thsoe innovations.  The developers who might be interested in bringing free software up to scratch are too busy with their day jobs, and the groups of users who might motivate such development are happy enough with GMail, Exchange, or less functional free software.

When it comes to big email providers, I’m often reminded of f a joke about Royal Mail cutbacks from a 40-year old radio comedy.  A spokesman announces that if you’ve posted a letter with a second-class stamp, it will receive a series of humiliating frankings (“second class is lower class!” or “who’s a pauper then?”), and may not be delivered at all.  Therefore you should write and tell the receipient to go and pick it up from the sorting office.

That’s the ridiculous conclusion you’re left with when a customer complains that their outgoing  email ends up in a Hotmail spam folder, or btinternet.com drop it silently.  There’s little you can do other than point them at the over-protective recipient’s mail provider, and tell them to make a phone call instead if it was important.  Go through this enough times with the same client, and they start to think it’s your problem, not the recipient’s.  It adds up to a less-than-ideal situation for smaller mail hosts like us.

Well, you would say that

As the MD of hosting company, I want people to host their own stuff on (our) rented servers.  Free software is the largest asset any ISP can wield in getting people to rent its servers, so if that base of free software falls behind what users want, all ISP services become less valuable.

All hosting companies should be falling over ourselves to sponsor free software development so we can continue compete with the mega-hosted applications like GMail – not  stating they’re sick of handling support for their email service, and people should use GMail (in a terribly nice way of course).

(aside: Uh, Dreamhost dudes, once Google work out a way of automating away the customer support side of web hosting, as they’ve done with email, they will try their hand at it.  Right now Google only have their app engine that they know only organised developers will look at, and it’s a bit nichey.  But they’re in this for the long haul, and I’m sure somebody at the chocolate factory has done the sums on hosting millions of PHP/MySQL applications for pennies each.  Once those customers can host at Google too, the circle is complete: punters will use their Google web browser, on their Google phones, to search Google, to find sites hosted at Google, all publishing adverts by Google.  Lack of customer support won’t matter if it works)

I’m not picking on DH as hosts (or indeed free software sponsors, I’m sure they’re that too), just pointing out the danger in their “outsourcing” advice, whether it be to Google or any other large potential competitor.

What’s new

So email is something hosting companies need to wrest back before anything else gets pulled away from us, and to do that  we need free software.  Here’s the best of the new:

  • Lamson SMTP is an application development framework for email, intended to break apart the muddle of MTA configuration languages, alias files, dot files, procmail and other mess  of building an email-based application.  It may take a few years to get mail administrators to learn new tricks, but this is definitely the direction they should be going in.
  • sup is a console email client incorporating all the good ideas from GMail.  Slightly worrying in its dependence on giant local indexes and occasional crashes.  Unfortunately no integration with IMAP other than gobbling messages from an inbound store, which meant I couldn’t re-file messages in shared mailboxes, so had to leave it aside for my own use.  But the author is keen to make it over as a network service and separate client, which is absolutely the right way – good luck Mr. Morgan.
  • Archiveopteryx is a database-backed mail store which breaks incoming messages into bits for fast searching.  The design and code quality looks really high (it uses C++, PostgreSQL and huge numbers of internal tests).  Unfortunately the promise of this approach over boring old Maildirs has yet to be fulfilled, as the webmail interface is just a proof-of-concept, and stodgy folder-based IMAP is its main user interface.  However the data store is a sound  base to build something more innovative, and the company is funding itself from paying clients – the best of old and new business models I hope.
  • Roundcube is an opensource web/IMAP client with a simple design reminiscent of Apple Mail – very nicely distilled.
  • Not forgetting good ol’ Dovecot – not new, but the author did a super job of making IMAP and POP3 mail stores simpler to build, and is leading the way in supporting new IMAP features with a recent 1.2 release.

That’s it for now – when I have some time later this year I’ll try to rework our company’s own internal mail to take advantage of the best of the new, and I’d be happy to hear of any projects I might have missed.

As I’ve said previously, we’re still very happy to offer free and (if necessary) very high-powered hosting to interesting free software projects, now with the qualification especially if it involves email!  Contact me directly if you contribute to a project that would fit the bill.

6 thoughts on “A brief history of email

  1. Great article Matthew,

    I am one of those who wanted customisation of their mail handling, my family has followed me and is using my host because the majority of mail hosting companies couldn’t account for connection oddities. I use MailWatch to track MailScanner filtering quickly and tweaking the filtering has been very important. I have one domain of mine hosted with Google Apps just because I can and it is good but perhaps it lacks the granularity for heavy business use?

    If a mail hosting company was able to provide the same granularity of control as MailScanner + Postfix + PostfixAdmin + Mailwatch + MySQL + Dovecot1.2 then I think it would give businesses a case to drop their internally hosted servers. Adding RequestTracker into that mix would just be the final leap to give them a feature not available with most off-the-shelf systems.

    On the mail client side what I have come to realise is that I can’t navigate my email quickly enough. I have many GB of email and at work I have thousands of emails waiting to be sorted into the myriad of sub-folders. I think the answer is tagging and bayes suggestions but I think tagging hasn’t truly been leveraged to it’s maximum. I think it should be possible to browse tags with depth to narrow down the emails. Thus the initial view should have every mail, there should be a tree list of every tag; then each tag should have every tag which is shared with the previous tag. Thus a matrix hierarchy of tags can evolve and with Bayes suggestions mails can be tagged as they arrive (I already use the Bayes sort add-in for Thunderbird). In this way the mass of mail can be filtered down to the target quickly and more importantly mail can be in many places! An email might be to do with marketing but also to do with a client.

    Perhaps I need to blog this….

    Anyway, good posting!

  2. I’m not convinced that the reason people use gmail is the interface. Indeed, if it wasn’t so damned slow I’d use it through the thunderbird interface by preference.

    It’s advantages are
    1) Free
    2) Big storage
    3) Great search

    The interface is pretty horrible IMHO and the threading into conversations has no real difference that I can see to thunder bird’s threading by subject line.

    It’s only the search that thunderbird lacks and that’s a server side not client side issue. Gmail does have the big advantage that the web client and the server are closely integrated but they don’t do too much with it.

  3. Speaking personally, the reason I host our email with Google Apps nowadays is nothing to do with features, or Bytemark support (which is awesome as always) or anything else – it’s that it’s close to 100% reliable.

    Maybe if you’re an expert sysadmin you can be confident you can keep your own mail server running smoothly, but for me, there was always the problem that every so often, I would embark on another trip in the “special” yellow half-trained sysadmin bus, do something hideous to my server, and have mail go down whilst I attempted to batter it back to life using half-understood config file commands and voodoo. (I think you may have been on the other end of tech support for that a couple of times…)

    TBH, if Bytemark offered a seperate mail server that Just Worked (TM) and stored the email in some close-to-impossible-to-lose format, I’d probably abandon Apps like a shot. But hosting email on my own box means that the buck stops with me when it comes to keeping it working, and given that I’ve got habits like occasionally doing development work on a live server (“It’ll be fine, it’s only a couple of lines of – oh, dear.”), I like the control of my vital services to be with someone more clueful than me!

    (Oh, Google spam filtering is also superb).

  4. Two points

    1) As a Fedora 11 desktop user, it looks to me as Thunderbird 3.0 beta is broken and is likely to be launched broken. Why? Because they are trying ti implement a structured LDAP address database instead of the old text based fudge of mork. Except mork works well and there is no standard data definition for the LDAP solution to work. So why do I care. Because Thundwerbird 3.0 beta will only find the primary email when typing in the To/CC/BCC fields. That means that after sticking with it for 6 weeks I’ve downgraded to Thunderbird 2.0 and will have to select a different client. I tried Zimbra but being web based its just too slow….

    2) This brings me to my next point. As an independent IT consultant more or my time is spent talking to people than at the command line. For me the holy grail of productivity for the ordinary business man/woman is contact synchronisation. This is generally a painful process, but once established is a god send. I think this synchronisation is the reason Outlook is popular, everything will sync to outlook, at least now everything syncs to google as well. Having synced Thunderbird and Blackberry and rationalised my contactbook is a big thing for me. It helps me at least 6 hours a day. And how did I do it, by creating a igoogle account which means google is used to store my contacts and calendar… and that is why people move to gmail. standalone communications clients are out.

    Mobile phones like blackberry or palm pre, will sync to facebook and gmail and oher apps and produce a fully synchronised and rationalised address book and that’s priceless

  5. I started off hosting my own email for the reasons you mentioned – i.e. the fun of running my own server and to develop my sysadmin skills in case I ever needed to run these sort of services for work.

    I could switch to Gmail and let Google do all of this now, but I don’t trust a US company which could easily start to (if they don’t already) mine my emails, move them out of the EU (with its strong data protection) etc. The reason I host my own email solution is because I don’t trust anyone else not to look at my emails, either automatically or manually.

    I’ve also had silly limits attached by ISPs in the past (e.g. maximum of 50 recipients on an email), which is why I run my own SMTP server too.

  6. Bizzarely, I’ve come across this post after just today deciding to bother to set up my VM to handle mail properly. Exim and Dovecot being the core components. I’ve been to lazy to do it in the past because I admin my own mail setup for work amongst many other things, and so have always used my work email for personal stuff. I’m lucky enough that I have a great deal of autonomy and control over my work set-up that nobody questions that, but it’s always been a nagging thing – some of the opinions I hold, that I use my work email for, might not exactly fit in with the ethos of the large public sector organisation I work for. One day, I reason, that may well come back to bite me. So I decided to get off my backside and set up my VM for handling e-mail for real. I just don’t get on with gmail.

    I don’t agree that Roundcube is that good – it’s pretty, but it’s feature light. But you’re right, there isn’t a lot going out there in the Open Source world for this. Squirrelmail is powerful and very robust, but very clunky. Open Exchange, which I’m investigating for work, is good, but very heavy for a personal email account. Zimbra, the same.

    I’ll be using Mulberry for IMAP access to my VM, and I guess, for webmail, it’ll have to be good old squirrelmail. Sigh.