Posted by Matthew Bloch
Sun, 07 Feb 2010 11:14:00 GMT
All internet providers in the UK would probably agree with me when I say
that Dr. Laurence Godfrey is ... a man. With a beard and glasses. The
legal precedent of his 10-year old court
victory against Demon Internet is still the most notable "chilling
effect" on free free speech in this country, and one which causes us service
providers to turn censor on a regular basis.
What his case established was that Internet Service Providers can be
held jointly responsible with their customers for any libellous statement
posted on their customers' web sites. When a solicitor is seeking
compensation for libel on the server hosted in the UK, he can bypass suing
the author of the libel, and instead threaten the owner of the computer on
which the libel is hosted, usually a more risk-averse target.
What happens with us typically is:
- a solicitor sends Bytemark a letter (and email, and fax, and
telegram, and carrier pigeon usually) stating that a statement at a
particular URL is libellous to their client, Mr Bastard. The letter usually
states that we are "on notice" of a libellous statement, and that we must
remove it or face the consequences;
- we find out which of our customers' servers is hosting the statement
offending Mr. Bastard, and contact them. We ask them to either take it
down immediately, or indemnify us against legal action with an initial deposit
of £20,000.
- we use our support system to put the customer in touch with the
solicitors, sometimes with Mr. Bastard too, and words are exchanged.
The offending page usually ends up being taken down, often with reference to Arkell v Pressdram.
Even where our customer wanted to face the complainant in
court, the precedent acts as a libel costs "amplifier" - for us to not
take down their site immediately, we need indemnifying against the costs of
our own defence, and at a much higher price than the customer might choose
to spend on their own. So rather than go to court, any statement that has
the sniff of libel about it can be taken down, almost immediately, without
any questions. That is very definitely a "chilling effect".
I don't believe that people should have a free pass for internet libel.
An malicious comment, however obscurely published, can be picked up by a
search engine as a match for someone's name, and be as devastating a
publication as the front page of The Times. And I would not object to
complying with a court order to reveal the identify of one of our customers,
in order that legal action could be brought against him directly.
But more recently, threats of libel actions are getting sloppier, and
after some advice, we had to draw a line in the sand. At Bytemark, we have
a policy of asking for the URL of a libellous statement before we
demand action from a customer. During a discussion in 2008, when a
complainant repeatedly refused to supply URLs of libellous statements
against him on a particular message board, he asked, exasperated:
Without our checking every hour, how do we know what has been
posted? The moment a comment is posted it immediately causes damage to our
companies professional reputation. We do not have the resources to put a
full time member of staff at a desk to check the hundreds of posts every
hour, and why should we?
And in a later email:
...it is not our responsibility to check this site each and every day and read
every single post to check for defamatory content.
To which I innocently answered: it's your reputation, not ours. If the
libels are not so serious that you can't identify them all, how do you
expect us to. Are they really so serious and urgent if you can't?
While that complaint didn't even to go a solicitor, we were recently
threatened with court in a similar situation. The solicitor gave us 3 URLs
by way of example, but when these were removed within 2 hours, advised us
and our customer that:
[it is] our clients' position is that it is now your obligation
to ensure that no similar posts remain or are allowed to be published in the
future. Accordingly, it may be easiest for you to either remove the
discussion threads in their entirety (and any others which come to be posted
in the future) and/or block access to the posters responsible for the
defamatory comments.
So, wait... now that you have told our customer that there might be
someone libelling you in future, they must make sure it never happens? And
if they don't, the ISP will have to defend themselves as if we'd written it
ourselves?
The message seemed clear: drop this customer, or we'll make trouble
in an area of law notoriously favourable to claimants. We were given a
deadline for court action, and after years of armchair defence, engaged
libel experts Carter-Ruck to find
legal defence in my indignation.
The bad news: internet hosting providers will likely remain
"publishers" in common
law thanks to Godfrey's precedent.
But Carter-Ruck found two strong defences from this shaky start.
Firstly,
Section 1, Clause 1 of the Defamation Act 1996 predates legislators
consideration of the internet. It states that a person has a defence
against defamation under certain circumstances, which hinge on whether the
person is the "publisher". But it goes on to define publisher as "a
commercial publisher, that is, a person whose business is issuing material
to the public". From that definition, publishing is not our business,
though I'm not clear how this interacts with the Common Law precedent.
Secondly, section 19 of the Electronic Commerce
Regulations 2002 is a lot more specific, and states that (heavily
elided, but accurate):
Where ... [hosting] ... is provided ... the service provider ...
shall not be liable for damages ... where the service provider does not
have actual knowledge of unlawful activity or information and ... upon
obtaining such knowledge or awareness, acts expeditiously to remove ... the
information.
That is a blanket exemption for us hosts service against any liability
for content we host, providing we didn't know about it first. And it has
been validated by Karim v Newsquest 2009 when
the defendant took down allegedly-libellous user comments the day they received
notice of their presence.
But my main worry was the vague notification, the expectation that
we (or our customer) should have to monitor and quickly remove future posts
that might be libellous. Again, we are advised we're on strong ground in
demanding URLs before we can take action. A detail of Metropolitan
International Schools v Google 2009 is interesting. This plaintiff
demanded that Google remove not just current, but future libellous extracts
contained in their search results. Google's barrister put forward that:
it is practically impossible, and certainly disproportionate,
to expect [Google Inc.] to embark on a wild goose chase in order to
determine where the words complained of, or some of them, might from time to
time 'pop up' on the Web.
And the judge agreed; Carter-Ruck think that this is a strong indication
that supplying URLs is the minimum amount of information needed for a
service provider to take action.
Joined up, those points ought to a strong defence against hand-waving
defamees; people who expect us to monitor their reputation for them without
providing specifics. While it's untested, the narrowest possible precedent I
could see coming from a libel victory is a pretty horrific one for free
speech.
Rather than just take down individual statements, ISPs and message board
admins who are "on notice" would be forced to implement moderation, and
a blanket ban on mentions of Mr. Bastard, and solicitors' notices would have
an enormous cost to the recipient, even without any court action. As ISPs would
remain jointly liable, they too would have to ensure that moderation of
discussions is effective, and would have to oversee this process. The
additional costs would be passed on to the owner of the board. Of course the
upshot would be that unmoderated message boards would likely close rather
than bear the bureaucracy imposed by a simple solicitors' letter - a letter
which would have to specify no more than the allegation of libel.
I don't see how any judge could bring this on the ISP industry, but then no ISP
expected the Godfrey precedent in 2000 either. And we won't be sure until
somebody risks half a million pounds on finding out.
3 comments
Posted by Matthew Bloch
Sun, 01 Nov 2009 12:02:00 GMT
There comes a time in every sufficiently ambitious Ruby programmer's life that one will butt up against gems, Ruby's own packaging system. Outside of the cosy confines of one's laptop, on servers and embedded systems, the system contains design mistakes that make them unmanageable. Here's how we cope at Bytemark. But how did it get here?
The world of programmers
So computer languages have always had some idea of a library, a set of functions, classes and other extensions that make the language more useful At the earliest and most basic, #include tells a C compiler that you wish to load a library of functions for input and output called stdio - printing lines of text, opening files and so on.
In Ruby, there are two sets of libraries which have shipped with the language. These are the core and standard libraries, and Ruby programmers know that they're always available. If you want to program with network sockets, you say require 'socket' and you know that you can use TCPServer and friends to start writing high-level network programs. In common with other scripting languages, if a programmer wanted to use another library on his system, he would install its files into /usr/local/lib/ruby, and could then 'require' those files in any program.
That's the same simple method that Perl, Python and many other scripting languages have used, and it's easy to describe and understand. It's also easy for Operating System vendors and Linux distributions to package.
The world of system administrators
Since the early 90s, the Debian project defined the state of the art in large-scale, long-term system administration. Before Debian (and yum, and the other systems it inspired), admins either routinely did their own software builds, or ran crude packaging systems which were no more complex than a tar file. Debian rejected the idea that system administrators needed to constantly reinstall to "blow away the cobwebs", and hammered out packages and a distribution structure that has allowed admins to avoid reinstalling some individual systems for 10+ years.
Debian have packages for everything from word processors to hardware drivers. An administrator can define the state of a whole computer system just through listing what packages are installed, and a handful of configuration files - no clicking through installers, or manual stages. Debian's thousands of volunteers have built and tested stable packages based on the creme of free software. Users of a Debian-based OS can have their pick of the best of it, and know it will work together. So in 2009 you can say "I want a system with openoffice installed", and Debian's package management system will perform a complex set of resolutions to find out what libraries openoffice depends on. It will install those in the correct order, right down to the operating system kernel and graphics drivers, so that after downloading, unpacking and running setup scripts, the user goes from having an empty system to a working word processor.
This same process works the same on a hundred Debian servers, even when those servers have very different hardware underneath. This is still an awesome achievement, and Debian are still leading the way in defining how software installation on computers should work.
Crucially, Debian package programming libraries for lots of languages, and authors writing for Debian systems can rely on scores of them being available in a predictable way. But right now, in Debian and most other Linux distributions, there are some major gaps in coverage for widely-used Ruby libraries, which means dependent applications can't be packaged. How did this situation come about?
Blame the web app startups!
Well, at some point in the last 10 years, the rise of Google caused all programmers everywhere to lose their minds. They decided that users shouldn't install their own programs on their own computers, but instead should trust installation and data storage to the programmer exclusively. Sophisticated, fast, reliable programs started being replaced by simple, slow flakey ones as a result.
Just a joke! We all love web applications really.
No - more reasonably speaking, a handful of smart programmers, all at once, had found a way out of the dreary mire of web applications in 2004-5, and a frontier was formed with Rails and various other programming libraries leading the charge.
On this frontier, distributing and installing applications doesn't matter. Paul Graham was probably the first to say that when you're selling a web-based application, you have an advantage that you can use any language you want because you don't have to distribute it. As long as the programmer can install his program on his own little set of servers, and handle his users' data storage, nobody else ever had to see his code, what libraries he used, or how it was put together. But also it means he doesn't have to conform to traditional system administration practices either, and I think this is the root of the problem.
Within the Ruby community, packaging for widespread distribution has been an afterthought to pushing forward the frontier of what the language can do. Distributable Ruby applications are thin on the ground (because right now, most people writing web apps want to sell access to them, rather than distribute them for free), but there are lots of great libraries out there.
Problem solved?
But instead of settling on a simple method of managing local installations like Perl folk did with CPAN, Rubyists settled on Rubygems.
In theory, a system administrator types gem install hoopystuff to install a gem called 'hoopystuff'. The gem program goes away, finds the hoopystuff gem, and installs it on his system. If there's anything that needs compiling, gem compiles it and puts it in the right place. Then any program that wants hoopystuff can start to use it. There is a master list of gems maintained, a network of mirrors and a signing mechanism, a lot like other packaging systems. There is a some wheel re-inventing going on, but it means Ruby programmers don't have to worry about supporting every possible system, which seems like a win. It even allows programmers to install libraries on a shared system, without needing full privileges over it, which is a useful feature.
But the biggest mistake made in Gems was to add to the language. In Java, or C, or Python, or any other language, to include a library, you do the same thing, regardless of who installed the library, or where. But in Ruby, a gem command was added to the language. And you need the rubygems library included first in order to use that command. So if a programmer wants to use the hoopystuff library he's installed as a gem, the obvious doesn't work any more:
require 'hoopystuff'
Instead he has to do:
require 'rubygems'
gem 'hoopystuff'
require 'hoopystuff'
But if he has installed hoopystuff through his system distribution, rather than Rubygems, this will fail! So a thorough way of including the library has to be a full six lines of code:
begin
require 'rubygems'
gem 'hoopystuff'
rescue LoadError => no_gems_error
# no Rubygems library installed, or no 'hoopystuff' gem
end
# either way we need to do this, if this fails the library definitely isn't here
require 'hoopystuff'
This is now the only portable way of asking for a library in Ruby - hardly the principle of least surprise. And through the fog, many library authors assume that the 'gem' command is available where it's not. So packaging almost any contemporary Ruby application involves altering its code. The Debian packagers are asking Ruby authors very nicely to bear this in mind, but to little effect.
To add to the confusion, the gem command also allows multiple versions of the same package on a single system. This means that instead of simply asking for gem 'hoopystuff', the programmer can ask for 'hoopystuff' version 1.23. Unfortunately another part of the same program can ask for 'hoopystuff' version 1.5, Ruby will die at that point, saying that it can't load two versions of the same gem in the same program. I don't think I'm going out on much of a limb when I say nobody needs this feature and if you think you do, you're not clever enough to use it propely. I have "fixed" plenty of conflicting gem invocations in live apps where both pieces of code are demanding different versions of a gem, where the same one works fine.
Why can't we convert?
It's this last requirement to allow multiple versions that makes gems fundamentally incompatible with every other package management system - Debian, Redhat, SuSE ... all of them allow one version to be installed on a system. So it's impossible to do a clean mapping of gems onto .debs or .rpms or any other mature packaging structure, because once you add enough applications into the mix, they can all be demanding different versions of the same Gem, and these demands are expected to be met.
At the start of 2009, the talented guys at Phusion made an attempt called DebGem where they took every version of every gem they could find, and made a Debian package out of it, baking the Gem version number into the name of the package. It looked weird, and appeared to work. But the project has been silent since April, and the Linux distributions they supported are fading into irrelevance. My guess is they couldn't stomach the amount of manual work needed to tweak every Ruby programmer's misunderstandings about Gems. (but Phusion dudes, if it was just the expensive hosting, Bytemark will still donate as many mirrors & build hosts as you need).
In contrast, the Perl folk have a simple site that routinely converts Perl packages into compatible debs, and allows them to be installed, and integrated into official distributions easily. All because the packaging system is simpler.
So what are the options left for a Ruby programmer who wants to ship portable software to a wide variety of users?
Option 1: A traditional build system
What I'm doing with a couple of our projects is to forget that Gems ever existed. Every shared library is frozen and checked into the project under an external directory where they stay unless I need newer versions. Then I have a Rakefile which runs two jobs more usually seen in compiled programs - a build, and an install.
build compiles any native extensions that the program needs, and install copies the whole program, and all its built dependencies into /usr/lib/myprogram. Finally it adds the binaries that I want to run to /usr/bin/myprogram but these are just stubs which set up the load paths to my "pure" Ruby environment.
In addition, because I don't want to have to fix the source code of all the gems I'm using (usually around 10-15), these loader stubs actually load a fake Rubygems library. The library just ensures that the gem command does nothing, and stops me having to worry about changes to the libraries' code.
The down sides are pretty well understood - my 2000 lines of code which would have been a tiny, architecture-independent package, has to be built a one large package, once for each architecture (we need two at Bytemark). If I wanted to distribute it any further I would probably want a wide variety more packages, for a few more distributions. But I have all the usual down sides of managing my own packages - checking for security bugs and doing my own code updates, much larger & slower-to-install packages, and so on. But the major up side: I can use Debian and apt-get to install and maintain it reliably on hundreds of servers.
Option 2: Repackage each library
For the long term, Patrick is working out how to gently modify the source of around 50 popular Ruby libraries so that they form normal Debian packages without needing Rubygems installed. That will help me kick all this duplicated library code out of individual projects and back into packages where they belong.
This is semi-automated but still has many manual elements that Pat is working through. The Debgems folk had given up at doing this in the general case, so we're focussing on just the set of Gems we need to run all our code, and trying to integrate our work with Debian's. I notice also an Ubuntu team has also got a bunch of reasonably new packages into Ubuntu's universe package list - I hope this will make it back into Debian, or that we can use them for Bytemark's systems.
I can even see scope for going all the way back to the start, and making a small fork of the core Ruby language & interpreter to fix the packaging problems - that is an extreme possibility, but with at least two new Ruby implementations becoming more relevant lately, it might not be the MRI (Matz's Ruby Implementation, the original one) that stays relevant in the long run, and leadership on packaging could easily change.
But right now this approach is going against the grain of what almost all Ruby authors are doing; surgery is needed to library authors' code, which is the cause of all this rot. But unfortunately that's the price that we have to pay to go back to a simpler, working library system.
Option 3: Wait... about five years?
I think that the worst mistake of Rubygems is being undone - from the next major Ruby version, the gem command is no longer necessary in the common case. So if you require 'hoopystuff' in Ruby 1.9, the require statement will implicitly look for a gem called 'hoopystuff'. This might seem like a trivial timesaver for library and application authors, after all, it was only six lines I was complaining about. But but but... it means that the sanctioned way of including libraries is back to how it used to be, just one require statement, one namespace.
That mean a lot less work in repackaging gems, but only when library authors have got the message and Ruby 1.8 installations cease to be relevant - so that's where my five year estimate comes from.
Start now, avoid the gem traps
If you're a Ruby author who cares about distributing your software to more than just other programmers' laptops, you only need to take some simple action with your existing Gems to make them compatible that the Debian folk wrote years ago. I'd add the following to these tips though:
- don't use the gem command in your main code at all, use a loader program that pulls it in if you need it. In almost all cases it is going away in 1.9, and good riddance;
- if you provide a Ruby library called foobar, make sure your gem is also called foobar, and preferably only provides a single module called Foobar;
- don't use capital letters in your gem name - amazingly there are already some gems in the namespace that differ only by case!
And finally, try building a native package for your favourite system! Debian at least is quite easy. It will take you a few more hours, but your library will simple be easier to manage for system administrators when you're done. Let's not wait years for these design mistakes to atrophy away - fix your libraries and help make Ruby a first-class component of every OS.
10 comments
Posted by Matthew Bloch
Mon, 12 Oct 2009 08:56:00 GMT
1) LUGRadio Live - it's back from the dead (though wasn't down for long). On 5pm on October 24th, the four gents from Wolverhampton will talk wittily about open source, hacker culture and further their own bizarre cult of personality. In the run up to the main event, they have organised a huge number talks from some very interesting British hackers - including Gervaise Markham of the Mozilla Project, Matthew Somerville from MySociety and Craig Rothwell, an insane genius who has designed and built his own handheld gaming unit. I'll be attending all day
2) coderack - from one of our customers Lunar Logic Polska, a competition for Ruby developers to help further the Rack framework by writing some interesting and useful plugins. I don't think there's an ungeeky way to sell that so I won't even try. Rack was a very sensible suggestion, two years ago, on how to plug web servers into Ruby programs in a very general way. It meant that people who wrote web servers, and people who wrote fancy web frameworks like Rails, could get along in a common language. In the world of Ruby programmers, where 20 minutes ago is ancient history, a technology that has been going after two years looks like it's here to stay. So the competition is to try to further Rack's usefulness by building plugins of the kind that you might already find in Apache - logging, authentication, redirection tricks, that kind of thing. Bytemark will definitely be benefitting from the results of this competition, so we're donating a year of very fast, free dedicated hosting (a Phenom 9750 X4, 8GB RAM, 1TB storage) for the enterprising Ruby programmer who comes up with the best plugin.
no comments
Posted by Matthew Bloch
Wed, 26 Aug 2009 15:06:00 GMT
This is expanded from a post to a mailing list in response to the recently announced government U-turn on broadband policy. After having dinner with a movie studio boss (or maybe also listening to Andrew Lloyd-Webber standing up for his cash-strapped friends) Peter Mandelson is now stating that "repeat offenders" who share films and music through their broadband internet connections will have their connections cut off.
I don't understand why the music & film industry gets to suggest an extrajudicial process to fight their battles, and why any politician would taking them seriously (unless they've just had a smashing lunch with a lobbyist). Kangaroo justice is easy, here are some more ideas!
- People who run out of restaurants without paying their bill should have to pay double at supermarkets for a month?
- Accused shoplifters should be excluded from parking in the city centre on a weekend?
- Marijuana growers have to run their house on a single 13A fuse?
You can make up all kinds of "short sharp shock" punishments that sound fair to a medieval baron, but they amount to the same thing: meddling with people's private contractual arrangements without due legal process.
The broadband cut-off proposal is not only unfair, it's technically impossible. PC are not yet closed systems which can tell copyright files from free, despite a spirited but failed effort from Microsoft. So if your computer can't tell which files are copyrighted, mine can't either, nor can those at a media company. To find out whether a file is copyright, you need to download it, then listen to or watch it. But instead the evidence that is routinely presented to us is in the form of an infringement notice from a subcontractor of a movie/music firm, usually demanding that we immediately cease service to a particular customer, but threatening no specific action. We used to pass them on as a bit of curio, now they come so frequently that we bin them.
Their implication is that of course you understand this is copyright material, of course we are trustworthy to impart this to you, and therefore of course you will accede to our demands. Well, no, no and therefore - no. Keeping up with movie releases is not my strong point, so a file called "District 9" could be someone's thesis on town planning, or a user mod for a video game, or just about anything other than 90 minutes of copyrighted hokey sci-fi. No I won't download it to check, it's neither instant nor obvious. And the movie and music industries have been suing both children and dead people's estates in years of litigation, so should I take these notices on trust? Doesn't seem likely.
If ISPs had all agreed that these infringement notices were terribly fair, that's one thing. But they are universally lashing out at Mandelson's proposal. The government can no more force ISPs to co-operate in abusing their own customers than force Lord Lloyd-Webber to stage Pee! The Musical!, the chorus prancing through the stalls urinating on the audience (and I have sat through By Jeeves, so I wouldn't rule that concept out as an improvement, L-W).
I simply don't believe there is a problem - true artists will always create, talent can always make a living, and their public will always find a way of being cheap. If the middle men can't hack being in the middle without resorting to legislating their right to a business model, they need to make a new one. I'm not interested in helping them out.
2 comments
Posted by Matthew Bloch
Wed, 29 Jul 2009 12:47:00 GMT
Not like this, but it's a start (spotted by Nick this morning at York railway station):
I checked, the server is working :-) So I'm sure this screen will look just fabulous when its owner plugs in the internet cord. By the by, that's the first display-fail I've seen showing a Firefox error page rather than an Internet Explorer one. We salute our anonymous client for the free advertising.
1 comment
Posted by Matthew Bloch
Thu, 09 Jul 2009 18:43:00 GMT
Phil Driscoll from the West Yorkshire Linux User Group invited me to do a talk at their monthly meeting, and I agreed! I'm still scribbling out what I'll speak about, but it should be a practical guide to virtualisation, focussing on KVM and explaining away the horrid networking details that confuse newcomers. I'm pretty sure it's an open house if any Bytemark customers wanted to come along and listen.
no comments
Posted by Matthew Bloch
Sun, 05 Jul 2009 21:57:00 GMT
Am I alone in thinking that open-source email technologies have completely stagnated over the last few years? It seems like 10-year old technology is still state of the art when it comes to building your own email servers these days, and I'd love to do something about it.
When Bytemark started back in 2002, we got a lot of geeks setting up virtual machines with us because they wanted control of their own email. What did a geek do if (s)he didn't self-host? The popular free-of-charge alternatives were your employer, school, dial-up ISP, Hotmail, fastmail, Yahoo - either you got stingy mailbox allowances measured in megabytes, an uncool-looking address, lack of access through IMAP or POP, lack of access outside of your network - there was always a down side to a free mailbox.
When we were first selling VMs, there was an aspirational aspect to our marketing. Run your own server! Stop using crappy shared hosting! Set your email up the way you want! And if you were willing to put in a little effort, or simply knew what you were doing, it was quite easy to set up email on lots of domains, for lots of people, in exactly the way you wanted, using exactly the same technology that Microsoft, or Yahoo, or your ISP were using. exim, qmail, postfix, cyrus, Squirrelmail, horde ... these were the tools of the trade, and there was a lot of overlap in what was free, and what the big boys used.
Then in 2004-5 GMail crept out, and started to push people's expectations of a free email service - a gigabyte of storage was a big deal! Microsoft and Yahoo had to react quickly to keep their users by upping their quotas, adding better searching and slicker webmail interfaces.
But I think that's when open source started to fall behind. Mozilla Thunderbird - which is pretty much the pinnacle of the open-source desktop email client (along with nearly-identical Evolution & KMail) hit 1.0, but I'm struggling to think of anything it does now that it didn't do then. While GMail was showing that email ought to be organised around conversations, fast and arbitrary sorting, and the assurance of ever-increasing storage, free software still wants you to push your email into folders and files, and fast indexing isn't at all easy to implement.
"Email is boring, I just want it to work"
I hear this a lot - from experienced sysadmins who know how to configure their own email systems, but don't want to bother any more. Now that GMail do a stonkingly good offering for free, the bar is raised and there is simply l.ess motivation for a geek to work on their own system "just because". It used to be that free software and a cheap host would give you a better email system than a free webmail service. But with the web interface being more important than it used to be, this ain't true any more.
Fashion is scarily relevant when wondering why free software developers work on the things they do. And there's nothing more fashionable than web development right now; it's moving fast enough for new coders to make a dramatic mark with the right idea, the right backing, or a really smart bit of technology. Plus giving away great applications for free isn't as popular it was; it's far more usual for a developer to host it himself and sell subscriptions.
Finally, email technologies are mature, and somewhat resistant to change. You might be able to think of a brilliant extension to SMTP or IMAP "if only everyone else would agree to it" but there's no chance. However good your idea, the last 10-15 years have shown that email technologies are too mature and widespread for anything new to become essential to everyone - the relatively minor additions of DomainKeys and friends from 4-5 years ago are still a long way from being universally useful.
So because of these roadblocks to innovation in free software, it's now easier to innovate if you're a commercial developer being paid to work on a black box - either Google on GMail, or Microsoft on Exchange. These products have the benefits of communicating with the whole world through open standards, but being able to innovate internally, and gain competitive advantage from thsoe innovations. The developers who might be interested in bringing free software up to scratch are too busy with their day jobs, and the groups of users who might motivate such development are happy enough with GMail, Exchange, or less functional free software.
When it comes to big email providers, I'm often reminded of f a joke about Royal Mail cutbacks from a 40-year old radio comedy. A spokesman announces that if you've posted a letter with a second-class stamp, it will receive a series of humiliating frankings ("second class is lower class!" or "who's a pauper then?"), and may not be delivered at all. Therefore you should write and tell the receipient to go and pick it up from the sorting office.
That's the ridiculous conclusion you're left with when a customer complains that their outgoing email ends up in a Hotmail spam folder, or btinternet.com drop it silently. There's little you can do other than point them at the over-protective recipient's mail provider, and tell them to make a phone call instead if it was important. Go through this enough times with the same client, and they start to think it's your problem, not the recipient's. It adds up to a less-than-ideal situation for smaller mail hosts like us.
Well, you would say that
As the MD of hosting company, I want people to host their own stuff on (our) rented servers. Free software is the largest asset any ISP can wield in getting people to rent its servers, so if that base of free software falls behind what users want, all ISP services become less valuable.
All hosting companies should be falling over ourselves to sponsor free software development so we can continue compete with the mega-hosted applications like GMail - not stating they're sick of handling support for their email service, and people should use GMail (in a terribly nice way of course).
(aside: Uh, Dreamhost dudes, once Google work out a way of automating away the customer support side of web hosting, as they've done with email, they will try their hand at it. Right now Google only have their app engine that they know only organised developers will look at, and it's a bit nichey. But they're in this for the long haul, and I'm sure somebody at the chocolate factory has done the sums on hosting millions of PHP/MySQL applications for pennies each. Once those customers can host at Google too, the circle is complete: punters will use their Google web browser, on their Google phones, to search Google, to find sites hosted at Google, all publishing adverts by Google. Lack of customer support won't matter if it works)
I'm not picking on DH as hosts (or indeed free software sponsors, I'm sure they're that too), just pointing out the danger in their "outsourcing" advice, whether it be to Google or any other large potential competitor.
What's new
So email is something hosting companies need to wrest back before anything else gets pulled away from us, and to do that we need free software. Here's the best of the new:
- Lamson SMTP is an application development framework for email, intended to break apart the muddle of MTA configuration languages, alias files, dot files, procmail and other mess of building an email-based application. It may take a few years to get mail administrators to learn new tricks, but this is definitely the direction they should be going in.
- sup is a console email client incorporating all the good ideas from GMail. Slightly worrying in its dependence on giant local indexes and occasional crashes. Unfortunately no integration with IMAP other than gobbling messages from an inbound store, which meant I couldn't re-file messages in shared mailboxes, so had to leave it aside for my own use. But the author is keen to make it over as a network service and separate client, which is absolutely the right way - good luck Mr. Morgan.
- Archiveopteryx is a database-backed mail store which breaks incoming messages into bits for fast searching. The design and code quality looks really high (it uses C++, PostgreSQL and huge numbers of internal tests). Unfortunately the promise of this approach over boring old Maildirs has yet to be fulfilled, as the webmail interface is just a proof-of-concept, and stodgy folder-based IMAP is its main user interface. However the data store is a sound base to build something more innovative, and the company is funding itself from paying clients - the best of old and new business models I hope.
- Roundcube is an opensource web/IMAP client with a simple design reminiscent of Apple Mail - very nicely distilled.
- Not forgetting good ol' Dovecot - not new, but the author did a super job of making IMAP and POP3 mail stores simpler to build, and is leading the way in supporting new IMAP features with a recent 1.2 release.
That's it for now - when I have some time later this year I'll try to rework our company's own internal mail to take advantage of the best of the new, and I'd be happy to hear of any projects I might have missed.
As I've said previously, we're still very happy to offer free and (if necessary) very high-powered hosting to interesting free software projects, now with the qualification especially if it involves email! Contact me directly if you contribute to a project that would fit the bill.
6 comments
Posted by Matthew Bloch
Fri, 12 Jun 2009 08:41:00 GMT
I'm close to flicking the switch on our new phone system - is anyone was interested in the design? I'm not certain myself, which is to say I'm not a phone engineer of any kind. Last I looked, phone engineers lived in a topsy-turvy parallel universe with their own private members clubs, secret handshakes from government and cultish ideas about circuit switching. Me, I just downloaded asterisk five years ago and found a rebel band of software engineers trying to crowbar their way in.
And there is something of the crowbar about asterisk - a box of bits featuring about four different scripting languages which all boil down to something that looks like 20-year old BASIC. U-turns abound in its documentation, even the simple task of setting a variable says "Version differences: This command is not available in Asterisk 1.0.9. Use SetVar instead. As of v1.2 SetVar is deprecated and we are back to Set." Somehow this shed of a program has been running our company phones - including home workers, voicemail, smart caller ID, smooth redirections to our Manchester call centre, for the last five years, with barely any maintenance. Whatever crimes of design it might have committed, asterisk is very capable, but I'd never thought very hard about how to make it robust.
Planning for the lights to go out
My first priority after last month was to make sure that we couldn't lose touch with our customers again, and because the current asterisk system is a bit of a toy (albeit a long-lived and very reliable one) I needed to rearrange it to survive in the face of network trouble. And the design work will be very similar for our key services - our email support, web site, forums and so on.
So this is what our network looks like for disaster recovery purposes:
Right now, we've got our awesome Fisher-Price FBI phones, GXP2000s in the office, one per desk, and a couple of home workers. Every phone to connect ot the ofifce phone server, and if that's down, the line is dead. If the office ADSL is down, the line is dead, even the home workers can't talk to our customers. If the London network stops working, the line is dead.
To take these points of failure out, I can take advantage of our network: we have two pretty separate parts to our core. Our London racks are rich with connections but very expensive, so we don't host much there. Our newer Manchester space is physically larger, but without the same richness in terms of criss-crossing minor connections. The failures that we've seen have only ever affected one or the other, so I've put one new phone server in each location.
I'm also commissioning another offsite in the Netherlands. This could prove to be a rotten idea because of the latency, but we'll have to see - it's intended as an absolute last resort in the unprecendented event of both sides shutting down.
Who's talking to whom
I can install all these servers, but how do they function as one unit?
We only have one advertised phone number, and one supplier for this phone number, the shadowy folk at Magrathea (whose services are the lynchpin to most of the UK VoIP industry, as far as I can tell). Unfortunately they will only try to route to one of my servers at a time, so I have to pick one to receive our incoming calls - their racks are in London, so I'll use the London server unless anything goes wrong. When it does, I can update Magrathea and ask them to send the calls elsewhere. But they can all send their outgoing calls through Magrathea at once if necessary.
At the other end, on our desks, the GXP2000s have a very handy function allowing them to connect to 4 SIP servers at once, each with its own button - aha:
So what I've been able to do is to tell every one of our handsets to connect to every one of our four servers, and all the individual servers to connect to each other in a big mesh. It's all over IP, so it's free! The wonder of VoIP. This is what it looks like:
At our desks we can all feel like Wall Street big shots- "yah yah, I'll just have to try routing that one via London, excuse me one second (beep) hang on let me send that one through Amsterdam (beep)", pressing buttons to send our outgoing calls through each different server server. No, not really, only a fool would do that.
But it does mean that we can receive incoming calls from any server. Instead of having to worry about which server is currently "live", I've told every server to try to dial out four times simultaneously. The first attempt is to a "local" SIP connection, our desk phones connecting directly to that server. The other three connections go to the other three peers, and try to connect to the same phone via that peer. So the command looks like this:
Dial(SIP/mbloch-desk&IAX2/manchester/301&IAX2/office/301&IAX2/offsite/301)
When any one of those connections picks up, Asterisk cancels the other dialling attempts, and the call proceeds. If any of the connections are down, whether to another server or the phone itself, Asterisk quietly gives up and carries on ringing the other connections. All the while, the caller only hears a single ring tone.
So the upshot is that if one server conks out, outbound calls still work by the user hitting "line 2" or "line 3", and the only thing I have to do is signal Magrathea to ask them to send incoming calls through another server. The meshing keeps any dirty secrets about our network status hidden from callers, and customers never hear a dead line again, hooray!
Testing and deployment
In order to make the setup as robust and easy to test as possible, I've used the excellent Capistrano automation tool to package up the whole configuration and startup routines into one place. So when I make a change I can just type "cap deploy" and all my changes go out to the four servers, and everything. It's very well suited to this, I hardly had to make any changes to the way it works.
But I've not implemented any automatic testing yet because I've not got my head around a couple of relevant tools, SIPp appears to be the only one that I could find, and that doesn't help me test the meshed IAX connections.
Because these are all new servers, I've just made most of it live, and placed heaps of calls. I think that's what real phone engineers do, at least in part. The worst that has happened is a cascade between servers (i.e. one inbound call caused four more internal calls, which caused four more internal calls, which caused etc.). I thought it was very funny at the time because it made my phone look like a Christmas tree. Ho ho ho. But I'd forgotten to take out our call centre's number as a fallback, so I'd accidentally placed 50 simultaneous silent calls to them. More than once I think. They told me so. Sorry!
Unsolved problems
I did say I was nearly ready... all that remains over the next couple of days is:
- when a single call comes in, all four lights on our phones flash up, which is panic-inducing as it makes me think all our customers are calling at once. That's never a good sign. How do I keep the robust signalling but remove all the blinking lights?
- how to make sure the calls are routed through the nearest phone server? I don't want to be talking to our customers via the Netherlands if I don't have to be;
- is it worth trying to automate re-routing of inbound calls? Currently I have assigned a magic phone number to each server which I can call to make it "claim" the main phone number from Magrathea. Trying to automate it seems risky.
- do I need to worry about local storage on each server? I am adding a few features to be able to leave "sorry we know about problem X" so that incoming callers know we know about problems when they happen. But I would prefer not to worry about updating each server individually with the same message.
So once these are fixed to my satisfaction, we should be moved over next week. Since real live Asterisk setups seem thin on the ground, I might publish this setup as a worked example (once it's worked for a month or two, ha) if anyone is interested.
And while it's bedding in, the same strategically-placed servers will also run self-contained copies of our main web site, forum and support email facilities, and I'll be testing the failover for those.
Oh my god, I've just noticed, it's a cloud. I've made a cloud. I've even drawn a bloody cloud.
1 comment
Posted by Matthew Bloch
Sat, 16 May 2009 10:37:00 GMT
So Pete and I have resolved the situation from last Sunday and I hope explained it in detail. It's a shame that we will start to treat our "carrier grade" routers like crabby Windows 98 PCs, and reboot them at the first sign of trouble. But I challenge any other network engineer to say they could come up with a design that would have withstood this fault, or a sensible method of diagnosing and fixing it.
The broken supervisor is now replaced, and we're moving on to the larger problem - which was that our support systems went dark during the outage. We'd not planned for a core network outage of longer than a few minutes, so I resorted to Twitter to post updates. Embarrassing, as Twitter itself presented its famous fail whale for a few minutes around 7pm. But definitely preferable to sitting on my thumbs while Pete was working on the fault itself - I went from 4 to 250 "followers" in a couple of hours, with links quickly making their way around the internet on other forums. Tim Anderson wrote a thoughtful piece about how useful the mechanism was, but Twitter-using customers are the minory - it took Google a couple of hours into the outage to pick up on other discussion threads and index them.
I'm not trying to gloss - it was a terrible mechanism, if only because I had to describe the problem in haiku, and most of our customers didn't know where to look.
Expecting the unexpected
So what are we doing to fix this? The starting point for my disaster planning is that our core network never goes down. That's the design. It's intended to be incredibly unlikely, and we go to great expense to ensure that it won't. While our entire network wasn't down on Sunday (London was still alive), the lesson to learn is that in the face of treacherous network equipment, all bets are off and this would have taken down anyone else's core network. If this is the kind of fault we might expect, we need to work around it.
To be clear, Bytemark has one core network, with one business goal, reliable hosting for a group of customers with similar needs. If that core breaks, all possible resources are diverted to fixing it. When that happens, we're not in a position to answer emails, phone calls or do anything other than fix the fault - but - we will frequently update customers on our progress and an ETA. As soon as our core network is functioning again, support emails can queue up, operators can take messages and so on. But I am planning a "red alert" network state where communication can only be one way - there is only one question customers will want to ask in these situations, and only one answer we will be able to give. Everything else should wait.
At least say you're sorry - no more timeouts
I'm still planning the details, but a common component is what I'm calling our "sorry server" to help both us, our customers, and our customers' customers, in the event of small or large outages.
At the moment, packets can come into our network for routing at one of four core routers - two in London, and two in Manchester. They are the front doors to our network, and have an enormous routing capacity, planned to take any kind of abusive traffic from the outside. We can manage if only one of them is working in each site.
When something is going wrong for a web site owner - whether it's their server being rebooted for a few minutes, or a major external event, that incoming traffic won't make it to their server. The visitor just sees a spinning hourglass and a "connecting to host..." message that eventually turns into an inexplicable "timeout error". That's not what we want.
I'm intending to place 2 or 3 "sorry servers" on the edge of the network, plugged directly in to all four core routers. These servers have no other function than to be ready to say "sorry" if something goes wrong. Just a simple web page, a URL to point visitors at for more information. Email can get delayed or turned away while the "sorry" server is answering.
For our own support services, we can make sure that these points at an off-site status page. All we need for this to function is one working core router out of four, one transit connection in, and the decision to artially shut down the network to turn on this facility if we have to. In the past six years we've never have been without this option, so I'd be confident that it can work.
I'd like to also open the service to all customers, so you can upload and selectively switch on your own "sorry page" when you're performing maintenance, moving between hosts, or for any other reason. I'll document this service on our main web site when it becomes available.
Why not DNS updates?
A couple of customers asked us why they couldn't update their Bytemark-hosted DNS quickly in the event of an outage. Answer: because it won't work fast enough - DNS changes really need days to percolate through the internet's various DNS caches. People will see your old IP even after you've made a change, but worse, when you want to switch it back again, you might be stuck pointing at your backup hosting for much longer than you intended.
The sorry server will be a smarter way of doing this, and allow you to send instant HTTP redirects or SMTP responses should be turn off-and-onnable instantly.
Getting our house in order
The sorry server is only a starting point, but a flexible one. I will still duplicate our forums, main web site and phone answering service onto an off-site server, and be able to flick a switch to take those services off our network if necessary. So Bytemark will always have the bones of our support operation present, even if a similar situation cropped up again. I'll let you know how we're doing on this.
From this and other recent hosting war stories and scares, I've been intending to complete our failsafe systems in creative ways, and fix niggling bugs, before striding ahead with new developments. So now is an excellent time for any customers to email me with your least favourite Bytemark bug, and I'll let you know what I'm doing about them.
no comments
Posted by Matthew Bloch
Mon, 20 Apr 2009 23:26:00 GMT
I've finally put up our new web site - there are a few rough edges which we'll address in the next few days, but it's a definite improvement overall. I would be interested to hear your comments below, or direct to matthew@bytemark.co.uk.
The main highlights are:
- 150MB virtual machines are now 256MB for the same price, and so on up the range;
- there's now a 2GB virtual machine available for £55 per month (that's only £10 more for double the memory);
- Atoms and Phenoms now come with 2 x 500GB drives as standard.;
- setup fees for upgrades are now zero, simplifying pricing somewhat;.and
- you can quickly experiment with prices on the front page.
We've stopped doing single and triple core systems, as well as single-drive systems, and have raised the price of new Atom systems which was necessary to keep the profitable amid rising data centre costs. The 2GB virtual machines are a lower-cost replacement, with probably faster storage.
We're now much more confident in selling Windows systems including Windows Data Center Edition. The licensing for this edition is unusually generous in that you're allowed to run as many copies of Windows on your dedicated server while paying only £50 per month per server extra. That's a lot of money for something you can do for free with Linux of course, but if you need .NET, IIS and the full stack of Microsoft technologies, it's a lot better value than it used to be. Windows Web Server edition is £15 per month, which is more suited to the lower-cost Atom.
The documentation for our Linux-based vhost system has had a shot in the arm, and is still ongoing. This is our tool to tame full root access, making it simple to deploy lots of web sites and mailboxes.
Finally we should soon have a documented set of tools for running KVM as our preferred virtualisation solution. Some customers are testing these already, and we are working on a migration path away from Xen. Sorry Xen guys, it was just too difficult to keep up, and we're glad to offer a better virtualisation system than we did before.
If you can make it down to Earls Court next week - Peter, Patrick, Yann and me will see you at Internet World.
no comments
