Posted by Matthew Bloch
Fri, 12 Jun 2009 08:41:00 GMT
I'm close to flicking the switch on our new phone system - is anyone was interested in the design? I'm not certain myself, which is to say I'm not a phone engineer of any kind. Last I looked, phone engineers lived in a topsy-turvy parallel universe with their own private members clubs, secret handshakes from government and cultish ideas about circuit switching. Me, I just downloaded asterisk five years ago and found a rebel band of software engineers trying to crowbar their way in.
And there is something of the crowbar about asterisk - a box of bits featuring about four different scripting languages which all boil down to something that looks like 20-year old BASIC. U-turns abound in its documentation, even the simple task of setting a variable says "Version differences: This command is not available in Asterisk 1.0.9. Use SetVar instead. As of v1.2 SetVar is deprecated and we are back to Set." Somehow this shed of a program has been running our company phones - including home workers, voicemail, smart caller ID, smooth redirections to our Manchester call centre, for the last five years, with barely any maintenance. Whatever crimes of design it might have committed, asterisk is very capable, but I'd never thought very hard about how to make it robust.
Planning for the lights to go out
My first priority after last month was to make sure that we couldn't lose touch with our customers again, and because the current asterisk system is a bit of a toy (albeit a long-lived and very reliable one) I needed to rearrange it to survive in the face of network trouble. And the design work will be very similar for our key services - our email support, web site, forums and so on.
So this is what our network looks like for disaster recovery purposes:
Right now, we've got our awesome Fisher-Price FBI phones, GXP2000s in the office, one per desk, and a couple of home workers. Every phone to connect ot the ofifce phone server, and if that's down, the line is dead. If the office ADSL is down, the line is dead, even the home workers can't talk to our customers. If the London network stops working, the line is dead.
To take these points of failure out, I can take advantage of our network: we have two pretty separate parts to our core. Our London racks are rich with connections but very expensive, so we don't host much there. Our newer Manchester space is physically larger, but without the same richness in terms of criss-crossing minor connections. The failures that we've seen have only ever affected one or the other, so I've put one new phone server in each location.
I'm also commissioning another offsite in the Netherlands. This could prove to be a rotten idea because of the latency, but we'll have to see - it's intended as an absolute last resort in the unprecendented event of both sides shutting down.
Who's talking to whom
I can install all these servers, but how do they function as one unit?
We only have one advertised phone number, and one supplier for this phone number, the shadowy folk at Magrathea (whose services are the lynchpin to most of the UK VoIP industry, as far as I can tell). Unfortunately they will only try to route to one of my servers at a time, so I have to pick one to receive our incoming calls - their racks are in London, so I'll use the London server unless anything goes wrong. When it does, I can update Magrathea and ask them to send the calls elsewhere. But they can all send their outgoing calls through Magrathea at once if necessary.
At the other end, on our desks, the GXP2000s have a very handy function allowing them to connect to 4 SIP servers at once, each with its own button - aha:
So what I've been able to do is to tell every one of our handsets to connect to every one of our four servers, and all the individual servers to connect to each other in a big mesh. It's all over IP, so it's free! The wonder of VoIP. This is what it looks like:
At our desks we can all feel like Wall Street big shots- "yah yah, I'll just have to try routing that one via London, excuse me one second (beep) hang on let me send that one through Amsterdam (beep)", pressing buttons to send our outgoing calls through each different server server. No, not really, only a fool would do that.
But it does mean that we can receive incoming calls from any server. Instead of having to worry about which server is currently "live", I've told every server to try to dial out four times simultaneously. The first attempt is to a "local" SIP connection, our desk phones connecting directly to that server. The other three connections go to the other three peers, and try to connect to the same phone via that peer. So the command looks like this:
Dial(SIP/mbloch-desk&IAX2/manchester/301&IAX2/office/301&IAX2/offsite/301)
When any one of those connections picks up, Asterisk cancels the other dialling attempts, and the call proceeds. If any of the connections are down, whether to another server or the phone itself, Asterisk quietly gives up and carries on ringing the other connections. All the while, the caller only hears a single ring tone.
So the upshot is that if one server conks out, outbound calls still work by the user hitting "line 2" or "line 3", and the only thing I have to do is signal Magrathea to ask them to send incoming calls through another server. The meshing keeps any dirty secrets about our network status hidden from callers, and customers never hear a dead line again, hooray!
Testing and deployment
In order to make the setup as robust and easy to test as possible, I've used the excellent Capistrano automation tool to package up the whole configuration and startup routines into one place. So when I make a change I can just type "cap deploy" and all my changes go out to the four servers, and everything. It's very well suited to this, I hardly had to make any changes to the way it works.
But I've not implemented any automatic testing yet because I've not got my head around a couple of relevant tools, SIPp appears to be the only one that I could find, and that doesn't help me test the meshed IAX connections.
Because these are all new servers, I've just made most of it live, and placed heaps of calls. I think that's what real phone engineers do, at least in part. The worst that has happened is a cascade between servers (i.e. one inbound call caused four more internal calls, which caused four more internal calls, which caused etc.). I thought it was very funny at the time because it made my phone look like a Christmas tree. Ho ho ho. But I'd forgotten to take out our call centre's number as a fallback, so I'd accidentally placed 50 simultaneous silent calls to them. More than once I think. They told me so. Sorry!
Unsolved problems
I did say I was nearly ready... all that remains over the next couple of days is:
- when a single call comes in, all four lights on our phones flash up, which is panic-inducing as it makes me think all our customers are calling at once. That's never a good sign. How do I keep the robust signalling but remove all the blinking lights?
- how to make sure the calls are routed through the nearest phone server? I don't want to be talking to our customers via the Netherlands if I don't have to be;
- is it worth trying to automate re-routing of inbound calls? Currently I have assigned a magic phone number to each server which I can call to make it "claim" the main phone number from Magrathea. Trying to automate it seems risky.
- do I need to worry about local storage on each server? I am adding a few features to be able to leave "sorry we know about problem X" so that incoming callers know we know about problems when they happen. But I would prefer not to worry about updating each server individually with the same message.
So once these are fixed to my satisfaction, we should be moved over next week. Since real live Asterisk setups seem thin on the ground, I might publish this setup as a worked example (once it's worked for a month or two, ha) if anyone is interested.
And while it's bedding in, the same strategically-placed servers will also run self-contained copies of our main web site, forum and support email facilities, and I'll be testing the failover for those.
Oh my god, I've just noticed, it's a cloud. I've made a cloud. I've even drawn a bloody cloud.
1 comment
Posted by Matthew Bloch
Sat, 16 May 2009 10:37:00 GMT
So Pete and I have resolved the situation from last Sunday and I hope explained it in detail. It's a shame that we will start to treat our "carrier grade" routers like crabby Windows 98 PCs, and reboot them at the first sign of trouble. But I challenge any other network engineer to say they could come up with a design that would have withstood this fault, or a sensible method of diagnosing and fixing it.
The broken supervisor is now replaced, and we're moving on to the larger problem - which was that our support systems went dark during the outage. We'd not planned for a core network outage of longer than a few minutes, so I resorted to Twitter to post updates. Embarrassing, as Twitter itself presented its famous fail whale for a few minutes around 7pm. But definitely preferable to sitting on my thumbs while Pete was working on the fault itself - I went from 4 to 250 "followers" in a couple of hours, with links quickly making their way around the internet on other forums. Tim Anderson wrote a thoughtful piece about how useful the mechanism was, but Twitter-using customers are the minory - it took Google a couple of hours into the outage to pick up on other discussion threads and index them.
I'm not trying to gloss - it was a terrible mechanism, if only because I had to describe the problem in haiku, and most of our customers didn't know where to look.
Expecting the unexpected
So what are we doing to fix this? The starting point for my disaster planning is that our core network never goes down. That's the design. It's intended to be incredibly unlikely, and we go to great expense to ensure that it won't. While our entire network wasn't down on Sunday (London was still alive), the lesson to learn is that in the face of treacherous network equipment, all bets are off and this would have taken down anyone else's core network. If this is the kind of fault we might expect, we need to work around it.
To be clear, Bytemark has one core network, with one business goal, reliable hosting for a group of customers with similar needs. If that core breaks, all possible resources are diverted to fixing it. When that happens, we're not in a position to answer emails, phone calls or do anything other than fix the fault - but - we will frequently update customers on our progress and an ETA. As soon as our core network is functioning again, support emails can queue up, operators can take messages and so on. But I am planning a "red alert" network state where communication can only be one way - there is only one question customers will want to ask in these situations, and only one answer we will be able to give. Everything else should wait.
At least say you're sorry - no more timeouts
I'm still planning the details, but a common component is what I'm calling our "sorry server" to help both us, our customers, and our customers' customers, in the event of small or large outages.
At the moment, packets can come into our network for routing at one of four core routers - two in London, and two in Manchester. They are the front doors to our network, and have an enormous routing capacity, planned to take any kind of abusive traffic from the outside. We can manage if only one of them is working in each site.
When something is going wrong for a web site owner - whether it's their server being rebooted for a few minutes, or a major external event, that incoming traffic won't make it to their server. The visitor just sees a spinning hourglass and a "connecting to host..." message that eventually turns into an inexplicable "timeout error". That's not what we want.
I'm intending to place 2 or 3 "sorry servers" on the edge of the network, plugged directly in to all four core routers. These servers have no other function than to be ready to say "sorry" if something goes wrong. Just a simple web page, a URL to point visitors at for more information. Email can get delayed or turned away while the "sorry" server is answering.
For our own support services, we can make sure that these points at an off-site status page. All we need for this to function is one working core router out of four, one transit connection in, and the decision to artially shut down the network to turn on this facility if we have to. In the past six years we've never have been without this option, so I'd be confident that it can work.
I'd like to also open the service to all customers, so you can upload and selectively switch on your own "sorry page" when you're performing maintenance, moving between hosts, or for any other reason. I'll document this service on our main web site when it becomes available.
Why not DNS updates?
A couple of customers asked us why they couldn't update their Bytemark-hosted DNS quickly in the event of an outage. Answer: because it won't work fast enough - DNS changes really need days to percolate through the internet's various DNS caches. People will see your old IP even after you've made a change, but worse, when you want to switch it back again, you might be stuck pointing at your backup hosting for much longer than you intended.
The sorry server will be a smarter way of doing this, and allow you to send instant HTTP redirects or SMTP responses should be turn off-and-onnable instantly.
Getting our house in order
The sorry server is only a starting point, but a flexible one. I will still duplicate our forums, main web site and phone answering service onto an off-site server, and be able to flick a switch to take those services off our network if necessary. So Bytemark will always have the bones of our support operation present, even if a similar situation cropped up again. I'll let you know how we're doing on this.
From this and other recent hosting war stories and scares, I've been intending to complete our failsafe systems in creative ways, and fix niggling bugs, before striding ahead with new developments. So now is an excellent time for any customers to email me with your least favourite Bytemark bug, and I'll let you know what I'm doing about them.
no comments
Posted by Matthew Bloch
Mon, 20 Apr 2009 23:26:00 GMT
I've finally put up our new web site - there are a few rough edges which we'll address in the next few days, but it's a definite improvement overall. I would be interested to hear your comments below, or direct to matthew@bytemark.co.uk.
The main highlights are:
- 150MB virtual machines are now 256MB for the same price, and so on up the range;
- there's now a 2GB virtual machine available for £55 per month (that's only £10 more for double the memory);
- Atoms and Phenoms now come with 2 x 500GB drives as standard.;
- setup fees for upgrades are now zero, simplifying pricing somewhat;.and
- you can quickly experiment with prices on the front page.
We've stopped doing single and triple core systems, as well as single-drive systems, and have raised the price of new Atom systems which was necessary to keep the profitable amid rising data centre costs. The 2GB virtual machines are a lower-cost replacement, with probably faster storage.
We're now much more confident in selling Windows systems including Windows Data Center Edition. The licensing for this edition is unusually generous in that you're allowed to run as many copies of Windows on your dedicated server while paying only £50 per month per server extra. That's a lot of money for something you can do for free with Linux of course, but if you need .NET, IIS and the full stack of Microsoft technologies, it's a lot better value than it used to be. Windows Web Server edition is £15 per month, which is more suited to the lower-cost Atom.
The documentation for our Linux-based vhost system has had a shot in the arm, and is still ongoing. This is our tool to tame full root access, making it simple to deploy lots of web sites and mailboxes.
Finally we should soon have a documented set of tools for running KVM as our preferred virtualisation solution. Some customers are testing these already, and we are working on a migration path away from Xen. Sorry Xen guys, it was just too difficult to keep up, and we're glad to offer a better virtualisation system than we did before.
If you can make it down to Earls Court next week - Peter, Patrick, Yann and me will see you at Internet World.
no comments
Posted by Matthew Bloch
Thu, 12 Mar 2009 16:36:00 GMT
We’re oot and aboot over the next few weeks:
The UKUUG conference is a merciless 3-day technical assault which you can endure with a couple of hundred other system administrators and programmers. And usually some nice drinks and dinner to soften the blows. It’s a very technical conference, but worth the time and money, at least if your job is to keep up to date with systems administration.
Internet World is an sprawling old monster of a trade show featuring a depressing number of shonky-looking SEO firms, but also lots of web hosts and specialist little providers jostling for attention. We hope to grab yours.
And the organisers couldn’t say this, but I will - CLOUD Expo Europe is going to be a better Linuxw*rld, formed from many of the embers of the last UK show. While relegating Linux to only an initial, it is organised by many of the same people, and has pulled the focus out to other UNIXes, cloud computing (oblig. sneer from me, for now) and developers, developers, developers. Some of our best customers are developers. Come on you lot, visit us and make our day.
I’m busy making the arrangements for all three, and hope our new web site will make an appearance just before (maybe during, maybe on the train down at this rate) the UKUUG conference. Any blog followers, customers or stalkers are welcome to meet me for a drink around any of these events, just drop me a line or comment below.
Also - new staff - we have a Yann! And next month, a David. I will welcome them both soon, but in the mean time please treat them nicely if you see them answering your emails.
no comments
Posted by Matthew Bloch
Fri, 20 Feb 2009 22:33:00 GMT
Today was Alex (and Fern’s) last day at Bytemark - they’re leaving us for new opportunities in Bath, greener pastures (literally) and closeness to family. He has recently helped move a large number of customers out of one of our data centres, maintained a lot of monitoring and logging systems, opened our eyes to some some interesting new kit and vendors over the two years he’s been with us, and many customers will miss his occasional firm hand on the support queue. Good bye, good luck!
His last act was sorting a pile of surprisingly powerful servers which we’ll be installing for the benefit of larger open source projects in need of hosting, build farms or other developer resources. If you’re helping to maintain any Free software, and need some infrastructure to help it develop, now is a very good time to ask me about hosting. We have servers, rack space and bandwidth to donate, especially to projects that we use ourselves.
So we have been looking for at least one replacement Alex all this week, and already met some very interesting and expert local engineers. The marathon finishes at the end of next week, and the one thing I can say for certain is I need to urgently fix the hot drinks situation. We’ve had 1.5-2.5hr interviews, 3 a day, and one cup per interview isn’t enough. What’s the etiquette in an interview when you have lots more to talk about, but don’t want to leave your interviewee alone to get another drink? Is it OK to go make another cup of tea and leave them alone for a few minutes? I will shift the coffee machine from the corridor, make a full pot, and get some kind of enormous samovar to dispense tea. So please excuse us if you’ve got a 4pm interview scheduled, and Pete and me are completely ripped by that time; we’re just trying to keep hydrated.
1 comment
Posted by Matthew Bloch
Mon, 16 Feb 2009 00:58:00 GMT
Since about 2-3 weeks ago, new virtual machines have been faster! A lot faster. That’s because we’ve tweaked our virtual machines to use the excellent Kernel Virtual Machine, finally completing our upgrade from User-Mode Linux which helped start our business six years ago. For the same money as before you’ll now get 256MB instead of 150MB, 512MB instead of 300MB, 768MB instead of 450MB and 1024MB instead of 600MB. Disc and bandwidth quotas are remaining the same. However the underlying technology is more efficient too - I/O bandwidth is much greater, with less work for the host, and we’ve seen loads on our hosts dropping or at worst, remaining constant. Like so:
That shows a drop in load after moving servers from UML to KVM (graphs from old and new servers spliced together). It’d be a cheat to say that this is entirely down to the more efficient technology, as we’ve upped the memory allocations by 70%. Unfortunately I can’t find any particularly recent technology tests to prove our point. But as a like-for-like commercial comparison it’s fair - you get more RAM for the same price. And customers have told us it their machines feel faster than they did.
Some VM administrator will already have spotted the upgrade as we’ve been moving servers around. Others will have to hang on up to 6 weeks while we set up £60,000 of new hosting hardware - but everyone will get the upgrade for free. The extra hardware will give us the time we need to move everyone off the old hosts, and on to the new. If all goes to plan, it won’t be the last improvements to our VM platform this year either. We’re not happy to sit on our laurels for too long, and once my current round of marketing and recruitment is out of the way, I’m going back to the coal face for a few weeks to improve our VM management.
(sorry, our site still lists the old specifications, but new VMs will get the new specs, and the web site shouldn’t be more than a week from catching up).
no comments
Posted by Matthew Bloch
Mon, 02 Feb 2009 17:46:00 GMT
Bytemark are now looking for candidates to fill three positions:
- A first-line technical support representative (£15-22k, from March)
- A Linux systems or programming expert (£20-40k, from March)
- A technical author for a 2-3 month contract (£15-£25 per hour, immediate start)
If you, or anyone you know is interested in Linux and wants to work with us in snowy York, please drop me a line! Here are the adverts as posted.
First-line technical support representative (£15-22k)
Bytemark Hosting is an Internet Service Provider based in York, and we’re looking to recruit a permanent first-line support representative to manage our steadily growing technical support queue.
We currently manage our support by allocating a single expert engineer per week to answering our customers’ calls and emails. The volume has now grown to the point where this is a full time job - many enquiries are simpler sales and technical questions, and the directors think that
an enthusiastic Linux "power user" could manage these on a permanent basis, while more complex questions can still go to our more experienced engineers.
The ideal candidate will be someone who uses Linux on a daily basis, or simply manages their own web site(s, the more ambitious the better). However neither experience is strictly necessary, we are willing to train the person with the right attitude on every aspect of our business so that they can fulfil this important role for our customers. There will also be opportunities to learn and practice programming and systems administration. The most important skill will be to maintain a calm, polite and organised front to our staff and customers.
Bytemark ask for a 37.5 hour working week to be worked from our offices in York, and offer 25 days paid holiday plus UK bank holidays. For this
role we can pay between £15000-£22000 per annum dependent on previous experience.
If you’re interested please send your CV to Matthew Bloch <matthew@bytemark.co.uk> and let me know in brief how you think you can help our company.
Linux systems and software engineer (£20-40k)
Bytemark Hosting is an Internet Service Provider based in York. We’re looking to recruit a Linux systems or programming expert to build and maintain various internal systems for our expanding business - these systems organise our network, racks, inventory and billing. They are increasingly business critical as we rely on their abilities to provide new services to our customers, and we need a larger team to work on both
their development and reliable deployment. You’ll also need to support our customers in using them, and help publish and promote some of their components of to the free software community.
Currently we have a many important internal systems written largely with the Ruby language and Rails framework, so these would be ideal skills to have. Perl, PHP, Java would also be useful to know, but nothing speaks louder than the experience of an open source project or previous commercial experience, in whatever language.
Bytemark ask for a 37.5 hour working week to be worked from our offices in York, and offer 25 days paid holiday plus UK bank holidays. For this role we can pay between £20000-£40000 per annum dependent on previous experience, and may consider some days working from home when working on defined projects.
If you’re interested please send your CV to Matthew Bloch <matthew@bytemark.co.uk> and let me know in brief how you think you can
help our company.
Technical author
Bytemark Hosting is an Internet Service Provider based in York, and we’re looking to hire a technical writer on a temporary basis. We need someone to bring our web site up to date with a large amount of original technical writing - the goal will be to make our hosting services as easy to use as possible for as wide an audience as possible.
We need someone who is understands hosting products, probably hosting and maintaining their own web site(s) already . You’ll need to explain the basics of uploading files, the domain name system and other hosting complexities with elegance and frequent illustration; your audience will be less experienced but enthusiastic users of Windows, Mac OS and Linux.
This is not a permanent position, and the initial brief will be 8-12 weeks work. However we are working hard on our web site at the moment, and may have immediate follow-on work. We can pay between £15-25 per hour depending on experience. The work does not need to done from our office in York, though you may feel that being on-site is advantageous to the job.
If you’re interested please send your CV to Matthew Bloch <matthew@bytemark.co.uk> and let me know in brief how you think you can help us.
no comments
Posted by Matthew Bloch
Tue, 20 Jan 2009 23:50:00 GMT
It’s time to drag our virtual machine platform into the 21st century and we’ve started to see the results today:
This disjoint graph shows the total load on one of our ordinary virtual machine servers, about as heavily loaded as we allow them to get. On the left we can see a couple of days ordinary load with spikes and relatively wide variation in responsiveness. On the right, that’s exactly the same set of virtual machines, on identical hardware, after being copied on our new platform and started up again (I omitted out the load ramp-up and ramp-down as we copied). That’s quite a big difference, and I suspect the customers with VMs on the server can feel it. I’m being a little guarded at the moment: we’re not quite done tweaking bits of the system and the emulator will need rebuilding a few more times before we’re completely happy with it (the "emulator" is KVM, not Xen, and I’m more than happy with its advantages). Assuming nothing goes terribly wrong in the next couple of weeks we’ll continue this roll-out and conversion of our customers’ systems, and our customers systems will get a fair amount faster, for free. This is a good thing. More for less. Applause.
That’s not the half of it, but the core performance improvement felt worth a glib graph. The marketing demon on my shoulder is shushing me now. Also saying we’re out of fizzy drinks.
3 comments
Posted by Matthew Bloch
Sun, 11 Jan 2009 07:05:00 GMT
I stumbled across the Deloitte Fast 50 and thought for a nanosecond, there’s a competition we should have entered! By my reckoning Bytemark would ranked somewhere between 10th and 25th depending on my recollection of our first year revenue. All that growth, and no recognition! But I am sworn off awards, league tables and so on after our first and only experience of the "awards industry" a few years ago. I’d eagerly put in a nomination for our company, and there was an expensive do in a London hotel which cost us £400 just for dinner. I wasn’t going to go but the organiser hinted strongly that we might have won. More fool me; no we hadn’t, but there were pricey tables to fill. And then I thought, what benefit is anyone getting from this? The winners get a certificate for their web site, the organisers might get a bit of press excitement for their publication if they can keep the awards going each year (this awards ceremony didn’t), but the rest of us? Just losers paying thousands of pounds for dinner!
Some in the awards industry realised this economic imbalance; there are a few sites whose awards are not just annual but monthly, voted for by its readers, ah, but only if your company already pays for a listing on their listings site, e.g. webhostdir and others. So you pay, you badger your customers to click a link, and your award is delivered within a week or two to be displayed on your web site forever. Some have so many award categories and such a fast turnaround for new winners that you’re statistically guaranteeed an award within a few months of your "paid listing". Wherever there’s an award, there’s a money trail to follow. So I don’t know what Deloitte would gain from their Fast 50; they probably only need to sign one new (flattered, beaming) client a year from their list to pay for the whole do.
This awards-as-advertising idea culminates in the glorious insanity of the UK product of the year award with its ability to compare air freshener, hair straighteners and dog food on the same criteria, and £20000 price demanded of the selected winner. Do they select another "winner" if you don’t pay? Thankfully our satisfied, talkative customers are still our best advertising and I get to spend time developing our products for our customers to recommend instead of chasing gongs.
no comments
Posted by Matthew Bloch
Wed, 24 Dec 2008 01:06:00 GMT
Here’s to all our customers and fans - we hope you have a happy Christmas! May your pagers be silent and backups be reliable well into 2009.
This is the after-dinner shot taken by the waiter at Biltmore Bar & Grill at the Bytemark Christmas do ten days ago. We had a fantastic time and a great year of growth to celebrate (cough, accounts pending, but we’re pretty sure it’s another few tens of percent by now). When Bytemark made the switch from being a consulting company 6 years ago, Pete and I realised that subscription revenue was going to keep us going through bad economic times and we’re very grateful to our customers for their continued faith in us. We absolutely don’t take it for granted, and will have some interesting news in the first couple of months next year. I hope we’ll make a few new fans and provide our loyal customers with some super upgrade options.
I had two monster articles half-written over the last month, but I think they’re destined for the bin now - you need a bit of speed and concentration for long rambling posts, and if they’re not done in a day or weekend, they’re never going to get done. For now, I’ll work on my New Year’s resolution to write here little and often, and come back with some more regular company news in 2009.
no comments
