Bytemark Hosting finally bought its own building last year. We got the keys in January and had converted the space into a data centre by July. We have previously rented data centre space in Manchester since 2008, and it wasn’t obvious that we would go to the hassle of moving people across — until we got our renewal quote from our suppliers in Manchester.
It was huge!
As a result, in July we made the decision to move two thirds of our servers in Manchester to York. The total number of servers moved or retired by this December will be nearly 1000. These are servers that customers expect to be online 24/7, many with difficult network setups.
We have done server moves before, and we learned a lot of valuable lessons from those events, but we had not attempted anything on this scale. We needed to keep disruption to a minimum, schedule the moves as precisely as we could, and keep all the processes in our own hands.
So here’s the story of how we successfully moved our data centre on the night of the 23rd September, and what we have learned from the process — from our initial planning to the finishing touches.
Step 1: Set a schedule
We’re not too proud to dump all our server data into a shared spreadsheet. This turned out to be the lynchpin of the operation, and a great tool for real-time co-ordination:
We could have built a Rails application, database and so on, but that takes time! You can’t beat the flexibility and speed of a spreadsheet for one-off, multi-team, multi-site co-ordination, and you can get results quickly.
Our goal was to empty 2 full cages; about 750 servers were identified as being candidates for moving. We tried to stick to the “easy” moves first, and approached the customers with more complex configurations later on.
We divided the moves into 5 nights and each night had 4 or 5 separate vans which would leave Manchester at half-hour intervals.
Every server went onto the sheet: old and new rack locations, IP address, dates, times and so on. From that point on, any changes to the plan were reflected clearly on the spreadsheet, and any member of the team could update it safely.
Step 2: Tell the customers
Once we had a schedule, we emailed the customers, several weeks in advance, to let them know that we were going to move their servers. We could tell them when they were going to get switched off, and (based on 90 minute journey time, plus loading and unloading) a reasonable estimate for when they’d be back on again.
Our email script read the move windows from the spreadsheet and opened tickets through our ticketing system (that’s another win for the spreadsheet “solution”).
Step 3: Tell them again, differently, and keep track of replies
The same script emailed our customers again at weekly intervals with a slightly differently-phrased message, to try to ensure our first email hadn’t been ignored. And the final email was different again.
Subject: <%= server %> is being moved on <%= date %> at <%= start %> Subject: REMINDER: <%= server %> is being moved, evening of <%= date %> Subject: FINAL WARNING: <%= server %> is being moved evening of <%= date %>
We changed wording in the body to try to try to get people’s attention any way we could, because we didn’t have the staff to ring round every customer, and needed to make sure we got heard!
Our script used RT‘s excellent command line interface to open new support tickets for customers, which lets us track replies. It is really useful to track each customer’s move individually through RT, rather than just send untracked email blasts. When customers replied, we had a full email chain recorded, and we could set global options per ticket which allowed the data centre team to search and categorise correspondence. RT’s custom fields are handy like that:
Step 4: Put stickers on everything
Before each move day, we label up every server that’s going to move with it’s name, exact destination and the move window (so A = the 9.30 pm van, B = 10.00 pm etc.). That let us spread the work of selecting and moving the servers among a lot of people — just look for the big letter, shut the server down at the keyboard, and pull it out.
As we started the moves, we had a lot of internal systems to update.
Our rack organiser keeps track of all the servers, racks and cabling inbetween them. It only lets us update one server and cable at a time, so earlier in the evening Peter had a lot of clicking to do, updating every server that was going to move.
Likewise, our network needed reconfiguring, so our routers would know that these servers’ IP addresses were in York rather than Manchester. Again, this has no automation involved and Tom needed to perform each configuration step by hand. We’d organised the vans by network so he could move each network as the servers were being powered off.
We debated whether to build some special-purpose automation to make this process less boring, but decided against it. Because 1) it seemed unlikely to take much less time than it would save, 2) we’d end up with user interfaces that we’d not test again for years, and 3) well-executed, well-reheared boring processes don’t go wrong.
There is certainly more automation we could build into our network and rack management, but it’s not the kind of thing we’d throw together for one occasion. So that night, there was a lot of clicking, and the job got done.
Step 6: On the night
The vans lined up in Manchester at 9pm and our Manchester team powered them off and piled them in, 20 or 30 at a time. Each van leaves at half-hour intervals and is greeted warmly by the York team, pictured here at midnight:
[Our data centre has contained “pods” where the cold air is piped up in the middle, and gets blown through into the main space of the DC. It’s more efficient because we are using less cold air, but servers won’t stay cool if they’re not sucking in chilled air.]
So before we could re-rack these servers, the next stage of the production line was to sort them and open up those that needed their fans reversing:
That was the cue for our sysadmin team upstairs to log into the box, check that the serial line was working, and that there was no stupid reason the box wasn’t booting. So we looked for long-running filesystem checks, dead disks and botched kernel upgrades. Once each box had a login prompt and pings, we updated the spreadsheet to say the box was done! and we moved onto the next.
At the same time our “on-call” sysadmin was watching for alerts; Nathan came into the office for the occasion (usually for this role, he’d be working from home, waiting for texts).
The alerting system lets them know if a customer has raised an urgent support request, or there is a monitoring failure of a manged client, or various other automated checks. Many of these will be expected, and Nathan’s job was to deal with the unexpected, answer the phones, as well as dealing with the usual load or urgent support requests unrelated to the move.
Between the failures we expected, and the failures that we didn’t, we got down the list until every server was back up and running, and marked the customer tickets as Resolved.
And that’s it — every box was moved, every one checked, and everyone could go home at around 5.30am. Well, it was nearly that smooth…
What went wrong: The “Never Events”
What we were trying to avoid was taking customers offline unexpectedly. Every customer was notified and agreed to be taken offline for particular time, or we made some other arrangement with them.
But in the two big moves so far, we’ve made four errors. That is, boxes that should have moved didn’t, or boxes did move that shouldn’t have. In both cases, we could make quick network fix-ups to get them back online on the same IP address, but it took a puzzled customer to notify us.
A British hospital would call these “Never Events”, things that should be optimised away at any cost. So we’ve gone back to the details of our communication between teams to ensure the same thing can’t happen, at least not for the same reasons.
Once the servers were all racked up, hooray, we’re nearly done! thought everyone. When we’re looking to get them all booting, there is a human tendency to go through the long list of servers, fix the easy problems. That shoves the non-obvious problems to the end of the night.
But that’s a mistake; by doing the easy servers first, we didn’t spot systemic problems e.g., a network misconfiguration in York affecting several servers. And fatigue makes trying to solve those problems much worse.
So we needed (at least) to ensure that we avoided our tendency to tick off easy cases, and to dig into deep problems sooner.
We were also considering a third wave of system administrators to start work at 3 am to look at these fresh, allowing the earlier move team to go home. That team might spot those systemic problems more quickly, but at the risk of spreading our team thinner. We’re still deciding what to do for the next move.
What went wrong: Wasting time
All our customers should do a test reboot before being moved, right? We emailed and we emailed and said pleeeeeease reboot, just once. But we ended up…
…a little disappointed at the number of people who didn’t perform that test. We don’t hold it against our customers of course!
But it’s proof that we should be trying to annoy people into doing it, because flushing these problems out beforehand can prevent issues with customers’ machines and reduces down time on the night. I think our reminders could be improved, and I can risk being more annoying where we know it hasn’t been done yet.
Finally we did spend a two hours sweating over a broken server that had been cancelled by the customer years earlier. No customer suffered, but palms hit faces and fists were shaken at the sky. That was just a reminder to check over our cancellation records.
We’ve done 2 out of 5 big moves, and I’m pleased that we’ve managed to co-ordinate this without any panic from staff or customers. (that’s not to say no panic, but it’s all been isolated and short-lived).
We still have to add further discipline, communication, and double-checking to ensure these “never events” are gone from the next move. As we build out more and more racks, we’ve got some ideas for tooling that could make these moves even easier and more efficient.
And that’s it! Many, if not all, gory details of our server moves, hopefully the last big ones we’ll need to do now that we own our own place.
I’d be interested to hear from any affected customers, or those who’ve experienced server moves with other ISPs – feel free to comment or talk to us @bytemark on Twitter.