If you’re a Linux nerd looking for a new challenge, your attention is kindly drawn to our recent job opening for a system administrator. The successful candidate will help out our managed & unmanaged customers, and work in our Manchester or York offices. As usual there is a little screener question to keep you amused. Full details and application information are this way.
There’s a new release of BigV’s client software – hurrah! So you are now able to “log in” or regenerate your profile on more than one machine. You can change your password rather than have to copy your .bigv folder around when (re)installing. The SSH key management is back, so you shouldn’t keep getting those prompts about host keys. You can create a VM with a CD-ROM ready inserted to speed installations.
More significantly you can also now create and delete usernames for other people – this means:
So if you have friends who would like BigV accounts, you now have the power to provide them with one.
There were also a two things I’d forgotten to document: did you know you can incorporate bigv invocation into other programs through YAML and batch modes? Or that there is an interactive mode which saves you entering your password every time you run bigv? They were always sat there in the interactive help, but I’d not written them up until now.
Update instructions are as usual: if you’re using Debian or Ubuntu you can “apt-get update; apt-get install bigv”. Everyone else, go to http://bigv.io/download to grab the latest tarball or installer. There are some Redhat / Fedora / CentOS RPMs that I’ve not fully tested but I hope to post in the next fortnight. Thanks to Jamie Nguyen for doing this work for us.
Finally, Bytemark is turning 10 years old this year; I should soon have good news about the end of BigV’s beta period, and a launch this summer.
Bytemark are hosting the 22nd UK Network Operators’ Forum meeting here in the Merchant Taylors Hall in York, on 3rd May. There are plenty of places left for the triannual “redistribution of clue”; so if you’re into network operations in the UK, please register! There will be talks and tea in the day, dinner and drinks at night. And maybe a teeny tiny pinball tournament if we take a wrong turn.
Here’s the paper I wrote for last week’s FLOSS UK Spring 2012 conference. I don’t really do slides, just scrolling through text and pointing at diagrams. But hopefully this is more useful afterwards – the paper spills all the details on how we’ve been implementing BigV, our excellent new hosting system. Have a read, and I’d be happy to answer any questions in the comments.
I’ve also dug out the paper I did for the UKUUG 2004 Bournemouth conference on what we were doing with User-Mode Linux. There’s actually plenty there that’s still relevant to planning a virtualisation system – everything from TAP interfaces to economies of scale. Both papers are a combination of half-imagined technology & business, but we did OK the first time round, so I’m thinking we have a shot with BigV!
Also at the UKUUG conference, I was fascinated to learn of Iustin Pop’s work on Ganeti, a VM supervision system developed at Google. If you’re looking to deploy a cluster of VMs on your own hardware without the bureaucracy of Openstack or Eucalyptus, I’d suggest you look at that; I certainly will be. If you do just want hugely flexible VMs with up to 8 discs and 120GiB RAM on our well-managed infrastructure, BigV is clearly where it’s at 
Here are the two papers:
0.5.2 is out! It fixes quite a few bugs and will prompt you to remove your plain-text password from the bigv config file (sorry, that was the default until now).
I’ve documented a slightly clunky method of installing FreeBSD on BigV which we’ll replace with a proper image shortly.
Another feature snuck in to the last release which I’d not documented until now was compatibility mode, which lets you run OpenBSD, Solaris and other operating systems which don’t support qemu’s efficient “virtio” hardware model.
Finally you can now undelete accidentally deleted machines within 45 days of deleting them – see my notes on vm undelete.
0.6 should follow reasonably soon which will allow you to change your password, set up more easily on multiple machines and manage your IP addresseses.
You can update with the usual – either apt-get update; apt-get upgrade on Debian / Ubuntu. Or go to our download page and follow the instructions for your OS.
If you’re a follower of the Bytemark blog you might have spotted that I’m moving the BigV diary posts over here, and some old posts might have shown up as new. Sorry!
What I came here to say though: Bytemark are sponsoring the FLOSS UK conference in Edinburgh next month, (20-22nd March), and I’m proud to be presenting a talk on the Design and Implementation of BigV, alongside lots of other interesting speakers. If you saw my talk in Bournemouth in 2004, this is the sequel you’ll no doubt have been waiting for. I’ll talk through some of BigV’s implementation, especially the bits that are easy to copy in your own setups. If you didn’t know, FLOSS UK is the new name for the UK UNIX User Group, and they always put on an interesting 2 days of cross-disciplinary technical talks. I’m hoping my talk will measure up.
There’s a new BigV release out now – documentation and some more news on the beta test should follow later tonight.
I’d be really happy if you could give the catchily-titled 0.4.1-1427 BigV client a go.
The main addition is that we have Windows support, so if your usual computer is Windows-based, there’s now an installer and command line for you.
You can also now install BigV servers running Windows Web Server 2008R2. There’s no charge during the beta period (even though it costs us), you just ask for the catchily-titled “winweb2k8r2″ distribution when you next run bigv vm new.
You might also want to read about the new privilege system.
This allows you to delegate some account management privileges to other BigV users, though at the moment a few features are missing to help you manage those privileges.
Finally please note that each account has a limit of 32GiB RAM and 500GiB of disc space – total. If you need more please ask, being sure to specify the account or machine that needs it.
For customers that are using BigV in production and need urgent assistance, you can now email support-urgent@bigv.io for 24 hour help, though as with our normal support channelsplease use it only when a production service is affected.
The commentary around Bulldozer, AMD’s latest processor line, is that it’s disappointing, a catastrophe, absolutely positively awful and so on, for miles and miles. And it’s a shame for AMD that Ars, Extreme Tech and the usual suspects have no imagination beyond their benchmarks when it comes to judging processors.
The numbers don’t lie of course – the benchmarks show that Bulldozer’s best is slower than Intel’s, by a long way, and most of those benchmarks care about absolute amount of data processed, numbers crunched and so on. Ars Technica concludes:
AMD compromised single-threaded performance in order to allow Bulldozer to run more threads concurrently, and that trade-off simply hasn’t been worth it.
But everyone’s performance tests make an assumption: that the processor is working for one person at a time, and that person wants it to crunch through as many numbers as possible for their benefit. Those numbers might be rendering frames of Battlefield 3, editing a huge photo or ripping a CD. Benchmarks sum it up the same way – however many ways a single job can be split to use a processor’s cores, they’re only interested in who gets to the finish line fastest.
And Intel still wins, I get it. Even AMD admit in a recent interview that their older processors still perform better for a lot of people:
We understand our customers make purchase decisions based on how they use their PCs, and in many cases our AMD Phenom™ II processors are a great (purchase).
They are struggling to pitch themselves against Intel to the gamers, power-hungry desktop PC users and the benchmark sites. They know they can’t get the same excitable press any more, not for years and another generation of processor. That’s probably why they’re temporarily giving up this fight on pure brawn, laying off hundreds in their PR and marketing departments a couple of weeks ago.
But instead of doing one thing for one person, let’s instead assume that a processor is under siege from 300 warring factions, all wanting to run separate and unpredictable work loads. The benchmark that interests us is: assuming we are running 299 “hostile” jobs, how quickly does that 300th job complete? If we vary those 299 jobs in nature, how reliable is the performance of that 300th? The time of that one lonely, slow, job is what I’m interested in.
For BigV where we are running a massively “multi-tenant” system, we really are planning to put in the order of 300 different customers on a single server. It’s far more important to have a reliable average performance for a virtual machine than the absolute fastest possible performance. I’ve not (yet) seen any benchmarks that test in this way, but it seems to us that more separate hardware cores must achieve this goal better than a single software-switched core.
If that weren’t the case I find it hard to understand investment in massively parallel server systems like the 768-core Atom server or super low-power 480-core ARM systems. Big multi-tenant systems don’t need to be the fastest, they just need to perform consistently.
For BigV, we just want more cores. The speed is almost irrelevant. While AMD’s performance is within the same ball park as Intel’s, they’ll work out very nicely for us virtual machine-mongers. And the low price of Bulldozer chips is just icing – a 32-core, 128GB system is extremely affordable and helps us keep our pricing to customers low.
AMD are gearing up for a different fight, and they don’t need press from the benchmark sites to prove them the winners. So while my gaming PC will stick with a Phenom for a while yet, BigV is going to be using an awful lot of Bulldozer chips in the near future.
In the beginning, there was “shared” hosting. And that was all the hosting there was. A UNIX beard would give you an account on their big UNIX server, and set up Apache for hundreds of users to put their web pages, CGI scripts and PHP. The Apache server invented hosting, it still works the same today as it did 15 years ago, and it’s still everywhere.
But demanding customers eventually hit a problem – a shared hosting platform only has one Perl or PHP version on the system, and eventually the UNIX beard has to upgrade it to make new users happy. Older customers suddenly “feel the beard” behind their hosting and their stuff breaks. That makes them grumpy when the beard hadn’t needed to intefere for years. It happens all the time, and gets us a steady stream of new customers.
So for a few years, the solution was to go Dedicated – pick a big company who would buy dirt cheap servers and rent them, whole, by the month. You can ditch the UNIX beard who broke your site, hooray! He also probably backed the server up, helped solve your programming problems, and worried about the server’s up time, but you won’t notice that until your server goes wrong in 2-3 years time.
This is about where we started our business in 2002, and the very clever User-Mode Linux project. We used it to offer dedicated-style hosting at £15 per month, where much larger companies were charging 3 times that price. And because it was grungy and unpolished and you had to be a kernel wizard to use it, the big hosts stayed away from our business a good long time!
We even went into Dedicated when we had the money, because we wanted people to be able to grow beyond VMs (It also helped that Pete didn’t mind driving 400 mile round trip to London – a lot).
Sometimes customers add more dedicated servers, or hire their own beard. But plenty feel that there’s something new in “the cloud”, which means either:
There are no hosting services out there that don’t fall into one of these camps, but it’s a basic distinction as to how complicated your infrastructure is, and who can screw it up for you.
The UNIX beard running his university Apache server in 1995 was the original “platform as a service”. And anyone renting UK2′s cheap servers ten years ago, when they were the cheapest option, were “infrastructure as a service” pioneers.
You’ve got up-to-the minute live replicas of your important data? Backups too? Your install is scripted, and rehearsed such that when your hosting provider goes down, you can simply redeploy elsewhere at a moment’s notice? You use chef or puppet or cfengine and you never make any manual changes to your server, that you haven’t tested and pushed via a source control system? And you’ve got accounts with multiple service providers?
If so, well done, you must be 1 in 1000 hosting customers. You realise that the real cloud isn’t your hosting provider, it’s every hosting provider, and your ability to both use the best of them, and depend on none of them. The cloud is your install script.
In reality, most companies without a dedicated 24/7 operations team just aren’t ready. They bed their services down in a reliable hosting provider, have a critical process or two here and there, graft on a server for a new business function and pay for 24 hour support. Sometimes they can even run their entire business successfully for years on a server, or a single redundant pair, and nothing goes wrong!
And the reason isn’t because they’re slapdash, or lazy, or stuck in the past, it’s because most businesses aren’t Amazon and don’t ever need that kind of scale, and preparing for it reduces your overall reliability and up time.
Maciej Ceglowski, creator of antisocial bookmarking site pinboard.in said this about modern hosting practices in an interview a few months back:
I believe that relying on very basic and well-understood technologies at the architectural level forces you to save all your cleverness and new ideas for the actual app, where it can make a difference to users.
His site runs on a handful of very large servers, and managed to handle a huge influx of users (caused by the bungled announcements around the rival delicio.us bookmarking service being closed) without needing any fashionable just-in-time hosting provision.
And John Kozubik of rsync.net wrote a lament two years ago about the failure of a rival caused by overcomplicated architecture:
When you don’t have stars in your eyes, and aren’t preparing for your IPO filing and the “hockey sticking” of your business model, you can do sensible things like keep regular files on UFS2 filesystems on standalone FreeBSD systems.
This is, of course, laughable in the “real world”. You couldn’t possibly support thousands and thousands of customers around the globe, for nearly a decade, using such an infrastructure. Certainly not without regular interruption and failure.
Except when you can, I guess.
…and illustrates with examples of two of his servers that have been up for 350 and 950 days respectively.
These are the people for whom we built BigV (not literally those two people, that would be a bit stalker-ish).
With BigV we might say it’s built for worst practices first, because we know they last longer than you might want. We know that relying on single servers for a while is more reliable than trying to build for Google-size overnight. We want to make it easy to throw up a server, back it up, back out of configuration mistakes and “push the walls out” when the server gets too small for the load. Right now you can go up to 120GiB RAM and 40TiB disc in a single server, and after the beta we’re not stopping there.
We can do that because we’re a lean company with lots of nice customers, not a huge number of staff, and we have plenty to drop on HP kit. HP do some very nice kit, and we want you to be able to build monster machines, and not be stuck having to glue together 8 or 16GiB machines with a cheaper hosting company. We want you to be able to use private VLANs, SSD storage, huge amounts of RAM, arbitrary disc snapshots, and all the toys that used to need expensive dedicated servers.
It’s not that you can’t build a minute-by-minute scalable cloud with lots of CPUs if you need them – you can! Look! But we recognise that most users won’t ever need to, and so we’re trying to replicate that “zen garden” isolated feel of a Dedicated server, without any of the overheads. We’re still doing the “big cloud” stuff, but looking at it from another angle: that you’ll start your site small and simple, and only maybe need to grow it big and complicated.
Our beta is still open for signups, and we’re both expanding our cluster and accelerating the pace at which we send out V-Keys. We’re looking forward to seeing what our customers build when they can shake off the vanity of scaling before they need to, and deliver their sites with a single cloud hosting company they trust.
For the foreseeable future, Bytemark will be helping to sponsor the Open Rights Group with as much hosting as they need to support their many and important campaigns for civil liberties in the UK.
The Open Rights Group have been campaigning for 7 years, funded by 700 seed supporters who joined up under the pledge: “I will create a standing order of 5 pounds per month to support an organisation that will campaign for digital rights in the UK but only if 1,000 other people will too.” (they had some leeway in that target!) Today the Open Rights Group have full-time staff and make a major contribution to the national debate on digital rights, putting journalists, campaigners and policy makers in touch with one another.
As a leading hosting company with a direct interest in the digital economy, we at Bytemark know that the Open Rights Group‘s campaigning works in our interests, and those of our customers. They led the charge against the the pointless Digital Economy Act, and exposed its many holes. They maintain constant vigilance against government snooping, and they continually oppose the entertainment industries’ demands to shackle our digital futures in the name of preserving profit. Their work is continual, important to lots of UK internet users who’ve never heard of them, and supported only by donations.
We’re glad to be associated with the Open Rights Group through our contribution, and look forward to working with their technical team.