Posted by Matthew Bloch
Sat, 25 Oct 2008 23:23:00 GMT
I got into this business for the software - I still like to write and direct software, but it has personally taken 4-5 years to start making progress on the software plans that I was really interested in. That’s because when Bytemark became profitable from our early efforts, the customer service really started to eat into the time I wanted to do development. But I realised that if we wanted to make a success of a hosting business, we had to commit to supporting our customers first, and developing new products second.
So for years we carried on answering difficult customer questions and maybe getting one day a week of difficult development done. But it paid off - we carried on rolling up new customers into a big ball, and made the smaller development time count. It’s taken years to be profitable enough to employ people to increase that development time, but now I’m able to look at what our next products will be.
It’s probably not a surprise that I’m focussing again on our virtual machines, and where to take them next, so it was a minor distraction to find that in the last few years, virtual machines seem to have acquired a new name: they’re not machines, they’re The Cloud (cue angelic chorus, ah-ahhhhhhh sound). The Cloud is "computing on demand". You only pay for the resources that you use, and cut your costs in the process. You can choose your own Cloud Computing provider and skip between them, hosting becomes completely interchangeable. Add resources on demand, drop them on demand, pay by the hour. And there are lots of people selling you this already. Hooray for the future! It’s here! Not at all like 1960s mainframes! I like the plan, but think the dream is further away than it looks, and that most of the work hasn’t been done yet.
Taking a shot at the cloud
Why’s that? Well firstly, an uncertainty with any hosting product is that the vendor tends to have no idea how fast their products will go; the typical answer is "fast enough for most things, sir" in the hope that the customer trusts them. On the other side, application developers also have no interest in benchmarking and are willing to accept the "fast enough" answer. Rarely, a developer will ask, "well how can I really measure what I’ll need?". I don’t always have a great answer, but I tend to point them at siege, a simple benchmark tool. It hammers their application in the same way that lots of visitors do, which allows the developer to set sensible limits for the number of visitors that the site will handle on particular hardware. Sometimes they’ll even use it.
This kind of wooliness is tolerable when you’re more than likely paying for some headroom in your hosting, but that’s not The Cloud! In The Cloud, you pay for *exactly* what you need, when you need it, because you’re saving money. Do you see developers coming up with benchmarks for their hosting providers? Do you see hosting providers advertising benchmarks to potential customers? No, nor do I. It’s still "faster than you’ll need" from the hosts, and "not sure about demand" from the customers. Actual decisions are made on reputation of the hosts and comparisons of the type "Joe is paying $100 for hosting X, and your application won’t do half that". You wouldn’t tolerate that from an electricity supplier, but it’s still OK with hosting because most people don’t (yet) pay for it by the hour.
When it comes to virtual machines, there is tons of competition in the $10-30 per month category, and almost all of the products are "good enough for most things". Lots of static sites, a single moderate use database application, thousands of mailboxes, life is pretty good for customers at that end of the market. Cloudy offerings compete pretty well at that end, but aren’t really any different, other than the promise of "scaling well" if the time comes.
Up through the atmosphere
The problem with The Cloud based on a few current offerings, is that the moderate-spec hosting becomes expensive, and the high-spec hosting is eye-watering. Take the example of one of our customers, a popular forum which gets regular mentions on morning television and has hundreds of people logged on and posting messages at a time. Right now the web node runs comfortably on a 4-core AMD Phenom 9550 processor with 8GB RAM and mirrored discs. Here’s how much it would cost to commit to those resources based on a price-comparison at various "cloud" providers, from cheapest to most expensive, per month:
- Plain old dedicated server with 200GB / month allowance £120
- Amazon EC2 (high spec server, 7.5GB, one core, excluding transfer fees) $288
- Flexiscale (assuming 50GB storage and 100GB / month transfer) £601.70
- Joyent Accelerator (two cores) $1000
I’ve picked on three companies who offer smooth "cloud" scalability promises, but I’m sure you can find others, and just looked at their advertised prices.
That’s not a "fair" comparison because the customer won’t be using all the cores and all the RAM all the time. Surely the Cloud providers should win by allowing the customer to dial up and down according to demand? The question is: who turns the web site power up to 11 when it gets mentioned on GMTV? Not the customer; they might be lucky enough to know when their spikes will occur, but not how far to turn the dial to be *sure* their web site stays up. Not the hosting provider, after all they’re just an on-demand utility. Even if they did, would you trust them to only give you what you need, and not simply make the most profit from a spike? And the customer’s application (along with 99.9% of other packaged web applications) is a PHP program that doesn’t know how or when to order a new server. So the typical hosting customer is on their own with managing demand for their "on demand" service, and will simply err on the side of paying for more than they need. If you’re doing that and need this amount of power, you may as well pay for a dedicated server.
To get those prices in perspective, the processor, board and RAM for this server cost £200 retail, and might draw £20-50 per month in running costs. Our £120/month price for a dedicated server is driven by the obvious concerns: the cost of rack space rental, power draw for the server, spare parts, support staff and the fact that about a zillion other companies are doing the same thing. I’m not that I’m claiming ours is the lowest price in the world, just that it’s cheaper than any Cloud computing provider is currently willing to offer, even Amazon as the oldest and best-established.
The last piece
What’s missing from the jigsaw is competent open-source applications with "cloud support" baked right in, applications that can say to their hosting platform "I’m struggling to fulfil demand on this host, order another server, and start serving customers on that too". San Francisco start-up types are adding this simple trick to their next-big-thing application, to make sure it stays up, and it really will save them up-front investment. Of course it will be at the cost of some programming wizardry, and they’ll still pay through the nose if their site takes off, see above. But compared to most of our hosting customers, mostly time-pressured developers and entrepreneurs who have something other you do than work on infrastructure, you do need to be a wizard to implement that kind of early economy for your own project. And it would be a Herculean effort to write a "cloud-aware" application that anybody could deploy, even more so if you wanted compatibility across more than one Cloud provider. The standards just aren’t there yet, so typically your cloud-aware application can only run on one Cloud. And if you want to test it outside of the single commercial Cloud you’ve written your for, you’re out of luck - none of the providers have made their provisioning software public.
The Cloud will need much more than the hardware and APIs to create reliable virtual machines; it’s about a programming framework too which has barely got started, and one which needs to be agreed among fast-moving competitors. While programmers are still writing applications where "throw a bigger server at it" is the only supported answer for scaling, The Cloud will still require expensive wizards to fulfil its economic promise. Whether those wizards sell you the application in the first place, or promise to adapt your application to The Cloud, or run a Cloud on which you host your stuff, you will be dependent on them for your application to run. And while you still need a wizard involved, it’s just not for everybody.
The older, unfashionable alternative remains the cheap one for now. If you think your application will scale, and are able to think ahead more than about two days at a time, call your hosting provider, tell them about your application, and ask them about a traditional scaling strategy with real or virtual boxes - for a few more years at least, real people thinking ahead and doing the provisioning can still beat The Cloud on economy.
Back to putting my development head back in the, umm, where it was.
no comments
Posted by Matthew Bloch
Sat, 04 Oct 2008 14:28:00 GMT
At Bytemark a couple of thousand people per month need to pay us money. We don’t offer barter or cash as payment options, which stops folk turning up to the office with livestock or pieces of eight (except sometimes on Talk Like a Pirate Day). The reason is that we’re only a staff of seven and we would probably spend all of our time meeting customers, offering them coffee, then assaying their bullion or sizing up the hind quarters of their cattle and taking them out to pasture in the car park. We did once receive a grubby tenner in the post in payment for a domain, but that’s as close as we came to actual physical exchange. The advantage of exchange is that, well, you meet your customer, you see what they’re offering for your services and you can decide on the spot whether you want to take it or not. If their coins are too clipped, or their sheep lame, well, you send em away, no sale.
Of course, for economic sanity, we take card. But in these days of credit crunch, we should probably know what that means. And I think it’s amazing it still works at all. The card system was invented by a New Yorker who forgot his wallet once at a restaurant, and promised to pay them later. Initially Frank McNamara issued Diners Club cards to 200 salesmen to spend in only 14 restaurants; the salesmen flash them at waiters, who would write down the card number, and the amount of the bill, then claim back the total balance from the Diners Club corporation at the end of the month, who would in turn bill their cardholders. Security must have been nice and easy - the embossed piececs of plastic were hard to come by, and the types of people who carried them were easy to spot - white, male, dressed-to-impress sales and executive types.
Despite trumpeted advances in credit card security over the last 10 years, this rather gentlemanly payment model is the one we, and our customers, rely on, to pay us for services, and the one which we’re going to have to keep relying on for the foreseeable future. We ask the customer for their secret numbers when they sign up. We ask the bank, every month, whether we may please have £15 or more to pay for that customer’s service, quoting their secret number back to the bank. Assuming the secret number is correct, the bank gives us our money and the customer’s account is paid.
The trouble is - thieves. They hand over other people’s card numbers to pay for our services, and the bank will give us their money without question. Bastards! We’re not a fancy New York restaurant, and the customer isn’t looking us in the eye and saying "a delightful meal, sir! Please bill my Diner’s Club club account!" So we don’t have the visual clues to size up a rogue. If it were a hypothetical Oriental or negro gentleman asking our hypothetical 1950s waiter to pay by Diner’s Club, the waiter might chuckle politely, make eyes at the manager, and firmly ask for cash - because those people couldn’t possibly be local salesmen (I would love to hear from any non-white 1950s New York salesmen, or waiters to confirm or deny this odd picture 
). At Bytemark, we know our customer profile just as well, and will check an order far more thoroughly if the customer claims to be a woman in Arkansas than if a network consultant from Leeds. It’s unpalatable in principle, but the shopkeeper’s best defence has always been to know his likely customers, and be wary of "strangers".
But fraudulent purchases isn’t our biggest concern with taking cards. Despite chip and PIN, CVVs and other security changes in the card system, it’s us merchants who need "continuous authority" that are the card industry’s biggest risk! That’s because we need to be able to store and re-use our customers’ card details so that they can pay for small items without handing over their card number for every few pounds they want to spend. The thieves we need to worry about are the ones who are targeting our customers’ account numbers, which our business model obliges us to store.
If those numbers got out en masse, the banks would easily spot that it was our fault because they would be able to correlate the charges made by us with fraud reports reported by our customers. They could potentially take away our ability to take payments by card in this scenario. However, a quiet penalty charge would be more in keeping with the card company business model, as overall confidence in the card system is more important to them than real security. When mine and my partners’ business debit cards have been included in a compromised haul of card numbers, we spot the fraudulent transactions and report them, getting our money back quickly. However the banks never tell us which merchant was likely to have led to the compromise. They might have identified and fined them, but they don’t want us to worry to whom we should or shouldn’t trust our card numbers.
As an aside, it would be better if the banks would protect their customers with fundamentally more secure cards. They could allow customers to make up new account numbers which worked for particular merchants, or for only a certain sum of money, to contain the damage from any data security problems. Instead they have gone for the "stick" solution: fines and sanctions of merchants who store credit card information improperly. Now that credit and debit cards are fundamental to many business’ revenue streams, they don’t need to spend on further security themselves, instead forcing the risk and cost out to the merchants.
Merchants are told how to store and process these lists of secret numbers- it’s called the Data Security Standard. It was initially a set of recommendations, but compliance is now a condition of getting a merchant number (at least from Barclays). If you store cards as we do, and haven’t planned how you do so, it might entail a complete overhaul of your company’s IT infrastructure and policies (which, given the financial risk to their customers of a compromised card database, is the only sensible option). It is arguable that full compliance is impossible in all but the most dogmatic bureaucracies, and the actual check for most merchants is nothing more than a port scan which costs you £100.
Our secret to complying with the DSS is segmentation! And this has just been recognised and recommended as a valid technique for compliance in the hot-off-the-press new version of the document (release only a few days ago). Now I’m going to talk about our card processing network and I really think it’s one of the best ideas out there - while I’m honest about its shortcomings and current (technical) non-compliance I should point out that it is secure in all the important areas, and far better than similar small businesses. We don’t store cards in a Microsoft Access database on a machine on the boss’ desk. We don’t store them written down in a filing cabinet. (both of these are examples from real companies - probably not the worst ones out there)
We have a whole isolated network dedicated to processing credit cards. It has one computer on it. It has a dedicated firewall box, the computer’s data is encrypted with keys held only by the company directors (so we need to be present to boot the thing up), kept physically in a locked cabinet, in a locked room, guarded by dogs. If we had the room to dig a moat, it would be full of sharks with lasers (I wasn’t joking about the dogs). All for one server, and a few kilobytes of precious data!
The software I’ve written is a "roach motel" credit card database. It allows us three operations: 1) store a new credit card, 2) update or delete an existing credit card, 3) charge one of the cards. The one thing you can’t ever do is read the card details back again. Not even staff. The system reads them only to transmit them to our payment services provider (then it’s their problem what to do with it), and that leaves over SSL.
Within a few of weeks, when customers call up on the phone, we will have finished the last couple of parts. One is a wireless terminal for entry of card data over the phone (an old Nokia N770 tablet device). That way when customers call, we don’t even trust the web browsers on our desktops to enter credit card data for an instant - we find the terminal in the office, and enter the secret numbers onto it, which communicates securely back to the card processor. And when customers enter credit card data themselves in an order or on our control panel, they will communicate directly with the payment processing machine, just their computer talking to our server with nothing in the middle.
So from a compliance point of view, our hands are clean, and the part of our network which processes the secret numbers (PANs, Primary Account Numbers, as the standard calls them) as small as it possibly can be. This is all thoroughly economically justified paranoia (and fun too) - as we grow and process more card transactions, the audit requirements for compliance to the Data Security Standard become even more onerous and expensive. If we were regularly entering card details through our normal web browsers for instance, the auditors (when they doubtless arrive) might say, what virus checkers are you running on these? And we might say, we’re not because these are Linux machines and we don’t know of any viruses that would affect them - look, here’s a company selling virus scanning software, why aren’t you buying it. And they would say, well, we had better check all your machines for non-existent viruses just to make sure, and here is a bill for fifty thousand quid by the way. With a small network, we can tell the auditors to keep their noses out of the rest of our network, and we can be surer that our card storage is really safe.
The only thing this leaves is - the software! I’ve written it myself. It’s in production already, and has been for nearly a year. It is quite paranoid about its input, and is guarded by the excellent pound URL filter, as well as an upstream firewall. The only trouble is, it’s custom software, and we’re the only people using it. The solution will definitely be for me to open it up to public scrutiny, to make it more popular for other companies to use and build confidence. However I’ll also be investigating the possibility of a paid audit procedure, with a view to it becoming suitable for a "tier 1" payment processor, i.e. the kind where the banks want to send in the men in black to check your company out.
For the moment if anyone else is interesting in helping with the software as-is or has implemented something similar/better for their own company, I’d love to talk to you and show off what we’ve done so far. It’s hard to find anything on the internet about this sort of thing in practice because I assume people are embarrassed about their homebrew card processing systems, or worried about non-compliance. I will take the leap and open up our software, but I am going to spend a few weeks giving it another coat of paint and some better tests first.
2 comments
