Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

General question here about upstream outages that can take down your site, e.g. DNS outage, AWS zone outage, etc.

What to tell your customers when an upstream service provider experiences an outage? I mean, if you're running ifttt.com your users might be savvy enough to understand that a DNS outage isn't your fault; but pinterest.com or whatever (painting with broad strokes here, forgive me) might not have a user base that would understand that events out of your control have made your site inaccessible.

How do you reassure your customers? What's the proper tone to take?



Good question. I believe in explaining as clearly as possible, but ultimately in taking responsibility. You chose the provider, decided on the level of risk you're comfortable with, and that ultimately failed. It was that choice that got in the way of your users.

If you can point to ways you'll improve the service in the future as a result of the outage, all the better.

When we had a large DoS attack at Posterous, I wrote two posts, one as soon as possible (http://blog.posterous.com/todays-outage-and-changes-for-cust...), and the next as a bit of a post-mortem (http://blog.posterous.com/moving-forward). Both explained that there were many factors beyond our control, but that the responsibility was ultimately ours, and we were working to learn from the event and improve our services as a result.

They weren't perfect posts, but I think they went a long way toward being open and honest with our users in the midst of a major negative event.


At the risk of professing an attitude that I generally dislike: there's no proper tone to take other than full responsibility. Like Steve Jobs said, when you're the janitor, reasons matter.

Once you're a service provider -- whether your services in turn rely on other services or not -- reasons stop mattering. As a practical matter, you and I know that there's no way you can build a fully scalable, fully redundant infrastructure from the ground-up in your first week. Hell, if you ever build that kind of infrastructure at all, you'll be way ahead of most companies.

But, that's the kind of infrastructure you should be working towards building, all the time. You should have a clear roadmap for ensuring data integrity, then dealing with security, then dealing with redundancy, and finally high availability.

If your service falls over for any reason, ultimately it's because you haven't done something on your roadmap yet. There's no way to explain that to your customers that doesn't sound like you're trying to pass the fault on to someone else -- because that's exactly what you're doing.

So just 'fess up to your customers: "one of the services that our business relies on had some serious technical problems that affected us today, but we recognize that ultimately it's our responsibility to make sure that our service is always available to you. We're constantly working on our infrastructure to make it more reliable, but we clearly still have more work to do. We will be changing some of our priorities so that this won't be a problem in the future. Thank you for sticking with us." (And then do it, otherwise this will backfire on you the next time you have an outage of similar cause.)

As an aside: I generally take a softer stance towards user responsibilities -- of course everyone should have backups, but Joe Schmoe just doesn't have time for that -- but a much harder stance towards businesses. Once you accept money from someone, you put yourself into a position of absolute responsibility for whatever it is that people rely on you for. If you can't guarantee the availability of your service or the safety of their data, then you shouldn't be taking their money.


Interesting, at the same time I both agree and disagree with what you wrote. You're saying that you want to offer 100% uptime, but how much does that cost? A client paying 5 USD/month has the same uptime expectancy as another one paying 2000 USD/month?


Oh man. I think I could write a long essay about this. Before I respond, I should mention that I've been running a business for several years that tries to break the rules of the price-quality-speed tradeoff, we've offered an alpha hosting service for a couple of years, and I've recently seen first-hand how bad it is for both you and your customers if you don't charge enough.

So, my reply in one sentence: a business should never charge less than it needs to meet its customers' expectations.

I don't think I've ever seen any hosted service brazenly advertise, "It only costs $4 a month and it's only down for a couple of days a year!" Instead, hosted services advertise their pricing, and then hide their uptime "guarantee" (in quotes because it's only a guarantee to the extent that there might be refunds involved if they don't meet it) somewhere in their fine print, or as a number that sounds impressive to people who don't know better, like, "99%!"

The problem with that is that your customers still expect your service to work. If your customers build a business of their own using your service -- and if you have enough customers, somebody is gonna try to do that -- then they can be seriously impacted if your service fails. At that point you have an adversarial relationship with your customer. As a business, you want to say, "but you're only paying $4 a month! What did you expect?", but as a customer, that's just about the worst possible response.

I think the race to the bottom in pricing is a really bad idea. GoDaddy's a really good example to use here. How many of their customers do you think couldn't afford an extra $1 a month, and at their scale, how much could they improve their service if they made an extra $1/month per customer? Conversely, how much damage does GoDaddy do to the hosting industry as a whole every time they piss off their millions of customers? I think you have to charge enough money to make your customer happy, and that includes a little extra to make sure your service continues to grow and improve and that you continue to be happy so that you'll continue to want to work on your business.

Once the business is up and running at some price point, and your infrastructure is, let's say, 75% complete, then start looking in to ways to reduce your price without compromising your service. Maybe if you find a thousand more customers, some network effects will kick in and you'll be able to charge everybody a little bit less. Great, go find those customers.

And, let's not ignore that the technology available to service providers right now is amazing. I first wanted to be an ISP in 1995. Back then, your startup costs were atrocious, the technology was unreliable, documentation was opaque (no such thing as howtoforge!), and you had to rely on industry fatcats that would shake with their right hand and shank you with their left.

Now there's Linode, Rackspace, Hurricane Electric, Slicehost, Heroku, Amazon, and a ton of others, all offering easy-to-use, low-cost, reliable services that you can build your business on. It's really amazing stuff. While individually they have sporadic issues, collectively they're rock-solid. So, unless you're offering a service that cracks hashes, I have trouble imagining that it would cost as much as $2,000 a month to provide a 100% uptime guarantee. If you just want to host a particular technology stack for customers, or provide SaaS, you should be able to engineer a really solid service for $50/month, tops.

Finally, I left room in my previous comment on this for businesses that are growing. I don't expect a business to have the perfect infrastructure in place the day that they launch. I do expect them to, at the very least, have solid backups in place and an idea of what to do if everything goes upside-down one day. Then, once they start getting customers, they should focus on improving their infrastructure. If a business is a year old and they have a few hours' downtime one day, I don't think to myself, "Pf, amateurs." But, if a business is two or three years old, and their customers' usernames and passwords just showed up on pastebin because they wrote their web app in PHP and didn't use PDO? Yeah, amateurs for sure.

As a thought experiment, imagine if GoDaddy took all the money that they sunk into Superbowl ads and pretty women and other stupid marketing, and instead put that money towards being the best damn domain registrar and hosting service on the internet. I bet you they could monopolize their market. There'd no longer be any reason at all for any of their customers to use any other service. There'd no longer be any reason for a potential new customer to not use their service.

So, want to own your market? Build an unbeatable service. (Or product.)

And you have to charge your customers enough money to do that.


I don't think there's a tone you can take that will make non-technical users understand that it's "not your fault". To the non-technical user your site is just down and, potentially, a service they're paying for is unavailable.

I'd argue that such an outage _is_ your fault. If you're worried about Customer perceptions as a result of outages of third-party services your site relies on then, I'd argue, you need to have redundancy in your choice of third-party services. If it's important enough to worry about it should be important enough to spend some money on and do something about. If that drives up your costs then your product's cost, to your Customers, needs to reflect that.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: