OSFlash down time and a warning about LiquidWeb
Update: OSFlash is up and running again but I'm looking for a new hosting provider as I don't trust LiquidWeb following how they handled this incident.
Update: Adam emailed me back following their investigation into the matter. Here's what they found:
This seems to be an isolated incident. That being said, I don't think an upgrade at this point is necessary. From the best I can tell, this was cause by an hung apache child process. I'm not able to narrow it down further than that unfortunately.
The big issue here, of course, is they way they handled this incident -- by shutting down the OSFlash server. Instead, they should have (a) restarted the server, (b) noted whether or not the load issue resurfaced to see whether it was an isolated incident or not, (c) investigated the cause of the issue, and only then should they have emailed me with information on the issue and, if necessary, asked me to upgrade.
This ticket has been created to inform you that your server load has climbed high enough to begin affecting the rest of the server that you reside on. Our other customers were affected by this and so, as a result, your virtual enviorment [sic] has been disabled to return things to stable. We ask that you either take steps to prevent this from happening, or upgrade to a VPS2 or higher. Overloading the server is in violation of our ToS.
When possible, please respond to the ticket with confirmation that you've read and understand it. Please feel free to ask any questions you have.
Thanks! -- Severin Systems Administrator
OSFlash has been running happily on a LiquidWeb's VPS server since October 2005 and this is the first I have heard of it taxing the server. I could go on about how angry I got at how LiquidWeb chose to handle this situation -- i.e., by switching off my server without warning me first and allowing me to upgrade to a larger account -- but I'll instead publish the email I sent back to them as it pretty much sums up how I feel:
Before anything else, I state for the record that I am very disappointed with how you have handled this situation.
Turning off my web site without even warning me is completely unacceptable. For this alone, I will immediately begin searching for a new hosting provider as it is apparent that you have no regard whatsoever for your customers. As someone who runs a popular web site, I expect my site to run without interruptions. If I am going over the allocated load or bandwidth, etc., I expect you to warn me and to give me the option to upgrade *not* to turn off my web site.
Now, to the practical matter at hand: Please upgrade my account to the next tier in your structure and please inform me how much this will cost. I will pay for this for the next month as I research alternatives to your service where they do not behave as rudely and unprofessionally as you have done in this case.
In a related matter, I just tried calling your support number (the International one since I am in the UK) and I get disconnected after selecting Technical Support (1) and VPS support (3). This has happened twice. The recorded voice tells me "exiting the system... goodbye." I was calling through Skype.
Also, please understand that everyone on the Open Source Flash community will hear about how you have handled this once the site is back up as I explain to them why their service was suddenly cut. Also, I will blog about this on my popular Flash blog at http://aralbalkan.com.
Important: Please upgrade my account to VPS2 and get my web site running again ASAP.
Also, please let me know what on my web site was causing the server load increase.
I expect that you will deal with this promptly and keep me updated about the status of my web site.
Sincerely, Aral Balkan http://aralbalkan.com http://osflash.org (now down thanks to LiquidWeb)
Some of my friends are very happy with RackSpace, so I am going to have a talk with them. I'd been meaning to move OSFlash over to a dedicated managed server but guess which company will not be getting my upgrade order for that move.
If you value not having your web server switched off without warning, I would strongly urge you to stay clear of LiquidWeb.
Update: Well, it's been over an hour since I replied to their email and there's still nothing from LiquidWeb and OSFlash is still down. Anger level: Orange.
Update: Almost three hours to the minute that LiquidWeb shut down the OSFlash server, the server is back up and running. I got email from Adam Elkins at LiquidWeb support that reads (emphasis mine):
Sorry for the delay. I've turned your server back on, and apologize for any inconvenience this has caused you.
It has been our policy to suspend accounts that cause a server or network to become unresponsive, or have any other negative effect on other customers. I do understand that prior notice would be greatly appreciated, and we often do our best to do this.
Again, our sincere apologies for the inconvenience.
Please let me know if there are any further questions or concerns.
I'm happy that OSFlash back up and running but, unfortunately, "we often do our best" isn't good enough when it comes to giving users warning before suspending their accounts. I still don't know what it was on the server that resulted in the high loads or what those high loads were.
If you have a host that you're extremely happy with and they do managed server hosting, I'd appreciate it if you refer them to me in the comments as I'm evaluating my options at the moment and looking to move OSFlash away from LiquidWeb as soon as possible before there is a chance that this incident is repeated.