The router is happier, Cisco support is helping.
Ironically, it seems that netflow (a protocol for traffic analysis, which we’ve been using to help figure out what might be going wrong) might be part of the problem.
And I think I mentioned this in a comment on another article, but amidst all the other stuff that went wonky after the power was restored, the router that’s being problematic now lost its active supervisor card (which is basically a router’s CPU) early yesterday. The hot spare supervisor kicked in just like it should have, but just another fun little heartstopper!
More word when we have it!
September 14th, 2005 at 11:30 am
Things to be working great exept for mysql taking 4-ever to work. Taking over a minute to get back the info from it. Any idea when that may be up?
Thanks
Phil
http://www.lowdowncustom.com
September 14th, 2005 at 11:38 am
Same problem here. At least our non-DB driven pages are loading now, and I’ve got a big red notice on our online shopping main page that says we appologize for the problems.
- Roach
September 14th, 2005 at 11:41 am
Unfortunately all my sites use the db and so everything for me is still slow
September 14th, 2005 at 11:54 am
My website also uses a db and is slower than a dialup connection…
September 14th, 2005 at 12:00 pm
That’s weird because I’m having latency issues to the site, but the DB seems to be acting normal.
September 14th, 2005 at 12:02 pm
we’re seeing 40ms of latency and sustained 10% packet loss between our webserver and our database server. Beyond that … database query response time is … horrible, even if you’re on the mysql console. 26 seconds to return a query that would normally take, oh, less than a second.
- Roach
September 14th, 2005 at 12:14 pm
Ping from a DH web server to a DH db server:
PING (66.33.194.154): 56 data bytes
64 bytes from 66.33.194.154: icmp_seq=0 ttl=63 time=35.1 ms
64 bytes from 66.33.194.154: icmp_seq=1 ttl=63 time=34.6 ms
64 bytes from 66.33.194.154: icmp_seq=2 ttl=63 time=38.7 ms
64 bytes from 66.33.194.154: icmp_seq=3 ttl=63 time=34.9 ms
64 bytes from 66.33.194.154: icmp_seq=4 ttl=63 time=34.2 ms
64 bytes from 66.33.194.154: icmp_seq=5 ttl=63 time=34.7 ms
64 bytes from 66.33.194.154: icmp_seq=6 ttl=63 time=34.3 ms
64 bytes from 66.33.194.154: icmp_seq=7 ttl=63 time=34.7 ms
64 bytes from 66.33.194.154: icmp_seq=8 ttl=63 time=34.8 ms
10 packets transmitted, 9 packets received, 10% packet loss
round-trip min/avg/max = 34.2/35.1/38.7 ms
September 14th, 2005 at 12:23 pm
Is nimitz totally down?
September 14th, 2005 at 12:37 pm
I can ping nimitz from bourbon, but if I try to telnet to it, it doesn’t respond – you never get a login prompt.
And the database problems aren’t getting any better. If anything, it’s getting worse. I’m getting timeouts now so our ability to sell things on the web is now pretty much nill.
- Roach
September 14th, 2005 at 12:56 pm
My actual webserver seems to be fine, but the MySQL server is indeed at dialup speeds or even slower…
September 14th, 2005 at 1:16 pm
All my sites are db driven so they are still unusable and timing out.
September 14th, 2005 at 1:18 pm
My sites are still sloooooooooooooow! What the hell is going on? It’s been like what? Two days?
September 14th, 2005 at 1:27 pm
Ours as well (Can I say that without it getting deleted?)
September 14th, 2005 at 1:28 pm
Ours as well. And it doesn’t seem to be improving. (Can I say that without it getting deleted?)
September 14th, 2005 at 1:51 pm
Still going very slow over here (Belgium 11 PM CET)
September 14th, 2005 at 1:52 pm
It is getting annoying now. I am getting timeouts to my domains. Any chance of an update?
September 14th, 2005 at 1:56 pm
Yeah, our site is still very slow also…I hope ya’ll are still working on it.
September 14th, 2005 at 2:37 pm
My site loads OK but any part with SQL loads slow like forums etc…. Hope they find a fix soon some of my visitors are getting annoyed along with me…. as slow loading site means slow moving income…..
September 14th, 2005 at 2:40 pm
My site revolves around phpBB, so it’s slow as well.
I backed up the MySQL database via file download today and was getting only 20k/sec down, where normally it’s over 300…
September 14th, 2005 at 2:50 pm
our DB is also super slow!
September 14th, 2005 at 2:52 pm
Everything I have is working great, even the databases. I guess my databases are on a different server from everyone else experiencing problems.
September 14th, 2005 at 3:06 pm
Anything having to do with MySql is creeping. i use a lot of Moveable Type, and it keeps timing out. can’t update my site very well at all. :( i know you guys are working on it. Hopefully things get worked out sometime soon. What a headache situation you guys must be going through. Ugh.
September 14th, 2005 at 3:15 pm
DB access is still dismally slow. My forum peeps are starting to bitch mightily. Nimitz is the server. :(
September 14th, 2005 at 3:16 pm
I know you guys are working hard, thanks for keeping us up to date, hopefully we can get this issue resolved, I can only try to understand the pain you’re going through dreamhost… although my ecommerce site is slow… i just know dreamhost is trying their best and that’s what matters… hopefully DH gets the power issue squared away after this fiasco… good luck DH!!
- a very happy customer
September 14th, 2005 at 3:19 pm
Last update was 10:24 AM probably California time. It’s 5:14 PM in Dallas and my system response time has not gotten any better. What is actually being done? I have not gotten an update since the one this morning. With all the people I’m sure are scrambling, could you designate one person to post an update as to what action is being taken? Does this problem move up a notch in priority? Somebody………… anybody………
September 14th, 2005 at 3:48 pm
I was about to add to the nag session, but things seem to be improving in the last few minutes. Fingers are crossed.
September 14th, 2005 at 4:19 pm
Yep all my database driven sites are quite slow. Thanks DreamHost for working your asses off for us!
September 14th, 2005 at 4:22 pm
Yeah, all my DB-driven sites are abysmally (sp?) slow. Visitors are starting to complain on my sites as well.
September 14th, 2005 at 4:44 pm
Same situation for me slow slow slow on my MYSQL driven sites. I am sure they are working on the issue and hopefully it is resolved SOOOOON.
September 14th, 2005 at 4:50 pm
Yes, my diary is also very slow. It’s unfortunate because my self-indulgent personal thoughts mean so much to so many! I am serious!
September 14th, 2005 at 4:55 pm
They’re working on it. Believe me, the company I used to work for resold web hosting. We had our own dedicated servers, and it was a pain then for about 20 domains. I can’t imagine 120,000! One bug and you have 512 emails. Granted it may be slow, it’s still not “down” (in most cases).
September 14th, 2005 at 5:05 pm
my site seems to improve now
i am using vbulletin and with the help of speed up performance, the site loading okie but still slow though
anyway, hope they will get it resolve soon
September 14th, 2005 at 5:27 pm
Robert makes a good point. I should have included in my comment that I actually love DreamHost–just to make clear that I wasn’t trying to come off as extremely disparaging.
September 14th, 2005 at 5:44 pm
Yeah, it is frustrating, but where else do you get such a full feature service with ssh for the price. Perhaps they should put some google ads on the emergency page- would pay for a new router ;)
an update would be nice though
September 14th, 2005 at 5:53 pm
Not to be Captain Obvious to all you people here, but they are WELL aware of the problems. Everyone likes to bitch about all their problems, but thats not going to make them work any faster then they already are. This wasn’t your usual power outage, and I am sure there are residual effects even after stuff is up and running that you have to deal with. Hindsight is always 20/20. Instead of giving them complaints, why not commend them on the excellent job that they have done so far in promply getting all the processes up and running that they could safely. It takes alot of work to run a hosting company.
I personally commend them for what they have done, and their quick response in trying to get everyone going again in the safest possible way.
September 14th, 2005 at 6:08 pm
i’d personally give DH a beer right now if i could… it’s stressful not getting any sleep i’m sure for some ppl over at DH.. keep up the good work fellas!!
September 14th, 2005 at 6:40 pm
Looks like speeds are starting to return to normal. WTG DH.
September 14th, 2005 at 6:42 pm
Looks like things are getting better. Keep up the good work guys!
September 15th, 2005 at 1:36 pm
All things are working great now!
Thanks!
Phil