What about sql passwords?

“Web panel passwords and email passwords were not accessed or affected. However we recommended in an update email to customers and their email users late yesterday that they reset their email passwords as well, as a precaution.”

Just so everyone knows, this is not strictly true. Dreamhost reset email passwords without disclosure or warning on at least one Dedicated Server hosting account (mine). Tech support took half a day to respond to my inquiries and basically said “Yeah, we did it. Sucks for you.”

I can understand the need for the action on FTP/Shell accounts but zapping several hundred email accounts used by my dispersed staff to run our business without any warning or significant response is pretty weak, especially since we pay about 400% more than the average shared hosting customer on an annual basis for a dedicated box.

So, great communication and transparency for the masses. A bit less so for the small number of us who pay big money.

SQL passwords were not compromised.

Zachary:- some more detail – our systems have stored and used encrypted passwords for a number of years, however the hacker found a legacy pool of unencrypted FTP/shell passwords in a database table that we had not previously deleted. We’ve now confirmed that there are no more legacy unencrypted passwords in our systems. And we’re investigating further measures to ensure security of passwords including when a customer requests their password by email (this was not the issue here, though). Re your shell accounts, I’d suggest that you select a new password just to be sure.

Jason:- apologies that your customer service experience was poor, and for the inconvenience of having email passwords reset as well. I’m not sure why that happened, we’ve been recommending that change but not forcing it. I’ll let my support team know to handle large customers better.

Question on what you mean by storing encrypted passwords:

Are you storing the password? or are you storing the hash(salt + password)?

I’ve been a customer since 2005 and am very happy with your service but I am very concerned about this security issue.

I learned about this on Facebook. I double checked my spam filters and did not see any e-mail from you. Not good. At least now I know why my phone I could not connect overnight (sftp).

I just changed some of my passwords. I’m still concerned. You say you encrypt everything now. If so, how is it that you display the PLAIN TEXT password on my screen after I hit submit?

@ Jonathon Joseph – “So you’d rather DH not change your passwords to hundreds of email accounts”
hundreds of email accounts? I have ~90 email accounts on a shared hosting… :)

I think … The best defense against hackers is to hire hackers!
It’s good that everything is resolved and were not more serious incidents. Love you all. Perhaps finally Dreamhost is the best choice.

Thank you for being so open about this incident.

Thanks for being as transparent as possible about this. One minor suggestion: in the control panel, you could give notifications such as the one regarding this incident a ‘close’ or ‘dismiss’ button. It’s a bit annoying to have that thing hanging in there even after I’ve reset all my passwords.

Just as Zachary Johnson notes above, for years I’ve received various emails and messages from DreamHost that disclose my passwords unencrypted and in plaintext. For years I’ve constantly warned tech support to stop this practice, to stop emailing me my secure information insecurely. Eventually after a few responses I learned that “not all of our customers are tech savvy and so may need reminders of their passwords” and that I could by specifically requesting get them to stop this for my account only, but if I wanted this practice stopped globally I should “submit a feature suggestion”. I did that, it’s largely been ignored for years, and now these types of amateur security practices end up in this kind of event.

This hack didn’t result from the practice of emailing plaintext passwords, but I can’t help but feel that if DreamHost cared enough and was serious enough about their security practices in the first place, then they would have made it policy to neither send out passwords in plaintext nor store them in any database, legacy or not, anywhere, ever.

Simon, you have to wonder sometimes why you bother opening the can of worms when you know there are going to be a few scorpions in there. I know it’s good customer service but about half of these people are just looking for something more to be pissed at above and beyond what their lives have to offer. You screamers out there need to:

1. Stop whining about a single incident in the all the years of perfect service and support DH has given you.

2. Understand that every day a successful company is running that things like this can and WILL happen.

3.Ones that it doesn’t happen to are either not popular and/or no one wants access to them anyway. Or they are EXTREMELY lucky (so far that is).

I opened one the first Internet Service Companies back before there was even GUI based services and it was bought up by a larger company not 6 weeks after I opened it. I can tell you from the limited time that this is the best service there is as I have had plenty of them since I sold. No one is better than DH (and no I don’t work for or have anything to do with the company other than a lot of accounts with them).

Now quite whining and go back to whatever you were doing before you decided to troll this blog and give your 2 cents. SKYRIM sounds like a great idea but WOW is the way to go!

I can’t access my ftp accounts as well. I’ve waited 30-45 minutes since changing passwords

Steve, I do know that updated shell or FTP passwords do not apply “immediately”. If you still see the clock icon next to the users in the Manage Users section, the changes have definitely not been applied to the accounts yet.

Thanks for answering my questions. And thanks for the quick action. I trust that you will review how you can be more secure in the future.

ALL my accounts are showing as ‘DNS Only’, despite the fact that all of them were fully hosted

Typically for a shell account I reset the password through the web interface to something random, log in to ssh with that, and then use the passwd command to change it to my real password. If I understand things correctly the hackers may or may not have had access to that initial random password I generated from the web interface but NOT the actual password I changed it to one I ssh in to the server?

” If so, how is it that you display the PLAIN TEXT password on my screen after I hit submit?”

You clearly don’t know anything, so just shut up. Really.

“a legacy pool of unencrypted FTP/shell passwords in a database table” – it would be good to know which accounts were in that table and whether it contained any other fields beyond username and password that could link accounts with domains, email addresses, etc.

@Jonathan Jospeh

“So you’d rather DH not change your passwords to hundreds of email accounts and possibly have your entire dedicated server hacked, defaced or whatever ? I mean really, think about what your saying.”

I’m fine with them doing it SO LONG AS I’M INFORMED (and given the correct information) ABOUT IT FIRST! All of the communications from DH prior to my experience with Tech Support said only Shell and FTP users needed to reset, which is perfectly fine and appropriate. Had those initial communications also said that emails were going to be reset or even “Hey Dedicated Customers: y’all will have to do emails too” then I wouldn’t have said anything about it.

But by saying one thing and doing another, we got hung out to dry. Doing this on a Friday and later in the day to boot with no forewarning virtually guarantees a weekend of headaches and frustration as I try to enable email access for all users across all accounts. Had I known it was coming, I could have prepped for it which, if you think about it, was the whole point of the status blog posts and email communications and Tweets etc. etc.

Am I really asking for too much as a high-end customer to get accurate communications and instructions from the company I entrust to handle our internet presence? Really?

I appreciate your vigilance and your transparency. I’ve been a customer since 2005 and have always appreciated the communication of your team. One thing I would have liked in this instance was an email notifying me as a Dreamhost customer with the same message that was in the panel. I didn’t learn of the breach until I could not get into my site via FTP, and then of course I logged into the panel and saw the message.

Thanks.

@ Carl,

Are you seriously that rude in person?

Please, enlighten me or go back to your mother’s basement.

@ Everyone else,

It has been more than 3 hours since my password change and I still cannot get into my ftp.

Thanks very much for posting this.

And kudos to the security team for catching this so quickly!

Thanks Kirk,

I was just noticing the clock icon. It’s usually gone after a minute or two, but has been there for over an hour now. I’m guessing it’s taking longer than normal for the changes to go through due to the fact that everyone else is probably doing the same thing…

Thanks for being on top of this,
Steve

I can understand you are having bad times, what I cannot understand is:
1) lately it’s more bad times than good times
2) chat support is offline
3) supports tickets are not solved or deployed adequately, for example : I asked the support team A) why all the files have been deleted from my sites and B) why I cannot upload files with my FTP account. they just answered me about the ftp problem and that they fixed it… Why my files were delete -from EVERY site- in the first place, or why don’t they restore a backup…no one is answering.

Thanks to the Team at Dreamhost, you all rock to say the least. I am just happy you kept us all up and running. This situation & resolution is only one reason I love Dreamhost.

I must say “Support” has always been there for me when I need them 24/7 keep up the great work people:)

Thanks
Michael

I hope DH has stopped automatically emailing new passwords back on a successful pwd change.

I just reset the first 5 of the 20 accounts I need to update and will be pissed if the new passwords are automatically emailed back to me. I like DH for several reasons but the practice of emailing my password is idiotic and something I’d expect the competition to do.

Otherwise, I appreciate your policy of open information.

Hi.

I am not very tech savvy, so please be patient with me.

I have a Dreamhost account and have quite a few domain names registered that I host for my self and some family members. I used WordPress to create all the websites and don’t ever recall ever having FTP/SFTP or Shell passwords. Is this done automatically? I have 5 user accounts that I set up, so does that mean I now need to change the passwords for these accounts? I don’t recall ever accessing these FTP/SFTP or Shell passwords in the past.

Thank you in advace.

Thanks

Pat, thank you for clearing that up. I did not mean to imply that private keys were going to be stored in the ~/.ssh/ folder. I implied that people doing a key exchange would have changed or added files in that folder. By doing an ls -la you would also see modified times, so if authorized_keys had been modified, it would be indicated that way.

“On at least one of my PS accounts the password was not reset by DH and remained as it has been until I changed it. This is at 4:30 CST. The statement that all SSH/FTP passwords were reset by DreamHost is not accurate.”

That happened on mine as well and I was greatful for it. The PS service was implemented after they stored everything encrypted, so it wouldn’t have been stored in that database. lucky for us.

Sorry the above happened 48 hours ago not yesterday.

There’s been no mention at all of the size of the “legacy pool” – is it too embarrassing to admit it included most if not all accounts?

Again – did this table contain other information beyond username and password that might directly link accounts with real names, domain names, email addresses and so on? Is it plausible that the intruder has downloaded a copy of the entire database?

It’s inadmissible that these points haven’t been addressed to allow customers to properly gauge the risk they’ve incurred through DH’s unthinkable, incompetent practice of storing plaintext passwords.

I’m sure there are users who use the same password for other services (email accounts, etc) and so this breach can have a fairly wide impact in some cases, especially if the table contained fields other than username and password.

I continue to appreciate the full disclosure ethic that DH has had, and continues to have. Who doesn’t have old slop data floating around that shouldn’t, it’s a good reminder to all. Especially as companies grow, remembering to ‘clean up the old mistakes’ always needs to be a part of the process.

My four requests:
– Alert everyone who has had a change in .ssh/* during the time between the compromise and the lockout (that’s gunna be a long find, but will be worth it).
– Since username.dreamhost.com went away; please send all users who use ssh in some form to login to their accounts, a list of all their hostnames and the DreamHost ssh fingerprint for that domain (I’ve noticed it hopping around some).
– Implement OAUTH or something a little cleaner for the Panel API
– PLEASE add a ‘change password’ panel API function, you don’t need to leave it out there all the time, but for today it sure would have been handy.

I’m sure you’re having a terrible rainy Saturday, but it’s better then being stuck in the LA traffic today, ufff.

@yudayuda Thank you for your help. I have now gone in and reset the user passwords.

Regards,

Peter

Hey, I had to put in my two cents…

I am really surprised at all the negative comments in regards to this security issue.
I myself was a little worried when I got an email from DreamHost but as usual I realized that this outfit was on the ball right out of the gate. Reading DreamHost Status was a great help as well.

So I did the changing of passwords, there was a little glitch, it didn’t work for a while, I had to start a ticket, but then; fixed, done, and everything was fine. In reality this happened quickly. Hats off to DreamHost!

I have been with this outfit for almost ten years and not once have I ever been let down or found anything disappointing. Yes there has been times of error, and it is usually resolved in less then twenty-four hours. In fact I can’t remember the last time I had to contact support. I really dig DreamHost.

Thank you, you folks at DreamHost, for forcing a password change and looking out for my best interests and yours. I wouldn’t host with anyone else even if they paid me. I feel so safe and secure and that is just fine with me. DreamHost Rocks !!!

evL

My concern in all this is that if FTP/sFTP/Shell passwords were considered suspect then everyone who has a Drupal/Wordpress/Joomla or other CMS website with a config file needs to change their MySQL passwords as well. Anyone that can access your site in a sFTP/Shell and hacked the DH database would know that these files are perfect targets.

Now on top of that add the possibility of user tables in these CMS systems being accessed. Doesn’t matter if they’re one way encrypted since most users fail to follow simple rules so at least one of them probably used “password” or some variant. With a Rainbow table attack most hackers would make quick work of these.

@Pat & @Kirk I hadn’t even thought about the possibility of setting SSH key logins thanks. This nightmare just keeps getting better and better.

@anyone This is not “bitching”, “moaning” or any other adjective. It is a legitimate concern for the security of my personal, my companies and our customers websites and users.

Knowing now that this is some legacy database table from years ago only confirms my suspicion that this was a gross over-reaction. The solution is worse than the perceived threat.

They’ve reset the sFTP/Shell logins for a PS created 3 months ago but left a VPS created a year ago undisturbed. The shared hosting part I can deal with, it sucks, but the inconsistent nature of the implementation of the DH solution leaves me wondering what the plan was in the first place? I’ve already reset my passwords on the shared and PS accounts but do I wait til the VPS passwords don’t work, then reset. Or reset them like some folks have done and then hours later the DH reset goes through, so I have to do it again? In the interim what is being compromised?

Our FTP accounts are STILL not accessible.

- I changed ALL our FTP passwords 24+ hours ago
- Still can’t cannot to any FTP account, across 3 VPSes

9-10 hours after sending a support ticket, I received 1 email from DH this morning saying “you should be able to access them now”. Finally got around to this now and still can’t access any FTP account.

@DDB Can you submit a support ticket attn: Oscar and I can look into that for you.

@Oscar: Thanks – I have 2 active tickets waiting for a reply, maybe you could take one of those. :) Or just look into this for me – account is my email danebaker@gmail.com.

As expected, DreamHost took care of the issue as fast as it could, with the best measures it could, as efficiently as it could.

As expected, a bunch of DreamHost’s competitors are spamming up the blog posts with their attempts to make it seem like DH has a whole load of mentally challenged customers. Oh wait, DH actually does have this many mentally challenged customers that aren’t grateful that, unlike their banks and credit card providers (and hospitals and governments), DH not only took care of the problem but disclosed the break in.

Hey there, retards, if your shitty bank got hacked, you would never know about it, and you’d be blamed by your bank (“you must have shared your PIN!!!”) when your money disappears. If this was your credit card provider, you’d call in after your card is used to buy things all over the world, and they would blame you for having your card info stolen.

@intern3ts – You’re the one coming across as some sort of mentally deficient idiot. Sit down, the grown-ups are talking.

@DH – I know crap happens. My Visa was compromised just this week (and, for the record, @intern3ts, my bank called me before I lost anything and immediately removed the bogus charges). It happens. But you have set our expectations that we just need to reset our passwords. But the reset has not propagated for many of us. If I check the status page I see a [much] earlier post on the status mentioning a 1-2 hour delay [again, much earlier in the day] mentioned.

But I reset my passwords 4 hours ago and they still don’t work.

Please update your status or your blog so we at least have a more realistic expectation. Even if it’s to say “we don’t know”, that’s better than what we have now… which is nothing.

For those of you looking for an important clue about the security breach, here is an important one.

I was going to ask Dreamhost for the last-modified date of the database table containing the unencrypted passwords, as this would allow me to assess which, if any, of my accounts were compromised. However, I no longer need to ask for that information, and here’s why.

Dreamhost knows full-well which accounts were compromised, as the list of usernames and passwords is obviously contained in the compromised database table. Therefore, Dreamhost needs only to reset the affected shell/ftp account passwords. Yet, Dreamhost publicly states that they have reset the passwords of “all” shell/ftp accounts. However, that is not true. And here is how I know.

I created a new shell/ftp account on October 11, 2012, and the account’s password has not been reset by Dreamhost. I can still access the shell/ftp account using the very same password as when I created the account on Oct 11.

So, what does that tell me? The fact that Dreamhost claims they changed all passwords, and the fact that this account’s password was not reset, and in fact, still uses the same password that was chosen about 3 months ago: it suggests that the compromised database table had not been updated in at least the past 3 months. But for how long it was not updated remains a mystery, and one that I hope Dreamhost will eventually disclose.

Nonetheless, I suspect the compromised table was being updated until fairly recently. Rhetorical question: If 99% of your shell/ftp accounts are affected, and you reset the password on those 99% of affected accounts, what is difference in telling the public that “all” shell passwords were reset, and “practically all” shell passwords were reset?

Anyway, for those of you wondering if your account’s password was compromised, it probably boils down to this. If Dreamhost reset your account’s password, well then, yes, your account’s password was probably compromised. However, if you find yourself able to access your shell/ftp account with the same password you were using before the announcement, well then, no, your account probably was not compromised.

Lastly, my suggestions do not necessarily imply an account created less than 3 months ago was not reset. As many people use the same password for multiple shells under the same account, I suspect that Dreamhost probably has checked every shell/ftp account against all unecrypted passwords found in the compromised database table. And, if any of those passwords succeeded, then they probably went ahead and reset the account password anyway, even if the shell account was created less than X months ago.

(Note, for the shell account created on Oct 11, 2012, the password is pretty damn unique and unlikely to be used coincidentally but any other Dreamhost user. Thus, in Dreamhost’s eyes, it would be a safe password/account, and therefore, not subject to needing a reset.)

Disclosure: The above may or may not be true, in whole or in part, but considering the facts, I think my claim has a high probability of being true. Dreamhost has not provided me with any of this information. Instead, I have simply made my assessment based on personal knowledge of my account’s shell/ftp usernames, passwords, and creation dates. Your mileage may vary.

PS, the reason why Dreamhost is not likely to require a change of user’s email passwords is because the compromised shell accounts were of the following nature.

username: janedoe
password: fluffy

In order to access an email account, a hacker would need to also know the domain.

username: janedoe@somedomain.com
password: fluffy

The perpetrator who accessed the compromised database table presumably did not gain access to a list of which shell accounts are linked to which domains. And thus, the perpetrator would not have the “@somedomain.com” necessary to use the compromised credentials to login to a domain. I suppose that the perpetrator could try all the compromised username/passwords against all the domains known to be hosted at Dreamhost. However, I also suspect that Dreamhost has security measures in place to protect against repeated unsuccessful brute force attempts against domains and/or from a specific source IP address.

Thank you.
We appreciate your response to this situation.

Simon, I was up today at 6am CST since I host a weekly web/internet tech podcast here in Minnesota. As a consequence all day long I’ve been monitoring this, attempting to get my password changes accepted in your queue, interacting with customer support, and here it is 11:45pm CST and nothing works.

Not only can I still not use SFTP to get in to my domains, all my sites are offline. I’ve had clients pinging us by email all day (there is a big trade show going on in Frankfurt, Germany and we just launched a new product last week people there are trying to access) wondering WTF is going on.

So imagine if your power company knocked you offline and you didn’t have backup generators. They did a few infrequent updates and sent you an email, but you’re effectively out of business. THAT is how I feel.

The other thing is that one of our businesses is site and web app development and we have historically recommended DH to our small clients due to our previously good experiences. I’ve had these clients pinging me all day as well.

To say that this is causing a negative impact on my businesses is an understatement. To say that I’m so f’ing pissed I can’t see straight is also one. While I appreciate what you’ve done thus far, and no question the ramifications of this security breach will likely cause DH to lose many customers, you are not doing enough.

It would have been nicer, honestly, if all accounts had been restricted, allowed to login, and had a message to this effect displayed prior to being kicked off demanding a password change. I would have gotten the information much sooner.

Simon you said, “The best way to get the fastest assistance is to submit your support issue through the panel, because we can’t necessarily identify your account or trouble shoot your issue from the information on a comment post here”

OK Simon…can you tell me why then:

a) My ticket #4694291 entered Jan 21, 2012 – 09:44:24 has gone unanswered nearly THIRTEEN HOURS later?

b) Why my ticket #4695917 entered Jan 21, 2012 – 21:34:00 has gone unanswered OVER THREE HOURS later?

In addition…

c) Why my comment above to you got no response?

d) Why my tweets to @DreamHostSimon @DreamHostCare @DreamHost over the last HOUR+ have had zero response?

I think I’m a reasonable guy and appreciate what you and your team must be going through. But over 19 hours of all sites being offline and 12 hours for a ticket response is not reasonable.

You know guys, there is no reason for you to store passwords in clear text!

Simon, my web site and email have been down all week. I have sent emails with no clear response. I changed the passwords as required in your email today, and my site and email are still down. What do you suggest I do? This is totally unacceptable.

Simon: Here is *my* status update.

I’m fully aware I’m being a complete dick. But if you were in my shoes — having clients emailing like crazy wondering WTF was going on with our ecommerce site since they want to buy — coupled with the small biz clients I serve whom I’ve recommended they host at DH and are now wondering if my team and I know what we’re doing.

One client just said to me in an email from France that just arrived, “Maybe if you used a *real* hosting company instead of a cheap one, you wouldn’t have this sort of problem”.

Ouch.

So yep, I’m going to be a dick until my sites are back up and running. Maybe no one is working?

Here are my outstanding tickets…the first one submitted over 14 hours ago:

Ticket: #4695998
Subject: Sites are *still* down, NO FTP
Submitted: 12 secs

Ticket: #4695975
Subject: FTP/Shell Password Not Working AND All Sites Down
Submitted: 37 mins 28 secs

Ticket: #4695917
Subject: Sites are *still* down, NO FTP
Submitted: 2 hours 14 mins

Ticket: #4694291
Subject: Why no status updates?
Submitted: 14 hours 4 mins

Thanks for staying responding a potential and spelling it all out in a way a non-techie (that would be me) could understand.

Several of my clients are with Dreamhost and when they asked me about the email about the passwords, they were all HAPPY that Dreamhost was on top of this.

OK, a second change of password on the panel finally pushed it through to my private server.

Never received any noticfication from Dreamhost that passwords , infections , viruses , or domains were down. I had to find out on my own.

Kirk… ls-la is useless as you can use ‘touch’ to change datestamps and then clear history

Brand new customer, moving our fundraising system to one of your VPS’s. Haven’t been able to FTP or Shell into my account for days.

“Customers who have ongoing concerns can contact our support team through the web panel” – good one!. Mostly I get automated responses, when I do hear back from an actual tech, the problem doesn’t get fixed.

Hope this gets remedied soon.

SQL passwords were not compromised.

As a precaution I changed mine anyway. After all, they are visible in my config files for wordpress. I suspect if someone could ftp to the site, they could read the config files.

Dear folks,

1) Dreamhost is NOT a reseller caliber provider. Never has been, never will be. They do a fine job with cheap “good enough” shared hosting. Want anything beyond that? Go elsewhere and pay more.

2) Acting childish in this space won’t change #1 or get you better service.

My site http://www.monticellolive.org won’t let me even login to it and i e-mailed your support@dreamhost.com and sales@dreamhost.com and i have not got a single reply back from them in a year so pleases tell them to e-mail me back and everybody too and start replying back to your comments on your dreamhoststatus.com comments

why is my comment still saying Your comment is awaiting moderation

@Sam Nelson said, “@ Steve Borsch, I recommend some professional help. You seem very obsessed. The world has not ended. Websites go down. They will be up again and life will go on.”

You are absolutely right. I do need some professional help. On the web hosting side. Am I obsessed? Yeah, WITH MY CUSTOMERS and >29 hours of downtime is why I’m acting as such a squeaky wheel (my sites are up right now with a temporary fix by DH, and the support guy was very accomodating).

the dreamhost support team has not answer any of my e-mails i sent to them about my site needing fixed

@Monticello I’m bet they got 1000 mail per hour and You asking them to answer Your mails fast?

Wow, you had a “legacy database” of plaintext passwords lying around? Such a thing should never even be created, or even be ABLE to be created in your environment, let alone get stored and forgotten somewhere.

Dreamhost is the most lackadaisical and unprofessional hosting company I have ever seen. Remember the time you charged all your customers $120 because of a “fat finger” incident?

I know everyone hates “process”, but this crap is why process exists.

@Sam Nelson
“Thank you DH for being proactive and we appreciate your hard work everyday. You can’t make everybody happy. Crap happens, and I personally appreciate what you are all doing to take care of your customers.”

How is having to reset passwords because of incompetent developers/management being proactive? Being proactive would have been to not EVER store any data that is sensitive in plain-text in the first place. I have trouble understanding how any one is happy that your data/customers could have been compromised because of complete laziness.

Once again people come and have a good old moan. The human species really does contain some weak DNA. Most of you are paying peanuts and still expect both perfection and infallibility. Just be VERY VERY thankful that they managed to detect it at all as I can speak from experience, being a security expert, that many places wouldn’t even know they’d been compromised. It sounds like they made a mistake storing the passwords in plain text for sure but these kinds of things happen no matter where you are and no matter how good you are. But they fessed up, put it right, gave a clear outline of what happened. Professional and prompt.

I get tired reading about the endless misguided people running endless “mission critical” services expecting to get a 24/7 service which never has problems. You’re idiots. Complete and utter idiots and you make my blood boil with your incessant whining. “Passwords are taking a long time to update” Really? Are they really? Well THERES a shock – a system designed to handle x password resets a day suddenly gets x*1000 that amount or more and buckles under the strain (but I hasten to add doesn’t completely fall over) and you come here to moan? Get over yourselves.

But no wait there are other moans “Tech support didn’t respond to me and my pissant site is down. I pay $130 a year dammit how DARE you not respond to me with 15 seconds” Unbelievably sad and incredibly stupid. They just hit the ENTIRE user community with a password change who then immediately hit tech support oblivious to the fact that they can’t access any more because their password is reset (mostly because they couldn’t be bothered to read email/website/webpanel) etc. And you expect that NOT to have a massive impact and detrimental effect on tech support? You don’t think past your own selfish, ignorant little worlds

I’ve said this in other posts and I’ll say it again. If you want a fully resilient, guaranteed uptime, instant support and other such things then go and damn well pay for it. You just want everything for nothing. With Dreamhost you get a staggering amount of service for a pittance and those of you moaning here have no idea whatsoever of the value you get. What sickens me most is you’ll pay more for a couple of latte’s from Starbucks than you do for hosting here for a month and I bet you don’t moan at Starbucks that you’re pissing that money down a toilet 12 hours later. But sure as eggs are eggs the second theres the slightest little (or major) hiccup here you come and whinge about how bad everything is. Grow up the lot of you.

Dave

I can’t seem to upload images to my blog using wordpress. I have to manually upload using FTP then set the link manually, which slows down my blogging. Is there any way to fix this? Perhaps I need to reset the config files or change the password settings in the database to what I already changed them in the Dreamhost panel?

BTW, I just wanted to say I appreciate Dreamhost taking the steps necessary to protect us. Mistakes happen, sometimes employees do retarded things that we find out about later. I know it happens. Glad you didn’t try to cover it up. Thanks for being real.

how can i fix my site

People say “these things happen”. These things don’t happen to top tier hosting companies, because they know how to safeguard against it. Dreamhost should probably consider just reselling Amazon Web Services hosting, instead of using their own hardware and team, because AWS has their shit together. Dreamhost doesn’t. Dreamhost are amateurs.

Dave Richards… well said. I just printed your post and will frame it on my wall. I feel for the people who were severely impacted by this, but more so because they should take precautions if their stuff is really mission critical. I use Dreamhost for hundreds of domains, but none of them are meant to be bulletproof. If I need that, I go dedicated, and use Linode or a similar service, at tremendous expense. I recommended Dreamhost yesterday, and I will recommend it again tomorrow for folks who want a good blend of functionality and affordability. Want uber-availability and perfect security? Call me, I will only charge you $350/hr dev/admin plus $2k/month per dedicated server.

ATTENTION ALL: Log into Dreamhost control panel.

Goto MySQL Databases

Click on any “Users with Access” on a database

Scroll down to “Would you like to change ##########’s password

What do you see?

Currently: “##the plain text password##”

WTF

My response from support almost a year ago after finding this was “if you don’t like the way we do things find another host”

Seriously, i have a copy of that ticket, and it still unbelievably is in my support history. They didn’t delete this one of all the many errors they have done and I have found and have deleted. I log everything for legal reasons.

You know what would be really cool? If when I changed my password as required, the system would actually recognize the new pass within a reasonable amount of time.

So far I changed my pass 3 times, waited up to 30 minutes for the magical process that allows the changes to take effect, and still have no FTP/Shell access to over 50 of my clients’ sites.

Never have this problem with HostGator, HostNine, Site5, VPS.net, Xilo, or any of the other hosting companies I work with. Ever.

Just my 2 cents.

LET ME INTO MY ACCOUNT!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

This is UNACCEPTABLE!! I don’t know what host the people on here who are singing praises of functionability and reliability have been using over the past few years, but it ain’t the Dreamhost that I’ve been using, because this incident is just another in a string of what seems to be every-increasing DUMB system-wide snags which hold up MY PRODUCTIVITY and I don’t care why!!

I have created a new password TWO TIMES for my FTP twice now. It’s been over an hour and no dice.

Passwords take almost an hour to update on any change, Email, FTP, SQL.

This has been ongoing since the beginning of Dreamhost and never addressed.

Maybe now it will since 100% of customers are complaining.

It is NOT just because of the mass passwords being changed.

…And if it is, maybe Dreamhost should move there memory slider a little more to the right ( /canned response ) :P

Coincidentally, my credit card number was stolen and used yesterday. Then I read of the security breach at DH. Could this be related? I’ve never had my number stolen before.

What would be nice would be if Dreamhost did not lie to its customers. When all this happened, I changed not only my user sftp/shell password but also my mail passwords. The new mail passwords worked fine for a day. Then they magically and mysteriously reverted to what they were before I changed them. That should be impossible! When I change a password, all record of the previous password should be overwritten. Instead, it happened the other way: Dreamhost did some sort of restore without telling us, and overwrote the new passwords with stale passwords that it was keeping somewhere else. There is stuff going on here that we’re not being told – plus this sort of folderol makes it very hard to check one’s email.

Simon Anderson, I respect you and I respect DreamHost, and would just like some simple questions answered:

1: Why did you have this table of plaintext passwords in the first place? My guess is that it had something to do with the password change feature and its way of warning you “your new password cannot differ from the old one only by case” that some users pointed out?

2: You call this a legacy table. When was it last updated? I had actually changed ALL my user passwords and email passwords on January 5th, 2012. Was my data in that table or am I safe?

3: What columns were in the table? I am guessing machine (like beid.dreamhost.com), username and password, and probably the account ID of the DH account that owns that user. Anything else that was lost?

4: Are you SURE this is all the hacker got access to?

These are VERY important questions. Please do not ignore them.

Looking forward to a response, Simon. F5 F5 F5,

Richie

UPDATED TO ADD QUESTION 5:

Simon Anderson, I respect you and I respect DreamHost, and would just like some simple questions answered:

1: Why did you have this table of plaintext passwords in the first place? My guess is that it had something to do with the password change feature and its way of warning you “your new password cannot differ from the old one only by case” that some users pointed out?

2: You call this a legacy table. When was it last updated? I had actually changed ALL my user passwords and email passwords on January 5th, 2012. Was my data in that table or am I safe?

3: What columns were in the table? I am guessing machine (like beid.dreamhost.com), username and password, and probably the account ID of the DH account that owns that user. Anything else that was lost?

4: Are you SURE this is all the hacker got access to?

5: You say that we should all change email passwords too. Why? In my case, they were last changed on January 5th, and had nothing to do with ANY other password I own. Why should I change these if the hacker didn’t get access to them? Please clarify what you know about the hack and what got out.

These are VERY important questions. Please do not ignore them.

Looking forward to a response, Simon. F5 F5 F5,

Richie

I for one, really appreciate the transparency of the situation. I was able to change FTP access with no problems. No harm done. Thanks Simon

found it. I think I’m set now. Thanks.

It’s sad to see our sites go down from time to time, because of unexpected outages like this event. However, I applaud Dreamhost’s security team and the corporate head of Dreamhost for their quick actions to protect customer info :).

FYI, the feature whereby your new password is compared to your old password can be implemented without storing passwords in plain text.

For example, here’s a md5 hash of the string “password”:
5f4dcc3b5aa765d61d8327deb882cf99

Rather than storing the actual password, you store the hash. When a user changes his password, you create a hash from the new password and compare it to the old hash. If they are the same, the passwords are the same.

Hashes are one-way, so if someone gets access to your database, all they have is the hashes not the passwords.

So let’s not jump to conclusions about why DH had plain-text passwords stored in their DB.

@panah, et al discussing SQL passwords. My guess is that Simon is saying that DH’s own database(s) of SQL passwords was not compromised.

Our own files may have been compromised, so if you think there’s a risk, I’d take precautions and change up your database credentials. It doesn’t take long and it’ll give you peace of mind.

Sweet. I’m glad, I have to go through all of the DH accounts I manage and change all of the passwords, and create a new passwords, then create a updated pasword list for all of my clients on DH, then call all of the folks to update…Maybe I’ll just send out a mass email DH style. Ahhh sorry.

So, yes. This happens doesn’t mean it doesn’t suck. I’ve started with DH early 2004. I know and run to many websites on this system/platform to list. I’ve been having some issues over the last month or so, a few of my websites have gone up and down, also locked files and folders out of my control. Overall DH has been great. Hope this is not growing pains. To many accounts to keep up with, now service goes down.

Please do your best to make your system…updates. My accounts were effected.

Just a note to Dreamhost with regard to passwords being encrypted:

Subversion passwords are not one-way hashed.

Also, seems our database account passwords are not one-way Hashed:

Would you like to change XXXX’s password?
New Password:
Currently yyyyyy

Well,

First: Jonathon, well commented. People who know too much always like to point out their superiority in these cases. I have been with a few hosting comapnies and DH is a new one for me. I have to say that I am very impressed with this response to a clever, but lucky hack.

I am not in any way a tech bod, I don’t understand most of the inner workings of all this stuff, but I know when a company makes an effort and when it doesn’t.

To the whingers: To reiterate comments, if you don’t like it, cancel your account, take your ‘big money’ somewhere else or start your own. You know so much but only complain. That is the mark of a coward. If you think you can do better, by all means do it. The rest of us will happily stay with a company with a track record and a commitment to it’s customers. I will recommend DH to anybody who asks and will burble enthusiastically about this on my blog and company website. In short, get on board and support or get out.

To DH: Thankyou so much for your quick response and, as it took me all of ten minutes to reset my passwords, no problem. A job well done. Hackers and Pokers will always be a thorn in the arse of anything digitally related. Vigilance is not always enough, response however can be vital. Again, thanks a bunch.

Dear Simon,

Can you tell me why my site has been down for SIX DAYS???

Nobody else at DreamHost seems to know why, or when it will be back.

Thanks,
Dave

What gives?
I’ve changed my passwords for login, mail, and ftp 4 times since getting the e-mail. The new passwords work for a few hours and then revert back to the original pre-hack passwords. Same thing again this morning. Anyone else having this problem?

@Chris Geisel. I disagree. It does take a long time when you run a 100+ blog network. :D Trust me…. I think we will have to do it. But it’s not ideal.

WHY ARE MY SITES DOWN?????????????

I have a bunch of sites that are STILL DOWN with a FATAL ERROR 68
When will you be fixing these?!

I appreciate your swift action and openness about this incident. However, I’m a bit alarmed that I had to find out about the incident through Lifehacker. I checked both my inbox and spam folders and did not find an email from DH.

My site was brought down so sites have definitely been affected.

All my VPS user folders are empty… I just want to access my files and move them elsewhere (my shared host is 100% ok for now). Where can we get information? $”#$”# individual emails, but please POST something useful to your clients.

Just FYI folks- after I complained Saturday that not all users passwords had been reset DH support confirmed this. And then they overwrote the passwords I had spent Saturday morning setting and recording…

@Erik
“Also, seems our database account passwords are not one-way Hashed”"

This does not mean they are stored in plain text.
There are other ways of encrypting passwords then one-way hashes….

@Dave Richards…well said. here’s a cookie.

@DreamhostSimon…a belated, nice to meet you. Do you know Chip Kelly (Ducks Coach)? Just askn, ’cause you’ve got the same brass ones to make this post, lol. Admirable man. You’ve shown that you are exactly where you should be in your personal career. Inviting, listening to and accepting client complaints is just as important as any other business operation.

Thanks for the quick action; tireless efforts; openess and honesty. A rare trait in business now-a-days. hugs…Bj.

Dreamhost is so screwed up I cant even add support to my account (for a fee) to find out why my registered domain doesn’t show up in my account!

This company has certianly gone down hill fast in the last year, premium fees and absoutly no service whatsoever. Security breaches, massive ammounts of outages; what a bunch of junk.

You should be fired. Seriously. It’s almost as if you sold short on Dreamhost and are trying your hardest to run it into the ground.

no company in their right mind should ever hire you as CEO.

I’M SO ANGRY I CAN’T FUNCTION!!!!!!!!!!!!!!!!!!!!!!!

Simon,

Thanks for the quick update, and full transparency … I appreciate it.

Cheers,
Steve

would like to know if i have to change all sql database passwords as well.

I can’t connect to my ftp server…

Nooo, al the domains are linked to pornhub :S:S:S help!!!

mehh nevermind, im going to Elitehosting.nl. faster, better, only a sad thing is its dutch.

Hi,
some days I get this written in my dreamhost panel
————
Due to some activity we detected Unauthorized Within one of our databases, we have forced to reset all of the FTP / SFTP and shell passwords as a precaution.
If you’re having trouble logging into your FTP or shell accounts, please visit the “Users> Manage Users” tab and update your passwords there.
You can keep up to date with this issue as it progresses on the DreamHost Status blog at:
http://www.dreamhoststatus.com/2012/01/20/changing-ftpshell-passwords-due-to-security-issue/
Our CEO, Simon Anderson, Also has posted a blog entry on this. You can read it here:
http://blog.dreamhost.com/2012/01/21/security-update/
——————-
I do not work anymore, the panels of my FTP sites and they are many and do not know how to fix them.
If the nchiedo can settle the problem as soon as possible I am having problems with my clients … What should I do??

Thank you.

I found a half-dozen foreign PHP files in my directories. I’ve removed them, but kept copies and can send them to you — in a ZIP — if you are interested.

I’ve enjoyed being a customer of yours these past six years, and I’m not ready to switch hosting providers — YET — but this is the third or fourth hack-attack I’ve had to endure, and I am absolutely sick and tired of repeatedly having to manually clean my directories.

I’ve changed my passwords and hope to never encounter this problem again, but I’d like some kind of compensation from you for the grief and lost time spent manually responding to this BS — preferably financial. Maybe a free month or free domain name registration for each attack.

What do you say?

@Jose

This is comments for a blog, not a support system. Posting 100 times that your domains are down here is utterly pointless and just spams up the comments for the rest of us.

@oneeyedwonderweezel

I think we get the picture. lol. Do you always scream and shout at 100MPH to effect change?, doesn’t work and just upsets and annoys everyone. If things are that bad and you’re soo angry why haven’t you changed host yet?

@DreamhostSimon

For the sake of full transparency, mentioning older passwords stored on un-encrypted database files should have been in your intial blog, NOT after someone had to ask you in comments and then you comment after it. That left me a bit dissapointed.

…however.. I have to go by my own experiences, been with Dreamhost over 7 years, rarely had any real problems. The worst was a period of downtime for about 2 days and that was it. I recognise that no other hosting site at the same or similar price point offers substantially better uptime or service, its just the nature of these things when attacks happen. So I’m going to stay put.

I HAVE CHANGED MY HOST!!!!!!

AND I’M HAPPY TO SAY I HAVEN’T LOST A BYTE OF DATA.

AS TO THE ” WHY I HAVEN’T CHANGED MY HOST YET?” QUESTION. I CHANGED MY HOST IN SPITE OF THE MALFUNCTIONING SCP.

MY MAJOR COMPLAINT IS THE GOD DAMN MIND RAPE YOU PUT ME THROUGH CHASING GHOSTS. MAJOR MALFUNCTIONS ARE OCCURRING AND I’M GETTING THE MESSAGE, “IT’S JUST YOU!!!”

AS FOR THE “YOU GET WHAT YOU PAY FOR” ARGUMENT, “IS DREAMHOST ONE HALF OF GATORHOST WHICH IS 2 TIME GODADDY BUT STILL COSTS JUST AS MUCH???????????????”

I CAN ACCEPT, “THESE THINGS HAPPEN.” BUT THERE WAS AN INFORMATION BLACKOUT. I WAS GIVEN NOTHING TO ACCEPT OR NOT ACCEPT. FOLLOWED BY A CLAIM OF TRANSPARENCY.

SUBVERSION IS SOLID ENOUGH TO TRANSMIT SOURCE AND MYSQL DUMPS TO OTHER HOSTS, WITH A LITTLE PATIENCE.

You probably have at most 70 years before the end of your world comes. None of this will be remembered then, especially your whining about your cheap shared hosting account. God bless DreamHost, but it will probably be gone by then, too.

Thanks for dealing with this issue promptly. And thanks for being open about what happened. On a related note, I’ve expressed my concern about the way you store (in reversible encrypted form) and transmit (in plaintext email!) passwords to your support staff on several occasions. Perhaps now, you will consider changing these policies? Keep in mind that AS LONG AS YOU CONTINUE TO HANDLE PASSWORDS THIS WAY, YOU WILL BE A PRIME TARGET FOR HACKERS. Just sayin’.

UPDATED TO ADD QUESTION 5 (I will keep asking until you answer or I get bored):

Simon Anderson, I respect you and I respect DreamHost, and would just like some simple questions answered:

1: Why did you have this table of plaintext passwords in the first place? My guess is that it had something to do with the password change feature and its way of warning you “your new password cannot differ from the old one only by case” that some users pointed out?

2: You call this a legacy table. When was it last updated? I had actually changed ALL my user passwords and email passwords on January 5th, 2012. Was my data in that table or am I safe?

3: What columns were in the table? I am guessing machine (like beid.dreamhost.com), username and password, and probably the account ID of the DH account that owns that user. Anything else that was lost?

4: Are you SURE this is all the hacker got access to?

5: You say that we should all change email passwords too. Why? In my case, they were last changed on January 5th, and had nothing to do with ANY other password I own. Why should I change these if the hacker didn’t get access to them? Please clarify what you know about the hack and what got out.

These are VERY important questions. Please do not ignore them.

Looking forward to a response, Simon. F5 F5 F5,

Richie

I can’t access any of my files and I’ve been getting a real run around from support. I don’t usually complain about these things on DH since I’m just a hobby site and not a business but this is really insane.

Hi,
I just realized that dreamhost keeps passwords in plain text… if I forgot my password and you email it to me plain text that’s not a good thing. This is upsetting. anyways you can watch the first 5 minutes of this show and they’ll explain.
http://www.jupiterbroadcasting.com/16331/answers-for-everyone-techsnap-42/

Anyone else having problems with virus attack on their hosting? I tried to clean it, but it is still there, and Google started to mark our sites as badware… All the SEO work may be wasted. Thanks

Sad to see you ignored all my important questions, Simon.

+1 @simon Please answer Richie’s questions.

Greetings, Simon!

I just made a re-visit at this moment on the off-chance that you had responded. I was very grateful to see that you had indeed!

Thank you so much for the very honest reply. It finally explains the details that had been bothering a lot of us more technical users.

Alright, so it *did* have up-to-date data as of the date of the attack (in other words not “legacy” as another DH employee had called it on the status-blog), and it was used for internal service authorization (but you’ve thankfully changed that system now), and you have determined that this is all the attacker had access to, and that there’s no evidence that the attacker even dumped this table (you just changed passwords out of precaution), and even better – the suggestion to change email passwords was based on the off-chance that someone had the same password on the email account as their shell account.

Fantastic answers and I am really glad to hear the reasons for each issue. It lays to rest any worries that I’ll have to change my email passwords once again, since they weren’t in the table, and had nothing to do with the shell passwords.

You know, I fell in love with DreamHost in 2007, when I learned that you had been founded by geeks interested in setting up a proper hosting service – “for geeks by geeks”. I tried your support back then and was amazed at the intelligent, technical replies. Since then, I’ve constantly had the satisfaction of knowing that any issue I had would be understood by support. This recent break-in hasn’t changed that opinion at all; you’ve taken care of the issue, and thoroughly explained what was affected, and as a result I am as happy as ever. A quick shell password change for my sites and we’re back on track.

Simon,
We are one of DH’s oldest (a decade+) and largest customer, and I’m sad to say that there has been a complete decline in both the support and the quality of the service since this summer. We are a DoD Federal Contractor who relies on DH to run our internal web based app, and when it goes down we lose time and money. While some problems are to be expected, the last 6 months have seen a level that surpassed downtime for us over the last 3 years! To add insult to injury, there is no acknowledgement or even apology, instead we get canned responses to our tickets (and if we ask for a call-back, if the tech can’t reach us they mark it as our call-back – absurd!).

In short, getting a live human to work with us was always tough, but it has reached unbearable levels now. When we go down, we want to know that our problem is being taken seriously by you, not that we now have to sign up another $10/mnth for the courtesy of maybe talking to somebody.
I would have sent this to your personal email, but even as a corporate customer, I don’t have a DH POC (the absurdity of posting this on a blog, shouldn’t escape you – I’d much rather discuss this privately).

That being said, I have been a loyal DH supporter and customer, and I think 10 years counts for something, so I am willing to speak with you – the question is whether you’re willing to reciprocate.

W/R,

Naveed Jamali
President/CEO
Books & Research

security audit of your website(s) HACKING OF WEBSITES & Hacking Accounts which include facebook,twitter this is pretty easy,myspace,skype,and email ids.I require either a Name, Friend ID, or E-mail address of the targets account(s). I have the help of a current 0-Day Exploit that allows me to gain remote access to the website servers and from there I find the password which is usually in an MD5 hash, from that I must decrypt to get the real password. The entire process takes about 30 minutes-1 hour to complete. All passwords are tested out 3 times before they get issued to any clients.I also rip Standards from websites i semd you a screen shot of the email to confirm.I accept payment through LR (Liberty Reserve) Only.I hardly ever USE WESTERN UNION!
YOU CAN REACH ME ON :kross303@yahoo.com (SEND ME AN IM THROUGH Y! MESSENGER OR MAIL)i also sell bank logins and credit cards