Question: For the database server that was breached, did the table that had the legacy “plain text” passwords ALSO contain the Full Name data associated with each password? (Real user name adjacent to plain text password?).

Or was this a breach of an entirely separate server, independent of any specific machine hosting the user accounts? Again, the real question being: was there any Full Name data stored in the table containing the legacy plain text passwords?

In short, getting a live human to work with us was always tough, but it has reached unbearable levels now. When we go down, we want to know that our problem is being taken seriously by you, not that we now have to sign up another $10/mnth for the courtesy of maybe talking to somebody.
I would have sent this to your personal email, but even as a corporate customer, I don’t have a DH POC (the absurdity of posting this on a blog, shouldn’t escape you – I’d much rather discuss this privately).

That being said, I have been a loyal DH supporter and customer, and I think 10 years counts for something, so I am willing to speak with you – the question is whether you’re willing to reciprocate.

W/R,

Naveed Jamali
President/CEO
Books & Research

The issue at hand is the extremely verbose headers appended to outgoing emails. Look at services like Gmail or any other provider set up for security; you’ll find NO “dangerous” info in the headers, and the receiver won’t be able to identify or harass or attack you. Compare that to these example DreamHost headers (sensitive data has been randomized below):

Received: from Mac-Pro.local (example-34324.isphostname.com
[123.123.123.123])
(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
(No client certificate requested)
(Authenticated sender: account@mydomain.com)
by homiemail-aXX.g.dreamhost.com (Postfix) with ESMTPSA id BF52FBF2
for support@somedomain.com; Wed, 23 Nov 2011 12:28:20 -0800 (PST)
Message-ID: 3ECD5761.8060304@mydomain.com
Date: Wed, 23 Nov 2011 21:28:17 +0100
From: Richie richie@mydomain.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0)
Gecko/20111105 Thunderbird/8.0
MIME-Version: 1.0

DreamHost reveals to the recipient:
* That I am running on a hostname of “Mac-Pro.local”, which reveals the type of machine I am using.
* That I am using “isphostname.com” as my ISP.
* That my external web IP is 123.123.123.123, which can be used for a GeoIP lookup to find my location down to the city I live in.
* That I am sending the email as originating “From: Richie richie@mydomain.com” but that my ACTUAL account name ON THE EMAIL SERVER is “Authenticated sender: account@mydomain.com“.

The user-agent headers are my own fault and are put there by the email client, revealing the OS version, etc, but that header alone would be ALRIGHT if there wasn’t also a revealed IP thanks to DreamHost.

An attacker could take this info, see the type of OS that the machine is running, and start a port probe with an exploit tool on that IP. They could also take your name and find your city via the IP and then look for you in the white pages. They could take the ISP name and look at coverage of that city to rule out all possible addresses if there’s more than one match in the white pages. Often ISP hostnames even helpfully give out the area name, such as “nyc-brooklyn-10203.someisp.com”

These are the reasons why security-conscious hosts, and even general-purpose hosts like Gmail, COMPLETELY skip such pointless and dangerous headers. They don’t have them AT ALL.

Why is this a problem? Because there is NO reason (other than nefarious purposes) for the entire world to know the internal IP and system name of the sender, along with the actual mailbox login name.

All email providers that take security really seriously will mask out the hostname and IP as follows, or omit it entirely:
Received: from localhost (unknown [127.0.0.1])

Worse yet, there are some of us really techy users that use one mailbox as the actual login, and then send using multiple aliased mailboxes; i.e. account@domain.com may be the login, but you might send as example-com@domain.com in all your communication with people from “example.com”. The purpose of this is that you never expose your true email, so WHEN (not IF) the spam starts rolling in, you just add an “auto-delete as garbage” filter to the alias that is now getting spam, and you’re once again spam-free. Well, that whole thing goes out the window when the headers proudly proclaim:

(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
(No client certificate requested)
(Authenticated sender: account@mydomain.com)

I’ve already had spam sent to my internal, secret mailbox name, due to the headers exposing this info, and have had to change the internal mailbox.

So, what should optimally be done to fix this reckless display of dangerous headers?

* Stop tagging the emails with the hostname and IP of the sender. (You can rely on the message-ID for anti-abuse purposes). There is no reason other than nefarious purposes for the rest of the world to see this info.
* Stop tagging the emails with the “authenticated sender” line, to prevent leaking the actual mailbox login information.

Sorry for putting this suggestion here, but it’s a serious issue that’s giving me headaches at DreamHost, and it’s a significant enough change that it would take configuration changes of your email servers across the board to get rid of this reckless security issue.

- Richie

The issue at hand is the extremely verbose headers appended to outgoing emails. Look at services like Gmail or any other provider set up for security; you’ll find NO “dangerous” info in the headers, and the receiver won’t be able to identify or harass or attack you. Compare that to these example DreamHost headers (sensitive data has been randomized below):

Received: from Mac-Pro.local (example-34324.isphostname.com
[123.123.123.123])
(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
(No client certificate requested)
(Authenticated sender: account@mydomain.com)
by homiemail-aXX.g.dreamhost.com (Postfix) with ESMTPSA id BF52FBF2
for ; Wed, 23 Nov 2011 12:28:20 -0800 (PST)
Message-ID:
Date: Wed, 23 Nov 2011 21:28:17 +0100
From: Richie
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0)
Gecko/20111105 Thunderbird/8.0
MIME-Version: 1.0

DreamHost reveals to the recipient:
* That I am running on a hostname of “Mac-Pro.local”, which reveals the type of machine I am using.
* That I am using “isphostname.com” as my ISP.
* That my external web IP is 123.123.123.123, which can be used for a GeoIP lookup to find my location down to the city I live in.
* That I am sending the email as originating “From: Richie ” but that my ACTUAL account name ON THE EMAIL SERVER is “Authenticated sender: account@mydomain.com“.

The user-agent headers are my own fault and are put there by the email client, revealing the OS version, etc, but that header alone would be ALRIGHT if there wasn’t also a revealed IP thanks to DreamHost.

An attacker could take this info, see the type of OS that the machine is running, and start a port probe with an exploit tool on that IP. They could also take your name and find your city via the IP and then look for you in the white pages. They could take the ISP name and look at coverage of that city to rule out all possible addresses if there’s more than one match in the white pages. Often ISP hostnames even helpfully give out the area name, such as “nyc-brooklyn-10203.someisp.com”

These are the reasons why security-conscious hosts, and even general-purpose hosts like Gmail, COMPLETELY skip such pointless and dangerous headers. They don’t have them AT ALL.

Why is this a problem? Because there is NO reason (other than nefarious purposes) for the entire world to know the internal IP and system name of the sender, along with the actual mailbox login name.

All email providers that take security really seriously will mask out the hostname and IP as follows, or omit it entirely:
Received: from localhost (unknown [127.0.0.1])

Worse yet, there are some of us really techy users that use one mailbox as the actual login, and then send using multiple aliased mailboxes; i.e. account@domain.com may be the login, but you might send as example-com@domain.com in all your communication with people from “example.com”. The purpose of this is that you never expose your true email, so WHEN (not IF) the spam starts rolling in, you just add an “auto-delete as garbage” filter to the alias that is now getting spam, and you’re once again spam-free. Well, that whole thing goes out the window when the headers proudly proclaim:

(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
(No client certificate requested)
(Authenticated sender: account@mydomain.com)

I’ve already had spam sent to my internal, secret mailbox name, due to the headers exposing this info, and have had to change the internal mailbox.

So, what should optimally be done to fix this reckless display of dangerous headers?

* Stop tagging the emails with the hostname and IP of the sender. (You can rely on the message-ID for anti-abuse purposes). There is no reason other than nefarious purposes for the rest of the world to see this info.
* Stop tagging the emails with the “authenticated sender” line, to prevent leaking the actual mailbox login information.

Sorry for putting this suggestion here, but it’s a serious issue that’s giving me headaches at DreamHost, and it’s a significant enough change that it would take configuration changes of your email servers across the board to get rid of this reckless security issue.

- Richie

I just made a re-visit at this moment on the off-chance that you had responded. I was very grateful to see that you had indeed!

Thank you so much for the very honest reply. It finally explains the details that had been bothering a lot of us more technical users.

Alright, so it *did* have up-to-date data as of the date of the attack (in other words not “legacy” as another DH employee had called it on the status-blog), and it was used for internal service authorization (but you’ve thankfully changed that system now), and you have determined that this is all the attacker had access to, and that there’s no evidence that the attacker even dumped this table (you just changed passwords out of precaution), and even better – the suggestion to change email passwords was based on the off-chance that someone had the same password on the email account as their shell account.

Fantastic answers and I am really glad to hear the reasons for each issue. It lays to rest any worries that I’ll have to change my email passwords once again, since they weren’t in the table, and had nothing to do with the shell passwords.

You know, I fell in love with DreamHost in 2007, when I learned that you had been founded by geeks interested in setting up a proper hosting service – “for geeks by geeks”. I tried your support back then and was amazed at the intelligent, technical replies. Since then, I’ve constantly had the satisfaction of knowing that any issue I had would be understood by support. This recent break-in hasn’t changed that opinion at all; you’ve taken care of the issue, and thoroughly explained what was affected, and as a result I am as happy as ever. A quick shell password change for my sites and we’re back on track.

Question 1: Why did you have this table of plaintext passwords in the first place? My guess is that it had something to do with the password change feature and its way of warning you “your new password cannot differ from the old one only by case” that some users pointed out?

[Simon] This was not anything to do with the password change feature. Our (very) old architecture used plain text passwords in some instances for internal service authorization purposes. We’d changed over to secure passwords a long time ago for this purpose. However this table was never cleaned up. FYI, we have cleaned all of this up now, and made some additional changes to password management as will be outlined in our newsletter to go out this week.

Question 2: You call this a legacy table. When was it last updated? I had actually changed ALL my user passwords and email passwords on January 5th, 2012. Was my data in that table or am I safe?

[Simon] The table was up-to-date as of the date of the intrusion.

Question 3: What columns were in the table? I am guessing machine (like beid.dreamhost.com), username and password, and probably the account ID of the DH account that owns that user. Anything else that was lost?

[Simon] No personally identifiable information was in the table. The FTP/shell password and host machine was in the table. Note that we have no evidence that the hacker actually got the table from the other internal machine it was dumped to. But in case they did, we took the precaution of resetting the FTP/shell passwords.

4: Are you SURE this is all the hacker got access to?

[Simon] Yes.

5: You say that we should all change email passwords too. Why? In my case, they were last changed on January 5th, and had nothing to do with ANY other password I own. Why should I change these if the hacker didn’t get access to them? Please clarify what you know about the hack and what got out.

[Simon] We suggested this as a precaution, because many users often use the same passwords across different services.

We understand that security is extremely important, and have taken a whole series of steps to protect and mitigate against this kind of intrusion happening again, including hardening our internal systems and firewalls. I trust you understand that we can’t go into too much detail on our current defenses, given this is an open forum. However we will be including more information on password security precautions in an email direct to customers to go out this week.