Phishing Phor Phishers

A funny thing happened to me on Tuesday.

Well, really it happened to my wife. But I hear being married is all about sharing.

We had just finished dinner when she casually mentioned we were getting a tax refund.

“Oh?” I responded…

“Yeah, I got an email”

“OH???????”…

I immediately had a sinking feeling.. had she been PHISHED?

I asked if she’d given her credit card number out?

“Yes.”

Social Security Number?

Yes.

MY Social Security Number?

NO! Sheesh, what do you take me for?!

Which credit card?

Our Visa check card.

Oi! That’s a bad one! I’m not sure the kind of fraud protection we have on it, and it’s tied to our bank account directly!

Before even inspecting the email, I called in and had them cancel the card. Hooray, no charges had gone through yet!

Honey, didn’t I warn you before about PHISHING scams?

Well, yes.. but I forwarded it to you on Monday and you never wrote back! So I just did it.

I never saw that email! (Sure enough.. it was caught in my spam filters. Makes sense!)

Couldn’t you have called me on the phone or even asked me in person on Monday night or Tuesday morning?!

I forgot about it until I checked my email again!

Anyway.. let me see the email you got.

And here it was..

Date: Mon, 28 Aug 2006 11:58:14 -0500
To: joshswife@yahoo.com
Subject: Tax Information - joshswife@yahoo.com - (Code 7863-3843)
From: “IRS.gov” Add to Address Book Add Mobile Alert

Account : joshswife@yahoo.com Number : 7863

After the last annual calculations of your fiscal activity we have determined that you are eligible
to receive a tax refund of $191,40. Please submit the tax refund request and allow us 5-7 days in orders to process it.

A refund can be delayed for a variety of reasons. For example submitting invalid records of applying after the deadline.

To access the form for your tax refund, please click here.

Regards,
Internal Revenue Service

Here are the immediate red flags that go off in my head when I get emails like this:

Right off the bat, any email I get from an address I’ve never received one from before has a 99% chance in my mind of being a spam, scam, or some kind of an annoyance.

I never get tax refunds! Ever ever ever. It’s not fair.

The IRS and state taxing authorities don’t send notices via email.

The IRS and state taxing authorities don’t have my email address.

They DO have my name and SSN, and would probably put those in an email, IF they had my email address and IF they sent emails.

There’s a typo in the email.. it says “of” where it should have said “or”.

They used a comma instead of a period for the decimal point in the dollar amount! That may fly in Europe, but god bless the IRS, this is America!

The link takes you to thistlejack.com!

But, believe it or not, my wife is not stupid. In fact, she has a PhD from Harvard!

For real.

Too bad she doesn’t run a web hosting company!

There’s no better training against phishing scams than having dozens of fraudsters a day attempting to send them from your servers!

But for the rest of you LOWLY Internet users, phishing scams work. And I think I know why:

They send a lot of phishing emails.

Just by sending a lot of messages, they’re going to catch a tiny percent of people who were specifically waiting for that email!

Even the almighty Josh nearly fell for an Ebay phishing scam once when I got the phish the very moment I had just won an auction.

And of course, a tiny percent of people are going to go for it even when they weren’t expecting an IRS refund, a paypal payment, or an ebay auction.

They prey on people’s greed or fears.

To my wife’s credit, (she claims) there were a LOT of red flags and alarms going off in her head while she filled out that form. But the lure of the $191,40 was just too strong!

And we’re rich!

People are getting really comfortable with “e-commerce”.

My wife doesn’t really care too much about giving out her credit card info online. Really, why should she? You’re not generally liable, and we should have the replacement card in the mail tomorrow. I do wish she was a little less comfortable with giving out her SSN though…

The thing is, how often in the real world do you come across an individual or business who is really trying to scam the crap out of you? Hopefully not too often in this country at least. It just doesn’t really happen. But on the Internet, it really does happen. Millions of times per day.

Fortunately, a lot of people are still deathly afraid of this “Internets”, and won’t give out anything to anybody! Or maybe that’s not fortunate.. because really, you’re not generally liable.

People are technically naive.

Honestly, it’s pretty easy to look at a URL and know if it’s legit.

Or is it?

I was trying to explain to my sister-in-law how to know. Basically the best I could do was “If the VERY first part of the URL is the correct domain name, and only the domain name, and doesn’t have a dash or something before it, but it’s okay if it has a dot before it, as long as it doesn’t have a slash before the dot, then it’s the right site!”

In fact, my wife was even like:

Well, I knew thistlejack.com wasn’t irs.gov, but you know how sometimes websites link off to some other server for their payment processing? And when I clicked all the links on the site, they were legit.

Because the links WERE to irs.gov!

Even the fact the page wasn’t secure didn’t faze her!

What was I to do?

I already canceled the credit card. But I wanted more! I wanted to shut this guy down, and I wanted to make sure nothing happened to my wife’s SSN.

First, I did a whois lookup on thistlejack.com and called the owner, Mr. Robert Stirling.

I knew he wasn’t the phisher. Nobody in the US phishes, and nobody uses real contact info when registering a domain for phishing! It looked like from the URL that the phisher had exploited a hole in a photo gallery script he had installed. (Which is why we have mod security for our happy hosters!)

Fortunately, he answered the phone.. I explained the situation and he was very, very, cooperative and helpful!

He logged in to his domain, took the phishing site down (it’s down now), and then at my request emailed me the source code for their web form. I wanted to see what was happening to the data.

Just as I might have guessed, it was being emailed off to two separate anonymous yahoo.com email addresses.

I immediately emailed abuse and postmaster@yahoo.com, got a tracking number back and started waiting. And waiting. (I’m still waiting…)

I couldn’t wait anymore!

I had to do something (besides call the credit reporting agencies and tell them what happened)!

And then it hit me!

Maybe I could fill this jerk’s mailboxes with enough BOGUS DATA that he’ll just give up on it all and not realize that my wife’s info was for reals!

Of course, it wouldn’t be too hard for him to realize all submissions after a certain time were fake.. but hey what did I have to lose?

I took the source code from that script and made up my own that sent an identical email to those two addresses, but with randomly generated info!

It was fun!

I set it up with a cron job to run every 20 minutes (but I put a random sleep of 1-20 minutes at the front so they didn’t come in too regularly).. it’s still going right now.

I’m going to keep it going until I hear back from Yahoo!.. and just FYI, here’s the output they were receiving from their phish:

Date: Thu, 31 Aug 2006 16:58:15 -0700 (PDT)
From: thistlej@server4.whmsecure.com
To: phisher@yahoo.com
Subject: IRS - Full

[ . . . : : : IRS FOUNDS : : : . . . ]
Social Security Number: 356 - 00 - 0258
Name On Card: Robert Rieger
Card Number: 6105341453830068
Expidation Date: 12 / 2007
CVV: 123
PIN: 5702
[ . . . : : : IRS FOUNDS : : : . . . ]

(Don’t worry, that’s a fake one I generated!)

In closing…

Phishing scams are pretty darn effective. They’re tricky, and they’re lucrative!

Anyway, my wife’s pretty embarassed about the whole thing and made me promise not to tell anyone.

Ask DreamHost Customers

I’ve got a question.

And I thought, who better to ask, than everybody?

Here goes…

We’ve got pretty serious storage needs. Like, in the next year, we’re estimating needing about 250TB (big T, big B) of additional centralized, networked, storage.

Besides needing a lot, we also need very high performance, redundancy, and thrift.

We want it ALL!

Our absolute requirements for our system are as follows:

* RELIABILITY .. we can never ever lose any data, ever.
* PERFORMANCE .. we need something that can serve approximately 3000 NFS ops per second per TB. (See spec.org). (It needs to do NFS.)
* PRICE .. that’s the whole reason we’re looking.

We’ve got a good system of RELIABILITY and PERFORMANCE already.. but the cost per usable GB is $10. The main problem is the 300GB Fiber Channel drives we use, which are $800 each. Is there anything out there that can do the same but with SATA drives that cost more like $100? Even if we needed twice or four times as many drives for the same performance and reliability, it seems possible!

There are also some REALLY WANT TO HAVE features, though possibly could be passed up if the top three are satisfied.

* SNAPSHOTS .. automatic versioning backups of all files by the OS. We’ve got this now, in a hidden “.snapshot” directory in every folder.. check it out!
* USER QUOTAS .. really, with the amount of space we’re giving out these days, quotas are almost a moot point. They’d be nice to have though.
* HIGH DENSITY/LOW POWER .. it’s always a plus to fit more storage in the same amount of space with the same amount of power, but it’s not really that big a deal.
* RAID 6 SUPPORT .. it’s cool.

Here are some vendors/solutions we’re considering..

* NETAPP .. what we use now.
* BLUEARC
* ONSTOR
* PANASAS
* CORAID (Open Source, ATA over Ethernet.. intereesssssting!)
* LUSTRE (Open Source, Clustering storage)

Soo.. basically, if people could post their suggestions, experience, other solutions, etc.. in the comments, it would be much appreciated. Not to mention you will be doing your patriotic duty to improve your hosting forever!

And remember, we’re talking serious NFS ops.. and we’d be willing to buy 256TB at once if our tests showed this system can do what we want and THE PRICE IS RIGHT!

YOU C@#KS#CK’N SPAMMERS!

As you’ve been reading on our blog over the past few weeks we’ve been dealing with a few ongoing problems. As Josh has told you we’ve sorted through most of the major issues and things have started to settle down lately! We’ll be sure to post more information when we have more to report right here, and for up-to-the-minute action don’t forget to check out dreamhoststatus.com.

I check our general voicemail box every day. We don’t offer traditional ‘dial-in’ phone support, so most messages that get left on our voicemail are either from our vendors or from smarmy sales people looking to partner up with the 25th largest web host in the world! (That’s us, thank you very much!)

Sometimes we’ll even get automated telemarketing messages from our competitors. Those are a real hoot.

On rare days we’ll receive messages that roundhouse kick us right in the moneymaker with their awesomeness. Today is one of those days.

I tend to have trouble keeping things under wraps, so I’m sharing. This has to be heard.

These voicemails contain pretty tough language and are definitely not work-or-child-safe.

We’ve bleeped out phone numbers and email addresses to protect the enraged…(I didn’t have a microphone handy so I did my bleeping through the business end of a pair of cheap headphones.)

Angry Voicemail Part 1
144k, .wav file

Wow! That guy’s really really mad! He gets as angry about spam as I did in oh, say, 1996.

When I first played the message to coworkers, some of them peed their pants in fear. One guy pooped and had to go home early. They were softies and needed to toughen up anyway. The Internet is serious business.

Hissy-fit notwithstanding, if there are spammers on our network we want to know about them! We did check out the email address he mentioned and it didn’t appear to be running afoul of our tough anti-spam policy. We’re pretty sure he was just on the receiving end of a few bounced messages with forged headers. And no, we didn’t call him back. As a general rule we don’t return abusive emails or phone messages.

Maybe all that threatening and yelling weighed heavily on his soul. It must have, because he called us back half an hour later, making sure to stress that he’s going to take us on LEGALLY. He’s going to kick our a$$ LEGALLY:

Angry Voicemail Part 2
478k, .wav

Someone should really let the postmaster in Huntington Park know that there’s a guy headed their way and he HATES SPAM. We’d do it ourselves, but we moved out a year and a half ago.

This is Brett at F$ckinDreamHost.com, signing off!

Reconstruction Efforts

Well, things could be worse.

We’ve pretty much got our whole network under control now.. the ongoing problem mentioned last post was finally figured out by Cisco support. It turns out it was a bug undocumented feature in IOS dealing with how they learn MAC addresses.

There was also another network problem we got resolved yesterday that was causing general slowness on web and mail servers. It’s complicated (i.e. I don’t understand it exactly myself), but in the end we took a distribution switch out of the network and that fixed it.

We still have one open ticket with Cisco for our core routers having some HSRP problems. It doesn’t seem like that’s having any real effect on our network, but we want it fixed!

We are also installing two new Ciscos to offload the BGP duties from the core routers so they’ll just have to handle switching. This set-up should be able to handle about 300% more traffic than our entire network now pushes at peak times!

Thanks to these network problems being resolved, we’ve also begun re-deploying in Alchemy, who at least didn’t have the second power outage.

We’re also still in the process of getting real UPS power on our network cabinet, plus our internal databases and a few internal servers. Basically, everything that keeps all the customer mail, web, database, and file servers from coming right back up quickly should there ever be another outage.

So, um.. that’s how it stands now! We hope this will all soon be nothing more than a long bad dream (that was real).

Anatomy of a(n ongoing) Disaster..

What a three weeks…

As I’m sure most of you already know, we’ve had nothing but troubles, large troubles, for pretty much the last three weeks. A lot of these troubles were our fault, a couple of them were at least ostensibly beyond our control, and they all compounded each other.

Here I’ll try and go into as much detail as possible about what happened, why, and the steps we’re taking to stop this sort of thing from ever happening again. I can’t excuse what happened, just apologize and hopefully elucidate.

Ironically, all the recent disasters stem somewhat from us attempting to take some proactive steps to head off any sort of future power outages like the kind we experienced last year.

The Back Story

As some of you may know, we are co-located with Switch and Data in The Garland Building in downtown L.A. To say we’re co-located is a bit misleading though, since we’re now basically 95% of their data center.

Why don’t we have our own data center?

Because, believe it or not, we’re still not big enough for it to make sense. Even now, we only use about 1000 sq ft of data center space.. for it to really start to make sense to get our own space, we’d have to be using around 2500 sq ft. Mainly because when you buy a data center, you want to get one big enough to handle a lot of growth.. and although it’s cheaper per square foot than co-locating, you have to pay for all the space you’re not using yet.

And really, The Garland Building is supposed to be an excellent place for data centers. There are more than a dozen in the building. Companies like iPowerWeb, Media Temple, BroadSpire, and even MySpace (now the most popular website in the whole US!) are in there. It’s got FIVE huge generators, UPS for the whole building, on two separate power grids, and a dedicated engineering staff to make it all work flawlessly. Or so we were all assured.

Around last June though, the building informed all its data center tenants that they had essentially run out of power! Not power altogether, but the “good” power that data centers need.. i.e. ups and generator-backed power. Because Wells Fargo, who holds the master lease on the building, wasn’t sure if they were going to renew the lease when it is up in three years, they didn’t want to invest the millions of dollars to add more generators and ups to increase capacity. This is in fact the primary reason we’re still not selling any more dedicated servers .. they use too much power per dollar!

Of course, none of that was supposed to have any affect on their ability to keep the current power going in the case of an outage. September 12th, 2005 we discovered they actually couldn’t… when two of the five generators failed!

However, since then, the building has repaired and replaced the faulty generators, and given all their tenants numerous assurances that what happened before would never ever happen again.

Why didn’t we move data centers right then?

That would have been a fairly massive undertaking, resulted in even more down time, been very expensive, and actually we did look around and there weren’t any really good options for moving… data center space is becoming pretty tight (in the LA area at least) and the Garland Building is still one of the best options, believe it or not. Also, this was the first time something like this had ever happened, and it seemed pretty reasonable that it wouldn’t happen again. We even asked around and none of those other tenants mentioned above were moving, so I guess it seemed like people were generally pretty confident it was a one-time freak occurrence.

Nevertheless, we started making contingency plans, searching around for another data center that had some power and would make sense for us. Eventually, we found Alchemy, just down the hall from S+D actually, and began making arrangements for getting some space from them. They had a little bit of power available because they were moving some of their clients out to El Segundo, and because they had gotten permission from the building to install their own generator. With that generator and some UPSes they were able to convert a “dirty” power feed into “clean” (i.e. good for data center use) power.

How the troubles began.

All this took a very, very, very long time. After months of searching and negotiating with Alchemy, we still had to get Switch and Data to allow us to put a cross-connect in from their data center over to their competitors down the hall. After even more months and teeth-pulling, we finally got that up and running. In fact, we finally got the first live server up in Alchemy a little less than a month ago.

All this in an attempt to head off future power problems.

Unfortunately, shortly after setting up the new footprint, we noticed something wasn’t right. Getting to Alchemy from Switch and Data we would lose huge buckets of packets. Just as we were trying to figure out the problem, we started to have problems with one of our file servers.

This resulted in a lot of problems across the board. The web servers that mounted that filer all had problems. The mail servers that mounted that filer all had problems. In fact, one of the mail servers was mis-configured and was logging thousands of errors a second to a remote logging machine… so many in fact that it was saturating its switch and clogging up a whole chunk of our network. Which in turn caused other machines to get slow and crashy because they couldn’t get to their filers, and so on and so on.

It turned out the filer problem seemed to stem from the fact that we had one shelf of 300GB disks and one shelf of 150GB disks on it. Apparently they’re not supposed to be able to support this, or at least it’s a bad idea. So, this was entirely our fault. However, we did have a number of other filers we did this on, and we’d never had problems before. Nonetheless, we will never mix disk shelf types on a file server again.

We eventually cleared all this up.

However, the Alchemy connection problems were still ongoing.

After trying all sorts of things, we eventually decided to replace one of our distribution switches that was acting strangely with a new one. This didn’t really seem to fix the problem either. This was on Friday, July 21st.

On Saturday, July 22nd, the building lost power.

This time, the generators actually worked, but the UPS failed! Honestly, it was much better than last year’s.. but unfortunately, even a brief power outage wreaks havoc on a data center. And this one wasn’t so brief.. here’s the building’s explanation:

At around 5:21pm, on Saturday July 22nd, a brown out occurred due to record high temperatures in downtown Los Angeles. Voltage dropped due to the high demand of electrical current along with equipment failure operated by the Department of Water and Power, City of Los Angeles. This condition caused failure of “ATS-B” switch and to UPS Module #3. Engineering crews were dispatched and began repair of this damaged equipment. A power interruption was required to replace contacts in “ATS-B”.

Repair of “ATS-B” failed contacts was completed on 7-24-06. Power was restored between 4:00am and 4:30am by the Engineering department.

Thank you,
Office of the Building

So, after all the emergency filer stuff going on the previous weekend, just about the entire admin team was back last weekend, working on getting everything back up when power came back on. Even when we had power, it was in a degraded state and so the cooling wasn’t working. As temperatures rose, file servers automatically shut themselves down rather than risk being damaged by the hostile environment. Apparently, MySpace made the decision to just keep all their servers off until cooling was restored.

More network troubles..

After the power outage, we decided to just yank everything back out of Alchemy (they lost power too!) until we could figure out what was going on with the network to there. Unfortunately, this didn’t seem to fix things, and our internal (”red”) network was still really fubar. When our red network isn’t working, the panel isn’t working, webmail isn’t working, and our server configuration system starts having problems (basically, anything that connects to our internal databases).

It took us just about all of Monday to figure out (and then fix) that a lot of the file servers had bad routes after being powercycled.. and so were sending ALL their traffic through the red network, saturating it. These things are generally pretty stable and a lot hadn’t been rebooted since September 12th, 2005.. and some had apparently had their networking set up by hand instead of correctly configured via our database. We’re making sure that doesn’t happen anymore either.

More network troubles..

Once that was fixed, things generally got better. Except there was STILL strange stuff going on (causing slowness and high loads around the system, but not an actual system-wide outage), even without NFS traffic going through red, and even without anything at Alchemy. It started to look like there was a problem with one of our core routers. We called our Cisco consultant and opened a trouble ticket with Cisco themselves..

More power problems..

On Friday, July 28th, we lost power again. The building wrote:

The Garland Building experienced a dead short which resulted in a brief power outage today, July 28, 2006. The air conditioning, elevators, and the electrical utility have all been restored.

While on generator power, a dead short occurred from one of our internal telecom users. We are investigating where the dead short occurred. A follow-up memo will be sent by the end of the business day reconfirming our transfer at 11:30pm tonight. We are currently on DWP power until further notice.

And then:

The Garland Building UPS System is back on-line supplied by DWP. Diesel generators have returned to an on-call status.

The 11:30pm transfer has been cancelled due to the dead short prematurely returning us to utility power. At 4:30pm the engineers engaged the UPS System to protect all tenants at the Garland Building.

Thank you,
Office of the Building

This time, we were able to get our entire system back up much quicker and with close to no problems. Of course, it had been less than a week since our last power outage.

Alchemy was the only data center in the building who did not lose power this time.

More network troubles..

Over the weekend (this last weekend), we kept having the same ongoing weird network problems I mentioned above, and Cisco hasn’t made much progress. Yesterday, we realized the new distribution switch (an extreme) was causing spanning tree problems with the older Ciscos. Jeremy got it all figured out, but in the process it erroneously blocked our “green” (public!) network for a few brief periods, taking down everything again.

Unfortunately, that fix STILL doesn’t seem to have fixed the ongoing core network problems. We were finally able to get our tickets escalated with Cisco yesterday. It is starting to look like something may have been damaged during the first power failure, although we’re not sure. The replacement/repair cost might be around $80,000 it looks like.

And that’s where things stand today.

Our number one priority right now is getting this nagging network problem understood and fixed. Once that’s the case, we should be able to put things back in Alchemy, who didn’t lose power on Friday at least. Once things are going good there, we’ll be able to add new servers and transition old ones slowly with little to no downtime.

We’re also going to be buying our own UPSes, since we have learned we can’t trust our data center OR our building to do it. We’ll start by putting the core routers on them, then our internal databases and servers, then our file servers, and finally the hundreds of customer mail, web, and database servers.

Finally…

We’re very sorry for what happened. We definitely don’t want it to happen again, and we’re trying to take all the practical steps we can to prevent it. We never want to have another July 2006 again.

Ironically, some of the network problems seem to have stemmed from us trying to better protect ourselves from power failures. I also want to say for the record that none of these problems in my opinion stemmed from “overselling”. Rather, I’d say it’s the result of bad luck. And incompetence on our (and the building’s) part.

I don’t know if we’ll be able to change our luck, but hopefully we’ve at least learned something and will be able to become a tiny bit less incompetent in the future.

I hope you’ll all stay with us to find out.