Yesterday I noticed Jason and Jeremy were both wearing the same shirt!

I’d call it a coincidence, except… they’re roommates!

Awwwwwww..

Hey, you know what’d be fun on a boring Monday in May? A little role play!

And I’m not talking about 12-sided dice and renaissance faires either, I’m just talking about some simple role reversal.

More specifically, I’m going to complain to you about a web host!

So, about three years ago I was trying out some competitors to, you know, test the waters in case I ever decided I wanted to switch hosts.

I used three places, and they all absolutely stank. I mean, they were horrible. I’m talking worse than us!

Every server I tried with these places was pretty much just not working. Besides that, their support was all universally useless, and their panels were a weird hodge-podge of different systems they’d cobbled together I guess. You also either couldn’t get shell access or had to fax them your driver’s license to enable it?!

The worst thing was, they were all difficult to cancel, and a few even tried to get out of giving me my money back (I was in their “unconditional” guarantee!)

Finally, I decided to splurge (I’d been spending like $7.95/month) and tried a VPS place for a whopping $49 a month!

Well, they were great! I mean, they still had a weird hodge-podge of different panels, and they sure laid stuff out differently than I was used to, but my VPS at least stayed up and I could do anything I wanted.

I never needed to contact support, which was fine with me, and luckily for them, I never really did too much with the account but kept paying them anyway for the last three years (I’m willing to bet a few of you are in this boat as well… thanks!)

Finally

Last month, I finally decided to transfer my little bit of crap I had with them over to a DreamHost PS! When I went to cancel, I decided to check my credit card statement and noticed that for February and March I’d been charged $89 instead of $49?!!

Eh? I searched through all the emails I received from them and the only thing I could find that seemed possibly related was one that mentioned they were upgrading all the features on their VPS, but don’t worry existing users would get them all at the same price!

So, I wrote them a nice email:

Helllooooo….

I just noticed this and that somehow you guys upgraded me without my permission from $49/month to $89/month!

Uh, what happened? It wasn’t my choice.. I did get one email saying resources were going up .. for FREE.

Please refund the extra $80 you’ve charged to my credit card asap.

Also, I’d like to cancel my service as of April 30th, I believe what I’ve already been charged for.

Thanks,
josh!

To which they replied:

Sir,

On 02/17/08 our support team notified you to tell you that your server had run out of resources, and that the only way they could keep your server from staying offline was to upgrade you. They did so for free for one week, and asked you to get back to them to work with them to resolve the issue. They stated that if they didn’t hear from you they would leave you on the higher package level instead of leaving you down completely.

After a week, and a followup reminder sent to this address that the account was being left at Signature level so that you could remain operational, your package was upgraded.

Admittedly this was an atypical situation, but most would probably agree that after not hearing from you the decision to leave you up and operational was preferrable to the decision to simply let your server fail.

As per the contract you agreed to at signup, we do require a 30 day written cancellation notice to close down your account. I can accept this as that notification and close your account 30 days from today, on May 18th. I hope that this helps.

All the best,
Christian

Ha, ha, ha… what?

So, because I was (somehow) crashing my own (private) server, they, without permission from me, started charging me an extra $40 a month, so it wouldn’t crash!

Gee, thanks guys!

I also appreciate it when my cable company notices that I haven’t been enjoying HBO and Showtime and most would probably agree that after not hearing from you the decision to give you all these great movies and original tv series was preferrable to the decision to simply let you suffer with Oxygen and TBS!

But actually, that never happened becuase that would be CRAZY!

I went back to look for this alleged email, and I found it:

Subject: 7 Day Trial upgrade to the Signature package for yourserver.com.

Hi,

This server has reached it’s limit on i-nodes which is number of files on the system.

Below is an output of where most of these I-nodes are being used:

357219 -> /vz/private/1753/root/var/qmail/mailnames/yourserver.com/user/Maildir/cur
457677 -> /vz/private/1753/root/var/qmail/mailnames/yourserver.com/user/Maildir/new

That is roughly 700,000 i-nodes for this mail account. Please clear this mail out and notify us within 7 days so that we can downgrade your account back to the Essential. Otherwise, you will be billed for the Signature package.

Thank you,
Tommy

First off, nice subject! No wonder I didn’t read that email!

Ah, I see.. I had a catch-all at the domain hosted there and it was filled with three years of spam!

It’s besides the point that there’s no mention of inode limits anywhere on their site or tos (I’m not saying who they are because there’s no such thing as bad publicity!), or that I guess their VPS solution has problems with some instances affecting others in certain inode-related areas.

The point is that it is crazy to assume that you may just UPGRADE your customer without hearing back from them, as opposed to say, just DISABLING their account.

I wrote back:

Hi Christian,

Um, actually no, I would have preffered to have the server fail.. I’m sorry I didn’t see those emails, but I did not agree to the upgrade!

Please refund the $80 extra dollars and set my service to cancel on May 18th, after downgrading back to the $49 plan for the rest of the time.

Thanks,
josh!

To which Christian replied:

Josh,

I understand that some people may feel this way. That’s why we gave you free time at Signature level before keeping you there, and the opportunity in successive messages to go ahead and downgrade. We made multiple contact attempts and then provided the service, which you used for two months.
I’ll need to look into the possibility of refund. I’m not sure what the protocol is offhand, so I’ll need to do some digging.

I’ll downgrade your account immediately but if the same problem exists I expect your server to start failing again shortly. If it does, you’ll need to upgrade an I won’t be able to authorize a free upgrade - not with a dispute pending. So make sure that if the server fails and you’re comfortable with that, that if you change your mind you will need to explicitly agree to the new $89 per month rate.

-Christian

HA! Man, at this point I was starting to get bemused and maybe even a little bit angry. Here I am, a guy who totally loved this host, had paid them about $1800 over three years while using virtually no resources, and they’re going to make me fight over $80 at the end?!

Especially when they have no chance in actually keeping it. I happen to know as something of a dabbler in the web host arts myself that it is very very hard for an Internet merchant to win a chargeback dispute with a consumer! My next email brought this up:

Hi Christian,

Please refund the $80 or I’ll have to take it up with my credit card company directly! Yuck!

Thanks,
josh!

Oooh, but he was not intimidated!

Josh,

I will need to take this up with our Controller. My personal opinion is that you were given clear and fair warning of the charges which were not put in place until after a lengthy period in which we provided that upgraded service for you free of charge. We made multiple efforts to contact you and it was your responsibility to keep your contact information updated with us, or in this case keep messages from your provider whitelisted so that we could communicate with you. As you were given plentiful and frequent notice of the upgrade and the consequences for not responding, as you utilized the resources and received benefit from them through multiple billing cycles, and as all of this can be documented, I am certain that we could be victorious contesting a chargeback request. However, as I stated previously this is not my call. What I will do is send this along to our Controller for review, and set your cancellation date to May 18th as promised. Though normally it is not allowed to downgrade and provide cancellation notice at the same time, given the odd circumstances I WILL allow that request to stand, which will save you some funds.

I hope this helps,
Christian

Oooohohhohoohoooo! Well! I hope it helps too! I am so grateful you are now allowing me to “downgrade” to the only plan I ever signed up for!

Anyway, long story short, they said it’d take two weeks to decide, so I contacted American Express and disputed the charges, and then a few days later they credited my $80.

And the moral is, billing issues are the biggest issues for consumers! Why burn up three years of good will at $49/month over $80? Before this, I honestly would have recommended them to people if I hadn’t been their direct competitor! I swear!

People can forgive a lot of bad service/bad product/headaches/incompetence/gross negligence if you just give them back their money. It’s kind of like saying, “the deal is off,” no hard feelings?

It is 100% worth it. Now, when they talk to their friends, they’ll be like “Well, I had a bunch of problems, but in the end they gave me my money back.”

As opposed to me who’ll be like, “They were fine until the end when they stole $80 and refused to return it! I PLEDGE ON MY UNBORN CHILDREN THAT DREAMHOST SHALL CRUSH THEM!”

That’s something that translates across all businesses too, because it’s just a universal way of doing business. It doesn’t matter what industry you’re in, nor what product or service you have, there are good ways of doing business, and there are not so good ways.

And I feel like although we don’t always succeed 100% at the specific details of trying to offer awesome web hosting for super cheap, we are generally successful at running a business that doesn’t lie, cheat, or steal, and always tries its best.

Now, you guys be me and please go write a ton of blog posts I can use the rest of my life.

Thanks!

We’re really sitting on something special because from what I’ve heard there’s not a lot of blue balls out there.

Today is Earth Day!

And to celebrate, we’re doing nothing! Not a thing. We’re not even singing to our plants.

We must hate the Earth and be horrible people, right?

Wrong!

We’re not doing anything special because we already rock the earth-friendly workplace every day!

With this being the Earthiest day of the year, it bears repeating that DreamHost is a carbon neutral company. In fact, we’ve been carbon neutral for an entire year!

To date we’ve neutralized over 2800 TONS of carbon emissions and are on track to wipe out a total of 3400 by June of this year.

To give you some idea of just how much that is…One ton of emissions are created when you…

• Travel 2,000 miles in an airplane.
• Drive 1,350 miles in a large sport utility vehicle.
• Drive 1,900 miles in a mid-sized car.
• Drive 6,000 miles in a hybrid gasoline electric car.
• Run an average U.S. household for 60 days.
• Have your computer on for 10,600 hours.

source

All those servers in our datacenter use up a lot of juice - life-giving electrical juice which in turn is created by burning lots of dead dinosaurs. And therein lies the problem!

We’ve already switched to using lower-power CPUs in our fleet of hardware, and just last year we introduced DreamHost PS!

A DreamHost Private Server gives our customers a greener alternative to truly dedicated hosting. Why get (and pay for) a complete dedicated server when in reality you may only need a fraction of its resources? On-demand resource scaling and flexible pricing ensure that servers aren’t sitting idle, sucking up all that dino-juice willy nilly. Customers can apply for PS hosting now from their web panel.

DreamHost PS represents a new earth-friendlier way of doing business and we’re proud to be the kind of company that’s able to offer it.

And remember, if you’re a DreamHost customer and want the world to know that your own website is green, just visit the Home > Green Hosting section of your DreamHost account control panel to get some green icons. After all, what good is being green if you can’t yammer on about it to anyone who will listen!

By the way, thanks for listening to me yammer on!

Okay, nothing silly this time, I promise…

Some of you may have noticed that we’ve been having what a problem that is, although maybe not the worst in DreamHost history, definitely in the top 5.

There has been a DreamHost Status post about it, but it’s been going on so long, there obviously needs to be more said.

The History

The events that conspired to cause this horrible performance for everybody in our “blingy” cluster actually started to take root 19 months ago.

That was when I made this post asking our customers for some suggestions on storage. I made the mistake in that post of mentioning the name of one particular storage vendor who apparently does a search for their name in rss feeds of all kinds of blogs. I won’t mention their name again here, to test if they REALLY read this blog, but they were the one on the list right after “Netapp”.

Anyway, immediately a sales guy from there was hounding me about how great their product was. It would have super-duper reliability, super-duper performance, and super-duper ease-of-management. It was super-duper expensive compared to our current solution (about 3x the price per GB), so in the end I declined.

But, over the next year he kept hounding me and hounding me, and eventually the price came down to something in line with our current costs, so we decided to try one unit for our new cluster, “Blingy”. After we were satisfied with our internal testing, Blingy went live with the new storage solution in December 2007.

Smooth Sailing

At first, everything was fine, performance was great, everybody was hunky and dory. But then, as usage started to go up, the new file system started acting up. Around the same time every night, the system would stop responding to NFS requests for a while, which would immediately break web and mail service for everybody in the entire cluster.. thousands of customers.

Our Bad

Now, it can be a big mistake to put live customers on any new system. But honestly, we’d tested it lots, researched it a ton, and we added people very slowly at first, and it performed great.

Our biggest mistake I believe had nothing to do with what specific vendor or hardware we went with.. it was simply putting so many eggs in one basket!

Even with our Netapps (which are pretty much awesome), there are problems from time-to-time. However, a typical hosting cluster will have a dozen or so Netapps, which means any problems are one twelfth as big.

With Blingy, EVERY customer is on this one “mega” filer, which in theory should make for better performance, reliability, and ease of management. And since we got the clustered solution (in an active-active configuration)… there really is no single point of hardware failure in this thing.

But, as it turns out, there are a lot of non-hardware failures in the world.

Their Bad

Well, the techs at the vendor couldn’t figure out what was causing the NFS freezing, and so they recommended us doing a major OS upgrade to hopefully fix it.

During this whole time, the fiber channel disks were slowly filling up, and we’d been trying to move large files off to the sata pool (it’s a two-tiered solution, and there’s a feature that automatically moves less-accessed data to lower tiers).. however the thing couldn’t move the data fast enough. It couldn’t finish doing a “move job” in a single day, and every day it’d sort of “crash”, which would screw up the move job, and nothing would get moved.

Also, as the disk kept getting more full, performance kept getting worse, creating a vicious cycle. We ordered some more fiber channel disk shelves at the end of February to grow the main FC volume, since we couldn’t get things off to SATA, and it was supposed to come on March 10th and be installed at the same time as the major OS upgrade.

However, the disks didn’t end up getting installed until March 25th, and at that point it turned out we could NOT grow the FC volume with these disks (well, it was technically possible, but their on-site techs recommended VERY VERY heavily against it.. it would severly impact performance), which was sort of the whole point. So now we had a new FC volume which we still had to migrate users to.

Your Bad

Of course, this whole time, new customers just kept signing up, and being added to Blingy. What were you guys thinking?

By this point we knew this was a bad idea, but we didn’t have a new cluster ready (we’d expected Blingy to grow for another couple of months), and we try to never ever grow old clusters again once they’ve been “shut off” from new signups (because in time they stablize and have very few problems).

However, the moving people off to the new FC vol, or the original SATA vol, or even the new Netapp we also added to Blingy, just wasn’t happening fast enough. So on April 2nd we bit the bullet and switched Blingy off as the “new customer” cluster and started growing good old “Postal” again. Once we did that, we were finally able to get ahead of the curve and total usage on our first fiber channel volume has been slowly dropping ever since.

We tried at that point to contact the vendor to see if we could just get more drives that WOULD allow us to grow fcvol1, but they said their manufacturers were closed for inventory for a week after the end of the quarter and we couldn’t get anything until Friday, April 11th at the absolute soonest. Later they said they could find us some they could get us by Tuesday, April 7th, and we preliminarily said we’d take them.

This whole time we had a support ticket open with the vendor about the crashes (the OS upgrade didn’t fix it), and finally on April 3rd we received notice that they’d fixed the bug that they believed was causing it! However, the patch still needed to go through their “QA”. Finally, this Sunday April 6th they said it was all ready to be deployed, so last night we did.

What Now

Well, right now, performance is still not great on fcvol1… but mail and web should be pretty much working. One thing we’ve noticed is a website that hasn’t been visited in a long time will have a big lag still upon the first visit.. but then subsequent reloads/visits seem much faster.

At least the total disk usage is coming down now, and hopefully by tomorrow it’ll be below 85% which is supposedly a magic number where performance is fine. We’re going to keep off-loading it until things are great, though. We’ve got plenty of disk space for it, the problem is just it takes so long to move it.

We also I guess will find out tonight if the NFS freezing bug is fixed by this new patch. Hopefully so.

It’s Too Late…

I realize this is probably too little too late for many of you, but I just wanted to sincerely apologize for this whole big Blingy cluster-f*ck. Also, if you’re on Blingy (you can tell from the panel by clicking “account status” and looking at “Your Email Server”, we’d like to offer you a month worth of hosting credit.

To get it, all you need to do is contact support from our panel and make the subject of your message “Blingy Account Credit”. That’s all you have to do, and we’ll credit everybody who asks (and is actually on Blingy!) next Monday (April 14th).

Well, it was a little over two months ago that we had what I think is pretty safe to call the worst disaster in DreamHost history.

In retrospect to me, it’s kind of funny that the worst disaster didn’t turn out to be due to a security breach, a power outage, a loss of data, or actually anything related to our actual hosting service. I guess it shouldn’t be a surprise that people care a lot more about their bank accounts than they do their websites.

I have realized that billing is the one issue where how important we feel it is is completely at odds with how important you guys feel it is.

What I’m trying to say is, we’ve always been ultra-flexible and lax about how people pay, when people pay, or even about giving credits, discounts, or refunds. We figure, whatever, pay us when you’re ready, we’re not sending anybody to collections or ruining anybody’s credit over some measly bandwidth bill.

What we’ve always tried to focus on more (even though it might not seem like it at times!) is our hosting system’s stability, performance, and features.

I guess I’ve always figured that any billing-related error can be easily undone (worst case scenario, it costs us a little money); there is no lasting harm done to the customer. Whereas having a website or email problem could potentially cause permanent damage to somebody’s business or personal life or something?

Well then, let’s go back and see just how little money a worst case scenario actually costs, shall we?

Credits and refunds to cover people’s bank fees: $52,000.

Sigh, if only everybody kept a big cushion of cash in their account! The main damage that can be caused by a billing snafu is for people who get their account overdrawn, and because of that aren’t able to make a critical purchase, or have a check bounce, causing hassles and incurring bank fees. We offered to pay people any amount their bank charged them for going negative, and in the end that total looks like it came to about $52,000.

Accidental refunds: $170,000.

The worst part of this whole process (for us) turned out to be just after the accidental billing, ironically when we were trying to make things right!

If you recall, our system was not actually charging about 75% of the time we thought it did.. and so we refunded thousands of people who were never charged (but, 75% of the refunds didn’t work either). Well, out of all that, and after two months, there are still about 600 accounts who were credited a total of $170,000 in excess of what we charged them that we haven’t been able to get back from them or their bank.

It is slightly annoying when the same guy who complains to the high heavens when he thought he’d been over-charged $9,000 by accident conveniently disappears when we realize that actually, he’s been over-refunded $9,000 by accident.

Extra credit card fees: $82,000.

Another slightly annoying thing is that credit card processors don’t credit you back any fees when you refund a transaction. Overall, the extra credit card processing we did resulted in extra fees of about $350,000! Fortunately, after a whole lot of groveling and explaining the situation (and waiting two months), we finally got all but $82,000 of that back from First Data, American Express, and Discover Card.

Extra support messages: 20,000.

As you may have surmised, people wrote to us about this thing. About 20,000 times… and it would have been tens of thousands more if we hadn’t put up an “emergency block” against new messages for a little while in there.

How much this extra support actually cost (in terms of your wased time, tech support overtime pay, and other questions taking longer to answer to) is hard to say, but normally we only get about 45,000 messages in a whole month!

Accounts canceled: 1000.

It’s also kind of hard to say how many people actually closed their account because of the incident, but in January we did have about 1,000 more accounts closed than average. Assuming each of those accounts would have stayed for maybe another year, that’s another $120,000 down the Intertubes. It’s crazy… from all our power problems back in 2006, we hardly lost any accounts at all.

Goodwill lost: Priceless.

Yeah, it turns out this whole blog post is nothing more than another clichéd MasterCard commercial parody.

P.S. I guess it’s nice to know, less than two hours away from our biggest data center move ever, that we’ll cause a tiny fraction of the disruption to our customers that one unexpected fat finger did!

P.P.S. Thanks RIM, for scheduling a blackberry outage exactly at the same time. It makes us look better. And, maybe some of our Happy Customers will blame their lack of email tonight on you!

It’s hot in LA!

Which makes me think of summer!

Which makes me think of interns!

Which makes me think of hot summer interns!

Which makes me think of how sad I am to not be in Advertising!

Now, I know there are lots of other cool-sounding summer marketing internships available around this Internet world of ours, but hey, they’re not going to be very cool when it’s 96 degrees with 98% humidity in Manhattan and they’re on a street corner handing out flyers!

Meanwhile, you could be nice and cool, deep underground in one of our (assuming there’s no power outages..) climate-controlled data centres! (That was for the Brits.)

Being Los Angelenos, we don’t really know when this “summer” you speak of IS, or even really how to spell it, and because of that we’ve already “jumped the shark” and are looking to hire FIVE (5) summer interns… RIGHT NOW IN THE MIDDLE OF MARCH!

I would explain how great it is to work at DreamHost, and all the details and what-ever-for-not here, except I already wrote it once on the jobs page there, and re-hashing old content is a job best left to the professionals!

I will at least mention here that #1 this is a PAID internship, at $18.50 / hour, and #2 we’re flexible with when you want to start and how long you want to do it for, and #3 sure, this could be a good way to perhaps eventually even get a real job!

Now, juuust to wrap things up, my friend sent me an email two days ago about his experience interviewing that was way funnier than everything I’ve ever written, so hopefully he’s okay with me posting a snippet here. I can only hope that any interviewing I do for this internship will score half as many douchebag points…

So then I went to this interview at this place where really the job description was 100% me, like every thing they needed, I have done, I had all the right skills, etc. The guy there was a bit wacky. First thing when we sat down he asked me who I WAS. Like to describe myself as a whole, what type of person I am, etc. Then he started telling me how he asked some receptionist applicant that, and she said “well I’m beautiful” and then he went on this story about how she WAS gorgeous but she went on to explain that she didn’t mean her outside appearance even though it was beautiful, but she meant her inner self, she’s a beautiful person, etc. We finally got on to the interview and during it he tells me to pretend I’m a shoe salesman and I just sold him a pair of shoes but he phones me and says he doesn’t know how to tie his shoes, can I explain it to him over the phone. I was like “haha oh umm ok, sure. Haha, I feel like this is going to be one of those trick questions!” and he assures me like no, no, it’s no trick, it’s cause communication is important in this job. So I go through it, put your left lace in your left hand, pull it over to the right side, yadda yadda. Then at the end he looks at me with this smug look and goes “That was good. But you forgot one thing. You forgot to tell me to put my foot in the shoe”. Extra douchebag points here because not only WAS this a trick question, but he said tell me how to TIE them. I just chuckled and was like “oh wow yeah, I didn’t, hah yeah”, but after I was thinking how awesome it would have been to just stare at him for like 2 seconds when he said that, say nothing, and then get up and walk out and drive away.

Agreed.

The other day I finally re-opened my favorite web browser and I noticed that some prankster had changed my home page to JoshJonesIsYourNewBicycle.com!

I was of course outraged, and since nobody spoke up, I had no choice but to have the entire admin staff TERMINATED!

Just a few days later, as the issues mounted, I re-started my browser again and, I assume due to the perspective lent by time, realized it was nothing more than a jest in good fun!

In fact, I was touched. And so, I immediately re-hired everybody (but at only 60% of their previous salaries so they knew not to pull that crap again)… only to find out that what I thought was such a kind and creative tribute to their illustrious leader was nothing more than a crude knock-off of BarackObamaIsYourNewBicycle.com!

Once again ENRAGED, I had no choice but to fire EVERYBODY. This time I got the entire support team, marketing, HR.. not even our gourmet chefs, fighter jet pilots, or doggie masseuses were safe. I even may have killed a few of the weaker employees.

I’ve finally got some solitude, and it got me to thinking about this whole Barack Obama thing.

To tell you the truth, I can’t get behind Obama. Let me tell you why.

Why?

You see that picture at the top? That’s Senator Obama’s actual name plaque from the actual Senate. Back in 2005 my dad had to testify before some senators about something or other, and I went along for fun. And before you ask, yes, that is how I roll.

One of the four Senators in the room was Mr. Obama, who was already semi-famous after his speech at the DNC in 2004. And, to be truthful, he actually seemed the most intelligent and educated on the issues of the four… but I still can’t get behind him.

Why Not?

Well, after the hearing was over, I took that picture with his name plate, and was THIS (yes, THIS) close to stealing it! Wouldn’t that have been 100% SWEET to put on my desk? But at the last moment, I wussed out.

I knew that thing was already worth a couple of bucks on eBay, or at least would make a cool momento to pass down to my grandchildren or bury on a deserted island… but man, if he becomes PRESIDENT? The first BLACK PRESIDENT?

It would just kill me.

So, that is why I hope he doesn’t win.

But the truth is, as I alluded to in the last newsletter, I can’t understand anybody voting for any politician they don’t personally know.

It is so hard to judge the true character of anybody, even people you’ve been friends or colleagues with for years… HOW could anybody feel comfortable voting for any politician whose entire career is based on projecting an “electable” persona? And that is any politician.

That is why I truly do like propositions. Unlike people, a proposition can’t back-pedal, change its mind, break campaign promises, cheat on its wife, pander to special interests or give in to the freaking UN. A proposition is simply a self-contained law, and before you cast your vote on it, you can do all the real research on it you want!

After a new proposition has been approved, nobody’s allowed to say “I know the law says 5,500 new slots, but now we need an extra 10,000 to stay the course.” (Not without putting it up to another direct public vote at least.)

Not to mention, even if your vote on a proposition ends up “losing”, at least you know that greater than 50% of your fellow voters “won.” Whereas a politician might often do things that are in nobody’s interest but their own.

Proposition Paradise!

So, what if every government decision were, like California’s propositions, put up to a direct public vote? Back in 1789, this would have been technically impossible, and a representitive democracy was the only feasible solution.

But the same is not the case in 2008. We’ve got the Internets now!

What if we kept everything the same about the current government, except that instead of the congressmen doing the final voting on laws, it was always put to the public directly in the form of propositions? Sort of a Government 2.0.

Or, as wikipedia puts it, a direct democracy!

Reading that article, the arguments against direct democracy are pretty weak. Let me debunk them now:

* Scale: The Internet make it easy.
* Practicality and efficiency: Again, thanks Internet! Plus, there should still be just one election per year. Everything would be voted on at once, making it more efficient and less of a burden on the voting populace. The fact that laws can’t be changed more than once a year is just gravy!
* Demagoguery: Please… if we can’t trust our public to vote intelligently, perhaps we’d all be better off in the Irans?
* Complexity: Haven’t you read The Wisdom of Crowds? The masses in aggregate are not fools and they do understand the issues when they affect them.
* Voter apathy: Again, people are apathetic only when the issues do not affect them; and if they feel the issues do not affect them, why does it matter whether they vote? If they did, they’d just vote randomly and cancel each other out anyways.
* Self-interest: I believe a thing it’s been proved time and again that the best way to make rational decisions is by acting in your own self-interest. I mean, that’s why we’re voting, right? To see if a proposed law would be in the majority’s best self-interest. Nobody should ever, ever, ever vote for something that would hurt them just because they think it will help society “as a whole”! Don’t worry, if it’s going to help more people that it hurts, it will win regardless of your measly vote.
* Suboptimality: Well gee, it’s also “sub-optimal” to have a competitive marketplace. It’s “sub-optimal” to have random mutations. It’s “sub-optimal” to buy an index mutual fund. There will be a lot of fumbling in the dark to be sure, but like natural selection, the “optimal” laws will bubble to the top eventually, and the “sub-optimal” ones will be voted the way of the dodo. And with a direct democracy, that will only happen faster.
* Manipulation by timing and framing: Again, all the voting would just be done on the first Tuesday in November each year!

Our founding fathers were also against the idea mostly due to the “Tyranny of the majority”… but as long as we still have the bill of rights and the Judicial branch everybody’s personal freedoms would stay intact.

Especially Interesting

Another bonus of a true direct democracy would be the end of special interests. Special interests exist whenever there is something that benefits a few people a lot while hurting everybody else a tiny bit. Those who stand to gain fight nooth and tail to keep the advantage, whereas there’s no single individual who feels enough pain to bother standing up to them. It’s thanks to this that industry subsidies, trade barriers, and real estate agents exist.

If every law was put to a vote of every citizen, say goodbye to subsidies, tariffs, and monopolies! I know I don’t give a damn if corn farmers have to compete harder as long as it means cheaper food for me. Now, I’m not going to go writing my congressman about it, but if it ever came up on a ballot that’d be a big fat NO vote.

And you’d vote against the law that’d show up every year requiring the state provide each DreamHost CEO a new SUPERCAR every month (so they could get to the data center quicker in the case of a power outage). Unfortunately, I think it’d be tough for me to get greater than 50% of you behind such a measure, no matter how much I plaster this blog with posts extolling the many virtues of such a proposed legislation.

Could It Work?

Yes. And how do I know? Because it has. And I’m not just talking about the DreamHost suggestions system either.

Switzerland has had the most direct democracy on the planet for over 160 years, and it’s the most competitive economy in the world, has a 3.1% unemployment rate, and hasn’t been in a war since 1815! Things are different when the people deciding whether to fight are the same people who would be fighting.

To Barack

So, Mr. Obama, if you really want to “bring real change to Washington,” why not (if/once elected) put every decision you ever come across up on whitehouse.gov, along with what you see the pros and cons to be. Send login info out to every single US citizen (you can include them in those mailed social security updates), and allow We, The People to at least be your guide. You know, radical transparency and all that.

Think of the clout you’d have with congress if every one of your decisions, proposals, and policies was backed by a direct “mandate from the people”! Not to mention, it seems like you should never have less than a 50% popular approval rating!

People say you don’t have much experience. So why not ask for advice? It’s worked out decent for us.

Remember Barack, you are not running for King of America. You are simply interviewing for a job. And that job is to provide guidance and support to us taxpayers, your direct supervisor.

Of course in the end, who knows if all this would make for a better presidency.

But at least it wouldn’t be any worse.

This week, I learned another five things I did not know before:

Monday: Although charging a credit card is instantaneous, refunding really does take 3-4-5-6-or-more business days to process.

Tuesday: You can erroneously credit an expired credit card. The money does leave your merchant account.

Wednesday: You can credit a canceled credit card. The money does leave your merchant account.

Thursday: You can credit a debit card tied to a checking account that has been closed for months. The money does leave your merchant account.

Friday: If you charge somebody with an international credit card and then refund their money, by the time the money gets back on, the dollar will have weakened!

Lucky you to learn these things the fun fun-facts way!

Hello.. how’s your morning going?

I hope it’s been a little better than mine.

We had a teensy eensy weensy little billing error last night… my first clue something was up when I saw this morning’s daily billing report (so far): $7,500,000.

It turns out due to my excessively fat fingers, nearly every one of our customers has been seriously over-billed in the last 12 hours.

I bet when you read this part of the last newsletter:

4. New Office!

Another important thing I’ve been doing instead of writing newsletters
is looking out the window of our NEW OFFICE:

http://blog.dreamhost.com/2007/12/21/were-so-high-right-now-you-dont-even-know

If your next web hosting bill from us is mysteriously tripled, now you
know why.

.. you thought it was a joke!

Ha, the joke is on you! I guess. Um, okay, no, not really, I’m sorry.

How on earth could something like this happen?

Let Me Explain

A couple of weeks ago, just around new years, we started beefing up some of our internal “controller” servers. These are the machines that run all of our “behind-the-scenes” services; things from adding a user to registering a domain to configuring apaches to rebilling customers.

I was on a little-bit-too-long vacation, but when I got back, I noticed our daily credit card payments seemed a tad low in the new year.

So, late last week I tried re-running the billing services for all the days back three weeks or so. I knew this was safe, because after 10 years, the one thing you DO get perfect is your billing system. Our biller is pretty bug-free and robust at this point, because we’d be broke and eating bugs if it weren’t.

In fact, it’s so robust you can just run it on any day you want, and it’s safe. It won’t double-charge people and it’ll even automatically find any missing charges and catch everything up to the day you said.

Anyway, I ran it, and things were fine.. and sure enough, it caught a lot of missed payments. I didn’t have time to look into it right then, but I made a note to myself to check up on it on Monday (yesterday) and see if things were fine or still messed up.

Come Monday

Monday came. I checked the reports and sure enough, things were still pretty low. So I looked at the logs for some of the biller services, and I noticed they were only failing on the machines that had been recently upgraded!

That explained why we were getting some money still (since not all the controllers have been upgraded yet), but not all of it.

Anyway, it turned out there was no 64 bit version of the PFProAPI module we use to interface to the credit card transaction server. No big deal, there’s a new module that interfaces with their new and preferred https interface, and it was only a couple of lines of code to change to get us switched over!

So anyway, I made the change, and it worked, and I even tested it, and things were fine!

But then… late last night, I realized: when I re-ran those biller services last week, they must not have fixed everybody then either! It’s just that by running it again I randomly got different people being charged on the working controllers who had been assigned an upgraded (and therefore broken) one before.

So why not just run it all one more time?

Sure, it should be no problem! So I did, manually running the biller (which is normally automatically scheduled) for 2008-01-14, 2008-01-13, 2008-01-12, 2008-01-11, 2008-01-10, 2008-01-09, 2008-01-08, 2008-01-07, 2008-01-06, 2008-01-05, 2008-01-04, 2008-01-03, 2008-01-02, and 2008-01-01.

I probably should have just stopped there. But then I thought better. I thought to myself, “When did we start upgrading these controllers anyway?”

I couldn’t remember. But, since the biller is super-safe and robust anyway, I went ahead and ran it for 2008-12-31, 2008-12-30, 2008-12-29, 2008-12-28, 2008-12-27, 2008-12-26, and 2008-12-25, just for the hell of it.

Notice Anything?

Don’t feel bad if you didn’t. I kind of missed it myself.

THOSE SHOULD HAVE BEEN 2007!!

Heh, uh.. um, er.. my bad?

So what happened?

Well, that super-robust and stable biller did what it was programmed to do, it ran as though today was December 31st, 2008!

And what did it see? Well, it saw a whole lot of accounts (essentially all of them) who for some unknown, mysterious reason hadn’t been charged at all for eleven and a half months!

So off it went, busily through the night, “fixing” everything up for “today”, December 31st, 2008.

Really, it’s sort of amazing this never happened before in the last ten years.

There IS a bug here.

I can imagine the half second or so of thought that sprinted through the programmer’s mind when he was adding the ability to allow you to pass in what day to run the biller as though today is:

Hmm.. well, I could see us POSSIBLY wanting to be able to bill for a future date.

Well guess what… NO! We will NEVER want to rebill as though today were a day that hasn’t happened yet! But instead, somebody along the line (Sage? Me? Somebody else?) figured, “What’s the harm in keeping it flexible?”

About $7,500,000 in harm, that’s what!

The serious part.

The end to this story is that of course, I’m very very sorry, we’re very very sorry, and I’m sure you’re very very sorry this happened. I really am. I understand the sort of problems that an unexpected large charge to your credit card (or worse yet, your debit card) can cause. If the tone of this blog post seemed a little light, I apologize I don’t mean to offend and I realize how serious an issue this is. I’ve been up since 3:50am trying to undo the damage and maybe I’m a little shell-shocked.

A new service is running right now (in parallel on all the controllers) that fixes all those future charges, re-enables your account if it was erroneously suspended, and if your credit card was automatically rebilled, refunds the payment automatically. You don’t have to contact us or your bank, and you’ll get an email when your account is finished fixing up. It’s going to take several more hours to complete. There are (or were, after this incident) a lot of you these days!

If, because of this billing mistake, you somehow incurred some fees from your bank or credit card company, please let us know after tomorrow (today we are just replying to all 10,000+ billing messages with a generic explanation) and we’ll do our best to make it right for you.

And of course, the biller no longer allows dates in the future.

The moral of this story is that “flexibility” is rarely desired in programming! The less a program will accept/the less a program will do/the less options and preferences it has, the more usable it is/the more understandable it is/the more stable it is.

Tough Love

When designing a program, you’ve got to make some tough decisions .. and when you really can’t decide if this is something your users will need someday, err on the side of leaving it out.

Otherwise, your users will someday err on the side of your face.

I recently wrote a self-described ‘rant’ describing some of the experiences DreamHost has had working with Ruby on Rails on our platform, and with some recommendations on how the Rails community might be able to improve the situation. That post has received some excellent comments, including some by DHH, the creator of Ruby of Rails, and I’d now like to follow-up with some clarifications and further comments of my own.

My original post was intended primarily as commentary for the Rails developer community as a whole and they are, of course, free to take it as simply that. 37 Signals and DHH obviously have their own agendas, business and personal, and those agendas are largely not in line with any agendas DreamHost is working to further. I hope my comments may be taken as ‘food for thought’ by the larger Rails developer community. The project will survive just fine with or without taking my advice, and the zealous user community will likely remain zealous.

DIY

Some of the response to my post was essentially ‘do it yourself’. DHH also went so far as to recommend we not treat the Rails community as a ‘paid vendor’ and to ‘wipe the wah-wah tears’ from my chin. While that is a very valid request, I don’t believe it applies in this situation.

Ruby on Rails is a pretty small part of our overall service. We have already put in a not insignificant amount of work to support Ruby on Rails, including a pretty good amount of user training to assist people new to Rails, and have in general worked to further the cause for it in the shared hosting environment. Aside from hardware, we don’t really have very many paid vendors of any kind now. We have ourselves developed all of the custom perl-based software our business relies on (warts and all). We very much believe in the DIY mentality and it’s been a major part of our business history. Additionally, our business will not be significantly affected one way or the other by the future actions of the Rails community. It simply does not have the critical mass necessary for that.

That said, I would like to see Ruby on Rails reach toward that critical mass of users, and thereby become a larger portion of our business. For that to become a reality I believe it needs to be simpler to use on the server side of things. A larger user community is a good thing for any open-source project.

Bride of PHP?

I mentioned PHP several times in original post, but I very much do not want Rails to become ‘another PHP’. That would be silly, as one PHP is plenty! Ruby on Rails seems to have a conflicted identity at the moment. It is simultaneously compared to enterprise technologies like Java Servlets as well as more average joe programmer technologies like PHP. There are more traditional web programmers experimenting with Ruby on Rails and exploring it, and there are professional enterprise programmers investigating it. The amount of attention Ruby on Rails has achieved in such a short time is awe-inspiring.

Programmers are Lazy

The fact remains that both of those groups of web application developers, and I’d go so far as to say all web applications developers, want development environments that more or less ‘just work’. They want to focus on programming and leave the server administration alone. Rails does a great job at saving programmers on programming time (which is exactly why programmers like it), but many reports I’ve heard indicate that in many cases it trades that programming time for back-end server administration time instead. The average joe programmers out there are typically using completely managed hosting environments that they do not have much control over, and the enterprise programmers are typically using a somewhat complex and large-scale established environment that they also do not have much personal control over. Even those VPS users who do have more control over their server environment do not want to be spending their time managing that VPS. They want to be spending time writing those great applications.

If Ruby on Rails were made to be simpler to use with a wider range of hosting environments, big and small, that would ultimately benefit Rails itself far more than it would benefit DreamHost or me personally.

I would be happy to provide a free hosting account to anyone who would like to help work on these issues with us. Just contact me through our contact form and we can talk about it.

UPDATE! : DreamHost makes using Mongrel much easier! This is one way we’re working to make Ruby on Rails easier to use.