One-Click Mystery

I was recently digging into our database collecting some information on PHP usage by looking at how many people were using our One-Click Web App Installs.  Nearly all of our One-Clicks are PHP so it was a quick way to estimate total PHP usage by our customers.  While looking at the data I noticed something strange, though.  Our users are using our One-Clicks less and less!  ”Weird”, I thought.

I collected this data by looking at every currently active account of ours and counting up the ones that have installed at least one One-Click (not the total number of One-Clicks in use!).  I also tracked the year each account was first made active in our system.  It doesn’t count people using PHP apps they installed themselves, and it does count people who may have set up a One-Click and then abandoned it. This is not a scientific analysis by any means, but any clear trends should still be pretty reliable.

The clear trend we see is a mostly upward trend with a spike in usage by accounts signed up in 2006, and then a pretty quick drop-off over the last couple of years.  I was expecting to see a continuous upward trend over the entire time and was surprised to see that drop-off there.  The graph shows percentages so the number of actual accounts isn’t a major factor.

I have a few theories about the slowing of One-Click Install Usage…

• Our users signing up over the last two years are more tech-savvy than those in earlier years and more of them are choosing software other than what we provide and installing it themselves.
• More people are going back to the days of static HTML.  (Seems unlikely.)
• Increased interest and awareness in more ‘hot’ tech like Ruby on Rails has resulted in users moving away from PHP itself, and our One-Click Installs along with it.
• More people are finding their Web App needs met by ‘Web 2.0′ websites like Flickr and Facebook.  (But then, what are they doing with their websites instead?)

The number of new users coming on board has not slowed at all so people are definitely still finding a need for tried and true, full power web hosting.

So… Tell us why you do or do not use DreamHost One-Click Installs for your website, and WHY!

Gutenberg…

Whenever people ask me what I like most about my job at DreamHost, I invariably think of Johannes Gutenberg.

Gutenberg was a German goldsmith who pioneered mechanical printing in the 1400’s. His inventiveness took the power of the pen – for good or for bad – and amplified it, fostering a spread of knowledge that, ultimately, resulted in a burgeoning Renaissance.

The impact this had on science, business, technology and the overall democratization of knowledge that come with it cannot be overstated, even though Gutenberg himself would benefit little from it personally in his lifetime.

This is, of course, not to compare DreamHost itself to Mr. Gutenberg – we’re just a web hosting company with a penchant for free pizza and off-the-wall jokes that people sporadically find amusing.

And, when you get down to it, the reality of the situation is that most of our customers opt to use the power of the ‘printing press’ that we offer to publish pictures of their cats, indulge in navel gazing or publish rants embued with varying degrees of coherence. Among, well, other things.

The beauty of this, though, is that each and every one of these things is important to somebody, somewhere, for some reason. Even if it’s just the author and a select handful of readers.

Taken at their face, this may not seem like much – but I believe it is actually pretty important.

• While most people couldn’t care less about someone else’s cat, forever will there exist a memory of a beloved pet that can be looked at years later. Self-publishing has caused people to treasure the little things that enrich their lives – and, from time to time, enrich others as well (the rise of digital photography owes a lot to the fact that people can now so easily share their creations with others, for instance).
• While the web’s penchant for facilitating so-called navel gazing garners criticism at times, I believe that this, too, serves a purpose. The more we explore our own thoughts and ideas and allow others to peek into our world, the better our relationships with those people will be. I, for one, am happy to see the web evolve into a tool to connect people. It has become a humanizing technology.
• Those rants that annoy so many also serve as an outlet of sorts. For many, writing is in itself a way to explore ideas. It links people with similar ideas and – yet – exposes people to a vast marketplace of ideas. If nothing else, we could always use more ideas.
• As for those “other things”, well, I’ll leave those to someone else. We’ve hosted some pretty weird stuff over the years.

Picture of my cats – because I can…

While we at DreamHost are ultimately a commercial entity and benefit in kind, I like to think that we’ve played a small but important role in facilitating the free-flowing transfer of ideas and creative expression across the Internet. Good ideas, bad ideas, those that are well thought out and those that could use a little refinement.

I’m quite proud to have been even just a small part of all this.

It’s for that reason, really, that I’ve had mixed feelings in the weeks since deciding to leave the company for other pursuits. When you’re proud of what you do and you’ve been doing it for nearly a decade, that’s a tough decision to make.

And now, finally, I’ve reached the last few hours of my employment here. While I’ve got some personal projects in the works that I’m excited about (shameless plug: one of which may be of particular interest to Spore players, once it’s ready: PanSporea) – and I’m certainly looking forward to sleeping in – it’s going to take me a while to get used to not being a part of it all.

While I’ve already thanked most of the great people whom I’ve had the pleasure to work with, I would like to again express my gratitude toward the DreamHost Honchos (Dallas, Sage, Josh and Michael) in particular for their support and for building this great company in the first place.

I also want to thank my fellow co-workers, both past and present, who made the experience so much fun. I’ll miss you all.

Finally, I want to thank our customers for filling our servers up with all sorts of interesting and peculiar things and generally making the web as a whole a more entertaining and useful place. I still get a huge kick any time I run across a really cool site only to scroll down and find a link to DreamHost (something that has become increasingly common in recent years). We’d be nothing without you.

Nor, for that matter, the inspiration of a certain Mr. Gutenberg.

PS: For those of you in or around southern California, you can see an original Gutenberg Bible at the Huntington Library in San Marino. It’s definitely worth the trip (the botanical gardens are also pretty impressive).

See Dallas Talk

I have a couple of talks coming up at events that you might be interested in. The first one is at WorldBlu Live , coming up in New York City next week, Thursday and Friday October 16-17, 2008. In their words: WorldBlu LIVE is the only conference in the world that connects you to directly to the top leaders of organizations on the WorldBlu List of Most Democratic Workplaces™ as well as other thought-leaders and mavericks who will share their unique stories, best practices, and leadership strategies for building a world-class democratic workplace.

I’ll be speaking about how democracy is used within DreamHost and how it makes our business stronger. A bit of DreamHost history will be thrown in to provide some basis. I’m looking forward to this conference and I think there will be a lot of really cool talks from really interesting people. Some of the speakers that stood out to me are Tim Westergren, the founder of Pandora, and Philip Rosedale, the CEO of Linden Lab (the Second Life people). Those are two companies doing things on the Internet that will have a long-lasting impact, in one way or another.

The other talk is in Atlanta on November 12-14, 2008 and is at php|works, a PHP conference with a sister Python conference, PyWorks, running simultaneously. I’ll be talking about how to optimize web applications for the users of shared and VPS hosting services like DreamHost. It’s an hour long talk so help me out by showing up with LOTS of questions. There’s a lot of other good talks too, and I’ll be there the whole time.

If you’re in NYC or Atlanta around those dates come say hi!

Hot off the presses

Earlier this year DreamHost was named as one of only twenty five companies worldwide to be recognized as having a democratic workplace by the WorldBlu Organization.

What’s that mean? It means that we listen to our employees. It means we’re transparent in how we do things and why we do them. It means that, coincidentally, we’re also a great place to work!

Axiom News of Canada recently recognized our commitment to the democratic workplace in a three-part series.

E-meetings Engage all DreamHost Employees in Corporate Decision-Making

We really don’t have too many meetings around here. We use email lists. A lot. We don’t restrict employees from joining any list – in fact, we encourage it. If someone in tech support wants to listen in on our developers or system administrators, they’re welcome to do so! If anyone has a suggestion for system improvements, or a general comment about our operations, it shows up on an email list first.

DreamHost’s Unique Profit-Sharing System Results in High Morale, Strong Retention

We’ve literally been giving our company away since the very beginning. Every DreamHost employee is issued units in the company. (Units are kind of like the equivalent of ’stock options’ for an LLC.) At most companies those options are sold to employees at a discount – but not at DreamHost. We just go ahead and GIVE them to our employees at no cost. Once you’ve been working here for a reasonable amount of time you’re actually made an owner of the company!

Flat Management Structure Promotes Productivity, Retention at DreamHost

There are no cubicles at DreamHost. Our open-door policy springs from our roots as a dot-com startup, when teams were small and titles were meaningless. Our teams have grown since then and we’re still not too big on titles, but one thing we haven’t done is closed the door on inter-company communication.

We’re honored to have been recognized by WorldBlu and Axiom News. It’s a good feeling to see people interested in promoting what we’ve considered to be basic business practices from day one.

DreamHost really is a great place to work, and we’re always on the lookout for new talent to add to our team. If any of what you’ve just read appeals to you, be sure to check out our open positions and send in your resume today!

Mobile Lan

Guess who got a 3G iPhone… two weeks early?

Try not to be too jealous of me, but… me!

Hooo yeah, it is SO sweet, being able to download web pages, like 90 times faster. Not to mention, I can access the iTunes Store anywhere now, so I can rock out to 30 second clips of whatever song I want, anytime I want, for FREE. Plus, YouTube works better now, with higher video quality and faster streaming.. not to mention downloading and updating apps is lickety-split!

To top it off.. my 3G iPhone is already UNLOCKED and on T-Mobile, AND I share the 3G network over wi-fi (802.11g) to my laptop, my Nintendo DS, and everybody else riding my bus!

Now, how did I finagle all this awesomeness? You might say, being the greatest corporate blogger IN THE WORLD is finally starting to pay dividends!

Or, you might say… I just got one of these:

The PHS300 from Cradlepoint

So, the truth is, I’ve still got my old iPhone.

Ain’t nobody beating down my door with free new gadgets to test and review. But, what this “Personal Hot Spot” does is take any one of the USB high-speed data modems on the market right now in the US from Verizon, Sprint and AT+T and turns it into a portable, rechargeable battery-powered, wi-fi hotspot!

I had heard about this thing before but I’d sort of forgotten about it. And with the new iPhone news, I was thinking, “Man, if only the iPhone could just share its 3G network over wi-fi, turning itself into a little portable hotspot, that would make it perfect.”

And that’s what made my brain remember about it; it was right about the time I was testing all the various USB modems and starting to accept that I was going to have a HUGE dongle sticking out of my laptop all the time. The PHS300 saved the day with its 100% pure awesomeness..

Pure Awesomeness

Oh, pure awesomeness, let me count thy ways:

1. You can plug any of the various USB modems out there and they immediately just work.
2. Therefore, you don’t have to install any of the relatively crappy software drivers on your laptop.
3. Therefore, my VPN now works fine on all of them.
4. Therefore, macs/pcs/linux machines all work fine… and with zero configuration.
5. Now, dozens of machines/phones/gaming systems can all share your one USB modem.
6. The web-based management it provides works great and has every feature you’d expect from any 802.11b/g wireless router.
7. From completely off to connected to the 3G network on wi-fi only takes about 30 seconds.
8. The battery lasts a bit over 2 hours, enough time for my daily bus rides.
9. The PHS300 also works while charging from the wall.
10. It only costs $145.

More Testing

Being a huge nerd, I decided to do some performance testing of my new 3G wi-fi setup. In this case, since I’d already returned the Sprint Compass 597 USB modem, I tested another Verizon modem (the USB720 this time) vs. the AT+T USBConnect 881 modem, both directly connected to my laptop, and through the PHS300.

(A few notes:

1. My apologies to Verizon. It looks like the slowness from my last test was more due to the PCMCIAM interface to my laptop, rather than the speed of their EVDO Rev. A network! Using this USB720 modem brought it up to the speeds I’d seen from Sprint’s modem.

2. I pinged craigslist.org this time instead of google.com. For some reason craigslist’s ping times were much better across all the various networks.)

Now, for the two things I found:

1. The PHS300 added about 10-30ms to average ping times it seemed. No big deal.

2. For some reason the AT+T down/upload speeds were essentially HALVED by the PHS300. I thought that I must have been hitting some (surprisingly low) threshold for throughput of the device, but then the Verizon modem performed the same whether it was through the wi-fi or not! Weird.

So…

That “halving” effect is a technical mystery to me… and it’s making it hard for me to decide if I should keep the AT+T modem or just stick with the (USB) Verizon one. HMM! The Verizon USB720 does have this annoying super-bright flashing green light on it, but that doesn’t matter if it just stays hidden in my bag all the time I guess.

Anyway, in case I haven’t made it clear, I really like this PHS300 thing. It’s 100% pure awesomeness. In fact, it’s so awesome, it’s small enough to fit in my pocket, so I can walk around and be a human hotspot, a full five years before the rest of America!

I also like having a hotspot in my car (again, a full five years before it comes standard in all new KIAs).

Especially since you can’t talk on the phone while driving in California anymore, I’m definitely going to have to be doing a lot more emailing, instant messaging, and pushing out while driving!

Good Reminiscing Friday

Well, it was a little over two months ago that we had what I think is pretty safe to call the worst disaster in DreamHost history.

In retrospect to me, it’s kind of funny that the worst disaster didn’t turn out to be due to a security breach, a power outage, a loss of data, or actually anything related to our actual hosting service. I guess it shouldn’t be a surprise that people care a lot more about their bank accounts than they do their websites.

I have realized that billing is the one issue where how important we feel it is is completely at odds with how important you guys feel it is.

What I’m trying to say is, we’ve always been ultra-flexible and lax about how people pay, when people pay, or even about giving credits, discounts, or refunds. We figure, whatever, pay us when you’re ready, we’re not sending anybody to collections or ruining anybody’s credit over some measly bandwidth bill.

What we’ve always tried to focus on more (even though it might not seem like it at times!) is our hosting system’s stability, performance, and features.

I guess I’ve always figured that any billing-related error can be easily undone (worst case scenario, it costs us a little money); there is no lasting harm done to the customer. Whereas having a website or email problem could potentially cause permanent damage to somebody’s business or personal life or something?

Well then, let’s go back and see just how little money a worst case scenario actually costs, shall we?

Credits and refunds to cover people’s bank fees: $52,000.

Sigh, if only everybody kept a big cushion of cash in their account! The main damage that can be caused by a billing snafu is for people who get their account overdrawn, and because of that aren’t able to make a critical purchase, or have a check bounce, causing hassles and incurring bank fees. We offered to pay people any amount their bank charged them for going negative, and in the end that total looks like it came to about $52,000.

Accidental refunds: $170,000.

The worst part of this whole process (for us) turned out to be just after the accidental billing, ironically when we were trying to make things right!

If you recall, our system was not actually charging about 75% of the time we thought it did.. and so we refunded thousands of people who were never charged (but, 75% of the refunds didn’t work either). Well, out of all that, and after two months, there are still about 600 accounts who were credited a total of $170,000 in excess of what we charged them that we haven’t been able to get back from them or their bank.

It is slightly annoying when the same guy who complains to the high heavens when he thought he’d been over-charged $9,000 by accident conveniently disappears when we realize that actually, he’s been over-refunded $9,000 by accident.

Extra credit card fees: $82,000.

Another slightly annoying thing is that credit card processors don’t credit you back any fees when you refund a transaction. Overall, the extra credit card processing we did resulted in extra fees of about $350,000! Fortunately, after a whole lot of groveling and explaining the situation (and waiting two months), we finally got all but $82,000 of that back from First Data, American Express, and Discover Card.

Extra support messages: 20,000.

As you may have surmised, people wrote to us about this thing. About 20,000 times… and it would have been tens of thousands more if we hadn’t put up an “emergency block” against new messages for a little while in there.

How much this extra support actually cost (in terms of your wased time, tech support overtime pay, and other questions taking longer to answer to) is hard to say, but normally we only get about 45,000 messages in a whole month!

Accounts canceled: 1000.

It’s also kind of hard to say how many people actually closed their account because of the incident, but in January we did have about 1,000 more accounts closed than average. Assuming each of those accounts would have stayed for maybe another year, that’s another $120,000 down the Intertubes. It’s crazy… from all our power problems back in 2006, we hardly lost any accounts at all.

Goodwill lost: Priceless.

Yeah, it turns out this whole blog post is nothing more than another clichéd MasterCard commercial parody.

P.S. I guess it’s nice to know, less than two hours away from our biggest data center move ever, that we’ll cause a tiny fraction of the disruption to our customers that one unexpected fat finger did!

P.P.S. Thanks RIM, for scheduling a blackberry outage exactly at the same time. It makes us look better. And, maybe some of our Happy Customers will blame their lack of email tonight on you!

Stock Tips

First Republic Bank just opened a new branch a block from my house. I was kinda bummed, because I was hoping the lot would become something cool like a Starbucks or a McDonald’s.

Anyway, First Republic has the ad you see above, and some other eerily similar ones, in the window. All the ads use such banal stock photography they never made much of an impression on me, despite passing them every single day.

They never made much of an impression, until yesterday that is! Which is when I passed the Bank of America less than two blocks from them and saw they had just put a new ad that was very eerily similar in their window:

That’s the same lady! Right? Am I right? Yeah, it is! Definitely. Right?

Ha, anyway, I thought that was kinda funny. I mean, geez, B of A, couldn’t you at least check the bank closest to you before picking from the stock gallery?!

The Web Hosting Angle

Now, I shouldn’t be one to bash stock photography… the whole concept is very much alive and well in the entire Web Hosting industry. Still I’ve never come across two hosts with exactly the same “employee” on their front page!

I know sub-prime lending is a mess right now, but come on, bankers!

Speaking of Web Hosting stock photography, I’ve decided to end this post with a little collection of some of my favorite Web Hosting stock photo hotties, each one linked to some actual people employed at the various companies…

MidPhase

(What, nobody wanted to cough up for the “in-focus” version?)

BlueHost/HostMonster

(Isn’t that the same laptop they have on all the desks at IKEA?)

FastHosts

(AIEE! How does she hold that paper without any finger tips?!)

Verio

(That’s some shiny floor they’ve got at Verio!)

1 and 1

(Hummuna hummuna hummuna…!)

DreamHost

(We try our best to make sure nobody ever uses the same stock art as us!)

A Strike on Strikes!

Ha, did you think you’d gotten through those stupid strike-themed posts?

Well, apparently you are!

The writer’s strike seems to be finally coming to an end, and I don’t know whether to be happy or sad. On the one hand, it means I can finally get back to writing awesome blog posts. On the other hand, it means I can no longer get away with writing these blog posts… which is bad news because this well of creativity is tapped, my friends.

The sad truth is, I did that entire “billing mistake” thing just so I’d have easy blog fodder for another week.

A Last Hurrah

I guess I’ll just quickly wrap up a bunch of stupid things I was planning on “striking” against but never got around to. I never expected this thing to end and was pacing myself.

Hopefully this strike really is settled or you ain’t going to be seeing any new posts here until at least the first Sunday after the Ecclesiastical Full Moon date after March 20th!

A Strike on Fax Machines!

How in the hell is it 2008 and everybody still uses fax machines?

Give me some widespread e-signature standard already, world!

A Strike on Social Networks

Is it just me, or do social networks only appeal to people who 1. are single 2. have no job or 3. care about what their friends are doing?

Because I, for one, am none of those things.

A Strike on Cell Phones

Why do cell phones still keep any data locally?

When you get a new cell phone, you should just have to log into it, like you do, say, a new email client, and whammo, all your contacts/pictures/text messages/themes/preferences/ETC.. are syncronized with a (non-proprietary) server.

We need IMAP for Phones.

(I lose my cell phone once a month.)

A Strike on Global Warming

Dallas warned me not to post this, but he’s in Thailand (trying to enjoy it while it’s still above the ocean.)

There’s just three things that bother me about global warming.

1. There’s literally no way we can be even reasonably sure about what will happen. There’s just no experiment we can run on our entire planet that we can set up an adequate control for!

2. Even if the earth does get warmer, we can’t really know (again, what would the control be?) all the effects that will have on us until it actually happens. The earth’s climate has changed a lot over the billions of years it’s been around, and yet here we are, over 6 billion strong and fatter than ever!

3. Even if the Earth does warm, and even if it is bad for us, there’s again no way we can possibly verify what actually caused it, nor if there was anything we could have done to prevent it.

I mean, I’m all for clean air and water and not wasting electricity and saving the whales, but isn’t just having clean air and water and more money and whales to ride reason enough?!

And if we want to focus on literally saving the human race as we know it, maybe we should be spending more R+D on stopping near Earth objects!

We know they’re out there, we know they’ve hit Earth before, and we know it’s very bad when they do!

A Strike on Getting Old

I broke my left foot playing basketball when I was 27.

It took about a year to heal, but it’s pretty much been fine since.

Now all of the sudden, 3 years later, everytime I get up after being inactive for a half hour or more my left foot kills!

And that’s the real reason why I don’t worry about global warming in the future … I refuse to get older.

Now, please feel free to hold your own stikes in the comments, before the writers settle!

Five Fun Facts For Friday

This week, I learned another five things I did not know before:

Monday: Although charging a credit card is instantaneous, refunding really does take 3-4-5-6-or-more business days to process.

Tuesday: You can erroneously credit an expired credit card. The money does leave your merchant account.

Wednesday: You can credit a canceled credit card. The money does leave your merchant account.

Thursday: You can credit a debit card tied to a checking account that has been closed for months. The money does leave your merchant account.

Friday: If you charge somebody with an international credit card and then refund their money, by the time the money gets back on, the dollar will have weakened!

Lucky you to learn these things the fun fun-facts way!

The Final Update

Okay, all the people who had still not gotten their refunds was starting to seem a little weird, so after further investigation yesterday, I think we’ve finally got things completely fixed.

It turns out, there was a glitch in our new PayflowPro.pm that resulted in only the first transaction in a single second actually going through! According to Paypal’s site, that PayflowPro.pm should be just a drop-in replacement for the old PFProAPI.pm… and it did seem to be, after changing two lines everything seemed okay.

However, there was one little difference. The new HTTPS interface requires you to pass a unique id for each transaction, and PayflowPro.pm generated that unique id as follows:

my $request_id=substr(time . $data->{TRXTYPE} . $data->{INVNUM},0,32);

The problem was, we never passed in the (optional) “INVNUM” field.. we had an invoice number, but we passed it in as the (also optional) “COMMENT1″. So, our “unique” request_id was pretty much just the current time (plus whether it was a sale or a credit)!

In my testing this didn’t fail, because I didn’t run multiple transactions in the same second. Also, they apparently still return the same old success code we test for when this happens! But when multiple biller services run in parallel on all our controllers, lots of transactions end up happening on the same second.

The Upside

It turns out of the actually closer to $9,600,000 we thought we mistakenly charged, only actually about 1/4 of them ever _actually_ hit people’s credit cards. Our system thought we charged them, and they received an email receipt, but that was where it ended. It turns out we actually billed “only” about $2,100,000 incorrectly.

The Downside

This bug still existed until late last night (around 4am).. so when we ran our super-refunder script, the same thing was happening. Only about 1/4 of the refunds successfully went through. This resulted in the following situation:

About 9/16th of our customers: weren’t actually billed OR actually refunded.
About 1/16th of our customers: were billed AND were refunded.
About 3/16th of our customers: were billed BUT WEREN’T refunded.
About 3/16th of our customers: weren’t billed BUT WERE refunded. (of course, nobody wrote in about it!)

Anyway, last night we fixed the bug (by passing our invoice in as INVNUM) and re-ran another fixer that took an actual log of successful transactions downloaded from our processor and cross-referenced everything with our system. This is what it did:

About 9/16th of our customers: marked their bill and refund as $0 amount.
About 1/16th of our customers: left everything alone.
About 3/16th of our customers: redid the refund.
About 3/16th of our customers: redid the charge.

Double checking now, there were no more of those glitches from before, so everything seems okay.

Once again, all the stuff mentioned in the last post still holds true (you may not see the correction on your statement yet, but if you call your processor they should see it coming, for REALs this time), and once again, I’m very sorry about this whole fiasco.

Sincerely,
Josh Jones

P.S. For people wondering how the “robust and stable” rebiller could have created multiple future charges for the same date… I guess I meant “robust and stable” in regards to normal use over the last ten years. It looks like in this case, when multiple instances were running in parallel on a future date, race conditions allowed some multiple charges for the same period to be created. That too should never happen again now that we don’t allow future bill dates.