Congratulations, Nastia!

It’s unfortunate about your name, but at least you’ve got a gold medal!

Which is more than you can say for most Happy DreamHost Customers!

Domain Name: NASTIALIUKIN.COM
Created on: 07-MAR-02
Expires on: 08-MAR-09
Last Updated on: 17-FEB-08

Administrative, Technical Contact:
Liukin, Nastia liukinfan2004@yahoo.com.au
nil
PO Box 5727
Hughes
Canberra, ACT 2605
AU
0407266797

Domain servers in listed order:
NS1.DREAMHOST.COM
NS2.DREAMHOST.COM
NS3.DREAMHOST.COM

(Early this morning, one of our shared web servers ovaltine started crashing. We quickly narrowed it down to nastialiukin.com. At her website manager’s request, we moved her temporarily to a different shared server with nobody else on it. It also kept crashing!

So, we switched her (static html) site from Apache to lighttpd and immediately the load dropped from 200 to .2 (unfortunately, lighttpd is really only good for static files).. but then the network port started to saturate!

To deal with that, (again at her website manager’s request) we turned off the auto-playing flash mp3 streamer on the front of her site (who wants those anyway), and then things were finally under control!)

Go visit and watch that visitor # in the corner GROW!

So, what with all this mobile spam and iPhone 3G talk, I spur-of-the-day decided last week to check out Sprint and AT+T’s current wireless data offerings.

Well, mostly spur-of-the-day. It was also because my current laptop is starting to die (just like my last three VAIOs, something comes loose somewhere deep inside its soul, and it begins to just randomly turn off), dammit. So, since I’m pretty sure my next laptop is not going to have a PCMCIA slot to fit my current Verizon EVDO modem, I need to get a USB dongle one anyway.

I also kinda wanted to check out how the coverage and speed was on AT+T’s 3G network, just in case I need to switch over to it come July 11th. I’m sorry T-Mobile, it’s been great, but you still ain’t got no high-speed Internets!

While at Radio Shack I also decided to get the USB modem Sprint makes and try it out. Despite the 2-year contract you need to sign to get $60/month (for only 5GB of transfer now!) and the $150 off the devices, both providers have a trial period so I figured might as well. And, Sprint has their new “Compass 597″ USB modem which is not only itty-bitty, but has a microSD card slot AND GPS.

A few things I noticed right at the store: Sprint’s rebate (at least at Radio Shack) was mail-in and left the price of the modem at $50, whereas AT+T’s was instant and made the modem free. But, as if to make up for that, AT+T did a sleazy thing where they automatically add one month of “free” roadside assistance to your account. You cannot opt out of it, and if you don’t call to cancel they’ll start charging you $3 a month for something probably all your credit cards already provide you for free. Boo!

The Tests

To test each card, I first did a bunch of pings to google.com, and recorded the dropped packets, minimum, maximum, and average ping times. Then, to measure the download speed I downloaded iTunes (56MB) from apple.com, and to measure the upload speed I then uploaded it right back to a DreamHost server via FTP…

At Home

At home, I have pretty close to perfect coverage for Verizon, Sprint, and AT+T. Under these ideal conditions I found:

Wow.. DreamHost is the bandwidth king! Ha!

Ha, well, I just threw in my home cable modem connection and DreamHost’s data center for comparison purposes… AT+T seems to be the current wireless bandwidth king! Sprint looks a bit better in the latency department, but only by a hair, and all three were fast enough to not even notice a lag in a remote shell window.

The Road

For my next test, I rode the bus to work like I do, down the 10 freeway in Los Angeles. Along the way, all three have great coverage; except actually Verizon’s EVDO Rev. A (”Broadband Access”) drops down to 1xRTT (”National Access”) a couple of times. 1xRTT, like everybody’s “2G” technologies, is slowwwwwwwww and suuuuuuuuuuucks. And, annoyingly Verizon’s “VZAccess Manager” software seems (at least for me) to never be able to jump back up to EVDO once it’s dropped down to 1XRTT without manually disconnecting and reconnecting.

At 60 MPH, both Sprint and AT+T’s download speeds were cut in about half, to 60 KB/s and 130 KB/s, respectively. Good scientist that I am, I, um, forgot to do the upload test (or test Verizon), but even if Verizon performed exactly the same as it did at home, it would have still been slower!

Ping times more or less stayed constant.

The Office

Our new office is way up there, and cell phone coverage of all kinds is a bit spotty. What I learned at the office was you do not want to use any of these in a location where they drop back to their “2G” speeds.

At 700 feet, all three of them are in the 10-30% packet loss with 400 ms ping times range… along with download speeds around 10 KB/s and upload speeds around 5 KB/s! I did learn that it seems like both Sprint and AT+T are better at automatically noticing when they can switch back to EVDO/HSDPA than Verizon was, and it’s nice not having to re-connect things yourself.

The Software

Something nobody seems to ever mention in reviews of these wireless data services is what the client software you have to install is like. Well, nobody bar me!

Sprint’s is garbage, Verizon’s is fine, AT+T’s seems good.

Sprint’s software took the longest to start each time (AT+T was a little slower than Verizon, but both were still twice as fast as Sprint), couldn’t minimize to the system tray, and, worst of all, doesn’t work with a VPN! At least not mine! If you connect to the VPN while connected to Sprint, about three seconds later you’re disconnected. Then, if you tried to re-connect, the Sprint software gives an error about being connected to another network device! DEAL BREAKER!

Also, the MicroSD slot is useless for me, and the built-in GPS I guess is interesting but I couldn’t figure out how to make it do anything besides give me my current latitude and longitude.. in degrees! Are you supposed to cut and paste it into google maps or something? Anyway, if I’ve got an Internet connection, I generally don’t need GPS.

As for Verizon’s software, the one thing I always wish they added was a signal-strength meter in the system tray when minimized. AT+T’s software does this, yay! Other than that, both are pretty much fine. They’re simple, don’t get in the way, and work with my VPN. (Er, I think.. I think I need to do a little more testing with AT+T…)

The Device

It’s true, Sprint’s modem is the smallest. However, it also sticks out the furthest from your laptop, like a USB stick. The AT+T one is about as big as one of those old Nokia brick phones, but at least it has a hinge so it is aligned veritcally. The Verizon one actually has the smallest form factor outside the laptop, just because the PCMCIA part is so big. But, my next laptop probably isn’t going to have a PCMCIA port, dammit!

ATT BFF

So, I guess it’s probable pretty clear by now, I’m going to go ahead and stick with the AT+T 3G device. It does have the biggest physical footprint, but I’ve already devised a plan to deal with that! Stay tuned for how that turns out…

I guess I’m sort of glad my laptop started dying, dammit.

(My mobile broadband is now four times faster!)

Can you believe it’s already that time of the year again…?
Apple iPhone rumor time-of-the-year that is!

I’m sure you remember back in early 2007 when I predicted Apple was about to announce the iTunes Mega Store, the iPod Crumple, and the iTV (which actually did get announced as Apple TV, but was missing the video games part).

Well, to not risk my perfect 0-3 record, this time I’m NOT making any predictions (after all, everybody already knows it’s just going to be the same phone but with 3G, GPS, more memory, 3rd-party apps, thicker, more expensive, and pretty colors). No, instead I’m just going to make a standard run-of-the-mill wish list.

After giving away an iPhone last year, I eventually got (a used 4GB unlocked/jailbroken) one for myself. And I gotta admit, it’s pretty durn good. I generally hate Apple, OS X, their stores, their ads, their employees, their employees shirts, Steve Jobs, and everything he stands for, but it seems like their design “mantra” just works for a portable device. Perfectly.

Except for three things. Well, four. Well, really zero.

The phone is just fine how it is.. but I just think it could be slightly better if it could only….

1. Control iTunes!

If your iPhone is on a wi-fi network and sees an iTunes share, then the iPod app should show a little select box at the top and let you toggle between music on your actual iPhone and music on the iTunes share. Plus, if your iPhone sees any remote speakers connected to Airport Expresses on that same wifi network, it should give you another select box to choose beetween outputting the music to your headset, the remote speakers, or both!

It’s stuff they already do with iTunes on a computer; if they just added it to the iPhone (and iPod Touch), they’d have a pretty dope competitor to the Sonos Digital Music System that everybody raves about.

2. Print!

Yeah, I know.. who prints anymore? I don’t that much.. but when I do it’s always an email attachment somebody wants me to sign and fax back to them.. and the office printer/fax is kinda far from my office.

Just this week there were forty-six times when I’ve gotten all the way to the printer, only to realize that I somehow missed printing the eighth attachment! Or even worse, got a PC LOAD LETTER! Of course, from there I can still VIEW the attachment right there on my iPhone, but I sure as toner can’t print it.

But forget me, if Apple allowed printing directly from an iPhone (again, by seeing a bonjour-enabled printer on the wireless, perhaps via an Airport Express), the iPhone could become a complete computer replacement for millions of parents and grand-parents world-wide!

Think about it… what do people over 55 use a computer for (if they use one at all) these days? Email, web browsing, photos, to do lists, maps… that’s pretty much it. The iPhone is honestly a completely passable way to handle these things, especially in the low quantities required by The Greatest Generation.

The only thing missing is, these people love to print. LOVE IT. L. O. V. E. I. T.
Web pages, email attachments, notes, maps, calendars, photos, whatever… if it deserves to be seen, then it deserves to be printed.

If all you had to buy your dad for Father’s Day was an iPhone, an Airport Express, a printer and some speakers for your folks to have a painless “Internet and Digital Music Experience”, I think that’d be a pretty solid value proposition. Especially with gas prices how they are these days. Or something.

3. Support UMA!

Unlicensed Mobile Access is the thing T-Mobile’s HotSpot @Home uses… it’s the technology that allows for “seamless” switching between free VOIP calls (when you’re on wi-fi) to regular cell network calls (when you’re not), all at the same phone number! I had it before my iPhone, and I loved it… since it meant I had great (not to mention unlimited free) reception in my house, office, and datacenters; all of which normally do not.

Unfortuantely, the iPhone was too awesome, and I dropped the HotSpot @Home service. But now when I’m in the office (where reception is spotty) I’m forced to set up call-forwarding to a Skype-in number that rings on my computer. Which is almost as annoying as it is stupid.

So hey, Apple… add UMA support to the iPhone! I know AT+T doesn’t support it, but think of the millions of people who’ve unlocked their iPhone and use it on a network that does. This feature alone would probably mean the switching of our entire admin team over to iPhones, since they all need “smart phone”-type devices and spend a lot of time in underground data centers where there’s wi-fi but no cell reception.

IN FACT.

I hereby officially offer a $10,000 bounty, (payable by DreamHost) to anybody who releases a UMA application (it can be an official app or only work with jailbroken phones) for the iPhone! The only requirement is that it work with T-Mobile’s HotSpot @Home service as well as any other UMA phone does. Ideally, once installed it’d just seamlessly run in the background, and if you’re on wi-fi it would just change your little “T-Mobile” carrier name in the top to the name of the 802.11 network you were on.

If you make this UMA app, feel free to charge whatever you’d like for it, but for the $10,000 BOOTY, DreamHost gets an unlimited license to use it for all their employees. Deal?

(4. Be a Universal Remote!)

Okay, this isn’t really something I expect Apple themselves to add, but it’s an app I think somebody should make. Or if somebody already has, please let me know in the comments.

I want to completely control my home entertainment system from my iPhone. I barely dare imagine it… no more six remotes sitting on the coffee table in the living room, just the one iPhone I already have in my pocket!

The system would have a single wi-fi/IR receiver/transmitter that hooks up to all your entertainment components. You would program that with all your current remotes, then throw them in the attic until their day on eBay comes. That base station thing would then serve a web page at some IP (which it’d show on an lcd screen) where you’d go to customize the interface and commands for your particular system.

(Personally, I’d just need a power button, volume up/down/mute, and an Xbox 360/Wii/cable toggle. If it was on cable tv, then there’d also be guide button, as well as fast-forward/rewind/pause, and a d-pad with a select button…)

I guess then your iPhone could just visit that web page to control the system, but it’d probably be a much slicker experience if there was an actual app that did it. Then you could do stuff like use the physical volume and mute buttons on the iPhone to control the TV! And change channels / navigate menus using motion gestures!

In Conclusion

If the iPhone could just do these four things, I would buy a mac. Well no not really, but I would control my TV, music, printer, as well as make free unlimited phone calls (with good reception) all from the same stupid device from the same stupid company that makes stupid macs.

Convergence, baby!

It’s what $200 stock prices are built on.

Okay, nothing silly this time, I promise…

Some of you may have noticed that we’ve been having what a problem that is, although maybe not the worst in DreamHost history, definitely in the top 5.

There has been a DreamHost Status post about it, but it’s been going on so long, there obviously needs to be more said.

The History

The events that conspired to cause this horrible performance for everybody in our “blingy” cluster actually started to take root 19 months ago.

That was when I made this post asking our customers for some suggestions on storage. I made the mistake in that post of mentioning the name of one particular storage vendor who apparently does a search for their name in rss feeds of all kinds of blogs. I won’t mention their name again here, to test if they REALLY read this blog, but they were the one on the list right after “Netapp”.

Anyway, immediately a sales guy from there was hounding me about how great their product was. It would have super-duper reliability, super-duper performance, and super-duper ease-of-management. It was super-duper expensive compared to our current solution (about 3x the price per GB), so in the end I declined.

But, over the next year he kept hounding me and hounding me, and eventually the price came down to something in line with our current costs, so we decided to try one unit for our new cluster, “Blingy”. After we were satisfied with our internal testing, Blingy went live with the new storage solution in December 2007.

Smooth Sailing

At first, everything was fine, performance was great, everybody was hunky and dory. But then, as usage started to go up, the new file system started acting up. Around the same time every night, the system would stop responding to NFS requests for a while, which would immediately break web and mail service for everybody in the entire cluster.. thousands of customers.

Our Bad

Now, it can be a big mistake to put live customers on any new system. But honestly, we’d tested it lots, researched it a ton, and we added people very slowly at first, and it performed great.

Our biggest mistake I believe had nothing to do with what specific vendor or hardware we went with.. it was simply putting so many eggs in one basket!

Even with our Netapps (which are pretty much awesome), there are problems from time-to-time. However, a typical hosting cluster will have a dozen or so Netapps, which means any problems are one twelfth as big.

With Blingy, EVERY customer is on this one “mega” filer, which in theory should make for better performance, reliability, and ease of management. And since we got the clustered solution (in an active-active configuration)… there really is no single point of hardware failure in this thing.

But, as it turns out, there are a lot of non-hardware failures in the world.

Their Bad

Well, the techs at the vendor couldn’t figure out what was causing the NFS freezing, and so they recommended us doing a major OS upgrade to hopefully fix it.

During this whole time, the fiber channel disks were slowly filling up, and we’d been trying to move large files off to the sata pool (it’s a two-tiered solution, and there’s a feature that automatically moves less-accessed data to lower tiers).. however the thing couldn’t move the data fast enough. It couldn’t finish doing a “move job” in a single day, and every day it’d sort of “crash”, which would screw up the move job, and nothing would get moved.

Also, as the disk kept getting more full, performance kept getting worse, creating a vicious cycle. We ordered some more fiber channel disk shelves at the end of February to grow the main FC volume, since we couldn’t get things off to SATA, and it was supposed to come on March 10th and be installed at the same time as the major OS upgrade.

However, the disks didn’t end up getting installed until March 25th, and at that point it turned out we could NOT grow the FC volume with these disks (well, it was technically possible, but their on-site techs recommended VERY VERY heavily against it.. it would severly impact performance), which was sort of the whole point. So now we had a new FC volume which we still had to migrate users to.

Your Bad

Of course, this whole time, new customers just kept signing up, and being added to Blingy. What were you guys thinking?

By this point we knew this was a bad idea, but we didn’t have a new cluster ready (we’d expected Blingy to grow for another couple of months), and we try to never ever grow old clusters again once they’ve been “shut off” from new signups (because in time they stablize and have very few problems).

However, the moving people off to the new FC vol, or the original SATA vol, or even the new Netapp we also added to Blingy, just wasn’t happening fast enough. So on April 2nd we bit the bullet and switched Blingy off as the “new customer” cluster and started growing good old “Postal” again. Once we did that, we were finally able to get ahead of the curve and total usage on our first fiber channel volume has been slowly dropping ever since.

We tried at that point to contact the vendor to see if we could just get more drives that WOULD allow us to grow fcvol1, but they said their manufacturers were closed for inventory for a week after the end of the quarter and we couldn’t get anything until Friday, April 11th at the absolute soonest. Later they said they could find us some they could get us by Tuesday, April 7th, and we preliminarily said we’d take them.

This whole time we had a support ticket open with the vendor about the crashes (the OS upgrade didn’t fix it), and finally on April 3rd we received notice that they’d fixed the bug that they believed was causing it! However, the patch still needed to go through their “QA”. Finally, this Sunday April 6th they said it was all ready to be deployed, so last night we did.

What Now

Well, right now, performance is still not great on fcvol1… but mail and web should be pretty much working. One thing we’ve noticed is a website that hasn’t been visited in a long time will have a big lag still upon the first visit.. but then subsequent reloads/visits seem much faster.

At least the total disk usage is coming down now, and hopefully by tomorrow it’ll be below 85% which is supposedly a magic number where performance is fine. We’re going to keep off-loading it until things are great, though. We’ve got plenty of disk space for it, the problem is just it takes so long to move it.

We also I guess will find out tonight if the NFS freezing bug is fixed by this new patch. Hopefully so.

It’s Too Late…

I realize this is probably too little too late for many of you, but I just wanted to sincerely apologize for this whole big Blingy cluster-f*ck. Also, if you’re on Blingy (you can tell from the panel by clicking “account status” and looking at “Your Email Server”, we’d like to offer you a month worth of hosting credit.

To get it, all you need to do is contact support from our panel and make the subject of your message “Blingy Account Credit”. That’s all you have to do, and we’ll credit everybody who asks (and is actually on Blingy!) next Monday (April 14th).

Quick, before it’s gone!

If you enjoy all the hilarious hijinks, illuminating illustrations, and jovial jokes of the DreamHost Blog, you better suck down a local copy TODAY…

We’re having a planned power outage tomorrow night!

(Click that link for some more details.. it’ll be from 11:15pm PST (GMT -0800) tomorrow night (Saturday) to hopefully much less than 5 hours from then.)

Not planned by us though, planned by our building. It would have been very nice if they could have given us a little earlier heads up, or avoided the outage at all, but no, they just can’t. And trust me, we want this to happen even a tiny bit less than you do!

So, this site will be down then, as well as all other DreamHost services, with the exception of ns2.dreamhost.com and dreamhoststatus.com, which are kept off-site for exactly this sort of situation.

Well, I just thought I better post something about it here too.. thanks for your understanding, and we’re really really really really sorry.

P.S. Here’s the pic the building emailed us of the problem:

So, um, yeah. I think what that shows is a piece of metal is vibrating next to that wire and cutting into the rubber insulation… and if it gets much further in, KABOOM!

Last night we made some moves.

Patrick and I moved about 60 servers!

And I only dropped one! (Sorry about that, bomberman.)

It took about two hours, and here we are, wrapping things up:

Stage 2? At 12:30 in the morning, after moving 60 servers, what else could we possibly want to move for a STAGE 2?

Hmm… something about “Brea”?

We passed this car in the parking lot.. and soon, we were at the OTHER DreamHost office.

We waited.. THE CON WAS ON!

Patrick had told Pete (who lives right by the office) that he was just in the area, at 1:30am on a Wednesday, and I wanted him to pick up some WWF glasses for the downtown office. But, HAD PETE PLAYED US FOR FOUR FOOLS?!

Apparently not….

We made short work of the coveted sign.

And then I decided to go raid the kitchen…. WHAAaaaa!!!!

Yes, very funny, Brea. But who’s wearing the cool shades now?!

As long as we were there, we thought we might as well have some fun…

And some more fun…

We took our time. We even checked out the Official DreamHost Museum!

Why hello there, Señor Corona, you sure are working late tonight!

Of course, we couldn’t just leave those poor, unsuspecting Breaites bare-walled!

Around 3:30, we were back “home.” Mission complete. Tired. Satisfied. Ugly.

The neon sign was finally where it has always been destined to be. Down in our NOC. The HEART of DreamHost.

Epilogue…

Happy New Year!

The snow’s not even dry on the rooftops of LA and we here at DreamHost already have a pile of resolutions for, as the cool sports video gamers call it, the 2K7.

In 2007 we do solemnly resolve to:

#1. Never get involved in a land war in Asia.

#2. Never go in against a Sicilian when death is on the line!

#3. Become once again renowned the Web-over as a stable, reliable, robust, high-performance webhost!

As those of you who’ve been playing the home game know, we had some troubles in 2006.

But actually, the root of those troubles began WAY back to June 2005, when the building our data center is in informed everybody they were unable to provide any new UPS and generator-backed power, period.

Moving data centers wasn’t really doable back then, and so for the next year or so we were forced into “low-power mode” .. scrapping our Dedicated Servers option and squeezing every last bit of power efficiency we could from our operations, even at a fair amount of expense.

Somehow, we kept going. And going. And going. And gahhhh, you get it.

And really, our service didn’t suffer for it.

But then, exactly one year ago today, something changed that seemed to affect our reputation for the worse ever since.

We started giving away a lot more disk and bandwidth. Like 8 times. As. Much.

That’s when things went downhill.

Well, not really.

In fact, we had exactly the same amount of problems (actually less, per customer!) we’d had the last eight years, but now finally people could put their finger on a REASON for them!

We were overselling!

Clearly, every problem we had stemmed from the simple fact that we gave away too much disk and bandwidth!

Well, I’ve already covered “overselling” plenty, and ALL the quota increases really did was increase the number of new customers we got!

But still, all through 2006 our rep seemed to slowly decline.

Every time we had a server crash; “Overselling.” A network fubar, “They’re overselling.” A panel bug: “Didn’t your mama ever teach you about overselling?” A power outage? “Oh yeah, sign up for DreamHost if you happen to like a fresh bunch of OVERSELLING!!!”

Of course, the power outages didn’t help. Nor did the weird problem between our two core routers that made our entire network suck eggs for six weeks this summer.

But in a way, those power outages were perhaps a blessing in disguise. A disguise that reminded me of a big mob of angry customers.

Those outages forced us, and our building, to really DO something about the power situation… which as you may recall is the real foundation for any stability problems we’ve had in the last 12 months.

After the power outages this summer, the building started literally BLEEDING data center tenants, figuratively.

This had two effects. Ichi, it forced them to start taking their UPS and generator problems seriously, and as of now they actually seem to have things in order. In fact, believe it or not, just TODAY the building experienced a power outage from DWP… and for the first time ever we were not affected at all!

Memo
DATE:             January 3, 2007
TO:                All Garland Tenants
FROM:            Timothy J. Moore
RE:                 DWP Power Outage Today

This is to advise you that at approximately 11:50am today, the Garland Building received a power outage from the Department of Water and Power.  The outage lasted less than one minute and all systems worked according to design.

The Building’s loads were transferred to the Emergency Generator System.  Upon stabilization of the DWP service, all ATS Switches transferred back to normal DWP power.

Should you have any questions, please do not hesitate to contact the Office of the Building.

Sincerely,
The Happy Office of the Building Team

Oh BOY was I ticked OFF when I saw how they stole our signature signature!

Ni, by bleeding those tenants, a lot of power was freed up for us! And by us I mean you! LITERALLY.

Also, we were now of a size (thanks, ironically, to our generous bandwidth and disk allocations!) that expanding to more than one data center was finally feasible.

So, this fall we expanded to two more facilities, with dark fiber connections between all three.

With all this new power available, we were finally able to spend more on hardware! So we did, and have been, and are, and will be! We’ve put many many fewer users per web server, mysql server, and mail server, added load balancers, beefed up our network equipment, and have added new targets (that we now have the power to attain) for server stability.

In fact, we spent over ONE MILLLLION DOLLARS on hardware in November alone! That’s more than we normally spend in a whole quarter! And in fact, things are quantitatively more stable now across our whole system than they’ve ever been in the past.

But our reputation as an “overselling host” seems to linger!

How can we fix it? Aren’t people just going to notice things are a lot better? And start telling their friends?

Won’t they just believe this blog post?

Probably not. It’s a stumper!

Fortunately, I pulled deep into my master-of-public-relations pouch, and pulled out this gem of wisdom:

People aren’t going to consider us a “stable” host until we offer LESS DISK AND BANDWIDTH!

But…ARGH! More disk and bw => more sign ups => more money => more resources => better service!

What to do?

Fortunately, I have a master-of-marketing pouch too (double-major).. so here’s what we’re doing:

Every day, starting tomorrow, the amount of starting disk and bandwidth we offer new customers (this does not affect existing customers at all!) will drop. You can see the amounts here.

(Don’t worry, once you sign up, your disk and bandwidth allocations will grow weekly just like before!)

And we’ll keep dropping them daily until our precious rep is restored!

(Or it cuts into our sign-ups too much.)

(Whichever comes first.)

(Reputation be damned.)

I’ve got a question.

And I thought, who better to ask, than everybody?

Here goes…

We’ve got pretty serious storage needs. Like, in the next year, we’re estimating needing about 250TB (big T, big B) of additional centralized, networked, storage.

Besides needing a lot, we also need very high performance, redundancy, and thrift.

We want it ALL!

Our absolute requirements for our system are as follows:

* RELIABILITY .. we can never ever lose any data, ever.
* PERFORMANCE .. we need something that can serve approximately 3000 NFS ops per second per TB. (See spec.org). (It needs to do NFS.)
* PRICE .. that’s the whole reason we’re looking.

We’ve got a good system of RELIABILITY and PERFORMANCE already.. but the cost per usable GB is $10. The main problem is the 300GB Fiber Channel drives we use, which are $800 each. Is there anything out there that can do the same but with SATA drives that cost more like $100? Even if we needed twice or four times as many drives for the same performance and reliability, it seems possible!

There are also some REALLY WANT TO HAVE features, though possibly could be passed up if the top three are satisfied.

* SNAPSHOTS .. automatic versioning backups of all files by the OS. We’ve got this now, in a hidden “.snapshot” directory in every folder.. check it out!
* USER QUOTAS .. really, with the amount of space we’re giving out these days, quotas are almost a moot point. They’d be nice to have though.
* HIGH DENSITY/LOW POWER .. it’s always a plus to fit more storage in the same amount of space with the same amount of power, but it’s not really that big a deal.
* RAID 6 SUPPORT .. it’s cool.

Here are some vendors/solutions we’re considering..

* NETAPP .. what we use now.
* BLUEARC
* ONSTOR
* PANASAS
* CORAID (Open Source, ATA over Ethernet.. intereesssssting!)
* LUSTRE (Open Source, Clustering storage)

Soo.. basically, if people could post their suggestions, experience, other solutions, etc.. in the comments, it would be much appreciated. Not to mention you will be doing your patriotic duty to improve your hosting forever!

And remember, we’re talking serious NFS ops.. and we’d be willing to buy 256TB at once if our tests showed this system can do what we want and THE PRICE IS RIGHT!

Well, things could be worse.

We’ve pretty much got our whole network under control now.. the ongoing problem mentioned last post was finally figured out by Cisco support. It turns out it was a bug undocumented feature in IOS dealing with how they learn MAC addresses.

There was also another network problem we got resolved yesterday that was causing general slowness on web and mail servers. It’s complicated (i.e. I don’t understand it exactly myself), but in the end we took a distribution switch out of the network and that fixed it.

We still have one open ticket with Cisco for our core routers having some HSRP problems. It doesn’t seem like that’s having any real effect on our network, but we want it fixed!

We are also installing two new Ciscos to offload the BGP duties from the core routers so they’ll just have to handle switching. This set-up should be able to handle about 300% more traffic than our entire network now pushes at peak times!

Thanks to these network problems being resolved, we’ve also begun re-deploying in Alchemy, who at least didn’t have the second power outage.

We’re also still in the process of getting real UPS power on our network cabinet, plus our internal databases and a few internal servers. Basically, everything that keeps all the customer mail, web, database, and file servers from coming right back up quickly should there ever be another outage.

So, um.. that’s how it stands now! We hope this will all soon be nothing more than a long bad dream (that was real).

Last week, as I was lazily thumbing through the May 2006 issue of AmericanWay, the AWARD-WINNING in-flight magazine of American Airlines, a familar face caught my eye.

Who WAS that cheery-eyed elf peering back at me from the upper-right hand corner of THE EXPERT SPEAKS?

I knew I’d never met the man, and yet I’d seen his face many times.. and yet I also knew he wasn’t famous. Where was he from? I couldn’t quite place it. It finally hit me when I read the introductory text… of course!

It was none other than CEO and publicity-hound extraordinaire of our favorite competitor CI HOST, C.F.! (One must never actually type his full name, lest he suddenly appear in a flash of smoke, lawsuit in hand.)

Great! So good to see other hosting guys moving up in the world!

I practically quivered in anticipation of the gleaming nuggets of insight soon to be bestowed upon me!

My practical quivering was soon rewarded as I came across this beautiful passage:

(Personally, I prefer my UMPCs with Nintendo DS.)