.info .redo!

For some reason .info is the only TLD besides .com/net/org that we offer for registration (of course you can host any domain) at DreamHost.

They were just one of the first “new” TLDs I suppose. Back before we got burnt out on them all.

Of course, there were a few people who got burnt by .info domains themselves… by the .info sunrise period, more specifically. The .info registry themselves!

I just got an email this morning that Afilias (the .info registry) will be reallocating a bunch of .info domains (see the end of this post for the full list) that were somehow “incorrectly” registered during their sunrise period many years ago.

I don’t know what comprises “incorrectly” .. maybe they just mean “squattedly” but the upshot is you can now submit an application to get one or more of the domains by filling out this form and emailing it as a PDF as well as mailing a paper copy with a check for $250 to

Attn: Sunrise Reallocation RFP
Afilias Limited
c/o Afilias USA, Inc.
300 Welsh Road, Bldg 3, Ste. 105
Horsham, PA 19044
USA

by December 22nd.

You’ve got to put on the application why want the domain(s) you’re going after, and I guess they’re just going to pick whomever is most “deserving”. Or their cousin.

I just thought this was kind of interesting because Afilias is opening themselves up to a whole lotta sour grapes from all the cyber squatters who don’t get the domains they wanted. And in the end all that changes is Afilias makes a lot of $250 application fees and new and different cyber squatters get the domains!

The only way I can think of to make this fair is to just randomly assign the domains among all applications. Or, if they really want to hold editorial control over who gets them, the criteria should be that the requester already be running an active website on a similar domain, that they then agree to relinquish over to Afilias in return for the .info domain they will start using in its stead.

Then I could swap information.la for losangeles.info!

114.info
24seven.info
3bo.info
401k.info
4-best.info
4flow.info
4runner.info
88edibleoil.info
aar.info
aar-treuhand.info
abenteuer.info
abfs.info
accessgroup.info
acousticwavemusicsystem.info
acousticwavemusicsystem.info
adata.info
adizesgraduateschool.info
adizesinstitute.info
adizesusa.info
aegonassetmanagement.info
aei.info
airline.info
airzone.info
alava.info
albacete.info
alicante.info
all.info
allergy.info
allianz-arena.info
allinfo.info
almeria.info
alte-leipziger-bauspar.info
alte-leipziger-holding.info
alte-leipziger-institut.info
amade.info
ameda.info
america.info
americantower.info
americantown.info
americantowns.info
amtrack.info
anchorage.info
andalucia.info
aniscastellana.info
anis-castellana.info
antalya.info
antiquitaeten.info
apartments.info
aqua.info
aquaculture.info
aragon.info
arthritis.info
arucana.info
asbestos.info
askamd.info
askthedude.info
asn.info
aspen.info
assetweb.info
asturias.info
athlon.info
athritis.info
auction.info
australien.info
auto.info
autoengineer.info
automation.info
autorouter.info
autos.info
aveny.info
aviation.info
avila.info
avnetexpress.info
avondale.info
ayurveda.info
a-z.info
baaworldpoints.info
bachelor.info
badajoz.info
badgirls.info
badwesternkotten.info
bahn.info
bairstow.info
baleares.info
baltimoresun.info
balve.info
bambinchen.info
bamboo.info
banff.info
bangkok.info
bank.info
bankshot.info
bargain.info
bargains.info
baseball.info
basell.info
bassfishing.info
bauernherbst.info
baukosten.info
bauzirkel.info
bdiu.info
beauty.info
beavercreek.info
bedandbreakfast.info
bedding.info
beer.info
beinhart.info
benetone.info
bettersex.info
bible.info
bier.info
big.info
bikehotels.info
bilbao.info
bisex.info
bit.info
blick.info
blinddater.info
blinds.info
blinds2go.info
bluetooth.info
boat.info
body.info
bollywood.info
bolsa.info
bonds.info
books.info
boot.info
bosecorporation.info
boston.info
boxoffice.info
branchenbuch.info
brauerei.info
braunschweig.info
bravoscreenfun.info
bregenzerwald.info
bremen.info
briefmarken.info
brigitte.info
britishamericantabacco.info
brixen.info
broadway.info
brotherhoodmutual.info
browser.info
brustkrebs.info
buchner.info
buffalo.info
buffaloboots.info
buffalo-boots.info
buildermarts.info
bunactravelservices.info
burgos.info
buydomain.info
buymine.info
byte.info
byteburg.info
caceres.info
cadiz.info
calciwise.info
caldaro.info
calence.info
calendar.info
california.info
camelotgroup.info
camelot-group.info
campingtirol.info
can.info
canarias.info
cancer.info
cargo.info
carpediem.info
cartagena.info
cas.info
casilan.info
casinos.info
castellon.info
catalunya.info
catelli.info
cbt.info
cellatlas.info
cellularphone.info
cerebys.info
certainteed-design.info
ceuta.info
charcol.info
charity.info
chat.info
chatrooms.info
cheapnights.info
chef.info
chemnitz.info
chemotherapie.info
chicagosports.info
chienes.info
childrensplace.info
chocolate.info
cholanhmb.info
cholan-hmb.info
cholesterol.info
christ.info
city.info
clearwindow.info
click4.info
click411.info
clinicaltrials.info
closs.info
clpcrowntechnicalservices.info
cms.info
coating.info
coburg.info
cocoapro.info
coins.info
cold.info
coldfusion.info
colfosco.info
colorado.info
colorcases.info
combipatch.info
comfortedgetechnology.info
comfortline.info
comfortpro.info
companyguru.info
companyinsight.info
computer.info
concrete.info
consulting.info
contender.info
copenhagen.info
cornwall.info
corp.info
corvara.info
costablanca.info
costabrava.info
costadelsol.info
craft.info
crbliss.info
creditcards.info
crown.info
crowntechnical.info
crowntechnicalservices.info
cruise.info
csi.info
cuenca.info
cyberdog.info
cyclotran.info
cygate.info
dachdecker.info
dagehtsmirgut.info
dairylicious.info
datadepot.info
dataflux.info
dateline.info
datingagency.info
dealdesk.info
deal-desk.info
deltaview.info
dental.info
denver.info
depression.info
depressionen.info
desenexmax.info
devies.info
dial-a-tech.info
diet.info
digital.info
digitaltimes.info
digitalvideo.info
directimagingcenter.info
direct-imaging-center.info
direktmarketing.info
dis.info
discounts.info
distancelearning.info
dobbiaco.info
doccontrol.info
doctor.info
doctors.info
dolcevita.info
dolomiten.info
dolomiti.info
door.info
dortmund.info
dotweb.info
dowagro.info
dowagrosciences.info
dowchemical.info
dowchemicalcompany.info
driveweb.info
druckmaschinen.info
dsl.info
dsv-saaten.info
duplium.info
durasolv.info
ear.info
earlydefense.info
earth.info
easytrade.info
ebank.info
ebia.info
ecitizen.info
economic.info
economics.info
ecopower.info
ediligence.info
edisonelectricinst.info
e-epoxy.info
e-fficiency.info
e-flood.info
eggdesign.info
egger.info
eggerwirt.info
egggroup.info
egginteractive.info
eisacktal.info
ejgallo.info
elearning.info
election.info
electric-blue.info
elevator.info
e-mail.info
energie.info
enews.info
eng.info
engineering.info
envigor.info
enviroment.info
envision.info
epilepsyaction.info
epoxy.info
equicareanddesign.info
erci.info
erektion.info
eres.info
ergo.info
eroski.info
escorts.info
etickets.info
etop.info
eucaris.info
eurekaconnection.info
eva.info
event.info
everlux.info
everything.info
e-watch.info
excact.info
exhibitions.info
exiban30.info
eyeclinic.info
eyewitness.info
fact.info
family.info
farmacia.info
farmbureauconnection.info
fashion.info
fedcup.info
feesten.info
fengshui.info
ferienwohnungen.info
ferrum.info
finance.info
financialadvisor.info
financialservices.info
findlaw.info
fineliving.info
fingersnackers.info
fish4cars.info
fish4jobs.info
fishing.info
flashlight.info
flexiblelogic.info
flexible-logic.info
flight.info
flkeys.info
floodwise.info
florida.info
flowers.info
flualert.info
flute.info
fmb.info
focalpoint.info
focuscomfortchallenge.info
focuscomforttrial.info
focusdayplus.info
focuseveryday.info
focusevery-day.info
focusexpress.info
foodinaminute.info
foodtv.info
foremost.info
forestbank.info
formentera.info
forsale.info
fostercare.info
franchise.info
frankfurt.info
frechdachs.info
freerider.info
freshfieldsbruckhaus.info
freshfields-bruckhaus-deringer.info
freshfieldsinternational.info
frijo.info
fristampumps.info
fuelcell.info
fuerteventura.info
fuerteventura.info
funday-times.info
furniture.info
fuschlamsee.info
fussball.info
futbol.info
futurephoto.info
galaxy.info
gamble.info
gambler.info
gambling.info
game.info
games.info
gapyear.info
gardengigglers.info
gazoline.info
gbg.info
gems.info
general.info
generika.info
genetics.info
genimap.info
gentleflex.info
geologie.info
geriatrie.info
getpacking.info
getsmartgetnextel.info
ghostgum.info
giancarlofisichella.info
giebel.info
gifts.info
gilead.info
girona.info
glen-elgin.info
global.info
gloss.info
glucon-d.info
glycemet.info
glycerine.info
go2orlando.info
gochina.info
god.info
godwad.info
goforit.info
gold.info
goldcard.info
goldegg.info
goldlinecoins.info
goldlineinternational.info
goldschmuck.info
golf.info
golf-alpin.info
goodidea.info
goodluck.info
goodwear.info
government.info
grancanaria.info
gravisand.info
graz.info
green.info
greenfingers.info
grossarl.info
guadalajara.info
guipuzcoa.info
gutachter.info
haas.info
hairloss.info
halbleiter.info
halfar.info
handibites.info
handi-bites.info
handicrafts.info
hangseng.info
hangsengbank.info
hardrive.info
hartfordcourant.info
haug.info
haut.info
hautecouture.info
haute-couture.info
headache.info
health.info
healthcare.info
healthhouse.info
heartshape.info
hechtl.info
heindl.info
heinzezsquirt.info
heinzwatties.info
heitersheim.info
helios.info
hema.info
hennesmauritz.info
hepatitis.info
herpes.info
highfly.info
high-fly.info
hints.info
hiv.info
hjheinz.info
holeinone.info
holidayclub.info
home.info
homecheck.info
home-check.info
homedecorating.info
homeoffice.info
home-office.info
homesales.info
homewinemaking.info
hondaracing.info
hophing.info
hophingedibleoil.info
hophinggroup.info
horse.info
hosting.info
hotels.info
hotsex.info
howbusinessgetsdone.info
huelva.info
huesca.info
humana.info
husten.info
hypnosis.info
ibexusersgroup.info
ibill.info
ichfreumichdrauf.info
ideas.info
idem.info
ils.info
ilxsystems.info
image.info
imanet.info
imc.info
immigration.info
immo.info
immobilien.info
immobilier.info
impactrecover.info
impotence.info
inc500.info
independentliving.info
indianapolis.info
industrialvision.info
indy.info
infanthealth.info
infochoice.info
infodienst.info
information-systems-control-association.info
inharmonywitheducation.info
inkasso.info
inkassoverband.info
inkasso-verband.info
innovation.info
innovations.info
inspiration.info
instant.info
internetsolutions.info
investigation.info
investment.info
investments.info
iowahealth.info
islam.info
islamicity.info
istanbul.info
itgovernanceinstitute.info
ixl.info
jaen.info
jagdhof.info
jaipur.info
jesus.info
jiffymailers.info
job.info
jobs.info
jokes.info
jovebook.info
jovebooks.info
joy.info
jungle.info
jura.info
kaltern.info
kamind.info
kamp.info
kaprun.info
karneval.info
katschberg.info
keen.info
keystonedevice.info
keystonelight.info
kids.info
kidtrition.info
kiens.info
kimchi.info
kinder.info
kings.info
kino.info
kirchberg.info
kirchbergintirol.info
kirche.info
kleinwalsertal.info
klopeinersee.info
k-market.info
koenigsallee.info
koenigsleiten.info
kontio.info
kopfschmerz.info
krankenkassen.info
krankenversicherung.info
krebs.info
kronplatz.info
kuchl.info
kufstein.info
kunstgalerie.info
labsafteycom.info
laketahoe.info
lana.info
land.info
landsales.info
langen.info
lanzarote.info
lapalma.info
larioja.info
last-minute-reisen.info
lasvardet.info
lasvegas.info
latelier.info
latimes.info
latsch.info
laval.info
lawfund.info
laxenburg.info
lecordonbleuinternational.info
lega.info
legal.info
legendofzelda.info
leisureliving.info
leitner.info
lemond.info
leogang.info
leon.info
leukaemie.info
lidpro.info
linksorganic.info
lippstadt.info
literotica.info
loan.info
loans.info
locks.info
locksmith.info
lofer.info
logistinet.info
logiweb.info
london.info
losangeles.info
lottery.info
lotteryextra.info
lottery-extra.info
louvre.info
ltd.info
ludwigsburg.info
luescher.info
lugo.info
lungau.info
lup.info
macromedia.info
macway.info
madrid.info
makelifeeasier.info
makeup.info
maktab.info
malaga.info
maps.info
mariazell.info
marketplace.info
marketresearch.info
masterbond.info
masterfoods.info
masterfoodservices.info
match.info
matrix.info
mccabesguitar.info
mdi.info
medi.info
media.info
media-vision.info
medical.info
medicaltravel.info
medizin.info
melt.info
memorable.info
memorycard.info
merida.info
merseytv.info
mesa.info
mgcorp.info
miami.info
miamibeach.info
michael.info
micheal.info
michelobamberrock.info
migraine.info
milenniumclub.info
millenniumhotel.info
minidisco.info
miniglobe.info
ministryfirst.info
mir.info
modelaircraft.info
momentous.info
momentus.info
mondsee.info
montana.info
montecarlo.info
monte-carlo.info
moodright.info
more.info
mortgage.info
mortgages.info
motgage.info
movies.info
mp3.info
mpeg.info
mto.info
multiple-sklerose.info
murcia.info
music.info
musik.info
musique.info
mutualofamerica.info
myfavourite.info
myhouse.info
nachrichten.info
nadnet.info
nano.info
national-lottery.info
natuerlichstein.info
natureconservancy.info
naturesgoodness.info
naturno.info
navarra.info
nebo.info
neca.info
negocios.info
netapps.info
netgifts.info
netis.info
netnation.info
netzgang.info
neuschwanstein.info
new.info
news.info
newsco.info
newsouthwales.info
newyorkcity.info
nexic.info
nextelhowbusinessgetsdone.info
nextelonline.info
nordenham.info
northyorkshiremoors.info
north-yorkshire-moors.info
northyorksmoors.info
norwalkadvocate.info
nrgdirect.info
nsw.info
ntx.info
numbernames.info
nursinghome.info
nutrilose.info
nycil.info
nynewsday.info
nzi.info
oberbayern.info
oberstdorf.info
obertauern.info
oberwart.info
office.info
oha.info
oil.info
olmetec.info
oneandonly.info
onlinecasino.info
onlineshop.info
only.info
optician.info
optilose.info
optometry.info
orlando.info
orlandosentinel.info
ourense.info
overseaspracticaltraining.info
oxygen.info
pain.info
painting.info
palencia.info
palmsprings.info
pamplona.info
panoramakamera.info
papawheelie.info
paris.info
partnershipadvantage.info
passivhaus.info
passporttoministry.info
pateacoller.info
patent.info
pcb.info
peacock.info
pebblebeach.info
pediatrician.info
penguinputnam.info
penguinputnaminc.info
pension.info
pensionen.info
personeelscentrum.info
pestcontrol.info
pharmacy.info
pharoridanddesign.info
philately.info
phoenix.info
photomodell.info
physik.info
planet.info
platespin.info
player.info
playstation3.info
playstationcom.info
plescia.info
pneumo-star.info
polarbears.info
pollen.info
polyethylene.info
polyurethane.info
pontevedra.info
pontis.info
porno.info
power2learn.info
prague.info
praha.info
printmediaacademy.info
privatvermieter.info
privilege.info
prize.info
prof.info
profume.info
progirls.info
project.info
property.info
pro-same.info
protection.info
pspsi.info
psthree.info
qosfs.info
qsc.info
quietcomfort.info
quote.info
race.info
racines.info
rackmaster.info
radsport.info
radstadt.info
radtouren.info
raj.info
rajasthan.info
rap.info
rasen.info
ratgeber.info
ratschings.info
rauch.info
raumfahrt.info
rcn.info
rdw.info
reality.info
realtv.info
rechtsanwalt.info
redcross.info
refractories.info
reignyourdomain.info
reindeergames.info
reindeer-games.info
reise.info
relocation.info
rendite.info
reno.info
renon.info
resin.info
resources.info
restaurant.info
retirement.info
revshare.info
rheumatism.info
riesterrente.info
riester-rente.info
ringier.info
rioja.info
ritten.info
ritz.info
riverheadbook.info
riverheadbooks.info
robitussin.info
rock.info
roplene.info
rosicrucian.info
rotenburg.info
rottenburg.info
rugby.info
s26materna.info
s-26materna.info
s26progress.info
s-26progress.info
saalbach.info
saalfelden.info
sachsen.info
saeugling.info
safe.info
salamanca.info
salonbiz.info
salzburgerland.info
salzkammergut.info
sansebastian.info
santorini.info
schach.info
schlaf.info
schmitten.info
schmittenhoehe.info
schneeberger.info
schnupfen.info
schools.info
schwaben.info
scl55.info
scotland.info
scubabiz.info
scusa.info
sdsu.info
secretshop.info
secretshoppercompany.info
securities.info
seep.info
seghorn.info
segovia.info
sesto.info
sevilla.info
sexbet.info
sex-bet.info
sexe.info
sexo.info
sexshop.info
sexten.info
shipping.info
shocktales.info
shop.info
shops.info
siam.info
sierragateway.info
signaturebrands.info
signatures-catalog.info
signetbook.info
signetbooks.info
silicon.info
simpad.info
skireport.info
smallbusiness.info
smhi.info
snapnstore.info
soccer.info
soccer.info
social.info
societe.info
solution.info
somnu-star.info
somnu-starph.info
sonja.info
soria.info
sound.info
southcarolina.info
southgeorgia.info
spabiz.info
space.info
spacecenterinc.info
spangenberg.info
specialist.info
specialists.info
sperti.info
spielkasino.info
sportbetting.info
sportsbet.info
sportsbetting.info
sports-betting.info
sportsbook.info
sprachkurs.info
spreewald.info
springbreak.info
springfield.info
stadtplan.info
stamfordadvocate.info
stamps.info
starcrestofcalifornia.info
starpower.info
stationery.info
staycalm.info
std.info
stefan.info
stemcells.info
stern.info
sterzing.info
stlatwork.info
stlouisatwork.info
stltoday.info
st-martin.info
stocks.info
storage.info
strafrecht.info
stressless.info
strideguard.info
stroke.info
stubaiergletscher.info
studenthomes.info
stuttering.info
stveit.info
suckit.info
sulden.info
summercampusa.info
sunar.info
susq.info
swiss.info
sybex.info
tahoe.info
tajmahal.info
talentbuilder.info
talkers.info
tanqueray-no-ten.info
tarragona.info
tauchen.info
tax.info
taxes.info
taxoffice.info
tcfbank.info
tcfexpress.info
tcfmortgage.info
tcmgroup.info
tech.info
technicalsupport.info
telecount.info
telefonbuch.info
tenderharvest.info
tenerife.info
terento.info
terlano.info
termeno.info
terminal4.info
teruel.info
test.info
textile.info
thai.info
thaichef.info
thatcorp.info
thatcorporation.info
thebathplace.info
thechristiansciencequarterly.info
thechristiansciencesentinel.info
thedowchemicalcompany.info
thenewsoftheworld.info
the-news-of-the-world.info
think.info
thirdfoods.info
ticketmedia.info
tickets.info
tigrisconsulting.info
timeshare.info
tinkerbell.info
tir.info
tirol.info
tivo.info
tivocentral.info
tivolution.info
toblach.info
tokyoelectron.info
toledo.info
tom.info
toomuch.info
topjobs.info
torrevieja.info
torusdevice.info
totalpowersolutions.info
total-power-solutions.info
toucan.info
touringbycar.info
tourist.info
traditionscatalog.info
traditions-catalog.info
tramin.info
transtec.info
traveling.info
travelling.info
travelpoint.info
treatment.info
trekbicycles.info
trekbike.info
trekbikes.info
triflur.info
trip.info
trucks.info
tuev-berlin.info
tuevdotcom.info
tuevdotcomservice.info
tuev-rheinland.info
tuv-berlin.info
tuvdotcom.info
ukevent.info
ukevents.info
undiscoveredcotswolds.info
unfall.info
urlaub.info
usa.info
useless.info
usfilter.info
usopen.info
uspca.info
uss.info
vacation.info
valdaora.info
valencia.info
valgardena.info
valladolid.info
valleisarco.info
vand.info
vaporpatch.info
veintain.info
verbier.info
verden.info
vereinswest-bank.info
vikingbook.info
vikingbooks.info
vinschgau.info
vipiteno.info
viread.info
visionsystems.info
vitalimages.info
vitamine.info
vitra.info
vivelledot.info
vizcaya.info
voiceofsmallbusiness.info
vue.info
wagrain.info
wahl2006.info
wald.info
waldorf.info
wallytool.info
warrant.info
watsonwyatt.info
watson-wyatt.info
wavepc.info
waveradiocd.info
weather.info
webdesign.info
webhost.info
webmaster.info
webshop.info
weesingbook.info
weesingbooks.info
wein.info
weine.info
welcome.info
wema.info
wengen.info
wertpapier.info
westerland.info
what2buy.info
whisky.info
whistler.info
whitehouse.info
whitepages.info
wicca.info
wiesenhof.info
wildbirds.info
wildcat.info
windsor.info
windsorpubliclibrary.info
wine.info
wisconsindells.info
wm2006.info
woergl.info
wolz.info
woodworking.info
work.info
workandtravelbritain.info
workandtraveluk.info
workingdog.info
workusa.info
world.info
worldofwatches.info
worldwidepants.info
wow.info
wpn.info
wppinvestors.info
wppplc.info
wtn.info
wyethkorea.info
wyethnutrition.info
xrml.info
xtend.info
xtra.info
xxx.info
yachts.info
yaletech.info
yell.info
yellowglen.info
yes24.info
yha.info
yogreats.info
yosips.info
youngandtherestless.info
zabadoo.info
zabix.info
zaditor.info
zaehne.info
zaragoza.info
zauchensee.info
zaxbys.info
zaxbyschicken.info
zaxbys-chicken.info
zellamsee.info
zemaphyte.info
zentralschweiz.info
zevalin.info
zitate.info
zwickau.info

Free Hosting from DreamHost is Here

When we started DreamHost back in 1997, we were in it to make money. Throw up a server, get some customers, collect fat checks, retire at 35. Simple!

Somewhere along the way (I think it was shortly after the fat checks showed up) we developed a conscience and decided – No fat checks!

For us it wasn’t enough to just not be evil – we realized that the only way to really feel good about ourselves was to lose money hand over fist, and just hope that we broke even somewhere along the way…kind of like a non-profit!

We took a multi-pronged approach to Operation Money Loss! We resolved to:

• Provide our employees the most comprehensive health insurance package we could find, free.
• Give away free hosting to non-profit organizations.
• Match customer donations to selected charitable organizations.
• Offset our carbon footprint through the purchase of renewable energy credits and carbon offsets.
• Provide financial support to open-source projects that we use ourselves – Jabber, MySQL, Linux-VServer, and grsecurity to name a few!
• Consistently offer ridiculous coupons that, even now, seem like a bad idea.
• Keep all our company cash in the stock market.

I’d call that a good start, but it’s just not enough.

Today we’re announcing DreamHost Apps, a completely free hosting service for web-based applications.

Well, it’s free for now anyway. We’re flirting with the idea of a $50 yearly fee once it’s out of beta…but if you sign up RIGHT NOW to help us test things out we’ll let you use DreamHost Apps for free for ever!

But wait! If you’re reading this and you’re already a DreamHost customer (and why else would you be reading this), you don’t need to worry.. you can already log in to DreamHost Apps with your same DreamHost Web Panel login!

For you, it’s really just an Ajax-ified way to manage your easy one-clicks and any domain registrations. But, for the hundred or so people out there who’ve read this blog and aren’t DreamHost customers… but do want some free hosting of a WordPress Blog, Drupal Site, Zenphoto Gallery, Mediawiki Wiki, and/or PhpBB Forum (plus Google Apps and Gmail) and aren’t afraid to do a little “beta testing” (and unlike some other web shops, when we say beta, we mean it), today must be your BIRTHDAY!

Right now usage of DreamHost Apps is by invitation only. Lucky for you I like your face and I’ve posted about 100 invitation codes in the comments of this post. If a code doesn’t work, it’s already been redeemed – try another one! When the codes run out, I may even post more!

If you’re a current DreamHost customer you’ve already had access to DreamHost Apps for months! That’s because the exact same functionality (minus the free part) can be found under “Easy Mode” in the “Goodies / One-Click Installs” section of your account control panel. There’s no need for you to sign up for DreamHost Apps because you’re already living the dream!

DreamHost Apps customers do miss out on a few perks that paying DreamHost customers take for granted, though. Apps users can just forget about FTP access, SSH access, email, and even technical support.

DreamHost Apps is provided ‘as-is’, and unfortunately technical support is not part of the ‘is’!

You try providing tech support for a free service and let us know how that works out.

For all intents and purposes, DreamHost Apps are entirely self-contained.

You can install and delete any number of apps through the DreamHost Apps management interface, and you’ll use each Apps’ own web-based interface to make application-level changes. Upgrading to full-featured hosting is just a click away!

DreamHost Apps was assembled over the course of a few months by a small team of four hardworking DreamHost Employees, Pete V, Mike P, Joshua H, and me, Brett D. In fact, this is the first major DreamHost project that Josh Jones hasn’t been personally involved in writing in a very long time, so rest assured that your data is safe and your credit card won’t be charged more than it should be!

If you find any bugs, please post them over in our DreamHost Apps discussion forum.

We’ve been working on this for a while, and we’re kind of tired. We’re just gonna take a quick little nap if you don’t mind…

Please enjoy DreamHost Apps!

Mobile Man

So, what with all this mobile spam and iPhone 3G talk, I spur-of-the-day decided last week to check out Sprint and AT+T’s current wireless data offerings.

Well, mostly spur-of-the-day. It was also because my current laptop is starting to die (just like my last three VAIOs, something comes loose somewhere deep inside its soul, and it begins to just randomly turn off), dammit. So, since I’m pretty sure my next laptop is not going to have a PCMCIA slot to fit my current Verizon EVDO modem, I need to get a USB dongle one anyway.

I also kinda wanted to check out how the coverage and speed was on AT+T’s 3G network, just in case I need to switch over to it come July 11th. I’m sorry T-Mobile, it’s been great, but you still ain’t got no high-speed Internets!

While at Radio Shack I also decided to get the USB modem Sprint makes and try it out. Despite the 2-year contract you need to sign to get $60/month (for only 5GB of transfer now!) and the $150 off the devices, both providers have a trial period so I figured might as well. And, Sprint has their new “Compass 597″ USB modem which is not only itty-bitty, but has a microSD card slot AND GPS.

A few things I noticed right at the store: Sprint’s rebate (at least at Radio Shack) was mail-in and left the price of the modem at $50, whereas AT+T’s was instant and made the modem free. But, as if to make up for that, AT+T did a sleazy thing where they automatically add one month of “free” roadside assistance to your account. You cannot opt out of it, and if you don’t call to cancel they’ll start charging you $3 a month for something probably all your credit cards already provide you for free. Boo!

The Tests

To test each card, I first did a bunch of pings to google.com, and recorded the dropped packets, minimum, maximum, and average ping times. Then, to measure the download speed I downloaded iTunes (56MB) from apple.com, and to measure the upload speed I then uploaded it right back to a DreamHost server via FTP…

At Home

At home, I have pretty close to perfect coverage for Verizon, Sprint, and AT+T. Under these ideal conditions I found:

Wow.. DreamHost is the bandwidth king! Ha!

Ha, well, I just threw in my home cable modem connection and DreamHost’s data center for comparison purposes… AT+T seems to be the current wireless bandwidth king! Sprint looks a bit better in the latency department, but only by a hair, and all three were fast enough to not even notice a lag in a remote shell window.

The Road

For my next test, I rode the bus to work like I do, down the 10 freeway in Los Angeles. Along the way, all three have great coverage; except actually Verizon’s EVDO Rev. A (“Broadband Access”) drops down to 1xRTT (“National Access”) a couple of times. 1xRTT, like everybody’s “2G” technologies, is slowwwwwwwww and suuuuuuuuuuucks. And, annoyingly Verizon’s “VZAccess Manager” software seems (at least for me) to never be able to jump back up to EVDO once it’s dropped down to 1XRTT without manually disconnecting and reconnecting.

At 60 MPH, both Sprint and AT+T’s download speeds were cut in about half, to 60 KB/s and 130 KB/s, respectively. Good scientist that I am, I, um, forgot to do the upload test (or test Verizon), but even if Verizon performed exactly the same as it did at home, it would have still been slower!

Ping times more or less stayed constant.

The Office

Our new office is way up there, and cell phone coverage of all kinds is a bit spotty. What I learned at the office was you do not want to use any of these in a location where they drop back to their “2G” speeds.

At 700 feet, all three of them are in the 10-30% packet loss with 400 ms ping times range… along with download speeds around 10 KB/s and upload speeds around 5 KB/s! I did learn that it seems like both Sprint and AT+T are better at automatically noticing when they can switch back to EVDO/HSDPA than Verizon was, and it’s nice not having to re-connect things yourself.

The Software

Something nobody seems to ever mention in reviews of these wireless data services is what the client software you have to install is like. Well, nobody bar me!

Sprint’s is garbage, Verizon’s is fine, AT+T’s seems good.

Sprint’s software took the longest to start each time (AT+T was a little slower than Verizon, but both were still twice as fast as Sprint), couldn’t minimize to the system tray, and, worst of all, doesn’t work with a VPN! At least not mine! If you connect to the VPN while connected to Sprint, about three seconds later you’re disconnected. Then, if you tried to re-connect, the Sprint software gives an error about being connected to another network device! DEAL BREAKER!

Also, the MicroSD slot is useless for me, and the built-in GPS I guess is interesting but I couldn’t figure out how to make it do anything besides give me my current latitude and longitude.. in degrees! Are you supposed to cut and paste it into google maps or something? Anyway, if I’ve got an Internet connection, I generally don’t need GPS.

As for Verizon’s software, the one thing I always wish they added was a signal-strength meter in the system tray when minimized. AT+T’s software does this, yay! Other than that, both are pretty much fine. They’re simple, don’t get in the way, and work with my VPN. (Er, I think.. I think I need to do a little more testing with AT+T…)

The Device

It’s true, Sprint’s modem is the smallest. However, it also sticks out the furthest from your laptop, like a USB stick. The AT+T one is about as big as one of those old Nokia brick phones, but at least it has a hinge so it is aligned veritcally. The Verizon one actually has the smallest form factor outside the laptop, just because the PCMCIA part is so big. But, my next laptop probably isn’t going to have a PCMCIA port, dammit!

ATT BFF

So, I guess it’s probable pretty clear by now, I’m going to go ahead and stick with the AT+T 3G device. It does have the biggest physical footprint, but I’ve already devised a plan to deal with that! Stay tuned for how that turns out…

I guess I’m sort of glad my laptop started dying, dammit.

(My mobile broadband is now four times faster!)

May de Mayo

Hey, you know what’d be fun on a boring Monday in May? A little role play!

And I’m not talking about 12-sided dice and renaissance faires either, I’m just talking about some simple role reversal.

More specifically, I’m going to complain to you about a web host!

So, about three years ago I was trying out some competitors to, you know, test the waters in case I ever decided I wanted to switch hosts.

I used three places, and they all absolutely stank. I mean, they were horrible. I’m talking worse than us!

Every server I tried with these places was pretty much just not working. Besides that, their support was all universally useless, and their panels were a weird hodge-podge of different systems they’d cobbled together I guess. You also either couldn’t get shell access or had to fax them your driver’s license to enable it?!

The worst thing was, they were all difficult to cancel, and a few even tried to get out of giving me my money back (I was in their “unconditional” guarantee!)

Finally, I decided to splurge (I’d been spending like $7.95/month) and tried a VPS place for a whopping $49 a month!

Well, they were great! I mean, they still had a weird hodge-podge of different panels, and they sure laid stuff out differently than I was used to, but my VPS at least stayed up and I could do anything I wanted.

I never needed to contact support, which was fine with me, and luckily for them, I never really did too much with the account but kept paying them anyway for the last three years (I’m willing to bet a few of you are in this boat as well… thanks!)

Finally

Last month, I finally decided to transfer my little bit of crap I had with them over to a DreamHost PS! When I went to cancel, I decided to check my credit card statement and noticed that for February and March I’d been charged $89 instead of $49?!!

Eh? I searched through all the emails I received from them and the only thing I could find that seemed possibly related was one that mentioned they were upgrading all the features on their VPS, but don’t worry existing users would get them all at the same price!

So, I wrote them a nice email:

Helllooooo….

I just noticed this and that somehow you guys upgraded me without my permission from $49/month to $89/month!

Uh, what happened? It wasn’t my choice.. I did get one email saying resources were going up .. for FREE.

Please refund the extra $80 you’ve charged to my credit card asap.

Also, I’d like to cancel my service as of April 30th, I believe what I’ve already been charged for.

Thanks,
josh!

To which they replied:

Sir,

On 02/17/08 our support team notified you to tell you that your server had run out of resources, and that the only way they could keep your server from staying offline was to upgrade you. They did so for free for one week, and asked you to get back to them to work with them to resolve the issue. They stated that if they didn’t hear from you they would leave you on the higher package level instead of leaving you down completely.

After a week, and a followup reminder sent to this address that the account was being left at Signature level so that you could remain operational, your package was upgraded.

Admittedly this was an atypical situation, but most would probably agree that after not hearing from you the decision to leave you up and operational was preferrable to the decision to simply let your server fail.

As per the contract you agreed to at signup, we do require a 30 day written cancellation notice to close down your account. I can accept this as that notification and close your account 30 days from today, on May 18th. I hope that this helps.

All the best,
Christian

Ha, ha, ha… what?

So, because I was (somehow) crashing my own (private) server, they, without permission from me, started charging me an extra $40 a month, so it wouldn’t crash!

Gee, thanks guys!

I also appreciate it when my cable company notices that I haven’t been enjoying HBO and Showtime and most would probably agree that after not hearing from you the decision to give you all these great movies and original tv series was preferrable to the decision to simply let you suffer with Oxygen and TBS!

But actually, that never happened becuase that would be CRAZY!

I went back to look for this alleged email, and I found it:

Subject: 7 Day Trial upgrade to the Signature package for yourserver.com.

Hi,

This server has reached it’s limit on i-nodes which is number of files on the system.

Below is an output of where most of these I-nodes are being used:

357219 -> /vz/private/1753/root/var/qmail/mailnames/yourserver.com/user/Maildir/cur
457677 -> /vz/private/1753/root/var/qmail/mailnames/yourserver.com/user/Maildir/new

That is roughly 700,000 i-nodes for this mail account. Please clear this mail out and notify us within 7 days so that we can downgrade your account back to the Essential. Otherwise, you will be billed for the Signature package.

Thank you,
Tommy

First off, nice subject! No wonder I didn’t read that email!

Ah, I see.. I had a catch-all at the domain hosted there and it was filled with three years of spam!

It’s besides the point that there’s no mention of inode limits anywhere on their site or tos (I’m not saying who they are because there’s no such thing as bad publicity!), or that I guess their VPS solution has problems with some instances affecting others in certain inode-related areas.

The point is that it is crazy to assume that you may just UPGRADE your customer without hearing back from them, as opposed to say, just DISABLING their account.

I wrote back:

Hi Christian,

Um, actually no, I would have preffered to have the server fail.. I’m sorry I didn’t see those emails, but I did not agree to the upgrade!

Please refund the $80 extra dollars and set my service to cancel on May 18th, after downgrading back to the $49 plan for the rest of the time.

Thanks,
josh!

To which Christian replied:

Josh,

I understand that some people may feel this way. That’s why we gave you free time at Signature level before keeping you there, and the opportunity in successive messages to go ahead and downgrade. We made multiple contact attempts and then provided the service, which you used for two months.
I’ll need to look into the possibility of refund. I’m not sure what the protocol is offhand, so I’ll need to do some digging.

I’ll downgrade your account immediately but if the same problem exists I expect your server to start failing again shortly. If it does, you’ll need to upgrade an I won’t be able to authorize a free upgrade – not with a dispute pending. So make sure that if the server fails and you’re comfortable with that, that if you change your mind you will need to explicitly agree to the new $89 per month rate.

-Christian

HA! Man, at this point I was starting to get bemused and maybe even a little bit angry. Here I am, a guy who totally loved this host, had paid them about $1800 over three years while using virtually no resources, and they’re going to make me fight over $80 at the end?!

Especially when they have no chance in actually keeping it. I happen to know as something of a dabbler in the web host arts myself that it is very very hard for an Internet merchant to win a chargeback dispute with a consumer! My next email brought this up:

Hi Christian,

Please refund the $80 or I’ll have to take it up with my credit card company directly! Yuck!

Thanks,
josh!

Oooh, but he was not intimidated!

Josh,

I will need to take this up with our Controller. My personal opinion is that you were given clear and fair warning of the charges which were not put in place until after a lengthy period in which we provided that upgraded service for you free of charge. We made multiple efforts to contact you and it was your responsibility to keep your contact information updated with us, or in this case keep messages from your provider whitelisted so that we could communicate with you. As you were given plentiful and frequent notice of the upgrade and the consequences for not responding, as you utilized the resources and received benefit from them through multiple billing cycles, and as all of this can be documented, I am certain that we could be victorious contesting a chargeback request. However, as I stated previously this is not my call. What I will do is send this along to our Controller for review, and set your cancellation date to May 18th as promised. Though normally it is not allowed to downgrade and provide cancellation notice at the same time, given the odd circumstances I WILL allow that request to stand, which will save you some funds.

I hope this helps,
Christian

Oooohohhohoohoooo! Well! I hope it helps too! I am so grateful you are now allowing me to “downgrade” to the only plan I ever signed up for!

Anyway, long story short, they said it’d take two weeks to decide, so I contacted American Express and disputed the charges, and then a few days later they credited my $80.

And the moral is, billing issues are the biggest issues for consumers! Why burn up three years of good will at $49/month over $80? Before this, I honestly would have recommended them to people if I hadn’t been their direct competitor! I swear!

People can forgive a lot of bad service/bad product/headaches/incompetence/gross negligence if you just give them back their money. It’s kind of like saying, “the deal is off,” no hard feelings?

It is 100% worth it. Now, when they talk to their friends, they’ll be like “Well, I had a bunch of problems, but in the end they gave me my money back.”

As opposed to me who’ll be like, “They were fine until the end when they stole $80 and refused to return it! I PLEDGE ON MY UNBORN CHILDREN THAT DREAMHOST SHALL CRUSH THEM!”

That’s something that translates across all businesses too, because it’s just a universal way of doing business. It doesn’t matter what industry you’re in, nor what product or service you have, there are good ways of doing business, and there are not so good ways.

And I feel like although we don’t always succeed 100% at the specific details of trying to offer awesome web hosting for super cheap, we are generally successful at running a business that doesn’t lie, cheat, or steal, and always tries its best.

Now, you guys be me and please go write a ton of blog posts I can use the rest of my life.

Thanks!

Good Reminiscing Friday

Well, it was a little over two months ago that we had what I think is pretty safe to call the worst disaster in DreamHost history.

In retrospect to me, it’s kind of funny that the worst disaster didn’t turn out to be due to a security breach, a power outage, a loss of data, or actually anything related to our actual hosting service. I guess it shouldn’t be a surprise that people care a lot more about their bank accounts than they do their websites.

I have realized that billing is the one issue where how important we feel it is is completely at odds with how important you guys feel it is.

What I’m trying to say is, we’ve always been ultra-flexible and lax about how people pay, when people pay, or even about giving credits, discounts, or refunds. We figure, whatever, pay us when you’re ready, we’re not sending anybody to collections or ruining anybody’s credit over some measly bandwidth bill.

What we’ve always tried to focus on more (even though it might not seem like it at times!) is our hosting system’s stability, performance, and features.

I guess I’ve always figured that any billing-related error can be easily undone (worst case scenario, it costs us a little money); there is no lasting harm done to the customer. Whereas having a website or email problem could potentially cause permanent damage to somebody’s business or personal life or something?

Well then, let’s go back and see just how little money a worst case scenario actually costs, shall we?

Credits and refunds to cover people’s bank fees: $52,000.

Sigh, if only everybody kept a big cushion of cash in their account! The main damage that can be caused by a billing snafu is for people who get their account overdrawn, and because of that aren’t able to make a critical purchase, or have a check bounce, causing hassles and incurring bank fees. We offered to pay people any amount their bank charged them for going negative, and in the end that total looks like it came to about $52,000.

Accidental refunds: $170,000.

The worst part of this whole process (for us) turned out to be just after the accidental billing, ironically when we were trying to make things right!

If you recall, our system was not actually charging about 75% of the time we thought it did.. and so we refunded thousands of people who were never charged (but, 75% of the refunds didn’t work either). Well, out of all that, and after two months, there are still about 600 accounts who were credited a total of $170,000 in excess of what we charged them that we haven’t been able to get back from them or their bank.

It is slightly annoying when the same guy who complains to the high heavens when he thought he’d been over-charged $9,000 by accident conveniently disappears when we realize that actually, he’s been over-refunded $9,000 by accident.

Extra credit card fees: $82,000.

Another slightly annoying thing is that credit card processors don’t credit you back any fees when you refund a transaction. Overall, the extra credit card processing we did resulted in extra fees of about $350,000! Fortunately, after a whole lot of groveling and explaining the situation (and waiting two months), we finally got all but $82,000 of that back from First Data, American Express, and Discover Card.

Extra support messages: 20,000.

As you may have surmised, people wrote to us about this thing. About 20,000 times… and it would have been tens of thousands more if we hadn’t put up an “emergency block” against new messages for a little while in there.

How much this extra support actually cost (in terms of your wased time, tech support overtime pay, and other questions taking longer to answer to) is hard to say, but normally we only get about 45,000 messages in a whole month!

Accounts canceled: 1000.

It’s also kind of hard to say how many people actually closed their account because of the incident, but in January we did have about 1,000 more accounts closed than average. Assuming each of those accounts would have stayed for maybe another year, that’s another $120,000 down the Intertubes. It’s crazy… from all our power problems back in 2006, we hardly lost any accounts at all.

Goodwill lost: Priceless.

Yeah, it turns out this whole blog post is nothing more than another clichéd MasterCard commercial parody.

P.S. I guess it’s nice to know, less than two hours away from our biggest data center move ever, that we’ll cause a tiny fraction of the disruption to our customers that one unexpected fat finger did!

P.P.S. Thanks RIM, for scheduling a blackberry outage exactly at the same time. It makes us look better. And, maybe some of our Happy Customers will blame their lack of email tonight on you!

A Strike on Strikes!

Ha, did you think you’d gotten through those stupid strike-themed posts?

Well, apparently you are!

The writer’s strike seems to be finally coming to an end, and I don’t know whether to be happy or sad. On the one hand, it means I can finally get back to writing awesome blog posts. On the other hand, it means I can no longer get away with writing these blog posts… which is bad news because this well of creativity is tapped, my friends.

The sad truth is, I did that entire “billing mistake” thing just so I’d have easy blog fodder for another week.

A Last Hurrah

I guess I’ll just quickly wrap up a bunch of stupid things I was planning on “striking” against but never got around to. I never expected this thing to end and was pacing myself.

Hopefully this strike really is settled or you ain’t going to be seeing any new posts here until at least the first Sunday after the Ecclesiastical Full Moon date after March 20th!

A Strike on Fax Machines!

How in the hell is it 2008 and everybody still uses fax machines?

Give me some widespread e-signature standard already, world!

A Strike on Social Networks

Is it just me, or do social networks only appeal to people who 1. are single 2. have no job or 3. care about what their friends are doing?

Because I, for one, am none of those things.

A Strike on Cell Phones

Why do cell phones still keep any data locally?

When you get a new cell phone, you should just have to log into it, like you do, say, a new email client, and whammo, all your contacts/pictures/text messages/themes/preferences/ETC.. are syncronized with a (non-proprietary) server.

We need IMAP for Phones.

(I lose my cell phone once a month.)

A Strike on Global Warming

Dallas warned me not to post this, but he’s in Thailand (trying to enjoy it while it’s still above the ocean.)

There’s just three things that bother me about global warming.

1. There’s literally no way we can be even reasonably sure about what will happen. There’s just no experiment we can run on our entire planet that we can set up an adequate control for!

2. Even if the earth does get warmer, we can’t really know (again, what would the control be?) all the effects that will have on us until it actually happens. The earth’s climate has changed a lot over the billions of years it’s been around, and yet here we are, over 6 billion strong and fatter than ever!

3. Even if the Earth does warm, and even if it is bad for us, there’s again no way we can possibly verify what actually caused it, nor if there was anything we could have done to prevent it.

I mean, I’m all for clean air and water and not wasting electricity and saving the whales, but isn’t just having clean air and water and more money and whales to ride reason enough?!

And if we want to focus on literally saving the human race as we know it, maybe we should be spending more R+D on stopping near Earth objects!

We know they’re out there, we know they’ve hit Earth before, and we know it’s very bad when they do!

A Strike on Getting Old

I broke my left foot playing basketball when I was 27.

It took about a year to heal, but it’s pretty much been fine since.

Now all of the sudden, 3 years later, everytime I get up after being inactive for a half hour or more my left foot kills!

And that’s the real reason why I don’t worry about global warming in the future … I refuse to get older.

Now, please feel free to hold your own stikes in the comments, before the writers settle!

Five Fun Facts For Friday

This week, I learned another five things I did not know before:

Monday: Although charging a credit card is instantaneous, refunding really does take 3-4-5-6-or-more business days to process.

Tuesday: You can erroneously credit an expired credit card. The money does leave your merchant account.

Wednesday: You can credit a canceled credit card. The money does leave your merchant account.

Thursday: You can credit a debit card tied to a checking account that has been closed for months. The money does leave your merchant account.

Friday: If you charge somebody with an international credit card and then refund their money, by the time the money gets back on, the dollar will have weakened!

Lucky you to learn these things the fun fun-facts way!

The Final Update

Okay, all the people who had still not gotten their refunds was starting to seem a little weird, so after further investigation yesterday, I think we’ve finally got things completely fixed.

It turns out, there was a glitch in our new PayflowPro.pm that resulted in only the first transaction in a single second actually going through! According to Paypal’s site, that PayflowPro.pm should be just a drop-in replacement for the old PFProAPI.pm… and it did seem to be, after changing two lines everything seemed okay.

However, there was one little difference. The new HTTPS interface requires you to pass a unique id for each transaction, and PayflowPro.pm generated that unique id as follows:

my $request_id=substr(time . $data->{TRXTYPE} . $data->{INVNUM},0,32);

The problem was, we never passed in the (optional) “INVNUM” field.. we had an invoice number, but we passed it in as the (also optional) “COMMENT1″. So, our “unique” request_id was pretty much just the current time (plus whether it was a sale or a credit)!

In my testing this didn’t fail, because I didn’t run multiple transactions in the same second. Also, they apparently still return the same old success code we test for when this happens! But when multiple biller services run in parallel on all our controllers, lots of transactions end up happening on the same second.

The Upside

It turns out of the actually closer to $9,600,000 we thought we mistakenly charged, only actually about 1/4 of them ever _actually_ hit people’s credit cards. Our system thought we charged them, and they received an email receipt, but that was where it ended. It turns out we actually billed “only” about $2,100,000 incorrectly.

The Downside

This bug still existed until late last night (around 4am).. so when we ran our super-refunder script, the same thing was happening. Only about 1/4 of the refunds successfully went through. This resulted in the following situation:

About 9/16th of our customers: weren’t actually billed OR actually refunded.
About 1/16th of our customers: were billed AND were refunded.
About 3/16th of our customers: were billed BUT WEREN’T refunded.
About 3/16th of our customers: weren’t billed BUT WERE refunded. (of course, nobody wrote in about it!)

Anyway, last night we fixed the bug (by passing our invoice in as INVNUM) and re-ran another fixer that took an actual log of successful transactions downloaded from our processor and cross-referenced everything with our system. This is what it did:

About 9/16th of our customers: marked their bill and refund as $0 amount.
About 1/16th of our customers: left everything alone.
About 3/16th of our customers: redid the refund.
About 3/16th of our customers: redid the charge.

Double checking now, there were no more of those glitches from before, so everything seems okay.

Once again, all the stuff mentioned in the last post still holds true (you may not see the correction on your statement yet, but if you call your processor they should see it coming, for REALs this time), and once again, I’m very sorry about this whole fiasco.

Sincerely,
Josh Jones

P.S. For people wondering how the “robust and stable” rebiller could have created multiple future charges for the same date… I guess I meant “robust and stable” in regards to normal use over the last ten years. It looks like in this case, when multiple instances were running in parallel on a future date, race conditions allowed some multiple charges for the same period to be created. That too should never happen again now that we don’t allow future bill dates.

The Aftermath

It seems like it’s about time for a follow-up on things from yesterday.

First, I just want to apologize for the regular-style blog post about it yesterday. Hopefully this will be the (picture, bold, and italics-free) blog post many of you would have liked to have seen yesterday.

The current status: we believe to have refunded everybody who was incorrectly billed at this point. This was pretty much finished yesterday at 3pm, but there were a few stragglers who we got today. If you were charged and haven’t seen the refund show up on your credit card / bank statement yet, try calling your bank. Lots of places take a day or two or three or even four to update their statements even if the money’s already back in, but they should see it (by tomorrow for sure) if you call them.

If this/these erroneous charge(s) by us resulted in you having any sort of overdraft/bounced check/nsf fee from your financial institution, please contact our support team from the web panel. We’d just like to request that you include a copy of your statement with the necessary info showing the fees. It can be either a paper statement or a print out of your online statement, or even a screenshot of your online statement and it can be scanned and attached to your support message via our support form or faxed to us at 714-990-2600. If you fax it, please be sure to write your domain name or DreamHost account number on the fax. When we get this, we will put money on your credit card equal to the amount your bank charged you, as well as give you a DreamHost account credit for the same amount on top of that.

Another thing… if you’ve decided because of this fiasco you’d like to cancel hosting with us, we will allow you to get a full credit card refund of any unused portion of your pre-paid contract, even if you’re past our standard 97 day money-back guarantee. To do so, just close your account as normal from our web panel (“Billing > Manage Account” area). Then, after it’s done, write into support and let them know you’d like to get your remaining account credit refunded to your credit card due to the billing snafu of January 15th and we’ll be happy to comply.

Checks to Protect Your Balances

Finally, here are the precautions we’ve now added to our billing system to make sure nothing like this happen ever again:

1. Our biller service will no longer accept a date in the future.
2. This whole time, we did have an option to specify “never automatically bill me more than $X in a day” on our web panel. Of course, not too many people had this set, and why would they have to? Nevertheless, we’ve made a change now that even if you don’t have a specific daily limit set our system will not allow billing you in one day more than 50% more than the most you’ve ever authorized in the past.
3. Our rebiller does an automatic filling-in of old charges when it finds some missing. This should never actually happen anyway, but we’ve added a new check that if it ever finds itself filling in more than 3 missing charges on any account it stops immediately and notifies our financial team.
4. We’ve also added an overall check where if the total number of payments in a day are more than double the average number of payments we’ve gotten on that calendar day for the last seven months it fails and notifies our financial team.

And that’s it.. I hope this puts things more or less behind us. And remember, if you have any specific issues, our support team is always there!

And of course, my sincere apologies for all of this.

Thanks,
Josh Jones

P.S. I apologize for that joke about the triple billing in the newsletter thing too, but you have to admit, it was kind of ironic that I actually did screw up billing less than a week later.

P.P.S. Some of you have attempted to email us directly with information about unresolved issues stemming from this billing fiasco and have received autoresponders telling you you can’t email us directly. That restriction was unintentional has now been removed so please re-send us your email if you have not already contacted us through other means.

Um, Whoops.

Hello.. how’s your morning going?

I hope it’s been a little better than mine.

We had a teensy eensy weensy little billing error last night… my first clue something was up when I saw this morning’s daily billing report (so far): $7,500,000.

It turns out due to my excessively fat fingers, nearly every one of our customers has been seriously over-billed in the last 12 hours.

I bet when you read this part of the last newsletter:

4. New Office!

Another important thing I’ve been doing instead of writing newsletters
is looking out the window of our NEW OFFICE:

http://blog.dreamhost.com/2007/12/21/were-so-high-right-now-you-dont-even-know

If your next web hosting bill from us is mysteriously tripled, now you
know why.

.. you thought it was a joke!

Ha, the joke is on you! I guess. Um, okay, no, not really, I’m sorry.

How on earth could something like this happen?

Let Me Explain

A couple of weeks ago, just around new years, we started beefing up some of our internal “controller” servers. These are the machines that run all of our “behind-the-scenes” services; things from adding a user to registering a domain to configuring apaches to rebilling customers.

I was on a little-bit-too-long vacation, but when I got back, I noticed our daily credit card payments seemed a tad low in the new year.

So, late last week I tried re-running the billing services for all the days back three weeks or so. I knew this was safe, because after 10 years, the one thing you DO get perfect is your billing system. Our biller is pretty bug-free and robust at this point, because we’d be broke and eating bugs if it weren’t.

In fact, it’s so robust you can just run it on any day you want, and it’s safe. It won’t double-charge people and it’ll even automatically find any missing charges and catch everything up to the day you said.

Anyway, I ran it, and things were fine.. and sure enough, it caught a lot of missed payments. I didn’t have time to look into it right then, but I made a note to myself to check up on it on Monday (yesterday) and see if things were fine or still messed up.

Come Monday

Monday came. I checked the reports and sure enough, things were still pretty low. So I looked at the logs for some of the biller services, and I noticed they were only failing on the machines that had been recently upgraded!

That explained why we were getting some money still (since not all the controllers have been upgraded yet), but not all of it.

Anyway, it turned out there was no 64 bit version of the PFProAPI module we use to interface to the credit card transaction server. No big deal, there’s a new module that interfaces with their new and preferred https interface, and it was only a couple of lines of code to change to get us switched over!

So anyway, I made the change, and it worked, and I even tested it, and things were fine!

But then… late last night, I realized: when I re-ran those biller services last week, they must not have fixed everybody then either! It’s just that by running it again I randomly got different people being charged on the working controllers who had been assigned an upgraded (and therefore broken) one before.

So why not just run it all one more time?

Sure, it should be no problem! So I did, manually running the biller (which is normally automatically scheduled) for 2008-01-14, 2008-01-13, 2008-01-12, 2008-01-11, 2008-01-10, 2008-01-09, 2008-01-08, 2008-01-07, 2008-01-06, 2008-01-05, 2008-01-04, 2008-01-03, 2008-01-02, and 2008-01-01.

I probably should have just stopped there. But then I thought better. I thought to myself, “When did we start upgrading these controllers anyway?”

I couldn’t remember. But, since the biller is super-safe and robust anyway, I went ahead and ran it for 2008-12-31, 2008-12-30, 2008-12-29, 2008-12-28, 2008-12-27, 2008-12-26, and 2008-12-25, just for the hell of it.

Notice Anything?

Don’t feel bad if you didn’t. I kind of missed it myself.

THOSE SHOULD HAVE BEEN 2007!!

Heh, uh.. um, er.. my bad?

So what happened?

Well, that super-robust and stable biller did what it was programmed to do, it ran as though today was December 31st, 2008!

And what did it see? Well, it saw a whole lot of accounts (essentially all of them) who for some unknown, mysterious reason hadn’t been charged at all for eleven and a half months!

So off it went, busily through the night, “fixing” everything up for “today”, December 31st, 2008.

Really, it’s sort of amazing this never happened before in the last ten years.

There IS a bug here.

I can imagine the half second or so of thought that sprinted through the programmer’s mind when he was adding the ability to allow you to pass in what day to run the biller as though today is:

Hmm.. well, I could see us POSSIBLY wanting to be able to bill for a future date.

Well guess what… NO! We will NEVER want to rebill as though today were a day that hasn’t happened yet! But instead, somebody along the line (Sage? Me? Somebody else?) figured, “What’s the harm in keeping it flexible?”

About $7,500,000 in harm, that’s what!

The serious part.

The end to this story is that of course, I’m very very sorry, we’re very very sorry, and I’m sure you’re very very sorry this happened. I really am. I understand the sort of problems that an unexpected large charge to your credit card (or worse yet, your debit card) can cause. If the tone of this blog post seemed a little light, I apologize I don’t mean to offend and I realize how serious an issue this is. I’ve been up since 3:50am trying to undo the damage and maybe I’m a little shell-shocked.

A new service is running right now (in parallel on all the controllers) that fixes all those future charges, re-enables your account if it was erroneously suspended, and if your credit card was automatically rebilled, refunds the payment automatically. You don’t have to contact us or your bank, and you’ll get an email when your account is finished fixing up. It’s going to take several more hours to complete. There are (or were, after this incident) a lot of you these days!

If, because of this billing mistake, you somehow incurred some fees from your bank or credit card company, please let us know after tomorrow (today we are just replying to all 10,000+ billing messages with a generic explanation) and we’ll do our best to make it right for you.

And of course, the biller no longer allows dates in the future.

The moral of this story is that “flexibility” is rarely desired in programming! The less a program will accept/the less a program will do/the less options and preferences it has, the more usable it is/the more understandable it is/the more stable it is.

Tough Love

When designing a program, you’ve got to make some tough decisions .. and when you really can’t decide if this is something your users will need someday, err on the side of leaving it out.

Otherwise, your users will someday err on the side of your face.