Zillions and zillions: an email fiasco that tore a hole in the Internet

 

I write emails to everyone I know every once in a while. About once a month. I use some tools to get these sent, use unsubscribe links, manage my lists/contacts, and generally try to be a good citizen.

Still, about 15,000 people are in my address book from my personal and professional dealings over the years. How do they get there? I just click “Add to Contacts” in Gmail pretty much. New people I meet… I don’t particularly troll for them.

I was excited to share this awesome update about our company with them on Tuesday:

After crafting my ‘very-rare’ update for hours, I hit send and put my feet up…for a sec.

Tweets began arriving:

 

 

Aaaaaaaaaaack!

People are getting zillions of mails from me?

Logging into the platform that sends our mails, I the following bar chart starting to take shape — what would eventually be about 300k mails sent through our platform.

:

My inbox was looking like those tweet screenshots too. Normally when we send out an update from me we get bounces and autoresponder noise. So I didn’t realize instantly what was happening.

Something crazy was happening. There are 12 Zendesk emails there and the rest might be too. They are all autoreplying to each other and cc’ing both me and the rest of the group. (Go Zendesk! This market share is insane!)

 

First of all I know exactly what mistake I made. There were at least two that were instantly obvious. There are a ton of emails in our user list that aren’t real personal addresses that forward somewhere else. In this case about 20 out of 30,000+, and the kind of thing that’s really hard to scrub out. Number two was also bad: we were trying out a new frontend to Mailgun that we almost never use, which lets you use a normal mail client to compose and send a campaign by emailing it to bcc:listname@whatever. Neither of these would be so bad. But then this happened:

The list name became the FROM name and every autoresponse was then sent back out to the entire list. This is insane for a mailing list. Even old-school computer science department faculty mailing lists from the 1980s admin this type of autoreply. For whatever reason Mailgun didn’t. That was not good. So even though we “bcc” the listname when we send, the mail comes from us, BUT THE reply-to is the list-name. Ouch.

The really insane part comes third as the autoresponders respond again and again to every message from another bot. 90% of the mails coming into our servers were from other company’s senders going back out to other humans. Argh.

So as this happened I was looking at it thinking: “Winner: None”

We get on touch with Zendesk and Mailgun and killed our accounts on the services and deleted private keys and hashes, and had support tech kill servers etc. There was still kind of a queue of instantly generated cannonballs that dribbled out for a bit. Then it stopped after a couple hours. It seems about 2,000 people got 100 emails each. Which is extremely horrible when it’s coming from Victoria’s Secret replying to Barkbox replying to whatever, and I started it all which made me feel like a total idiot (which I am — also Jeff Jarvis asked me to apologize to the “netizens” for “frictioning” them, which I did, then I got more angry tweets for the infelicitous elocution).

ANYWAY, THE POINT IS I RESOLVED TO TRY TO FIX THE UNDERLYING PROBLEM SOMEHOW. 

No matter what it takes! I would start a Kickstarter, I would do a march or something…

So, I started by contacting the Zendesk and Mailgun support people and laying out my best case, sending the screenshots and stats.

A few hours later:

 

!!!!!

Wow!

These two amazing vendors literally realized the fixes needed on the spot and promised to implement them this week/near future!

Go Zendesk and Mailgun!

(I’m still not going to do that dumb hack again…back to safety. Sorry to all that I frictioned.)