By David Gewirtz
You may have noticed that Connected Photographer has been offline for the past few days. It's back, and what happened makes for quite a story.
About two months ago, I noticed increased traffic on our Web sites -- and the traffic was causing a load on the servers that didn't seem to be right. In particular, I noticed that our email-to-a-friend page was getting accessed repeatedly, at a rate disproportionate to what regular traffic would generate. I reasoned that a spammer was using the page to send junk mail out through our email-to-a-friend interface, and promptly turned that page off.
"A million individual computers hit our servers in the space of a day."
Traffic was one or two accesses a minute from different IP addresses, all over the world. I traced IP addresses to Russia, Brazil, UK, Turkey, Korea, Ukraine, Australia, Canada. There were also a bunch of IPs that wouldn't give up their real locations.
Current incident
Beginning Tuesday night, performance of our Web servers began to degrade. It took me until Thursday morning to determine that the performance degradation was due to an increase in traffic to a particular set of Web pages. This was, in part, because the server was performing so slowly that accessing any information took a very long time.
Eventually, I was able to determine that the email-to-a-friend page (which no longer existed) was being requested for our Connected Photographer Web site. Each request caused a server error, slowing the system down. Unfortunately, there was no way to stop the server errors, since the code that generated them was compiled into the server's kernel.
Yes, I have access to the kernel code and have added features in the past, but I didn't want to muck with code at such a low level while trying to sustain our level of quality service. It would have just taken too much time.
Through the use of a software firewall, I was able to determine that requests to the email-to-a-friend URL were comining into the server at the rate of thousands of requests per second. I configured the software firewall to ban requests to this particular page, and then ban the IP addresses that originated the request.
However, within about ten minutes, the software firewall ceased to function. It had banned more than 10,000 individual IP addresses, (about 1,000 per minute), exceeded its available memory, and pushed the server to 100% utilization.
I tried re-routing and even turning off the DNS pointing to the server. The requests still kept coming in. My guess is that the URL they were requesting was cached, and so the spamming system knew the IP address, ignoring the DNS completely.
