To give an example, back in 2002 before we had anything, the most cumbersome task was when we had, say, a spammer, and we might have 100 complaints in the abuse box in the morning. I would take a single email, check the IP address, search the inbox for that single IP address and move all emails containing it to a temp folder. Because new complaints would be pouring in steadily I would shout over my cubicle that “I took 18.104.22.168, don't touch that”. Going through the radius logs, DHCP logs, or whatever was applicable for that case to determine the right customer took around 20 minutes.
Even when we had already shut the customer, complaints of previously sent spam would keep on coming for weeks to come, so we had to remember that 22.214.171.124 had already been handled. Then we had the additional problem of dynamic IP addresses. We had to go through the 20 minute trouble multiple times only to find our that the same customer was behind a lot of the different addresses.
So, the first thing we did was automate the log browsing part by putting DHCP, radius and other logs into a database that our system could use to resolve the customers behind the IP addresses, then we opened up APIs to our customer management systems to get the customer information, choosing credible sources of intel to automate and integrate to our system and creating a webGUI for the handling part. When we had that done, we only had a single case instead of a hundred emails with multiple IP addresses in a messy mailbox. I would have a single row in our handling system saying Customer ID12345 and all those emails would be behind that link.
The next step was to notify the customer and/or shut the connection. Notifying the customer was easy because we already got the customer information automatically. Shutting the customers with a button was a bit trickier, but we started with the connection types where we had the most incidents.
The most tricky part was to get the connections up and running again – how would the customer be able to inform us that the box has been fixed. So we opened our system to our customer helpdesk and gave them the “unshut” button. When we realized that instead of copies of Viagra ads and firewall logs they really needed descriptions on each malware type, links for more information, etc. so they could pass that information to the customer, we started providing them exactly with that.
But this is all ancient history and was finished back in 2003. There's always something new popping up that we need to adjust to. Right now we're struggling with NAT a bit.
By subscribing to our early morning news update, you will receive a daily digest of the latest security news published on Help Net Security.
With over 500 issues so far, reading our newsletter every Monday morning will keep you up-to-date with security risks out there.