But this is all ancient history and was finished back in 2003. There's always something new popping up that we need to adjust to. Right now we're struggling with NAT a bit.
The trickiest part was and still is to detect the malware on our own instead of depending on 3rd parties. The most challenging part on that has been making the solutions compatible with our legal framework. One trick we got into production about four years ago is something I like to call a reversed darknet. With it, we more or less detect 100% of worms and other malware that try to scan the network. In a nutshell, we log all outbound traffic where the destination is not found from the routing table at that time.
When our customer has enough traffic towards unannounced IP space, the evidence is pushed as an incident ticket against that customer. Even in the current IPv4 space, there's still plenty of unannounced space practically behind every /8, so any malware trying to scan, say, random 10 000 addresses per hour will get caught thousands of times during that hour. We even detect malware trying to spread solely within internal networks, because customers tend to route the private IP address space not used by themselves up to their ISP.
How many people work on the team dedicated to fighting infections on the endpoint and what are their roles?
Our CSIRT team consists of five security specialists. None of us dedicate our work solely on customer infections, but rather see handling them as a part of our teams basic activities. Customer incidents contribute to our “other job”, which is to handle all internal IT security incidents, because instead of having to prepare for new threats by reading about them from media, we've usually already seen them targeting a customer of ours. We also detect our own infected workstations with our system, which is a nice additional benefit.
One person is mainly responsible for running the system and bringing new features to production, though he's running plenty of other systems not related to customer abuse as well. Additionally, most of us can code so we all contribute. Handling the customers is done by all of us, but it probably takes less than few hours a day altogether. I mean, when we get information from a credible source that our customer is, say, infected with Zeus, it's just a matter of clicking the “Zeus” -button. That takes a fraction of a second - and even that could be automated if we wanted to, but we've decided against it for now.
We don't have a dedicated helpdesk for these cases. When a customer needs support, the case goes to anyone that happens to answer our tech support number. There's the additional benefit that our helpdesk is more “security aware” than most as they are always reading about the latest threats. When the helpdesk needs help, we have an internal IRC server and a channel dedicated to this.