They managed to bypass the browser's DEP and ASLR protection with a 0-day heap overflow vulnerability, and then used a separate memory corruption bug to break out of its Protected Mode.
As VUPEN founder Chaouki Bekrar claims, the memory corruption bug they used to do that is one of many they found, but he also admitted that the new IE 10 will be much harder to break into as Microsoft has added new protection mechanisms.
For those wondering exactly what kind of improvements IE 10 will bring, Forbes Higman, Security Program Manager for Internet Explorer, shared details about some of them on the IEBlog.
"Memory protections aim to safely terminate a browser process under attack before a vulnerability can be successfully exploited to run the attacker’s code," he explains. "In many cases, protections allow vendors time to produce and distribute a fix before a vulnerability can be exploited to cause damage."
Some of the memory, compile time and run time mitigation techniques he mentions are already present in previous IE versions, but have been improved over time. Others, such as ForceASLR and High Entropy ASLR (HEASLR) are new.
"Randomizing the location of objects and functions in memory helps prevent an attacker from discovering where they are, which helps prevent a technique called Return Oriented Programming," Higman explains Address Space Layout Randomization.
"This randomization can be thought of as securing the attacker’s payload with a combination lock. If an attacker doesn’t have the combination, they only get one guess. Guessing wrong means the attack will fail and the process will safely be terminated."
But some modules loaded by the browser are not compiled with the /DYNAMICBASE flag and couldn't - until now - be assigned a randomized location when loaded. ForceASLR solves that problem by (as the name says) forcing IE 10 to instruct the operating system to randomize the location of all modules loaded by the browser - even the ones previously mentioned.
The (also) new High Entropy ASLR makes the attackers' lives even more difficult as it takes advantage of the larger address space available on 64bit Windows machines to increase entropy.
This, in turn, increases the number of addresses that can be assigned to a 64bit process, i.e. increase the randomness of the assigned location of objects and functions.
For those worried how the new enhancements to IE will work with Windows 7, another manager offers the explanation: "IE10 on Windows 7 will opt-in to ForceASLR on Windows 7. HEASLR and other ASLR improvements are new to Win8 and only available on that platform. IE10 on Win7 also benefits from the compile-time mitigations mentioned here, and may be configured to run in 64bit mode for enhanced security."