Exploiting Trade-offs in Symbolic Execution for Identifying Security Bugs

Over the past 20 years, our society has become increasingly dependent on software. Today, we rely on software for our financial transactions, our work, our communications, even our social contacts. A single software aw is enough to cause irreparable damage, and as our reliance on software increases, so does our need for developing systematic techniques that check the software we use for critical vulnerabilities. In this dissertation, we investigate trade-os in symbolic execution for identifying security-critical bugs. In the first part of the dissertation, we present symbolic execution systems capable of demonstrating control ow hijacks on real-world programs both at the source, and binary level. By exploiting specific trade-os in symbolic execution, such as state pruning and careful state modeling, we show how to increase the efficacy of vanilla symbolic execution in identifying exploitable bugs. In the second part of the dissertation, we investigate veritesting, a symbolic execution technique for exploiting the trade-o between formula expressivity and number of program states. Our experiments on a large number of programs, show that veritesting finds more bugs, obtains higher node and path coverage, and can cover a fixed number of paths faster when compared to vanilla symbolic execution. Using veritesting, we have checked more than 33,248 Debian binaries, and found more than 11,687 bugs. Our results have had real world impact with 202 bug fixes already present in the latest version of Debian.