Design errors in computer systems, i.e. bugs, can cause
inconvenience, loss of data and time, and in some cases catastrophic
damages. One approach for improving design correctness is formal methods:
techniques aiming at mathematically establishing that a piece of hardware
or software satisfies certain properties. For some industrial cases in
which formal methods are utilized, quantified first order formulas in
satisfiability modulo theories (SMT) are useful. This dissertation
presents several novel techniques for solving quantified formulas in SMT.

In general, deciding a quantified formula in SMT is undecidable. The
practical approach for general quantifier reasoning in SMT is
heuristics-based instantiation. This dissertation proposes a number of new
heuristics that solves several challenges. Experimental results show that
with the new heuristics a significant number of more benchmarks can be
solved than before.

When only consider formulas within certain fragments of first order logic,
it is possible to have complete algorithms based on instantiation. We
propose several new fragments, and we prove that formulas in these
fragments can be solved by a complete algorithm based on instantiation.
For satisfiable quantified formulas in these fragments, we show how to
construct the models.

As SMT solvers grow in complexity, the correctness of SMT solvers become
questionable. A practical method to improve the correctness is to check
the proofs from SMT solvers. We propose a proof translator that translates
proofs from SMT solver CVC3 into a trusted solver HOL Light that actually
checks the proofs. Experiments with the proof translator discover a faulty
proof rule in CVC3 and two MIT-labeled quantified benchmarks in the SMT
benchmark library SMT-LIB.