General relativity has passed all Solar System experiments and
neutron star based tests, such as binary pulsar observations, with
flying colors. A more exotic arena for testing general relativity is
in systems that contain one or more black holes. Black holes are the
most compact objects in the universe, providing probes of the
strongest-possible gravitational fields. We are motivated to study
strong-field gravity since many theories give large deviations from
general relativity only at large field strengths, while recovering
the weak-field behavior. In this article, we review how one can
probe general relativity and various alternative theories of gravity
by using electromagnetic waves from a black hole with an accretion
disk, and gravitational waves from black hole binaries. We first
review model-independent ways of testing gravity with
electromagnetic/gravitational waves from a black hole system. We
then focus on selected examples of theories that extend general
relativity in rather simple ways. Some important characteristics of
general relativity include (i) only tensor gravitational degrees of
freedom, (ii) the graviton is massless, (iii) no quadratic or higher
curvatures in the action, and (iv) the theory is 4
dimensional. Altering a characteristic leads to a different
extension of general relativity: (i) scalar-tensor theories, (ii)
massive gravity theories, (iii) quadratic gravity, and (iv) theories
with large extra dimensions. Within each theory, we describe black
hole solutions, their properties, and current and projected
constraints on each theory using black hole-based tests of
gravity. We close this review by listing some of the open problems
in model-independent tests and within each specific theory.