Brown attended the University of Texas at Austin in college. He played cornerback and wore #8. Brown was a reshirt freshman in 2006, and attended college for five years. He had eleven tackles in 2007, eight on special teams in thirteen games. In 2008, Brown recorded 28 tackles and a sack. He was limited by injuries, but returned and regained a starting spot before the Fiesta Bowl. In 2009, Brown had his best year. He recorded 48 tackles (31 solo), had two interceptions, and two sacks. He started just nine games in 2010, and his season ended with a forearm injury against Kansas State University. Brown managed to log in 19 tackles (14 solo), and a sack in his final season with the Texas Longhorns.

Brown was drafted in the 5th Round of the 2011 NFL Draft by the Baltimore Ravens. In the final preseason game, Brown returned an interception for a touchdown. Brown recorded four tackles for Baltimore in 2011. Brown was often used on special teams as he wore #23.