The 5'5" actor was born in Houston, Texas to parents Sharon and Daniel. He attended Harvard University and graduated in 2007. While there, he was president of Hasty Pudding Theatricals, a student society known for its burlesque cross-dressing musicals.

He first stepped in front of the camera for a 2009 short film called The Condom Guy. He followed that up with three other shorts before he won a part on the comedy series Glory Daze. From 2010-11, Josh played Zack Miller in 10 episodes of the show, which profiled the life of college friends in 1980s Wisconsin. Also in 2011, he filled the role of Dale on an episode of the highly successful, Golden Globe- and Emmy-winning comedy series The Big Bang Theory. He would reprise his role in a 2013 episode.

Josh transitioned from portraying Lyle to playing Kyle, an assistant, on the comedy series Maron. Between 2013 and 2015, he acted in 12 episodes of the show, which he told Paste Magazine in May 2016 is his "absolute favorite thing in the world." In 2014, he acted in one episode of the Patricia Heaton comedy series The Middle.

In 2015, Josh scored his most lucrative part to date. He began playing Nelson “Big Head” Bighetti on the Golden Globe-nominated and Emmy-winning comedy series Silicon Valley. He has worked on 28 episodes and told Yahoo! in 2016 that he never predicted the show's success. He said, "This has all been a happy, wonderful surprise... it's been an exciting and wonderful and wild great ride. And I'm just lucky to be a part of it all."

Josh returned to film with the 2015 quirky dramedy Welcome to Happiness, alongside Keegan-Michael Key and Frances Conroy. He played Ripley in the movie and spoke to Paste Magazine about the project in 2016. He said, "The script was just so very unique. I'd really never read anything quite like it." He added that since he was a child, he's been fascinated with stories that mix fantastical elements and components of realism. For him, Welcome to Happiness combined both. He said the film features "people who are very real, and very grounded and have real problems and real sadnesses, but there's this little spark of magic."

The same year, he also acted in the musical-comedy The Rumperbutts and in an episode of Shonda Rhimes' Emmy-winning drama series Scandal with Kerry Washington. He played Gavin Price, a man accused of murdering his father.

Josh, who told the publication The Roosevelts that he's motivated by abject fear and would have become a doctor if he wasn't an actor, firmly believes in finding a director he trusts and can collaborate with on a consistent basis. Discussing his philosophy with Paste Magazine, he said, "I'd like to think of an acting career as blind betting on a horse race. You gotta pick a director early and stick with them and hope that they'll carry you to the finish line for the rest of your career."

The actor also confessed to The Roosevelts that while he was growing up, his favorite TV shows were The Simpsons and Seinfeld, and that his favorite meal is macaroni and fish sticks.