Josh was All-State in high school. In his first season with Central Michigan University, in 2005, Collmenter went 7-1 with a 2.70 ERA in 16 games (10 starts). Louisville Slugger named him a Freshman All-American. He was third in the Mid-American Conference in ERA. The following season, Collmenter went 8-5 with a 3.41 ERA in 15 games (14 starts). That summer, he had a 1-1, 2.42 record for the Hyannis Mets of the Cape Cod League. He went 9-4 with a 1.93 ERA in 15 starts in 2007. He led the Mid-American Conference in wins, strikeouts (117) and ERA for a pitching Triple Crown. He was named the MAC Pitcher of the Year. He was 15th in NCAA Division I in ERA. He set a school record with 288 career strikeouts. In the 2007 amateur draft, he was taken by the Diamondbacks in the 15th round. He was signed by scout Matt Haas and began his professional career that season.

Collmenter played for the Yakima Bears in 2007, going 6-3 with a 2.71 ERA in 14 games (12 starts). He led the Northwest League in ERA, .26 ahead of Bruce Billings, but did not win All-Star honors. In 2008, he pitched for the South Bend Silver Hawks, going 12-8 with a 3.41 ERA in 27 starts. The following season, Collmenter went 8-10 with a 4.15 ERA in 27 starts for the Visalia Rawhide. As well, he struck out 152 batters in 145 1/3 innings that season. He led Diamondbacks farmhands in strikeouts and opponent average. In the high-scoring California League, he was 9th in ERA and led in whiffs. Baseball America ranked him as owning the best changeup in the league. He split the 2010 season between the Rawhide, Mobile Bay Bears and Reno Aces, going a combined 14-8 with a 3.38 ERA in 25 starts.

Collmenter joined the Diamondbacks' staff two weeks into the 2011 season, pitching out of the bullpen at first. He made his first major league start on May 14, facing the Los Angeles Dodgers and held them scoreless for six innings to get credit for a 1-0 win; his opponent that night was Chad Billingsley, who gave up only one hit - a double by Stephen Drew - but Collmenter's teammates managed to turn that hit into a run and he made the short lead stand. In fact he did not give up a single run in his first 15 innings as a major league starter. He finished the season with a record of 10-10, and an excellent ERA at 3.38. He pitched 154 1/3 innings, with a K/W ration of 100/28. He started Game 3 of the NLDS against the Milwaukee Brewers with the D-Backs down 2 games to 1, but he pitched a very solid game, giving up a single run in 7 innings on his way to an 8-1 win. He finished 5th in voting for the 2011 National League Rookie of the Year Award, trailing Craig Kimbrel, Freddie Freeman, Vance Worley and Wilson Ramos.

Collmenter got off to a rough start in 2012. After four appearances as a starter, he was 0-2 with a 9.82 ERA. He had given up a National League-leading 20 earned runs and 6 homers at that point, and ceded his spot in the starting rotation to rookie Patrick Corbin on April 30th. He managed to turn things around, however, ending the season with an ERA of 3.69 in 28 games and a record of 5-3. He pitched 90 1/3 innings, making 11 starts. In 2013, he pitched exclusively in relief, making 49 appearances with a record of 5-5. His 92 innings in relief were the highest total in the National League that season. He went back to starting on April 14, 2014, after making his first four appearances of the year out of the bullpen; in the last of these, he recorded his first career save with a four-inning outing in a 7-3 win over the San Francisco Giants on April 9th. His first start in almost two years was no great shakes, as he gave up 3 runs in 4 innings against the New York Mets and was charged with a 7-3 loss. He lost again the next time out, but managed to keep his spot in the rotation and began to pitch better. That culminated on May 29th when he pitched his first career complete game and shutout in defeating the Cincinnati Reds, 4-0. He faced the minimum 27 batters that day, in spite of giving up three hits, as all three Reds baserunners were erased on double plays, including Brayan Pena who was thrown out at third trying to advance on a fly out after hitting a lead-off double in the 3rd. He was the first D-Backs pitchers to face the minimum number of batters since Randy Johnson had pitched a perfect game on May 18, 2004, and he needed only 94 pitches to complete the gem. He improved to 4-2, 3.34 on the year.

Collmenter has a very unorthodox pitching motion. He throws completely over the top. Indeed, as he winds up, it seems that he is trying to throw the ball straight upwards, but his wrist breaks at the last moment, and he does throw towards the plate, with surprisingly good control. He doesn't throw hard, topping around 87 mph, but still has an excellent change-up.