Smith was the first player to make his Major League debut during the 2007 season when he was used as a reliever on Opening Day by the Mets on April 1, 2007. Through his first 16 2/3 innings of his major league career, Joe had a 0.00 ERA. he went 3-2, 3.45 in 54 games, pitching only 44 1/3 innings. His usage pattern was very similar to that of the typical LOOGY, but from the right side. His sidearm delivery makes him extremely tough on right-handed batters, but much more vulnerable to left-handers, and his extreme platoon splits justify using him only in carefully chosen situations. In 2008, he pitched 82 games for the Mets, placing him in a tie for 3rd most in the National League, 4 behind teammate Pedro Feliciano. He went 6-3, 3.55, with 53 strikeouts in 63 1/3 innings that season.

Smith was traded to the Cleveland Indians before the 2009 season, as part of a three-team deal also involving the Seattle Mariners in which 12 players, all of whom were or became major leaguers, changed uniforms. He pitched only 37 times for the Indians in his first season with the team, with no decisions and a 3.44 ERA, then was 2-2, 3.83 in 53 games in 2010 and 3-3 with an excellent 2.01 ERA in 71 appearances in 2011. In 2010, he also pitched 20 times for the AAA Columbus Clippers, with a 1.96 ERA. His total of games pitched in 2011 tied him for second in the American League, 4 behind leader Jose Valverde of the Detroit Tigers. He struck out 45 in 67 innings, continuing his particular usage pattern of pitching only in middle innings and for one inning or less, sprinkled with a few up mop-up appearances when the Indians are trailing. In fact, through his first 5 major league seasons, he had not picked up a single save in spite of making 297 appearances out of the bullpen, although he had 51 games finished.

In 2012, he realized that being a specialist middle reliever with the sort of generic name that customers use to evade scrutiny when registering in a seedy motel does not cut much weight outside the world of baseball geeks. Trying to enter an exclusive bar in the upscale community of Put-in-Bay, OH without identification on May 21st, he tried vainly to convince gatekeepers that he was a local celebrity, leading to a minor scuffle that required police intervention. Ironically, his girlfriend at the time, Cleveland television personality Allie Laforce, a former Miss Teen USA, would probably have been more recognizable to bouncers than the anonymous Smith. In spite of that humbling experience, he had two solid seasons back-to-back in 2012 and 2013. That first year, he went 7-4 with a 2.96 ERA in 72 games, then followed that with a 6-2 record and 2.29 ERA in 2013. That year he finally recorded his first three career saves, being used as closer in place of the struggling Chris Perez in the last week of the season as the Indians were pushing for a playoff spot. They did secure one of the two wild card slots in the 2013 American League, giving Smith the opportunity to make his first postseason appearance, as he pitched two-thirds of an inning in Cleveland's 4-0 loss to the Tampa Bay Rays in the Wild Card Game. It turned out to be his last appearance in an Indians uniform, as on November 23rd, he signed a three-year free agent contract with the Los Angeles Angels of Anaheim, worth $15.75 million.