LOS ANGELES (Variety.com) - Jake Gyllenhaal has a razor-thin scar on the palm of his hand. It's a permanent souvenir from the set of "Nightcrawler," the Toronto Film Festival thriller, in which the actor plays a twisted crime paparazzo. On the Los Angeles shoot last fall, director Dan Gilroy was filming Gyllenhaal simmering alone in a house after his character, Lou, suffers a professional setback. "We were in the middle of a scene with a mirror," Gyllenhaal recalls on a recent afternoon. "I hit the mirror." The violent act wasn't in the script, and Gyllenhaal still isn't sure what propelled him to do it. "It was just a choice in that moment that happened," says Gyllenhaal, who accidentally sliced open his hand on a shard of glass.

Doctors at Cedars-Sinai eventually stopped the bleeding and stitched him up, and Gyllenhaal returned to work eight hours later, with his wrist wrapped in gauze. "He's a total trouper," Gilroy says. Gary Michael Walters, the CEO of Bold Films, which financed the $8.25 million indie, remembers seeing the bandage and asking if it was a prop. "They were like, 'No, Jake went so deep, he hurt himself pretty badly,' " Walters says. "It was pretty fucking gnarly."

Gyllenhaal's career trajectory has cut just as deep. Four years ago, the 33-year-old actor made a professional U-turn. He left behind popcorn movies like "Prince of Persia" and "The Day After Tomorrow," and decided he wanted to work only with directors who pushed him out of his comfort zone. He veered toward edgier material like 2012's "End of Watch," in which he played an L.A. cop; 2013's "Prisoners," portraying an obsessed detective; and the head-scratcher "Enemy," as a pair of mysterious twins.

His new direction may have taken some by surprise, but audiences and critics are responding well to his gonzo side. "One of the interesting things about these roles that Jake has taken is that they are darker, but still commercial," says Tom Ortenberg, CEO of Open Road Films, which will release "Nightcrawler" on Halloween following its Sept. 5 premiere at Toronto. The distributor's "End of Watch," which cost only $6 million, made $41 million at the domestic box office, and "Prisoners" grossed $122 million worldwide last year for Warner Bros. "A lot of people see Jake as a nice young Jewish kid from Beverly Hills," says Antoine Fuqua, who cast him as a professional boxer in the upcoming Weinstein Co. release "Southpaw," which had strong buzz out of Cannes based on a teaser clip. "I said, 'Nah!' He's a man now. There's something in his eyes that people don't see. He's got anger in him."

Gyllenhaal grew up in an industry family -- his father, Stephen, is a veteran film and TV director; his mother, Naomi Foner, is an Oscar-nominated screenwriter; and his older sister, Maggie, is, of course, a movie star too -- and he made his acting debut at 11 as Billy Crystal's kid in 1991's "City Slickers." When Gyllenhaal first broke out as an actor, he was best known for dramatic roles in smaller movies like 2001's "Donnie Darko" and 2002's "The Good Girl." He also boldly played the part of Heath Ledger's lover in 2005 hit "Brokeback Mountain," for which he received a supporting actor Oscar nomination. "I never would have thought the movie would have gotten the response that it did," Gyllenhaal says. "It was a life-changing story, and career changing for all of us. It was inexplicable."

Gyllenhaal has returned to his roots with his latest roles, even if it means taking smaller paychecks. "I don't have my own family right now," he says. "I have an opportunity to make those choices." Nick Meyer, CEO of Sierra/Affinity, which sold "Nightcrawler" to international distributors, says Gyllenhaal's star status is the reason the project got made. "He's a significant artist with weight, and he took his role on this movie seriously," Meyer says.

"Nightcrawler" centers on Lou, a fast-talking sleazeball who chases ambulances to catch footage of accident victims on his camcorder. Like Michael Douglas in "Wall Street," he's a self-starter who will do anything to succeed. Gyllenhaal walks the difficult line of making heinous look charming, insisting the film wouldn't work if audiences don't root for its antihero. "I remember getting the script while I was shooting 'Prisoners,' " he says. "It seemed really political to me in a subversive way."

The story touches on the idea of privacy in an Internet age -- and the proliferation of tabloid journalism in even serious news organizations. But Gyllenhaal was most taken by the poetic soliloquies in the script (by Gilroy). He memorized every sentence of dialogue before the 28-day nocturnal shoot through the streets of Los Angeles. "I would always be sad when we wrapped a scene," he says, "because I'd think, 'That's the last time I'm ever going to say the line, and it's so good. It deserves to be said again.' "

Gyllenhaal, who is also a producer on "Nightcrawler," spent three months in pre-production brainstorming with Gilroy on the look of his character. He had the radical idea of making Lou look gaunt, which meant he had to drop 30 pounds from his 180-pound frame. When the film's trailer debuted over the summer, his physical transformation shocked the blogosphere. "I would try to eat as few calories as possible," he says. "I knew if I was hungry that I was in the right spot. Physically, it showed itself, but chemically and mentally, I think it was even a more fascinating journey. It became a struggle for me."

On some nights, he wouldn't eat at all, or he'd only nibble on small pieces of meat, crackers or kale salad. "When you watch the film and see the angularity of his face, the hollow cheeks, the way that his eyes become prominent," Gilroy says, "it's such a haunting look for a night shoot." Gyllenhaal would run 15 miles from his house to the set in the evenings to stay lanky. The inspiration for the fast-talking character came from the animal world. "There was a general sense that he was a coyote," Gyllenhaal says. "I just wanted to live that way."

In the film, Lou takes an interest in his boss, a local TV news producer played by Rene Russo, who happens to be married to Gilroy. The director laughs when asked if those scenes were uncomfortable for him to choreograph. "This was nothing," he says. "My God -- she's kissed every leading man in Hollywood. That never bothered me. I grew up in the business." Gilroy is following a similar path as his older brother, "Bourne Identity" scribe Tony Gilroy, who made his directorial debut late in life, with 2007's "Michael Clayton." "Nightcrawler" is Dan's first go in the director's chair at 55, after writing scripts for films like 2005's "Two for the Money." He says Gyllenhaal topped his wish list to play Lou -- who figures in almost every scene in the film -- because of his willingness to experiment. "There's this tremendous desire to push himself, and create something that's unique," Gilroy says.

Even so, some of Gyllenhaal's methods were unconventional. He went on an undercover scavenging mission to find Lou's 1990s-style wardrobe. "I would drive around L.A. sometimes in rehearsal and take pictures of guys I'd see on the street, and send them to Dan," he says. He went on a ride-along with real paparazzi ambulance-chasers. "There was a car that went over the 101 Freeway down a ridge, and crashed into a wall," he says. "We arrived 30 seconds before the cops got there." Gyllenhaal did similar research with L.A. police when he prepared for "End of Watch," and he playfully sparred with director Denis Villeneuve before settling on his character's haircut and twitch for "Prisoners." Hugh Jackman, his co-star in that film, describes Gyllenhaal as an actor who relies on his instincts. "There's one key scene in the car where I've just gotten out of a liquor store," Jackman says. "Most of that was ad-libbed. We kind of got the scene, and Jake looked and me and said, 'I'm on the verge of something. Come on -- let's go for more.' The next take is the one that's in the movie."

Gyllenhaal admits that playing a dark character like Lou will sometimes sneak into his dreams. "I always have nightmares," he says, before catching himself. "I don't really believe in nightmares. I don't believe the things that scare us are in our dreams. I think they are us communicating with ourselves. Even if I'm scared, I think they are helpful sometimes." Gyllenhaal speaks abstractly at first when asked what drove him to more vulnerable roles. He says he had an epiphany when he turned 30. "You wake up and you say, 'I know who I am. Why am I not able to communicate through my art?' " he says. "I just realized I was sitting in the wrong place."

But he soon opens up about how his parents' divorce in 2009 affected him. "My father got remarried, and my mother moved to New York," says Gyllenhaal, who relocated to New York to be close to her. "My family became a different entity, and I think a more honest one." He wanted to find the same honesty in the characters he played. "My family is so strong right now," Gyllenhaal says. "In a way, that's given me the strength to say trust yourself."

A year after completing "Nightcrawler," Gyllenhaal is still running. On a Sunday afternoon in Pittsburgh, where he's about to wrap "Southpaw," Gyllenhaal is drenched in sweat on a gym treadmill. His arms and torso are now buff. He's spent six months working out six hours a day to build enough muscle to look like a professional boxer. "We made a deal," says his trainer, Terry Claybon. "If you're going to learn how to box, you're going to have to get in there and act like a boxer."

Part of Gyllenhaal's routine includes an eight-mile run, followed by time in the ring with real boxers -- he's trained at gyms in Los Angeles and at the Mayweather Boxing Club in Las Vegas. He even brings along a Variety reporter as he does some of his core exercises: pull-ups (I passed), abs (not so bad) and a squatting drill that involves flipping a 200-pound tire across the floor of a gym (I almost pulled out my back).

Wearing a hoodie and fake tattoos on his arms and neck, Gyllenhaal huff outs fragments of conversation during his sprint. It's a rainy summer day in Pittsburgh, and the sky lights up with electricity. "Thunder is amazing," Gyllenhaal says, making an imploding gesture with his fist. After four miles, he gets into the ring for a different lightning round. The gym thrums with hip-hop music as Gyllenhaal, his fingers wrapped in tape, flicks jabs at his trainer and another boxer. He shot most of the boxing scenes for "Southpaw" in the first two weeks of production earlier in the summer, but he's now trying to prep for one last fight before he hangs up his gloves. "Two people have done this in their lives," Fuqua says. "Robert De Niro in 'Raging Bull' and Denzel Washington in 'The Hurricane.' Those kinds of guys are rare."

After his workout, Gyllenhaal sits on a couch in the back of the gym. Even though he's exhausted, and his eyes look drowsy from just four hours of sleep, he says he feels reinvigorated. Many actors become proficient at a different trade to use as part of a role. "But for me, I learn a skill for what you don't see onscreen," Gyllenhaal says. "A line is a line is a line. But a line is something else when underneath it, there's the experience of the line." He admires actors most who understand what he calls "the art of the instant," and then recites a line from one of of T.S. Eliot's "Four Quartets": "Quick now, here, now, always -- "

He says that when he's working, he gives his director a conventional choice, which he'll follow up with a bolder take for the editing room. "I always think of it as a bucket," he says. "I'm throwing all these choices into a bucket. The director can pull them out and put them up there and cut them up the way he wants for the character." In addition to "Southpaw," Gyllenhaal has "Everest," in which he plays a climber waylaid by a snowstorm, out next year, and he'll soon begin shooting "Demolition," directed by Jean-Marc Vallee ("Dallas Buyers Club"), about an investment banker whose life crumbles when his wife dies tragically. After that, he'll make his Broadway debut in Nick Payne's "Constellations" in December. "I always loved being onstage," says Gyllenhaal, who has appeared in plays on the West End and Off Broadway. Conveniently, his sister will be performing down the street in "The Real Thing."

It's somewhat ironic that the one scene in "Nightcrawler" that's true to life features Gyllenhaal shattering his own image. Gilroy says he was taken by surprise that the star did that on set, but it's become a crucial part of the story. "We didn't plan on the mirror breaking," he says. "It was honest. It was dramatic. It becomes a pivot point in the character." Now Gilroy can't imagine "Nightcrawler" without the scene. "It was an accident we put in the film because it worked so well," he says. But he's recently thought about how Gyllenhaal will always carry the character of Lou with him. "The movie has become a permanent part of his physical being," Gilroy says.

Gyllenhaal isn't fazed by the role that scarred him. "As an actor, you're constantly true to your feelings and the feelings of a character," he says. "The most interesting moments are always the most unexpected. I'm not saying it's a positive thing to get hurt and have to get stitches in your hand. But to me, the scar is about a certain type of commitment."