As Rubin told Malcolm Gladwell on their Broken Record podcast (embedded below), the genesis for Wildflowers came from the sessions that resulted in the bonus tracks for the Heartbreakers' 1993 compilation, "Mary Jane's Last Dance" and the cover of Thunderclap Newman's "Something in the Air." After the meticulous work they did with Jeff Lynne, Rubin freed them in the studio, and it resulted in what Rubin called a more "organic" sound, that "felt more alive and more human," and they decided to continue working together.

It was a particularly "fertile" and "prolific" period in Petty's career, as Rubin noted. Plus, Petty was eager to please his new producer and open to suggestions. "Hope You Never," one of the leftovers that wound up on the She's the One soundtrack, was a particular example.

"We did that very in a straightforward way, kind of almost Jeff Lynne-y drumwise," he said. "Very straight. And we probably played that a bunch of different ways before we decided, 'Oh, we like it this way.' [We] probably played it more like band-style and then it's like, 'This lends itself more to the kind of hypnotic, locked-in sound.' It's a more down-tempo, moody piece, sort of the sarcastic Tom -- 'I hope you never fall in love with somebody like you.'"

Rubin said that they recorded "between 26 and 28 songs," but Warner Bros. felt a single LP would have greater commercial potential. Petty, who repeatedly fought with his former label MCA, agreed with his new bosses and they went about figuring out which songs to include and in what order. Wildflowers was eventually released in November 1994 with 15 tracks. But Petty always hoped to put out the others.

"He thought it was really important because the legacy of the Wildflowers album loomed large in his career," Rubin continued. "And he knew that the second half of Wildflowers was an important statement. His issue was [that] he didn't want to put it out as a new Tom Petty album, 'cause it's not a new Tom Petty album -- it was recorded 25 years ago -- and he didn't want to release it as an old catalog album because he thought it deserved more than being a catalog album. He felt like it was too good to just put out and was sort of looking for the right story where it would have the exposure that it deserved. And he never came up with it."

About two and a half years ago, Petty went to Rubin's house and played the unreleased tracks, which he had since made a few changes to, for Rubin, and the quality "floored" him. "I had, like, a vague memory of them," he said, "but some of them just hit me like, 'Wow, what a great song! How did we ever miss this?'"

But during that listening session, Petty opened up about how he knew they had channeled something magical on those tapes, and could never get it back.

"He told me Wildflowers scares him, because he's not really sure why it's as good as it is," Rubin said. "So it has this, like, haunted feeling for him. ... He loves it, but it's not like he can turn that on again. He couldn't make Wildflowers 2 today. That was the point. The point was, 'I can't do this now. This was then, and it was where I was then and it was a prolific period. This is an extension of that moment.'"