Lorde: ‘Make no mistake about who I am.’ Photograph: Perou for the Guardian

Lorde talks the way she dances, with the full and free use of all limbs. Sitting outside a hotel in California – where the New Zealand-born musician, real name Ella Yelich-O’Connor, has come to perform at the Coachella festival – she flails, fidgets, and flings out her fingertips for emphasis. She runs her hands into her great mane of hair (Pixar-springy and the colour of burnt toast) and talks with her palms pressed into her head. Discussing her imminent second album, Melodrama, Lorde waggles her legs excitedly. And to help illustrate what it felt like, five or six years ago, to be an ambitious kid in far-flung Auckland, making music that she hoped would one day be heard by a wider world, she stretches out her arms. “There was an element of reaching out. Do you see me? Do you hear me? I’m over here.”

There was a time back there – in 2013 and 2014 especially – when she was everywhere. Her debut single Royals, released when Lorde was still some months shy of 17, propelled her to the top of the charts on three continents (her name, she has said, comes of being “obssessed with aristocracy”). A subsequent album, Pure Heroine, sold over three million copies. She was fast-tracked on to the cover of Rolling Stone, endorsed by an array of musical elders, taken up as a pal by Taylor Swift, and subjected pretty well immediately to weighty predictions about her great and glittering future. David Bowie described Lorde’s warbled, flinty outsider-pop as “like listening to tomorrow”. Lily Allen wrote a lyric that summed up the awe (the terror, no doubt) that Lorde’s super-youth inspired in her peers. “Lorde smells blood,” went Allen’s line. “Kid ain’t one to fuck with.”

Sitting with her now in the Californian hotel, four years on from that enchanted, faintly ridiculous period of her emergence, I make a suggestion to Lorde: that she pretend it all happened to somebody else.

“OK,” she says, pushing her blue-tinted sunglasses further up her nose. Pretend you were an outsider, and that the Lorde phenomenon of 2013 was just something you followed online and on TV. What does it look like?

Lorde, still frowning and remembering back, says, “I think I would have been annoyed by me. This kid... This very self-assured 16-year-old, who seemed to think she knew everything. Writhing around on TV with bad, kind of broken movements. Wearing these weird clothes.”

She wonders if she would have been able to recognise the dual personalities that were in conflict inside the young pop star at the time. “A two-part self. One, this ugly, incorrect, fledgling person-trying-to-be-a-person. And the other, someone who could roll their shoulders back on live television and do it.” Either way, she hopes she would have been sympathetic. “I hope that if I was a removed stranger, I would have thought: ‘Fair enough, she’s just a weird kid, doing her thing.’”

Lorde blinks herself out of the game. “It’s a strange one to think about. I feel very protective of that little person.”

Her essential weirdness has survived the intervening years. She still crackles out her songs in that unique and unaccountable voice, part Édith Piaf, part Gaga, part (what is that?) Prohibition-era gangster. When Lorde dances, it is still with the jagged, unchoreographed exuberance of the nearly-so-drunk-they-must-go-home. During renditions of her graver songs, she still plants herself on stage with her feet apart and describes peculiar marionette movements with her arms.

‘I’m like an old witch when I talk.’ Photograph: Perou for the Guardian

Eccentricity was, and remains, central to her appeal as an artist. It seems perfectly right, in the desert today, that her outfit should admit no surrender to the conditions. It’s 99F (37°C) outside, but Lorde wears an oversized black T-shirt, a complicated tangle of necklaces, Adidas Superstars, and blue jeans so wide in the leg that, when she moves about, she has to be supporting about a kilo of denim. “One of my favourite things,” she says slowly, watching me scribble notes about her outfit and no doubt anticipating this paragraph, “is reading in magazines what I wear from the perspective of a male journalist. Always interesting.”

She is like this: relentlessly alert. In conversation, she plunges into new ideas with the enthusiasm of an undergraduate enrolled on her dream course. She searches for the better-expressed phrase or analogy – which isn’t always necessary, because she’s very good off the cuff. (On the difficulty of maintaining a friendship with someone as famous as Taylor Swift: “It’s like having a friend with very specific allergies. There are certain places you can’t go together. Certain things you can’t do. There are these different sets of considerations within the friendship. It’s like having a friend with an autoimmune disease.”) And though her youth finds an outlet, for instance when she agrees with something and yells, “Totally!” or “Truly!”, she expresses herself for the most part with the fluency of someone at least a decade older.

“I’m like an old witch when I talk,” she says, her Kiwi accent flattening certain vowels. “I do meander. But I always come back around.”

There has been some professional meandering since those months when Royals made her a star. In 2013, the textbook next move would have been an immediate, gleeful emigration to New York or Los Angeles – then studio time with some big-name producers and a rush to get the next album out. But Lorde has never been one for textbooks.

After 2014, she pulled back and slowed the pace of her life. She returned to Auckland and bought a house on the water. Ever since she was 13, when a friend sent a tape of her covering a Duffy song to Universal Records in New Zealand and she was signed to a deal, Lorde had been managed by a Universal A&R man called Scott Maclachlan. Post-Pure Heroine, she changed management – a simple case, she insists, “of wanting to poke my nose into different areas”. All of her significant songs up to this point (the four that formed a 2012 EP, The Love Club, and the 10 of her debut album) were co-written with a talented New Zealand songwriter called Joel Little. But Lorde and Little also parted company. These dissolutions were amicable, Lorde insists. “It was really just a case of going where the work needed me to go. It’s hard to have ill-will with that. I’m the nucleus of what’s happening. And I’m going to take that to a lot of places in my career. I don’t think there’s anything strange about that.”

Through 2015, it became trickier for those outside Lorde’s sphere to follow what she was up to. She is not much of a presence on social media (vowing at one point that “all words [will] go into songs instead of tweets”), and though she performed the odd one-off show, provided vocals for a Disclosure track, and worked on the soundtrack for a Hunger Games film, she faded from frontline view. Having been seen and heard by the world, she wanted to resume, as much as possible, her old life with her friends. “Hugs and dinners,” she describes it. “Trips to the beach. Going to dumb bars in the middle of intersections.”

‘I probably did drink too much. But it was an enlightening time.’ Photograph: Perou for the Guardian

She moved into the waterside house in Auckland on her own. “I come from a big family,” she explains. (Her mother, Sonja Yelich, is a poet and academic, her father, Vic O’Connor, is a civil engineer; Lorde is the second of four siblings.) “We grew up in a house full of love and activity and screaming and crying. When I bought the house, people asked: ‘Isn’t it going to be strange living by yourself?’ And I was like: ‘No! I adore the feeling of being able to spread out my brain.’”

Aware that she couldn’t put off work on her second album for ever, she started assembling notes and ideas on the coffee table in her lounge. This was hastily covered with a towel whenever friends visited: she wasn’t especially proud of the half-formed work. “I was writing about nothing. I wasn’t in the right zone. [These songs] would pass off fine, but they meant nothing to me. I was writing stuff that maybe sounded cool, that were trying to be cool. But they weren’t vividly me.”

Throughout the Royals era, she had been in a relationship with a photographer from Auckland, James Lowe. When that relationship ended some time in 2015, Lorde says she was seized by a frenzy of energy, creative as well as social. She wrote, and went out partying. Newly single and inclined to enjoy herself, “I probably did drink too much. But it was, uh, an enlightening time.” How so? “Obviously break-ups are very complex,” Lorde says. “And, for me, I found that it wasn’t this year of mourning followed by a year of light. I think you can feel a rushing freedom and be elated by that, and then that same day have this sadness hit you and for two hours feel the worst you’ve ever felt. It was a marbled experience.”

By this point, Lorde had a new writing collaborator. She first met the American musician Jack Antonoff, who records as Bleachers, at an industry party in LA. They clicked immediately, bonding over cans of pineapple juice. Antonoff was based in New York, where he lives with his long-term partner, the writer and actor Lena Dunham. Through 2016, Lorde started flying from Auckland to New York for weeks-long writing stints in Antonoff and Dunham’s apartment. Then she’d fly back to Auckland for a bit, to bar-crawl between intersections with her friends.

“I guess there was an element of topping-up to it,” Lorde says of her returns home. “Jack would always say: ‘Every time you come back from New Zealand, you just have all this stuff in your head.’ We would always write a few real good songs as soon as I got in from the airport.”

It is almost as though making her first album was like school, I suggest, while the second has been like leaving that behind to go to university – the first manager and the former collaborator had to go because, however much we like them, we don’t take our teachers with us into life. “Yes,” Lorde says, “I moved out of home. And even though I’m not necessarily going through the exact same things as my peers any more, the trajectory is similar. There’s this element of newfound independence, this total unfamiliarity with your surroundings, of building a life for yourself with a new group of people, and just going from there.”

While making Melodrama, Antonoff and Dunham became especially important to her. “I love that family. That apartment. I’d go there every day, root around in the fridge… There was definitely an element of popping down the hall and Lena being, like, ‘What are you working on today?’ I felt like their child.” At the time, Dunham was polishing the scripts for the final season of her HBO drama Girls. It presents an irresistible image: the high priestess of millennial TV beavering away at her work at one end of the hallway, the wonder child of pop doing the same at the other. Some substantial creative reputations were on the line in that apartment, and it must have felt imperative, during those months, that no laptops got splashed. “Truly!” Lorde laughs. “No. Laptops. Splashed.”

In the summer of 2016, she and Antonoff wrote a Royals-calibre track called Green Light. It charted the contradictory impressions of a person in the aftermath of a break-up, someone waiting for internal permission – a green light – before they can get on and be high-spirited again. They wrote another song, Homemade Dynamite, about just such a phase of high-spiritedness. On stage at Coachella, Lorde says Homemade Dynamite is about that time in an evening when “maybe the drugs are just kicking in”. For a track called Writer In The Dark, she composed a neat central lyric that seemed to summarise a year of ruthless story-gathering back in Auckland. “Bet you rue the day,” Lorde sings at an unnamed lover, “you kissed a writer in the dark.”

Won’t this sort of thing start to have a personal cost, when it’s known that she reaps private experience for public art? Might people not start to keep their distance?

“I think it’s great,” Lorde answers. “That there’s going to be a lyric like this out in the world, and that anyone who is potentially interested in meeting me hears it.” Under the table, she twitches her white Superstars. “It is a difficult thing, the relationship between writing about people and knowing them. Loving them. But it comes with the package. Because, y’know, make no mistake about who I am.” (When she says this, I think of Allen’s line: “Kid ain’t one to fuck with.”)

Her crowd of mates back in New Zealand are a mixed group. “I have my surfer friends. My friends in the city. Girlfriends who are, like, 25, and at art school, or who work in retail, or who are overseas for a year, trying to figure out what’s next. I have friends who are parents.”

When Lorde’s in New York, however, I get the sense that a different support system gathers; a loosely affiliated bunch of influential, young women who motivate each other when necessary. Dunham. “Tay” Swift (who threw Lorde’s 20th birthday party back in November), Florence Welch, of Florence + the Machine, and Tavi Gevinson, editor of the teen website Rookie. Lorde explains that there’s a practical reason famous people should seek out famous people as friends. “It’s an element of wanting to stick together. Because it can be so scary to do this. These friendships, they’re about having a hand put on your shoulder by someone who knows. They’re about hearing, ‘I’ve done it, too.’”

Some of her new music has been quietly influenced by this New York brains trust. Gevinson was played early cuts of songs. Green Light was written after Lorde and Antonoff went to a Florence + the Machine gig in Brooklyn, the collaborators fooling about in the studio the next morning and obsessing over the joyful, “British” way that Welch’s keyboardist had pounded her instrument. In this way, they dreamt up a piano riff – buoyant, trippy, introducing into Green Light a sudden and incongruous change of key – that would come to define the song.

I’m trying to make stuff that looks like the inside of my brain

When the track was released in March, it secured a respectable top-20 place in the American and UK charts. Though it entered mass radio rotation, it didn’t have quite the culture-arresting impact of Royals. I was a fan, but I wasn’t sure if Bowie would have heard this song and thought, “Yes, tomorrow has arrived.” The Swedish composer Max Martin, this century’s patron saint of popcraft, and the man behind hits by Adele, Katy Perry and Taylor Swift, told Lorde that Green Light was an example of “incorrect songwriting” (it “had to do with the melodic math – shortening a part”, she says). I read more complaints from fans that the song was too up-tempo and club-compatible, that it was a departure from the more emaciated, antisocial, “How the hell do we dance to this?” flavour of Lorde’s earlier work.

“It’s a complex song,” she admits. “A lot of people we played it to hated it when they first heard it.” Lorde had worried about her new work being met with less enthusiasm than last time, of course. “When your first record is so well received, at such a fledgling time, you think: ‘God, what if I develop? And what if they don’t like who I develop into? What if they want me to go back to being an embryo?’” She tries to clad herself against criticism by issuing frequent notes-to-self: “That I’m trying to make stuff that looks like the inside of my brain. So how can criticism touch me? As long as I know I’m transcribing my brain faithfully and vividly?”

There has even been a perverse satisfaction in not being able to win over everyone this time. Lorde is like one of those kids who were slightly too clever at school; they find it difficult to stay motivated when tasks are too simple. So if not everyone has yet been won over by the 2017 incarnation of Lorde, that leaves more winning-over to be done as an incentive for the rest of the year. “I’ve talked to other artist friends about a moment in their careers when they’ve felt themselves stop getting better. So it was almost nice, in a way, to come to this second record and realise: ‘Ooooh, I can still be crap at this.’ It meant there was still so much to learn.” She says she’s come to appreciate that a second album, though an appallingly pressurised thing in the moment, should only be a part of a wider career collage.

“I want to be really, really good one day,” Lorde says. Her legs start to twitch again. Her arms begin to flail, making their shapes. “I think I’m pretty good now. I think I’ve made a good start. But I want to be Paul Simon.” She thumps her hands down hard on the table. “I want to be Leonard Cohen.” (Thump.) “I want to be Joni.” (Thump.) “Fucking.” (Thump.) “Mitchell.” (Thump.) “And that takes time.”

Perhaps drawn by the commotion – the percussive hand-thwaps of honest ambition – a publicist from Universal comes by our table. A quick walk-by to check that Lorde’s OK. “Mmm!” says the pop star. When we’re alone again, she whispers: “I love it when they do the walk-by.”

It’s so you don’t have to be the one, I say, to ask the interviewer to bugger off when it’s over. Lorde nods. Suddenly she’s not so much the old witch, or the erudite professional, or the Joni hopeful. She’s 20 years old, a bit silly, inclined to indulge her innate weirdness. She tilts her head and wonders what would happen if the publicist forgot to come back and wrap up our interview. “Because, if it was up to me, I don’t think I’d be able to ask someone to leave. We’d keep talking, and baking in the sun, and then we would die.”

Before that can happen I ask Lorde about something she’d said earlier – about how when she first blasted onto the scene, she felt as if she had “a two-part self”: half doubtful young person, half self-possessed performer, neither side fully reconciled with the other. Is that still the case?

“That gap has started to lessen,” she says. “I’m starting to be able to recognise myself in the work. It’s like there’s more of an overlap?... They’re almost...” For the first time today Lorde can’t summon the precise words she’s after. So she uses her hands instead. “Now it feels more like this,” she says, cupping one hand inside the other. “See?”