Burst around truth: an excursion in recurrent neural network poetry

Like hunter-gatherers
armed with pens, paper and acute observation skills, six Dunedin writers have
composed poetry collected from the fabric of the city. Come along and fill up
on a diet of found poetry foraged for you by some of our finest.

False startsI’ve been
living in Dunedin just shy of four months, which is an awkward length of time. It’s too
long to be able to feel like a tourist and blithely, touristically, make poetry
without feeling every misstep. But it's too short to know exactly what I want to
say, even if I’m to say it with other people’s words.

Point and case, my first misstep when I knew I had to find some poetry. The idea came to me
when I first visited Forsyth Barr Stadium (for the Warriors v Bulldogs, alas,
rather than a Highlanders game – spot the out of towner!). I looked at the names
of the stands. There was the Speights Stand (of course), the Mitre 10 Mega
Stand (fitting as the huge orange shed is visible from pretty much every
lookout), the Otago Daily Times stand, which is only there when it’s needed (eg
for All Blacks games, super rugby finals, not NRL games), which seemed to say
something about traditional media and local media. Very quickly I thought of a
range of other temporary, interstitial stands that could be inserted in the
seating plan of Dunedin’s stadium to give a more meaningful map of what it
means to live in Dunedin in 2017. Emersons needed a stand beside Speights, for
starters. And what were the local businesses displaced by, or put out of
business by, Mitre10 Mega?

But the more I looked at my seating plan, the more I
realised, I didn’t know enough.

That, and making a seating plan poetic seemed to pull the
thing in less meaningful, but funnier directions (The Lemonade Stand stand, The
Stand By Your Man stand).

Verdict: abandon!

Moving on

My next and more enduring projectwas inspired by the screens in Toitu (Otago Settlers Museum) that
played Dunedin Sound music videos. I sat there for ages, suddenly recognising streets and beaches of temporary hometown.

The music was something I could identify with, even before I’d
moved down here. In my first week as the Burns Fellow I sat in Poppa’s Pizza
and stared in awe at the orientation posters from the late eighties and early
nineties: The Chills, Straitjacket Fits, The Bats, The 3Ds... It was one of those I just wasn't meant for these times times.

Back in 2008, while questing for a million words, I
generated a lot of poetry with the help of Google Translate. I'd take song lyrics from bands
like The Tragically Hip, Procul Harum and Monster Magnet, translate into (for
example) Russian, then Indonesian, then Chinese (Simplified), then English,
then Italian, then Latin, then English, then Welsh, then Hindi, then English.
Each time the text slipped back into English, it’d scavenge any interesting
collision of words and keep going until I’d built up a stockpile of phrases to
then construct something new.

If you’re interested, here are some examples of finished poems that are
available online:

Himalyan
White(started life as ‘Nights
in White Satin’ and ‘Whiter Shade of Pale’)

Glen
Coe (started life as ‘Whipping Post’ by The Allman Brothers Band)

(I played around with other things too. A series of warped Shakespearean soliloquies. Translating poems from languages I didn’t know without any help — analog or digital (‘The Bumblebee’ came from my ignorant ‘translation’ from the Turkish of Mustafa Ziyalan’s poem ‘Ad Bulamadiğim’).

This is all to say I have an interest in alternative means of text generation, the place of the writer in the process, and the relationship between song lyrics as text and poetry — so it was not a very big leap to make to decide to take lyrics from Dunedin Sound bands and generate something new from this corpus.

The final piece of the puzzle is my interest in Recurrent Neural Networks (RNN). I guess I have a soft spot for AI that can’t quite pass the Turing Test. I can listen to ChatBots talking to each other for hours. And when I read a list of recipe titles generated by an RNN that had been trained on cookbooks, I seriously lost my shit.

Match-making

So, I wanted to learn more about RNNs, and after a bit of
reading up I knew I couldn’t do it alone. One of the great things about being
part of a university, however temporary, is the possibility of collaboration
just seems that much more achievable. I reached out to Anthony Robins in the
University of Otago’s Computer Science Faculty, because his name appeared most
often with respect to neural networks. We met up for coffee and he seemed
excited by my project but was in the throes of editing a book and didn’t have
time to help. He sent an email to five of his colleagues and after a few
nibbles I met up for another coffee with Lech Szymanski.

Lech had developed a basic RNN for teaching purposes last
year, fed it Pride and Prejudice and it was able to spit out Austeny prose with
the right prompts. Lech offered to set up a web interface on one of the
university’s servers so I could access this RNN, upload my own corpus, select
how to train it (more on this in a second), then play around with the outputs.

And in a matter of days I had the keys (read: password) to
this amazing text generator. In terms of what can be achieved in language
generation, it’s still highly basic. But it has proved an amazing sandpit in
which to cut my teeth (and to generate some poetry for tonight’s event!).

Method statement

Step 1:Collect the text (corpus) you’re going to
feed into the RNN.

For me this involved about 10 hours of scouring the internet
for all available song lyrics by Dunedin Sound bands up to about 1995. This was
quite hit or miss. I got quite a lot of Chills, Bats, Verlaines, Clean and
Straightjacket Fits. Others (eg The Gordons, Look Blue Go Purple) were much
harder to come by, a reflection of the fact these lyrics sites are heavily
weighted towards contemporary puff and hosted overseas. All up, I got about 300
songs (45,000 words). I’d hoped to have about double that, and maybe someone
will read this and help me out. But I didn’t have time to transcribe lyrics
from some of the great bands that were unfairly under-represented in my corpus,
and thems the breaks.

A note about the size: the RNN basically learns to speak
English from the text you upload and train it on. If a word isn’t in the
corpus, it won’t know it. If a word is only in there once, the model will only
know one what that word can occur. There’s an element of randomisation in word
selection, but to get unique but genuine sounding outputs requires a huge and
varied corpus and a really well calibrated model. It’s fair to say my first
attempts were going to be sitting at the more cursory end of the spectrum.

Step 2: Upload the
.txt file containing these lyrics to the RNN.

Step 3: Specify the
training parameters for the new model:

Number of epochs – how many times the network
goes over the entire training data during its training. The more epochs, the
better at speaking this particular brand of English it’s going to be.

Number of hidden neurons – the size of the
network. More neurons, more capacity to store patterns, but the longer it’ll
take to train.

Predict from a sequence of X words – the network
takes a fixed number of words in order to predict the next one.

For my first model, I went with the default (500 epochs, 128
hidden neurons, sequence of 5 words)

Step 4: Start training

And…

Waiting for a
command...

Received command
to train...

Error! Couldn't
find dunedin...

Waiting for a
command...

Received command
to train...

Error! Couldn't
find dunedin...

Waiting for a
command...

Received command
to train...

Error! Couldn't
find dunedin...

Okay, so
there were a few hiccups to start with (no spaces in .txt filenames, please).

When we got
it going, the first model took around 12 hours to train, which meant I had a
new toy to play with when I got into the office the next morning.

Generating text

Once you’ve
trained your model, you need to give it some starting text that it will then
follow on from, the number of word to generate, and the number of top most
likely words to choose at random from.

Version 1

This first
model, with 500 epochs of training and a fairly grunty 128 neurons, was so well
schooled in the 300-ish song lyrics, that if you set the number of top most
likely words to choose from at random around 5 or lower, it would very quickly
start spitting out verbatim Verlaines lyrics or just saying soft bomb soft bomb soft bomb (Thanks,
The Clean!) until it had dutifully fulfilled the order of words it needed to
generate.

By turning
up the randomness, it got a bit more interesting.

Very early
on it became clear this exercise, while it might not produce the greatest
poetry, it’s a really good way for humans (or me, at least) to synthesise a
decent amount of text and look for patterns.

Because I
was hoping to create a ‘found’ poem or two that reflected back on Dunedin, I
was on the lookout for items of local colour in the generated text. But unlike
the music videos, and that guitar
jangle, there wasn’t a lot of quintessential Dunedin nounage in the corpus. ‘Dunedin’
never once came up. No suburbs (I don’t think). No ‘rugby’ or ‘Speights’ or ‘Mitre
10’. Which is telling, perhaps, in that this was counter-culture, even for
Dunedin. The music videos on George St or St Clair beach weren’t so much
parochial proclamations as the result of fiscal imperatives.

As I
continued to tinker, I found the most glee in using Dunedin-themed seed words (eg
‘Welcome to Dunedin’, ‘Take
Your Place In the World’), as a kind of antagonistic gesture to drawn these
fiercely geo-generic lyrics into the service of overt local commentary.

But first I
needed to tinker with the model.

Version 2I got the
RNN to retrain on the same corpus but only for 200 epochs, using 32 neurons and
5 word predictive sequence. The hope was that this would provide a model that
would be prone to more random combinations of words, without producing entire
gibberish.

And it did
seem to work better, for my purposes at least.

In amongst
the noise…

There's wonder whetherÊ, sweet

(Where bark enough

But(Why?,It’s hipper on once awake

…there were moments of almost
poetry:

I'm burst around truth

Tremble cos it's time for love

We want a pain that grows

But blind away our worries

Nah surprise, mindless idiot

Poetry? Or
a direct address to the mindless idiot fiddling with the knobs?

To generate
proto-poems, I’d feed the model the same phrase three or four times, tell it to
generate 40 or 50 words at different degrees of randomness, and tape them
together as verses.

I’d then
cut and paste the verbatim output and do a light edit, getting rid of the
nonsense characters, adding spaces to awkwardly conjoined words, and fixing
tenses.

Then I’d
cut and paste that text and do a heavier edit, removing errant words or moving
them within a line, changing pronouns to maintain the unity of the verse.

If it was
going really well, I might do an even heavier edit, which would leave just the
cream of the model’s poetic output.

Anyone who
spends any time tinkering with methods of text generation for creative means is
basically willing themselves into a state of extreme credulity. You have to
take what you’re given and assign it meaning when the machine had no such
intention. For me, there were two impulses, at times complimentary, at times
competing. The text from the model was:

The disembodied voice of Dunedin
itself; and/or

Autobiographical (i.e. the words
Craig really wants to say, but didn’t know it until he pushed “Generate”

An example of
a ‘poem’ using this second model with 3 rounds of editing from me:

Take your place in the world

Take your place in the world

Ride on the air, on sorrow,

through the world and all you’re

Keeping back

You think I could pass the living touch to you?

Done.

A plundering drive inside the water

Look at this head,

it's really my cactus upon me

Take your place in the world, cross the blue

And accept a light

tomorrow they came from today

take me up, love that's more than money,

just go, baby, without my sense.

Version 3Ideally, I
wouldn’t have to resort to anything but a light edit and let the model talk for
itself. The biggest drawback seemed to be the limited corpus. If you could only
learn to speak from 300 song lyrics, you’d struggle to be articulate no matter
how many different ways you studied them.

But I’d
exhausted the online well and wasn’t about to transcribe lyrics for 30 hours
just to double the size of my corpus. So, in the spirit of experimentation, I
added the Beatles songbook (30,000 words) to my Dunedin Sound lyrics. I chose
the Beatles because they were a known quantity, the lyrics were readily available and have relatively little
local flavor, and adding some more Lennon and McCartney songwriting DNA into
the model could only be a good thing if the aim was to create more recognisable
song lyric text.

And it
worked pretty much as I’d hoped. There were hardly ever words that were
obviously The Beatles' (begone Pam and Maxwell) and the dark and brooding nature of the
previous models was still retained.

So I
started feeding the model seeds of Dunedin and in return it gave me the raw
materials for Dunedin-y poems. I still needed to edit, and the more I edited
the better they became as poems (especially ones with the intent of reading
aloud) — but of course I’d think that about my own editing.

I don’t
want to do much more analysis of the poems, they should speak for themselves,
but I will add that, knowing what you now know about their genesis, it opens up a range of reading approachs that wouldn't normally exist, like:

Looking for identifiable words or phrases from Dunedin Sound or (Beatles) songs

Looking for ‘the joins’ – where I might have wielded the heaviest hand in the editing for structural reasons

Reading my palms – the search for autobiographical elements despite my efforts to create a distance between me and the text

Dobbing Dunedin in – the search for the truth of Dunedin from the (mangled) mouths of its jean-clad prophets

So to
close, I’ll leave you with two poems (I’m still not sure which I’ll read tonight
– I’m the 6th of 7 poets, so I’ll pick which fits best on the fly),
both generated using the third model (Dunedin Sound + Beatles lyrics, 300
epochs, 32 neurons, 10 word sequence). The titles are the seed text I used for
each generative act.

Dunedin

Dunedin,

you call you with your old trees

that time cannot bite:

Welcome to Think Small.

Dunedin the one, it's just,

How was I doing? Was I right?

You’re inside me and you've got a summer!

Rivers will rise,

They'll question you:

North Dunedin, I spend my life leaving,

trapped, refrain.

Another Lady hard from any year, any horizon.

The artist using her

kaleidoscope

knows to touch the heads of my bedroom

for silence.

Yeah, she doesn't.

Dunedin,

If I can’t get a better wave of submarine aches,

Sail and sunset and back Monday,

that distant other lover’s boat slips in two:

you we I me

I can guess what they're about to say,

It's:

Dunedin with the night darkness,

should overtime bring you round

all the west commences brown

easy-pleaser sling stood still

There's the little madman’s papers-Yes

He put on your love song,

Pyromaniac,

Take in darkness what we seem,

a bad danger.

Dunedin with a little hand

everybody’s ticked.

She was bringing us apart.

Dunedin, don't prove ugly!

Welcome to Dunedin

Welcome to Dunedin at dark.

You don't force dreams.

People see only the thread that ever dreamed

And, for you,

a star.

Welcome to Dunedin.

Ain't they run out of naked sorrow gloom?

Now you'll stuff shaking bright eyes where

the wind he reaps soft passes

of a blue town, and

Baby, you're dead.

What? Please.

Now gone in confuséd linger.

Welcome to Dunedin!

[Chorus: dance to question the mind]

Welcome to Dunedin,

The impairing ocean is now blue.

Say you would never never do the lie by land.

The message turned the pot satin.

A seized day hauled off.

We belong that tell the prayer.

Welcome to Dunedin,

Old hallows joyride and no-one

gazing beside me come snow.

Trust that I'll get beauty,

That any chaos will.

-->

Roll credits

Thanks very much to Lech Syzmanski and Anthony Robins. Thanks also to Clare Mabey & the team at Dunedin Writers Festival for putting me on the Found Poetry session and starting me off on this path. I have a lot of links to other interesting stuff people are doing with neural networks, but I'll save that for another time.