Turing Test: Passed, using computer-generated poetry

Hi everyone, its been quite awhile since I’ve posted. I have a number of great things in the works, so please stay tuned. In the meantime, I’d like to tell you a story about how I developed another kind of artificial intelligence. This AI can create poetry indistinguishable from real poets. The real Turing Test of this AI was to get it accepted to a literary journal, which was accomplished – this poetry was successfully accepted into a literary journal at a prestigious university. The story is below and the code is at the bottom for those who want to make their own!

What do you think of the following poem?

Orange Light

I conduct myself in a windy manner because I am drunk and enchanted in this field. The oxygen around my head is rabid and filled with orange light like a equinoctial tiger to its flesh. My heart moves violently on this neon ship.

I promise as I were a rotting ghost forced half-open in love in front of the gray agony of the darkness and decaying droplets of acidulous gold.

I reply, only fear and geology are the leaves of belligerence.

I’d do it for the geology and I’d do it for the fear of your response.

If you answered anything other than “That’s not a poem!” to that question, then you probably believe more or less that it is a poem, and by some standards then the artificial intelligence that generated this poem once again passed the Turing Test.

Turing test, what’s that?

The Turing test is a test of the ability of a machine to exhibit intelligence that is indistinguishable from a human. It can exist in many varieties – chat bots, intelligence bots, artist bots. This poetry generator is a very simple type of artificial intelligence, yet it is indistinguishable from any sort of poetry you may encounter from other humans.

There have been generators among the years that are capable of passing Turing tests. In 2005, students wrote a generator for academic CS papers. One of the generated papers was accepted to a conference before conference organizers realized it was a joke. There is also a great post-modernism generator developed by Andrew Bulhak and Josh Larios, which demonstrates the ability to generate cohesive but incoherent post-modernist literature academic papers.

How does the poetry generator work?

This Poetry generator uses a Context-free grammar using the notation of Backus-Naur Form. Context-free grammar systems are a generalized system of formal grammar defined by production rules which allow sentences to be recursively built from smaller phrases. The formalism was developed by Noam Chomskey in the 1950’s.

Essentially this poetry generator works by having the poem dissected into smaller components: stanzas, lines, phrases, then verbs, adjectives, and nouns. When a call to create a poem is made, then it randomly selects components of the poem and recursively generates each of those.

In that case the title is generated as “A ” and then it looks up and selects one of the possible “” words to finish it before it returns “A grape” or something similar.

Does this computer-generated poetry pass the Turing test?

I began by submitting my poetry to your-poetry.com which was a popular poetry website of the time (in 2010). I never conferred that these poems were computer-generated. I published under the name Antikythera. The responses I received were, for the most part, quite flattering. Here are some of the responses I got:

“Many layers to this piece, it demands a second and even third reading. I like the challenge you’ve set out here, well done.”

“I like this! Its like it is straight from the pages of a mystical story,filled with delightful colours and wonderful landscapes.”

“I like you style of just giving the reader a small taste of what you are expressing. It opens up the thought process and gives us something to ponder without fully knowing the subject. Nice!”

“What a wonderful piece of writing. you paint a vivid picture and I love the picture you have painted here. well done.”

“Its like you just sang a beautiful song…”

“you certainly know how to make words jump off the page, well written, well done.”

The negative criticism I received typically found that my poems were incomprehensible or nonsensical, such as the following:

“Sorry to say you have a very strange way of expressing yourself and this poem I have really not understood.”

The overwhelming positive responses compelled me to use a more stringent test of authenticity – seeing whether this computer-generated poetry could get published in a poetry or literary journal.

Final test: Getting computer-generated poetry published

As many poets often do, I tried submitting my poetry to several places. When I submitted these poems to publications I never specified that they were computer generated. I submitted to the Memoir Journal (now defunct) and First Writer Poetry contest but both rejected the poetry without comment. I then turned to local sources and submitted to The Archive, one of the oldest continuously published literary magazines in the United States and the oldest student publication at Duke University.

The Archive agreed to publish one of my poems! The email chain is below. I actually submitted 26 poems (one for each letter of the alphabet) and selected a good one to publish.

Email correspondence

The Archive published a great poem and selected an image of a bulldozer to be accompanied with it. The bulldozer is interesting, I think that they implied that the poem’s central theme is something about the destruction of the environment? The poem and art is shown below:

Page 31 of The Archive – The poem I entered into a literary journal

Page 32 of the Archive, photograph accompanying the poem

In conclusion it seems very possible to create an artificial intelligence to do specialized tasks, like writing poetry, that can sufficiently pass as a human being. Perhaps in the future we will need to question the source of creative materials to determine whether they are indeed human or machine made.

Interested in making your own poem?

The source code for my poetry generator (now implemented in Python instead of Java) is open-source and available. Click the button below to generate your own poem, or check out the source code on Github.

Thanks. It depends on what you think AI is. To me, AI is intelligence exhibited by software/machines. Being intelligent may comprise any/all/some of a computer being able to: 1) reason, 2) store knowledge, 3) plan/navigate, 4) learn, 5) manipulate environment. For example, a good speech recognition AI should be able to know Fourier spectrums of different speech patterns (store knowledge) and figure out which pattern best matches what it hears (reason). Then when it hears something it doesn’t know it should be able to fill in that pattern on the fly (learning).

This poetry generator is not innovative in the way that it reasons (Backus-Naur syntax has been around awhile) so the only unique and minor accomplishment here is the “stored knowledge” of poetic words and verses – the brain. Though its not a big step towards higher-level AI (being able to do #1-5), its still a step. 🙂

Excellent work, and a fun post. As a computational linguist, I find this sort of stuff fascinating.

You might really enjoy a short story by Stanislaw Lem called “The Electronic Bard.” It’s part of a collection of short stories in The Cyberiad. His hero invents a machine that generates poetry, and some of what it produces is pretty hilarious. Of course a human author pretended to write the poetry that the bard produces in this story, but it’s still quite a feat, especially when you appreciate that the poems all had to be translated from Polish to English while retaining their rhyme and meter. 🙂

The assessment of poetry is a faculty requiring the linking together of sometimes disparate concepts into a syncretic whole. Poetry itself, in the days following the general dissolution of strict formal qualities, merely requires a string of moderately grammatical phrases to be declared to be poetry in order to trigger this particular mode of assessment. Therefore it is unsurprising that a poem, submitted as a poem, would be recognized as a poem. This is not an illustration of the success of an algorithm at producing poetry, but of a feature of language and human perception that has been widely recognized by scholars of semiotics and literature since the 1960s. It’s generally expressed as the phrase “the author is dead”, and it means that the interpretive value of any signifying object is always displaced from its origin. The point of authorship literally does not matter, which is why algorithmic art is even a thing at all. There is no “speaker” except for the object – the poem – itself. Declaring these poems to have passed the Turing Test for meeting a standard of aesthetic judgement is like looking at a sunset and saying “that is beautiful”, and concluding that the universe has therefore passed the Turing Test for producing something that we can interpret as beautiful.

[…] while ago, I came across Zack’s post entitled “Turning test: passed using computer generated poetry”, who had apparently generated a poem that was accepted into a literary journal. While I thought […]

[…] Yes, it’s an idea that goes beyond computer-generated music. For example, at the intersection of artificial intelligence and computational linguistics, you’ll find algorithms that can generate poetry. One computer scientist even managed to get a computer-generated poem published in a literary journal. […]

[…] Yes, it’s an idea that goes beyond computer-generated music. For example, at the intersection of artificial intelligence and computational linguistics, you’ll find algorithms that can generate poetry. One computer scientist even managed to get a computer-generated poem published in a literary journal. […]

[…] Yes, it’s an idea that goes beyond computer-generated music. For example, at the intersection of artificial intelligence and computational linguistics, you’ll find algorithms that can generate poetry. One computer scientist even managed to get a computer-generated poem published in a literary journal. […]

I’m guessing you’re a kid, so I won’t be too harsh (It’s also possible that during the last year or so you’ve learnt more about the test). However I need to point out that your experimental methodology is flawed. The Turing test does not work by passing off computer generated texts as having been written by a human.

An observer is presented with two terminals; one controlled by a human, and the other controlled by the artificial intelligence agent that is being tested.

The observer knows that one terminal is a human and the other is a machine. They interact with both and after a period of time are asked to identify which is the human and which is the computer.

This test is repeated many times. If the observer is able to correctly identify the agent too often, it fails the test. On the other hand, if the observer cannot readily tell the difference, the expected strike rate falls to 50% – what you would get if you were guessing each time. If that happens, the machine is indistinguishable from the human, and it passes the test.

What you have done is taken your computer generated poetry and passed it off as your own work. Of course other humans have reacted as though you wrote it – it’s an underlying assumption that you’re taking advantage of.

A better way to proceed would be to write some poetry of your own, and then go back to that poetry site, put up both poems, and then run a poll asking people which of the two they thought had been written by a computer. Rather than asking one observer many times, you would be asking many observers one time.

To be a bit more rigorous you could put up more comparisons – three, or five, or ten, and poll them all.

Then you’ll see how often they can tell apart the computer generated poem from the human one.

The imitation game you are proposing was definitely the original conception by Turing. I think that this is a good experiment. However, I think there should be double-blinding so that the interrogator isn’t biased by knowing that a computer has replaced one of the humans.

I also agree that a quantitative approach is better. Here I’ve taken a very qualitative (very preliminary) approach since it was just for fun. However, this approach I am benefited by double-blinding since none of the evaluators knew it was computer generated.

[…] A home transformed by the lightning the balanced alcoves smother this insatiable earth of a planet, Earth. They attacked it with mechanical horns because they love you, love, in fire and wind. You say, what is the time waiting for in its spring? I tell you it is waiting for your branch that flows, because you are a sweet-smelling diamond architecture that does not know why it grows. (2010 AI, Zackary Scholl) […]