MozCon 2018 is nearing and it’s almost time to brush off that microphone. If speaking at MozCon is your dream, then we have the opportunity of a lifetime for you! Pitch us your topic and you may be selected to join us as one of our six community speakers.

What is a community speaker, you ask? MozCon sessions are by invite only, meaning we reach out to select speakers for the majority of our talks. But every year we reserve six 15-minute community speaking slots, where we invite anyone in the SEO community to pitch to present at MozCon. These sessions are both an attendee favorite and a fabulous opportunity to break into the speaking circuit.

Katie Cunningham, one of last year’s community speakers, on stage at MozCon 2017

Interested in pitching your own idea? Read on for everything you need to know:

If you submit a pitch, you’ll hear back from us regardless of your acceptance status.

What you’ll get as a community speaker:

15 minutes on the MozCon stage for a keynote-style presentation, followed by 5 minutes of Q&A

A free ticket to MozCon (we can issue a refund or transfer if you have already purchased yours)

Four nights of lodging covered by Moz at our partner hotel

Reimbursement for your travel — up to $ 500 for domestic and $ 750 for international travel

An additional free MozCon ticket for you to give away, plus a code for $ 300 off of one ticket

An invitation for you and your significant other to join us for the pre-event speakers dinner

The selection process:

We have an internal committee of Mozzers that review every pitch. In the first phase we review only the topics to ensure that they’re a good fit for our audience. After this first phase, we look at the entirety of the pitch to help us get a comprehensive idea of what to expect from your talk on the MozCon stage.

Want some advice for perfecting your pitch?

Keep your pitch focused to online marketing. The more actionable the pitch, the better.

Be detailed! We want to know the actual tactics our audience will be learning about. Remember, we receive a ton of pitches, so the more you can explain, the better!

Keep the pitch to under 1200 characters. We’re strict with the word limits — even the best pitches will be disqualified if they don’t abide by the rules.

No pitches will be evaluated in advance, so please don’t ask

Using social media to lobby your pitch won’t help. Instead, put your time and energy into the actual pitch itself!

Linking to a previous example of a slide deck or presentation isn’t required, but it does help the committee a ton.

You’ve got this!

This could be you.

If your pitch is selected, the MozCon team will help you along the way. Whether this is your first time on stage or your twentieth, we want this to be your best talk to date. We’re here to answer questions that may come up and to work with you to deliver something you’re truly proud of. Here are just a handful of ways that we’re here to help:

Topic refinement

Helping with your session title and description

Reviewing any session outlines and drafts

Providing plenty of tips around best practices — specifically with the MozCon stage in mind

Comprehensive show guide

Being available to listen to you practice your talk

Reviewing your final deck

A full stage tour on Sunday to meet our A/V crew, see your presentation on the big screens, and get a feel for the show

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

You’re busy and (depending on effective keyword targeting) you’ve come here looking for something to shave months off the process of learning to produce your own chat bot. If you’re convinced you need this and just want the how-to, skip to “What my bot does.” If you want the background on why you should be building for platforms like Google Home, Alexa, and Facebook Messenger, read on.

Why should I read this?

Do you remember when it wasn’t necessary to have a website? When most boards would scoff at the value of running a Facebook page? Now Gartner is telling us that customers will manage 85% of their relationship with brands without interacting with a human by 2020 and publications like Forbes are saying that chat bots are the cause.

The situation now is the same as every time a new platform develops: if you don’t have something your customers can access, you’re giving that medium to your competition. At the moment, an automated presence on Google Home or Slack may not be central to your strategy, but those who claim ground now could dominate it in the future.

The problem is time. Sure, it’d be ideal to be everywhere all the time, to have your brand active on every platform. But it would also be ideal to catch at least four hours sleep a night or stop covering our keyboards with three-day-old chili con carne as we eat a hasty lunch in between building two of the Next Big Things. This is where you’re fortunate in two ways;

When we develop chat applications, we don’t have to worry about things like a beautiful user interface because it’s all speech or text. That’s not to say you don’t need to worry about user experience, as there are rules (and an art) to designing a good conversational back-and-forth. Amazon is actually offering some hefty prizes for outstanding examples.

I’ve spent the last six months working through the steps from complete ignorance to creating a distributable chat bot and I’m giving you all my workings. In this post I break down each of the levels of complexity, from no-code back-and-forth to managing user credentials and sessions the stretch over days or months. I’m also including full code that you can adapt and pull apart as needed. I’ve commented each portion of the code explaining what it does and linking to resources where necessary.

I’ve written more about the value of Interactive Personal Assistants on the Distilled blog, so this post won’t spend any longer focusing on why you should develop chat bots. Instead, I’ll share everything I’ve learned.

What my built-from-scratch bot does

Ever since I started investigating chat bots, I was particularly interested in finding out the answer to one question: What does it take for someone with little-to-no programming experience to create one of these chat applications from scratch? Fortunately, I have direct access to someone with little-to-no experience (before February, I had no idea what Python was). And so I set about designing my own bot with the following hard conditions:

It had to have some kind of real-world application. It didn’t have to be critical to a business, but it did have to bear basic user needs in mind.

It had to be easily distributable across the immediate intended users, and to have reasonable scope to distribute further (modifications at most, rather than a complete rewrite).

It had to be flexible enough that you, the reader, can take some free code and make your own chat bot.

It had to be possible to adapt the skeleton of the process for much more complex business cases.

It had to be free to run, but could have the option of paying to scale up or make life easier.

It had to send messages confirming when important steps had been completed.

The resulting program is “Vietnambot,” a program that communicates with Slack, the API.AI linguistic processing platform, and Google Sheets, using real-time and asynchronous processing and its own database for storing user credentials.

If that meant nothing to you, don’t worry — I’ll define those things in a bit, and the code I’m providing is obsessively commented with explanation. The thing to remember is it does all of this to write down food orders for our favorite Vietnamese restaurant in a shared Google Sheet, probably saving tens of seconds of Distilled company time every year.

It’s deliberately mundane, but it’s designed to be a template for far more complex interactions. The idea is that whether you want to write a no-code-needed back-and-forth just through API.AI; a simple Python program that receives information, does a thing, and sends a response; or something that breaks out of the limitations of linguistic processing platforms to perform complex interactions in user sessions that can last days, this post should give you some of the puzzle pieces and point you to others.

What is API.AI and what’s it used for?

API.AI is a linguistic processing interface. It can receive text, or speech converted to text, and perform much of the comprehension for you. You can see my Distilled post for more details, but essentially, it takes the phrase “My name is Robin and I want noodles today” and splits it up into components like:

Intent: food_request

Action: process_food

Name: Robin

Food: noodles

Time: today

This setup means you have some hope of responding to the hundreds of thousands of ways your users could find to say the same thing. It’s your choice whether API.AI receives a message and responds to the user right away, or whether it receives a message from a user, categorizes it and sends it to your application, then waits for your application to respond before sending your application’s response back to the user who made the original request. In its simplest form, the platform has a bunch of one-click integrations and requires absolutely no code.

I’ve listed the possible levels of complexity below, but it’s worth bearing some hard limitations in mind which apply to most of these services. They cannot remember anything outside of a user session, which will automatically end after about 30 minutes, they have to do everything through what are called POST and GET requests (something you can ignore unless you’re using code), and if you do choose to have it ask your application for information before it responds to the user, you have to do everything and respond within five seconds.

What are the other things?

Slack: A text-based messaging platform designed for work (or for distracting people from work).

Google Sheets: We all know this, but just in case, it’s Excel online.

Asynchronous processing: Most of the time, one program can do one thing at a time. Even if it asks another program to do something, it normally just stops and waits for the response. Asynchronous processing is how we ask a question and continue without waiting for the answer, possibly retrieving that answer at a later time.

Database: Again, it’s likely you know this, but if not: it’s Excel that our code will use (different from the Google Sheet).

Heroku: A platform for running code online. (Important to note: I don’t work for Heroku and haven’t been paid by them. I couldn’t say that it’s the best platform, but it can be free and, as of now, it’s the one I’m most familiar with).

How easy is it?

This graph isn’t terribly scientific and it’s from the perspective of someone who’s learning much of this for the first time, so here’s an approximate breakdown:

Label

Functionality

Time it took me

1

You set up the conversation purely through API.AI or similar, no external code needed. For instance, answering set questions about contact details or opening times

Half an hour to distributable prototype

2

A program that receives information from API.AI and uses that information to update the correct cells in a Google Sheet (but can’t remember user names and can’t use the slower Google Sheets integrations)

A few weeks to distributable prototype

3

A program that remembers user names once they’ve been set and writes them to Google Sheets. Is limited to five seconds processing time by API.AI, so can’t use the slower Google Sheets integrations and may not work reliably when the app has to boot up from sleep because that takes a few seconds of your allocation*

A few weeks on top of the last prototype

4

A program that remembers user details and manages the connection between API.AI and our chosen platform (in this case, Slack) so it can break out of the five-second processing window.

A few weeks more on top of the last prototype (not including the time needed to rewrite existing structures to work with this)

*On the Heroku free plan, when your app hasn’t been used for 30 minutes it goes to sleep. This means that the first time it’s activated it takes a little while to start your process, which can be a problem if you have a short window in which to act. You could get around this by (mis)using a free “uptime monitoring service” which sends a request every so often to keep your app awake. If you choose this method, in order to avoid using all of the Heroku free hours allocation by the end of the month, you’ll need to register your card (no charge, it just gets you extra hours) and only run this application on the account. Alternatively, there are any number of companies happy to take your money to keep your app alive.

For the rest of this post, I’m going to break down each of those key steps and either give an overview of how you could achieve it, or point you in the direction of where you can find that. The code I’m giving you is Python, but as long as you can receive and respond to GET and POST requests, you can do it in pretty much whatever format you wish.

1. Design your conversation

Conversational flow is an art form in itself. Jonathan Seal, strategy director at Mando and member of British Interactive Media Association’s AI thinktank, has given some great talks on the topic. Paul Pangaro has also spoken about conversation as more than interface in multiple mediums.

Your first step is to create a flow chart of the conversation. Write out your ideal conversation, then write out the most likely ways a person might go off track and how you’d deal with them. Then go online, find existing chat bots and do everything you can to break them. Write out the most difficult, obtuse, and nonsensical responses you can. Interact with them like you’re six glasses of wine in and trying to order a lemon engraving kit, interact with them as though you’ve found charges on your card for a lemon engraver you definitely didn’t buy and you are livid, interact with them like you’re a bored teenager. At every point, write down what you tried to do to break them and what the response was, then apply that to your flow. Then get someone else to try to break your flow. Give them no information whatsoever apart from the responses you’ve written down (not even what the bot is designed for), refuse to answer any input you don’t have written down, and see how it goes. David Low, principal evangelist for Amazon Alexa, often describes the value of printing out a script and testing the back-and-forth for a conversation. As well as helping to avoid gaps, it’ll also show you where you’re dumping a huge amount of information on the user.

While “best practices” are still developing for chat bots, a common theme is that it’s not a good idea to pretend your bot is a person. Be upfront that it’s a bot — users will find out anyway. Likewise, it’s incredibly frustrating to open a chat and have no idea what to say. On text platforms, start with a welcome message making it clear you’re a bot and giving examples of things you can do. On platforms like Google Home and Amazon Alexa users will expect a program, but the “things I can do” bit is still important enough that your bot won’t be approved without this opening phase.

I’ve included a sample conversational flow for Vietnambot at the end of this post as one way to approach it, although if you have ideas for alternative conversational structures I’d be interested in reading them in the comments.

A final piece of advice on conversations: The trick here is to find organic ways of controlling the possible inputs and preparing for unexpected inputs. That being said, the Alexa evangelist team provide an example of terrible user experience in which a bank’s app said: “If you want to continue, say nine.” Quite often questions, rather than instructions, are the key.

2. Create a conversation in API.AI

API.AI has quite a lot of documentation explaining how to create programs here, so I won’t go over individual steps.

Key things to understand:

You create agents; each is basically a different program. Agents recognize intents, which are simply ways of triggering a specific response. If someone says the right things at the right time, they meet criteria you have set, fall into an intent, and get a pre-set response.

The right things to say are included in the “User says” section (screenshot below). You set either exact phrases or lists of options as the necessary input. For instance, a user could write “Of course, I’m [any name]” or “Of course, I’m [any temperature].” You could set up one intent for name-is which matches “Of course, I’m [given-name]” and another intent for temperature which matches “Of course, I’m [temperature],” and depending on whether your user writes a name or temperature in that final block you could activate either the “name-is” or “temperature-is” intent.

The “right time” is defined by contexts. Contexts help define whether an intent will be activated, but are also created by certain intents. I’ve included a screenshot below of an example interaction. In this example, the user says that they would like to go to on holiday. This activates a holiday intent and sets the holiday context you can see in input contexts below. After that, our service will have automatically responded with the question “where would you like to go?” When our user says “The” and then any location, it activates our holiday location intent because it matches both the context, and what the user says. If, on the other hand, the user had initially said “I want to go to the theater,” that might have activated the theater intent which would set a theater context — so when we ask “what area of theaters are you interested in?” and the user says “The [location]” or even just “[location],” we will take them down a completely different path of suggesting theaters rather than hotels in Rome.

The way you can create conversations without ever using external code is by using these contexts. A user might say “What times are you open?”; you could set an open-time-inquiry context. In your response, you could give the times and ask if they want the phone number to contact you. You would then make a yes/no intent which matches the context you have set, so if your user says “Yes” you respond with the number. This could be set up within an hour but gets exponentially more complex when you need to respond to specific parts of the message. For instance, if you have different shop locations and want to give the right phone number without having to write out every possible location they could say in API.AI, you’ll need to integrate with external code (see section three).

Now, there will be times when your users don’t say what you’re expecting. Excluding contexts, there are three very important ways to deal with that:

Almost like keyword research — plan out as many possible variations of saying the same thing as possible, and put them all into the intent

Fallback contexts don’t have a user says section, but can be boxed in by contexts. They match anything that has the right context but doesn’t match any of your user says. It could be tempting to use fallback intents as a catch-all. Reasoning along the lines of “This is the only thing they’ll say, so we’ll just treat it the same” is understandable, but it opens up a massive hole in the process. Fallback intents are designed to be a conversational safety net. They operate exactly the same as in a normal conversation. If a person asked what you want in your tea and you responded “I don’t want tea” and that person made a cup of tea, wrote the words “I don’t want tea” on a piece of paper, and put it in, that is not a person you’d want to interact with again. If we are using fallback intents to do anything, we need to preface it with a check. If we had to resort to it in the example above, saying “I think you asked me to add I don’t want tea to your tea. Is that right?” is clunky and robotic, but it’s a big step forward, and you can travel the rest of the way by perfecting other parts of your conversation.

3. Integrating with external code

I used Heroku to build my app . Using this excellent weather webhook example you can actually deploy a bot to Heroku within minutes. I found this example particularly useful as something I could pick apart to make my own call and response program. The weather webhook takes the information and calls a yahoo app, but ignoring that specific functionality you essentially need the following if you’re working in Python:

#start
req = request.get_json
print("Request:")
print(json.dumps(req, indent=4))
#process to do your thing and decide what response should be
res = processRequest(req)
# Response we should receive from processRequest (you’ll need to write some code called processRequest and make it return the below, the weather webhook example above is a good one).
{
"speech": “speech we want to send back”,
"displayText": “display text we want to send back, usually matches speech”,
"source": "your app name"
}
# Making our response readable by API.AI and sending it back to the servic
response = make_response(res)
response.headers['Content-Type'] = 'application/json'
return response
# End

As long as you can receive and respond to requests like that (or in the equivalent for languages other than Python), your app and API.AI should both understand each other perfectly — what you do in the interim to change the world or make your response is entirely up to you. The main code I have included is a little different from this because it’s also designed to be the step in-between Slack and API.AI. However, I have heavily commented sections like like process_food and the database interaction processes, with both explanation and reading sources. Those comments should help you make it your own. If you want to repurpose my program to work within that five-second window, I would forget about the file called app.py and aim to copy whole processes from tasks.py, paste them into a program based on the weatherhook example above, and go from there.

Initially I’d recommend trying GSpread to make some changes to a test spreadsheet. That way you’ll get visible feedback on how well your application is running (you’ll need to go through the authorization steps as they are explained here).

4. Using a database

Databases are pretty easy to set up in Heroku. I chose the Postgres add-on (you just need to authenticate your account with a card; it won’t charge you anything and then you just click to install). In the import section of my code I’ve included links to useful resources which helped me figure out how to get the database up and running — for example, this blog post.

I used the Python library Psycopg2 to interact with the database. To steal some examples of using it in code, have a look at the section entitled “synchronous functions” in either the app.py or tasks.py files. Open_db_connection and close_db_connection do exactly what they say on the tin (open and close the connection with the database). You tell check_database to check a specific column for a specific user and it gives you the value, while update_columns adds a value to specified columns for a certain user record. Where things haven’t worked straightaway, I’ve included links to the pages where I found my solution. One thing to bear in mind is that I’ve used a way of including columns as a variable, which Psycopg2 recommends quite strongly against. I’ve gotten away with it so far because I’m always writing out the specific column names elsewhere — I’m just using that method as a short cut.

5. Processing outside of API.AI’s five-second window

It needs to be said that this step complicates things by no small amount. It also makes it harder to integrate with different applications. Rather than flicking a switch to roll out through API.AI, you have to write the code that interprets authentication and user-specific messages for each platform you’re integrating with. What’s more, spoken-only platforms like Google Home and Amazon Alexa don’t allow for this kind of circumvention of the rules — you have to sit within that 5–8 second window, so this method removes those options. The only reasons you should need to take the integration away from API.AI are:

You want to use it to work with a platform that it doesn’t have an integration with. It currently has 14 integrations including Facebook Messenger, Twitter, Slack, and Google Home. It also allows exporting your conversations in an Amazon Alexa-understandable format (Amazon has their own similar interface and a bunch of instructions on how to build a skill — here is an example.

You are processing masses of information. I’m talking really large amounts. Some flight comparison sites have had problems fitting within the timeout limit of these platforms, but if you aren’t trying to process every detail for every flight for the next 12 months and it’s taking more than five seconds, it’s probably going to be easier to make your code more efficient than work outside the window. Even if you are, those same flight comparison sites solved the problem by creating a process that regularly checks their full data set and creates a smaller pool of information that’s more quickly accessible.

You need to send multiple follow-up messages to your user. When using the API.AI integration it’s pretty much call-and-response; you don’t always get access to things like authorization tokens, which are what some messaging platforms require before you can automatically send messages to one of their users.

You’re working with another program that can be quite slow, or there are technical limitations to your setup. This one applies to Vietnambot, I used the GSpread library in my application, which is fantastic but can be slow to pull out bigger chunks of data. What’s more, Heroku can take a little while to start up if you’re not paying.

I could have paid or cut out some of the functionality to avoid needing to manage this part of the process, but that would have failed to meet number 4 in our original conditions: It had to be possible to adapt the skeleton of the process for much more complex business cases. If you decide you’d rather use my program within that five-second window, skip back to section 2 of this post. Otherwise, keep reading.

When we break out of the five-second API.AI window, we have to do a couple of things. First thing is to flip the process on its head.

What we were doing before:

User sends message -> API.AI -> our process -> API.AI -> user

What we need to do now:

User sends message -> our process -> API.AI -> our process -> user

Instead of API.AI waiting while we do our processing, we do some processing, wait for API.AI to categorize the message from us, do a bit more processing, then message the user.

The way this applies to Vietnambot is:

User says “I want [food]”

Slack sends a message to my app on Heroku

My app sends a “swift and confident” 200 response to Slack to prevent it from resending the message. To send the response, my process has to shut down, so before it does that, it activates a secondary process using “tasks.”

The secondary process takes the query text and sends it to API.AI, then gets back the response.

The secondary process checks our database for a user name. If we don’t have one saved, it sends another request to API.AI, putting it in the “we don’t have a name” context, and sends a message to our user asking for their name. That way, when our user responds with their name, API.AI is already primed to interpret it correctly because we’ve set the right context (see section 1 of this post). API.AI tells us that the latest message is a user name and we save it. When we have both the user name and food (whether we’ve just got it from the database or just saved it to the database), Vietnambot adds the order to our sheet, calculates whether we’ve reached the order minimum for that day, and sends a final success message.

6. Integrating with Slack

This won’t be the same as integrating with other messaging services, but it could give some insight into what might be required elsewhere. Slack has two authorization processes; we’ll call one “challenge” and the other “authentication.”

Slack includes instructions for an app lifecycle here, but API.AI actually has excellent instructions for how to set up your app; as a first step, create a simple back-and-forth conversation in API.AI (not your full product), go to integrations, switch on Slack, and run through the steps to set it up. Once that is up and working, you’ll need to change the OAuth URL and the Events URL to be the URL for your app.

Thanks to github user karishay, my app code includes a process for responding to the challenge process (which will tell Slack you’re set up to receive events) and for running through the authentication process, using our established database to save important user tokens. There’s also the option to save them to a Google Sheet if you haven’t got the database established yet. However, be wary of this as anything other than a first step — user tokens give an app a lot of power and have to be guarded carefully.

7. Asynchronous processing

We are running our app using Flask, which is basically a whole bunch of code we can call upon to deal with things like receiving requests for information over the internet. In order to create a secondary worker process I’ve used Redis and Celery. Redis is our “message broker”; it makes makes a list of everything we want our secondary process to do. Celery runs through that list and makes our worker process do those tasks in sequence. Redis is a note left on the fridge telling you to do your washing and take out the bins, while Celery is the housemate that bangs on your bedroom door, note in hand, and makes you do each thing. I’m sure our worker process doesn’t like Celery very much, but it’s really useful for us.

You can find instructions for adding Redis to your app in Heroku here and you can find advice on setting up Celery in Heroku here. Miguel Grinberg’s Using Celery with Flask blog post is also an excellent resource, but using the exact setup he gives results in a clash with our database, so it’s easier to stick with the Heroku version.

Up until this point, we’ve been calling functions in our main app — anything of the form function_name(argument_1, argument_2, argument_3). Now, by putting “tasks.” in front of our function, we’re saying “don’t do this now — hand it to the secondary process.” That’s because we’ve done a few things:

We’ve created tasks.py which is the secondary process. Basically it’s just one big, long function that our main code tells to run.

In tasks.py we’ve included Celery in our imports and set our app as celery.Celery(), meaning that when we use “app” later we’re essentially saying “this is part of our Celery jobs list” or rather “tasks.py will only do anything when its flatmate Celery comes banging on the door”

For every time our main process asks for an asynchronous function by writing tasks.any_function_name(), we have created that function in our secondary program just as we would if it were in the same file. However in our secondary program we’ve prefaced with “@app.task”, another way of saying “Do wash_the_dishes when Celery comes banging the door yelling wash_the_dishes(dishes, water, heat, resentment)”.

In our “procfile” (included as a file in my code) we have listed our worker process as –app=tasks.app

All this adds up to the following process:

Main program runs until it hits an asynchronous function

Main program fires off a message to Redis which has a list of work to be done. The main process doesn’t wait, it just runs through everything after it and in our case even shuts down

The Celery part of our worker program goes to Redis and checks for the latest update, it checks what function has been called (because our worker functions are named the same as when our main process called them), it gives our worker all the information to start doing that thing and tells it to get going

Our worker process starts the action it has been told to do, then shuts down.

As with the other topics mentioned here, I’ve included all of this in the code I’ve supplied, along with many of the sources used to gather the information — so feel free to use the processes I have. Also feel free to improve on them; as I said, the value of this investigation was that I am not a coder. Any suggestions for tweaks or improvements to the code are very much welcome.

Conclusion

As I mentioned in the introduction to this post, there’s huge opportunity for individuals and organizations to gain ground by creating conversational interactions for the general public. For the vast majority of cases you could be up and running in a few hours to a few days, depending on how complex you want your interactions to be and how comfortable you are with coding languages. There are some stumbling blocks out there, but hopefully this post and my obsessively annotated code can act as templates and signposts to help get you on your way.

Bonus #1: The conversational flow for my chat bot

This is by no means necessarily the best or only way to approach this interaction. This is designed to be as streamlined an interaction as possible, but we’re also working within the restrictions of the platform and the time investment necessary to produce this. Common wisdom is to create the flow of your conversation and then keep testing to perfect, so consider this example layout a step in that process. I’d also recommend putting one of these flow charts together before starting — otherwise you could find yourself having to redo a bunch of work to accommodate a better back-and-forth.

Bonus #2: General things I learned putting this together

As I mentioned above, this has been a project of going from complete ignorance of coding to slightly less ignorance. I am not a professional coder, but I found the following things I picked up to be hugely useful while I was starting out.

Comment everything. You’ll probably see my code is bordering on excessive commenting (anything after a # is a comment). While normally I’m sure someone wouldn’t want to include a bunch of Stack Overflow links in their code, I found notes about what things portions of code were trying to do, and where I got the reasoning from, hugely helpful as I tried to wrap my head around it all.

Print everything. In Python, everything within “print()” will be printed out in the app logs (see the commands tip for reading them in Heroku). While printing each action can mean you fill up a logging window terribly quickly (I started using the Heroku add-on LogDNA towards the end and it’s a huge step up in terms of ease of reading and length of history), often the times my app was falling over was because one specific function wasn’t getting what it needed, or because of another stupid typo. Having a semi-constant stream of actions and outputs logged meant I could find the fault much more quickly. My next step would probably be to introduce a way of easily switching on and off the less necessary print functions.

The following commands: Heroku’s how-to documentation for creating an app and adding code is pretty great, but I found myself using these all the time so thought I’d share (all of the below are written in the command line; type cmd in on Windows or by running Terminal on a Mac):

CD “””[file location]””” - select the file your code is in

“git init” – create a git file to add to

“git add .” – add all of the code in your file into the file that git will put online

“heroku git:remote -a [the name of your app]” – select your app as where to put the code

“git push heroku master” - send your code to the app you selected

“heroku ps” – find out whether your app is running or crashed

“heroku logs” – apologize to your other half for going totally unresponsive for the last ten minutes and start the process of working through your printouts to see what has gone wrong

POST requests will always wait for a response. Seems really basic — initially I thought that by just sending a POST request and not telling my application to wait for a response I’d be able to basically hot-potato work around and not worry about having to finish what I was doing. That’s not how it works in general, and it’s more of a symbol of my naivete in programming than anything else.

If something is really difficult, it’s very likely you’re doing it wrong.While I made sure to do pretty much all of the actual work myself (to
avoid simply farming it out to the very talented individuals at
Distilled), I was lucky enough to get some really valuable advice. The
piece of advice above was from Dominic Woodman, and I should have
listened to it more. The times when I made least progress were when I
was trying to use things the way they shouldn’t be used. Even when I
broke through those walls, I later found that someone didn’t want me to
use it that way because it would completely fail at a later point.
Tactical retreatis an option. (At this point, I should mention he wasn’t
the only one to give invaluable advice; Austin, Tom, and Duncan of the
Distilled R&D team were a huge help.)

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

“As a business owner/marketer … it’s a huge cost saver. Before we used to have to manually search for websites that were (a) relevant and (b) have a good Google PageRank. The tool does that automatically for us – it’s a treasure trove of information.”

“We’ve used a lot of link building tools over the years, but in the first half hour of using Link Builder we knew that is was going to be a tool that we’d be using for our SEO services from that point onwards.”

“It’s clean, it’s intuitive, simple and easy to use – that makes working with it very enjoyable.”

So what are we calling it today? Link building, link prospecting, content marketing, linkbait, socialbait, PR ? Whatever it is and whatever sub-definitions exist for the process of finding quality, related websites to link back to yours is difficult and time-consuming work.

As with most processes associated with SEO campaigns, or website marketing campaigns in general, enterprising folks have built tools to make our lives a little easier and our time more fruitful and productive. A couple of those enterprising fellows are Garrett French and Darren Shaw (from Whitespark.Ca) over at Citation Labs.

Garrett has a suite of link building tools available, many of them complement his flagship tool; The Link Prospector.

Link Prospector Review TOC

To help you navigate to specific sections of the review we’ve included in-content links below.

Selecting a Report

The nice thing about this tool is that it’s designed for a specific purpose; link prospecting. It’s not bloated with a bunch of other stuff you may not need and it’s easy to use, yet powerful, because it focus on doing one thing and doing it very well.

The UI of this tool is right on the money, in my opinion. Garrett has built in his own queries to find specific types of links for you (preset Reports). Here you can see the reports available to you, which are built to help you find common link types:

Customizing Your Prospecting

As you can see, there are a variety of built in queries available which run the gamut of most of the link outreach goals you might have (interviews, resource pages, guest posts, directories, and so on). Once you settle on the report type it’s time to select additional parameters like:

Region

Web or Blog, or Web AND Blog results

Search Depth (You can go up to 1,000 deep here, but if you make use of your exclusion lists you shouldn’t have to dive that deep)

Try to make your queries as relevant but broad as possible to get the best results. Searches that are too specific will either net to few results or many of your direct competitors. Here, you can see my report parameters for interviews I may want to do in specific areas of SEO (Garrett includes a helpful video on that page, which I highly recommend watching):

Using Exclusions

The use of exclusions is an often overlooked feature of this toolset. Brands are all over the SERPs these days so when you have the Link Prospector go out to crawl potential link sources based on keywords/queries, you’ll want to make sure you exclude sites you are fairly certain you won’t get a link from.

You may want to exclude such sites as Ebay, Amazon, NewEgg, and so on if you are running a site about computer parts. You can put your exclusions into 2 categories:

Global Exclusions

Campaign Exclusions

Global exclusions apply to each campaign automatically. You might want to go out and download top 100 site lists (or top 1,000) lists to stick in the Global Exclusions area or simply apply specific sites you know are irrelevant to your prospecting on the whole. To access Exclusion lists, just click on the exclusion option. From there, it’s just a matter of entering your domains:

Campaign exclusions only apply to a specific campaign. This is good news if you provide link building services and work with a variety of clients; you are not constrained to one draconian exclusion list. In speaking with Garrett, he does mention that this is an often overlooked feature of the toolset but one of the most effective features (both Global and Campaign exclusions).

Working With the Data

So I ran my report which was designed to find interviewees within certain broader areas of the SEO landscape. The tool will confirm submission of your request and email you when it’s complete, at any time you can go in and check the status of your reports by going to Prospects -> View Prospects. Here’s what the queue looks like:

The results are presented in a web interface but can be easily exported to excel. From the web interface, you can see:

Total Domains

Total Paths (pages on the domain where relevancy exists, maybe we would find a relevant video channel on YouTube where it makes sense to reach out)

TLD

LTS – Link Target Score

PR of Domain

Export Options

LTS is a proprietary score provided by Citation Labs (essentially a measure of domain frequency and position within the SERPs pulled back for a given report).

If we expand the domain to see the paths, using Search Engine Land as an example, we can see pages where targets outside of the main domain might exist for our interviewing needs:

This is where Citation Labs really shines. Rather than just spitting back a bunch of domains for you to pursue at a broad level, it breaks down authoritative domains into specific prospecting opportunities which are super-relevant to your query/keyword relationship.

If you are on Windows (or run Windows via a virutal machine) you can use SEO Tools for Excel to take all these URLs, or the ones you want to target, and pull in social metrics, backlink data, and many other data points to further refine your list.

You can also import this data right into Buzzstream (export from Citation Labs to a CSV or Excel, then import into Buzzstream) and Buzzstream will go off and look up relevant social and contact details for outreach purposes.

Creating Your Own Queries

Another nice thing about Citation Labs’s Link Prospector is that you can enter your own query parameters. You are not locked in to any specific type of data output (even though the built in ones are solid). You can do this by selecting “Custom” in the report selection field

In the Custom Report area you can create your own search operators along with the following options:

Region

Web or Blog, or Web AND Blog results

Search Depth (You can go up to 1,000 deep here, but if you make use of your exclusion lists you shouldn’t have to dive that deep)

Garrett’s Pro Tips

One objection I hear from folks who test the link prospector is “my results are full of competitors.” This is typically because the research phrases they’ve selected don’t line up with the type of prospects they’re seeking. And more often than not it’s because they’ve added their target SEO keywords rather than “category keywords” that define their area of practice.

The solution is simple though – you just need to experiment with some “bigger head” phrases. Instead of using “Atlanta Divorce Lawyer” for guest post prospecting, try just “Divorce Lawyer,” or even “Divorce.”

And I’d definitely recommend experimenting with the tilde “~Divorce” as it will help with synonyms that you may not have thought of. So if you’re looking for guest posting opportunities for a divorce lawyer your five research phrases could look like this:

divorce
~divorce
~divorce -divorce
Divorce ~Lawyer
“family law”

The link prospector tool will take these five phrases and combine them with 20+ guest posting footprints so we end up doing 100+ queries for you. And there WILL be domain repetitions due to the close semantic clustering of these phrases. This overlap can help “float up” the best opportunities based on our LTS score (which is essentially a measurement of relevance).

All this said there are PLENTY of situations where using your SEO keywords can be productive… For example in guest posting it’s common for people to use competitive keywords as anchor text. You could (and yes I’m completely contradicting my example) use “Atlanta Divorce Lawyer” as a guest posting research phrase along with your other target SEO KWs. The prospects that come back will probably have been placed by competitors.

How do you fine-tune your research phrases?

I often test my research phrases before throwing them in the tool. Let’s go back to the divorce guest posting example above. To test I simply head to Google and search [divorce "guest post"]. If I see 4 or more results in the top 10 that look like “maybes” I consider that a good keyword to run with. The test footprint you should use will vary from report-type to report-type.

A good links page test is to take a potential research phrase and add intitle:links. For content promoters you could combine a potential research phrase with intitle:”round up”.

I find that this testing does two things. For one it helps me drop research phrases that are only going to clog my reports with junk.

Secondarily I often discover new phrases that are likely to be productive. Look back at the list of divorce research phrases above – the last one, “family law,” is there because I spotted it while testing [~divorce "guest post"]. Spending time in Google is always, always productive and I highly advise it.

What tips can you give us regarding proper Search Depth usage?

Depth is a measure of how many results the link prospector brings back from Google. How often do you find useful results on the third page of Google? How about the tenth page? There’s a gem now and again, but I find that if I’ve carefully selected 5 awesome research phrases I save time by just analyzing the results in the top 20.

Your mileage may vary, and the tool DOES enable users to scrape all the way down to 1000 for those rare cases where you have discovered a mega-productive footprint. Test it once for sure, don’t just take my word for it – my guess is you’ll end up with tons of junk that actually kills the efficiency that the tool creates.

Any more expert tips on how to best use phrases and search operators?

You can addadvanced search operators in all your research phrases. Combine them with your research phrases and try them out in Google first (see tip 2) and then use them as you see fit. I use the heck out of the tilde now, as it saves me time and aids in research phrase discovery when I vet my phrases in Google. The tilde even works in conjunction with the wildcard operator (*).

So if you’re looking for law links pages you could test [~law* intitle:links] and then add ~law* as one of your research phrases if it seems productive. It’s not super productive by the way, because the word “code” is a law synonym… but I wouldn’t have known if I didn’t test, and if I didn’t test I’d end up with link prospetor results that don’t have anything to do with the targets I’m seeking.

Any tips on how to best leverage Exclusions (beyond putting in sites like google.com into your Global Exclusions )

If you have junk, not-ops that keeps turning up in your reports, add the domain as domain.com and www.domain.com to the exclusions file. Poof. It’s gone from future reports you run.

You can even add the domains you’ve already viewed so they won’t show up anymore. Be careful though – make sure you’re adding them to your campaign-level excludes rather than Global.

How often do you update the tool and what is coming down the pike?

If you sign up and you find yourself asking “I wonder what would happen if I…” please write me an email. If I don’t have an answer for you I will send you credits for you to do some testing. I will end up learning from you. I have users continually pushing the limits with the tool and finding new ways to use it.

We’ve added PR for domains, titles and snippets for each URL, blog-only search, and fixed numerous bugs and inefficiencies based on requests from our users. We’re also bringing in DA, MozRank and an API because of user requests.

Thanks Garrett!!

Free Trial and Pricing

Citation Labs is currently offering a free trial. They have monthly and per credit (love that!) pricing as well. You can find their pricing structure here.

This is a guest post written by Dave Kurlan, a top rated speaker, best selling author, and sales development thought leader. His top-rated business blog, Understanding the Sales Force, is read by thousands of sales and marketing leaders.