Categories

Meta

Tag: Python

Before we start make sure you can ssh into your machine and run
$sudo apt-get update

It’s a four step process:
1. Open the appropriate port on Azure
2. Install pip, virtualenv, virtualenvwrapper, and flask
3. Write our code and run it
4. Keep it running with Gunicorn

1. Open the appropriate ports on Azure
Go to you virtual machines landing:

Then select the resource group in the top left corner:

Resource groups are the way Azure breaks down how our VM interacts with the internet, other vms, storage, and public/private networks.

Top open the port we need to change our network security group, which is represented by the shield. (Underlined in the screenshot above)

Then select settings -> Inbound Security Rules:

This will allos us to open up our VM to the public internet so we can visit what’s presented at the port like a regular website.

You should see SSH already included, that’s the port we’re using in our ssh client/terminal.

We’re now going to add two new Inbound Security Rules one called FlaskPort where we’ll set the destination port range to 5000 and use for debugging. The second will be called FlaskProduction that we’ll use to deploy our complete app.
Here’s the configuration panel for FlaskPort:
Press okay to accept the settings.

And the other panel for FlaskProduction:
Again press okay to accept the settings.

Notice how the ‘Source Port Range’ is ‘*’ that just means that we’ll accept connections from the port of any machine. This tripped me up the first time.

In a couple seconds the port will be open we’ll be ready to visit it, but nothing will be there because we haven’t create an application server.

To do that we’ll install the basics.

2. Install pip, virtualenv, virtualenvwrapper, and flask

To use Python effectively we utilize virtual environments to help keep our various python project and required libraries in order.

If you get lost in these steps or want more context Gerhard Burger provides the same setup on a very helpful post on askubuntu.

First we install pip:$ sudo apt-get install python-pip

Second we install virtualenv and virtualenvwrapper

$ sudo pip install virtualenv
$ sudo pip install virtualenvwrapper

Third we configure virtualenv and virtualenvwrapper

Create a WORKON_HOME string which will contain the directory for our virtual environments. We’ll name it vitualenvs

$ export WORKON_HOME=~/.virtualenvs

Now we’ll create this directory.

$ mkdir $WORKON_HOME

And add this to our bashrc file so this variable is defined automatically every time we hit the terminal.

$ echo "export WORKON_HOME=$WORKON_HOME" >> ~/.bashrc

Then we’ll setup virtualenvwrapper by importing its functions with bashrc.

$ echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.bashrc

You can see the additions to our bashrc file by opening it with nano. Scrolling down to the bottom you should see two lines like this:

Then implement your changes.

$ source ~/.bashrc

Here’s what all that looks like all together:

Fourth, let’s create our first virtualenvironment

$ mkvirtualenv venv

And take a look at the currently installed packages$ pip list

Like so:

Now we can install all of the python packages we want without risk of needing to reinstall python!

Fifth, install flask:

$ pip install flask
$ pip list

3. Write our code and run it!
Our first app is a simple site that shares an image.

We’re going to create a folder called Photo-App that contains two folders and an app.py that will serve our clients.

To change to our home directory:$ cd ~
And create our new folder:$ mkdir Photo-App

You’re using gunicorn to start app hosted at 0.0.0.0:8000 with the reload tag configured.
The reload will look for changes in your code and reload the server everytime you change any server side stuff. It won’t auto-reload for HTML changes, but will reload them once you make a change to the python code.

When I search:
Node JS Server Azure, Ubuntu, JavaScript, Mongo, Postgres, Flask, VM
I turn up with all sorts of unhelpful results.
So I dedicated a couple days to creating a couple guides for common Cloud Stacks on Azure VMs to make it as simple as possible to start deploying your code to the cloud.

This is the introduction and at the bottom of this blog post you’ll see other workflows fill in.

First up is Storage:
This will configure what we want to name the storage account for our vm. I’ve changed mine to ‘resourcegrouponestorage’, but I could have selected any of my previous storage account in the same region, in this case westus.

Second is Network:
We can configure a Virtual Network to allow our virtual machines to connect to other resource on our network by default. We can also change this later. So in this case I’m creating the default virtual network.

Again, I could have selected a previously created Virtual Network Called ‘Databases’ which is in the same region.

Third is Extensions:
We won’t add any extensions

Fourth is Monitoring:
Which we’ll disable for simplicity sake, but is a very powerful tool one you start needing to make scaling decisions.

Fifth and finally is Availability:
We won’t use an availability set, until we need to scale out our app.

Here’s what the lower portion of our settings pane looks like:

And we’ll select OK to finish with our settings. This will take us to the summary page so we can do a one more check on our machine, don’t get to anxious about making mistakes because we can always tear this one down and spin up another if we messed something up!

Hit Okay one last time!

You’ll then be taken to your dashboard where you’ll see a nice loading tile:

It’ll take ~5 minutes to spin up and then we’ll be ready to take on the world!

Once ready it’ll look like this:

Click the tile to hit the landing page for our VM:

See that public IP address?
We’ll use that to SSH into our machine.

In my case: 13.88.180.170 !

3. Check VM using SSH

Let’s SSH into our box.

Pull out your preferred SSH client. Here’s bash on Windows and Putty Side by Side:

Notice ‘Timothy’ Triple underlined?
That’s the User Name we set during basic configuration and is paired with the password that we also set in Azure.

When you connect you might have to accept the ras2key fingerprint. It’ll look like this when using putty. Or it’ll be in the terminal using bash. Type ‘yes’ or Select Yes to continue.

Now you’ve created a virtualenv inside of your my_project_folder directory. Which is cool, but can be confusing with git, sharing code, and testing package versions.
So we use VirtualEnvWrapper to keep our virtualenvs in the same place.

This is what it looks like in nano for me.
Ctrl+X to exit and y-enter to save.

Then either:
Source ~/.bashrc

Or start a new command prompt->bash and try “workon” or “lsvirtualenv”

See the next image for a simple workflow.

9. Install some packages

Now lets install ‘requests’ into our newly created virtualenv:

Isn’t that nice!

10. Test Flask
Finally we’re going to test this with flask.
First we install the required files using pip into our activated ‘venv’
Then runserver -> navigate to the designated address -> and see our site.

As of March 2015 the download you want for a standard windows machine is Windows x86-64 MSI installer (The other download is for servers). Its circled here:

Run the installer!
You’ll come across this page in the installer:

You’ll want to scroll down and add it to the path. If you don’t that’s okay. You can add it later.
Adding Python to the PATH will allow you to call if from the command line.

After the installation is complete double check to make sure you see python in your PATH. You can find your path by opening your control panel -> System and Security -> System -> Advanced System Settings -> Environment Variables -> Selecting Path -> Edit ->

Now you’re looking at your Path. Be Careful, if you delete or add to the path accidently you may break other programs.

You need to confirm that C:\Python27; and C:\Python27\Scripts; is part of your path.

If you do not see it in your path you can simply add it at the beginning or end of the variable value box. As you can see in the image below.

Install Pip:

As of Python Version 2.7.9 Pip is installed automatically and will be available in your Scripts folder.

If you install a later version of Python I would recommend installing it according to this helpful stackoverflow post.

Pip is a Package manager for python which we will use to load in modules/libraries into our environments.

An example of one of these libraries is VirtualEnv which will help us keep our environments clean from other Libraries. This sounds really confusing but as you start using it you’ll begin to understand how valuable this encapsulation of modules/libraries can be.

To test that Pip is installed open a command prompt (win+r->’cmd’->Enter) and try ‘pip help’

You should see a list of available commands including install, which we’ll use for the next part:

Install virtualenv:

Now that you have pip installed and a command prompt open installing virtualenv to our root Python installation is as easy as typing ‘pip install virtualenv’
Like so:

Now we have virtualenv installed which will make it possible to create individual environments to test our code in. But managing all these environments can become cumbersome. So we’ll pip install another helpful package…

Install virtualenvwrapper-win:

This is the kit and caboodle of this guide.

Just as before we’ll use pip to install virtualenvwrapper-win. ‘pip install virtualenvwrapper-win’
Like so:

Excellent! Now we have everything we need to start building software using python! Now I’ll show you how buttery smooth it is to use these awesome tools!

Lets call it HelloWold. All we do in a command prompt is enter ‘mkvirtualenv HelloWold’
This will create a folder with python.exe, pip, and setuptools all ready to go in its own little environment. It will also activate the Virtual Environment which is indicated with the (HelloWold) on the left side of the prompt.

Anything we install now will be specific to this project. And available to the projects we connect to this environment.

Connect our project with our Environment:

Now we want our code to use this environment to install packages and run/test code.

First lets create a directory with the same name as our virtual environment in our preferred development folder. In this case mine is ‘dev’

Now to bind our virtualenv with our current working directory we simply enter ‘setprojectdir .’
Like so:

Now next time we activate this environment we will automatically move into this directory!
Buttery smooth.

Deactivate:

Let say you’re content with the work you’ve contributed to this project and you want to move onto something else in the command line. Simply type ‘deactivate’ to deactivate your environment.
Like so:

Notice how the parenthesis disappear.
You don’t have to deactivate your environment. Closing your command prompt will deactivate it for you. As long as the parenthesis are not there you will not be affecting your environment. But you will be able to impact your root python installation.

Workon:

Now you’ve got some work to do. Open up the command prompt and type ‘workon HelloWold’ to activate the environment and move into your root project folder.

Like so:

Pretty sweet! Lets get working.

Pip Install:

To use flask we need to install the packages and to do that we can use pip to install it into our HelloWold virtual environment.

Make sure (HelloWold) is to the left of your prompt and enter ‘pip install flask’
Like so:

This will bring in all the tools required to write your first web server!

Flask:

Now that you have flask installed in your virtual environment you can start coding!

Open up your favorite text editor and create a new file called hello.py and save it in your HelloWold directory.

I’ve simply taken the sample code from Flask’s website to create a very basic ‘Hello World!’ server.

I’ve named the file hello.py.

Once the code is in place I can start the server using ‘python hello.py’ this will run the python instance from your virtual environment that has flask.

I was struggling with pulling the data from a Database using flask and serving it to the client for an Ajax request.
The end goal is a heat map of sorts with paths being mapped and colors being changed for certain conditions.

I’m still working out the best practices for this, but its working! So take it with a grain of salt.

Take that list and pass it as JSON to a template.
Still not sure if this is best practice, as on the client/javascript side it picks it up as an array object.
But this is the route we use in flask to send our data.

This week I’ve been making progress on the Huggable Cloud Pillow website.

In the process I’ve learned about some sweet stuff you can do with Javascript, Python, and Flask-SocketIO.

The first thing to take note of is Flask.

Flask is the tiny server that allows us to host websites using Python to deliver content to the client. While on the server side you can manage complicated back ends or other processes using Python in conjunction with Flask.

This type of development is nice, because you can start seeing results right away but can take on big projects easily.

It might not be the most robust Framework, but its great for small projects…

If you want to get into Flask Web Development checkout this extensive MVA.

Small and simple, Flask is static on its own. This allows us to present static content, like templates and images easily and deals with input from the user using RESTful calls to receive input. This is great for static things with lots of user actions, but if we want something a bit more dynamic we’re going to need another tool.

In this case I’ve found Flask-SocketIO, similar to Flask-Sockets but with a few key differences highlighted by the creator (Miguel Grinberg) here.

Sockets are great for is providing live information with low latency. Basically, you can get info on the webpage without reloading or waiting for long-polling.

There are lots of ways you can extend this functionality to sharing rooms and providing communication with users and all sorts of fun stuff that is highlighted on GitHub with a great chunk of sample code. The following demo is based off of these samples.

For my project, I need the webpage to regularly check for differences in the state of the cloud and present them to the client, while also changing the image the user sees.

At first I tried to implement it using sockets passing information back and forth, but that wasn’t very stable.

The solution I’ve found, uses a background thread that is constantly running while the Flask-SocketIO Application is running, it provides a loop that I use to constantly check state of our queue.

Let’s break it down…
a. I need my website to display the current state of the cloud.
b. The Flask application can get the state by query our azure queue.
c. Once we determine a change of state we can display that information to the webpage.
d. To display the new state to the webpage we need to use a socket.
e. And pass the msg to be displayed.

This demo intends to break down problem a, c, d, and e.

I’ve created this little guide to help another developer get going quickly, with a nice piece of code available on GitHub.

The five steps to this little demo project are as follows:
1. Install Flask-SocketIO into our Virtual Environment
2. Create our background thread
3. Have it emit the current state to our client
4. Call the background thread when our page render_template’s
5. Have the Javascript Catch the emit and format our HTML.
Celebrate! Its Friday!

1.

Flask-SocketIO is a python package that is available for download using

This emit will be sending to the client (Javascript) a message called ‘message’.

When the Javascript catches this message it will be able to pull from the python dicionary msg.data and msg.time to get the result of this package.

4

So how do we call background_stuff?

We can call it wherever we want, but for this simple example we’ll put it right in our ‘/’ base route. So when we navigate to 127.0.0.1:5000 (Local Host) we’ll see the result of our background thread call.

I’m working with a raspberry pi and I’m learning how to collect and manage sensor data.

There is a lot of data and that’s awesome, but dealing with remote persistent storage has helped me understand some good ways of keeping stuff organized and safe.

For example: ‘Keys’ or ‘Tokens’ (Not to be confused with the Late English writer Tolkien)

These are the keys to the door of your data.
To access the contents behind these doors, you must establish that you are the key-holder. One of the ways to do this in software is to create a long complicated string that can passed to the database so you can input or output data.

As long as you keep this token secret nobody will be able to mess with your data, or use it without having to pay for it.

Now this sounds pretty simple, but what if you use that token in code that is stored in a public repository on GitHub? This is essentially like printing a bunch of keys to your house and leaving them all over town. Not safe or smart.

So…

1. We need to keep these keys separate from the rest of your code.2. We need to keep the file that contains those keys away from your public repository.

To do this I created a separate file called tokens.py that contains the strings of my tokens and a function called getStorageKey() that returns those strings.
Cool, we’ve satisfied our first goal.

Now we need to keep this file away from our repository. To do this we create a .gitignore file. This is a special ‘git’ file that allows us to specify which files will not be added to the remote repository.
To create a .gitignore file simply enter:

touch .gitignore

touch is a Unix command to create a file and update the access data, but not make any edits. Its the same as opening and closing without saving any changes.

After you’ve created this file you can edit it with any editor you like. I do it with sublime or VS Code because it helps me keep track of the separation occurring between visual studio and GitHub.

All you need to do is add on a separate lines the files you don’t want GitHub or just git to include.

When I first started all my .gitignore file had was:

tokens.py

Its expanded to include…
tokens.py
*.pyc

# the asterisk* acts as a wild card and will match any files with .pyc at the end
# anything that starts with a # is a comment and will be ignored from .gitignore

I saved this file (.gitignore) then added and committed it to the repository.

Now any files I add that end in .pyc or match tokens.py will not be added to my repository. And you will not be prompted to add them when you check git status.

What about files we’ve already have added???

If we add the rule after we added the file we want to ignore… We need to remove it from tracking on GitHub using the rm –cache command.

This will not remove it from your computer, but it will stop if from being tracked by GitHub, and will essentially be treated as if it was removed from your repository.

To pull this off simply type this command

git rm --cached filename.py

In this case I’m removing a file named ‘filename.py’.
After checking our git status we’ll see that this file is going to be removed.

Commit the changes and push to your remote repo.

You’ll see the files are still in your local directory, but are no longer in the remote or local repository. Hooray for keeping things safe!

Please let me know if you have any questions or comments.
The documentation is excellent and I recommend you spend some time reading through the resources below to round out your learning:

I ran into some troubles migrating/configuring my tables for a new app in my Django project.

I’ve been following this excellent tutorial, and ran into a bump I thought needed some clarification/update. As I’m not sure if the guide is up to date with the current version of Django.

Things I searched:
no module named models
manage.py sqlall
django not creating db
models are not being created in sqlite3 Django
sqlite3 not creating table django
No migrations to apply
django sqlite3 unable to create tables
manage.py shell
sqlite3 python package

Do these correlate with what you’re having issues with?
If so this was my solution.

First Install SQLite3 and add it too your Environment Variables -> Path
Install Page — Select: Source Code -> Zip for windows machines
I extracted it to C:/.
Now I have SQLite.exe in my root directory.
This is what the end of my Path looks like:
C:\Program Files\Microsoft\Web Platform Installer\;C:\SQLITE3.EXE\;

Sweet, now we can use SQLite in Powershell.

Configuration of Visual Studio:
Create a new app by right clicking on your Project file.
Then “Add” -> Select “Django App”
In this case my app is named book.

Sweet, now we have another Python app.
Go into your settings file and add it to the INSTALLED_APPS tuple.
eg. ‘book’,

Okay, now we’re configured make sure you’re SQLiteDB is properly configured as well.