Author: py-guy

This post is about working with containers and python web applications. But first, a little bit on containers and their use in a development ecosystem. Docker is a technology that provides abstraction and automation at an operating system level offering reusability, automation control, version control, per review, and testing capabilities.
This virtualization of bare-bones operating systems into containers enables a micro-services model where all the units of work are divided into separate units of work, facilitating scalability, relability and testing. In essence, docker containers allows developers to be accountable for programming features without having to worry about machine dependencies.

The following code blocks gives a walk-through on building a dockerized web app, the hello world of docker.

First create a new directory called “hello_docker” to contain our webserver. Then inside the hello_docker directory create a another directory creatively named “app.” Inside the app directory, create a file called hellodocker.py with the following contents.

This web server will run off a docker container so lets create a dockerfile that will be used to create our docker image. In the hello_docker directory create a file called Dockerfile with the following contents.

Dockerfiles contain a set of instructions for docker to create an image to specifications. The first line pulls the python 3 image as a base installation and installs flask, the next lines copy the code from our directory to the image on build and runs the server.

Python’s lightweight dynamic interface is proven excellent for networking, data scraping and gui generating tasks. Python’s powerful and possibly overlooked os module enables you to take a dynamic approach to operating system programming. With python you can read or write to and from files across different areas on your hard drive and interface with the cmd line simply utilizing a few lines of code. This becomes useful in managing dependencies and project states.

First import sys and os

# lister.py
import sys, os

Then we will create a new method lets call it lister which will take an argument root to create our directory tree. The for loop will iterate through each directory containing files and os.walk() will generate a list of directories either top-down or bottom-up. This will print each directory to the command line console encapsulated by braces.

In this post I will talk about how to handle JSON data from an external API utilizing python. Making calls to web services is made simple with python, with just a few lines of code you can track the International Space Station’s (ISS) position and time, realtime with a sleek graphical user interface. The following is a link to the project files download, https://codeclubprojects.org/en-GB/python/iss/.

The Turtle module is an object oriented graphics tool that draws to a canvas or screen. Turtle’s methods derived include forward(), backwards(), left() and right() like telling a turtle in what direction to draw. Turtle will draw over a NASA curated 2D map of Earth, so you should place the ‘map.jpg’ file in your project directory.

So one of the first things we need to do is instantiate a turtle screen with the following command.

Our location object will tell turtle to write the ISS png file to the screen at a specific position given the latitude and longitude of the ISS. Instantiate a Turtle() to create an object with the following code.

Now, before we can tell our turtle to draw the ISS overhead-time we need the actual latitude and longitude coordinates of the passing ISS. A quick google search gives us the coordinates to store in a dictionary.

Then to make the call to the url use urllib.request to access the url, querying for each given location. The data is then stored as a result, loaded in json format. Json stands for JavaScript Object Notation and is used to conveniently organize data.

The lines above are the contents of the json data, data is accessed similar to a python dictionary utilizing keys and indices.

The above code block makes a call to the api, loads the json data, parses the overhead position at the current geographic coordinates and plots the iss icon. The while loop is infinite to constantly track the iss.

If you like these blog posts or want to comment and or share something do so below and follow py-guy!

There is not much literature out there praising the usefulness of lambda functions, so I thought I would make this post about pythonic lambda functions.

Lambda functions originate in lambda calculus as abstractions and application of functions using variable binding and substitution. Such functions take nested form to encapsulate non-specific code with out cluttering code. Because lambda functions are not be bound to any identifier they are also less error-prone.

Given a large data set with the following structure, where dashes represent some data entry such as a string, int or datetime data value, this is often useful in building complex queries of big data.

The code below instantiates lambda functions that will select sub columns from a sequential level column for each given category.

The above statement would return all the data from the table, selecting “Snack” skill as the basis column and all the tutorial levels from “peeling bananas” subskill. As such these little functions can be efficient components, critical to complex dependency heavy programming.

If you like these blog posts or want to comment and or share something do so below and follow py-guy!

In this post I will talk about how to use pypack to program clean and reusable python code. For programming larger complex applications in python, import statements tend to clutter code readability and isn’t practicable to reuse for different projects. Let’s say if you want to code a data science app you have your “go-to” packages like numpy, matplotlib, math etc. or a web crawler like selenium, beautiful soup, and requests with compact packaging and reusable configuration programming is streamlined.

First specify the packages used in your program in a configuration file named ‘config,’ defining imports and statements in key value declaration spaced by one line.

The above code snippet is the result of the configuration file contents listed at the beginning of the post. pypack is a simple program written in python with less than 37 lines of code that reads the specified packages from the config file and writes those packages to a new python file for specialized coding projects.

import sys
# config file should be in same folder as pypack
# if not, specify
f=open('config','r')
s=f.read()

Python syntax is such that assigning elements is as simple as encapsulating a loop with brackets. The first four lines of this snippet comma delimit the the config file and assign imports and statement elements to separate arrays.

In python object oriented programming is a simple way to build powerful applications. Consider a real-world object like a pair of shorts. This pair of shorts has a set of attributes and properties to make that pair of shorts unique. For example this pair of shorts might have pockets, buttons and zippers to put on and take off the shorts. Essentially, we have a blueprint to make any pair of shorts (give or take a few unique properties), this is known as a class and is the fundamental concept of object oriented programming and design. Each class defines attributes and methods instantiated by objects. Let’s take a look at some example code of out shorts class.

In python each class has an __init__ constructor to define unique parameters for each object. Self is the reference to the object at reference and initializes attributes unique to that class. Above, the constructor class takes parameters self, waist length and color and initializes those values as arguments utilized later in the program. Put on and take off methods pass those arguments by reference and updates self.wearing to false to let us know the shorts are off.

The code above defines methods to handle attributes of the shorts object ie. self.waist, self.length, self.color and self.wearing. When executed the passed attributes are printed to the console. The code below shows the class, instantiated as an object calling the defined methods.

The last post about VR technology I wrote with really no preface as to why write about VR other than I wanted to so this post serves (I should hope) as a preface as to why VR. VR Technology is not groundbreaking, its been around for years along with the buckets of scifi tropes giving VR the center stage, so what makes VR exciting?

There are the Oculus rift, the HTC Vive, PlayStation VR headsets positional tracking and specs to boast and each come with there own set of accessories these however require an expensive high end host PC. But these days everyone with access to a super computer in their pocket has the option of buying one of the mobile headsets to begin their own VR experience.

With mobile headsets like the Google Daydream, Samsung Gear, and flavors of Google Cardboard VR is an affordable option for anyone and everyone to develop and or consume VR content. Unity Game Engine, Unreal game engine to name a few support application development integrating the hardware sdk libraries and a great wealth of developer tools to quickly get develop a VR app.

Then the next question is what makes a great VR experience? I have the Samsung Gear with out any of the peripheral accessories, so I brainstormed simplicity. How can I make a VR experience enjoyable using only interaction supported by the headset, ie motion and touchpad? I reflected upon my early childhood playing litebrite with my friends and how that was such a fun experience and thought that would port to a great VR experience, and have started prototyping.

I decided the simpler the experience would immerse and ultimately give the user an intuitive sense of presence. The controls utilize user head movements to explore a world of spherical lites and the touchpad to select different colored lites and clear lites. BriteLites will be available to download through the oculus store for free in the near distant future.

If you like these blog posts or want to comment and or share something do so below and follow py-guy!

This week I’m deviating from posting about data science and python modules to explore virtual reality with my new samsung gear vr to create a 360 image viewer application. This can be done with not much code simply utilizing Unity3d and the OVR sdk to some satisfying results. Following the rest of this post is an abstract walk-through of the steps to create your own 360 image viewer application.

For this tutorial you will need the latest version of Unity 5, and the OVR sdk.

First, make a new Unity Project file with the name “360Viewer”. Choose where you want to save the project on your computer, make sure 3D is selected and click Create Project.
Select OVR from the downloaded folder, drag and drop it into your Assets folder.
Before we jump into our 360Viewer application make sure you’ve created a Plugins folder for your oculus signature file with the structure: Plugins > Android > Assets and then your oculus signature file.

This file is necessary for the development to access the low-level VR functionality of your device. You can download your oculus signature file when you sign up as an Oculus developer at https://developer.oculus.com/.

Next we want to configure the Unity3D environment to develop for mobile VR. Navigate to File > Build Settings and select Android.

Leave the Development Build field unchecked. This is how you build your application for testing. Click Player Settings and navigate to Player Settings in the Inspector.
Make sure Virtual Reality Supported is checked and select the Oculus SDK. Navigate to identification and set your package name.

Now we will create a sphere 3D gameobject by navigating to the Hierarchy, and selecting your scene. Right click and select 3D Object > Sphere.

You will see the Sphere object appear in the scene, let’s make sure it is centered at the origin of our scene at position (0, 0, 0) and set the scale to (100, 100, 100).
Make sure Blend Probes are selected in your sphere’s Mesh Renderer.

Then create a folder for your 360 images or panorama photos and add your images to the folder. These image files will be applied to our sphere’s texture later, but first we need to create a shader that will map the images to the sphere. Create the following shader file and name it DoubleSided from which you will cycle textures.

Create a C# script with name “textureCycler,” and copy and paste the code below. This will enable us to browse our images utilizing the oculus touchpad. Attach the script to your sphere, and select the size and images for your viewer.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

Next delete the main camera, navigate to Assets > OVR > Prefabs and drag and drop OVRPlayerController onto your sphere. In the Hierarchy select the OCRPlayerController, if you expand the contents you will see LeftEyeAnchor and Right EyeAnchor children objects. These are utilized to calibrate the virtual environment to the hardware’s optics. We want the OVRPlayerController object to see the inside of the sphere, the OVRPlayerController sends a raycast in the direction the samsung gear is facing and returns the first object it hits.

This is the sphere, we want to see the side of the sphere facing the camera, this is the texture, so select clear flags to skybox, culling mask to default and set field of view to 60. Do the same for the left and right.

Select your sphere object and make sure you’ve checked static and apply to all children objects. Add some in game lighting so you can view the scene!

If you like these blog posts or want to comment and or share something do so below and follow py-guy!

So I still haven’t figured out if I want to make one blog post a week or make more than one post a week but I will try to effectively post at least once a week on topics in computer science. We’ll see where it goes, it will be very exciting and most certainly worth the click.

This week I plan on exploring a data set of over 5,000 film entries scraped from imdb in an effort to briefly discuss machine learning, particularly Latent Dirichlet Allocation. I will not go into any of the theory because that is beyond the scope of this blog, these aren’t the droids you’re looking for.

However, nltk and gensim provide extensive apis that enable processing human language. Anything from stemming down to root words and or tokenizing a document to perform further analysis it is made easy with the above modules.

Latent Dirichlet Allocation is used to estimate word topic assignments and the frequency of those assignments for a fixed number of words called documents. Let’s assume each document exhibits multiple topics. So we will be looking at columns plot_keywords and genres.

movie['plot_keywords']

Next let’s remove the pipe with some list comprehension and check if successful.

Stemming reduces words down to their root word and is particularly useful in developing insightful NLP models.

docs=[d for d in keyword_strings if d.count(' ')==5]
len(docs)
texts=[]
#create english stop words list
en_stop= get_stop_words('en')
# create p_stemmer of class PorterStemmer
# stemmer reduces words in a topic to its root word
p_stemmer= PorterStemmer()
# init regex tokenizer
tokenizer= RegexpTokenizer(r'\w+')
# for each document clean and tokenize document string,
# remove stop words from tokens, stem tokens and add to list
for i in docs:
raw=i.lower()
tokens=tokenizer.tokenize(raw)
stopped_tokens=[i for i in tokens if not i in en_stop]
stemmed_tokens = [p_stemmer.stem(i) for i in stopped_tokens]
texts.append(stemmed_tokens)

The next block of code transforms the granular data into sets of identifiable tokens to manipulate later. To do so, let’s create a dictionary for the terms and value and a matrix for each document and term relationship.

The immediate next line of code generates the Latent Dirichlet Allocation model taking the corpus, the number of topics and the number of training iterations. Printing the model we see there is an estimate of observed words assigned to each topic, effectively (or ineffectively) predicted.

Sci-kit learn is a fantastic set of tools for machine learning in python. It is built on numpy, scipy, and matplotlib introduced in the first py-guy post and makes data analysis and visualization simple and intuitive. sci-kit learn provides classification, regression, clustering, dimensionality reduction, model selection, and preprocessing algorithms making data analysis in python accessible to everyone. We will cover an example of linear regression in this weeks post exploring Solar Radiation data from a NASA hackathon.

First after importing packages let’s read in the SolarPrediction.csv data set. The link to the data set is commented in the code block.

Taking a first look at the data set, specifically, UNIXTime and Date, note it is not formatted to a particular type so we will look at this later.

df.shape
df.describe()

Calling the describe method on the data frame returns some descriptive statistics on the data set and tells us there might be a relationship between radiation, humidity and or temperature.

So let’s look at a correlation plot to get a better feel for any possible relationships.

truthmat= df.corr()
sns.heatmap(truthmat, vmax=.8, square=True)

There is a strong relationship between radiation and temperature (unsurprisingly or surprisingly) so let’s choose two features with some ambiguity. Pressure and Temperature will do fine, we will use seaborn, a statistical visualization library based on matplotlib to explore the relationship between the two features.

First we will convert to date time to manipulate later then add hour, month and year columns for a granular scope. Much Better!

With sklearn linear regression we can train python to model the data and then test the model for its accuracy. We will drop temperature column from the dependent variables because that is what we want to learn.

If you like these blog posts or want to comment and or share something do so below and follow py-guy!

Note: I referenced kaggler Sarah VCH’s notebook in making todays blog post, specifically the feature engineering code in the fifth code block. If you want to see her notebook I’ve listed the link below.

Posts navigation

Previous Post

Welcome to py-guy! py-guy blog explores science, culture and technology with simple examples and thoughtful discussions. I will list a “preview” link to the previous blog post here once I figure out how to do so and after I’ve written more blog posts.