Python on Quakkels.comhttp://quakkels.com/tags/python/index.xml
Recent content in Python on Quakkels.comHugo -- gohugo.ioen-usLet's Scrape the Web with Python 3http://quakkels.com/posts/lets-scrape-the-web-with-python-3/
Sun, 10 Mar 2013 00:00:00 +0000http://quakkels.com/posts/lets-scrape-the-web-with-python-3/
<p><img src="http://quakkels.com/images/pythonWebScrapeCover.jpg" alt="" /></p>
<p>In the back of my mind I&rsquo;ve always been intrigued by writing an application that can retrieve web pages over HTTP. It&rsquo;s a fairly simple thing to do. We have a myriad of web browsers that do it for us. But there is just something about writing an application that operates independently of a browser and reaches out to touch the internet that I find fun and intriguing. So let&rsquo;s do it&hellip; in Python.</p>
<p>First let&rsquo;s define some specifications for this project. Basically we&rsquo;re going to &ldquo;scrape&rdquo; Craigslist.org listings and display them in our terminal (command line). It should be able to scrape any (or nearly any) of Craigslist&rsquo;s regions and categories.</p>
<ul>
<li>Will be a command line application</li>
<li>Will read and display posts from Craigslist.org</li>
<li>Will be able to define desired Craigslist.org subdomain and path from the command line</li>
<li>Will display results to the terminal</li>
</ul>
<h2 id="separating-the-web-scraper-into-pieces">Separating the Web Scraper into Pieces</h2>
<p>As I see it, there are three pieces to this application:</p>
<ol>
<li>The user interface which handles input and displays ouput</li>
<li>The HTTP client which actually accesses the HTML page and gets the information therein</li>
<li>The HTML parser which reads the HTML and collects the parts we want to keep</li>
</ol>
<p>Now that the application&rsquo;s functionality is defined and it is broken down into pieces. We can start thinking about the project&rsquo;s name and structure. Let&rsquo;s just call it MyScrape and let&rsquo;s structure in the following folder and files.</p>
<ul>
<li>MyScrape/ (the application folder)
<ul>
<li>MyHttp.py (This will be responsible for the HTTP client that gets the web pages)</li>
<li>MyParser.py (To read the HTML and collect the good parts)</li>
<li>MyScrape.py (This will be the executable that handles user input and displays the results from MyParser)</li>
</ul></li>
</ul>
<h2 id="the-http-client-and-python-3">The HTTP Client and Python 3</h2>
<p>Python 3 has a handly little module that we can use to make our lives easy. We&rsquo;re going to import http.client into our MyHttp.py file and use it in our class.</p>
<pre><code>import http.client
</code></pre>
<p>Now that we imported http.client, we can create our class to handle a page. Let&rsquo;s keep things simple and just call this class Page. Page just needs to connect to a server, request a page using a path, and provide the result to the application. Here is the complete MyHttp.py file:</p>
<pre><code>'''GET a webpage using http.'''
import http.client
class Page:
def __init__(self, servername, path):
'''This initialize function sets the servername and path'''
self.set_target(servername, path)
def set_target(self, servername, path):
'''This is a utility function that will reset the servername and the path'''
self.servername = servername
self.path = path
def __get_page(self):
'''This is a private function that actually goes out
and gets the response from the server'''
server = http.client.HTTPConnection(self.servername)
server.putrequest('GET', self.path)
server.putheader('Accept', 'text/html')
server.endheaders()
return server.getresponse()
def get_as_string(self):
'''This function provides the webpage as a string'''
page = ''
reply = self.__get_page() # gets the page
if reply.status != 200:
page = 'Error sending request {0} {1}'.format(reply.status, reply.reason)
else:
data = reply.readlines()
reply.close()
for line in data:
page += line.decode('utf-8')
return page
</code></pre>
<p>Now that we have our class, we need to make sure it works by testing it. We can do that by using the Python interpreter to execute our code for us. First, start the Python interpreter by sending the command:</p>
<pre><code>$ python3
</code></pre>
<p>to the shell prompt (or the command prompt if you&rsquo;re on Windows). This should give you a prompt that looks like this:</p>
<pre><code>&gt;&gt;&gt;
</code></pre>
<p>To exit the interpreter, just enter exit() and press enter. <em>Note: More information about the Python interpreter can be found <a href="http://docs.python.org/3.3/tutorial/interpreter.html">here</a> on the Python.org website</em>.</p>
<p>To test our code in the Python interpreter, first navigate to the MyScrape folder that has the MyHttp.py file in it.</p>
<pre><code>$ cd path/to/your/MyScrape
</code></pre>
<p>Then start the interpreter and enter the following code:</p>
<pre><code>&gt;&gt;&gt; import MyHttp
&gt;&gt;&gt; page = MyHttp.Page('quakkels.com', '')
&gt;&gt;&gt; print(page.get_as_string())
</code></pre>
<p>You should now see the HTML source code for quakkels.com in your terminal. It works!</p>
<h2 id="html-parsing-and-python-3">HTML Parsing and Python 3</h2>
<p>The next part of this project we need to write is the HTML parser that allows us to identify the pieces of the Craigslist page that we want to keep. There are several different techniques for doing this including: regular expression matching (don&rsquo;t use this technique), <a href="https://developer.mozilla.org/en-US/docs/DOM">DOM</a>, and <a href="http://sax.sourceforge.net/">SAX</a> parsing.
The DOM (Document Object Model) technique basically involves navigating an XML or HTML document through a tree of nodes. The SAX (Simple API for XML) technique does not involve navigating like the DOM technique does. Rather, it reads the file through once, sending information to the application as the file is read. This means it&rsquo;s pretty quick, but because there is no navigation the application will need to keep track of the state of the document as the SAX style parser reads it. Our MyParser.py file is going to execute a SAX style parser using the html.parser module.</p>
<p>Python 3 has a handy module called html.parser that we&rsquo;ll use in our application. Our parser class is going to be designed to just read Cragslist.com listings. I&rsquo;m going to name the parser class ClParser. ClParser will need to inherit from HTMLParser (which is in the html.parse module) so that we can override the methods that get executed as the file is read in a SAX manner.</p>
<p>Here&rsquo;s the complete MyParser.py file:</p>
<pre><code>'''
Parse html from craigslist
'''
from html.parser import HTMLParser
class ClParser(HTMLParser):
# parser state
# These variables store the current state of the parser as it reads the file
date = '' # The date for the current listing
title = '' # The title of the current listing
link = '' # The link to the current listing's details
collectFor = None # will use this to keep track of what kind of data we
# are currently collecting for. valid options are:
# 'date', 'title', and 'link'
insideRow = False # This flag keeps track of whether we are inside a &quot;row&quot;
# &quot;rows&quot; have listing information
# parser output
results = '' # the parser's output will be stored here
def handle_starttag(self, tag, attrs):
'''This function gets called when the parser encounters a start tag'''
if tag == 'a' and self.insideRow:
self.collectFor = 'title'
for key, value in attrs:
if(self.collectFor == 'title'
and key == 'href'
and not self.link): # and not self.link makes sure it doesn't overwrite a preexisting value
self.link = value
if key == 'class':
if value == 'row':
self.insideRow = True
if value == 'ban':
self.collectFor = 'date'
def handle_endtag(self,tag):
'''This function is called when the parser encounters an end tag'''
if tag == 'p':
self.insideRow = False
# is there data to output?
if self.title + self.link:
self.results += &quot;\nDate: \t{0}\nTitle:\t{1}\nLink:\t{2}\n&quot;.format(
self.date,
self.title,
self.link)
self.__reset_row()
def handle_data(self, data):
'''This function is called when the parser encounters data inside to tags'''
if self.collectFor == 'date':
self.date = data
if self.collectFor == 'title' and not self.title:
self.title = data
self.collectFor = None # when we're done collecting the data, reset this flag
def __reset_row(self):
'''This is a utility function to reset the parser's state after a row'''
self.title = ''
self.link = ''
self.summary = ''
self.collectFor = None
self.insideRow = False
</code></pre>
<p>The HTMLParser class that we are inheriting from has a feed(string argument) function that has been applied to our ClParser class. To execute our parser, we just need to make an instance of the class and call the feed(string argument) function.</p>
<p>We can test this in the Python interpreter in the same way that we tested MyHttp. In the interpreter enter the following code:</p>
<pre><code>&gt;&gt;&gt; import MyHttp, MyParser
&gt;&gt;&gt; page = MyHttp.Page('milwaukee.craigslist.org', '/sya/')
&gt;&gt;&gt; parser = MyParser.ClParser()
&gt;&gt;&gt; parser.feed(page.get_as_string())
&gt;&gt;&gt; print(parser.results)
</code></pre>
<p>This should print a list of nicely formatted Craiglist listings for computers in the Milwaukee area. We&rsquo;re almost done!</p>
<h2 id="the-last-piece">The Last Piece!</h2>
<p>Alright, we have two of our three pieces built. The last thing to do is handle user input and display results. We&rsquo;re going to implement these features in the MyScrape.py file. Here&rsquo;s the whole MyScrape.py file:</p>
<pre><code>import sys, MyParser, MyHttp
# try to assign the subdomain and path values
# if the assignment fails, just use default values
try:
subdomain, path = sys.argv[1:]
except:
subdomain, path = 'milwaukee', '/sya/'
# instantiate the parser
parser = MyParser.ClParser()
# instantiate the page
page = MyHttp.Page(subdomain + '.craigslist.org', path)
# get the page and feed it to the parser
parser.feed(page.get_as_string())
# display the results
print('################\n Results:\n################\n', parser.results)
</code></pre>
<p>There you have it. MyScrape.py is the entry point to our application. It allows the user to set a subdomain and a path when calling the script. It brings the MyHttp and MyParser modules together. and it displays results to the screen. To use this application, enter the following command in your shell or command prompt:</p>
<pre><code>$ python3 MyScrape.py
</code></pre>
<p>&hellip;or&hellip;</p>
<pre><code>$ python3 MyScrape.py sierravista /ata/
</code></pre>
<p>You can download the entire source <a href="http://quakkels.com/files/MyScrape.zip">here</a>.</p>
<h2 id="improving-the-scrape">Improving the Scrape</h2>
<p>Feel free to take this code and experiment with it. Expand on it. Make it spider sub pages. Make it return a list of dictionaries instead of a string. Save the data in a sqlite database, or to a text file. Maybe make it into a web service. Do whatever you want with it. (Keep it legal.)</p>
Let's talk to a SQLite database with Pythonhttp://quakkels.com/posts/lets-talk-to-a-sqlite-database-with-python/
Sun, 20 Jan 2013 00:00:00 +0000http://quakkels.com/posts/lets-talk-to-a-sqlite-database-with-python/
<p>As I write this, it&rsquo;s the weekend. My wonderful wife brought home some amazing Jet Fuel XBold Dark Roast Coffee that is sure to keep me wired for the next several hours. Now what should I do? Let&rsquo;s explore Python a little more by using it to talk to a database.</p>
<p><img src="http://quakkels.com/images/JetFuel.png" alt="Turning caffeine into code!" /></p>
<p>We&rsquo;re a long way from my bread and butter. My goto technologies for database interaction are .NET, C# and Entity Framework CodeFirst with SQL Server <sup>2005</sup>&frasl;<sub>2008</sub>. These technologies lend themselves to enterprise level web and application development. They require a fair amount of software licenses and infrastructure to get rolling. In contrast, SQLite (as you may have guessed) is a very lightweight implementation of a relational database. It is completely self contained. There is no SQLite server or configuration requirements. You just make a database and boom! You&rsquo;re in business. Because of this light footprint and its ease of use, it is a very common database to use in mobile applications.</p>
<p>First, let&rsquo;s create a SQLite database file. I&rsquo;m going to call my database &ldquo;Blog.sqlite&rdquo;. You can do this from the command line with Python, or if you prefer a graphical interface, you can use a snappy little Firefox add-on called <a href="https://addons.mozilla.org/en-us/firefox/addon/sqlite-manager/">SQLite Manager</a>. It runs inside Firefox and offers a simple interface for interacting with SQLite databases. And, since it runs in Firefox, it works on any platform that runs Firefox (E.G. MacOS, Windows, Linux).</p>
<p>Once we have our database created, then we need to make a table. Once again, we can do this using straight SQL and Python from the command line, but I prefer to use the SQLite Manager add-on in Firefox.</p>
<p><img src="http://quakkels.com/images/CreatePostsTableResized.png" alt="Screenshot of creating a table with SQLite Manager." /></p>
<p><em>Note that the Id field is an auto incremented integer value and a primary key.</em></p>
<p>Now, let&rsquo;s dive into code. What will this application do? Well, since this project is in the spirit of exploration, the application we&rsquo;re going to build won&rsquo;t be very useful. Instead it will have an academic focus. Let&rsquo;s just perform basic CRUD operations (Create, Read, Update, Delete) and then exit. We won&rsquo;t worry about user interaction or GUI or web. This is going to just be a console app.</p>
<p><strong><em>This should go without saying, but this won&rsquo;t work without installing Python first. I&rsquo;m using Python3.2.3. Don&rsquo;t worry about downloading SQLite. There is no SQLite server, remember? If you have Python, you can use SQLite.</em></strong></p>
<p>In the same folder as the Blog.sqlite database, create your Python file using your favorite text editor. I called mine sqliteConnect.py. Once you have your file, you can run it from the command line by first browsing to the folder where you saved the file, then just enter the following command:</p>
<pre><code>$ python3 sqliteConnect.py
</code></pre>
<p>That will run the Python script. Though, nothing will happen until we put some code in there.</p>
<h2 id="connecting-to-your-sqlite-database-from-python">Connecting to Your SQLite Database from Python</h2>
<p>To perform CRUD functionality, first we need to establish a connection with the database, and then create what&rsquo;s known as a cursor which will allow us to execute commands.</p>
<pre><code>import sqlite3
conn = sqlite3.connect('Blog.sqlite')
cursor = conn.cursor()
</code></pre>
<h2 id="creating-a-record">Creating a Record</h2>
<p>Now that we have our open connection and our cursor, we can insert data into the Posts table.</p>
<pre><code>cursor.execute('insert into Posts (Headline, Body) values (?, ?)', ('This is my Headline', 'This is the body of my blog post.'))
firstPostId = cursor.lastrowid
cursor.execute('insert into Posts (Headline, Body) values (?, ?)', ('Jet Fuel XBold Coffee', 'Jet Fuel XBold Dark Roast Coffee will make you code like a madman.'))
conn.commit()
</code></pre>
<p>Here, we call <code>cursor.execute()</code> and pass it two parameters. A SQL command with placeholders for the data (the question marks), and a sequence containing the data to be inserted. Since we are hard-coding the insert, we could have just placed our data directly into the SQL. But in the real world, we could be inserting user generated input. In that case, we would want to use this parameterized technique to keep the SQL query safe and keep our database safe from SQL Injection style hacking attempts.</p>
<p>We have explicitly inserted the data for Headline and Body while ignoring the value for the Id field. This is purposeful because when we created the Posts table, we defined Id to be an auto-incrementing integer value. This is good because it saves us from having to generate a unique identifier ourselves. However, we are going to be editing this same record later in our application. In order to be able to get the same record, we need to know the auto generated Id value.</p>
<p>The last inserted row id is stored in <code>cursor.lastrowid</code>. So, when our insert is complete, if we want to grab the new Id value, we just need to assign <code>cursor.lastrowid</code> to our <code>firstPostId</code> variable.</p>
<p>It&rsquo;s worth noting here, calling <code>cursor.execute()</code> will execute the command on the database. But, we need to commit any changes by calling conn.commit(). If we don&rsquo;t call conn.commit() before the connection is closed, then our changes will be undone as if they never happened.</p>
<h2 id="reading-our-new-records">Reading Our New Records</h2>
<p>Now that we have some data inserted into the database table, let&rsquo;s read it out.</p>
<pre><code>cursor.execute('select * from Posts')
print('Current records: ')
for row in cursor.fetchall():
print('\t', row)
</code></pre>
<p>We are using the same <code>cursor.execute()</code> method to run a select query on the table. Then we can print each row by using <code>cursor.fetchall()</code> in a for loop.</p>
<h2 id="update-that-first-record">Update That First Record</h2>
<p>Now, let&rsquo;s make some changes to the record we inserted first. This is when the <code>firstPostId</code> comes in handy.</p>
<pre><code>cursor.execute('update Posts set Headline=?, Body=? where Id=?', ('This is my NEW Headline', 'This is the NEW body of my blog post.', firstPostId))
conn.commit()
print('Records after update: ')
cursor.execute('select * from Posts')
for row in cursor.fetchall():
print('\t', row)
</code></pre>
<p>Now when we print our read results, we can see that the first record has been updated with new data.</p>
<h2 id="delete-everything">Delete Everything</h2>
<p>You don&rsquo;t have to delete everything, but that&rsquo;s what I&rsquo;m going to do.</p>
<pre><code>cursor.close()
conn.close()
</code></pre>
<p>Now, when we print our query results, we see nothing.</p>
<h2 id="wrapping-up-closing-down">Wrapping Up, Closing Down</h2>
<pre><code>cursor.close()
conn.close()
</code></pre>
<p>Since we&rsquo;re done with our cursor and connection, we can close them down.</p>
<h2 id="complete-code">Complete Code</h2>
<pre><code>import sqlite3
conn = sqlite3.connect('Blog.sqlite')
cursor = conn.cursor()
# (C)reate a new blog post
cursor.execute('insert into Posts (Headline, Body) values (?, ?)', ('This is my Headline', 'This is the body of my blog post.'))
firstPostId = cursor.lastrowid
cursor.execute('insert into Posts (Headline, Body) values (?, ?)', ('Jet Fuel XBold Coffee', 'Jet Fuel XBold Dark Roast Coffee will make you code like a madman.'))
conn.commit()
# (R)ead our new posts
cursor.execute('select * from Posts')
print('Current records: ')
for row in cursor.fetchall():
print('\t', row)
# (U)pdate the first post
cursor.execute('update Posts set Headline=?, Body=? where Id=?', ('This is my NEW Headline', 'This is the NEW body of my blog post.', firstPostId))
conn.commit()
print('Records after update: ')
cursor.execute('select * from Posts')
for row in cursor.fetchall():
print('\t', row)
# (D)elete all the records
cursor.execute('delete from Posts')
print('Records after delete: ')
cursor.execute('select * from Posts')
for row in cursor.fetchall():
print('\t', row)
conn.commit()
cursor.close()
conn.close()
</code></pre>
ASP.NET MVC Compared to Djangohttp://quakkels.com/posts/asp-net-mvc-compared-to-dango/
Sun, 13 Jan 2013 00:00:00 +0000http://quakkels.com/posts/asp-net-mvc-compared-to-dango/
<p>The MVC pattern is widely used in web development today. Regardless of whether
you develop in C#, PHP, Python, Java, JavaScript, or Ruby, you won’t have any
trouble finding a popular MVC pattern for your language of choice.</p>
<p>Being primarily a C# developer, I’ve been using ASP.NET MVC since version 1.
It’s a great framework. It can be very good at getting a project off the ground
quickly, especially when coupled with a good ORM such as NHibernate, or Entity
Framework.</p>
<p>For a side project, I decided to set aside the familiar ASP.NET MVC framework in
favor of learning a new language and a new corresponding framework. After a
period of time spent deliberating over the virtues and drawbacks of each
language, I chose Python for the language, and Django for the framework&hellip; and
there was much rejoicing.</p>
<p>As I learned the language and read the Django docs, it became apparent to me
that the MVC pattern can be implemented in various ways. MVC is not just
whatever Microsoft says it is. The pattern (as with almost everything) is open
to a little interpretation. So I needed to come to terms with the absence of
some tools that I was used to having in ASP.NET MVC, as well as the unfamiliar
terminology in the Django framework.</p>
<h2 id="introducing-django-it-is-not-a-cms">Introducing Django (it is not a CMS)</h2>
<p>The first thing I would like to point out, is that Django is not a CMS. Django
is to Python as ASP.NET MVC is to C#. I was surprised to find that out. My first impression of Django was that it was like Drupal, or Wordpress. I thought it was more like a CMS than a framework.</p>
<p>Having said that, Django’s administration scaffolding is amazing. Almost to the point that it could appear to be a pre-built CMS. Django is literally capable of generating an entire administration section that allows you to manage data in your database as defined by your MVC models. That is something that ASP.NET MVC is capable of doing to a certain degree, when an application uses Entity Framework, but they do it a little differently. ASP.NET MVC Scaffolding is completely dependant on Visual Studio’s ability to read your models and then generate code for your controllers. Though, in my experience I often have to go back and rewrite most of the generated code, and many-to-many relationships are sketchy, if they’re generated at all.</p>
<p>That isn’t the case for Django, which has it’s own built in ORM. Django is capable of complete scaffolding of many-to-many relationships, image and file uploaders, field validation, as well as basic input fields.</p>
<h2 id="mvc-actually-stands-for-model-template-view">MVC actually stands for Model Template View</h2>
<p>It’s semantics, I know. Read on.</p>
<h2 id="django-controllers-aren-t-called-controllers">Django Controllers Aren’t Called Controllers</h2>
<p>To fulfill the MVC pattern, Django uses three primary divisions of code, the Model, the View, and the Template. Notice that there is no Controller.</p>
<p>Since I come from and ASP.NET MVC background, I was expecting code divisions that are labeled Model, View, and Controller (hence MVC). But no, this is not the case in Django. This fact caused some initial confusion for me as I was learning Django, but it doesn’t need to be confusing. If you’re familiar with ASP.NET MVC, then we just need to associate the Model, View, Controller concepts to the correct labels in Django.</p>
<table>
<thead>
<tr>
<th align="right"></th>
<th>Model</th>
<th>View</th>
<th>Controller</th>
</tr>
</thead>
<tbody>
<tr>
<td align="right"><strong>ASP.NET MVC Term</strong></td>
<td>Model</td>
<td>View</td>
<td>Controller</td>
</tr>
<tr>
<td align="right"><strong>Django Term</strong></td>
<td>Model</td>
<td>Template</td>
<td>View</td>
</tr>
</tbody>
</table>
<h2 id="django-s-views-are-not-asp-net-mvc-views">Django’s Views are not ASP.NET MVC Views</h2>
<p>When I think of Views in MVC, I think of the presentation layer which consists of markup, CSS, JavaScript, and some server-side scripting for basic display logic. That’s not what Views are in Django. Rather, Django’s Views do the equivalent job of the Controller in ASP.NET MVC.
Templates are the Real Views</p>
<p>ASP.NET MVC uses the term &lsquo;View&rsquo; to describe their presentation layer. HTML markup, css styles and JavaScript come together in a View file with help from Razor. Razor is very powerful and it is easy to pick up because it allows C# syntax directly in the view file to render a complete webpage. In my experience, this can also lead to an unbalanced View that has too much logic embedded in the View file instead of in the controller.</p>
<p>Django uses the term “Template&rsquo; to describe their presentation layer. It also brings HTML markup, css styles, and JavaScript together. Though, it uses it’s own template rendering system. Whereas ASP.NET has the power of Razor that leverages C# syntax, Django’s template rendering syntax is unique, and it introduces a steeper learning curve than Razor.</p>
<h2 id="if-you-re-looking-for-partial-views-in-django">If You’re Looking for Partial Views in Django&hellip;</h2>
<p>&hellip;You’re probably not going to find them. There is nothing called &ldquo;Partial Template&rdquo; in Django, though you can accomplish similar functionality.</p>
<p>Let’s say you have a data driven navigation bar that you want to display on all your template pages. You don’t want to write all your views to explicitly query that data, create the nav object for the template, and then have the navigation markup located in each template. Rather, you would like to just have located in your template file a reference to a &ldquo;Partial View&rsquo; which would do all that for you and it is independent of View-specific processes. In ASP.NET, this can be accomplished with Partial Views. In Django, this can be accomplished with Template Tags. It seems to me to be more cumbersome than a Partial View concept, but that could just be because I am still relatively new to the Django framework.
Models are Pretty Much What You Would Expect</p>
<p>Django has its own built in ORM. This allows you to define database tables, fields, validation, and relationships using Python classes. When you make changes to you Model’s classes, Django provides a simple command to sync the database:</p>
<pre><code>$ python manage.py syncdb
</code></pre>
<p>These Model classes are editable through Django’s excellent scaffolding in the admin area, and they are queryable from Views for displaying in templates.</p>
<h2 id="is-django-better-than-asp-net-mvc">Is Django Better than ASP.NET MVC?</h2>
<p>No, not really. Neither is ASP.NET better than Django. ASP.NET MVC and Django both have their weaknesses. They need to be selected based on the project specs. Also, familiarity is huge when considering what technology to use when starting a new project. I was looking for a low cost learning experience with my project, Python + Django was perfect for that. The project is now done, or perhaps more accurately described as ‘abandoned’, and I am glad that I explored it. I would probably use Django again, if the situation called for it. Though, I think first exploring other Python frameworks would be a good idea.</p>