Navigation

List and Items

An entity is an instance of a data model, analogous to a data record in a database table.
When you store such a chunk of data, sometimes you need to locate it quickly, using a unique identifier. This is the entity's key.

Keys for data entities are typically unique relative to the data model.
They are commonly hidden in a web page so that, when the user clicks
something, the appropriate key is passed back to the server. This
"passing of keys" (from the server, to the user, then back to the
server) happens in any web application that empowers the user to create
many-to-one relationships on a web page -- for example, to create a list with items.

The Google App Engine (GAE) hosting environment provides a database with a difference: when created, each entity is assigned a key universal to Google's "Big Table", guaranteed never to change, and guaranteed to be accessible only from your GAE application.

This implies an unusual approach to passing keys to the browser. When
the webapp generates a web page the data choices can be hidden in the
page, not with ID's generated on-the-fly, but with permanent, universal
keys. There are clues on how to do this, mostly implied or embedded,
scattered across the Google App Engine Group's discussion forum, the GAE cookbook, etc. But there are no simple, deployed examples. So I thought I would provide one.
Unfolding of an Example

To present this, I'm explicitly introducing an ancient illustrative
technique to the world of software development, one that I think is
particularly pleasant, unusually effective and full of possibility: the unfolding of a program. It's a polished sequence
of stepwise software development. This is rather different than a
walk-through of finished code. You may have seen it before in public programming performances ... in fact, Google introduced GAE with just such a performance, which I'll analyze in a separate paper.

For the moment, let's think of a program as a living organism.

During an organism's development, its structure and shape changes,
moving from very simple to very complex. We marvel at nature's ability
to maintain a coherent morphology amidst this massive engineering
project. Amazingly, the elegant developmental sequences we see in
nature, which generate such robust and complex mechanisms, are very
flexible and adaptive. Living things tap a superb library of
transformative solutions and methods. Since humans have only about
30,000 genes, life somehow creates robust, unique people from something analogous to 30,000 small functions. The Linux Kernal has about 10 million lines of code. We need to pay much
closer attention to nature's approach.

If you watch this gradual, incremental growth, you'll see difficult
engineering problems overcome fluidly, simply, at each stage, at all
scales. A great deal of ad hoc adaptation is going on, adaptations synthesized from available information, used in a persistent, continuously balanced approach to tackling the emergence of the entire organism, not just tiny parts of it.

This holistic, stepwise approach to growth can be seen as following, or borrowing from, many sequences of steps. Each step creates a morphological differentiation,
building upon the previous step, providing the context for the next.
Surprisingly, although the number of sequences borrowed by a developing
organism is vast, the number of differentiations is not. A human being
develops in approximately 50 cell divisions, for example.

Assuming that programmers are called developers because they engage in something strongly akin to this kind of development,
it's unfortunate that one of the least explored areas in programming is
the study of the steps and sequences we use. When we decide to change
code, at that moment, what are we doing? What are the problems, and what is our resolution? And where
are we going? Clearly it would be useful to pass these steps and
sequences around, among ourselves, for the sake of improving our
discipline. The study of developmental programming
could provide a quicker, more efficient and effective absorption of new
contexts, in the fast-changing world of engineering. Sequences could
provide another tool for Open Source computer science.

If all this talk seems redolent of Design Patterns, there's a reason.
The notion to identify sequences of steps that humans can take, to build
complex coherent systems, comes from Christopher Alexander, whose
practical architectural philosophy inspired the computing movement to
mine patterns, and pattern languages. Research by Alexander and myself
in the 1990's indicated, among many other things, that people take from a
sequence whatever they need, absorbing information about dependencies,
and borrowing solutions, and sensibilities, very like a developing
organism making use of its genome.

I believe that polished sequences are a complement to patterns and
refactoring ... after you've refactored towards various patterns, you
tend to think: "if only I'd known about A! At this point I would have
done A instead of B" or "Y turned into a huge time-wasting problem later
on. I'm sure someone else has run into this."

Sequences give you the opportunity to save vast numbers of people from
the frustration with poor solutions, and teh difficult path to
rediscovery of an already known solution. Sequences can make us more
effective, and help us to produce quality work more easily and more
often.

So, to our example. I hope this new approach is useful, and that the
sequence itself provides solutions large and small, from which you can
borrow.

Preliminary: Scope and Environment

To get started, note that I'm not supplanting the steps found within the Google App Engine Getting Started
tutorial, which is still the only coherent tutorial for Google App
Engine development. If you're making use of my sequence, below, I assume
that you're already running the development environment, based on
instructions in the GAE tutorial.

Again I'm focussed on simply passing keys. This sequence will not
develop code for editing, ownership, user authentication, sessions,
sorting, timestamps, AJAX, exception handling, CSS, backups or any
number of otherwise perfectly reasonable and necessary features. But,
clearly, those tasks too can be broken into small sequences, "snippets"
if you will, which could be applied in an additive fashion to any
application derived from this article.

In this example I'm also using a very simple subset of Python, free of 'dunder' methods, and avoiding the Django helper.
I want to limit the number of technical features I'm describing here,
while allowing as broad a swath of programmers as possible to read it.
Similar limits were placed on the tutorials produced by Google.

We'll call this List-Item sample application "Passing Keys", or "passingkeys" to GAE. It's hosted at passingkeys.appspot.com, with links to the source. The primary Python module is passingkeys.py .

Step 1: Data and Purpose

There is a simple data model that reflects the purpose of the application

What will I do with this application? I will create lists, with items. So, let's define List and Item,
two different Datastore models derived from the 'db' class ... db is a
Google App Engine wrapper for the Datastore API, which reads and writes
from Google's Big Table (or, in the development environment on your own
machine, a simulation of Big Table).

Code Note: the relationship between List and Item is made explicit by Item's "ReferenceProperty (List)" declaration, in bold. When the time comes, Item's list_key will be assigned the instance of List that the Item belongs to.

General Note: if you are using these steps to guide the development of a
different application, it's good to keep data models that are
understood visually separate from your exploratory models, or your
exploratory properties.

Step 2: Functional Centers and the First Bridge

There are functional centers, which reflect the primary actions of the application, and the technology in which the application runs.

We'll define three clumps of code that I'll call functional centers. They reflect the three primary centers of activity in this app. These aren't classes yet. But this outline bridges the gap between, on the one hand, the application activity the user will see, and on the other hand, web technology (GETs, POSTs, FORMs, HTML) and the simple framework of supporting classes and methods found in this Google App Engine environment.

# The Home Page:
# * get: displays the current lists and items
# * render: called by all other functions to render a version of the home page

# Creating Lists:
# * get: renders the home page with a List form
# * post: handles the CGI POST to create a List

# Creating Items:
# * get: renders the home page with an Item form
# * post: handles the CGI POST to create an Item

Step 3: The Class and Method Bridge

The functional centers are expressed in the form of methods and classes. Among these, the Handlers form a distinct group.

I associated render above with the home page. But, when I step over the next bridge, to a class and method outline, Python's object idiosyncrasies kick in. 'Class methods' are not normal in Python, so render needs to be an isolated function definition. Hence, my class and method outline is:

The handler classes, and the calls to render, have explicit arguments.

Let's make the class and method outline more rigid, 'hardening' the shells of both.

Some might complain that this approach is too top-down, that I'm not following an end-to-end incremental methodology here. But, this is runnable code. Over the decades, programming languages and environments have shifted towards pseudocode, so that we can ease our way through these early differentiating stages more naturally. The previous steps could be end-to-end, and runnable, if the language and environment shifted a little further in this direction. And if it included higher-level descriptions for web pathways.

There's a hidden assumption in what follows -- it's in my mind, so I might as well spell it out early. For simplicity, I'll be using one Django template, which I'll name index.html, to create the HTML presented to the user. The template will be populated with Item and List data by my render function, which in turn is passed a Python dictionary (a collection of name-value pairs). This dictionary also includes switches to turn the forms on and off.

Here's an implementation outline of the five handler methods + 1 render function:

I call my render function for GETs, and I redirect to home after POSTs. I could consistently call the render function, or with a session layer I could consistently redirect. I'm just trying to reduce overhead, for this example.

Step 5: Handler Map

There is a section of code that maps incoming requests to the handlers

As per GAE's default webapp WSGI framework, there's a mapping, at the end of passingkeys.py, between our defined URL regex's and our request handlers. So the end of every main web application module in GAE-land looks like this:

A critically important feature used here, relative to passing keys in a Python/GAE implementation, is the "(.*)" in the regex. This passes the contents of that URL sub-string (a key passed back from the user) as an argument to CreateItem's CGI GET handler. In two steps, you'll see this assigned to the variable "list_key" inside CreateItem.get.

Step 6: Template Valve

The render function is the only one which calls the template rendering methods.

The Django template index.html will be rendered by my render function, which initially just looks like this:

Clearly, the understanding behind the smooth encoding of this template outline would require additional sequences to explain. I'm sticking with fairly large morphological changes in this sequence, but many other worthwhile sequences, and approaches, remain unexplored. For example, I could have started to create pieces of the template in earlier steps.

Note that from a user perspective, a web implementation pathway or cycle begins here. I start with the "Create" links. Developers tend to follow this cycle, this mini sequence, to put themselves in the user's shoes, and to provide a reality check, during live webapp development: create links -> create form -> entity storage -> render data. It's a very natural cycle, and so common that, if you watch development in time-lapse, it'll make you dizzy. You can see it in public programming events like the Google App Engine Introduction. I did it too, and it's reflected here. It's useful both in exploratory development and recapitulation.

What did I know at this "template logic outline" stage? With an evangelical passion for the Model-View-Controller pattern, Django templates attempt to force logical constraints on the developer, literally, by forcing as much logic as possible out of the template. The result is a few extra conditional tags here and there -- for example, a test for item_form and a separate "equal" test for the desired list_key, resulting in a duplicate "Create an item" link in the template.

Step 9: List Form Pathway

There is a form in the template for creating lists. There is a method in the CreateList handler for handling HTTP POST, and creating a list.

Next I need to generate the List form, when the user clicks the "Create a List" link. Here's CreateList.get again, unchanged:

A repackaged payload of List and Item data is an argument to the template renderer, to compensate for limitations in the Django system.

We need to display the list itself. I already have an idea for displaying items, so I'll write the code for rendering in one try.

Which brings us to the biggest consequence of the Django template philosophy, at this stage -- our function render needs to pre-package a new structure, to pass to the Django template rendering system. This is used by the "{% for list in lists %}" and "{% for item in list.items %}" template tags in my index.html.

To do this, render needs to fetch Lists and Items from the datastore and create a kind of populating payload, which I build as "new_lists" below, and rename as "lists" to pass to the template, within the "template_values" dict structure:

This sequence cannot represent the intentions of the builders of Python, Django or the GAE implementation. It is only the story of my resolutions to my problems. It will only let you understand what I've done. But, you may find something useful for your application.

Of course, there are many problems with this sequence. It's not perfect. However, I believe that many such sequences, ironed-out, polished and improved, analyzed and argued over, can serve as inspiring resources for programmers. This practical, introspective approach may guide us to a future of smoother software development, for increasingly complex and coherent systems. It might also shape the future of tools and languages to support good engineering and design.