I am sitting on a plane flying eastbound at 37,000 feet over middle of America. A few hours ago we left San Francisco the host city for State Of The Map US 2013. State of the Map US is the American regional conference for OpenStreetMap participants and enthusiasts. A common theme for the weekends events was building community.

I go to open-source conferences so that I can spend time informally talking with other members of my open source community. In San Francisco this week I was able to meet OpenStreetMap celebrities such as Richard Fairhurst (author of Potlach) and Frederick Ramm (former JOSM maintainer, Geofabrick founder and OSMF board member). I also spent time informally chatting with dozens of people who attended the conference to learn more about OpenStreetMap. Many people I spoke with told me they were interested in OpenStreetMap and were looking into using the data but had only done a small amount of mapping.

Conferences typically have three sets of activities the most popular of these are the sessions. My favorite sessions of this conference was the session Ryan Closner gave on on using Voxel.js to render OpenStreetMap data and a set of sessions on who is doing mapping. Voxel.js is a open-source clone of Minecraft written in Javascript. The presenter has built software that allows your character to ‘walk through’ a world consisting of real streets and building locations through OpenStreetMap data. My other favorite session was the track on community analysis. Richard Fairhurst, Alan McConchie, and Martjin van Exel presented a set of very related talks on who is doing the mapping. I hope to use some of their techniques and code to perform an analysis of activities in the Canadian community.

The second conference track is the hall track where everyone talks informally between sessions in the hall or conference while waiting for a session to start. I find that I get the most out of a conference by making a point of saying hi to the person sitting next to me while waiting for a session to start. I might ask them where they are from, why they are at the conference and what types of things they have mapped. I learned that there are many people new to the OpenStreetMap community. I have been adding to data to OpenStreetMap for almost 5 years and have spoken on OpenStreetMap at at least 3 conferences and numerous mapping parties and hack days. This was the first OpenStreetMap conference for many of the people I spoke with. Many of them are coming either from a traditional geography background or web background where they are starting to use OpenStreetMap for stuff. As a community we, need to keep these people engaged as regular casual mappers to make sure that they help us keep the map strong. I spoke with many people that we were able to convince too spend their weekend at a OpenStreetMap conference but we haven’t yet been able to convince them to spend their weekend adding to the map. How do we convince these people to occasionally take half a day and spend it on improving the map?

We also need to make sure that they can easily use our data. We need to reach out and build bridges with communities that can use our map. A group of us met in a BOF to discuss improving the diversity of the people involved in OpenStreetMap. We agreed to create a diversity-talk mailing list to discuss ways we can expand the OpenStreetMap community. I hope to spend some time thinking of ways I can grow our community by making inroads into other established communities.

The third conference track I attended was the party track. I attended three parties in San Francisco. There was a party Saturday night at The Raven a popular bar on Mission in San Francisco. On Sunday, Stamen design welcomed the OpenStreetMap world into their office for a party. The Stamen office is filled with cool maps. Monday night Mapbox also hosted a small party at a pub following the code-sprint. I missed the Code for America party on Friday but I don’t doubt that was fun. Eating and drinking with other mappers allows you to get to know them in a more relaxed atmosphere. I find that it is much easier to ask someone for help or their honest opinion when you’ve spent time socializing with them.

I want to thank OpenStreetMap.US and the sponsors for organizing this conference and all of the folks from Europe and Asia (I met people who flew in from Japan and New Zealand).

]]>https://scanningpages.wordpress.com/2013/06/13/state-of-the-map-us-2013/feed/0scanningpagesCode Sprint PhotoMaproulette Hack Weekendhttps://scanningpages.wordpress.com/2013/03/13/maproulette-hack-weekend/
https://scanningpages.wordpress.com/2013/03/13/maproulette-hack-weekend/#commentsWed, 13 Mar 2013 02:11:29 +0000http://scanningpages.wordpress.com/?p=589]]>This weekend we had the second annual Toronto OpenStreetMap developer weekend. The nice folks at the Ryerson Department of Geography hosted us. My focus this weekend was to work the Serge and Martijn on maproulette

Maproulette is software that presents an easy to do mapping task to users which they can complete and then mark the task as completed. Examples of past maproulette mapping challenges include fixing connectivity errors or fixing objects touched by the license change.
Serge and Martijn wanted to redesign parts of maproulette to make it easier develop more challenge modules. I am going to discuss some of the design decisions we debated during the weekend.

Challenges
A Challenge is a type of mapping activity that map roulette can assign to users. Examples of challenges might be ‘Add bridges to roads with where we suspect a bridge exists’ or ‘Add the street address tags to restaurants that don’t have addresses’.

Tasks
A task is a specific work item within a challenge. In the ‘Add bridges to roads’ challenge a task might be to add a bridge to a specific intersection (or determine that bridge should be there)

Maproulette needs to provide a web based interface that displays challenge tasks to users(task assignments). This requires us to somehow store information about the tasks that make up a challenge. Challenge specific modules will compute tasks (ie find missing bridges). Figuring out how the web interface and the programs that find challenge tasks should be connected was one of the questions we discussed this weekend.

Proposal 1: Discrete Modules

Each challenge has a challenge module that supports operations such as

Give me some available tasks near this lat/lon

Mark this task as assigned to a user

Mark this task as complete/skipped

Each challenge module would be responsible for storing task data in it’s own
format (or using database tables specific to the module). The user’s web-browser would talk to a common front-end component that would invoke the individual challenge modules. The challenge modules would return HTML/Javascript/JSON suitable that the browser would use in displaying information about the task to the user.

Proposal 2: Shared Task Representation

Challenge modules compute a list of tasks by examining a mixture of OSM and external data sources. The challenge modules then convert these tasks into a common representation made up of a task and associated sets of task_objects. A task to find add a missing bridge might be made up of three objects. An object for the place we think the bridge belongs, an object for the road the bridge is part of and a third object for the road/stream that the bridge crosses over. These tasks and associated objects get stored in a database. Tasks and associated objects for different challenges get stored in the same pair of database tables. The web application that users interact with can then display and manipulate information about tasks from all challenges.
Contrasting the options

When designing software systems you often have two or more different approaches to a given problem. Deciding which option to proceed with is the challenge software designers have to face. Some of the questions I asked myself this weekend included

How can code between different challenges be shared or re-used in each of the options? . The second approach sounds like it has more places where code can be re-used because all challenges use the same data model for assigning tasks

Is one approach more flexible than the other? Are there things different challenges might deal with differently that will be hard to accommodate with one of the designs? Our concern with the second proposal was that we were generalizing all challenges to object + geometry and not much else. What if some challenges needed to display extra things such as textual hints?

How do the proposals make it easier or harder for someone to come along and develop a new challenge? In the first approach a challenge developer can has free reign to implement things in any fashion they like as long as they implement the task assignment API I discuss above. In the second proposal challenge authors can still pick their programming language and methods for detecting challenges but they must use the database schema we provide for storing information about tasks. With the second approach challenge developers only need to worry about writing programs to find challenge tasks and store them in the database. Task assignment/management is handled by the common code

A third proposal

We were still debating the merits of these different approaches on late-morning on Sunday when we decided to take a break and go for lunch. One would think that a group of mappers would know lots of places nearby for lunch in downtown Toronto but we seemed to be wondering aimlessly back and forth down the street. While we were doing this we came up with the idea of a shared database schema approach like proposal 2 but letting each challenge store a blob of JSON with the task. Each challenge would also implement challenge specific rendering/display could that would know how to take the JSON describing the task and display information about the task to the user in a meaning full way.

I want to thank Richard Weait for organizing this weekend and Serge and Martijn for letting me participate in the design process. Next we need to finish implementing what we discussed. This should help us figure out if our ideas were good ideas.

]]>https://scanningpages.wordpress.com/2013/03/13/maproulette-hack-weekend/feed/0scanningpagesPython urllib2, cookies and OpenStreetMaphttps://scanningpages.wordpress.com/2012/12/29/python-urllib2-cookies-and-openstreetmap/
https://scanningpages.wordpress.com/2012/12/29/python-urllib2-cookies-and-openstreetmap/#commentsSat, 29 Dec 2012 06:00:36 +0000http://scanningpages.wordpress.com/?p=576]]>I found the documentation on urllib2 a bit unclear about how to get cookie handling working properly.

I was working on a python script that needed to contact the OpenStreetMap web server, login with my OSM credentials and interact with the website.

The first step is to setup a urllib2 opener instance that is configured to store cookies.

This will create an opener that can be used to retrieve URL’s. Any cookies set in the HTTP response will be stored in the cookie’s cookie jar. If I needed to add additional openers (ie for special redirect handling) I would just add them as additional parameters to the build_opener call. ie urllib2.build_opener(handler1, handler2,handler3)…

Next we need to contact OpenStreetMap to get a blank login form. The blank login screen has a hidden variable ‘authenticity_token’ that needs to be passed back as part of the POST with my login credentials

inptag = '{http://www.w3.org/1999/xhtml}input'
formtag = '{http://www.w3.org/1999/xhtml}form'
# fetch the blank login form
response_tokenfetch = opener.open(request)
html = response_tokenfetch.read()
htmlfile=StringIO.StringIO(html)
# parse the HTML elements in the form
# extract any input fields for later resubmission
# this will pick up the authenticity_token and anything else
xml_tree = ElementTree.parse(htmlfile)
for form in xml_tree.getiterator(formtag):
for field in form.getiterator(inptag):
if 'name' in field.attrib and 'value' in field.attrib:
login_payload[field.attrib['name']] = field.attrib['value']
login_payload['username'] = username
login_payload['password'] = password
login_payload['remember_me'] = 'yes'
login_payload['cookie_test'] = 'true'

Next we submit the LOGIN request as a POST. Any session cookies returned as part of the blank form will be added to the second request.

You could then parse the HTML to extract a list of messages.
If your using the formal OpenStreetMap API (ie calls under /api/0.6/…) then you should instead use oauth for authentication instead of logging in through the website. Some OSM features such as messaging can only be accessed by pretending to be a web session and parsing/faking HTML.

]]>https://scanningpages.wordpress.com/2012/12/29/python-urllib2-cookies-and-openstreetmap/feed/2scanningpagesLearning from mistakeshttps://scanningpages.wordpress.com/2012/08/16/learning-from-mistakes/
https://scanningpages.wordpress.com/2012/08/16/learning-from-mistakes/#commentsThu, 16 Aug 2012 23:38:02 +0000http://scanningpages.wordpress.com/?p=549]]>Last year I attended and wrote about FOSS4G 2011 in Denver. The 2012 FOSS4G event was supposed to take place in Beijing this September. Last month OSGEO announced that the conference was cancelled.Cameron Shorter has posted a detailed analysis on how the conference fell apart. His post does a great job of looking at the sequence of events that lead up to the current situation.

In the corporate world when an event, company or product falls on its face the detailed reasons are hushed up and rarely get a public airing. When the government screws up sometimes we get an expensive public inquiry where lawyers argue about how someone else did or didn’t screw up. OSGEO has a tradition of building consensus and discussing non-technical issues on their mailing lists. Good open-source projects not only be give us amazing software but they allow us to learn valuable lessons on public relations, finances and management by letting us watch their inner-workings.

When I was learning too fly airplanes I made a point of regularly reading the investigation reports of plane crashes. A common theme was that a single mistake didn’t cause the accident but a chain of mistakes, malfunctions and bad luck all added up to the crash. Common thinking in the aviation community is that if you learn about the mistakes others have made then you will be more likely to avoid making the same mistakes. The story Cameron tells is about limited options, ignoring red flags, communication break downs and bad luck. These are all issues we see everyday across all aspects of life.

The mailing list discussion about the lessons that can be learned for future FOSS4G events is still on-going but as outsiders we can follow their discussion and try to learn from their mistakes. When our events, initiatives or products don’t succeed we should also conduct a carefully look at how we got to that state and what things, with 20/20 hindsight, we should have paid more attention too.

]]>https://scanningpages.wordpress.com/2012/08/16/learning-from-mistakes/feed/0scanningpagesPGCon2012 – Making Mapshttps://scanningpages.wordpress.com/2012/05/22/pgcon2012-making-maps/
https://scanningpages.wordpress.com/2012/05/22/pgcon2012-making-maps/#commentsTue, 22 May 2012 01:16:34 +0000http://scanningpages.wordpress.com/?p=515]]>Slides from my 2012 PGCon presentation Making your Own Maps are now available

The presentation covered

Common reasons people render their own maps

Where to get OpenStreetMap data and how to load it into your PostGIS database

How to use Tilemill to design your own map style

How to render map tiles, both statically and dynamically

How to use OpenLayers to display your map

The presentation was recorded. I will update this post when the recording comes online.

I’ve been a regular attendee of PGCON since the first year it was held in Ottawa. Like past years I enjoyed the conference and I would like to thank Dan Langille for putting together another first-rate conference. My favourite part of PostgreSQL conferences is meeting and reconnecting with users people in the community.

I have published a new version of the Peninsula Lake map. This edition includes new cartography and map updates based on my survey work last summer.

I had written the previous map style in Cascadnik, a CSS like language that translates to mapnik XML. Cascadnik has since been replaced by Carto so I have decided to rewrite the map style using TileMill (Map design software that uses Carto and Mapnik).

One of my goals while redesigning the map was to put to use what I have learned about cartography over the past year. The original PenLake map style was based on a sample Cascadnik style adapted to my colour scheme, nautical features and a print scale. The new style was designed by me from scratch to focus on

The lake and nearby rivers, streams and marshes

Recreational attractions around the lake including hotels and golf courses

Roads near the lake

The water depth of the lake including suspected rocks

This style is much smaller than the previous one ( the style is less than 300 lines of CSS) and it behaves well at different zoom levels between 12 and 18. I suspect the number of layers can still be further reduced for faster rendering times.

Some of the water depth data in this version of the map is based on observations I manually made while kayaking on the lake in the summer of 2011. I have tried to correct this data to match the sounding datum used by the CHS in their observations. I don’t expect the observations taken by me with a kayak, rope and a tape measure to match the accuracy of the CHS with their fancy sonar but we have to work with the equipment that we have available to us. The depths on the map should be based on a datum of 913 feet.

]]>https://scanningpages.wordpress.com/2012/05/06/penlake-map-may-2012/feed/0scanningpagespenlake.2012.05.smallOSM Hack Weekend (Toronto 2012)https://scanningpages.wordpress.com/2012/03/18/osm-hack-weekend-toronto-2012/
https://scanningpages.wordpress.com/2012/03/18/osm-hack-weekend-toronto-2012/#commentsSun, 18 Mar 2012 15:34:02 +0000http://scanningpages.wordpress.com/?p=485]]>The Toronto OSM hack weekend (2012) started on friday with an introduction to OpenStreetMap presentation at the Ryerson Geography department. Experienced OpenStreetMap presenter, Richard Weait gave a talk to a room of Geography students, staff and faculty. After Richard explained the basics of OpenStreetMap A.J. and Tristen gave a short presentation on some of the work that they have been doing for MapBox.

After the presentations we did a QA session with the audience. The OSM ‘experts‘ in the room tried to answer the questions from the audience. Most people in the audience were geographers and GIS people who had already heard of OSM. A number of the attendees already had worked with various components in the OSM stack. We had questions on projections, layers, TileMill, Mapnick, tagging and an assortment of other topics.

After the Q/A I went with A.J and Tristen to Chiplote for some Burritos. At Chipotle we learned three things. 1) Chipotle in Washington does more of the meat preperation in the back room than they do in Toronto; Toronto should take a hint. 2) Keeping the Burrito in the tinfoil wrap it in keeps it from falling apart as you eat it. 3) Next time I need to remember to bring Buritos back to my fellow hackers. Friday night we had a social at the hack weekend social HQ, there was plenty of beer and the room was packed.

Saturday morning I took the train back downtown for the first day of hacking. We had local OSM developers and out of town hackers from three different countries participate along with a group of Ryerson Geography students. I spent some time Saturday morning helping someone get the rails port working on their OSX laptop. The part they seemed to have the most difficulty with was getting PostgresSQL running. This tells me two things. First that the work people in the OSM community have done to make the rails port easier to install has paid off and some PostgreSQL binary distributions could make their user experience a bit better.

I had suggested the EDB one-click installer for OSX but they opt’d for the KyngChao binaries. They were okay installing the package but they weren’t clear how to startup PostgreSQL and connect to it. I feel that the EDB installer makes it easier for an inexperienced PostgreSQL user to get up and running.

In addition to helping people with installing PostgreSQL I also helped some people with SQL queries and helped with changes to the website look cleaner for users logged in but haven’t uploaded any GPX traces.

Saturday night we had another party at social HQ. Party highlights included beer with ingredients from seven continents, comparision of political systems, talk of election rigging. Thai food, wine and beer were also spotted. Most of the geography students from the hack sessions were able to make it out and saw how fun OSM parties can be. I hope to see them at the regular Toronto OSM pub nights.

The GO train back to Oakville was pretty full. This was St. Patricks day and the streets of Toronto were filled with drunk people in green. The 11:43pm train to the suburbs is a bit too early for serious leprecons. Many of the riders had been at a Van Halen concert, it didn’t seem like many of them were interested in making maps.

There is a saying in the OpenStreetMap world that if you want to build an local OpenStreetMap community you should import a bunch of Germans. If you can’t import some Germans the next best thing is to import Richard Weait. He almost single-handedly organized the Toronto hack weekend, secured a venue and provided food, drinks and social entertainment. I also want to thank the Ryerson Geography department and Claus Rinner for letting us use their facilities and providing eager GIS students.

The neighborhood I live in is between 20 and 30 years old, other parts of Oakville are under constant growth. When a new sub-division is built the developer will put in sewers, roads and start building houses. Sometimes the street signs will look home-made other times proper street signs will already be up.

When I see a new subdivision going up near my I try drive (or walk) the roads with my GPS capturing the data for OpenStreetMap. Often my GPS traces in OpenStreetMap will be the first time these roads show up on a map outside of the towns GIS. It might take years before some of the commercial maps send trucks to map these roads.

On Sunday I was driving past a sub-division that was being built and decided to map some of the roads. I got about GPS traces and names for about half of the roads. I need to make a point of going back and finishing the soon. Sub-division construction sites usually have construction workers and heavy equipment moving about. It is important to pay close attention to your surroundings and the condition of the road your mapping often it is a work in progress.

I doubt anyone is going to be moving into these houses until the spring. In the spring when people move into their new houses they will be able to use OpenStreetMap to invite there friends to the house-warming party.

]]>https://scanningpages.wordpress.com/2011/11/17/virgin-roads/feed/1scanningpagesDSC_0400.1PostGIS Replication @ FOSS4Ghttps://scanningpages.wordpress.com/2011/09/18/postgis-replication-foss4g/
https://scanningpages.wordpress.com/2011/09/18/postgis-replication-foss4g/#commentsSun, 18 Sep 2011 21:42:23 +0000http://scanningpages.wordpress.com/?p=413]]>My talk on PostGIS replication at FOSS4G 2011 went well. It looked like there were about 150 people in the room. Most of them had not yet deployed a PostGIS replication solution.

My talk covered Slony and streaming replication. It gave an overview of different replication patterns that can crop on in the GIS space. I then gave an overview of the key features and limitation of Slony and streaming replication.

]]>https://scanningpages.wordpress.com/2011/09/18/postgis-replication-foss4g/feed/0scanningpagesBig Tent Conferenceshttps://scanningpages.wordpress.com/2011/09/17/big-tent-conferences/
https://scanningpages.wordpress.com/2011/09/17/big-tent-conferences/#commentsSat, 17 Sep 2011 22:49:50 +0000http://scanningpages.wordpress.com/?p=401]]>This past week I attended FOSS4G in Denver, a conference run by the open-source geospatial foundation (OSGEO) that also happened to be the largest PostgreSQL conference in North America.

FOSS4G is a big tent conference that attracted about 900 attendees from all over the world with over 400 of them from the United States, over 50 from Canada.
A big tent conference is a conference is similar to a big-tent political party. The idea is appeal to a broad base of people by catering to as many groups as possible while keeping true to some key common themes. Out of the 900 people at the conference I’d say 75% raised their hands when asked if it was their first FOSS4G. Last years FOSS4G was held in Europe and also had over 800 attendees.
At FOSS4G the common theme is open source geo-spatial software. This includes open-source desktop GIS systems like QGIS and GRASS. It includes open source map rendering platforms like Mapserver and Mapnik. It also includes open source spatial databases such as PostGIS and MongoDB. The tent is also big enough to include commercial vendors such as ESRI, Safe(authors of FME) who produce commercial products that interface with some open-source geospartial software. Many of the software packages (open-source and commercial) represented at the conference compete with each other but they are able to share the same tent. At the opening reception one attendee (who I won’t name) said “I’m need to find Chris Schmidt and tell him why OpenLayers sucks” conferences that get everyone under one roof for a week allow us to learn from each other so we can improve our software, our documentation and our communities. It is much harder to have a flame-war on a mailing list with someone you have sat down and had a beer with than it is to have a flame-war with someone you have never met.

At FOSS4G almost everyone I spoke to used PostGIS in some fashion, some of them used couchDB or MongoDB for tasks, others had legacy Oracle Spatial or ESRI systems but everyone was aware of PostGIS and most had used it in some fashion. If you are running PostGIS then you are running PostgreSQL. FOSS4G 2011 was the largest collection of PostgreSQL users under one roof in North America to date. As a member of the PostgreSQL community I feel we need to do a better job at making sure that geospatial types feel welcome in our community. Excluding the core PostGIS developers I recognized less than 5 people from the PostgreSQL community at the conference and most of us were there because of our connections to other communities. Having said that, most people I spoke to said that they loved PostGIS and PostgreSQL.

The flight home from FOSS4G is also a good time to reflect on what it means to be a community conference.

Each year OSGEO asks for submissions/proposals from the community on groups that want to host the next FOSS4G. An OSGEO committee examines these submissions and selects a host city for the next conference.

OSGEO moves the conference around the world each year. They won’t hold the conference in the same continent two years in a row. This makes it easier for people in different parts of the world too attend. It also means that when the conference is in your backyard you make a special effort to attend because it probably isn’t coming back for at least three years.

FOSS4G has a call for papers, the papers are then voted on by the wider community as an input into the paper selection processes. Anyone in the community can have input into which talks are selected.

After FOSS4G a financial report on the conference along with many other documents used in planning the conference are published in a subversion repository. This acts as a resource for future groups considering hosting FOSS4G. It also means that the revenues and expenses are public so people can can see where the money goes.

FOSS4G does not just appeal to developers. In addition to developers and sysadmins there where many cartographers and geographers at the conference

Next year FOSS4G will be held in Bejing. Various political, cultural, logistical and financial reasons will make it hard for many of the American attendees to go to FOSS4G. Some people in Denver were talking about how there should be a North American FOSS4G next year as well. My thoughts on this are

It is important to not dilute the FOSS4G brand, FOSS4G is the big geo-spatial conference that travels around the world. If you start splitting the FOSS4G brand you will soon end up with a FOSS4G-EU, FOSS4G-US , FOSS4G-Asia instead of a FOSS4G. Other communities do this (*BSD as an example). Figure out the relationship between FOSS4G, OSGEO and regional conferences before someone announces FOSS4G-US, maybe these regional conferences should be called something other than FOSS4G

If you are thinking about creating another North American geo-spatial conference think about what other communities your conference can reach out to (OSM, PostgreSQL, Python, etc…). State of the Map was fun for a lot of the same reasons PGCon is fun. You get to spend a few days meeting and hanging out with people that you regularly chat with online and everyone shares a deep passion of a very specific project. A lot of the fun I had with FOSS4G was from the energy of being at such a large conference.

If you do hold some sort of regional conference, for god sakes don’t hold the conference two weeks before the Bejing conference. Space the conferences out so that it is easier for speakers, attendees and sponsors to attend both. I think the next year will turn out to be a great year for the struggling OpenStreetMap community in the US because of the interest and enthusiasm generated at SOTM and FOSS4G. I hope FOSS4G in Bejing brings a lot more great Asian contributors into the open source geospatial tent next year.

It feels like the plane is descending. I will try to send this off once I’m home.