This course will show how one can treat the Internet as a source of data. We will scrape, parse, and read web data as well as access data using web APIs. We will work with HTML, XML, and JSON data formats in Python. This course will cover Chapters 11-13 of the textbook “Python for Everybody”. To succeed in this course, you should be familiar with the material covered in Chapters 1-10 of the textbook and the first two courses in this specialization. These topics include variables and expressions, conditional execution (loops, branching, and try/except), functions, Python data structures (strings, lists, dictionaries, and tuples), and manipulating files. This course covers Python 3.

강사:

Charles Russell Severance

Professor

스크립트

Hello and welcome to Python for everybody. We are doing some code samples here. If you want to follow along you can download the sample code or it's in the big zip file. I've got it. We are going to be working with the Google Maps API. In the old days, this Maps API was free and did 2,500 requests per day. But now, they've made it so that parts of it are behind API keys and you start over using OAuth and stuff but they haven't put it all behind this one address service that we've been using, that continues to work. The basic idea of an API as you go read the documentation, you find a URL and this is going to Google servers and you pass in the address. We have to pass in the address using what's called URL encoding. So, spaces are pluses. That's a comma and then that's a space. So, we have to pass this in a certain way. But if we do it right, we hit this we're going to get ourselves some JSON back, and that's really cool. So, deep inside here we get the real address. A good address. We get a geometry. We have the location. We got the latitude and longitude. We can extract stuff out of here. So, we're talking and this one here is still rate limited to 2,500, but it's one of the few parts of the Google Maps API that is not hidden behind an API key. In a later chapter, we'll show you how to actually talk with the API key in the geodata code. The geoload, shows you how to use an API key if you want to jump ahead and take a look at that. But for now, we're just going to take a look at GeoJSON which is going to retrieve one page and tear it apart. So, let's take a look. So, we're going to grab the urllib stuff and import JSON. So now, we're going to use JSON but we're going to actually pull the data out of the Internet. So, I just take that service URL for Google Maps API. I found that somewhere in documentation. Then I'm going to have a loop that's going to run forever. I'm going to add for the add the location. Then if I hit enter that's what this is saying, get out of the loop. Then what I'm going do is, I'm going to concatenate the the service URL which is this. This URL parse URL encode gives a dictionary of address equals. This bit right here gives me the string that leads to putting this address equals, but they're encoding these spaces the right way. So, if you type a space that bit of code turns it into the plus. So, that's important and I've got the question mark sitting here at the end of that. Then what we're going to do is, we're just going to do a urlopen to get a handle. We're going to read the whole document and because it's UTF-8 coming from the outside world and we want it turned into unicode inside our application, we say.decode. We can ask how many characters we got and we put our JSON loads. Now, up till now we've been just doing loads's from internal strings. But this is now a string that came from the outside world. We'll put a try except in and we'll set js to be non and that'll be our little trigger. They give us, we take a look at the output. They give us this okay and that status can be a problem and it can complain about things. So, we have to check to see if we got a good status. So, at this point, if you look at the outer bit of this, the outer bit that we get is a curly brace so it's an dictionary. Then there is within that dictionary and key results which is a list. But then the second thing in the outer dictionary is status. So, we can ask if we got a fall. So, forgotten nothing that will quit. If we don't have a status key in that job and that object or that dictionary or it's not equal to okay, any number of those things. If this, or this, or this, are all either of those are true, we're going to quit for failure to retrieve and print the data out. When you start to restuff on the net, you often have to put debugging in here like this, something quit I got to figure out and so debugging. Next thing we're going to do is called JSON dumps which is the opposite of loads which takes this array that dictionary and includes arrays and we're going to pretty print it with an indent of four and then we're going to print that out. So, if you look at my code we'll see that the first thing we do once we parsed it as we print it back out so we can see it. Then we're going to dig into it. So, let's go ahead and run this code. Python geojson.py. One of these days I will always type Python3 Ann Arbor Michigan, okay. So it ran, and so you see that it retrieve this URL. This URL was constructed and retrieved 1736 characters and it's JSON pretty printed with an indent of four. This is that JSON dumps all the way down to here. So, that's just JSON dumps. Then it starts extracting. So, it's going to pull things out. Now, when you write this code is really easy to look at this and say, "Oh, great it's easy." I tend to have to print this stuff out over, and over, and over, and as I construct this expression. But if we look at it, the outer dictionary sub results leads to this array. If you go look at this array careful, you find there is only one thing in it. So, that the results is an array sub zero gets us this dictionary. I keep on I saying object because that's what it's called and that goes all the way down to here. So, that's what we get there. Then within that, we now have an object and we look for geometry within that object, where is geometry? Right there, geometry. Geometry goes from there to there. There's geometry in there. Got here used to it that's why it's nice to have this stuff indented. Geometry sub low. Oops come back, come back and then we go to location within that, so jump location within geometry. Then well then let's job location we have lat and long. So, this is pulling out this 42 and 83. Then, so we print that out. Take a look, and that prints that out. Pulls that right out of the JSON. These are tricky to write but after awhile you win and you get it right and it's just fine, okay. So, we do the same thing results sub zero formatted address gets us this. So, that's how we print the location out. So, that's a real quick look at how we would do that with the JSON talking to the Google Maps API. Okay, hope this helps.