Friday, June 22, 2012

R and the web (for beginners), Part II: XML in R

This second post of my little series on R and the web deals with how to access and process XML-data with R. XML is a markup language that is commonly used to
interchange data over the Internet. If you want to access some online data over
a webpage's API you are likely to get it in XML format. So here is a very simple example
of how to deal with XML in R.

Duncan Temple Lang wrote a very helpful R-package which makes it quite easy to parse, process and generate XML-data with R. I use that package in this example. The XML document (taken from w3schools.com) used in this example describes a fictive plant catalog. Not that thrilling, I know, but the goal of this post is not to analyze the given data but to show how to parse it and transform it to a data frame. The analysis is up to you...

How to parse/read this XML-document into R?

# install and load the necessary package

install.packages("XML")

library(XML)

# Save the URL of the xml file in a variable

xml.url <- "http://www.w3schools.com/xml/plant_catalog.xml"

# Use the xmlTreePares-function to parse xml file directly from the web

xmlfile <-
xmlTreeParse(xml.url)

# the xml file is now saved as an object you can easily work with in R:

One can already assume how this data should look like in a matrix or data frame. The goal is to extract the XML-values from each XML-tag <> for all $PLANT nodes and save them in a data frame with a row for each plant ($PLANT-node) and a column for each tag (variable) describing it. How can you do that?

# To extract the XML-values from the document, use xmlSApply:

plantcat <- xmlSApply(xmltop, function(x) xmlSApply(x, xmlValue))

# Finally, get the data
in a data-frame and have a look at the first rows and columns

I'm not very familiar with the RHTMLForms-package, thus I might be the wrong guy to answer this question. Nevertheless, I guess the problem occurs already in your application of createFunction(), with your code I get from that line:

A good general starting point is to use Firebug (a Firefox extension) to inspect the website with the data you are interested in.

What you refer to in your example as "embedded spreadsheet" seems to be in the end a HTML-table (for which the same techniques as described in my post on web scraping should work: http://giventhedata.blogspot.com/2012/08/r-and-web-for-beginners-part-iii.html)

Mind though that scraping data from a web site, such as in your example, is often a lot more tricky than querying/extracting data from a XML-document.

you've correctly pointed out that the XML package also comes with a convenient function (xmlToDataFrame) to "extract data from a simple XML document". There are mainly two reasens why I didn't want to point to that function in this post:

1) if you are a novice in xml/R you don't learn anything by just using xmlToDataFrame in the above example. The explicit aim of the post is to give some insights into how one can work with XML documents in R.

2) as the documentation of xmlToDataFrame mentions, this function is made for "simple" XML documents. You will notice what this means as soon as your trying to use xmlToDataFrame in a more complex xml structure as the very simple example above.

a third, rather minor point is that even if xmlToDataFrame works in your setting it is likely to be less efficient than a self-made function written with the functions pointed out in the example.

anyway, thanks for pointing this out! mentioning the convenient function as concluding remarks in my post would not have been a bad idea.

Free online HTML tutorial for beginners with examples - HTML tutorial will help you in creating website, after study the tutorial you will just one step ahead of creating your own website. HTML is easy to understand and you will enjoy it to learn. HTML tutorial contains hundreds of examples to better understand.

About

I use this blog to share my experiences in learning and applying R in my daily research as a PhD-student in the field of Political Economics. In particular, I post lines of code and small functions that proofed to be helpful in my work and/or might be inspiring to people getting to know R.