Comments for CommonCrawl http://commoncrawl.org Thu, 16 Feb 2012 02:53:00 +0000 hourly 1 http://wordpress.org/?v=3.3.1 Comment on Answers to Recent Community Questions by Audris http://commoncrawl.org/answers-to-recent-community-questions/#comment-426 Audris Thu, 16 Feb 2012 02:53:00 +0000 http://www.commoncrawl.org/?p=12082#comment-426 no. I wanted to pursue http://matpalm.com/blog/2011/12/10/common_crawl_visible_text/ but had no chance to do that yet. There appears to be a mention of first copying "The first thing was to get the data into a hadoop cluster. It's made up of 300,000 100mb gzipped arc files stored in S3. I wrote a dead simple distributed copy to do this." and: "Pulling from S3 to EC2 is network bound so I ran using the MultithreadedMapRunner to ensure I could get as much throughput as possible." no. I wanted to pursue http://matpalm.com/blog/2011/12/10/common_crawl_visible_text/
but had no chance to do that yet. There appears to be a mention of first copying
“The first thing was to get the data into a hadoop cluster.
It’s made up of 300,000 100mb gzipped arc files stored in S3.
I wrote a dead simple
distributed copy to do this.”
and:
“Pulling from S3 to EC2 is network bound so I ran using the
MultithreadedMapRunner to ensure I could get as much throughput as possible.”

]]> Comment on Answers to Recent Community Questions by Marcus http://commoncrawl.org/answers-to-recent-community-questions/#comment-424 Marcus Sat, 11 Feb 2012 19:55:00 +0000 http://www.commoncrawl.org/?p=12082#comment-424  Did you determine how to increase throughput?  Did you determine how to increase throughput?

]]> Comment on SlideShare: Building a Scalable Web Crawler with Hadoop by Building a Scalable Web Crawler with Hadoop « Another Word For It http://commoncrawl.org/slideshare-building-a-scalable-web-crawler-with-hadoop/#comment-421 Building a Scalable Web Crawler with Hadoop « Another Word For It Fri, 27 Jan 2012 21:31:35 +0000 http://www.commoncrawl.org/?p=12456#comment-421 [...] Building a Scalable Web Crawler with Hadoop [...] [...] Building a Scalable Web Crawler with Hadoop [...]

]]> Comment on Common Crawl on AWS Public Data Sets by Stephan http://commoncrawl.org/common-crawl-on-aws-public-data-sets/#comment-420 Stephan Fri, 27 Jan 2012 06:16:00 +0000 http://www.commoncrawl.org/?p=12491#comment-420 Bittorrent is not an option? :-) Bittorrent is not an option? :-)

]]> Comment on MapReduce for the Masses: Zero to Hadoop in Five Minutes with Common Crawl by snyff http://commoncrawl.org/mapreduce-for-the-masses/#comment-419 snyff Thu, 26 Jan 2012 05:39:00 +0000 http://www.commoncrawl.org/?p=12242#comment-419 If someone is interrested by having a direct access to the data, you can use aws(https://github.com/timkay/aws): % perl aws get "x-amz-request-payer:requester"   commoncrawl-crawl-002/2010/01/07/18/1262876244253_18.arc.gz > 1262876244253_18.arc.gz The file is around 100Mo If someone is interrested by having a direct access to the data, you can use aws(https://github.com/timkay/aws):

% perl aws get “x-amz-request-payer:requester”   commoncrawl-crawl-002/2010/01/07/18/1262876244253_18.arc.gz > 1262876244253_18.arc.gz

The file is around 100Mo

]]> Comment on MapReduce for the Masses: Zero to Hadoop in Five Minutes with Common Crawl by State of Data #82 « Dr Data's Blog http://commoncrawl.org/mapreduce-for-the-masses/#comment-416 State of Data #82 « Dr Data's Blog Fri, 20 Jan 2012 06:45:32 +0000 http://www.commoncrawl.org/?p=12242#comment-416 [...] #architecture – Zero to Hadoop in 5 minutes [...] [...] #architecture – Zero to Hadoop in 5 minutes [...]

]]> Comment on MapReduce for the Masses: Zero to Hadoop in Five Minutes with Common Crawl by George http://commoncrawl.org/mapreduce-for-the-masses/#comment-415 George Thu, 19 Jan 2012 03:16:00 +0000 http://www.commoncrawl.org/?p=12242#comment-415 Great step by step. The hardest part was working through the constrains of case and spaces. Once I learned to use all lower case and no spaces all went smooth. My process ran 12 for minutes on the example archive. Great step by step. The hardest part was working through the constrains of case and spaces. Once I learned to use all lower case and no spaces all went smooth. My process ran 12 for minutes on the example archive.

]]> Comment on MapReduce for the Masses: Zero to Hadoop in Five Minutes with Common Crawl by Peter Quirk http://commoncrawl.org/mapreduce-for-the-masses/#comment-412 Peter Quirk Sat, 14 Jan 2012 00:13:00 +0000 http://www.commoncrawl.org/?p=12242#comment-412 I got this working a couple of weeks ago with the only problem being that the bucket name needed to be all lower-case. The output data format is supposed to be CSV but it looks like no CSV format I've ever seen. The record separator is single-quote and the field separator is comma. most CSV-compatible tools cannot process this format. Is there a way to condition the output to be more easily digested by other tools? I got this working a couple of weeks ago with the only problem being that the bucket name needed to be all lower-case. The output data format is supposed to be CSV but it looks like no CSV format I’ve ever seen. The record separator is single-quote and the field separator is comma. most CSV-compatible tools cannot process this format. Is there a way to condition the output to be more easily digested by other tools?

]]> Comment on MapReduce for the Masses: Zero to Hadoop in Five Minutes with Common Crawl by Luke Stanley http://commoncrawl.org/mapreduce-for-the-masses/#comment-411 Luke Stanley Fri, 13 Jan 2012 15:42:00 +0000 http://www.commoncrawl.org/?p=12242#comment-411 I'm very grateful for this how to, and at the same time I'm frustrated that it's 5 minutes of intense clicking and prodding just to have a few java files run on Amazon. (Albeit manipulating some amazing data sets). It would have been good to see a little coverage of the code and it's output.If it was a Sikuli script or a shell script to setup it would be easier too. Surely there are tools that allow command line deployment and show the output in real time too? I really thought this would be for the masses, it isn't clear that it is yet.If I get around to playing with this it would be interesting to compare how long it ACTUALLY takes to do the same thing as the video from start to finish, and to perhaps screen capture THAT more authentic process (despite ).There must be some web app alternative where code can just be edited and deployed without fuss? (Just given Amazon login data etc.) What about Jython also? I’m very grateful for this how to, and at the same time I’m frustrated that it’s 5 minutes of intense clicking and prodding just to have a few java files run on Amazon. (Albeit manipulating some amazing data sets).
It would have been good to see a little coverage of the code and it’s output.If it was a Sikuli script or a shell script to setup it would be easier too.
Surely there are tools that allow command line deployment and show the output in real time too?
I really thought this would be for the masses, it isn’t clear that it is yet.If I get around to playing with this it would be interesting to compare how long it ACTUALLY takes to do the same thing as the video from start to finish, and to perhaps screen capture THAT more authentic process (despite ).There must be some web app alternative where code can just be edited and deployed without fuss? (Just given Amazon login data etc.) What about Jython also?

]]> Comment on MapReduce for the Masses: Zero to Hadoop in Five Minutes with Common Crawl by MapReduce for the Masses: Zero to Hadoop in Five Minutes with Common Crawl « Another Word For It http://commoncrawl.org/mapreduce-for-the-masses/#comment-410 MapReduce for the Masses: Zero to Hadoop in Five Minutes with Common Crawl « Another Word For It Fri, 13 Jan 2012 00:32:31 +0000 http://www.commoncrawl.org/?p=12242#comment-410 [...] MapReduce for the Masses: Zero to Hadoop in Five Minutes with Common Crawl [...] [...] MapReduce for the Masses: Zero to Hadoop in Five Minutes with Common Crawl [...]

]]>