Host- and Domain-Level Web Graphs Nov/Dec/Jan 2019 – 2020

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, December 2019 and January 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which host all scripts and tools required to construct the graphs.

Host-level graph

The graph consists of 1.24 billion nodes and 4.54 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 1.17 billion dangling nodes (95%) and the largest strongly connected component contains 45 million (3.6%) nodes.

You can download the graph and the ranks of all 1.24 billion hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-20-nov-dec-jan/host/. Alternatively, you can use https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2019-20-nov-dec-jan/host/ as prefix to access the files from everywhere.

Download files of the Common Crawl Nov/Dec/Jan 2019-20 domain-level webgraph

SizeFileDescription
7.23 GBcc-main-2019-20-nov-dec-jan-host-vertices.paths.gznodes ⟨id, rev host⟩, paths of 12 vertices files
20.16 GBcc-main-2019-20-nov-dec-jan-host-edges.paths.gzedges ⟨from_id, to_id⟩, paths of 24 edges files
8.42 GBcc-main-2019-20-nov-dec-jan-host.graphgraph in BVGraph format
2 kBcc-main-2019-20-nov-dec-jan-host.properties
10.80 GBcc-main-2019-20-nov-dec-jan-host-t.graphtranspose of the graph (outlinks inverted to inlinks)
2 kBcc-main-2019-20-nov-dec-jan-host-t.properties
1 kBcc-main-2019-20-nov-dec-jan-host.statsWebGraph statistics
16.32 GBcc-main-2019-20-nov-dec-jan-host-ranks.txt.gzharmonic centrality and pagerank

Note that the host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 85.8 million nodes and 1.9 billion edges. 51% or 44 million nodes are dangling nodes, the largest strongly connected component covers 34 million or 39% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-20-nov-dec-jan/domain/ resp. https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2019-20-nov-dec-jan/domain/.

Download files of the Common Crawl Nov/Dec/Jan 2019-20 domain-level webgraph

SizeFileDescription
0.59 GBcc-main-2019-20-nov-dec-jan-domain-vertices.txt.gznodes ⟨id, rev domain, num hosts⟩
7.65 GBcc-main-2019-20-nov-dec-jan-domain-edges.txt.gzedges ⟨from_id, to_id⟩
4.10 GBcc-main-2019-20-nov-dec-jan-domain.graphgraph in BVGraph format
2 kBcc-main-2019-20-nov-dec-jan-domain.properties
4.13 GBcc-main-2019-20-nov-dec-jan-domain-t.graphtranspose of the graph
2 kBcc-main-2019-20-nov-dec-jan-domain-t.properties
1 kBcc-main-2019-20-nov-dec-jan-domain.statsWebGraph statistics
1.86 GBcc-main-2019-20-nov-dec-jan-domain-ranks.txt.gzharmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 86 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Nov/Dec/Jan 2019-2020)

harmonic
centrality
rank
hc valuepage rankpage rank
value
reversed hostname
13059839810.019072com.googleapis
22911313630.012214com.facebook
32747513820.013236com.google
42561048040.007452com.twitter
52494712650.007174org.w
62490471260.006611com.youtube
72328150490.004269com.instagram
82244629670.005561org.gmpg
92215475080.005033com.googletagmanager
1022107784130.003001com.linkedin
1121307220100.003433org.wordpress
1221290688200.001717com.gravatar
1321096944110.003266com.cloudflare
1421086168230.001516com.pinterest
1520869868160.002242com.gstatic
1620855286150.002366com.wordpress
1720713268250.001234org.wikipedia
1820641712170.002130com.apple
1920584368140.002460com.bootstrapcdn
2020371046330.001102com.vimeo
2120336732420.000833com.blogspot
2220237058180.001787com.jquery
2320220572500.000732be.youtu
2420218806320.001130com.microsoft
2520119820490.000737com.wp
2620059828190.001764com.adobe
2720029544520.000709com.amazon
2820013346440.000784gl.goo
2919972608350.001020com.amazonaws
3019941934670.000471com.tumblr
3119937390640.000518ly.bit
3219862906290.001164com.macromedia
3319817494300.001151com.baidu
3419816382380.000958com.google-analytics
3519808066310.001142com.googlesyndication
3619781460340.001092net.cloudfront
3719770754240.001254ru.yandex
3819749302530.000693com.flickr
3919700112220.001568com.github
4019698814800.000368com.yahoo
4119676534580.000644eu.europa
42196325061150.000287com.reddit
4319603190410.000918com.addthis
4419561006720.000403com.weebly
4519559908430.000823org.w3
4619550116630.000524me.wp
47195466141080.000313com.googleusercontent
4819531876450.000777io.github
49195232241820.000140org.wikimedia
5019521602700.000422com.medium
5119516098470.000743org.schema
5219496222460.000754net.jsdelivr
5319495662760.000374org.creativecommons
54194726101730.000153com.imgur
5519459848360.000989net.doubleclick
5619451512510.000711com.wix
57194160541860.000138uk.co.bbc
58194083121490.000181com.forbes
5919405426930.000332com.weibo
6019404990600.000601co.t
6119398622280.001192com.fontawesome
6219382484480.000741com.paypal
63193759142100.000114com.cnn
64193722921440.000190org.archive
6519368680610.000583org.mozilla
66193481821890.000137net.sourceforge
67193047743150.000079edu.mit
68193029361740.000152com.theguardian
69192937822780.000089edu.harvard
70192833721790.000144com.bing
71192713901170.000286com.jimdo
72192703681400.000202com.nytimes
7319260752270.001195com.qq
74192594062370.000103com.wsj
7519257754210.001581org.apache
7619256254560.000658com.googleadservices
77192411102150.000111com.washingtonpost
78192378162690.000092com.bloomberg
79192371642590.000094com.techcrunch
80192105625440.000049com.deviantart
81192059461620.000162org.ietf
82191983462470.000098com.oracle
83191946962790.000089com.android
8419193448690.000430com.list-manage
85191749683970.000065com.ted
86191678203210.000078com.reuters
87191619643120.000080com.wired
88191587681540.000173com.wixsite
89191548743410.000073com.ft
90191520683300.000076uk.co.telegraph
91191516664270.000060com.theverge
92191450781550.000172gov.nih
93191438862720.000091com.myspace
94191421723770.000068gov.nasa
95191360222910.000085com.bbc
96191261383390.000074com.example
97191199162390.000103org.python
9819117868820.000361com.whatsapp
99191115501220.000267com.unpkg
100191103461880.000138uk.co.google
101190895047070.000041com.economist
102190887662550.000095com.appspot
103190823003840.000067uk.co.dailymail
104190819042090.000115org.gnu
105190817082620.000093com.githubusercontent
106190801821310.000232com.ytimg
107190771563200.000078org.un
108190758061630.000162com.giphy
109190688603980.000065com.latimes
110190676061690.000157com.twimg
111190664044310.000060com.googleblog
112190561621760.000148com.blogger
113190543102320.000104com.dribbble
114190528242070.000115com.npmjs
115190505245640.000047org.arxiv
116190451946660.000042edu.upenn
117190427701710.000154com.eventbrite
118190366123790.000068com.springer
119190324222770.000090org.ampproject
120190313545570.000047com.gitlab
121190256165960.000045com.vice
122190252362060.000116com.disqus
1231902397810360.000031com.hatenablog
124190234068350.000039edu.columbia
125190186708180.000040io.readthedocs
126190082802050.000116me.t
127190058423900.000066com.w3schools
128190040309410.000034org.chromium
129190039004180.000062com.nature
130190014527160.000041com.slate
131190008621570.000171jp.co.yahoo
132189975663250.000077com.time
133189971644300.000060com.statista
134189903367440.000040com.ubuntu
135189854661580.000167com.yelp
136189825506320.000043org.worldbank
137189824281430.000191com.spotify
138189810783470.000072com.skype
139189786609350.000034com.playstation
140189765243060.000082com.fc2
1411897331213040.000024org.coursera
142189694381210.000281com.stripe
143189685628160.000040com.qz
144189681686170.000044com.git-scm
145189664484860.000053uk.co.independent
146189658101990.000126com.eepurl
147189644449610.000034com.500px
148189643424050.000063net.researchgate
149189626542410.000101com.bandcamp
15018959274550.000669net.facebook
151189564783890.000066com.outlook
152189554902290.000105com.unsplash
153189504606310.000043com.mysql
154189493304190.000062com.theatlantic
155189484561160.000286com.soundcloud
156189483641800.000143com.amazon-adsystem
157189476701230.000259org.networkadvertising
158189407685710.000046org.bitbucket
1591894029811630.000027com.jetbrains
160189364744080.000063com.mozilla
161189361044980.000052com.nationalgeographic
162189329063160.000079com.usatoday
163189308904390.000059com.criteo
164189273428370.000039uk.ac.ox
165189255604680.000054com.fortune
166189249564660.000055com.pixabay
1671892222412780.000024uk.co.thesun
168189214762300.000104net.behance
1691891670015470.000019com.amd
170189155748220.000039com.evernote
17118909918400.000932com.vk
172189093187990.000040com.about
173189079105050.000051uk.co.blogspot
1741890400011930.000026se.haxx
175189036862510.000097gle.forms
176189001327190.000041com.docker
177189000209670.000033uk.co.guardian
178188995763030.000082org.doi
179188983304970.000052me.about
180188968625250.000050gg.discord
1811889443213830.000022com.instructables
182188915821470.000189com.dropbox
1831888843210000.000032com.scientificamerican
184188852283320.000076jp.co.rakuten
185188815489030.000036google.blog
186188757041940.000130com.feedburner
1871887075810810.000030org.altervista
188188690686340.000043org.unesco
1891886896610820.000030org.eclipse
190188684882750.000090gov.ca
191188665528750.000037jp.livedoor
1921886516414900.000020org.phys
193188646742920.000085com.sciencedirect
194188606742120.000113jp.ameblo
195188557005720.000046gov.loc
196188548929930.000033org.cambridge
197188455124030.000064ca.google
198188425586630.000042edu.washington
19918836168710.000408net.slideshare
200188328085430.000049com.cisco
2011882983813640.000023edu.rutgers
202188288083040.000082com.nbcnews
203188272942430.000099ru.rambler
204188265948230.000039au.net.abc
205188260349980.000032uk.co.thetimes
2061882555414720.000021com.bankofamerica
20718821912620.000549com.fb
208188216708630.000037org.sciencemag
209188138448870.000036com.speakerdeck
210188109864120.000063jp.ne.sakura
211188067482200.000109org.iana
2121880211212580.000025com.wikidot
2131879970616500.000018pt.sapo
214187928809050.000036uk.co.mirror
215187906202180.000110edu.stanford
2161878773411430.000028org.kernel
217187865264500.000057com.elsevier
2181878442014520.000021edu.osu
2191878344616220.000019com.googlesource
2201878242613950.000022com.vogue
22118781902740.000401net.akamaihd
222187812588420.000038gov.fcc
2231878092612400.000025ms.1drv
2241877910613770.000022edu.asu
225187788282540.000095com.businessinsider
226187785348440.000038co.ibb
2271877838618430.000016com.wolfram
228187710907210.000041com.trello
229187691481060.000323com.paypalobjects
230187662983010.000083net.windows
2311876223812160.000026jp.geocities
232187621505870.000045com.box
233187618408760.000037com.sciencedaily
234187588022270.000106com.wpengine
235187529864930.000052com.herokuapp
2361875202610300.000032edu.princeton
237187488149150.000035edu.academia
238187482322840.000087com.googlecode
2391874614611770.000027com.asahi
2401874004413750.000022com.newscientist
2411873371814280.000021blog.home
242187324583140.000080com.tinyurl
243187320105160.000051com.udacity
2441873174021030.000014com.wizards
245187299063860.000067com.cnet
2461872964414210.000022com.ndtv
247187293703350.000075com.getpocket
2481872927613490.000023com.fandom
2491872641212410.000025net.seesaa
250187233761670.000158com.imdb
251187184403330.000076org.debian
252187155526990.000041site.business
253187136382610.000093com.live
2541871149810590.000031jp.ne.goo
2551871077815040.000020io.itch
2561870401611550.000027org.greenpeace
2571870225410470.000031com.netlify
2581870156818080.000017net.pixnet
259187005524020.000064com.squareup
2601869985211450.000028co.elastic
261186997563100.000081com.ibm
262186987842030.000118com.stackoverflow
263186981464940.000052com.indiatimes
2641869631234200.000010com.armorgames
265186931482650.000092com.aliyuncs
266186923782260.000106com.optimizely
2671868594417770.000017uk.co.timesonline
268186795709750.000033com.mixcloud
2691867753016180.000019com.itv
270186744601980.000128org.bbb
27118674340570.000648net.fbcdn
2721867414822360.000014com.opendns
2731867268233020.000010tw.com.gamer
274186726464060.000063com.go
275186724864370.000059com.msn
27618672432370.000966com.wixstatic
2771866760614850.000021org.archlinux
278186665464480.000058org.pewresearch
27918664590870.000345com.shopify
280186629148380.000039jp.shinobi
281186625509950.000032com.bmj
2821866051615790.000019com.diigo
283186596881600.000167com.opera
2841865933218220.000017com.youdao
2851865768612590.000025com.angelfire
286186573769060.000036jp.naver
2871865693410380.000031com.thelancet
2881865361015850.000019uk.bl
289186521868640.000037br.com.google
290186499144900.000053com.bigcartel
2911864879610070.000032com.sky
2921864759614080.000022net.daringfireball
2931864259217790.000017uk.ac.kcl
2941864116817520.000017org.maven
295186395445340.000049me.m
2961863929811070.000029com.reverbnation
2971863729016650.000018net.cnki
2981863410210890.000030com.theconversation
299186305044600.000056it.placehold
3001862906411170.000029com.podbean
301186286608810.000036org.fao
3021862695611440.000028co.g
3031862626413700.000023com.dw
304186236461190.000283com.mailchimp
3051862126016310.000019jp.ne.so-net
3061862026413460.000023com.livescience
3071862011020710.000015edu.kit
3081861190013080.000024ca.utoronto
3091861061813100.000024com.webnode
310186083889570.000034au.gov.nsw
3111860696414110.000022com.citrix
3121860370410310.000032jp.jugem
3131860237411150.000029gov.wa
314186000024580.000056com.quora
31518598840990.000326com.godaddy
3161859774013340.000023com.bloglovin
3171859683411940.000026com.serving-sys
318185959769000.000036gov.dhs
3191859512016300.000019org.edx
320185926962440.000099me.wa
3211859258218300.000017com.pearltrees
3221859202215760.000019com.twitpic
3231859174417760.000017cn.people
3241859035011710.000027com.britannica
3251858938018930.000016sg.edu.nus
3261858813017910.000017com.kinja
3271858740422960.000013com.authorstream
3281858592215660.000019ca.mcgill
329185851323800.000068com.kickstarter
3301858163612370.000025com.lulu
3311857694825910.000012com.colourlovers
3321857506419190.000016com.hm
333185742423370.000075com.rackcdn
3341856799422610.000013uk.ac.sussex
3351856395419450.000016org.vim
3361856294610630.000031com.healthline
3371856286019110.000016org.wikibooks
3381856265018670.000016io.soup
3391856005217040.000018nl.blogspot
340185557405090.000051com.mashable
341185537302640.000093com.typepad
3421855348610710.000031com.adjust
343185514563960.000065com.photobucket
3441854452817410.000017org.bitcoin
3451854295222270.000014tw.edu.ntu
3461854107211000.000030com.ecwid
3471854079818200.000017com.indianexpress
3481854028818890.000016co.ello
349185388705750.000046edu.berkeley
3501853737220260.000015com.upi
351185372641320.000232com.squarespace
352185356822800.000089uk.org.ico
3531853492411360.000028com.ssrn
3541853438221520.000014com.viki
3551853381812190.000025it.scoop
356185326062700.000092com.surveymonkey
3571853201616010.000019com.fastcodesign
3581853062017820.000017org.unep
3591852958810570.000031uk.parliament
3601852735619660.000016org.haskell
361185271402240.000107com.etsy
3621852706414420.000021com.shutterfly
3631852538815690.000019uk.org.tate
3641852453028620.000011co.electrek
3651852313426930.000011jp.doorblog
366185228381560.000171com.issuu
3671851914820180.000015com.dezeen
3681851791024300.000013sh.now
3691851753011570.000027com.tradedoubler
3701851502811730.000027gov.weather
3711851361611090.000029com.imageshack
3721851268216930.000018com.channel4
3731851213411160.000029gov.dot
3741851101827030.000011cn.edu.sdu
3751851055411640.000027com.wikia
376185092442820.000088com.huffingtonpost
377185091249530.000034uk.co.pinterest
378185086769240.000035com.arstechnica
379185071562710.000091com.rawgit
380185058124840.000053tv.twitch
3811850572219170.000016th.co.google
3821850313423900.000013uk.ac.nhm
3831850236017640.000017com.netvibes
3841850115618710.000016edu.emory
385185009649180.000035in.amazon
386185002529630.000034com.strikingly
3871849922417730.000017net.bplaced
3881849778633560.000010tw.edu.ntnu
3891849569218110.000017edu.iu
390184945428330.000039com.brightcove
391184916242250.000107com.hubspot
3921849136614700.000021com.wattpad
3931849091414760.000021gov.michigan
3941848922219160.000016nl.tudelft
3951848806414360.000021org.c-span
396184877083940.000065com.meetup
3971848341221100.000014com.kaggle
3981848166212990.000024edu.brookings
39918478490860.000345net.jsfiddle
4001847842026210.000012sh.surge
4011847553022480.000014com.rsa
4021847522017810.000017gov.ahrq
403184741288250.000039org.mediawiki
404184739223460.000072edu.yale
405184726648260.000039com.intel
4061847228815120.000020gov.faa
4071847192619750.000015io.material
4081847173210730.000031com.thenextweb
4091847170618470.000016net.earthlink
4101846961019080.000016jp.blog
411184690928310.000039com.pexels
4121846476010290.000032uk.gov.nationalarchives
4131846091619730.000016com.smashwords
414184590889390.000034org.ieee
4151845742021850.000014com.smore
416184567243450.000072com.livejournal
4171845636033660.000010hk.edu.hkbu
418184535424140.000063com.nypost
4191845346417850.000017com.business-standard
4201845344029070.000011com.yam
4211845124812330.000025org.aarp
4221845040819990.000015com.oprah
4231844994220530.000015org.jpn
4241844988015030.000020org.amnesty
4251844979016510.000018com.avvo
4261844964825400.000012com.cleantechnica
427184495045220.000050edu.cornell
4281844812425860.000012com.mysanantonio
429184475944730.000054io.shields
4301844754414440.000021org.hrw
4311844424024870.000012org.neocities
4321844281019430.000016com.care2
4331844070217140.000018com.snopes
4341844019411480.000027com.gizmodo
4351844005824740.000012com.googledrive
4361843927225940.000012com.iflscience
4371843752415940.000019org.pypi
438184372662680.000092net.php
4391843690219270.000016org.rsc
4401843628813890.000022com.pbworks
4411843597827350.000011com.itsnicethat
4421843536220920.000015ae.thenational
4431843532624610.000012com.hsbc
444184346423380.000074com.hp
4451843263613540.000023uk.co.standard
4461843176418070.000017com.instapaper
447184315964650.000055io.codepen
448184313905530.000047com.buzzfeed
4491843102019100.000016com.secondlife
4501843025824250.000013jp.go.ndl
4511842975621060.000014io.gitlab
452184284323730.000069int.who
4531842712823000.000013org.lds
4541842697624710.000012uk.mod
4551842695417110.000018google.ai
45618426290960.000330de.google
4571842389814270.000021com.thehindu
4581842372417490.000017com.curbed
4591842290216380.000019no.google
460184217383400.000074com.cnbc
4611842068610610.000031com.thedrum
462184197801650.000160com.ebay
463184187906270.000043com.zdnet
4641841845423300.000013pl.cba
4651841639224410.000013com.minds
466184134882010.000125com.salesforce
4671841325215740.000019com.moonfruit
4681841235811560.000027com.mixpanel
4691841167028180.000011tl.page
4701840919619820.000015com.name
4711840908022820.000013jp.hateblo
4721840783025070.000012org.tvtropes
4731840731825800.000012jp.hatenadiary
4741840634825100.000012de.dw
4751840563418540.000016com.googlegroups
4761840550818760.000016mx.com.google
4771840513421980.000014org.aiga
4781840340428830.000011uk.co.birminghammail
479184033403670.000069com.booking
4801840160223140.000013vn.com.google
4811840155017290.000018gov.pa
4821839997216660.000018org.hrc
483183996188820.000036gov.nist
4841839847227420.000011com.exxonmobil
4851839737618410.000016ar.com.google
486183963569890.000033net.clickbank
487183956609760.000033com.matterport
4881839240224290.000013ua.at
4891839052220110.000015uk.ac.leeds
490183874443090.000081gov.cdc
4911838652815580.000019int.unfccc
4921838640823420.000013com.eklablog
493183857004590.000056com.gmail
494183855984010.000064org.npr
4951838483216720.000018gov.maryland
496183843903570.000070com.office
4971838395022400.000014se.liu
4981838381020670.000015com.discovermagazine
4991838340022040.000014com.ipage
5001838162611100.000029com.stackexchange
5011838159424180.000013it.justpaste
502183809744490.000058fr.free
5031838068217180.000018sg.com.google
5041837967210600.000031com.engadget
5051837823824210.000013my.com.thestar
5061837728212730.000024dk.google
5071837713622100.000014org.biorxiv
5081837706218610.000016com.weheartit
5091837419415980.000019uk.gov.tfl
510183712745080.000051gov.whitehouse
5111836933017230.000018ly.snip
5121836900618090.000017com.yourstory
5131836635631540.000011com.bonanza
5141836565028330.000011com.scienceblogs
5151836548414310.000021com.ebayimg
5161836543617740.000017gov.ky
517183635008580.000038com.venturebeat
5181836292411600.000027se.google
5191836245413500.000023com.firebaseapp
520183620261780.000147com.zendesk
5211836015020040.000015uk.gov.metoffice
522183599909280.000035com.windowsphone
5231835975023360.000013com.rediff
524183583885180.000051com.alibaba
5251835525622250.000014com.blogfa
526183552324150.000063com.fastcompany
5271835321214260.000021com.surveygizmo
5281835235220210.000015au.com.telstra
5291835145411340.000028org.sphinx-doc
5301835050220480.000015ro.google
5311835012619040.000016org.tigris
5321834952428350.000011be.lesoir
5331834943026980.000011cz.centrum
5341834937220470.000015link.page
535183492604790.000054org.nodejs
5361834902819600.000016com.marketwire
5371834767222420.000014com.mystrikingly
5381834701822600.000013ch.unige
5391834685027530.000011cat.uab
5401834681828890.000011com.zynga
5411834516415100.000020us.mn.state
5421834162222750.000013com.articulate
543183400129910.000033edu.psu
5441833942221410.000014com.thecvf
5451833902021500.000014es.csic
5461833892228800.000011co.carrd
5471833738016110.000019gov.mo
5481833736022970.000013com.newatlas
5491833569039080.000009jp.rdy
5501833463019900.000015org.iea
5511833359825650.000012com.db
5521833271623100.000013com.webstarts
5531833258424880.000012jp.hatenablog
5541833197623310.000013ly.rebrand
555183313443700.000069com.mapbox
556183312284850.000053com.livechatinc
5571832535219980.000015org.mozillazine
5581832494422710.000013de.uni-freiburg
5591832447213720.000023com.tinypic
560183242528830.000036com.steampowered
5611832384220720.000015uk.ac.york
5621832218610970.000030com.thinkwithgoogle
5631832058225890.000012ru.msu
5641832015624580.000012org.kotlinlang
5651831954016290.000019gov.oregon
5661831891435070.000010com.ingress
5671831812018060.000017gov.wi
568183180565410.000049com.aol
5691831804019690.000016gr.google
5701831782427410.000011lv.draugiem
5711831672023050.000013org.iucnredlist
5721831582220350.000015com.broadwayworld
573183140761340.000221com.youtube-nocookie
5741831395415110.000020net.openid
575183137041680.000158com.tripadvisor
576183135164350.000059com.dailymotion
5771831339815480.000019net.leadpages
5781831336033890.000010com.brother
5791831318627550.000011com.webcindario
5801831311631610.000011es.usal
5811831296223380.000013bg.google
582183127289070.000036com.xiti
5831831233822730.000013us.oh.state
584183107607200.000041fm.last
5851831070016620.000018net.ucoz
586183069363530.000071org.acm
5871830407439310.000009com.worldlingo
5881830325433790.000010com.embarcadero
5891830311017890.000017com.eiseverywhere
5901830283822300.000014org.wri
591183027463950.000065com.pubmatic
592183025164700.000054com.goodreads
5931830082622120.000014com.thehindubusinessline
5941830019622890.000013com.mihanblog
5951829998820370.000015com.intensedebate
5961829818432300.000011com.hellomagazine
5971829729834740.000010net.hypermart
598182949902350.000103uk.co.amazon
5991829442827110.000011nf.co
60018294164730.000401me.fb
601182940885420.000049com.entrepreneur
6021829306225430.000012com.futurelearn
6031829264423890.000013com.iconarchive
6041829230612750.000024com.cognitoforms
6051829213817300.000018org.khanacademy
6061829107220410.000015com.financialpost
6071829100017440.000017us.pa.state
6081828869826530.000012com.fatcow
609182885643720.000069com.staticflickr
6101828827218510.000016io.bower
6111828787836570.000010nz.govt.tepapa
6121828500021990.000014org.prlog
6131828498027430.000011ca.shaw
6141828473621040.000014com.bravesites
6151828303026750.000012de.uni-erlangen
6161828256824460.000012org.lacity
6171828202415340.000020fi.google
6181828201822670.000013de.uni-koeln
6191828051024640.000012uk.co.spectator
620182793263340.000076com.typeform
6211827932427250.000011is.good
6221827929033990.000010com.114la
6231827884232770.000010net.freeforums
624182783849200.000035com.zoho
6251827389824650.000012uk.ac.jisc
6261827340424890.000012com.mnn
6271827335223930.000013ca.dal
628182727401140.000290com.statcounter
629182727224800.000054com.netflix
6301827231815670.000019com.flashtalking
6311827221219030.000016com.prweek
6321827080631150.000011site.negocio
6331827079420800.000015org.lung
6341827050627270.000011com.mouser
6351827034025690.000012uk.co.profilebusiness
6361826984036160.000010uk.gov.number10
6371826814638060.000009net.dead
6381826773433750.000010jp.ac.kobe-u
6391826762819250.000016uk.org.nice
64018267512880.000343com.oculus
6411826719831960.000011build.bazel
6421826654618780.000016org.gentoo
6431826616621810.000014ie.thejournal
644182661481090.000310com.sharethis
6451826597619060.000016org.gnupg
646182647281480.000186ru.mail
6471826337618600.000016com.doodlekit
6481826223819480.000016com.crashlytics
6491826215618310.000017org.alz
6501826195425490.000012us.ms.state
6511826116224590.000012com.instructure
652182605408200.000040com.cbsnews
6531825984428770.000011ee.ut
6541825982612110.000026com.msdn
655182596107770.000040com.samsung
6561825700413380.000023com.emailmeform
657182549345490.000048edu.cmu
6581825482224960.000012uk.co.osoo
65918254762830.000354com.livestream
6601825465622260.000014com.atavist
6611825287622080.000014fr.archives-ouvertes
6621825228227920.000011com.cnsnews
6631825201823480.000013io.pantheon
664182511488980.000036com.createjs
6651825102617550.000017us.fl.state
6661825073023210.000013com.rabbitmq
6671825062827120.000011uk.co.newmedianow
6681824857614220.000022com.123formbuilder
6691824703220860.000015gov.nh
6701824350422330.000014org.crossref
6711824231422290.000014us.nm.state
672182422542960.000084com.scribd
6731824136632540.000010ca.qc.montreal
6741824090832850.000010uk.co.lrb
675182408281350.000215com.youku
676182397505170.000051com.slack
6771823965826770.000012com.hatenadiary
6781823965622920.000013com.itsmyurls
6791823763626710.000012uk.org.oxonaa
680182369022460.000099com.constantcontact
6811823686233480.000010com.outlookindia
6821823585438930.000009in.ac.nptel
6831823554026810.000012uk.org.oxfam
6841823523623440.000013com.yext
685182338122560.000094com.getbootstrap
6861823332421070.000014org.jenkins-ci
6871823058420550.000015com.broadcastingcable
6881823047816860.000018uk.gov.direct
6891823041626630.000012com.wmtransfer
6901823037419770.000015gov.mt
6911823016428210.000011uk.ac.stir
6921822854010520.000031com.marketwatch
6931822774422660.000013com.tmcnet
6941822744031360.000011uk.co.hsbc
6951822708617980.000017org.nfpa
6961822679229390.000011com.batchgeo
6971822584432750.000010com.weightwatchers
698182256362340.000103to.amzn
6991822463235740.000010com.orgfree
7001822377813550.000023org.whatbrowser
7011822181428430.000011com.adn
7021822127611900.000026org.weforum
703182205064810.000054org.hbr
7041821988028200.000011au.edu.deakin
7051821973414550.000021org.js
7061821911824450.000013in.ernet
7071821796228540.000011hu.elte
7081821751630250.000011pl.edu.uw
7091821727423670.000013uk.org.rspb
7101821652822200.000014com.healthgrades
7111821626427790.000011org.carbonbrief
712182142143660.000069com.prnewswire
7131821395620880.000015com.tapatalk
7141821318024310.000013org.grist
7151821275034230.000010id.co.kaskus
716182106384560.000057com.oreilly
7171821010635870.000010com.skepticalscience
718182099505390.000049gov.sec
7191820992230810.000011com.deccanherald
7201820966819050.000016tl.we
7211820877023110.000013us.ma.state
7221820686011010.000030uk.ac.cam
7231820599436300.000010ua.meta
7241820573835260.000010app.web
7251820446223980.000013uk.co.zoopla
7261820196632100.000011org.oceanconservancy
7271819963034210.000010org.atsjournals
7281819896235320.000010ru.my1
7291819844431620.000011com.mozello
7301819560015620.000019com.pastebin
7311819458028670.000011de.freenet
7321819341411370.000028edu.ucla
7331819310030520.000011com.telegraphindia
7341819302628570.000011com.chagasi
735181927589370.000034br.com.uol
7361818899426300.000012com.atwebpages
7371818862630360.000011com.remind
7381818792211320.000028com.redhat
739181877486080.000044com.wikihow
7401818765833770.000010edu.utep
7411818726434550.000010ru.nnov
7421818683418810.000016uk.gov.defra
7431818656823590.000013net.portfoliobox
7441818562426100.000012com.blogsky
7451818543438560.000009uk.co.mailonsunday
7461818543227230.000011jp.xxxxxxxx
7471818412214250.000021edu.ucsd
7481818396214490.000021com.digitaltrends
749181837381960.000130jp.ne.hatena
7501818246425630.000012uk.co.inews
7511818172823130.000013gov.la
7521818165612660.000024ly.ow
7531818036034410.000010gr.sch
7541817980230550.000011com.sc
7551817862833730.000010com.cummins
7561817756623630.000013com.activerain
7571817602638010.000009com.kazeo
7581817600229010.000011net.onlinewebshop
7591817542236890.000010com.galvanize
7601817490234730.000010ru.pr-cy
761181748265030.000052com.dmca
7621817352833280.000010com.kaywa
763181733488210.000040com.psychologytoday
7641817211828530.000011uk.co.heatall
76518171416840.000350me.ogp
7661816812826010.000012gov.ks
7671816778215160.000020ca.blogspot
7681816755821700.000014com.cityam
7691816728436040.000010gov.cabq
7701816643618130.000017org.reactjs
7711816605232830.000010org.escardio
7721816573410640.000031com.foxnews
7731816568018970.000016com.fifa
774181648602040.000117com.naver
7751816440437610.000009com.carscoops
7761816268029280.000011com.ecowatch
7771816239015070.000020com.literatumonline
778181619985350.000049net.2mdn
779181618004760.000054com.force
780181605781590.000167gov.privacyshield
7811816027018960.000016com.pcworld
7821816019229860.000011com.theyworkforyou
78318159730810.000365com.messenger
7841815970039390.000009com.anghami
785181594264240.000061edu.nyu
7861815799012940.000024com.indiegogo
7871815782818690.000016kr.or.kisa
788181578163640.000070com.discordapp
7891815701431860.000011uk.org.38degrees
7901815685036280.000010com.insideevs
7911815549614880.000020com.placeholder
7921815507232500.000010google.design
7931815504437640.000009gle.goo
794181544624540.000057com.walmart
795181533604280.000060com.flipboard
7961815204429020.000011pl.lublin
797181519524220.000062com.wufoo
7981815119811230.000029com.shutterstock
7991815068425370.000012org.iihs
8001814944627880.000011in.businessworld
801181486369810.000033com.pinimg
8021814776024070.000013jp.e-shops
8031814773422500.000014com.codecademy
8041814634026420.000012com.zx2c4
805181463281290.000243info.aboutads
8061814594421380.000014ca.ubc
8071814553828740.000011com.bnef
8081814435432400.000011uk.ac.rcplondon
8091814425437180.000009com.wsoctv
8101814390239500.000009com.monbiot
8111814334234630.000010com.droppages
8121814314823660.000013gov.arts
8131814245426440.000012us.wi.state
8141814204634770.000010org.usatf
8151814087816240.000019com.nvidia
8161813886636360.000010com.elmercurio
8171813883815380.000020com.businessweek
8181813846221760.000014com.tutsplus
819181383825540.000047com.atlassian
8201813735611840.000026com.searchengineland
8211813727835940.000010com.glu
8221813712436450.000010es.consumer
823181359742400.000102cn.com.sina
8241813559639480.000009com.allmyfaves
8251813534234460.000010com.businessgreen
826181336423500.000072com.163
8271813326832920.000010org.jython
828181332304710.000054com.smugmug
8291813281638640.000009org.thechicagocouncil
8301813212635760.000010gov.azdot
8311813047011760.000027com.ycombinator
8321812983833390.000010org.transportenvironment
8331812853829930.000011gov.ferc
834181279109360.000034com.aliexpress
835181261543560.000070com.wiley
836181257906960.000042com.moz
8371812499627560.000011uk.gov.environment-agency
8381812488630120.000011org.zsl
8391812413637040.000009org.ssireview
8401812352023780.000013uk.gov.scotland
8411812297815950.000019tv.ustream
8421812252230960.000011org.dailystrength
843181220385980.000045com.caniuse
8441812099624850.000012net.privacypolicytemplate
845181208667680.000040gov.noaa
8461812081815730.000019jp.makeshop
8471812051830400.000011org.rspo
8481811994623030.000013com.seetickets
8491811945421830.000014com.ign
850181188964040.000064mp.mailchi
851181180003110.000081com.digg
8521811800028550.000011gov.txdot
8531811736634120.000010uk.ac.ceh
8541811716414790.000021com.crunchbase
8551811707411270.000029com.highcharts
8561811587026450.000012com.9to5mac
8571811464810900.000030com.withgoogle
858181143148890.000036com.webs
8591811407224810.000012uk.co.streetmap
8601811250838650.000009com.pushwoosh
8611811170832040.000011ca.uwaterloo
862181111308170.000040com.shinystat
863181110783050.000082fr.google
8641811105034670.000010com.baomoi
8651811097439160.000009uk.ac.tyndall
8661811039617660.000017com.webmasterplan
8671811018036860.000010dk.bloggersdelight
8681810990834010.000010uk.gov.hm-treasury
8691810926217930.000017uk.org.cqc
8701810894812480.000025com.smashingmagazine
871181081383310.000076com.automattic
8721810757215300.000020com.ning
8731810698428290.000011com.linkwithin
8741810652230020.000011uk.org.greenpeace
875181037689560.000034com.libsyn
8761810353812390.000025com.sap
8771810295620910.000015edu.uci
878181025646280.000043com.patreon
8791810202035030.000010com.climatechangenews
880181017584090.000063com.xinhuanet
8811810133634640.000010com.kapook
882181006188850.000036com.newyorker
8831810047436400.000010com.spruz
884181001964780.000054com.inc
8851810006226760.000012jp.aikotoba
886180992689140.000035org.eff
8871809879436620.000010com.platts
8881809855635350.000010org.c2es
8891809855027470.000011com.mykaratestore
8901809678617700.000017com.ikea
8911809639414230.000022com.billboard
8921809509210700.000031com.hootsuite
8931809494835250.000010com.jkp
8941809349628240.000011org.mcsuk
8951809262212540.000025es.agpd
8961809243833490.000010net.edie
897180923585330.000050com.ea
898180921123760.000068org.opensource
8991809156829030.000011ru.drom
9001809016226390.000012com.yelloyello
9011808996825440.000012uk.co.intersol
902180897401390.000202com.alicdn
9031808942240510.000009com.mforos
9041808699014730.000021com.fiverr
905180863529340.000035com.foursquare
9061808591817370.000017org.freecsstemplates
9071808481041420.000009uk.org.indymedia
9081808467420490.000015uk.gov.education
9091808369438430.000009com.thinkbroadband
910180821642310.000104jp.co.amazon
9111808011438680.000009org.sciencenewsforstudents
912180800342210.000108org.drupal
9131807972610960.000030com.variety
914180786662900.000086com.stumbleupon
9151807803832690.000010net.scienceontheweb
9161807758217560.000017com.nba
9171807745225610.000012org.webring
9181807650210330.000031com.visualstudio
9191807595840050.000009io.raindrop
9201807454427440.000011jp.zouri
9211807376639040.000009org.corporateeurope
9221807247014020.000022com.storify
923180714363750.000069gov.ftc
9241807137216030.000019net.with2
9251807092614480.000021com.nike
9261807022240480.000009io.dataquest
9271807006612550.000025org.unicef
9281806967235670.000010bnpparibas.group
9291806917236850.000010com.thestatesman
9301806886634270.000010uk.org.rya
931180675083830.000068com.airbnb
9321806720416350.000019de.zeit
9331806719025550.000012com.hackernoon
9341806627434510.000010ca.pe.gov
9351806526640310.000009com.raamdev
9361806438824380.000013io.postach
9371806412614870.000020edu.purdue
938180635084070.000063com.tripod
9391806322812280.000025gov.fbi
9401806315413690.000023com.lifehacker
9411806313010690.000031com.uk
9421806187834320.000010in.gov.mhrd
9431806113035270.000010org.gmplib
9441806010038790.000009com.gitimmersion
9451805957828070.000011jp.at-ninja
9461805900430100.000011com.shichihuku
9471805882636290.000010com.h2database
9481805773634820.000010uk.org.rcn
9491805764037370.000009com.writetothem
9501805659213660.000023com.parsiblog
951180565869840.000033com.dropboxusercontent
9521805595013060.000024com.prweb
9531805562836950.000009com.websiteseguro
9541805510411180.000029com.vox
9551805427213970.000022us.imageshack
9561805396420320.000015com.howstuffworks
9571805292015310.000020com.yoast
9581805228012980.000024com.pcmag
9591805139830080.000011uk.org.woodlandtrust
9601805093635230.000010gle.posts
9611805080038380.000009org.priceofoil
9621804958016140.000019com.ccbill
9631804906637500.000009com.fourfour
964180472149450.000034gov.census
9651804648613280.000023edu.wisc
966180458761510.000179jp.co.google
9671804571012200.000025com.blackberry
9681804541411030.000030edu.umich
9691804539019520.000016com.w3layouts
970180438941460.000190me.line
9711804381615930.000019edu.usc
9721804235628420.000011com.zatunen
973180422405000.000052com.nasdaq
974180421305670.000046net.daum
9751804157031180.000011vn.tuoitre
9761804055625730.000012com.hisupplier
9771803944420230.000015com.nfl
978180393709270.000035com.ggpht
9791803932415490.000019com.vmware
9801803902038270.000009com.realtytimes
9811803836232610.000010net.batcave
9821803811633410.000010org.mygamesonline
983180378667340.000040com.mckinsey
9841803767439830.000009org.eia-international
985180376042580.000094com.sohu
9861803759437000.000009io.dropwizard
9871803739410260.000032gov.nps
9881803724421310.000014au.com.news
9891803660836520.000010de.epubli
9901803419813810.000022com.unity3d
9911803407229920.000011net.nend
9921803304840980.000009com.easyhits4u
9931803189011620.000027com.steamcommunity
9941803162214510.000021edu.uchicago
9951803157010860.000030com.uber
9961803147053060.000007com.plurk
997180304905970.000045com.adweek
9981803018236350.000010com.jal
9991802967017860.000017com.techradar
10001802924412710.000024com.ifttt

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

January 2020 crawl archive now available

The crawl archive for January 2020 is now available! It contains 3.1 billion web pages or 300 TiB of uncompressed content, crawled between January 17th and 29th. It includes page captures of 960 million URLs not contained in any crawl archive before.

Improvements and Fixes

  • date time values in the column "fetch_time" of the columnar index are now stored using the "int64" data type. For details and compatibility issues please see cc-index-table#7
  • WARC request records now show the HTTP protocol version sent with the HTTP request which can be different from the version received in the HTTP response message, cf. NUTCH-2760

Archive Location and Download

The January crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-05/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://commoncrawl.s3.amazonaws.com/ to each line, you end up with the S3 and HTTP paths respectively.

 File List#FilesTotal Size
Compressed (TiB)
SegmentsCC-MAIN-2020-05/segment.paths.gz100
WARC filesCC-MAIN-2020-05/warc.paths.gz5600059.94
WAT filesCC-MAIN-2020-05/wat.paths.gz5600022.3
WET filesCC-MAIN-2020-05/wet.paths.gz5600010
Robots.txt filesCC-MAIN-2020-05/robotstxt.paths.gz560000.25
Non-200 responses filesCC-MAIN-2020-05/non200responses.paths.gz560002.28
URL index filesCC-MAIN-2020-05/cc-index.paths.gz3020.23

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-05/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

December 2019 crawl archive now available

The crawl archive for December 2019 is now available! It contains 2.45 billion web pages or 234 TiB of uncompressed content, crawled between December 5th and 16th. It includes page captures of 850 million URLs not contained in any crawl archive before.

Archive Location and Download

The December crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-51/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://commoncrawl.s3.amazonaws.com/ to each line, you end up with the S3 and HTTP paths respectively.

 File List#FilesTotal Size
Compressed (TiB)
SegmentsCC-MAIN-2019-51/segment.paths.gz100
WARC filesCC-MAIN-2019-51/warc.paths.gz5600047.47
WAT filesCC-MAIN-2019-51/wat.paths.gz5600017.6
WET filesCC-MAIN-2019-51/wet.paths.gz560008.06
Robots.txt filesCC-MAIN-2019-51/robotstxt.paths.gz560000.26
Non-200 responses filesCC-MAIN-2019-51/non200responses.paths.gz560003.5
URL index filesCC-MAIN-2019-51/cc-index.paths.gz3020.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-51/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

November 2019 crawl archive now available

The crawl archive for November 2019 is now available! It contains 2.55 billion web pages or 250 TiB of uncompressed content, crawled between November 11th and 23rd with a short operational break on Nov 16th. It includes page captures of 1.1 billion URLs not contained in any crawl archive before.

What’s new?

We’ve added two new fields to the URL indexes (CDX and columnar):

  • the redirect target location is stored in the CDX JSON field "redirect" resp. the column "fetch_redirect". The value is extracted from HTTP header field "Location" if the HTTP status code indicates a HTTP redirect. A relative URL path is converted to an absolute URL using the page URL as base URL. The key is absent (resp. the field value is null) in case the "Location" value is missing, not a valid URL or not a valid relative URL path.
  • truncation of the WARC record payload is indicated by the key "truncated" resp. the column "content_truncated". The reason for the truncation is given only for truncated records following the WARC header field "WARC-Truncated".

Additional details and examples can be found in the corresponding PR #15.

We’ve fixed a bug affecting the capture time (WARC-Date) in the the robots.txt subset which has been extracted from the HTTP "Date" field of the HTTP header and appeared to be occasionally wrong. Please see issue #14 for further details.

Archive Location and Download

The November crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-47/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://commoncrawl.s3.amazonaws.com/ to each line, you end up with the S3 and HTTP paths respectively.

 File List#FilesTotal Size
Compressed (TiB)
SegmentsCC-MAIN-2019-47/segment.paths.gz100
WARC filesCC-MAIN-2019-47/warc.paths.gz5600053.95
WAT filesCC-MAIN-2019-47/wat.paths.gz5600018.50
WET filesCC-MAIN-2019-47/wet.paths.gz560008.34
Robots.txt filesCC-MAIN-2019-47/robotstxt.paths.gz560000.24
Non-200 responses filesCC-MAIN-2019-47/non200responses.paths.gz560003.05
URL index filesCC-MAIN-2019-47/cc-index.paths.gz3020.20

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-47/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

Host- and Domain-Level Web Graphs Aug/Sep/Oct 2019

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of August, September and October 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which host all scripts and tools required to construct the graphs.

What’s new?

The following improvements have been made for this webgraph release:

  • the graphs now also included edges stemming from HTTP 303 “See Other” redirects (in addition to other HTTP redirect status codes)
  • the Common Crawl robots.txt WARC files are used to get additional host-level redirects including hosts which exclude the entire content in their robots.txt
  • links from robots.txt files to sitemaps are now extracted directly from the robots.txt WARC files, see the Feb/Mar/Apr 2018 web graph announcement for more details about this type of host-level links

Host-level graph

The graph consists of 820 million nodes and 4.55 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 752 million dangling nodes (92%) and the largest strongly connected component contains 50 million (6%) nodes.

You can download the graph and the ranks of all 820 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-aug-sep-oct/host/. Alternatively, you can use https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2019-aug-sep-oct/host/ as prefix to access the files from everywhere.

SizeFileDescription
5.29 GBcc-main-2019-aug-sep-oct-host-vertices.paths.gznodes ⟨id, rev host⟩, paths of 28 vertices files
20.73 GBcc-main-2019-aug-sep-oct-host-edges.paths.gzedges ⟨from_id, to_id⟩, paths of 56 edges files
8.15 GBcc-main-2019-aug-sep-oct-host.graphgraph in BVGraph format
2 kBcc-main-2019-aug-sep-oct-host.properties
10.00 GBcc-main-2019-aug-sep-oct-host-t.graphtranspose of the graph (outlinks inverted to inlinks)
2 kBcc-main-2019-aug-sep-oct-host-t.properties
1 kBcc-main-2019-aug-sep-oct-host.statsWebGraph statistics
11.59 GBcc-main-2019-aug-sep-oct-host-ranks.txt.gzharmonic centrality and pagerank

Note that the host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 92.7 million nodes and 2.4 billion edges. 52% or 48 million nodes are dangling nodes, the largest strongly connected component covers 36 million or 40% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-aug-sep-oct/domain/ resp. https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2019-aug-sep-oct/domain/.

Download files of the Common Crawl Aug/Sep/Oct 2019 domain-level webgraph

SizeFileDescription
0.64 GBcc-main-2019-aug-sep-oct-domain-vertices.txt.gznodes ⟨id, rev domain, num hosts⟩
9.06 GBcc-main-2019-aug-sep-oct-domain-edges.txt.gzedges ⟨from_id, to_id⟩
4.64 GBcc-main-2019-aug-sep-oct-domain.graphgraph in BVGraph format
2 kBcc-main-2019-aug-sep-oct-domain.properties
4.82 GBcc-main-2019-aug-sep-oct-domain-t.graphtranspose of the graph
2 kBcc-main-2019-aug-sep-oct-domain-t.properties
1 kBcc-main-2019-aug-sep-oct-domain.statsWebGraph statistics
1.97 GBcc-main-2019-aug-sep-oct-domain-ranks.txt.gzharmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 92 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Aug/Sept/Oct 2019)

harmonic
centrality
rank
hc valuepage rankpage rank
value
reversed hostname
13219322210.020989com.googleapis
22993949030.012691com.facebook
32920276220.012925com.google
42682358440.007369com.twitter
52630480450.006660org.w
62602310660.006435com.youtube
72433093090.003914com.instagram
82395469870.004993org.gmpg
92353721680.004863com.googletagmanager
1023456362130.002913com.linkedin
1122601398120.003086org.wordpress
1222511672100.003602com.cloudflare
1322484028220.001698com.gravatar
1422366260230.001509com.pinterest
1522337722190.002143com.wordpress
1622168804140.002422com.bootstrapcdn
1722152656320.001134org.wikipedia
1821939876200.001777com.apple
1921666694420.000842com.blogspot
2021595338210.001736com.jquery
2121576018490.000713be.youtu
2221574068350.001064com.vimeo
2321525328300.001154com.microsoft
2421517514180.002164com.gstatic
2521444512170.002176com.adobe
2621427572390.000964com.amazonaws
2721426674500.000695com.wp
2821393434510.000681com.amazon
2921314258650.000516com.tumblr
3021291646460.000767gl.goo
3121256658250.001309com.macromedia
3221253410290.001173com.baidu
3321136800670.000501ly.bit
3421087412270.001200com.flickr
3521083642240.001391com.github
3621068510890.000381com.yahoo
3721063868410.000928com.google-analytics
3821063204310.001139com.googlesyndication
3921053852570.000608eu.europa
4021051522610.000541org.mozilla
41209978121060.000300com.reddit
4220934970370.001002net.cloudfront
4320930738280.001184ru.yandex
4420907836380.000964com.addthis
4520875048480.000734co.t
4620863874470.000744net.doubleclick
4720860222700.000482org.w3
4820822794980.000329com.googleusercontent
4920819594430.000814com.squarespace
5020815248750.000462com.medium
5120812946910.000376org.creativecommons
52208110761750.000140org.wikimedia
5320788398860.000417com.weebly
5420786242630.000534me.wp
55207641441290.000221com.nytimes
5620754576880.000400io.github
5720744406550.000625com.paypal
58207310501650.000148uk.co.bbc
5920729776580.000557net.jsdelivr
60207236081080.000297com.soundcloud
61207206141720.000141com.imgur
62206964741300.000210com.dropbox
63206762261370.000181com.forbes
64206410781730.000141net.slideshare
6520640886540.000634org.schema
66206370161530.000163com.theguardian
67206194101870.000136com.cnn
68206144822040.000118com.businessinsider
69205893762170.000109com.wsj
70205753842810.000086edu.harvard
71205728061670.000147com.bing
72205710442410.000098com.techcrunch
73205672922900.000084edu.mit
74205571442850.000084com.reuters
75205540743750.000067com.msn
76205498523290.000075com.cnet
77205421041400.000178org.archive
78205381042500.000094com.bloomberg
7920522694330.001120com.fontawesome
80205213741410.000175gov.nih
8120513408930.000355com.shopify
82205131402710.000089com.myspace
83205070482070.000116edu.stanford
8420496152530.000647com.wix
85204891022000.000120com.stackoverflow
86204879384340.000057com.googleblog
87204846321540.000163org.apache
88204781942290.000102com.oracle
89204753922140.000110com.washingtonpost
90204723862600.000091com.android
91204692982670.000090com.bbc
92204664401940.000123org.ietf
93204385423100.000079com.time
94204363522980.000081uk.co.telegraph
95204179763690.000067com.ted
96204156623720.000067gov.nasa
97204123163680.000067com.githubusercontent
98204098881850.000136com.npmjs
99204016043940.000063com.quora
100203961466010.000042com.thenextweb
101203956741610.000156com.giphy
102203887787260.000037com.wikia
103203807283430.000072uk.co.dailymail
104203796522940.000082com.usatoday
105203783963710.000067com.latimes
106203702127130.000037org.chromium
107203697003060.000079org.un
108203681481440.000174com.wixsite
109203666104930.000050com.economist
11020361312260.001226com.qq
111203434262680.000090com.appspot
112203392664800.000052com.pixabay
113203373984910.000050com.zdnet
114203283083150.000079com.example
115203254223580.000070com.livejournal
116203223343800.000066com.mashable
117203082003020.000080com.cnbc
118203080662530.000093org.ampproject
119203069844420.000056com.nationalgeographic
120202934265050.000049com.venturebeat
121202923804040.000062com.dailymotion
122202855021390.000178com.twimg
123202841644760.000052org.bitbucket
124202823685470.000046com.pexels
125202807143270.000075com.springer
126202799922180.000108com.huffingtonpost
12720279190940.000355com.whatsapp
128202779284590.000054com.cisco
129202684161460.000170com.blogger
130202676841230.000234com.ytimg
131202647304130.000061com.fortune
132202630146410.000040uk.ac.ox
133202622582310.000100com.getbootstrap
134202616488470.000035org.cambridge
135202612686290.000040org.weforum
136202508541970.000123com.typepad
137202506982790.000086com.sciencedirect
138202501625120.000048com.about
139202471922860.000084com.wired
140202401303170.000078com.skype
141202352025580.000045org.worldbank
142202301921340.000186com.issuu
143202250045040.000049com.mysql
144202209966500.000039org.sciencemag
145202209725310.000047org.arxiv
146202182966240.000041uk.co.guardian
147202161944070.000062com.nature
148202140121270.000226com.unpkg
149202136381430.000175com.spotify
150201955008240.000036com.playstation
151201953521770.000139uk.co.google
152201952604400.000057gov.noaa
153201935743230.000077com.staticflickr
154201935123660.000068com.gmail
1552019193410370.000028org.eclipse
156201918323950.000063net.researchgate
157201859343420.000072com.fc2
158201791946030.000042org.ieee
159201771401320.000201com.zendesk
160201771083830.000065com.theatlantic
161201738505900.000043com.git-scm
162201737221820.000136me.t
163201694462820.000085com.googlecode
164201679642120.000113net.behance
165201669603640.000068com.w3schools
166201654086570.000039com.stackexchange
167201475661280.000222com.youtube-nocookie
168201442664300.000058com.buzzfeed
169201431685730.000043br.com.uol
170201412228280.000036ca.blogspot
171201385285920.000042com.evernote
172201375368540.000034com.scientificamerican
173201230002270.000102com.dribbble
174201229664950.000049com.vice
175201198121800.000137com.feedburner
176201187865740.000043net.azurewebsites
177201133705360.000046com.alexa
178201107804180.000059com.outlook
179201033824240.000059com.gitlab
180200925884220.000059me.about
181200922324090.000061com.goodreads
1822009184211020.000026com.nvidia
183200824504190.000059com.mozilla
184200785244470.000056com.entrepreneur
185200737402360.000099com.ft
186200715344520.000055com.wikihow
187200661242450.000096com.disqus
1882006494210920.000026com.jetbrains
1892006375613270.000023org.phys
190200620666020.000042org.greenpeace
191200614743860.000065org.hbr
192200594681780.000139com.salesforce
193200585325370.000046com.adage
194200560123000.000080org.doi
1952005591411060.000026org.ap
196200540688600.000034com.500px
197200518244880.000051gov.loc
198200513429570.000030com.sap
199200505006260.000041com.marketwatch
2002004982412650.000024com.siemens
2012004958411730.000025ca.utoronto
202200493004280.000058uk.co.independent
203200480342220.000104com.hubspot
204200457885930.000042com.slate
205200420183490.000071gg.discord
2062002495614350.000021com.hackernoon
207200220964870.000051uk.co.blogspot
2082001213014510.000021org.tensorflow
209200076824010.000062com.indiatimes
2102000748610350.000028org.kernel
211200016985300.000047com.trello
212199990346660.000038com.searchengineland
2131999708410090.000029com.unity3d
214199969404730.000052com.computerworld
215199962325490.000045com.withgoogle
2161999307813690.000022edu.osu
217199918809490.000030edu.si
218199902366120.000041au.net.abc
2191998808814280.000021com.lego
220199875322870.000084com.nbcnews
2211997748213560.000022com.angelfire
222199760804990.000049com.moz
223199753581990.000122net.sourceforge
224199692366670.000038co.ibb
2251996811416180.000019org.edx
226199670725150.000048com.box
227199614589860.000029com.huffpost
228199613705980.000042gov.state
2291995641815630.000019blog.home
2301995560816780.000018com.oregonlive
231199542846310.000040com.pinimg
232199531808630.000034gov.usgs
2331994989220480.000016com.sputniknews
2341994895010470.000027co.elastic
2351994746011960.000025edu.rutgers
236199466142110.000115com.optimizely
2371994541814090.000021org.maven
2381994266813730.000022net.seesaa
239199395122370.000099com.aliyuncs
240199393002910.000083com.tinyurl
241199391821880.000134com.eepurl
242199381522240.000103com.wpengine
2431993653822350.000014com.slides
244199355866590.000039com.sciencedaily
245199332621360.000183com.addtoany
246199330889460.000031com.storify
247199321941420.000175com.yimg
248199270323540.000070com.getpocket
249199256427150.000037com.vox
25019922530600.000546com.vk
251199209941710.000142org.allaboutcookies
2521991999011650.000025com.vogue
253199183643350.000074com.wufoo
2541991467612820.000023ms.1drv
2551990648414810.000020io.itch
256199063128340.000035com.techtarget
257199051626000.000042org.change
258199015305970.000042com.uk
259199012584210.000059com.squareup
2601989757614080.000021com.itv
261198968029540.000030com.thehill
2621989677212910.000023com.scmp
2631989451417770.000017com.diigo
264198931923160.000079es.google
265198902446510.000039com.lifehacker
266198887866710.000038gov.fcc
267198869807390.000037com.chicagotribune
2681988618023090.000014com.pearltrees
2691988551615540.000019org.unep
270198819603130.000079net.windows
271198818422480.000094ru.rambler
272198806425060.000049us.icio
27319877580920.000358com.weibo
274198765561090.000290com.paypalobjects
275198748268910.000033com.strikingly
2761987359811780.000025com.netlify
277198676544560.000055gov.epa
278198663502920.000083com.criteo
279198640807140.000037org.pewresearch
280198611365330.000047org.plos
2811986095412250.000024com.newscientist
282198608368490.000035uk.co.mirror
2831986070010100.000029com.mediafire
2841986029810720.000027com.sky
285198599469280.000031com.buffer
2861985891012280.000024com.aljazeera
2871985816813390.000022it.scoop
288198580402090.000116org.iana
2891985726020700.000016com.coca-colacompany
290198569126830.000038com.flipboard
2911985390018010.000017jp.ac.u-tokyo
2921985311610180.000028uk.co.metro
293198510543090.000079com.ibm
294198469683220.000077com.go
2951984683815520.000019uk.bl
2961984155612640.000024com.nikkei
29719840090520.000667com.fb
2981983984425060.000013it.unimi
2991983685815950.000019com.googlesource
300198345044740.000052com.udacity
301198340248350.000035uk.co.thetimes
302198322621680.000144com.imdb
303198316608430.000035gov.congress
304198281426680.000038org.fao
3051982665611910.000025org.acs
3061982523817280.000018com.toptal
3071982473610650.000027edu.duke
308198239826210.000041site.business
3091982092011330.000026com.trendmicro
310198178229550.000030com.theconversation
311198142589830.000029co.g
312198130348510.000034com.bmj
313198122021700.000143com.amazon-adsystem
3141980839810450.000027com.searchenginewatch
3151980612813760.000022edu.gatech
3161980347422070.000015com.viki
3171980338811350.000026edu.brookings
318198031789710.000030com.reverbnation
3191979896010690.000027au.com.smh
32019797938440.000797com.googleadservices
321197961644750.000052org.freecodecamp
322197928066580.000039br.com.google
3231979189617660.000017jp.co.japantimes
324197912344000.000063me.telegram
3251979020813320.000022com.msnbc
3261978967219150.000016org.wikibooks
3271978935612960.000023com.dw
3281978762213660.000022com.hostgator
329197841544770.000052com.theverge
3301978091615740.000019com.bankofamerica
331197769869940.000029com.yoast
332197757429970.000029com.socialmediaexaminer
333197741468410.000035org.apa
334197727484260.000058com.elsevier
335197714044580.000055com.bigcartel
3361977019022400.000014com.kinja
3371977002417010.000018com.mediaplex
3381976908010580.000027uk.co.huffingtonpost
3391976682016820.000018org.bitcoin
3401976566814300.000021com.grammarly
3411976522020710.000016com.mathworks
3421976466212530.000024com.livescience
343197642022490.000094com.live
3441976351622650.000014org.biorxiv
3451976202417940.000017com.makeuseof
346197607009420.000031com.econsultancy
347197592965180.000047com.bigcommerce
348197590889530.000030com.searchenginejournal
34919757028620.000537net.akamaihd
3501975586617640.000017com.colourlovers
351197512323140.000079com.rackcdn
3521974916218340.000017com.sas
353197469622230.000104org.gnu
3541974239024900.000013com.itsnicethat
3551974169422960.000014uk.ac.sussex
356197392128200.000036com.neilpatel
357197385541620.000156com.opera
358197385409510.000030com.gumroad
359197334348680.000034com.business2community
360197311389090.000032uk.co.pinterest
361197305706170.000041uk.parliament
362197295608980.000032com.ecwid
363197290245260.000047me.m
3641972830211860.000025com.thelancet
3651972747616770.000018uk.co.timesonline
3661972556816620.000018edu.iastate
367197209488900.000033com.thedrum
3681971820012340.000024com.seattletimes
369197167721160.000258com.jimdo
3701971515817480.000018org.rsc
371197133183180.000078me.wa
3721971305223120.000014io.soup
373197121742400.000098net.php
374197105249960.000029com.healthline
375197066621030.000317net.facebook
376197006623890.000064com.meetup
3771969816813970.000021int.unfccc
3781969784223640.000014com.autoblog
3791969718411470.000026uk.co.ebay
3801969628415120.000020com.channel4
381196961023450.000072int.who
382196958428560.000034com.photoshelter
383196934262970.000081org.python
3841969316821030.000016edu.miami
3851969310824450.000013com.mysanantonio
3861969305213140.000023com.bustle
3871969300424160.000013com.smore
3881969087212440.000024uk.co.express
3891968949618820.000016com.smashwords
3901968934614540.000021com.gawker
3911968926614920.000020org.hrc
3921968857013780.000022uk.gov.blog
393196882102660.000090com.rawgit
394196853862510.000094uk.org.ico
3951968437222290.000015org.vim
3961968369421480.000015uk.ac.york
3971968304819020.000016com.discovermagazine
3981968246620170.000016com.dummies
3991968227028110.000012com.iht
4001967870214980.000020fr.lesechos
4011967719016430.000019org.amnesty
4021967718410870.000026org.aarp
403196759128360.000035uk.gov.legislation
4041967566615820.000019com.pbworks
4051967522811970.000025com.cio
4061967503615410.000020com.googlegroups
407196736968880.000033uk.gov.nationalarchives
408196717864890.000051com.nwsource
4091966919013440.000022com.thestar
4101966832819930.000016com.treehugger
4111966827616020.000019com.brainyquote
412196678685130.000048com.livechatinc
4131966726211950.000025org.heart
414196660462590.000091com.unsplash
4151966593814750.000020ie.independent
4161966566224440.000013org.sciencenews
4171966400814780.000020fi.google
4181966289612010.000025uk.co.standard
419196624041630.000156com.eventbrite
4201966185019970.000016com.timesofisrael
4211966134013040.000023com.surveygizmo
4221965977812450.000024org.ohchr
4231965671619890.000016com.nationalreview
4241965426020220.000016com.gucci
425196532546050.000041org.mediawiki
426196512349720.000029com.wordstream
4271965110215840.000019com.netvibes
4281964956619760.000016org.bitcointalk
4291964822823720.000014com.deepmind
4301964812417730.000017org.iucn
4311964790414960.000020com.startribune
432196460242930.000082com.ebay
4331963938813550.000022com.convinceandconvert
434196371005220.000047edu.yale
435196366143840.000065com.kickstarter
436196357761000.000321com.godaddy
4371963491221570.000015com.instapaper
4381963383017670.000017uk.co.ibtimes
4391963137812610.000024com.imageshack
440196301461100.000284com.mailchimp
4411962700828870.000011net.openreview
442196269244810.000052gov.whitehouse
4431962688413010.000023ch.ipcc
444196258589590.000030com.bandsintown
445196255983880.000064com.office
4461962403220390.000016edu.udel
4471962363618180.000017uk.ac.kcl
448196198669880.000029org.ilo
4491961863618800.000016tl.we
4501961812820920.000016io.gitlab
4511961669819750.000016com.digitaljournal
45219615278840.000440com.list-manage
45319614194150.002224com.wixstatic
4541961192817910.000017com.secondlife
4551960499811710.000025uk.gov.tfl
4561960364619940.000016org.peta
4571960288012520.000024com.medicalnewstoday
4581960184417440.000018com.teenvogue
45919601126450.000773net.fbcdn
4601960076818130.000017com.upi
461196004102050.000117com.etsy
4621959880015770.000019no.google
4631959770620970.000016com.shell
4641959673215350.000020com.quicksprout
465195966224060.000062com.fastcompany
4661959622613240.000023org.hrw
467195961645590.000045edu.berkeley
468195957368260.000036com.intel
4691959340819110.000016com.tomsguide
4701959276216550.000018ca.pinterest
471195914623650.000068com.hp
472195903126490.000039org.nodejs
4731958929621350.000015com.politifact
4741958851624000.000013com.towardsdatascience
4751958835622920.000014com.dailykos
4761958805817490.000018com.oprah
4771958523830390.000011org.arkive
478195847328590.000034com.engadget
4791958423817400.000018com.shareholder
480195842289670.000030ly.snip
4811957764613590.000022com.smallbiztrends
4821957760423840.000014com.hsbc
483195774141040.000312com.statcounter
484195773345660.000044com.photobucket
4851957646821610.000015org.jenkins-ci
4861957402410170.000028com.contentmarketinginstitute
4871956923824470.000013uk.co.spectator
4881956795819660.000016com.thecut
4891956739826550.000012uk.ac.mmu
4901956303014580.000021net.convio
4911956262618970.000016org.project-syndicate
492195626028570.000034com.deviantart
4931956231216580.000018google.ai
4941956091219210.000016com.ogilvy
4951956052817750.000017com.csoonline
496195594349900.000029com.cognitoforms
4971955839820290.000016link.page
4981955745222240.000015com.upworthy
4991955535616700.000018com.kinsta
500195515743930.000063com.getclicky
5011954879419070.000016ms.nyti
5021954829419510.000016uk.ac.leeds
5031954682212470.000024st.po
504195466903590.000069com.mapbox
5051954595823410.000014com.sciencealert
5061954512023870.000013com.instructure
5071954389423430.000014org.theiet
5081954329226200.000012com.ksl
5091954005421680.000015com.webbyawards
5101953788628520.000011com.brandyourself
5111953556427350.000012jp.hatenablog
5121953455227410.000012com.zynga
513195337803820.000066org.acm
5141953232218410.000017com.cmswire
515195319504310.000058io.codepen
5161953103213430.000022org.pocoo
5171953011229310.000011uk.co.autocar
518195299001600.000158com.tripadvisor
519195293722340.000099org.drupal
520195280289910.000029com.gizmodo
5211952514423170.000014org.aei
522195241488640.000034com.matterport
5231952314218710.000017uk.co.thesundaytimes
5241952123010410.000027com.tinypic
525195209448130.000036com.netflix
5261952042024390.000013com.newatlas
5271951876424100.000013com.triplepundit
528195186663810.000066com.booking
5291951832029780.000011fr.hellocoton
5301951736622010.000015org.unfpa
5311951630016030.000019pt.google
5321951400217150.000018net.openid
5331951133230800.000011com.blogsky
5341951124017630.000017com.bloglines
535195080142720.000089com.adnxs
5361950723221060.000015org.royalsociety
5371950658626590.000012com.asiaone
5381950428423080.000014com.waterstones
5391950385823420.000014com.financialexpress
5401950321216390.000019uk.org.nationaltrust
5411950277216460.000019org.pypi
542195012028990.000032com.highcharts
5431950079018890.000016org.panda
5441950070228980.000011org.ifaw
5451950070018280.000017org.thinkprogress
546194997249010.000032com.arstechnica
5471949823622030.000015com.kaggle
5481949765619480.000016org.wri
5491949480426930.000012co.electrek
5501949378623060.000014uk.org.wwf
5511949342624360.000013com.mongabay
5521949328233190.000010com.carscoops
5531949216210820.000027com.mixpanel
5541948655015020.000020io.fabric
5551948625812690.000023com.firebaseapp
556194858309060.000032edu.psu
5571948486818480.000017com.infolinks
5581948405616470.000018com.coschedule
5591948194016720.000018us.pa.state
5601948020022090.000015uk.ac.nhm
5611947965013020.000023com.clicky
562194777265000.000049tv.twitch
563194775445320.000047edu.cornell
564194770848720.000033edu.washington
56519476626710.000478com.livestream
5661947560023070.000014com.autonews
5671947452026600.000012pt.publico
5681947448619290.000016org.americanprogress
5691947419025780.000012com.nordvpn
5701947397222060.000015org.sonatype
5711947193014570.000021com.activecampaign
572194716126250.000041com.samsung
5731947130627300.000012com.delawareonline
5741947086028480.000011com.topgear
575194682409990.000029edu.upenn
5761946549417600.000017uk.gov.metoffice
5771946435227330.000012com.sc
5781946429825730.000013br.inpe
5791946038618730.000017com.prweek
5801946008625890.000012com.ecowatch
58119459484720.000477net.jsfiddle
5821945859032930.000010com.algorithmia
5831945721420270.000016com.scotsman
584194571264290.000058com.slack
5851945537218870.000016com.impactbnd
5861945374810080.000029uk.ac.cam
5871945331622630.000014com.articulate
5881945314027800.000012com.nouw
5891945126628960.000011com.flock
5901944903825710.000013org.globalcitizen
591194470065380.000046com.proofpoint
5921944599823350.000014com.googledrive
5931944426224340.000013nz.co.radionz
5941944422427630.000012jp.riken
5951944369023880.000013de.greenpeace
596194431901190.000244com.youku
597194421181740.000141jp.co.yahoo
5981944159828310.000011com.mumsnet
5991943992418740.000017com.crashlytics
600194391749650.000030edu.umich
6011943902821140.000015uk.org.rspb
602194380282080.000116uk.co.amazon
603194374481010.000321de.google
6041943579027480.000012com.quickanddirtytips
6051943183426680.000012au.com.huffingtonpost
6061943121618960.000016uk.gov.london
6071943069825410.000013com.thejakartapost
6081942948630970.000011com.shanghaidaily
609194288604150.000061com.xinhuanet
6101942861430690.000011com.theminimalists
6111942848612710.000023com.sprinklr
6121942649612080.000025org.iea
6131942646625120.000013ie.thejournal
6141942615217850.000017com.jeffbullas
6151942490229790.000011com.art
6161942464028370.000011it.polito
6171942300818080.000017com.martechtoday
6181942242625990.000012uk.co.profilebusiness
6191942149225340.000013com.db
6201942075628510.000011org.onegreenplanet
6211941839623400.000014net.opendemocracy
6221941695218690.000017org.iucnredlist
6231941390826880.000012uk.org.savethechildren
6241941261423790.000014com.theyworkforyou
625194116666950.000037com.xiti
6261940919826610.000012org.oceanconservancy
6271940871826830.000012com.dreamgrow
6281940797622540.000014com.rabbitmq
6291940737225680.000013com.shoutmeloud
6301940717010280.000028com.mcafeesecure
631194068664490.000055fr.free
632194036403620.000069org.npr
6331940207218650.000017com.copyscape
6341940130827910.000012com.sitesell
635194008803120.000079gov.cdc
6361939982824230.000013com.cleantechnica
6371939968628090.000012pl.edu.uw
638193972743990.000063com.nypost
639193968285690.000044com.aol
6401939644631670.000010com.seeker
6411939639027600.000012uk.org.amnesty
642193962122650.000090com.sohu
6431939596216130.000019com.flashtalking
6441939530825160.000013com.generalmills
6451939347220490.000016com.cityam
6461939247433800.000010com.dremel
647193923703960.000063com.163
6481939176230200.000011com.brothersoft
6491939167020610.000016org.gnupg
65019388022360.001003com.createjs
6511938766010270.000028edu.ucla
652193866305110.000048com.dmca
6531938544214950.000020scot.gov
6541938380623470.000014org.grist
6551938359224740.000013uk.org.oxfam
6561938176624570.000013uk.co.thisismoney
6571938048032590.000010org.aqicn
6581937984825660.000013uk.org.rspca
6591937919011690.000025com.hollywoodreporter
6601937874627260.000012org.irena
6611937782629080.000011org.kuow
6621937586629340.000011eu.i-scoop
6631937528231370.000011com.winefolly
664193742302440.000096com.bandcamp
6651937380613500.000022net.leadpages
6661937129818550.000017net.noscript
6671937072614380.000021com.pastebin
6681937012026920.000012com.targetmarketingmag
6691936852435160.000010co.edureka
6701936837627730.000012com.ipsos-mori
6711936828425460.000013org.zsl
6721936804423930.000013com.moodys
6731936789611700.000025gov.fbi
6741936768621820.000015com.thermofisher
6751936619828000.000012uk.ac.ceh
676193654842730.000089com.surveymonkey
6771936445617030.000018uk.co.which
6781936311814310.000021uk.gov.defra
6791936209226260.000012com.wikidot
6801936186421120.000015com.problogger
6811936143227940.000012com.pnsegypt
6821936048631320.000011com.hatenadiary
683193595721690.000143com.taobao
684193595063330.000074com.pubmatic
685193587703770.000066com.scribd
6861935874829850.000011org.storyofstuff
6871935810631680.000010org.heartland
6881935699829020.000011com.nationalgrid
689193557283520.000070com.wiley
690193550148860.000033com.windowsphone
6911935152825110.000013uk.gov.forestry
6921934981827460.000012org.spie
693193495968160.000036com.mobirise
6941934682229630.000011uk.ac.mdx
695193459364630.000054com.oreilly
6961934522822980.000014com.iconarchive
6971934497432130.000010edu.uah
698193441308930.000032edu.columbia
6991934384621960.000015uk.gov.food
7001934249227700.000012edu.dukeupress
7011934192825180.000013com.wral
7021933730612390.000024google.blog
703193371804530.000055com.sxsw
704193371086860.000038com.steampowered
7051933297228910.000011com.almanac
706193324969150.000031com.docker
707193321384330.000057com.force
708193308909130.000032org.reactjs
7091933043431580.000011com.dbs
7101933001233200.000010uk.org.bornfree
7111932994412830.000023uk.org.greenpeace
7121932832811000.000026com.redhat
7131932800412480.000024com.elpais
714193279247850.000036com.webs
7151932493434010.000010org.sciencenewsforstudents
7161932454834760.000010org.sharktrust
7171932367834470.000010uk.org.caat
718193222183050.000080com.digg
719193203843250.000076com.typeform
7201932019627560.000012com.batchgeo
7211931955821160.000015com.fifa
7221931748023890.000013org.chathamhouse
7231931711613220.000023org.whatbrowser
7241931709420980.000016org.fsc
7251931602417060.000018com.nike
7261931592623570.000014uk.co.inews
7271931582413620.000022edu.ucsd
7281931545834000.000010com.artstation
729193153868550.000034org.unesco
7301931526026540.000012com.ingress
7311931341415610.000019com.technologyreview
7321931275823750.000014io.pantheon
7331931184629520.000011com.climatechangenews
7341931108229810.000011org.c2es
7351930971417710.000017com.ikea
7361930950630100.000011com.foodsafetynews
7371930659825740.000012uk.org.38degrees
7381930574426760.000012com.thecvf
7391930547825880.000012org.carbonbrief
7401930545829900.000011org.sourcewatch
741193049685710.000043com.cbsnews
7421930459429860.000011com.moneysupermarket
743193041684690.000053com.statista
7441930409434140.000010me.start
7451930150828440.000011com.tiddlywiki
7461929969226450.000012com.bnef
7471929862030950.000011uk.co.bristolpost
748192974461980.000122io.polyfill
7491929700230590.000011jp.ac.kobe-u
750192968021220.000238org.networkadvertising
751192963185020.000049com.atlassian
752192940763380.000073com.prnewswire
7531929152211280.000026com.canva
7541928897830120.000011org.twinery
7551928882827370.000012com.adcolony
7561928845831170.000011no.forskning
7571928624627850.000012com.doctoroz
7581928485035560.000010com.cmgdigital
7591928467831430.000011com.sunherald
7601928406231720.000010com.ibmbigdatahub
7611928399235170.000010com.2createawebsite
7621928371629960.000011net.organicfacts
7631928285822430.000014com.privacypolicies
7641928212229050.000011com.winemag
7651928174610560.000027com.ubuntu
7661928151214190.000021uk.co.thesun
767192810864700.000053com.inc
7681928101021430.000015org.cites
7691928099022900.000014uk.gov.dft
7701927928031460.000011com.insideevs
7711927917427340.000012de.ksta
7721927842226840.000012com.e-activist
7731927837614120.000021com.speakerdeck
7741927689427470.000012com.chubb
7751927391626080.000012org.rspo
776192738949640.000030net.2mdn
7771927314232650.000010com.jordantimes
778192720343190.000078gov.ca
7791926891035060.000010com.idt
7801926842627570.000012com.theinnovationenterprise
7811926754223490.000014uk.gov.environment-agency
7821926747834960.000010com.sutori
783192664061510.000163ru.mail
784192662241640.000152com.yelp
7851926551031840.000010com.galvanize
7861926480034250.000010com.thewritepractice
7871926477832120.000010org.carbontracker
7881926457034640.000010org.earthworksaction
7891926354817130.000018com.martechseries
790192626389810.000029com.visualstudio
7911926216833830.000010com.nutraingredients
7921926169432220.000010com.quandl
7931926145214840.000020uk.co.foe
794192609242320.000100to.amzn
7951926017417310.000018org.khanacademy
7961926013026990.000012com.businessgreen
797192599205240.000047com.airbnb
7981925963432000.000010com.thedrinksbusiness
7991925870433840.000010com.monbiot
8001925848826850.000012au.com.mumbrella
8011925710230720.000011fr.thelocal
8021925672833300.000010org.cnduk
803192566286600.000039org.eff
8041925647614410.000021com.tutsplus
8051925592230900.000011ai.fast
8061925542227230.000012com.goinswriter
8071925517033580.000010org.thechicagocouncil
8081925393630290.000011jp.hatenadiary
8091925273027430.000012gov.ferc
8101925263413840.000022com.uber
8111925209434440.000010com.visitdublin
8121925095425820.000012nz.govt.mfat
8131924984422230.000015uk.gov.charitycommission
8141924940611920.000025edu.utexas
8151924911232730.000010com.chemistryworld
8161924899833000.000010org.alaskapublic
8171924898414180.000021fr.lemonde
8181924881231440.000011com.tuck
8191924722631560.000011com.marksdailyapple
8201924628410050.000029com.americanexpress
821192462045790.000043com.patreon
8221924506228140.000012com.ing
823192450321660.000147jp.co.google
8241924424419320.000016uk.gov.education
8251924289627530.000012com.webestools
8261924250225040.000013com.instructables
8271924246011850.000025edu.princeton
8281924055236450.000010com.theppk
8291924053633050.000010com.machinelearningmastery
8301923886417160.000018se.haxx
8311923871211490.000026com.digiday
832192384628960.000032com.zoho
8331923826846690.000009com.9to5mac
8341923760237610.000010org.muslimaid
835192358365410.000046com.alibaba
8361923573628170.000012uk.ac.rcplondon
837192338825560.000045gov.sec
8381923288030430.000011com.platts
8391923268826510.000012com.recyclenow
8401923261834410.000010org.thebestschools
8411923199433520.000010com.beruby
842192318262020.000119com.constantcontact
8431923100223540.000014net.privacypolicytemplate
8441923014232070.000010com.gpsvisualizer
8451922777431040.000011com.rabobank
8461922721633060.000010com.seat61
8471922719834120.000010uk.co.lep
848192261223110.000079com.marriott
849192246662390.000098cn.com.sina
850192242827530.000036com.css-tricks
851192235322460.000095jp.co.amazon
8521922284612990.000023gd.is
8531922182823500.000014uk.co.vogue
8541922142413810.000022com.dell
855192211187220.000037fm.last
8561922110420090.000016io.getmdl
8571922043037560.000010uk.org.stopwar
8581922019626270.000012org.ramsar
8591921798819870.000016com.instapage
860192174345950.000042com.psychologytoday
8611921720235920.000010com.fox13memphis
8621921639631340.000011uk.org.sja
8631921634235380.000010com.breakingenergy
8641921607034360.000010com.star2
8651921578431030.000011org.scielo
86619215692970.000332com.sharethis
867192156868810.000033com.aliexpress
8681921532036130.000010it.diggita
869192148362100.000116jp.ne.hatena
8701921461411250.000026com.firefox
871192144926340.000040gov.nist
8721921294032520.000010org.beatthemicrobead
8731921237236030.000010nl.zoom
8741921231012320.000024com.convertkit
875192078205450.000046uk.co.eventbrite
8761920733431450.000011com.abnamro
8771920638429040.000011org.wildlifetrusts
8781920608816370.000019org.whales
8791920575010680.000027com.shutterstock
8801920467639810.000009com.visitguatemala
8811920384631280.000011uk.org.scope
8821920328810300.000028com.foxnews
8831920314826750.000012org.soilassociation
8841920284210190.000028com.cbslocal
8851920084837020.000010no.haugenbok
8861919958629140.000011com.ironsrc
887191994269520.000030com.variety
8881919934426220.000012com.feedreader
889191988765170.000048com.ea
8901919832235950.000010uk.co.theboltonnews
8911919805211900.000025com.globo
8921919681828630.000011com.itsma
8931919609815640.000019org.freecsstemplates
8941919597820780.000016com.hulu
8951919567231480.000011com.rebekahradice
896191952622890.000084com.discordapp
8971919476235430.000010info.e-ir
8981919454633640.000010org.swi-prolog
8991919228831820.000010com.wpxi
900191916984860.000051com.nasdaq
9011919086431700.000010uk.co.dennis
9021919066835510.000010com.alaskadispatch
9031919046011500.000026com.java
904191902762300.000100com.googletagservices
9051918967635040.000010es.ree
9061918956231620.000010com.sgx
9071918862637210.000010br.org.imazon
9081918833437760.000010com.citymayors
9091918818235820.000010au.com.hotfrog
9101918706632720.000010uk.org.cat
9111918628032790.000010aq.ats
912191861886780.000038com.newyorker
9131918469040020.000009net.politicalscrapbook
9141918434436060.000010com.southernfriedscience
9151918348431250.000011app.web
916191825782200.000106com.naver
9171918177017360.000018com.techrepublic
9181918097836320.000010com.theoildrum
9191918072837280.000010org.worldnuclearreport
920191806741560.000162gov.privacyshield
9211917911828710.000011uk.co.realbusiness
9221917725414630.000021edu.uchicago
9231917672614530.000021tv.ustream
9241917516418070.000017com.nba
9251917335232710.000010uk.org.cpre
9261917318817880.000017org.golang
9271917240229550.000011com.writetothem
9281917236820410.000016com.howstuffworks
9291917089614070.000021uk.co.theregister
930191706684640.000054com.adweek
931191706302430.000096com.stumbleupon
9321917048015790.000019edu.unc
9331916944422110.000015edu.virginia
9341916886036190.000010com.renewablesnow
9351916851813900.000022com.over-blog
9361916780014430.000021com.digitaltrends
9371916778240730.000009uk.co.moblog
9381916514014060.000021us.imageshack
9391916472436040.000010com.at0086
9401916449221440.000015org.coursera
9411916442837990.000010com.avivaromm
942191625829840.000029com.thinkwithgoogle
9431916245036290.000010com.eremnews
944191616604660.000053com.snapchat
9451915991814420.000021com.billboard
9461915990433940.000010uk.gov.peterborough
9471915950635300.000010org.campaigncc
948191586006420.000039org.pbs
9491915759032990.000010uk.co.siemens
9501915757434700.000010org.ilga-europe
951191562589780.000029com.dropboxusercontent
952191543808940.000032com.uservoice
9531915425215780.000019com.ssllabs
9541915399233670.000010com.trafficgenerationcafe
9551915225616140.000019com.warnerbros
956191520429220.000031com.libsyn
9571915185236650.000010uk.org.biofuelwatch
9581915171836170.000010uk.org.garyhall
9591915154823990.000013com.ehow
9601915082037710.000010no.universitetsforlaget
9611914843035590.000010br.org.idec
962191482908390.000035com.qz
9631914816429110.000011net.nend
964191474226900.000038com.webmd
9651914723816940.000018com.codeplex
9661914485213740.000022com.fiverr
9671914458439220.000009net.kjokkenutstyr
968191445724970.000049edu.cmu
9691914415840060.000009org.freedom-now
9701914393211670.000025com.smashingmagazine
9711914360431470.000011uk.org.refill
9721914337219710.000016com.invisionapp
9731914225628640.000011com.dzone
9741914215834900.000010io.dataquest
9751914153839840.000009org.alqaws
9761914123032420.000010io.dropwizard
9771914066238210.000010com.superiorthreads
9781914042033980.000010uk.co.firstnews
979191389943550.000070org.debian
9801913815421340.000015com.w3layouts
981191343888770.000033com.foursquare
9821913404030350.000011com.vungle
9831913371632050.000010org.corporateeurope
984191336448660.000034gov.census
9851913349639950.000009com.tinnedtomatoes
9861913338214000.000021com.blackberry
9871913333613350.000022jp.livedoor
9881913247633340.000010com.drillordrop
9891913183633290.000010com.ovoenergy
9901913117238040.000010com.descarteslabs
9911913077810640.000027com.politico
9921912888837880.000010org.ianfairlie
9931912872818660.000017com.nokia
9941912778626690.000012in.bbc
9951912751229870.000011org.vegsoc
9961912710833870.000010com.figure-eight
997191258183570.000070gov.ftc
998191255962420.000097org.icann
9991912535814010.000021com.xkcd
10001912521636090.000010br.com.ambev

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

October 2019 crawl archive now available

The crawl archive for October 2019 is now available! It contains 3.0 billion web pages or 280 TiB of uncompressed content, crawled between October 13th and 24th. It includes page captures of 1.1 billion URLs not contained in any crawl archive before.

Archive Location and Download

The October crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-43/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://commoncrawl.s3.amazonaws.com/ to each line, you end up with the S3 and HTTP paths respectively.

 File List#FilesTotal Size
Compressed (TiB)
SegmentsCC-MAIN-2019-43/segment.paths.gz100
WARC filesCC-MAIN-2019-43/warc.paths.gz5600059.56
WAT filesCC-MAIN-2019-43/wat.paths.gz5600021.7
WET filesCC-MAIN-2019-43/wet.paths.gz560009.94
Robots.txt filesCC-MAIN-2019-43/robotstxt.paths.gz560000.15
Non-200 responses filesCC-MAIN-2019-43/non200responses.paths.gz560001.69
URL index filesCC-MAIN-2019-43/cc-index.paths.gz3020.22

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-43/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

September 2019 crawl archive now available

The crawl archive for September 2019 is now available! It contains 2.55 billion web pages or 240 TiB of uncompressed content, crawled between September 15th and 24th. It includes page captures of 1.0 billion URLs not contained in any crawl archive before. The other 1.5 billion pages have been already captured in prior crawls and are now revisited.

Archive Location and Download

The September crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-39/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://commoncrawl.s3.amazonaws.com/ to each line, you end up with the S3 and HTTP paths respectively.

 File List#FilesTotal Size
Compressed (TiB)
SegmentsCC-MAIN-2019-39/segment.paths.gz100
WARC filesCC-MAIN-2019-39/warc.paths.gz5600055.99
WAT filesCC-MAIN-2019-39/wat.paths.gz5600018.01
WET filesCC-MAIN-2019-39/wet.paths.gz560008.16
Robots.txt filesCC-MAIN-2019-39/robotstxt.paths.gz560000.14
Non-200 responses filesCC-MAIN-2019-39/non200responses.paths.gz560001.63
URL index filesCC-MAIN-2019-39/cc-index.paths.gz3020.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-39/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

August 2019 crawl archive now available

The crawl archive for August 2019 is now available! It contains 2.95 billion web pages or 260 TiB of uncompressed content, crawled between August 17th and 26th.

The August crawl contains page captures of 1.1 billion URLs not contained in any crawl archive before. New URLs are sampled based on the host and domain ranks (harmonic centrality) published as part of the May/Jun/Jul 2019 webgraph data set from the following sources:

  • a random sample of 2.1 billion outlinks extracted from July crawl WAT files
  • 1.8 billion URLs mined in a breadth-first side crawl within a maximum of 6 links (“hops”), started from
    • the homepages of the top 60 million hosts and domains and randomly selected samples of
    • 2 million human-readable sitemap pages (HTML format)
    • 3 million URLs of pages written in 130 less-represented languages (cf. language distributions)
  • 1 billion URLs extracted and sampled from 20 million sitemaps, RSS and Atom feeds

Starting with this crawl the following fixes and improvements are applied to the provided data formats:

Archive Location and Download

The August crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-35/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://commoncrawl.s3.amazonaws.com/ to each line, you end up with the S3 and HTTP paths respectively.

 File List#FilesTotal Size
Compressed (TiB)
SegmentsCC-MAIN-2019-35/segment.paths.gz100
WARC filesCC-MAIN-2019-35/warc.paths.gz5600053.53
WAT filesCC-MAIN-2019-35/wat.paths.gz5600020.85
WET filesCC-MAIN-2019-35/wet.paths.gz560009.29
Robots.txt filesCC-MAIN-2019-35/robotstxt.paths.gz560000.18
Non-200 responses filesCC-MAIN-2019-35/non200responses.paths.gz560001.79
URL index filesCC-MAIN-2019-35/cc-index.paths.gz3020.22

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-35/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

Host- and Domain-Level Web Graphs May/June/July 2019

We are pleased to announce a new release of host-level and domain-level web graphs based on the published crawls of May, June and July 2019. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark on GitHub which host all scripts and tools required to construct the graphs.

What’s new?

Links from Content-Location and Link HTTP headers are now also used to span up the web graphs. This is in accordance with RFC 5988 which defines the Link HTTP header as semantically equivalent to the element in HTML. It also fits previous web graph releases which used to include all kinds of links including technical ones and redirects.

Host-level graph

The graph consists of 445 million nodes and 3.14 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 382 million dangling nodes (86%) and the largest strongly connected component contains 48 million (11%) nodes.

You can download the graph and the ranks of all 445 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-may-jun-jul/host/. Alternatively, you can use https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2019-may-jun-jul/host/ as prefix to access the files from everywhere.

Download files of the Common Crawl May/June/July 2019 host-level webgraph

SizeFileDescription
3.02 GBcc-main-2019-may-jun-jul-host-vertices.paths.gznodes ⟨id, rev host⟩, paths of 28 vertices files
14.75 GBcc-main-2019-may-jun-jul-host-edges.paths.gzedges ⟨from_id, to_id⟩, paths of 56 edges files
6.42 GBcc-main-2019-may-jun-jul-host.graphgraph in BVGraph format
2 kBcc-main-2019-may-jun-jul-host.properties
7.01 GBcc-main-2019-may-jun-jul-host-t.graphtranspose of the graph (outlinks inverted to inlinks)
2 kBcc-main-2019-may-jun-jul-host-t.properties
1 kBcc-main-2019-may-jun-jul-host.statsWebGraph statistics
7.22 GBcc-main-2019-may-jun-jul-host-ranks.txt.gzharmonic centrality and pagerank

Note that the host names are reversed and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 88 million nodes and 1.9 billion edges. 52% or 46 million nodes are dangling nodes, the largest strongly connected component covers 35 million or 40% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2019-may-jun-jul/domain/ resp. https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2019-may-jun-jul/domain/.

Download files of the Common Crawl May/June/July 2019 domain-level webgraph

SizeFileDescription
0.61 GBcc-main-2019-may-jun-jul-domain-vertices.txt.gznodes ⟨id, rev domain, num hosts⟩
7.50 GBcc-main-2019-may-jun-jul-domain-edges.txt.gzedges ⟨from_id, to_id⟩
4.06 GBcc-main-2019-may-jun-jul-domain.graphgraph in BVGraph format
2 kBcc-main-2019-may-jun-jul-domain.properties
3.99 GBcc-main-2019-may-jun-jul-domain-t.graphtranspose of the graph
2 kBcc-main-2019-may-jun-jul-domain-t.properties
1 kBcc-main-2019-may-jun-jul-domain.statsWebGraph statistics
1.91 GBcc-main-2019-may-jun-jul-domain-ranks.txt.gzharmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 90 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (May/June/July 2019)

harmonic
centrality
rank
hc valuepage rankpage rank
value
reversed hostname
12997766810.020841com.googleapis
22786770430.011812com.facebook
32741998020.012857com.google
42519603040.007273com.twitter
52455883650.006439org.w
62453370260.005984com.youtube
72259209890.003799com.instagram
82206065070.004857org.gmpg
921829028130.002863com.linkedin
102159544680.004481com.googletagmanager
1120930920220.001704com.gravatar
1220912076240.001531com.pinterest
1320730700110.003384com.cloudflare
1420698732170.002180com.wordpress
1520613210120.003087org.wordpress
1620607942260.001241org.wikipedia
1720408594140.002452com.bootstrapcdn
1820351540200.001823com.apple
1920148418410.000904com.blogspot
2020103846300.001124com.vimeo
2120036764210.001719com.jquery
2219874716500.000673com.wp
2319870332290.001130com.microsoft
2419839912430.000816gl.goo
2519828406450.000769com.amazon
2619793040180.002021com.gstatic
2719790998190.002015com.adobe
2819788744570.000573com.tumblr
2919754126310.001104com.amazonaws
3019619798250.001407com.macromedia
3119616602340.001057com.googlesyndication
3219585788470.000744be.youtu
3319585670390.000937com.google-analytics
3419583342620.000531ly.bit
3519572994680.000440com.yahoo
3619549710330.001080com.flickr
3719526876350.001023net.cloudfront
3819526762230.001676com.github
3919503814600.000553me.wp
4019467672270.001170ru.yandex
4119467424580.000568org.mozilla
42194548901060.000305com.googleusercontent
4319411724490.000725net.doubleclick
4419374766520.000658co.t
4519366860440.000776com.baidu
4619322188700.000401com.weebly
47193217541050.000310com.reddit
48193170941230.000234com.nytimes
4919313908460.000749com.paypal
50193080941040.000312com.soundcloud
5119278436670.000448com.medium
5219268558660.000451io.github
5319266970630.000517org.w3
5419255616800.000379org.creativecommons
55192280341840.000143uk.co.bbc
56192194701750.000151com.imgur
57191914841370.000194com.forbes
58191843201680.000154net.slideshare
5919169524560.000588org.schema
60191662141620.000162com.bing
61191633881800.000144net.sourceforge
62191558821820.000143org.wikimedia
6319145618480.000738com.googleadservices
64191438402150.000109com.businessinsider
65191360402330.000104com.techcrunch
66191251982730.000089com.reuters
67191127301520.000169com.theguardian
68190917081770.000147com.imdb
6919081148640.000496net.jsdelivr
70190766421450.000177org.apache
71190689122020.000120org.gnu
72190677202500.000097com.ibm
73190659042740.000089com.cnet
74190604021940.000124com.washingtonpost
75190561621640.000159com.blogger
76190496223360.000073gov.nasa
77190434062710.000090com.android
7819038878320.001080com.fontawesome
79190308241960.000123com.huffingtonpost
80190227642430.000100com.oracle
8119022114990.000323com.shopify
82190100921780.000147com.stackoverflow
83190088662640.000092com.bbc
84189915041380.000194com.wixsite
85189797941930.000128org.ampproject
86189796063310.000074com.latimes
87189669243340.000073com.livejournal
88189543521480.000171com.eventbrite
89189529144060.000061com.zdnet
9018951470380.000950com.addthis
91189411682600.000093com.usatoday
92189303062610.000093com.wired
93189299484730.000052com.economist
94189248941220.000237com.ytimg
95189158202950.000083com.prnewswire
96189077841070.000304com.whatsapp
97189055622410.000101com.appspot
98189037502890.000086org.npr
99188998266050.000046com.thenextweb
100188987321390.000192com.issuu
101188971301980.000122org.ietf
102188931881810.000143jp.co.yahoo
103188890961420.000183com.spotify
104188887604490.000055com.venturebeat
10518888186550.000590eu.europa
106188862403820.000064com.goodreads
10718880882370.000994com.qq
108188808806010.000046org.ieee
109188769882090.000114com.bandcamp
110188744483590.000068com.quora
111188726664260.000058com.cisco
112188696402110.000112net.behance
113188665604740.000052org.arxiv
114188520803940.000062com.buzzfeed
11518844806950.000330com.sharethis
116188345024270.000058com.deviantart
117188341668990.000031com.ibtimes
118188297621850.000141com.giphy
11918828960960.000328com.statcounter
120188250746490.000043com.stackexchange
121188236241700.000152uk.co.google
122188188482830.000087com.cnbc
123188173848250.000034org.eclipse
124188145663330.000074com.aol
125188143924850.000051com.pixabay
126188069442060.000117com.disqus
127188009124580.000054com.about
12818793968420.000849com.squarespace
129187935725220.000048com.mysql
130187927401440.000180com.yelp
131187907943550.000068com.theatlantic
132187874244170.000059me.about
133187870063170.000077com.skype
134187826364760.000052com.visualstudio
135187805382320.000104me.t
136187726669480.000030com.nvidia
137187725604680.000053com.wikihow
138187683582760.000089com.sciencedirect
139187678222240.000106com.dribbble
140187622663240.000075com.scribd
141187592367120.000039google.blog
142187568861830.000143com.salesforce
143187562365510.000048com.slate
144187539681310.000208com.dropbox
145187516964070.000061uk.co.independent
146187512422990.000081com.fastcompany
147187465902570.000094com.googlecode
148187461422130.000111com.hubspot
149187444704400.000057com.newyorker
150187444524300.000058com.box
151187433321200.000249org.networkadvertising
152187369566670.000042org.chromium
153187359184630.000053gov.loc
154187341902970.000082com.example
155187337622000.000121com.cnn
156187314286710.000041com.tinypic
157187281602690.000090com.fc2
158187261047900.000035com.nymag
159187231847070.000039com.smashingmagazine
160187197046160.000045com.evernote
161187185762720.000090com.nbcnews
162187163965480.000048net.azurewebsites
163187106062190.000108com.npmjs
164187097701550.000167org.archive
165187087683060.000079com.w3schools
1661870509010240.000028ca.utoronto
167187038801910.000130jp.ne.hatena
168186999744770.000052io.codepen
16918699212610.000544com.vk
170186990169690.000029com.ign
171186946927030.000039com.speakerdeck
172186942568530.000033com.mediafire
173186916285060.000049com.foursquare
174186861028940.000031com.nike
175186841366080.000046com.trello
176186793021190.000251info.aboutads
177186761683760.000066com.mozilla
17818670790530.000604com.wix
179186697806390.000044uk.ac.ox
180186642641460.000174com.amazon-adsystem
181186611621030.000317com.paypalobjects
18218658320840.000366com.bizjournals
183186534383420.000072com.getpocket
184186390783160.000077ca.google
185186363244980.000050com.indiatimes
186186283165960.000047com.pinimg
187186261626240.000045com.cbslocal
188186242783110.000078edu.mit
189186238789420.000030com.chron
190186224201140.000272net.windows
1911861878611580.000025org.tensorflow
192186183267260.000038ca.blogspot
193186176028420.000033com.sap
194186156788410.000033com.css-tricks
195186121443600.000068com.entrepreneur
196186060506230.000045com.libsyn
197186033401340.000205com.unpkg
198186023021170.000253com.stripe
199186003523080.000079edu.harvard
200185974642260.000106com.wsj
2011859521410700.000026com.hackernoon
202185941748360.000033com.thehill
20318592786590.000557com.fb
204185905106250.000045ca.cbc
205185901729120.000031org.unicode
206185866107920.000035com.buffer
207185858803690.000067com.elsevier
208185811267940.000035com.theglobeandmail
20918580570150.002238com.wixstatic
210185793263630.000068me.telegram
211185788586620.000042com.searchengineland
212185764021790.000147org.bbb
213185749006560.000043site.business
214185744764810.000051com.withgoogle
215185743462520.000097es.google
216185726168740.000032org.kernel
217185724986440.000044com.flipboard
218185711067250.000038co.ibb
219185650146580.000042com.huffpost
2201856358810050.000028edu.rutgers
221185627888480.000033uk.co.wired
222185607447590.000036com.ssrn
223185606061130.000272com.weibo
2241855764610390.000027com.aljazeera
225185558607360.000037gov.archives
226185543383460.000071com.mapbox
227185540086370.000044org.d3js
228185532781510.000170com.yimg
2291855109810930.000026org.hrw
230185491046030.000046gg.discord
2311854683414680.000020com.hm
2321854670811460.000025ly.visual
233185459689850.000029com.geekwire
234185454642010.000120com.optimizely
2351854475412510.000023ca.huffingtonpost
236185441142120.000111edu.stanford
237185428609430.000030uk.co.huffingtonpost
238185423346180.000045co.elastic
2391853924218770.000017com.pearltrees
2401853588018290.000017cn.people
2411853069014020.000021com.diigo
242185288042960.000082com.tinyurl
243185280344550.000054com.mapquest
244185255509790.000029org.slashdot
2451852427611060.000025edu.osu
24618523478650.000473net.akamaihd
247185223168820.000032com.theconversation
248185174362780.000089org.purl
249185173623750.000066com.mashable
2501851427210970.000026com.dw
251185137709340.000030com.bt
252185115128440.000033com.today
253185114908770.000032com.marketwired
2541851032412670.000023jp.co.ntv
2551850984210990.000026com.mentalfloss
256185095889860.000029com.computerworld
2571850527613760.000021jp.ac.u-tokyo
258185052248400.000033co.g
259185030728470.000033com.healthline
260185027047820.000035com.ecwid
2611850161014280.000021com.sas
262185015264650.000053com.yoast
2631849900611010.000025edu.gatech
264184900784540.000054com.moz
2651848797616360.000019com.kaggle
2661848777213710.000021com.makeuseof
267184874005010.000050me.m
268184871642800.000088com.bloomberg
269184871547710.000036com.econsultancy
270184868388800.000032uk.parliament
271184837708200.000034com.newsweek
2721848373013000.000022com.googlesource
2731848064014260.000021blog.home
274184802547610.000036com.outbrain
2751847830610770.000026com.sfchronicle
276184755702040.000119org.iana
277184736423130.000077com.scorecardresearch
278184714661540.000169gov.nih
2791846176212790.000022com.avg
280184598224600.000054com.theverge
281184577326360.000044jp.shinobi
2821845503610090.000028org.postgresql
2831845205614150.000021com.dailydot
284184495729400.000030com.foxbusiness
285184483308140.000034com.adjust
286184470547640.000036edu.brookings
287184460747670.000036com.business2community
2881843792818610.000017com.uniqlo
2891843586614460.000020com.dezeen
290184333103470.000071com.trustpilot
291184327228050.000035com.contentmarketinginstitute
2921843196810360.000027com.trendmicro
293184312229090.000031org.aarp
294184309228500.000033com.searchenginewatch
295184280423090.000078org.python
296184279841600.000163com.twimg
297184274545080.000049edu.berkeley
298184252007680.000036uk.co.pinterest
299184233964460.000055com.bigcommerce
3001841826613580.000021edu.iastate
3011841717213070.000022com.motherjones
302184169307150.000039com.techtarget
303184140682810.000087com.myspace
3041841352211540.000025com.hostgator
3051841131410460.000027com.medicalnewstoday
3061841069610250.000028com.bustle
30718410084690.000420com.list-manage
308184095623230.000076uk.co.telegraph
309184093623300.000074com.meetup
3101840869011680.000024org.openoffice
3111840601212960.000022com.contently
312184035327200.000038com.cdbaby
313184020045140.000049com.adage
3141840140413370.000022org.wnyc
315184002867140.000039com.neilpatel
3161839861015270.000020com.mathworks
317183975184780.000052net.researchgate
318183945989380.000030co.apple
319183945863290.000074com.go
32018393232940.000347com.godaddy
321183920264110.000060com.msn
322183916744040.000061com.ted
3231839014814840.000020io.material
324183899948170.000034com.arstechnica
325183895088600.000033com.wikia
326183881889700.000029com.vogue
327183849123770.000066me.wa
3281838202014750.000020se.blogspot
329183806707890.000035edu.washington
330183803621650.000158com.opera
331183771062480.000098com.rawgit
332183769388190.000034com.bandsintown
3331837455810660.000026com.convinceandconvert
3341837415412380.000023com.convertkit
3351837385018710.000017io.soup
3361837031014380.000020com.secondlife
3371836615217210.000018com.zara
338183644382870.000086com.live
339183626282380.000102com.surveymonkey
340183586541880.000132com.etsy
341183563881690.000153com.feedburner
3421835609419440.000016edu.uark
3431835504819110.000017com.mysanantonio
344183547262660.000092uk.org.ico
345183526304290.000058org.hbr
346183524066020.000046com.livechatinc
3471835205814930.000020com.thenation
348183515867500.000037com.yellowpages
349183499221120.000281com.mailchimp
350183491268150.000034com.wordstream
3511834906215060.000020com.toptal
3521834703612160.000024io.itch
353183426264940.000050com.kickstarter
354183415722350.000104com.typepad
355183406084200.000059com.googleblog
356183380463660.000068com.aliyuncs
3571833776016700.000018com.manta
3581833760014630.000020com.amcharts
3591833665214190.000021com.indiewire
360183353784560.000054com.fortune
36118333310510.000663net.fbcdn
362183332862310.000105uk.co.amazon
3631833109215870.000019ly.adobe
364183297609220.000030com.searchenginejournal
3651832860215690.000019ms.nyti
366183257443740.000066com.ft
3671832351620040.000016com.zoominfo
3681832344211790.000024com.grammarly
3691832178016200.000019li.paper
3701832175012100.000024com.csmonitor
3711832151221480.000015com.brandyourself
3721832071620760.000015me.websta
373183108423400.000072com.getclicky
374183104009110.000031uk.gov.nationalarchives
375183068468630.000033com.engadget
376183040661590.000165com.zendesk
377183009809620.000029com.cio
37818300896870.000360de.google
3791830085216250.000019id.co.blogspot
3801830075217350.000018org.unfpa
381182994786860.000040com.intel
382182977164310.000058com.nationalgeographic
3831829768419420.000016com.cinemablend
3841829549219390.000016com.wral
385182953726630.000042com.vice
386182946684430.000056com.oreilly
3871829455410200.000028com.weddingwire
388182930344610.000053com.nature
3891829299814400.000020com.harpercollins
390182915702900.000085gov.cdc
391182906583640.000068com.githubusercontent
392182903685200.000048com.photobucket
393182903649260.000030com.socialmediaexaminer
394182900209980.000028com.firebaseapp
395182891508750.000032com.angieslist
396182888429010.000031com.sendpulse
397182886288220.000034edu.columbia
398182877508230.000034com.pexels
3991828660015410.000019com.mindbodygreen
4001827945215160.000020com.mailjet
401182783561490.000171com.tripadvisor
402182781683190.000077com.wiley
4031827693018500.000017com.merchantcircle
404182764542680.000090com.digg
4051827608818900.000017fr.huffingtonpost
4061827574616950.000018com.thoughtworks
4071827376010140.000028org.ocks
4081827322020620.000015jp.pinterest
409182727684840.000051com.cbsnews
410182718783520.000069int.who
411182705288160.000034com.format
412182701082550.000096net.php
4131826992414640.000020com.thecut
4141826865820550.000015org.spie
415182645542140.000110org.aboutcookies
4161826330012310.000023com.mynewsdesk
417182617324090.000060com.office
4181826162410710.000026com.fastcodesign
4191826085614520.000020fr.liberation
420182607743350.000073com.time
421182603664440.000056org.freecodecamp
4221826002016060.000019com.dummies
4231825940017780.000018com.instapaper
424182589307550.000036com.mediapost
425182558426300.000044com.proofpoint
4261825411818780.000017it.binged
4271825408613210.000022ly.snip
428182528584160.000059uk.co.dailymail
429182492606040.000046org.nodejs
430182485903920.000062fr.free
431182484924640.000053com.statista
432182473568790.000032com.gizmodo
433182466463150.000077com.st-hatena
4341824538816600.000018com.superpages
4351824407811200.000025com.theknot
436182436783570.000068com.unsplash
4371824149413970.000021com.jeffbullas
4381823620815220.000020com.biography
4391823594621460.000015de.huffingtonpost
4401823482014320.000021com.csoonline
4411823472614860.000020com.louisvuitton
442182335121210.000246com.jimdo
4431823292010400.000027uk.ac.cam
4441823234813380.000022google.ai
4451823158621900.000014com.mango
4461823090212270.000023com.activecampaign
447182263369640.000029com.netlify
448182261729530.000030com.eater
4491822398410040.000028com.smallbiztrends
4501822356421050.000015site.negocio
451182231002770.000089com.ebay
4521822177813010.000022ca.yellowpages
453182204226890.000040com.windowsphone
454182203667750.000035com.marketwatch
4551821971411470.000025com.redhat
4561821797221700.000015edu.scad
4571821766012900.000022com.digitaltrends
4581821731811230.000025org.mathjax
4591821667016580.000019com.politifact
4601821554622250.000014com.dexknows
461182147904900.000050gov.whitehouse
4621821004412250.000023com.quicksprout
463182074941760.000150com.slack
4641820520816550.000019uk.co.bbci
4651820319413360.000022com.cmswire
46618202308790.000382net.jsfiddle
4671819967416830.000018com.nyt
4681819849019280.000016com.itsnicethat
469181974928350.000034edu.psu
470181968563540.000068com.booking
471181967966880.000040com.webs
472181958529600.000030edu.ucla
473181913647010.000039gov.nist
474181911389450.000030com.sprinklr
475181911023070.000079gov.ca
47618188332760.000389com.livestream
4771818690813750.000021net.openid
4781818675011310.000025gov.fbi
479181858344750.000052tv.twitch
4801818349819820.000016google.design
4811817695017900.000017com.psmag
482181757887740.000036com.oath
4831817381614980.000020org.gnupg
484181721443510.000069com.hp
485181716302910.000085org.acm
4861816729624880.000013org.travelblog
4871816703222430.000014com.ingress
4881816557812640.000023com.coschedule
4891816476617460.000018com.financialexpress
4901816464818680.000017com.allafrica
4911816436011100.000025edu.princeton
4921816367223050.000014com.tommy
4931816337616290.000019org.whatbrowser
4941816234412990.000022com.kinsta
4951816131224410.000014com.algorithmia
496181611405300.000048net.brightcove
4971815843220240.000016jp.riken
498181576668310.000034com.msdn
499181576405110.000049edu.cornell
5001815656822790.000014com.theminimalists
501181539402360.000103to.amzn
5021815384014600.000020net.noscript
503181470563050.000079com.typeform
5041814683817270.000018com.iconarchive
505181453609280.000030org.weforum
506181446228380.000033com.git-scm
5071814354022010.000014net.organicfacts
5081814024217240.000018com.gap
509181389007090.000039org.bitbucket
510181366304030.000061com.dailymotion
511181346483930.000062com.nypost
5121813442422440.000014com.bonfire
5131813383220950.000015it.polito
514181325729030.000031com.sfgate
515181305442390.000101com.stumbleupon
5161813038622730.000014net.brownbook
5171812959420730.000015com.zynga
518181272965230.000048edu.yale
5191812613416560.000019com.wayfair
520181259982540.000096org.drupal
521181259263810.000065org.un
5221812381223000.000014com.23hq
523181228366140.000045gov.sec
524181201564150.000059com.gmail
5251811968411960.000024com.playstation
5261811823417170.000018org.polymer-project
5271811332817720.000018za.co.iol
5281811254819940.000016au.com.huffingtonpost
5291811092223440.000014com.marksdailyapple
5301811089415070.000020com.impactbnd
531181097406480.000043com.jwplatform
5321810923615130.000020com.instapage
5331810729213740.000021com.ning
5341810699020350.000015com.dreamgrow
535181061222920.000085cn.com.sina
5361810501021880.000014net.openreview
5371810480616040.000019com.aolcdn
538181040042100.000113com.constantcontact
5391810387418890.000017uk.ac.jisc
5401810365619930.000016com.towardsdatascience
5411810183016690.000018com.thermofisher
5421810027019240.000016com.city-data
543180999009410.000030uk.co.guardian
5441809982621360.000015com.whitepages
5451809950018910.000017com.deepmind
546180984966110.000046com.mobirise
547180974403560.000068com.springer
5481809627819290.000016org.elasticsearch
549180943907430.000037com.steampowered
5501809204810910.000026com.auth0
551180920081920.000128com.eepurl
5521809169412140.000024kr.or.kisa
553180908008320.000034gov.senate
55418090404710.000398me.fb
5551809018839300.000010com.artstation
556180901106840.000040org.eff
5571808850620750.000015com.quickanddirtytips
5581808822018560.000017com.googledrive
5591808789022670.000014lb.com.dailystar
5601808737010830.000026de.spiegel
5611808718421890.000014com.oilprice
5621808659813770.000021io.bower
5631808658619970.000016com.batchgeo
5641808636010600.000027com.clicky
5651808599011720.000024com.merriam-webster
5661808474619270.000016com.nytco
567180842722840.000087com.histats
5681808385616130.000019org.jenkins-ci
5691808358019660.000016com.underconsideration
5701808309022210.000014com.swatch
571180818686170.000045uk.co.blogspot
572180789363430.000071com.sxsw
573180785746190.000045com.patreon
5741807738814710.000020io.getmdl
5751807650610810.000026com.hollywoodreporter
576180753946100.000046com.163
577180753481560.000166ru.mail
5781807494018450.000017com.rabbitmq
5791807463617830.000017com.lexology
5801807455016650.000018com.invisionapp
5811807427219870.000016com.lightreading
5821807390613510.000021edu.northwestern
583180735569960.000028com.ubuntu
5841807345421110.000015edu.dukeupress
5851807185221730.000015org.onegreenplanet
5861807148021410.000015com.hotfrog
5871807059223610.000014edu.uah
5881806887213690.000021org.khanacademy
5891806843814610.000020uk.co.thesun
5901806690434870.000012com.wikidot
5911806637016140.000019com.digitaloceanspaces
5921806560224320.000014net.sott
5931806550015470.000019com.technologyreview
594180652823490.000070com.staticflickr
59518063590780.000383org.reactjs
596180619546600.000042com.xinhuanet
5971806177625550.000013com.idt
598180616782470.000098de.amazon
599180612687390.000037com.qz
6001805793619310.000016com.googleapps
6011805788417530.000018io.pantheon
6021805776821320.000015net.eenews
603180571167790.000035com.deloitte
6041805703816510.000019com.checkatrade
605180546226570.000043com.psychologytoday
606180545969000.000031gov.nps
6071805107819730.000016com.shoutmeloud
6081804917427610.000013ca.411
6091804862014960.000020com.citysearch
6101804818417480.000018com.tutsplus
6111804499021260.000015io.flutter
6121804403620640.000015com.vanguardngr
6131804324214730.000020edu.unc
6141804317419340.000016com.gimletmedia
6151804262616270.000019com.fifa
6161804143624630.000013org.simile-widgets
617180412309320.000030edu.upenn
6181804093020980.000015com.designobserver
619180408346610.000042org.pbs
6201804055224100.000014com.ubu
6211804042211180.000025net.recode
6221803956612080.000024jobs.amazon
623180384463450.000071com.tripod
6241803656213150.000022edu.purdue
625180351969210.000030com.variety
626180345029800.000029com.alexa
6271803415012110.000024us.imageshack
6281803317421980.000014edu.arizona
6291803286220080.000016in.huffingtonpost
6301803073415920.000019com.yell
631180302789740.000029org.sciencemag
6321802972813200.000022uk.co.theregister
6331802824616790.000018com.verywellmind
634180259548520.000033org.worldbank
635180256388650.000033io.readthedocs
636180251041300.000208com.youku
6371802471421780.000015com.epochtimes
6381802430621860.000015info.bem
639180233982210.000107com.taobao
6401802228811440.000025com.elpais
6411802180019630.000016org.dartlang
6421802156610880.000026org.altervista
643180212943580.000068org.debian
644180209924450.000056com.force
6451802094012750.000023com.ifttt
6461802046622090.000014com.youm7
6471801964010730.000026com.vox
6481801956819330.000016com.hulu
6491801903222560.000014au.com.yellowpages
6501801898025050.000013com.pushwoosh
6511801661211770.000024com.nydailynews
652180161306980.000039gov.noaa
6531801460016570.000019com.yext
654180140229580.000030com.shutterstock
6551801362823200.000014com.gifyu
6561801332012620.000023com.storify
657180132566760.000041com.samsung
6581801294410950.000026edu.ucsd
659180119784220.000058edu.nyu
660180097366960.000040com.tandfonline
661180095824470.000055com.atlassian
662180092468960.000031com.geocities
663180088124390.000057edu.cmu
6641800874624330.000014com.yelloyello
665180086027800.000035com.netflix
6661800744012910.000022tv.ustream
667180071046200.000045us.icio
6681800681211380.000025edu.utexas
669180059244480.000055com.gitlab
6701800579020930.000015com.targetmarketingmag
6711800430621660.000015com.cargurus
672180042068860.000032com.docker
6731800293211910.000024com.trustedshops
6741800221824790.000013com.analyticsvidhya
6751800143424450.000013com.2findlocal
676179985208450.000033com.foxnews
6771799714620800.000015jp.huffingtonpost
6781799573626370.000013com.instructables
6791799523819450.000016com.nokia
6801799510011970.000024edu.academia
681179926647560.000036com.gettyimages
682179912302450.000099com.wpengine
6831799108422120.000014ca.uwaterloo
6841798868625470.000013com.cmgdigital
685179870908660.000033edu.umich
686179869746930.000040com.symantec
687179866348100.000034net.2mdn
6881798662621290.000015com.mondaq
689179861649520.000030com.ycombinator
6901798579422060.000014com.keepersecurity
691179850963880.000063com.newrelic
6921798474620540.000015com.doctoroz
693179845349080.000031com.uservoice
694179838622070.000115com.naver
6951798268415570.000019com.pastebin
696179804161890.000132com.xing
6971797873618570.000017com.duckduckgo
698179783129560.000030com.thinkwithgoogle
6991797812819070.000017se.haxx
7001797698420070.000016com.thecvf
7011797592622550.000014au.com.truelocal
7021797464035340.000012com.9to5mac
7031797453421300.000015uk.co.yelp
7041797416214440.000020fm.last
705179740868910.000032com.dropboxusercontent
7061797335415540.000019com.sankei
7071797292422050.000014com.tiddlywiki
7081797185823150.000014com.galvanize
7091797124021490.000015es.huffingtonpost
710179711082490.000098com.automattic
711179697289200.000031com.investopedia
7121796799422350.000014com.bizcommunity
7131796745811560.000025org.cambridge
7141796729612200.000023com.freeprivacypolicy
715179672869170.000031org.change
7161796654221450.000015com.winemag
7171796632424440.000014com.maritime-executive
7181796542410520.000027gov.uspto
7191796446425560.000013com.alternion
7201796335818340.000017com.autodesk
7211796312424110.000014com.communitywalk
7221796272618390.000017org.coursera
7231796220212550.000023com.upwork
7241796068223410.000014net.futurecdn
7251795997420890.000015com.kudzu
7261795985823520.000014com.ericsson
7271795832018320.000017com.adespresso
7281795692225270.000013edu.alamo
7291795678412600.000023com.irishtimes
7301795677823420.000014com.filedn
7311795666013530.000021edu.usc
7321795615810410.000027com.wunderground
733179557228640.000033br.com.uol
734179557186970.000039com.gartner
7351795538422540.000014com.gamespot
7361795525420740.000015com.btplc
7371795443220580.000015com.showmelocal
7381795425623860.000014com.massimodutti
7391795377420200.000016edu.virginia
7401795360017310.000018com.ikea
7411795347622600.000014com.insiderpages
7421795341612740.000023com.indiegogo
7431795267620300.000016com.goinswriter
7441794969424400.000014com.bershka
7451794935021840.000015com.almanac
746179491607700.000036gov.census
7471794688012330.000023com.intuit
748179459144130.000060com.inc
7491794453243470.000009com.programmableweb
7501794326811320.000025com.pcmag
7511794266621940.000014com.writersdigest
7521794213422830.000014com.citysquares
7531794164615350.000020com.fiverr
7541794131618720.000017com.csswizardry
7551794129613730.000021com.vanityfair
7561794117219030.000017jp.sankeibiz
7571794079624560.000013com.live5news
7581793923411170.000025gov.usgs
759179389169140.000031com.zoho
7601793828234000.000012com.freep
761179374088300.000034com.blackberry
7621793725621630.000015jp.booklog
7631793684423510.000014com.thedrinksbusiness
7641793542610570.000027com.politico
7651793538821970.000014com.winefolly
766179347686870.000040com.alibaba
7671793436629700.000013com.jeeran
7681793408024940.000013io.stackedit
7691793386219580.000016ca.ubc
770179337641610.000163me.line
7711793353839400.000010org.greenpeace
7721793306223810.000014com.yellowbook
7731793233624590.000013za.co.bdlive
7741793221223960.000014com.asianage
7751793209016310.000019com.udemy
7761793205834150.000012com.glamour
7771793183016350.000019com.chrome
7781793164211940.000024com.techrepublic
779179316148490.000033com.unity3d
7801793159020330.000015mp.j
781179315365980.000047gov.usda
7821793100423650.000014net.islamweb
783179293348080.000034int.wipo
7841792846623550.000014com.wsoctv
785179275926650.000042com.marketo
7861792704810490.000027edu.umn
787179269324140.000060mp.mailchi
788179263549680.000029com.aliexpress
7891792590026080.000013org.torproject
7901792536023220.000014com.utah
791179252289540.000030com.sciencedaily
7921792450219320.000016org.ap
793179240987240.000038gov.house
7941792397622080.000014com.chamberofcommerce
7951792359419530.000016com.urbandictionary
7961792355825700.000013com.spoke
7971792279428070.000013com.salespider
7981792117623690.000014com.ibmbigdatahub
799179211269810.000029au.net.abc
8001792107415650.000019com.problogger
801179210085330.000048com.snapchat
8021792092214250.000021fr.lemonde
803179193461410.000185jp.co.google
8041791759255390.000006cc.co
8051791702020170.000016com.posterous
8061791688211290.000025com.canva
8071791610416380.000019com.britannica
8081791578223790.000014com.wpxi
8091791546820400.000015edu.cuny
8101791545025190.000013com.americantowns
811179143646750.000041gov.hhs
8121791399622870.000014org.themoth
8131791332814200.000021com.rollingstone
8141791302212450.000023com.xkcd
8151791298435820.000011edu.brown
816179128806340.000044com.feedly
8171791245824470.000013com.hdnux
8181791239626090.000013com.zionsbank
8191791231625150.000013com.pacegallery
8201791143426450.000013com.tupalo
821179111366400.000044au.com.google
822179110608430.000033com.uk
823179090561280.000215com.youtube-nocookie
8241790886213220.000022com.vmware
8251790856820940.000015org.semanticscholar
8261790834219700.000016com.sanspo
8271790824810130.000028com.java
8281790817822390.000014it.scoop
829179078424700.000053com.adweek
8301790755023010.000014uk.co.dennis
8311790747421080.000015jp.co.sankei
8321790695025760.000013za.co.sowetanlive
833179067485040.000049gov.copyright
834179062503620.000068com.wufoo
8351790556223100.000014edu.uci
8361790490820520.000015jp.ne.iza
8371790481025430.000013org.foodrevolution
8381790445624910.000013com.thewritepractice
8391790445421470.000015com.parksassociates
8401790441011110.000025fr.blogspot
8411790403426360.000013au.com.whitepages
8421790387813290.000022com.billboard
8431790336012720.000023com.prezi
8441790221621720.000015com.local
845179014143200.000076gov.ftc
8461789997018310.000017edu.illinois
847178995869750.000029com.indeed
848178990668110.000034org.unesco
8491789898618520.000017com.hatenablog
8501789818424970.000013dk.brics
8511789800621180.000015uk.ac.ed
8521789761811730.000024org.unicef
853178972324250.000058com.criteo
8541789639821510.000015org.linuxfoundation
8551789606822150.000014com.vendio
8561789571819810.000016uk.ac.ucl
857178949962790.000089com.marriott
8581789419667530.000005com.blog
8591789376211870.000024com.steamcommunity
860178935828340.000034com.gofundme
8611789335420220.000016net.privacypolicytemplate
8621789303840670.000009com.virustotal
863178920184670.000053com.iconfinder
8641789161625410.000013com.lacartes
8651789127422690.000014ai.fast
8661789123217960.000017com.howstuffworks
8671788898412220.000023com.dell
8681788886624730.000013com.ibegin
8691788816614700.000020com.over-blog
870178874043500.000069net.themeforest
871178873763020.000080com.netdna-ssl
8721788727235760.000011edu.tufts
8731788657224870.000013za.co.moneyweb
8741788613616890.000018com.twilio
875178860428870.000032com.hootsuite
8761788457612300.000023com.gallup
8771788433223940.000014com.machinelearningmastery
8781788290623890.000014io.dropwizard
879178825129920.000028com.att
8801788141623080.000014com.ehow
8811788078836600.000011com.discogs
8821788072423330.000014com.blogs
8831788068821390.000015com.dandb
884178796964860.000051com.squareup
8851787953210370.000027gov.bls
886178794784010.000061com.bitly
8871787854036650.000011com.twitpic
8881787835830640.000013com.invoicesherpa
889178775786500.000043com.herokuapp
8901787710222650.000014ru.narod
8911787657018750.000017com.tunein
8921787525215700.000019com.com
8931787465019800.000016jp.co.zakzak
894178740526130.000046com.airbnb
8951787348821230.000015uk.co.realbusiness
896178722088370.000033gov.justice
8971787183819510.000016co.gcdn
898178716182670.000091com.myshopify
8991787084435040.000012de.bild
900178704782340.000104jp.co.amazon
9011786997619050.000017org.filezilla-project
9021786972225740.000013com.growtix
9031786971019220.000016com.newsfactor
9041786862627750.000013org.earthmagazine
9051786824035950.000011cc.tiny
906178681023390.000072org.opensource
9071786762817100.000018org.owasp
9081786743416780.000018org.cancer
909178652203700.000067org.doi
9101786475212150.000024ly.ow
9111786445828200.000013co.iglobal
9121786355613300.000022edu.uchicago
913178632621330.000206de.bund
914178628542590.000094com.getbootstrap
915178621864990.000050com.nasdaq
9161786182410000.000028com.lifehacker
9171786174812710.000023org.pnas
918178616443950.000062io.atom
9191786132414580.000020in.blogspot
9201786030626440.000013ai.becominghuman
9211786024826800.000013com.googlemaps
9221785853020090.000016net.nend
9231785686847300.000008com.colourlovers
9241785679814130.000021com.splashthat
925178566769820.000029com.jetbrains
926178561769150.000031jp.livedoor
927178561523030.000080com.ssl-images-amazon
9281785422426430.000013nl.zeelandnet
929178536208690.000032com.pingdom
9301785353430780.000013com.sophos
9311785290825250.000013gr.huffingtonpost
9321785214210020.000028de.blogspot
9331785068227600.000013com.fox13memphis
9341785048821140.000015com.richmediagallery
9351785037818210.000017com.hotmail
93617850366720.000395com.messenger
9371785027622310.000014edu.asu
938178500529950.000028org.iso
9391784997613890.000021com.imimg
9401784937211450.000025com.uber
9411784912023560.000014com.tuck
9421784855617260.000018com.nba
9431784823224040.000014jp.news24
9441784773215590.000019com.ogilvy
9451784731827720.000013com.addustour
9461784680028310.000013org.grayarea
9471784671421030.000015com.homestars
9481784665011360.000025com.seattletimes
949178465802650.000092ru.rambler
9501784598823620.000014edu.utah
9511784586838620.000010com.starwars
952178456404790.000051jp.ne.sakura
9531784471810630.000027gov.congress
9541784310214100.000021dk.datatilsynet
955178429328590.000033com.stitcher
9561784279829710.000013com.oilandgas360
9571784248617850.000017edu.umd
958178424307580.000036com.yandex
9591784010018850.000017com.wetransfer
9601783962824570.000013ms.1drv
961178382129770.000029com.prweb
962178380864230.000058com.smugmug
9631783770224140.000014com.delta
9641783635623060.000014edu.bu
9651783615611410.000025com.500px
9661783466827960.000013org.cmlibrary
9671783424825650.000013com.fixr
9681783376413120.000022com.firefox
9691783336820500.000015edu.ufl
9701783161024090.000014ca.ualberta
9711783138639770.000010com.thingiverse
972178308884000.000061com.discordapp
9731783071438170.000010edu.unl
9741782974827460.000013tw.com.ibon
9751782913424900.000013au.com.hotfrog
9761782896623500.000014de.mpg
9771782892811600.000025com.timeanddate
9781782858024950.000013com.figure-eight
9791782857423700.000014com.codecademy
980178279648900.000032gov.usa
981178275182560.000096it.google
9821782706424270.000014com.outboundengine
9831782687413080.000022com.strikingly
9841782684012430.000023com.target
9851782575824550.000013com.theblogpress
9861782530025850.000013com.expressbusinessdirectory
9871782528822160.000014com.nfl
9881782519226070.000013com.elocal
9891782512026280.000013au.com.news
9901782431411160.000025com.scientificamerican
9911782416813250.000022co.vine
992178237107470.000037com.cargocollective
993178235306910.000040com.caniuse
9941782193021070.000015com.angelfire
9951782078830050.000013com.hbo
9961782064816390.000019uk.co.screamingfrog
9971782030426170.000013com.ovoenergy
998178200107370.000037uk.co.eventbrite
9991781972626910.000013com.normacomics
1000178195767520.000037com.sagepub

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

July 2019 crawl archive now available

The crawl archive for July 2019 is now available! It contains 2.6 billion web pages or 220 TiB of uncompressed content, crawled between July 15th and 24th.

The July crawl contains page captures of 810 million URLs not contained in any crawl archive before. New URLs are sampled based on the host and domain ranks (harmonic centrality) published as part of the Feb/Mar/Apr 2019 webgraph data set from the following sources:

  • a random sample of 2.0 billion outlinks taken from June crawl WAT files
  • 1.8 billion URLs mined in a breadth-first side crawl within a maximum of 6 links (“hops”), started from
    • the homepages of the top 60 million hosts and domains and randomly selected samples of
    • 2 million human-readable sitemap pages (HTML format)
    • 2 million URLs of pages written in 130 less-represented languages (cf. language distributions)
  • 900 million URLs extracted and sampled from 20 million sitemaps, RSS and Atom feeds

Archive Location and Download

The July crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2019-30/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://commoncrawl.s3.amazonaws.com/ to each line, you end up with the S3 and HTTP paths respectively.

 File List#FilesTotal Size
Compressed (TiB)
SegmentsCC-MAIN-2019-30/segment.paths.gz100
WARC filesCC-MAIN-2019-30/warc.paths.gz5600046.10
WAT filesCC-MAIN-2019-30/wat.paths.gz5600017.62
WET filesCC-MAIN-2019-30/wet.paths.gz560007.80
Robots.txt filesCC-MAIN-2019-30/robotstxt.paths.gz560000.14
Non-200 responses filesCC-MAIN-2019-30/non200responses.paths.gz560001.63
URL index filesCC-MAIN-2019-30/cc-index.paths.gz3020.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2019-30/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.