Blog – Common Crawl

Host- and Domain-Level Web Graphs September/October, November/December 2022 and January/February 2023

March 15, 2023Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the September/October, November/December 2022 and January/February 2023 crawls. For more information about the data formats and the processing pipeline, please see the announcements of previous webgraph releases. You may also visit the cc-webgraph and cc-pyspark projects which contain all the scripts and tools needed to construct the graphs. Instructions for exploring the graphs in the webgraph format can be found in our collection of webgraph notebooks.

Host-level graph

The graph has of 325 million nodes and 2.63 billion edges. Both hyperlinks, HTTP redirects and link headers are used as edges to span up the graph. All types of links are included, including purely “technical” links pointing to images, JavaScript libraries, web fonts, etc. However, only hostnames with a valid IANA TLD are used. As a result, URLs with an IP address as host component are not taken into account for building the host-level graph.

There are 268 million dangling nodes (82.7%) and the largest strongly connected component contains 43.1 million (13.3%) nodes. Dangling nodes come from

hosts that are not crawled, but are referenced by a link on a crawled page
hosts with no links pointing to another hostname
or hosts that only returned an error page (e.g. HTTP 404).

Hostnames in the graph are in reverse domain name notation with the leading www. removed: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 325 million hosts from AWS S3 at s3://commoncrawl/projects/hyperlinkgraph/cc-main-2022-23-sep-nov-jan/host/ (this requires an account on AWS). Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2022-23-sep-nov-jan/host/ as prefix to access the files from everywhere.

Note that the text representation of the host-level graph is delivered in 10 gzip-compressed files listed in two path listings – one for the nodes (vertices), and one for the edges (arcs). First, download the path listing and decompress it with “gzip -d” or “gunzip”. Adding the prefix s3://commoncrawl/ or https://data.commoncrawl.org/ to each line in the path listing will give you the list of URLs to download the entire graph.

Download files of the Common Crawl Sep/Nov/Jan 2022-2023 host-level webgraph

Size	File	Description
2.34 GB	cc-main-2022-23-sep-nov-jan-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 32 vertices files
11.40 GB	cc-main-2022-23-sep-nov-jan-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 64 edges files

5.51 GB	cc-main-2022-23-sep-nov-jan-host.graph	graph in BVGraph format
2 kB	cc-main-2022-23-sep-nov-jan-host.properties

5.88 GB	cc-main-2022-23-sep-nov-jan-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2022-23-sep-nov-jan-host-t.properties

1 kB	cc-main-2022-23-sep-nov-jan-host.stats	WebGraph statistics

5.56 GB	cc-main-2022-23-sep-nov-jan-host-ranks.txt.gz	harmonic centrality and pagerank

Domain-level graph

The domain graph is built by aggregating the host graph at the pay-level domain (PLD) level based on the public suffix list maintained on publicsuffix.org. Version (commit) 0bbf864 of the public suffix list was used (commit date 2023-03-08).

The domain-level graph has 88 million nodes and 1.68 billion edges. 52% or 46 million nodes are dangling nodes, the largest strongly connected component covers 34 million or 39% of the nodes.

All domain graph files are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2022-23-sep-nov-jan/domain/ or on https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2022-23-sep-nov-jan/domain/.

Download files of the Common Crawl Sep/Nov/Jan 2022-2023 domain-level webgraph

Size	File	Description
0.61 GB	cc-main-2022-23-sep-nov-jan-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
6.89 GB	cc-main-2022-23-sep-nov-jan-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

3.90 GB	cc-main-2022-23-sep-nov-jan-domain.graph	graph in BVGraph format
2 kB	cc-main-2022-23-sep-nov-jan-domain.properties

3.81 GB	cc-main-2022-23-sep-nov-jan-domain-t.graph	transpose of the graph
2 kB	cc-main-2022-23-sep-nov-jan-domain-t.properties

1 kB	cc-main-2022-23-sep-nov-jan-domain.stats	WebGraph statistics

1.90 GB	cc-main-2022-23-sep-nov-jan-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 88 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Sep/Nov/Jan 2022-2023)

harmonic centrality rank	hc value	page rank	page rank value	reversed domain name
1	30208264	1	0.016763	com.googleapis
2	29783728	3	0.010997	com.facebook
3	29384286	2	0.015692	com.google
4	26032564	6	0.005934	com.youtube
5	25805292	5	0.006482	com.twitter
6	25487994	8	0.005484	com.instagram
7	25225590	7	0.005863	org.w
8	25002416	4	0.007149	com.googletagmanager
9	23952870	9	0.004622	org.gmpg
10	23337268	12	0.003349	com.linkedin
11	23278606	10	0.004162	com.gstatic
12	22405388	15	0.002066	com.gravatar
13	22178872	11	0.003793	com.cloudflare
14	21844608	13	0.002400	org.wordpress
15	21681482	25	0.001450	com.pinterest
16	21559576	32	0.001217	org.wikipedia
17	21429174	17	0.001813	com.apple
18	21227448	31	0.001226	com.wordpress
19	21216668	34	0.001088	com.vimeo
20	21195402	39	0.000900	be.youtu
21	20932166	14	0.002311	net.cloudfront
22	20898430	18	0.001711	com.bootstrapcdn
23	20823002	35	0.001080	com.microsoft
24	20748850	48	0.000709	com.amazon
25	20741894	55	0.000581	com.blogspot
26	20704246	26	0.001442	com.jquery
27	20677290	47	0.000780	gl.goo
28	20670994	43	0.000852	com.amazonaws
29	20665338	16	0.001840	io.polyfill
30	20631324	44	0.000844	eu.europa
31	20618144	50	0.000696	com.wp
32	20564226	29	0.001298	net.jsdelivr
33	20528750	45	0.000812	org.mozilla
34	20516286	28	0.001369	com.wixstatic
35	20513258	65	0.000505	ly.bit
36	20484072	23	0.001470	com.adobe
37	20443722	21	0.001544	com.fontawesome
38	20411612	40	0.000896	com.google-analytics
39	20400328	22	0.001485	com.github
40	20349558	54	0.000635	com.paypal
41	20346992	19	0.001674	com.googleusercontent
42	20328172	30	0.001297	com.whatsapp
43	20292300	103	0.000334	com.tumblr
44	20192382	33	0.001113	ru.yandex
45	20186350	85	0.000378	com.medium
46	20183944	123	0.000299	com.reddit
47	20180914	108	0.000331	com.yahoo
48	20178722	62	0.000529	com.shopify
49	20170012	58	0.000564	com.flickr
50	20167766	69	0.000483	io.github
51	20163916	63	0.000522	co.t
52	20152240	128	0.000260	com.nytimes
53	20124068	125	0.000293	com.spotify
54	20104136	36	0.001066	com.baidu
55	20098412	59	0.000553	org.w3
56	20094912	37	0.001000	com.qq
57	20016100	57	0.000569	com.vk
58	20005808	41	0.000881	com.googlesyndication
59	20003706	114	0.000313	com.weebly
60	19975298	158	0.000186	com.forbes
61	19967874	122	0.000302	org.creativecommons
62	19967726	130	0.000251	com.soundcloud
63	19962190	142	0.000211	org.archive
64	19942482	150	0.000198	gov.nih
65	19918748	126	0.000264	com.tiktok
66	19915782	61	0.000538	org.schema
67	19912766	60	0.000544	com.unpkg
68	19911234	181	0.000153	com.bing
69	19911002	249	0.000107	com.imdb
70	19905552	264	0.000100	edu.harvard
71	19902624	80	0.000387	me.t
72	19901168	196	0.000137	org.wikimedia
73	19873990	163	0.000174	com.dropbox
74	19872264	288	0.000091	net.slideshare
75	19869992	192	0.000142	int.who
76	19862262	203	0.000131	com.cnn
77	19862142	174	0.000157	gov.cdc
78	19858886	214	0.000123	com.theguardian
79	19847642	184	0.000151	com.unsplash
80	19845106	121	0.000307	com.list-manage
81	19831690	272	0.000097	net.researchgate
82	19819188	253	0.000105	com.wsj
83	19816030	293	0.000090	com.bbc
84	19812290	64	0.000507	com.macromedia
85	19810452	297	0.000089	uk.co.bbc
86	19805166	304	0.000087	com.reuters
87	19788972	246	0.000107	com.washingtonpost
88	19786314	279	0.000094	com.statista
89	19775652	318	0.000083	edu.stanford
90	19775346	432	0.000060	gov.nasa
91	19774590	67	0.000497	com.addthis
92	19770972	46	0.000807	net.doubleclick
93	19769890	151	0.000195	com.wixsite
94	19765714	227	0.000117	com.businessinsider
95	19763270	215	0.000123	com.imgur
96	19754014	367	0.000072	com.go
97	19750586	204	0.000129	com.live
98	19746004	107	0.000332	com.wix
99	19745530	372	0.000070	com.wired
100	19738366	152	0.000195	us.zoom
101	19723422	135	0.000225	gle.forms
102	19718148	225	0.000118	com.etsy
103	19717824	277	0.000095	com.ibm
104	19700092	447	0.000058	com.theverge
105	19699788	356	0.000074	com.nature
106	19696656	120	0.000309	me.wp
107	19693524	176	0.000156	org.ietf
108	19687268	49	0.000700	com.fb
109	19686956	157	0.000187	com.ytimg
110	19685800	492	0.000053	com.msn
111	19685436	315	0.000084	com.android
112	19684204	316	0.000084	com.cnbc
113	19684024	140	0.000214	com.mailchimp
114	19682468	42	0.000858	net.fbcdn
115	19682198	255	0.000104	com.stackoverflow
116	19681202	511	0.000051	edu.berkeley
117	19679968	275	0.000096	org.un
118	19679582	265	0.000100	com.bloomberg
119	19677792	220	0.000121	com.outlook
120	19673502	178	0.000155	org.apache
121	19666706	269	0.000098	com.oracle
122	19658542	382	0.000069	com.example
123	19650660	343	0.000077	org.npr
124	19649560	350	0.000075	com.quora
125	19649030	129	0.000254	com.youtube-nocookie
126	19646800	601	0.000045	com.zdnet
127	19646398	99	0.000336	com.giphy
128	19644128	195	0.000140	com.hubspot
129	19643752	219	0.000122	org.doi
130	19632490	542	0.000048	com.myspace
131	19630886	222	0.000120	gov.ca
132	19618726	395	0.000067	com.time
133	19612954	359	0.000074	com.slack
134	19610836	70	0.000455	com.ft
135	19610350	462	0.000056	com.appspot
136	19606234	139	0.000217	com.opera
137	19605258	261	0.000102	com.sciencedirect
138	19604780	476	0.000054	com.ted
139	19604180	342	0.000077	com.springer
140	19602362	401	0.000065	org.arxiv
141	19598276	393	0.000067	com.usatoday
142	19594554	194	0.000140	com.issuu
143	19588288	332	0.000080	org.acm
144	19585042	205	0.000129	com.npmjs
145	19581664	331	0.000080	uk.co.amazon
146	19577244	307	0.000086	com.githubusercontent
147	19576784	242	0.000108	com.blogger
148	19576054	311	0.000086	com.wiley
149	19574732	355	0.000074	com.pexels
150	19570782	468	0.000055	edu.cornell
151	19569412	482	0.000054	com.theatlantic
152	19567006	440	0.000059	org.python
153	19564494	593	0.000045	org.worldbank
154	19563390	565	0.000047	uk.co.telegraph
155	19563008	732	0.000037	edu.psu
156	19553204	428	0.000060	com.cnet
157	19539618	149	0.000199	org.ampproject
158	19537568	623	0.000043	org.weforum
159	19537332	296	0.000089	uk.gov
160	19533658	444	0.000059	com.huffingtonpost
161	19532878	490	0.000053	com.latimes
162	19528578	645	0.000042	org.unesco
163	19527222	655	0.000041	com.livejournal
164	19526534	449	0.000058	com.pixabay
165	19508444	411	0.000063	com.sagepub
166	19506694	480	0.000054	com.goodreads
167	19506188	499	0.000052	uk.co.google
168	19503402	258	0.000103	net.behance
169	19503138	270	0.000097	com.bandcamp
170	19503070	727	0.000037	org.chromium
171	19495988	525	0.000050	com.cbsnews
172	19495180	326	0.000081	ee.linktr
173	19493452	527	0.000049	edu.yale
174	19489730	286	0.000092	com.w3schools
175	19483130	175	0.000157	com.yelp
176	19482412	299	0.000089	edu.mit
177	19481882	68	0.000496	com.googleadservices
178	19480090	78	0.000393	me.wa
179	19477430	528	0.000049	uk.co.independent
180	19474080	134	0.000228	com.statcounter
181	19472938	300	0.000089	com.tinyurl
182	19472136	509	0.000051	com.fortune
183	19471318	714	0.000038	edu.columbia
184	19471218	690	0.000039	com.vox
185	19461296	422	0.000061	gov.whitehouse
186	19460546	416	0.000062	org.nodejs
187	19457810	512	0.000051	uk.co.dailymail
188	19455462	537	0.000049	com.indiatimes
189	19452626	349	0.000075	com.businesswire
190	19451618	290	0.000090	org.pewresearch
191	19450276	559	0.000047	edu.cmu
192	19442676	643	0.000042	com.marketwatch
193	19436888	494	0.000052	com.tandfonline
194	19431628	598	0.000045	org.pbs
195	19429220	657	0.000041	com.usnews
196	19428984	777	0.000035	edu.upenn
197	19428100	278	0.000095	com.twimg
198	19421114	683	0.000039	com.buzzfeed
199	19420594	590	0.000045	gov.loc
200	19420022	504	0.000052	com.fc2
201	19419582	674	0.000040	com.git-scm
202	19418554	901	0.000033	com.qz
203	19416166	885	0.000034	edu.washington
204	19415650	778	0.000035	com.trello
205	19415444	675	0.000040	com.apnews
206	19411912	1121	0.000026	com.techradar
207	19411736	517	0.000050	com.investopedia
208	19410610	652	0.000041	com.mysql
209	19408094	138	0.000218	info.aboutads
210	19407400	880	0.000034	me.about
211	19403682	276	0.000096	org.gnu
212	19403122	736	0.000037	com.economist
213	19402862	677	0.000040	com.box
214	19397500	713	0.000038	com.scribd
215	19396454	1276	0.000023	com.techrepublic
216	19395758	370	0.000071	com.gitlab
217	19395036	489	0.000053	com.walmart
218	19394568	348	0.000075	com.techcrunch
219	19388888	406	0.000064	co.ibb
220	19388054	487	0.000053	com.nationalgeographic
221	19387258	734	0.000037	com.venturebeat
222	19386248	541	0.000048	com.inc
223	19385564	165	0.000171	com.staticflickr
224	19385242	141	0.000211	me.line
225	19382874	534	0.000049	com.theconversation
226	19380928	510	0.000051	com.nbcnews
227	19380802	582	0.000046	com.digg
228	19379752	1117	0.000026	edu.northwestern
229	19378952	787	0.000035	org.semver
230	19377466	1002	0.000029	edu.jhu
231	19377050	737	0.000036	ca.cbc
232	19376942	539	0.000049	com.googleblog
233	19375096	1397	0.000021	edu.rutgers
234	19371812	647	0.000042	com.photobucket
235	19370806	1099	0.000027	edu.usc
236	19369504	600	0.000045	gov.senate
237	19367834	198	0.000134	com.calendly
238	19367222	231	0.000116	net.windows
239	19365020	1262	0.000023	org.kernel
240	19359670	1287	0.000023	co.elastic
241	19358788	808	0.000035	com.shutterstock
242	19358402	691	0.000039	org.cambridge
243	19356378	1091	0.000027	fm.last
244	19355798	321	0.000082	tv.twitch
245	19355788	187	0.000147	page.g
246	19355748	719	0.000037	com.newyorker
247	19354116	748	0.000036	org.bitbucket
248	19345608	566	0.000047	com.oup
249	19344142	930	0.000031	org.sciencemag
250	19343430	250	0.000107	com.jotform
251	19340588	191	0.000143	com.cloudinary
252	19340216	538	0.000049	org.unicef
253	19339906	1020	0.000029	edu.princeton
254	19339090	588	0.000045	io.codepen
255	19338824	982	0.000030	gov.usgs
256	19338794	879	0.000034	uk.ac.ox
257	19336670	562	0.000047	com.xinhuanet
258	19336118	1184	0.000025	com.alexa
259	19333656	515	0.000051	org.js
260	19331480	864	0.000034	edu.asu
261	19329334	884	0.000034	com.nvidia
262	19328550	1187	0.000025	com.mediafire
263	19325282	221	0.000121	net.sourceforge
264	19322430	1433	0.000021	com.euronews
265	19321422	335	0.000079	com.prnewswire
266	19320932	937	0.000031	com.foxnews
267	19320900	1083	0.000027	gov.fbi
268	19314904	1174	0.000025	org.coursera
269	19313470	627	0.000043	com.biomedcentral
270	19312726	1367	0.000022	com.500px
271	19310966	88	0.000374	com.stripe
272	19308392	237	0.000112	com.tripadvisor
273	19306794	186	0.000148	com.xing
274	19306438	248	0.000107	com.wpengine
275	19306070	154	0.000191	com.sharethis
276	19304476	929	0.000031	com.nypost
277	19301278	740	0.000036	com.politico
278	19297772	325	0.000081	com.automattic
279	19297516	1450	0.000020	com.digitaltrends
280	19297208	1072	0.000027	co.g
281	19297054	1058	0.000028	org.pnas
282	19295368	1142	0.000026	com.axios
283	19295228	410	0.000063	org.openstreetmap
284	19293638	1191	0.000025	uk.co.guardian
285	19293548	1111	0.000026	com.scmp
286	19290228	595	0.000045	ca.canada
287	19289660	1431	0.000021	org.eclipse
288	19288644	1261	0.000023	uk.co.blogspot
289	19288300	678	0.000040	com.huffpost
290	19286476	1322	0.000022	org.semanticscholar
291	19284068	637	0.000042	gov.census
292	19283194	469	0.000055	gov.usda
293	19282746	66	0.000504	com.trustpilot
294	19281282	765	0.000036	com.hp
295	19281216	556	0.000048	io.readthedocs
296	19276912	1240	0.000023	com.nymag
297	19276304	256	0.000104	org.iana
298	19276166	870	0.000034	com.ssrn
299	19276092	962	0.000030	edu.umn
300	19274986	993	0.000029	au.net.abc
301	19273958	1448	0.000020	com.sky
302	19273816	1211	0.000024	edu.purdue
303	19273324	625	0.000043	org.apa
304	19272362	217	0.000122	com.eventbrite
305	19271512	167	0.000167	gov.privacyshield
306	19271404	1064	0.000028	com.about
307	19271058	340	0.000078	com.dribbble
308	19270226	583	0.000046	gov.house
309	19269684	907	0.000033	com.sciencedaily
310	19269194	571	0.000047	gov.noaa
311	19269034	445	0.000059	com.arcgis
312	19267554	363	0.000073	com.feedburner
313	19266714	591	0.000045	fm.anchor
314	19266698	1533	0.000020	ru.spb
315	19264280	567	0.000047	site.business
316	19264260	1308	0.000022	com.nikkei
317	19261166	1030	0.000029	com.ggpht
318	19259980	868	0.000034	org.change
319	19257436	1053	0.000028	com.evernote
320	19256818	1360	0.000022	edu.illinois
321	19255628	162	0.000174	com.office
322	19255582	1343	0.000022	org.postgresql
323	19252970	882	0.000034	org.pypi
324	19252962	581	0.000046	com.163
325	19251350	574	0.000047	com.dailymotion
326	19248956	1617	0.000019	org.aclu
327	19247890	1051	0.000028	edu.uchicago
328	19245190	953	0.000030	com.mdpi
329	19245010	889	0.000034	de.spiegel
330	19243662	496	0.000052	gov.hhs
331	19242990	400	0.000065	com.indeed
332	19241006	696	0.000039	gov.justice
333	19239998	621	0.000043	gov.state
334	19237584	1665	0.000019	org.greenpeace
335	19236460	106	0.000333	net.jsfiddle
336	19233688	789	0.000035	gov.congress
337	19233144	689	0.000039	com.bigcartel
338	19231252	2010	0.000017	edu.gatech
339	19230230	484	0.000054	gov.epa
340	19229306	1923	0.000017	com.openai
341	19228750	619	0.000043	org.ohchr
342	19227578	893	0.000033	org.fao
343	19227162	390	0.000067	com.atlassian
344	19227150	1812	0.000018	org.science
345	19224842	768	0.000035	com.jetbrains
346	19223786	1223	0.000024	com.foursquare
347	19223692	454	0.000057	com.squareup
348	19220130	124	0.000297	com.alicdn
349	19219200	1996	0.000017	org.phys
350	19218212	1090	0.000027	cn.com.chinadaily
351	19217284	339	0.000078	com.ebay
352	19216816	347	0.000075	com.surveymonkey
353	19216726	883	0.000034	com.chrome
354	19216716	1289	0.000023	uk.co.thetimes
355	19216348	260	0.000102	com.webflow
356	19216090	2023	0.000017	com.foxbusiness
357	19213972	913	0.000032	app.netlify
358	19213822	385	0.000068	com.disqus
359	19212532	1229	0.000024	com.hollywoodreporter
360	19211550	766	0.000036	gov.archives
361	19210746	423	0.000061	com.getpocket
362	19210526	396	0.000066	com.samsung
363	19207318	371	0.000071	com.proofpoint
364	19206792	1119	0.000026	edu.utexas
365	19206132	132	0.000248	com.zendesk
366	19204834	397	0.000066	com.substack
367	19202152	702	0.000038	com.mashable
368	19201060	1081	0.000027	org.jstor
369	19200068	649	0.000042	net.azurewebsites
370	19198950	200	0.000133	org.allaboutcookies
371	19198558	421	0.000062	com.freepik
372	19197812	529	0.000049	com.netdna-ssl
373	19196780	547	0.000048	com.snapchat
374	19194372	679	0.000040	com.gumroad
375	19194238	144	0.000206	com.paypalobjects
376	19193934	76	0.000397	me.fb
377	19192814	526	0.000050	ch.admin
378	19191374	863	0.000034	com.pinimg
379	19191250	743	0.000036	com.britannica
380	19191082	1489	0.000020	au.com.smh
381	19190906	704	0.000038	com.vice
382	19189032	554	0.000048	gov.copyright
383	19183662	1157	0.000025	com.dw
384	19183626	345	0.000076	net.themeforest
385	19182584	452	0.000058	com.patreon
386	19182178	1235	0.000024	uk.co.mirror
387	19176944	1363	0.000022	de.sueddeutsche
388	19173466	1014	0.000029	uk.ac.cam
389	19172198	333	0.000080	fr.cnil
390	19169612	3327	0.000010	google.research
391	19167346	784	0.000035	cc.postimg
392	19165170	609	0.000044	gov.nist
393	19163734	1493	0.000020	ca.sfu
394	19162676	507	0.000051	com.gmail
395	19162016	2342	0.000015	com.martinfowler
396	19160966	1297	0.000023	org.imf
397	19160094	983	0.000030	edu.si
398	19158666	709	0.000038	org.oecd
399	19157194	402	0.000064	ru.gov
400	19156178	1059	0.000028	com.chicagotribune
401	19154640	1257	0.000023	com.crunchbase
402	19154290	404	0.000064	com.optimizely
403	19153126	75	0.000404	net.akamaihd
404	19152672	1005	0.000029	com.intuit
405	19151200	156	0.000188	org.networkadvertising
406	19150938	2139	0.000016	app.web
407	19150100	947	0.000031	com.history
408	19147258	2498	0.000014	com.ibtimes
409	19146882	161	0.000174	com.rawgit
410	19146734	284	0.000093	net.azureedge
411	19146432	478	0.000054	nl.google
412	19144122	475	0.000055	com.meetup
413	19143658	2728	0.000013	com.cbs
414	19143614	1549	0.000019	org.unhcr
415	19143010	119	0.000312	de.google
416	19140470	977	0.000030	com.sap
417	19140308	577	0.000047	com.kickstarter
418	19139232	431	0.000060	com.media-amazon
419	19138468	1340	0.000022	com.aljazeera
420	19138260	346	0.000076	net.php
421	19136610	1894	0.000018	com.straitstimes
422	19135416	52	0.000645	com.godaddy
423	19134718	1103	0.000027	com.insider
424	19134162	1017	0.000029	gov.treasury
425	19133204	1601	0.000019	us.imageshack
426	19132232	1269	0.000023	org.sphinx-doc
427	19131826	1073	0.000027	link.page
428	19131230	508	0.000051	cn.com.people
429	19130856	1457	0.000020	de.mpg
430	19129168	514	0.000051	org.debian
431	19124870	2597	0.000014	au.com.news
432	19123686	267	0.000098	jp.co.yahoo
433	19122658	336	0.000078	com.typepad
434	19121732	73	0.000429	com.wsimg
435	19121444	941	0.000031	com.podbean
436	19121320	1066	0.000028	uk.gov.service
437	19121300	235	0.000113	gg.discord
438	19120430	1490	0.000020	com.over-blog
439	19119364	306	0.000086	com.eepurl
440	19119274	636	0.000042	gov.usa
441	19118634	633	0.000043	com.stumbleupon
442	19116958	354	0.000074	org.hbr
443	19114264	1789	0.000019	ms.1drv
444	19114220	673	0.000040	google.blog
445	19112010	1796	0.000019	com.buzzfeednews
446	19110954	1093	0.000027	org.ilo
447	19110296	1962	0.000017	com.mystrikingly
448	19108200	90	0.000367	net.facebook
449	19107938	1446	0.000021	de.zeit
450	19107838	471	0.000055	com.tripod
451	19106728	1065	0.000028	int.coe
452	19106358	1389	0.000021	com.teachable
453	19106154	1136	0.000026	com.thehill
454	19106002	56	0.000570	net.typekit
455	19105858	2051	0.000016	uk.co.standard
456	19103522	2071	0.000016	com.newscientist
457	19102398	2527	0.000014	com.channel4
458	19102058	2631	0.000013	com.storify
459	19098660	1421	0.000021	edu.duke
460	19094500	596	0.000045	com.healthline
461	19093638	1302	0.000023	au.gov.nsw
462	19091574	2525	0.000014	org.maven
463	19090438	622	0.000043	org.worldwildlife
464	19086786	1022	0.000029	com.brightcove
465	19086756	1541	0.000020	int.unfccc
466	19086616	896	0.000033	com.withgoogle
467	19085682	292	0.000090	com.squarespace
468	19085616	2578	0.000014	com.instructables
469	19084862	2082	0.000016	com.rt
470	19084252	2430	0.000015	org.tensorflow
471	19083306	263	0.000101	me.telegram
472	19081082	638	0.000042	com.cisco
473	19080766	809	0.000035	watch.fb
474	19079900	465	0.000056	com.steampowered
475	19079012	760	0.000036	com.deviantart
476	19074242	603	0.000044	com.googlecode
477	19072502	1813	0.000018	uk.parliament
478	19070624	572	0.000047	com.airbnb
479	19068360	543	0.000048	com.matterport
480	19067152	2020	0.000017	org.edx
481	19063090	2909	0.000012	com.dreamstime
482	19063038	1875	0.000018	com.googlesource
483	19062408	1144	0.000026	com.dell
484	19061850	2934	0.000012	me.ogp
485	19059726	1131	0.000026	org.hrw
486	19059152	2058	0.000016	edu.cuny
487	19056892	436	0.000059	com.elsevier
488	19056600	961	0.000030	gov.dhs
489	19055372	1086	0.000027	com.bostonglobe
490	19053316	283	0.000093	com.salesforce
491	19051650	2876	0.000012	org.icrc
492	19050700	1479	0.000020	gov.defense
493	19049286	169	0.000163	com.discord
494	19048846	2672	0.000013	cc.arduino
495	19048696	172	0.000160	com.addtoany
496	19048338	2122	0.000016	com.padlet
497	19047634	2187	0.000016	uk.co.thesun
498	19047204	1933	0.000017	edu.georgetown
499	19046984	563	0.000047	com.deloitte
500	19044280	2026	0.000017	ca.blogspot
501	19044124	2180	0.000016	edu.ucsb
502	19040766	2965	0.000012	org.wikibooks
503	19040244	2116	0.000016	edu.tufts
504	19039882	251	0.000107	org.bbb
505	19037044	3519	0.000010	com.deepmind
506	19036140	362	0.000073	net.secureservercdn
507	19035692	3370	0.000010	google.ai
508	19032558	1436	0.000021	eu.politico
509	19032004	1965	0.000017	edu.wustl
510	19031454	1040	0.000028	com.istockphoto
511	19029618	761	0.000036	com.thinkwithgoogle
512	19029288	3052	0.000011	com.diigo
513	19029204	2945	0.000012	com.snap
514	19027680	1181	0.000025	us.icio
515	19027416	1447	0.000020	ch.ipcc
516	19026814	148	0.000199	com.jimdo
517	19024796	2119	0.000016	com.france24
518	19024010	3548	0.000010	com.ulule
519	19021632	1138	0.000026	com.arstechnica
520	19020080	2471	0.000014	com.instructure
521	19019232	960	0.000030	edu.brookings
522	19019042	2409	0.000015	edu.caltech
523	19018294	252	0.000105	com.aliyuncs
524	19018192	24	0.001461	cn.gov.miit
525	19018120	667	0.000041	ca.amazon
526	19016070	1403	0.000021	org.rfc-editor
527	19013594	1151	0.000026	com.verizon
528	19009160	273	0.000096	me.m
529	19007720	289	0.000091	ru.ok
530	19006594	1212	0.000024	uk.nhs
531	19002870	708	0.000038	com.intel
532	19001948	1983	0.000017	gov.lbl
533	19001690	2411	0.000015	ru.kremlin
534	18999232	2443	0.000015	edu.oregonstate
535	18998558	426	0.000060	com.fastcompany
536	18998428	505	0.000052	com.ssl-images-amazon
537	18997736	2873	0.000012	fr.archives-ouvertes
538	18995582	2089	0.000016	org.archlinux
539	18995234	576	0.000047	com.wufoo
540	18994966	1422	0.000021	com.people
541	18994790	2484	0.000014	gov.cia
542	18994448	2564	0.000014	tl.we
543	18994368	2249	0.000015	org.unwomen
544	18993384	2942	0.000012	com.kaggle
545	18991740	2808	0.000012	com.aboutamazon
546	18991620	4094	0.000008	com.sho
547	18991158	313	0.000085	gov.ftc
548	18991106	659	0.000041	com.docker
549	18990118	728	0.000037	com.zoominfo
550	18987498	3417	0.000010	com.pearltrees
551	18984800	2096	0.000016	io.gitlab
552	18984398	3046	0.000011	org.scala-lang
553	18983188	317	0.000083	com.typeform
554	18981496	1956	0.000017	com.asahi
555	18979704	368	0.000071	net.imgix
556	18978984	168	0.000164	com.youronlinechoices
557	18977728	738	0.000036	com.symantec
558	18977396	2904	0.000012	jp.co.japantimes
559	18975970	1361	0.000022	com.buymeacoffee
560	18975394	1589	0.000019	com.justia
561	18974980	2786	0.000012	uk.co.huffingtonpost
562	18974464	552	0.000048	com.gartner
563	18974226	2928	0.000012	jp.ac.u-tokyo
564	18973684	398	0.000065	com.force
565	18971782	2893	0.000012	no.nrk
566	18969492	3113	0.000011	cc.taplink
567	18967596	1330	0.000022	org.amnesty
568	18967562	2090	0.000016	com.thestar
569	18966190	2353	0.000015	tv.ustream
570	18965606	3480	0.000010	tv.blip
571	18964426	2212	0.000016	com.peatix
572	18961114	755	0.000036	com.redhat
573	18960518	1877	0.000018	com.firebaseapp
574	18960296	2075	0.000016	com.flipboard
575	18959466	989	0.000029	com.stackexchange
576	18959354	446	0.000058	com.herokuapp
577	18958890	495	0.000052	com.campaign-archive
578	18958728	1986	0.000017	org.nber
579	18958182	1172	0.000025	com.ecwid
580	18957588	2719	0.000013	hk.com.google
581	18957146	2787	0.000012	blog.home
582	18957046	2917	0.000012	com.rakuten
583	18956066	2639	0.000013	org.biorxiv
584	18955174	1123	0.000026	gov.wa
585	18953020	558	0.000048	com.netflix
586	18952656	2681	0.000013	com.gamespot
587	18950358	555	0.000048	com.canva
588	18947616	3041	0.000011	org.rsf
589	18947462	435	0.000059	com.mckinsey
590	18945824	1952	0.000017	com.reverbnation
591	18944198	1271	0.000023	net.clickbank
592	18944066	319	0.000082	jp.co.amazon
593	18941406	2109	0.000016	com.jimdosite
594	18939532	3017	0.000011	com.self
595	18938332	201	0.000132	ru.mail
596	18937396	1942	0.000017	gov.eia
597	18937228	3073	0.000011	org.oas
598	18936846	656	0.000041	com.iheart
599	18935892	2163	0.000016	com.haaretz
600	18935478	3045	0.000011	edu.syr
601	18935194	978	0.000030	com.icons8
602	18934738	243	0.000108	to.amzn
603	18934162	2562	0.000014	org.computer
604	18933548	1460	0.000020	site.notion
605	18929554	651	0.000042	org.iso
606	18929436	1837	0.000018	com.livescience
607	18928992	2618	0.000013	com.infogram
608	18928636	2155	0.000016	gov.usembassy
609	18928430	1348	0.000022	com.mapquest
610	18928262	3111	0.000011	com.tutorialspoint
611	18928028	665	0.000041	com.qualtrics
612	18927130	2022	0.000017	cn.gov.fmprc
613	18926910	697	0.000039	org.ieee
614	18926652	1078	0.000027	com.pcmag
615	18926588	2435	0.000015	com.popsugar
616	18925646	2157	0.000016	com.iconfinder
617	18925600	688	0.000039	com.entrepreneur
618	18925436	344	0.000077	com.visualstudio
619	18925272	1032	0.000028	com.dropboxusercontent
620	18924720	2863	0.000012	it.scoop
621	18922978	2767	0.000013	com.pbworks
622	18921774	2043	0.000016	ph.telegra
623	18920704	957	0.000030	me.onelink
624	18919802	3326	0.000010	org.grist
625	18917230	2833	0.000012	com.fineartamerica
626	18917112	2939	0.000012	au.edu.unimelb
627	18916394	1800	0.000019	mil.army
628	18914670	4477	0.000007	com.mail
629	18914274	3514	0.000010	com.afp
630	18914002	1345	0.000022	org.consumerreports
631	18911486	3760	0.000009	net.docdroid
632	18911096	862	0.000034	com.oreilly
633	18910022	2929	0.000012	com.novell
634	18908850	981	0.000030	org.mediawiki
635	18908110	1887	0.000018	com.bol
636	18908096	2755	0.000013	com.gq
637	18903732	1352	0.000022	com.maxmind
638	18903710	1241	0.000023	com.licdn
639	18903220	1791	0.000019	gov.cancer
640	18902968	2951	0.000012	com.eonline
641	18902644	3186	0.000011	com.theonion
642	18902286	2697	0.000013	net.openid
643	18901832	2060	0.000016	com.dictionary
644	18901570	2523	0.000014	com.foreignpolicy
645	18900276	2555	0.000014	org.c-span
646	18898920	506	0.000052	net.fastly
647	18898588	1941	0.000017	edu.tamu
648	18897290	699	0.000039	int.wipo
649	18896958	1101	0.000027	com.merriam-webster
650	18896876	3366	0.000010	org.freedomhouse
651	18896364	111	0.000328	com.livestream
652	18896362	1832	0.000018	com.verywellmind
653	18895668	379	0.000069	jp.ameblo
654	18894814	979	0.000030	com.forrester
655	18893636	1550	0.000019	com.wikia
656	18893130	2061	0.000016	org.unep
657	18892602	2340	0.000015	com.patch
658	18892586	159	0.000184	com.weibo
659	18890690	548	0.000048	com.sxsw
660	18889804	2722	0.000013	com.motherjones
661	18889768	1254	0.000023	com.jekyllrb
662	18889752	874	0.000034	gov.federalregister
663	18889268	3930	0.000009	com.instapaper
664	18888650	2844	0.000012	com.thecut
665	18887192	1107	0.000027	net.authorize
666	18886308	2493	0.000014	edu.gwu
667	18885348	3225	0.000010	org.csis
668	18884650	2514	0.000014	gov.ky
669	18884342	2709	0.000013	com.theintercept
670	18883802	3213	0.000011	ua.com.google
671	18883552	1530	0.000020	com.snopes
672	18883178	3655	0.000009	au.com.businessinsider
673	18883096	5172	0.000007	com.ixbt
674	18882438	2757	0.000013	org.fas
675	18882418	1070	0.000027	com.tableau
676	18882310	1462	0.000020	gov.uscourts
677	18881746	3551	0.000010	com.teacherspayteachers
678	18880066	664	0.000041	gov.sec
679	18877954	550	0.000048	com.scorecardresearch
680	18877136	2126	0.000016	org.ncsl
681	18877074	3975	0.000009	org.cpj
682	18876192	230	0.000117	com.naver
683	18875692	2680	0.000013	uk.ac.imperial
684	18875654	1205	0.000024	com.findlaw
685	18874462	2829	0.000012	si.gov
686	18874298	1113	0.000026	edu.ucla
687	18873866	2380	0.000015	com.voanews
688	18871112	3856	0.000009	org.edublogs
689	18868180	2869	0.000012	org.marketplace
690	18867852	3725	0.000009	net.aljazeera
691	18867276	2376	0.000015	com.channelnewsasia
692	18867194	723	0.000037	org.plos
693	18866674	1041	0.000028	net.atlassian
694	18866468	4110	0.000008	edu.ua
695	18866372	1210	0.000024	gov.uspto
696	18864712	2418	0.000015	com.goodhousekeeping
697	18863308	1328	0.000022	org.altervista
698	18862148	1867	0.000018	com.billboard
699	18861664	1686	0.000019	gov.govinfo
700	18860214	2238	0.000015	ru.ria
701	18860032	2679	0.000013	com.nationalpost
702	18859750	4921	0.000007	com.viki
703	18859350	3294	0.000010	com.hm
704	18858510	2867	0.000012	com.treehugger
705	18857788	932	0.000031	com.termsfeed
706	18857590	3457	0.000010	ru.interfax
707	18857080	1251	0.000023	com.squarespace-cdn
708	18856272	2966	0.000012	com.sandiegouniontribune
709	18856050	1506	0.000020	io.termly
710	18854390	4288	0.000008	com.dailycaller
711	18853340	2758	0.000013	com.html5rocks
712	18850964	3377	0.000010	is.archive
713	18850298	3057	0.000011	com.nextdoor
714	18849796	3786	0.000009	me.site123
715	18848064	1338	0.000022	org.mitre
716	18847026	5985	0.000006	com.fanpop
717	18846742	2599	0.000014	org.pewtrusts
718	18846728	3574	0.000009	org.britishcouncil
719	18846392	197	0.000136	com.caniuse
720	18845760	2369	0.000015	va.vatican
721	18845434	259	0.000102	com.getbootstrap
722	18843938	3714	0.000009	com.worldpopulationreview
723	18843390	530	0.000049	com.adweek
724	18842378	2234	0.000016	gov.oregon
725	18842176	906	0.000033	com.digitaloceanspaces
726	18841406	3644	0.000009	org.transparency
727	18841194	1324	0.000022	com.windows
728	18841082	3247	0.000010	com.tomsguide
729	18840980	535	0.000049	com.gofundme
730	18839198	2875	0.000012	org.unfpa
731	18838976	971	0.000030	com.imageshack
732	18838954	3992	0.000008	com.metacritic
733	18838084	4023	0.000008	org.carnegieendowment
734	18837572	498	0.000052	com.bigcommerce
735	18836694	916	0.000032	com.libsyn
736	18836082	1924	0.000017	com.kaltura
737	18835258	2926	0.000012	org.wikisource
738	18835144	3021	0.000011	org.gnupg
739	18835004	3164	0.000011	org.signal
740	18834884	493	0.000052	com.aol
741	18834658	3055	0.000011	no.uio
742	18834006	5000	0.000007	ua.nv
743	18833898	3110	0.000011	ru.vedomosti
744	18833794	2169	0.000016	com.wakelet
745	18833066	867	0.000034	com.zoho
746	18832896	640	0.000042	jp.ne.sakura
747	18832118	3963	0.000009	com.theweek
748	18831308	2113	0.000016	com.proquest
749	18830330	1118	0.000026	com.slate
750	18829038	1868	0.000018	com.speakerdeck
751	18828890	2825	0.000012	jp.nicovideo
752	18828214	241	0.000110	jp.co.google
753	18826472	3223	0.000010	com.tradingeconomics
754	18824538	2526	0.000014	com.radio
755	18824466	5366	0.000006	org.bakerlab
756	18824444	1168	0.000025	org.webaim
757	18823292	166	0.000169	org.whatwg
758	18823246	3038	0.000011	com.bloglovin
759	18822092	3620	0.000009	edu.temple
760	18821762	1071	0.000027	com.engadget
761	18821604	1193	0.000025	io.powr
762	18821134	671	0.000040	org.eff
763	18821128	4228	0.000008	com.virgin
764	18820708	330	0.000080	com.wistia
765	18820470	4223	0.000008	com.scotsman
766	18820390	3533	0.000010	ly.plot
767	18819496	3657	0.000009	de.diplo
768	18818614	1855	0.000018	com.ticketmaster
769	18817694	2700	0.000013	com.me
770	18816976	71	0.000441	com.oculus
771	18816804	2918	0.000012	com.digitaljournal
772	18816466	1580	0.000019	com.cbssports
773	18815294	3761	0.000009	io.fabric
774	18814250	2485	0.000014	com.surveygizmo
775	18811680	4234	0.000008	io.meduza
776	18811246	594	0.000045	fr.free
777	18810210	3782	0.000009	org.neocities
778	18810150	2685	0.000013	com.jpost
779	18809838	2701	0.000013	com.washingtontimes
780	18809742	3610	0.000009	org.annualreviews
781	18809160	3333	0.000010	int.nato
782	18808792	1994	0.000017	com.trustwave
783	18808532	3214	0.000011	org.heritage
784	18808074	3486	0.000010	org.repec
785	18807740	1809	0.000019	co.carrd
786	18807316	4100	0.000008	uk.co.timesonline
787	18806184	3358	0.000010	re.appsto
788	18805892	81	0.000387	org.nginx
789	18805568	1314	0.000022	com.playstation
790	18805506	3261	0.000010	uk.ac.leeds
791	18805074	521	0.000050	org.drupal
792	18804212	3656	0.000009	com.citylab
793	18803430	1185	0.000025	com.gizmodo
794	18803162	3903	0.000009	com.nationalreview
795	18802824	3095	0.000011	org.nrdc
796	18802742	3972	0.000009	net.openreview
797	18802624	2783	0.000012	com.wpcomstaging
798	18802584	2589	0.000014	org.sleepfoundation
799	18801620	4771	0.000007	com.bizcommunity
800	18801430	1304	0.000023	com.udemy
801	18801120	2874	0.000012	com.towardsdatascience
802	18800914	3649	0.000009	com.glitch
803	18800894	2162	0.000016	com.unity
804	18800888	549	0.000048	com.globenewswire
805	18800242	3836	0.000009	com.bepress
806	18800046	2567	0.000014	com.thespruce
807	18799488	1921	0.000017	ru.rbc
808	18798872	3774	0.000009	com.pbase
809	18798732	1486	0.000020	br.com.uol
810	18798690	3885	0.000009	ru.mid
811	18797228	3569	0.000009	org.wilsoncenter
812	18796512	4905	0.000007	it.justpaste
813	18795920	2460	0.000014	ru.rutube
814	18795064	866	0.000034	com.newsweek
815	18794898	3272	0.000010	au.edu.sydney
816	18793792	1975	0.000017	fr.blogspot
817	18793384	648	0.000042	com.mimecast
818	18792904	3771	0.000009	it.eventbrite
819	18792676	3090	0.000011	com.financialpost
820	18792614	1742	0.000019	com.technologyreview
821	18792574	4677	0.000007	edu.csun
822	18792430	4810	0.000007	org.scala-sbt
823	18792342	705	0.000038	net.b-cdn
824	18792186	3982	0.000008	com.indystar
825	18791040	2580	0.000014	ru.tass
826	18790612	1863	0.000018	ch.ethz
827	18790492	2764	0.000013	com.newrepublic
828	18790304	3622	0.000009	ca.uvic
829	18790238	614	0.000044	com.fandom
830	18789298	4500	0.000007	com.kinja
831	18788432	2923	0.000012	int.wmo
832	18786950	1036	0.000028	com.akamai
833	18785250	3444	0.000010	ru.lenta
834	18783154	3886	0.000009	com.slidesharecdn
835	18783124	4278	0.000008	org.elifesciences
836	18782898	2720	0.000013	com.fivethirtyeight
837	18782704	2770	0.000013	com.verywellhealth
838	18782398	729	0.000037	org.reactjs
839	18782236	1055	0.000028	org.unicode
840	18780268	2032	0.000016	org.americanbar
841	18779610	4850	0.000007	co.aeon
842	18779582	754	0.000036	com.moz
843	18779480	4695	0.000007	com.jigsy
844	18779480	202	0.000131	com.jimcdn
845	18779168	1990	0.000017	com.kxcdn
846	18779096	1802	0.000019	com.images-amazon
847	18778590	4028	0.000008	com.thediplomat
848	18778408	4189	0.000008	com.allafrica
849	18778074	1854	0.000018	gov.medlineplus
850	18777630	1027	0.000029	com.emarketer
851	18776948	3103	0.000011	com.blogtalkradio
852	18776668	3148	0.000011	com.biography
853	18775650	1355	0.000022	com.xkcd
854	18774718	1387	0.000021	com.thenextweb
855	18774182	1035	0.000028	com.css-tricks
856	18773746	2765	0.000013	io.redis
857	18773632	1830	0.000018	io.kubernetes
858	18771512	3430	0.000010	fr.rfi
859	18770806	4308	0.000008	au.edu.adelaide
860	18770028	3176	0.000011	org.nationalgeographic
861	18769896	1060	0.000028	com.yandex
862	18767830	3264	0.000010	org.panda
863	18767292	271	0.000097	de.amazon
864	18766934	4471	0.000008	fi.hs
865	18766498	2992	0.000011	com.euractiv
866	18766350	5125	0.000007	edu.umt
867	18765526	3720	0.000009	net.ipsnews
868	18765048	224	0.000118	org.icann
869	18764642	3971	0.000009	gov.ornl
870	18763736	4042	0.000008	org.thinkprogress
871	18763532	3536	0.000010	vn.com.google
872	18763496	1474	0.000020	edu.umd
873	18763446	441	0.000059	org.opensource
874	18762766	2839	0.000012	fi.yle
875	18761268	1129	0.000026	com.glassdoor
876	18760974	3809	0.000009	com.crashlytics
877	18760580	383	0.000069	it.google
878	18760534	3392	0.000010	cn.globaltimes
879	18759470	3841	0.000009	com.sputniknews
880	18759316	3296	0.000010	gov.doi
881	18759284	1048	0.000028	ly.cutt
882	18759154	3769	0.000009	com.clarin
883	18759124	3833	0.000009	uk.gov.metoffice
884	18758894	5758	0.000006	org.cgsociety
885	18758746	1452	0.000020	com.rollingstone
886	18758550	1458	0.000020	com.smashingmagazine
887	18758014	3209	0.000011	org.cfr
888	18757926	3933	0.000009	gov.fec
889	18757834	3530	0.000010	ru.rosminzdrav
890	18756978	1243	0.000023	org.golang
891	18756894	5382	0.000006	edu.chapman
892	18756824	4544	0.000007	uk.ac.nhm
893	18756496	5110	0.000007	au.edu.uts
894	18756282	1373	0.000021	edu.ucsd
895	18756272	3427	0.000010	edu.unh
896	18755430	3498	0.000010	jp.ne.docomo
897	18755124	2300	0.000015	com.w3techs
898	18754824	950	0.000031	com.ubuntu
899	18754654	1889	0.000018	com.indiegogo
900	18754566	4075	0.000008	org.tigris
901	18754136	1879	0.000018	int.itu
902	18753386	4143	0.000008	com.coca-cola
903	18752518	3825	0.000009	ru.gazeta
904	18752498	4300	0.000008	ch.swissinfo
905	18750682	2507	0.000014	se.haxx
906	18750434	5143	0.000007	com.chinatimes
907	18749260	4702	0.000007	edu.upf
908	18747924	1862	0.000018	sh.brew
909	18747896	4795	0.000007	kr.co.koreatimes
910	18747192	3805	0.000009	mt.gov
911	18746096	4217	0.000008	com.motor1
912	18745808	5756	0.000006	com.tv
913	18745512	2104	0.000016	net.vnexpress
914	18744084	2565	0.000014	gd.is
915	18744012	2793	0.000012	ru.hh
916	18743684	1270	0.000023	org.wiktionary
917	18743628	4299	0.000008	uk.ac.exeter
918	18743132	3471	0.000010	com.bhg
919	18742516	1904	0.000017	org.linuxfoundation
920	18742090	3908	0.000009	build.bazel
921	18741624	1401	0.000021	com.freeprivacypolicy
922	18741410	3334	0.000010	cn.org.china
923	18740876	2573	0.000014	com.pcworld
924	18740070	4403	0.000008	com.bravesites
925	18740044	3218	0.000010	com.nyt
926	18739968	4027	0.000008	com.usmagazine
927	18739330	1203	0.000024	com.webs
928	18739298	5530	0.000006	com.gust
929	18739228	5683	0.000006	tv.eurovision
930	18738996	6003	0.000006	ke.co.google
931	18738768	5498	0.000006	tw.org.rti
932	18738216	899	0.000033	com.elpais
933	18737996	2574	0.000014	ru.rg
934	18737310	5139	0.000007	com.defensenews
935	18736786	4709	0.000007	com.alignable
936	18736772	2241	0.000015	ru.kommersant
937	18736336	650	0.000042	com.accenture
938	18736124	3843	0.000009	tr.com.aa
939	18736094	1019	0.000029	com.buzzsprout
940	18735764	2554	0.000014	ru.mos
941	18735662	3192	0.000011	com.post-gazette
942	18733672	4867	0.000007	com.revolut
943	18733416	5107	0.000007	org.siggraph
944	18732882	2100	0.000016	com.hackerone
945	18732676	4096	0.000008	uk.ac.core
946	18732092	7116	0.000005	com.orgfree
947	18731434	4215	0.000008	org.jenkins-ci
948	18731344	373	0.000070	mp.mailchi
949	18731210	4764	0.000007	fr.huffingtonpost
950	18728952	5603	0.000006	net.zshare
951	18728946	4191	0.000008	com.encyclopedia
952	18728822	3020	0.000011	com.devpost
953	18728208	4792	0.000007	com.iconarchive
954	18728168	3356	0.000010	com.washingtonexaminer
955	18727110	5507	0.000006	uk.org.rspb
956	18726884	2250	0.000015	org.donorbox
957	18726876	954	0.000030	edu.wisc
958	18726550	3373	0.000010	org.rferl
959	18726378	3789	0.000009	nl.wur
960	18725900	5637	0.000006	jp.riken
961	18725706	1829	0.000018	com.homeadvisor
962	18725558	1466	0.000020	org.owasp
963	18725218	1159	0.000025	com.imrworldwide
964	18724578	93	0.000344	com.messenger
965	18723300	2897	0.000012	ru.kp
966	18723096	3931	0.000009	gov.ustr
967	18722424	878	0.000034	edu.umich
968	18722176	3363	0.000010	int.iom
969	18721796	1408	0.000021	com.sfgate
970	18721554	2776	0.000013	com.cloudwaysapps
971	18721416	660	0.000041	com.psychologytoday
972	18721272	4996	0.000007	org.geogebra
973	18720848	1263	0.000023	edu.hbs
974	18720786	4336	0.000008	com.podomatic
975	18720500	3758	0.000009	ru.avito
976	18720174	1194	0.000025	com.searchengineland
977	18718614	927	0.000032	com.wikihow
978	18717880	5397	0.000006	com.nippon
979	18717810	4662	0.000007	org.democracynow
980	18716504	519	0.000050	gov.fda
981	18716444	6716	0.000005	uk.ac.aber
982	18716304	4204	0.000008	com.vancouversun
983	18716098	5646	0.000006	re.cli
984	18715622	3847	0.000009	edu.sc
985	18714542	1153	0.000025	to.dev
986	18713766	1012	0.000029	org.frontiersin
987	18713302	405	0.000064	com.constantcontact
988	18713076	4221	0.000008	org.sonatype
989	18711534	2510	0.000014	com.etonline
990	18711418	597	0.000045	com.figma
991	18710884	1207	0.000024	edu.nyu
992	18709766	3679	0.000009	org.ets
993	18709648	6296	0.000006	org.sfpl
994	18708946	2403	0.000015	com.alibabagroup
995	18708874	4758	0.000007	net.thedailystar
996	18708726	3477	0.000010	com.bp
997	18708662	2647	0.000013	ca.citizenlab
998	18708340	2682	0.000013	com.discogs
999	18707350	5517	0.000006	com.maxpreps
1000	18705518	364	0.000073	com.heroku

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope you find the data useful for any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

January/February 2023 crawl archive now available

February 16, 2023Sebastian Nagel

The crawl archive for January/February 2023 is now available! The data was crawled January 26 – February 9 and contains 3.15 billion web pages or 400 TiB of uncompressed content. Page captures are from 40 million hosts or 33 million registered domains and include 1.3 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The January/February crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2023-06/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively, please see accessing the data for detailed instructions.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2023-06/segment.paths.gz	100
WARC files	CC-MAIN-2023-06/warc.paths.gz	88000	88.02
WAT files	CC-MAIN-2023-06/wat.paths.gz	88000	21.72
WET files	CC-MAIN-2023-06/wet.paths.gz	88000	9.05
Robots.txt files	CC-MAIN-2023-06/robotstxt.paths.gz	88000	0.13
Non-200 responses files	CC-MAIN-2023-06/non200responses.paths.gz	88000	2.04
URL index files	CC-MAIN-2023-06/cc-index.paths.gz	302	0.23
Columnar URL index files	CC-MAIN-2023-06/cc-index-table.paths.gz	900	0.26

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2023-06/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

November/December 2022 crawl archive now available

December 14, 2022Sebastian Nagel

The crawl archive for November/December 2022 is now available! The data was crawled November 26 – December 10 and contains 3.35 billion web pages or 420 TiB of uncompressed content. Page captures are from 44 million hosts or 34 million registered domains and include 1.2 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The November/December crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2022-49/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2022-49/segment.paths.gz	100
WARC files	CC-MAIN-2022-49/warc.paths.gz	88000	92.59
WAT files	CC-MAIN-2022-49/wat.paths.gz	88000	22.89
WET files	CC-MAIN-2022-49/wet.paths.gz	88000	9.58
Robots.txt files	CC-MAIN-2022-49/robotstxt.paths.gz	88000	0.15
Non-200 responses files	CC-MAIN-2022-49/non200responses.paths.gz	88000	2.43
URL index files	CC-MAIN-2022-49/cc-index.paths.gz	302	0.25
Columnar URL index files	CC-MAIN-2022-49/cc-index-table.paths.gz	900	0.28

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2022-49/. Also the columnar index has been updated to contain this crawl.

September/October 2022 crawl archive now available

October 11, 2022Sebastian Nagel

The crawl archive for September/October 2022 is now available! The data was crawled September 24 – October 8 and contains 3.15 billion web pages or 380 TiB of uncompressed content. Page captures are from 44 million hosts or 34 million registered domains and include 1.3 billion new URLs, not visited in any of our prior crawls. This crawl includes improvements made in extracting clean text in WET files and WAT anchor texts.

Archive Location and Download

The September/October crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2022-40/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2022-40/segment.paths.gz	100
WARC files	CC-MAIN-2022-40/warc.paths.gz	80000	82.71
WAT files	CC-MAIN-2022-40/wat.paths.gz	80000	21.49
WET files	CC-MAIN-2022-40/wet.paths.gz	80000	9.11
Robots.txt files	CC-MAIN-2022-40/robotstxt.paths.gz	80000	0.13
Non-200 responses files	CC-MAIN-2022-40/non200responses.paths.gz	80000	1.96
URL index files	CC-MAIN-2022-40/cc-index.paths.gz	302	0.23
Columnar URL index files	CC-MAIN-2022-40/cc-index-table.paths.gz	900	0.26

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2022-40/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs May, June/July and August 2022

September 23, 2022Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of May, June/July and August 2022. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases. You may also visit the projects cc-webgraph and cc-pyspark which include all scripts and tools required to construct the graphs. Instructions to explore the graphs in the webgraph format are given in our collection of webgraph notebooks. See below for a summary of changes and improvements implemented for the current web graph release.

Changes, improvements and bug fixes

Unicode internationalized domain names are always converted into their ASCII equivalents (IDNA). This is now ensured for node labels in the host-level webgraph (see cc-pyspark#35) and consequently also for the domain-level webgraph where non-ASCII characters were replaced by question marks (see cc-webgraph#6)
The nodes of the domain graph are now strictly sorted lexicographically by node label (the reverse domain name). This should allow for more efficient compression of the list of domain nodes.
The strict sorting was implemented to address a bug (cc-webgraph#3) which may cause duplicated nodes (two or more nodes with the same label) in the domain graph.
The domain graph includes domain names equal to multi-part public suffixes. Previously the assumption was that names of registered domains are exactly one level below any ICANN suffix in the public suffix list and host names which are equal to multi-part suffixes (including at least one dot) were excluded. Such host names are now included, eg. gov.uk, freight.aero or altoadige.it. No further validation (eg. DNS lookup) is performed, so also invalid domain names may be included. Generally, except for a valid domain name string with a valid TLD or public suffix, no further validation is performed for any domain name. For more details, see cc-webgraph#1.

Host-level graph

The graph consists of 449 million nodes and 2.69 billion edges. Both hyperlinks and HTTP redirects and link headers are used as edges to span up the graph. All types of links are included, including pure “technical” ones pointing to images, JavaScript libraries, web fonts, etc. However, only host names with a valid IANA TLD are used. Consequently, URLs with an IP address as host component are not taken into account for building the host-level graph.

There are 389 million dangling nodes (86.6%) and the largest strongly connected component contains 46.4 million (10.3%) nodes. Dangling nodes stem from

hosts that have not been crawled, yet are pointed to from a link on a crawled page
hosts without any links pointing to a different host name
or hosts which did only return an error page (eg. HTTP 404)

Host names in the graph are in reverse domain name notation and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 449 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/host/ (this requires an account on AWS). Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/host/ as prefix to access the files from everywhere.

Please note that the text representation of the host-level graph is shipped in 96 gzip-compressed files listed in two path listings – one for the nodes (vertices), one for the edges (arcs). First, download the paths listing and decompress it using “gzip -d” or “gunzip”. By adding the prefix s3://commoncrawl/ or https://data.commoncrawl.org/ to each line in the path listing you get the list of URLs to download the entire graph.

Download files of the Common Crawl May/Jun/Aug 2022 host-level webgraph

Size	File	Description
3.09 GB	cc-main-2022-may-jun-aug-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 32 vertices files
11.91 GB	cc-main-2022-may-jun-aug-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 64 edges files

5.76 GB	cc-main-2022-may-jun-aug-host.graph	graph in BVGraph format
2 kB	cc-main-2022-may-jun-aug-host.properties

6.20 GB	cc-main-2022-may-jun-aug-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2022-may-jun-aug-host-t.properties

1 kB	cc-main-2022-may-jun-aug-host.stats	WebGraph statistics

7.46 GB	cc-main-2022-may-jun-aug-host-ranks.txt.gz	harmonic centrality and pagerank

Domain-level graph

The domain graph is built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org. Version (commit) e5ff0c7 of the public suffix list was used (commit date 2022-09-15).

The domain-level graph has 91 million nodes and 1.57 billion edges. 50% or 45 million nodes are dangling nodes, the largest strongly connected component covers 37 million or 40% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/domain/ or on https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2022-may-jun-aug/domain/.

Download files of the Common Crawl May/Jun/Aug 2022 domain-level webgraph

Size	File	Description
0.63 GB	cc-main-2022-may-jun-aug-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
6.52 GB	cc-main-2022-may-jun-aug-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

3.77 GB	cc-main-2022-may-jun-aug-domain.graph	graph in BVGraph format
2 kB	cc-main-2022-may-jun-aug-domain.properties

3.59 GB	cc-main-2022-may-jun-aug-domain-t.graph	transpose of the graph
2 kB	cc-main-2022-may-jun-aug-domain-t.properties

1 kB	cc-main-2022-may-jun-aug-domain.stats	WebGraph statistics

1.96 GB	cc-main-2022-may-jun-aug-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 91 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (May/Jun/Aug 2022)

harmonic centrality rank	hc value	page rank	page rank value	reversed domain name
1	32914686	1	0.018077	com.googleapis
2	32131562	3	0.012273	com.facebook
3	31770190	2	0.015371	com.google
4	27971714	5	0.007018	com.twitter
5	27839074	7	0.006164	com.youtube
6	27662708	6	0.006892	org.w
7	27254468	8	0.005701	com.instagram
8	26739486	4	0.007602	com.googletagmanager
9	25688664	10	0.004673	org.gmpg
10	25578590	9	0.004792	com.gstatic
11	25033650	12	0.003435	com.linkedin
12	24026588	11	0.004116	com.cloudflare
13	23679430	17	0.002013	com.gravatar
14	23526052	13	0.002488	org.wordpress
15	23497052	24	0.001546	com.pinterest
16	23111634	28	0.001244	org.wikipedia
17	23073066	14	0.002254	com.apple
18	22828954	25	0.001434	com.wordpress
19	22794652	31	0.001150	com.vimeo
20	22725476	39	0.000940	be.youtu
21	22500412	18	0.001913	com.bootstrapcdn
22	22420202	32	0.001128	com.microsoft
23	22394764	15	0.002193	net.cloudfront
24	22370010	22	0.001568	com.jquery
25	22285798	23	0.001553	io.polyfill
26	22278972	51	0.000652	com.blogspot
27	22275856	44	0.000799	gl.goo
28	22242208	35	0.001012	com.amazonaws
29	22199000	47	0.000701	com.amazon
30	22170846	27	0.001252	net.jsdelivr
31	22147346	46	0.000764	eu.europa
32	22143092	41	0.000874	ly.bit
33	22058786	42	0.000835	org.mozilla
34	22050970	38	0.000958	com.google-analytics
35	22028542	21	0.001626	com.fontawesome
36	21967818	36	0.001001	com.adobe
37	21947388	20	0.001865	com.github
38	21939440	94	0.000371	com.tumblr
39	21919148	19	0.001882	com.googleusercontent
40	21916910	49	0.000687	com.wp
41	21896858	52	0.000647	com.paypal
42	21790948	61	0.000550	co.t
43	21769982	48	0.000695	com.whatsapp
44	21761882	54	0.000605	com.flickr
45	21753952	99	0.000356	com.yahoo
46	21729404	69	0.000515	io.github
47	21713518	133	0.000248	com.nytimes
48	21675054	34	0.001031	ru.yandex
49	21669788	91	0.000382	com.medium
50	21638440	30	0.001195	com.wixstatic
51	21614440	67	0.000526	com.shopify
52	21601372	119	0.000315	com.reddit
53	21594356	158	0.000193	com.forbes
54	21576744	40	0.000925	com.googlesyndication
55	21576166	63	0.000546	org.w3
56	21562668	131	0.000257	com.soundcloud
57	21539526	108	0.000328	com.weebly
58	21506586	59	0.000571	org.schema
59	21485114	122	0.000306	org.creativecommons
60	21473180	153	0.000207	gov.nih
61	21464862	179	0.000156	int.who
62	21460442	65	0.000529	com.vk
63	21421598	177	0.000158	com.theguardian
64	21419292	203	0.000129	com.cnn
65	21414034	147	0.000213	org.archive
66	21410902	218	0.000122	uk.co.bbc
67	21408640	50	0.000660	net.doubleclick
68	21393638	62	0.000550	com.unpkg
69	21390194	217	0.000122	com.businessinsider
70	21384262	149	0.000212	com.tiktok
71	21379450	198	0.000134	com.imgur
72	21369906	106	0.000332	me.wp
73	21362530	78	0.000407	com.android
74	21361648	148	0.000213	com.wixsite
75	21359636	56	0.000603	com.addthis
76	21350540	281	0.000098	com.bloomberg
77	21338508	60	0.000564	com.fb
78	21332614	337	0.000083	edu.stanford
79	21330884	355	0.000078	com.theverge
80	21308772	57	0.000588	com.macromedia
81	21306812	240	0.000109	com.imdb
82	21305630	117	0.000324	me.t
83	21301714	181	0.000154	com.bing
84	21299082	92	0.000379	com.giphy
85	21277366	300	0.000093	com.bbc
86	21270958	100	0.000353	com.list-manage
87	21266506	43	0.000827	net.fbcdn
88	21265192	143	0.000218	gle.forms
89	21263378	252	0.000106	com.wsj
90	21237826	368	0.000075	com.go
91	21236654	321	0.000087	com.reuters
92	21236614	220	0.000120	org.ietf
93	21233526	132	0.000253	com.statcounter
94	21223988	93	0.000375	com.stripe
95	21222332	194	0.000137	uk.gov
96	21221406	302	0.000093	edu.mit
97	21219964	254	0.000105	org.un
98	21218302	295	0.000096	edu.harvard
99	21218140	184	0.000151	com.issuu
100	21215566	175	0.000159	gov.cdc
101	21213292	120	0.000314	de.google
102	21212876	285	0.000097	com.oracle
103	21208848	150	0.000209	com.ytimg
104	21206504	396	0.000068	com.cnet
105	21204790	338	0.000083	com.techcrunch
106	21203072	365	0.000075	gov.nasa
107	21198090	157	0.000198	com.dropbox
108	21197416	476	0.000055	com.msn
109	21196722	249	0.000107	com.twimg
110	21191914	357	0.000077	com.quora
111	21190924	367	0.000075	com.wired
112	21184172	289	0.000097	net.slideshare
113	21184136	190	0.000142	com.unsplash
114	21183394	73	0.000469	com.wix
115	21181708	174	0.000160	org.apache
116	21171102	455	0.000058	com.googleblog
117	21169330	135	0.000237	com.mailchimp
118	21168736	182	0.000153	com.etsy
119	21167548	364	0.000075	org.hbr
120	21162976	125	0.000284	com.spotify
121	21159700	248	0.000107	com.stackoverflow
122	21145710	206	0.000127	com.blogger
123	21144796	397	0.000067	org.arxiv
124	21140336	292	0.000096	com.slack
125	21139536	270	0.000101	net.researchgate
126	21138540	263	0.000104	uk.co.amazon
127	21136788	345	0.000080	org.npr
128	21134310	374	0.000073	com.example
129	21128534	156	0.000200	us.zoom
130	21128018	236	0.000110	com.washingtonpost
131	21124494	363	0.000076	com.appspot
132	21117290	124	0.000298	com.ft
133	21115674	325	0.000086	com.cnbc
134	21115136	309	0.000091	com.wiley
135	21112604	356	0.000078	com.nature
136	21104960	510	0.000052	edu.berkeley
137	21100090	483	0.000055	com.myspace
138	21095354	216	0.000122	com.outlook
139	21093680	298	0.000095	org.acm
140	21091496	155	0.000203	com.weibo
141	21088936	152	0.000208	org.networkadvertising
142	21080382	572	0.000047	com.cbsnews
143	21079152	279	0.000099	org.gnu
144	21074760	437	0.000061	uk.co.telegraph
145	21073516	68	0.000520	com.godaddy
146	21068068	291	0.000096	uk.co.google
147	21065050	130	0.000266	com.youtube-nocookie
148	21065030	195	0.000136	org.wikimedia
149	21062066	393	0.000069	com.usatoday
150	21061852	645	0.000041	com.intel
151	21052752	478	0.000055	com.goodreads
152	21048112	378	0.000072	com.time
153	21041138	481	0.000055	com.theatlantic
154	21040374	633	0.000042	com.box
155	21038502	274	0.000100	com.squarespace
156	21031308	204	0.000129	com.eventbrite
157	21030242	37	0.000977	com.qq
158	21030040	185	0.000150	com.yelp
159	21025664	137	0.000230	com.opera
160	21022330	360	0.000076	ee.linktr
161	21022182	1141	0.000026	com.wikia
162	21020670	349	0.000079	com.springer
163	21017524	465	0.000056	com.latimes
164	21017202	170	0.000165	com.zendesk
165	21015596	424	0.000062	com.huffingtonpost
166	21014202	162	0.000185	org.ampproject
167	21011874	574	0.000046	com.indiatimes
168	21009830	145	0.000217	info.aboutads
169	21006182	886	0.000035	com.qz
170	21005110	704	0.000039	org.chromium
171	21003538	682	0.000040	com.buzzfeed
172	21000898	221	0.000120	org.doi
173	20999442	585	0.000045	com.vice
174	20989458	1116	0.000027	com.thenextweb
175	20987914	304	0.000092	com.typeform
176	20983612	261	0.000104	com.sciencedirect
177	20982702	506	0.000053	edu.cornell
178	20982294	544	0.000049	com.mashable
179	20977172	626	0.000043	com.scribd
180	20973696	523	0.000051	edu.yale
181	20971220	501	0.000053	uk.co.independent
182	20970866	258	0.000105	net.behance
183	20970776	679	0.000040	com.economist
184	20968290	747	0.000037	edu.upenn
185	20964282	278	0.000099	org.pewresearch
186	20960828	545	0.000049	com.cisco
187	20960582	451	0.000058	com.bigcommerce
188	20956062	564	0.000047	com.psychologytoday
189	20942726	513	0.000052	com.fortune
190	20942618	193	0.000139	page.g
191	20940302	382	0.000071	com.gitlab
192	20939136	462	0.000057	uk.co.dailymail
193	20936146	432	0.000061	com.pixabay
194	20933922	306	0.000091	com.tinyurl
195	20932536	497	0.000053	com.deloitte
196	20932040	956	0.000031	com.evernote
197	20925436	542	0.000049	io.codepen
198	20924450	212	0.000125	com.calendly
199	20923226	694	0.000039	com.vox
200	20919494	731	0.000038	com.git-scm
201	20918582	610	0.000044	org.unesco
202	20917448	1008	0.000030	com.about
203	20916974	71	0.000469	net.facebook
204	20915832	571	0.000047	org.weforum
205	20915084	419	0.000062	com.w3schools
206	20914978	326	0.000086	com.typepad
207	20911268	515	0.000052	com.squareup
208	20907742	904	0.000034	com.arstechnica
209	20900952	473	0.000055	com.nbcnews
210	20899924	373	0.000074	co.ibb
211	20899532	632	0.000042	com.withgoogle
212	20898928	809	0.000036	edu.washington
213	20896616	521	0.000051	com.inc
214	20891820	898	0.000034	uk.ac.cam
215	20886310	405	0.000066	com.sagepub
216	20878134	543	0.000049	fm.anchor
217	20876826	683	0.000040	com.apnews
218	20875670	967	0.000031	com.slate
219	20875650	442	0.000059	gov.whitehouse
220	20872664	689	0.000040	com.venturebeat
221	20871102	530	0.000050	com.pexels
222	20866632	242	0.000109	org.iana
223	20865428	260	0.000105	de.amazon
224	20862054	549	0.000048	gov.noaa
225	20860884	755	0.000037	me.about
226	20858432	33	0.001073	com.baidu
227	20856406	1312	0.000023	org.eclipse
228	20854214	609	0.000044	com.mysql
229	20847014	244	0.000108	com.live
230	20846064	654	0.000041	com.nationalgeographic
231	20844358	876	0.000035	edu.asu
232	20842882	299	0.000094	com.ibm
233	20839080	196	0.000136	jp.co.google
234	20835838	351	0.000078	com.dribbble
235	20835472	716	0.000038	ca.cbc
236	20828032	558	0.000048	org.worldbank
237	20827500	1278	0.000023	com.nike
238	20814914	459	0.000057	gov.fda
239	20813098	603	0.000044	org.pbs
240	20811434	586	0.000045	gov.loc
241	20810490	467	0.000056	gov.usda
242	20810284	485	0.000054	com.gofundme
243	20807808	316	0.000088	com.feedburner
244	20807006	329	0.000084	net.windows
245	20805276	1132	0.000027	com.hollywoodreporter
246	20804930	161	0.000187	com.staticflickr
247	20804690	1003	0.000030	org.greenpeace
248	20802310	492	0.000054	com.tandfonline
249	20802308	339	0.000081	eu.youronlinechoices
250	20801724	992	0.000031	app.netlify
251	20801376	1281	0.000023	com.billboard
252	20799338	642	0.000042	com.newyorker
253	20798194	875	0.000035	edu.wisc
254	20796936	702	0.000039	au.net.abc
255	20796272	916	0.000033	org.pypi
256	20795900	176	0.000159	com.office
257	20795318	1266	0.000024	com.technologyreview
258	20784974	477	0.000055	com.theconversation
259	20782898	887	0.000035	org.sciencemag
260	20782640	253	0.000105	com.jotform
261	20779480	984	0.000031	com.gizmodo
262	20778708	673	0.000040	org.cambridge
263	20777714	1294	0.000023	com.500px
264	20777238	730	0.000038	com.walmart
265	20775900	525	0.000051	com.oup
266	20773216	608	0.000044	com.xinhuanet
267	20772144	429	0.000061	com.getpocket
268	20770590	1181	0.000025	edu.umd
269	20768772	500	0.000053	gov.epa
270	20767862	709	0.000039	org.bitbucket
271	20767390	1144	0.000026	edu.purdue
272	20763440	1383	0.000022	ms.1drv
273	20762974	1084	0.000028	co.elastic
274	20760120	891	0.000034	org.semver
275	20755544	430	0.000061	org.debian
276	20753630	1308	0.000023	org.kernel
277	20749768	757	0.000037	com.britannica
278	20749716	963	0.000031	com.nypost
279	20747130	638	0.000042	com.elpais
280	20744652	929	0.000032	com.foxnews
281	20738360	502	0.000053	com.dailymotion
282	20736612	1154	0.000026	com.sky
283	20735678	1000	0.000030	com.uk
284	20729688	246	0.000108	com.wpengine
285	20728890	1623	0.000019	com.googlesource
286	20726846	1007	0.000030	edu.princeton
287	20725440	548	0.000048	gov.house
288	20722436	592	0.000045	com.mozilla
289	20721772	86	0.000393	com.wsimg
290	20721658	1404	0.000021	com.over-blog
291	20718906	488	0.000054	com.ted
292	20716838	1660	0.000018	com.lego
293	20715928	754	0.000037	gov.justice
294	20714832	1005	0.000030	uk.co.guardian
295	20713856	425	0.000062	com.arcgis
296	20713486	1319	0.000023	com.digitaltrends
297	20711750	795	0.000036	edu.umich
298	20710650	428	0.000061	org.openstreetmap
299	20709586	241	0.000109	net.sourceforge
300	20708624	947	0.000032	com.ssrn
301	20703088	1697	0.000018	org.usenix
302	20700354	386	0.000070	com.netdna-ssl
303	20698318	935	0.000032	com.ggpht
304	20697518	232	0.000113	com.amazon-adsystem
305	20696838	314	0.000090	tv.twitch
306	20696330	950	0.000032	uk.co.blogspot
307	20696136	1416	0.000021	com.hatenablog
308	20692884	1149	0.000026	co.g
309	20691964	230	0.000114	gov.ca
310	20689588	801	0.000036	com.politico
311	20689246	1315	0.000023	com.socialmediatoday
312	20686440	728	0.000038	org.change
313	20685528	239	0.000110	uk.org.ico
314	20685498	223	0.000119	jp.co.yahoo
315	20685212	588	0.000045	uk.gov.service
316	20684354	171	0.000162	com.rawgit
317	20684232	280	0.000098	net.azureedge
318	20681826	1210	0.000025	io.itch
319	20680346	1318	0.000023	de.mpg
320	20678714	1552	0.000019	com.euronews
321	20676490	964	0.000031	edu.jhu
322	20676186	940	0.000032	edu.umn
323	20675058	531	0.000050	site.business
324	20672708	169	0.000166	com.addtoany
325	20671774	474	0.000055	gov.hhs
326	20670114	412	0.000064	com.ebay
327	20668938	1550	0.000019	com.urbandictionary
328	20664866	1182	0.000025	com.axios
329	20664008	1242	0.000024	org.semanticscholar
330	20663034	1103	0.000027	com.udemy
331	20662500	1395	0.000021	com.reverbnation
332	20659858	1505	0.000020	edu.indiana
333	20656824	1481	0.000020	au.com.news
334	20654924	1079	0.000028	edu.uchicago
335	20654274	752	0.000037	org.fao
336	20653112	622	0.000043	gov.census
337	20652698	1178	0.000025	net.speedtest
338	20650808	1771	0.000017	org.phys
339	20650016	74	0.000424	net.akamaihd
340	20647938	229	0.000115	com.hubspot
341	20642594	995	0.000030	com.scientificamerican
342	20641466	1328	0.000023	com.nymag
343	20638870	1788	0.000017	com.martinfowler
344	20638382	1663	0.000018	edu.gatech
345	20637680	555	0.000048	com.kickstarter
346	20635558	187	0.000146	com.xing
347	20635506	1129	0.000027	org.wiktionary
348	20634592	1042	0.000029	edu.utexas
349	20633912	2314	0.000015	com.flipboard
350	20633634	567	0.000047	com.snapchat
351	20633246	3204	0.000011	com.openai
352	20630316	1423	0.000021	ch.ethz
353	20629522	1420	0.000021	com.businessweek
354	20629100	873	0.000035	watch.fb
355	20626018	154	0.000206	com.sharethis
356	20625672	948	0.000032	com.timeanddate
357	20625036	720	0.000038	org.d3js
358	20624578	1744	0.000017	com.itv
359	20620690	1267	0.000024	uk.ac.ucl
360	20618834	1455	0.000020	uk.co.metro
361	20617718	320	0.000087	com.statista
362	20617602	529	0.000050	com.googlecode
363	20617378	1147	0.000026	com.jetbrains
364	20617292	614	0.000044	org.ohchr
365	20617266	915	0.000033	de.spiegel
366	20616654	472	0.000055	com.meetup
367	20616580	322	0.000086	com.disqus
368	20615966	399	0.000067	com.optimizely
369	20615414	2802	0.000013	com.diigo
370	20615048	287	0.000097	jp.ne.hatena
371	20614360	1285	0.000023	com.smithsonianmag
372	20614098	1152	0.000026	com.scmp
373	20614010	1110	0.000027	com.foursquare
374	20610490	2636	0.000014	blog.home
375	20610202	2020	0.000016	com.knowyourmeme
376	20607856	353	0.000078	net.themeforest
377	20607506	733	0.000038	au.gov.nsw
378	20606218	1078	0.000028	com.chicagotribune
379	20603548	1164	0.000026	au.com.smh
380	20603248	1589	0.000019	uk.co.express
381	20599986	1121	0.000027	edu.nyu
382	20599508	268	0.000102	com.npmjs
383	20598566	641	0.000042	gov.senate
384	20594974	639	0.000042	com.zdnet
385	20594264	1128	0.000027	link.page
386	20591568	968	0.000031	com.usps
387	20588732	890	0.000035	gov.congress
388	20586888	293	0.000096	com.eepurl
389	20585314	1002	0.000030	com.history
390	20584024	677	0.000040	com.pinimg
391	20582266	141	0.000221	com.paypalobjects
392	20581216	66	0.000528	com.googleadservices
393	20580344	450	0.000058	es.google
394	20579052	2736	0.000014	edu.byu
395	20577748	899	0.000034	au.com.google
396	20577588	1450	0.000021	uk.co.standard
397	20576632	711	0.000039	com.istockphoto
398	20572810	97	0.000357	net.jsfiddle
399	20572202	283	0.000097	me.telegram
400	20568536	1333	0.000022	cn.com.chinadaily
401	20568322	552	0.000048	ca.google
402	20567936	1174	0.000025	de.bild
403	20566704	1394	0.000022	com.producthunt
404	20566074	392	0.000069	com.proofpoint
405	20564788	955	0.000031	edu.si
406	20562416	635	0.000042	org.oecd
407	20559584	1479	0.000020	ca.ubc
408	20559174	1467	0.000020	com.wattpad
409	20558142	2132	0.000015	app.web
410	20557956	888	0.000035	google.blog
411	20557880	1095	0.000028	com.dw
412	20554318	719	0.000038	gov.archives
413	20553172	1491	0.000020	com.buzzfeednews
414	20552996	503	0.000053	nl.google
415	20551926	1921	0.000016	com.mystrikingly
416	20551384	458	0.000057	com.criteo
417	20550866	1035	0.000029	uk.co.thetimes
418	20549656	352	0.000078	com.prnewswire
419	20548982	1463	0.000020	uk.ac.lse
420	20548708	974	0.000031	in.co.google
421	20548238	380	0.000071	com.sohu
422	20544956	1448	0.000021	uk.co.wired
423	20544686	389	0.000069	com.atlassian
424	20544326	359	0.000077	net.php
425	20542034	527	0.000050	com.matterport
426	20540666	1638	0.000018	de.ebay
427	20536506	777	0.000036	com.livejournal
428	20535454	328	0.000085	ru.ok
429	20535130	1059	0.000029	gov.treasury
430	20534250	1194	0.000025	com.sun
431	20533698	1787	0.000017	com.channel4
432	20532900	381	0.000071	net.imgix
433	20532638	1932	0.000016	gov.cia
434	20532370	1054	0.000029	org.telegram
435	20531500	1053	0.000029	uk.parliament
436	20531096	2759	0.000013	ph.telegra
437	20530556	1509	0.000020	uk.co.thesun
438	20529928	593	0.000045	edu.cmu
439	20529694	1070	0.000028	int.coe
440	20528058	494	0.000053	com.media-amazon
441	20528052	1814	0.000017	com.hindustantimes
442	20527716	919	0.000033	com.iconfinder
443	20526664	1004	0.000030	org.jstor
444	20525290	1590	0.000019	com.straitstimes
445	20524680	1767	0.000017	edu.tufts
446	20523352	418	0.000062	com.elsevier
447	20520868	490	0.000054	ru.gov
448	20520208	1083	0.000028	gov.fbi
449	20516310	1371	0.000022	edu.duke
450	20514968	408	0.000065	com.adroll
451	20514226	1344	0.000022	int.itu
452	20513026	1382	0.000022	de.zeit
453	20512422	1654	0.000018	com.newscientist
454	20511574	372	0.000074	com.githubusercontent
455	20511532	1454	0.000021	com.unity3d
456	20509014	1712	0.000018	org.maven
457	20508604	988	0.000031	de.focus
458	20508282	2525	0.000015	com.storify
459	20506534	1475	0.000020	com.irishtimes
460	20506474	627	0.000043	gov.state
461	20505268	705	0.000039	uk.nhs
462	20505188	1711	0.000018	com.mercurynews
463	20505146	1196	0.000025	edu.unc
464	20504400	311	0.000090	com.mapbox
465	20503420	600	0.000044	net.ctfassets
466	20503084	1406	0.000021	jp.ne.goo
467	20501372	1690	0.000018	org.propublica
468	20499916	900	0.000034	gov.sba
469	20499600	2765	0.000013	me.ogp
470	20498960	1541	0.000020	com.mcafee
471	20498292	1564	0.000019	com.nydailynews
472	20497012	1322	0.000023	org.unhcr
473	20492506	1980	0.000016	com.csmonitor
474	20491416	1645	0.000018	ca.mcgill
475	20490972	496	0.000053	org.python
476	20488474	259	0.000105	gg.discord
477	20487728	3569	0.000010	net.docdroid
478	20485818	1881	0.000016	app.vercel
479	20484976	2557	0.000015	com.instructure
480	20483828	1263	0.000024	ch.ipcc
481	20481150	1943	0.000016	io.gitlab
482	20481144	267	0.000102	com.aliyuncs
483	20480398	1963	0.000016	com.thoughtco
484	20478522	1025	0.000030	gov.dhs
485	20477934	1635	0.000019	com.lenovo
486	20477440	1124	0.000027	gov.usgs
487	20475314	1037	0.000029	org.ilo
488	20472878	1246	0.000024	org.hrw
489	20472770	95	0.000363	me.wa
490	20472152	453	0.000058	com.samsung
491	20470420	142	0.000219	com.salesforce
492	20467196	2818	0.000013	com.oxforddictionaries
493	20466550	2539	0.000015	au.com.sbs
494	20465652	436	0.000061	com.filesusr
495	20464104	2051	0.000016	com.brave
496	20462132	1107	0.000027	com.thehill
497	20462008	1249	0.000024	com.aljazeera
498	20461328	1327	0.000023	com.brightcove
499	20460532	780	0.000036	com.thinkwithgoogle
500	20460298	576	0.000046	org.worldwildlife
501	20457830	2834	0.000013	sg.edu.nus
502	20455892	435	0.000061	com.visualstudio
503	20454454	3817	0.000009	com.minds
504	20454024	1029	0.000029	edu.brookings
505	20452798	1088	0.000028	sg.com.google
506	20452018	296	0.000095	gov.ftc
507	20451250	2081	0.000016	com.rt
508	20450574	1335	0.000022	de.welt
509	20450326	889	0.000035	com.fandom
510	20449110	1307	0.000023	de.sueddeutsche
511	20448850	498	0.000053	com.fastcompany
512	20448284	768	0.000037	com.oreilly
513	20448152	3182	0.000011	cc.uxdesign
514	20447408	905	0.000034	com.deviantart
515	20442684	449	0.000058	com.ssl-images-amazon
516	20442572	2891	0.000013	org.accessnow
517	20442500	3809	0.000009	org.edublogs
518	20441072	159	0.000192	com.jimdo
519	20439556	2245	0.000015	tl.we
520	20438646	3143	0.000012	com.instapaper
521	20437842	208	0.000125	ru.mail
522	20436396	457	0.000057	com.patreon
523	20436198	2841	0.000013	com.bloglovin
524	20434984	1539	0.000020	com.firebaseapp
525	20432342	3587	0.000010	com.pearltrees
526	20429974	2565	0.000015	edu.oregonstate
527	20428120	369	0.000074	com.surveymonkey
528	20426222	403	0.000066	com.businesswire
529	20425900	2907	0.000013	org.wikibooks
530	20423662	1108	0.000027	de.stern
531	20423310	1653	0.000018	com.warnerbros
532	20418994	1407	0.000021	be.google
533	20418380	1148	0.000026	ly.rebrand
534	20416106	1913	0.000016	edu.ucsb
535	20415774	598	0.000044	com.airbnb
536	20414102	98	0.000356	com.messenger
537	20413100	1566	0.000019	org.rfc-editor
538	20413010	303	0.000093	net.secureservercdn
539	20412834	1911	0.000016	co.carrd
540	20412638	2555	0.000015	it.scoop
541	20411920	787	0.000036	com.zoho
542	20411722	539	0.000050	com.gmail
543	20411284	923	0.000033	com.thelancet
544	20410478	2023	0.000016	com.dictionary
545	20408662	4662	0.000008	com.folkd
546	20408416	953	0.000032	edu.psu
547	20408248	1975	0.000016	org.documentcloud
548	20407678	1203	0.000025	org.undp
549	20406464	697	0.000039	io.readthedocs
550	20406272	1403	0.000021	net.codecanyon
551	20405676	3142	0.000012	com.hubpages
552	20403958	640	0.000042	com.entrepreneur
553	20402310	1855	0.000017	com.france24
554	20400524	237	0.000110	to.amzn
555	20399576	2550	0.000015	gov.lbl
556	20397996	3296	0.000011	google.ai
557	20397392	2812	0.000013	com.aboutamazon
558	20395738	1284	0.000023	com.snopes
559	20394748	1415	0.000021	int.unfccc
560	20394362	954	0.000032	com.ubuntu
561	20394264	213	0.000125	com.aspnetcdn
562	20393036	561	0.000047	com.steampowered
563	20392344	3041	0.000012	com.dreamstime
564	20391666	1527	0.000020	gov.defense
565	20390246	1829	0.000017	org.iea
566	20387722	2991	0.000012	com.oregonlive
567	20383972	3395	0.000011	org.neocities
568	20383752	1652	0.000018	io.ghost
569	20379568	2625	0.000014	org.nature
570	20378934	1180	0.000025	com.prweb
571	20378242	565	0.000047	com.netflix
572	20377878	1459	0.000020	mil.army
573	20377388	499	0.000053	org.nodejs
574	20377144	2229	0.000015	uk.bl
575	20377008	2049	0.000016	org.archlinux
576	20376392	1119	0.000027	com.dell
577	20376074	3049	0.000012	org.paho
578	20375762	2103	0.000016	com.thefreedictionary
579	20373064	701	0.000039	com.docker
580	20372496	2721	0.000014	org.computer
581	20372436	2650	0.000014	com.googlegroups
582	20370720	2233	0.000015	org.ap
583	20369470	3116	0.000012	com.webbyawards
584	20369394	138	0.000229	me.line
585	20369374	606	0.000044	com.investopedia
586	20369154	3126	0.000012	org.scala-lang
587	20368348	2738	0.000014	com.msnbc
588	20365992	1547	0.000019	ca.sfu
589	20363512	1764	0.000017	com.patch
590	20362914	1239	0.000024	net.clickbank
591	20362088	3223	0.000011	de.chip
592	20359640	3207	0.000011	org.vim
593	20357948	936	0.000032	org.js
594	20357518	1390	0.000022	io.shields
595	20357372	2965	0.000012	org.rsf
596	20352942	1798	0.000017	gov.usembassy
597	20351154	1385	0.000022	com.mixpanel
598	20350540	102	0.000349	com.uservoice
599	20350118	4086	0.000009	com.bravesites
600	20350052	2739	0.000014	edu.iastate
601	20349940	3202	0.000011	com.slides
602	20349860	893	0.000034	com.office365
603	20349118	3443	0.000010	org.aclweb
604	20349104	3375	0.000011	org.google
605	20347138	3411	0.000011	uk.co.yougov
606	20345188	579	0.000046	org.unicef
607	20345092	3195	0.000011	com.dummies
608	20345072	4395	0.000008	it.justpaste
609	20344796	3394	0.000011	org.globalcitizen
610	20344590	1874	0.000016	ca.globalnews
611	20343884	507	0.000053	com.fc2
612	20342034	524	0.000051	com.adweek
613	20341362	2867	0.000013	jp.co.japantimes
614	20340504	1270	0.000023	com.loom
615	20339376	937	0.000032	com.digitaloceanspaces
616	20339258	72	0.000469	com.oculus
617	20339126	744	0.000038	uk.co.pinterest
618	20338770	1091	0.000028	com.webs
619	20338486	3416	0.000011	com.thecvf
620	20332428	2657	0.000014	ca.ualberta
621	20331874	2252	0.000015	com.channelnewsasia
622	20331560	2968	0.000012	in.businessinsider
623	20330608	982	0.000031	org.mediawiki
624	20330438	1525	0.000020	com.bol
625	20328810	2325	0.000015	com.foreignpolicy
626	20328038	557	0.000048	com.digg
627	20326124	288	0.000097	com.bandcamp
628	20325846	959	0.000031	com.variety
629	20325100	1293	0.000023	org.imf
630	20324962	1130	0.000027	ly.cutt
631	20323872	3221	0.000011	org.freedomhouse
632	20323384	1746	0.000017	us.mn.state
633	20323358	4970	0.000007	com.sendspace
634	20322480	4337	0.000008	org.marxists
635	20322396	64	0.000540	com.trustpilot
636	20321834	332	0.000084	me.fb
637	20320766	1699	0.000018	com.ipsos
638	20320202	1698	0.000018	gov.uscis
639	20319680	211	0.000125	org.whatwg
640	20319120	1811	0.000017	eu.politico
641	20318744	5519	0.000006	com.edocr
642	20318266	3351	0.000011	de.diplo
643	20316524	2077	0.000016	com.spreaker
644	20316416	2961	0.000012	com.space
645	20315134	1866	0.000017	com.voanews
646	20314896	2750	0.000014	org.wikidata
647	20313896	2047	0.000016	dk.google
648	20309998	1033	0.000029	me.onelink
649	20309160	3695	0.000010	com.prweek
650	20308644	3578	0.000010	com.virgin
651	20307882	3206	0.000011	com.slidesharecdn
652	20306230	605	0.000044	com.canva
653	20305826	1615	0.000019	com.indianexpress
654	20305362	3379	0.000011	com.reason
655	20303486	908	0.000034	com.imageshack
656	20303212	3815	0.000009	org.cpj
657	20302922	1157	0.000026	com.att
658	20302106	759	0.000037	uk.co.eventbrite
659	20300748	3233	0.000011	com.hm
660	20300012	761	0.000037	com.gumroad
661	20299804	3151	0.000012	de.taz
662	20297678	3671	0.000010	uk.ac.nhm
663	20297472	945	0.000032	com.fiverr
664	20297324	2729	0.000014	com.verywellhealth
665	20297110	573	0.000046	com.globenewswire
666	20296832	883	0.000035	com.wikihow
667	20293998	2251	0.000015	org.ocks
668	20293488	3211	0.000011	org.iucnredlist
669	20293434	3008	0.000012	edu.uoregon
670	20293204	2658	0.000014	com.gfycat
671	20293094	3409	0.000011	org.oxfam
672	20293066	805	0.000036	int.wipo
673	20292834	2851	0.000013	com.fineartamerica
674	20292720	1501	0.000020	pl.gov
675	20292124	4536	0.000008	com.backblazeb2
676	20291708	1818	0.000017	com.jimdosite
677	20290958	1774	0.000017	com.thestar
678	20290480	3139	0.000012	org.eji
679	20290432	4260	0.000008	com.theodysseyonline
680	20289286	1533	0.000020	com.routledge
681	20286712	3550	0.000010	uk.co.timesonline
682	20285432	2900	0.000013	org.gnupg
683	20284918	2570	0.000015	com.infogram
684	20284746	1961	0.000016	uk.org.greenend
685	20284694	2269	0.000015	org.rand
686	20284380	1901	0.000016	com.surveygizmo
687	20283570	621	0.000043	br.com.uol
688	20283218	511	0.000052	org.drupal
689	20282834	3423	0.000011	org.democracynow
690	20281418	1057	0.000029	org.unicode
691	20279680	4248	0.000008	com.roche
692	20279102	4978	0.000007	re.cli
693	20278472	2959	0.000012	com.kaggle
694	20278344	1208	0.000025	cn.news
695	20278082	2182	0.000015	cc.tiny
696	20277564	3513	0.000010	org.bitcointalk
697	20276968	3180	0.000011	com.gawker
698	20275524	3459	0.000010	com.bigthink
699	20275396	1362	0.000022	com.jekyllrb
700	20274250	1735	0.000017	com.justia
701	20273300	1077	0.000028	com.css-tricks
702	20272378	2821	0.000013	com.motherjones
703	20272156	2850	0.000013	edu.nd
704	20272076	1691	0.000018	org.ourworldindata
705	20271098	1885	0.000016	ca.on.gov
706	20270166	2803	0.000013	com.timesofisrael
707	20270140	3646	0.000010	org.project-syndicate
708	20269998	509	0.000052	com.mckinsey
709	20269596	192	0.000140	com.discord
710	20269514	2553	0.000015	net.openid
711	20269466	1405	0.000021	org.amnesty
712	20269452	2842	0.000013	net.vnexpress
713	20268296	4161	0.000009	com.crayola
714	20268292	1446	0.000021	gov.uscourts
715	20267682	2029	0.000016	gov.faa
716	20267344	484	0.000055	com.onesignal
717	20266670	2380	0.000015	com.lexisnexis
718	20265730	3297	0.000011	com.nme
719	20265006	1231	0.000024	ms.aka
720	20264948	2043	0.000016	gov.usaid
721	20263632	1066	0.000028	com.pcmag
722	20263480	2976	0.000012	com.mathworks
723	20263422	2796	0.000013	uk.ac.kcl
724	20263002	2746	0.000014	fr.gouv.diplomatie
725	20262136	1854	0.000017	org.worldcat
726	20260522	553	0.000048	ca.youradchoices
727	20257736	3050	0.000012	org.csis
728	20257216	3330	0.000011	org.repec
729	20257192	2003	0.000016	de.ndr
730	20256910	1193	0.000025	com.playstation
731	20256308	3083	0.000012	ru.kp
732	20254870	3376	0.000011	no.uib
733	20254712	660	0.000041	gov.nist
734	20253674	3129	0.000012	org.ewg
735	20253570	2588	0.000014	de.web
736	20253132	901	0.000034	com.mobirise
737	20252678	3005	0.000012	au.com.businessinsider
738	20252022	3586	0.000010	org.polymer-project
739	20251846	541	0.000049	com.sxsw
740	20249958	688	0.000040	com.usnews
741	20248448	209	0.000125	com.myshopify
742	20247520	342	0.000081	mp.mailchi
743	20247494	812	0.000036	net.b-cdn
744	20246430	4236	0.000008	com.mail
745	20246388	2571	0.000015	com.sina
746	20245658	1540	0.000020	com.pastebin
747	20244724	4633	0.000008	com.mysanantonio
748	20244720	2656	0.000014	org.unctad
749	20243924	3492	0.000010	com.thejakartapost
750	20243400	1289	0.000023	org.coursera
751	20243002	1296	0.000023	com.smashingmagazine
752	20242396	3714	0.000010	io.fabric
753	20242344	164	0.000176	de.bund
754	20241670	3606	0.000010	com.shell
755	20241424	3064	0.000012	com.biography
756	20241116	3751	0.000010	com.nwsource
757	20240468	4289	0.000008	build.bazel
758	20240144	2529	0.000015	org.medrxiv
759	20236914	2914	0.000013	com.coca-colacompany
760	20236502	946	0.000032	com.shutterstock
761	20236242	949	0.000032	uk.gov.legislation
762	20235876	518	0.000052	com.herokuapp
763	20234654	629	0.000042	it.placehold
764	20234380	5770	0.000006	com.filedropper
765	20234228	4331	0.000008	org.globalnetworkinitiative
766	20233566	1598	0.000019	org.altervista
767	20233562	3298	0.000011	com.sacbee
768	20233426	2538	0.000015	org.biorxiv
769	20232540	3213	0.000011	fr.rfi
770	20232534	2974	0.000012	com.ericsson
771	20232530	4073	0.000009	com.kinja
772	20230382	991	0.000031	com.trello
773	20228230	3108	0.000012	org.oas
774	20227650	1183	0.000025	com.ycombinator
775	20226700	1810	0.000017	org.donorbox
776	20226310	2717	0.000014	com.e-monsite
777	20225712	1022	0.000030	gov.fcc
778	20225274	2099	0.000016	org.unodc
779	20224068	1159	0.000026	com.tableau
780	20223780	75	0.000419	net.cpanel
781	20222306	3580	0.000010	org.tigris
782	20222078	1366	0.000022	com.alexa
783	20221990	1248	0.000024	gov.uspto
784	20221206	3828	0.000009	com.wasabisys
785	20221088	1809	0.000017	com.speakerdeck
786	20219794	2416	0.000015	com.miamiherald
787	20219138	3771	0.000010	com.bangkokpost
788	20218368	1125	0.000027	gov.cms
789	20216750	1143	0.000026	org.reactjs
790	20216026	562	0.000047	com.gartner
791	20215672	1111	0.000027	com.jwplayer
792	20215008	2844	0.000013	edu.usf
793	20214258	2903	0.000013	com.thenation
794	20213606	2730	0.000014	com.washingtontimes
795	20213058	3355	0.000011	com.wikidot
796	20212972	960	0.000031	com.hp
797	20210904	651	0.000041	gov.sec
798	20210232	2082	0.000016	com.squarespace-cdn
799	20209450	2647	0.000014	jp.nicovideo
800	20208430	4413	0.000008	de.otto
801	20207608	2679	0.000014	ru.kremlin
802	20207088	251	0.000106	com.cloudinary
803	20206412	580	0.000046	fr.free
804	20206384	1016	0.000030	com.podbean
805	20206236	6475	0.000006	com.uberant
806	20206196	714	0.000039	org.apa
807	20204880	2626	0.000014	se.haxx
808	20204770	4090	0.000009	com.bloombergquint
809	20203792	1996	0.000016	org.khanacademy
810	20203308	1081	0.000028	com.engadget
811	20203230	3705	0.000010	com.allafrica
812	20203190	3286	0.000011	vn.com.google
813	20202746	5123	0.000007	to.gplus
814	20201740	3468	0.000010	my.com.thestar
815	20201484	3617	0.000010	uk.org.asa
816	20200948	2896	0.000013	com.simonandschuster
817	20200830	2904	0.000013	com.lowes
818	20200804	2223	0.000015	org.wto
819	20199982	207	0.000126	com.caniuse
820	20199982	224	0.000118	com.getbootstrap
821	20199966	2668	0.000014	tv.ustream
822	20199814	4118	0.000009	uk.co.spectator
823	20198870	227	0.000117	org.icann
824	20198394	653	0.000041	org.eff
825	20197522	3471	0.000010	com.sputniknews
826	20196340	3982	0.000009	com.manta
827	20195994	3509	0.000010	uk.ac.qmul
828	20195930	3378	0.000011	com.eiu
829	20195406	3185	0.000011	com.financialpost
830	20195398	2954	0.000012	uk.gov.metoffice
831	20193982	269	0.000102	com.naver
832	20193900	2240	0.000015	gov.gao
833	20193132	1114	0.000027	edu.ucla
834	20192420	1713	0.000018	fr.blogspot
835	20192342	3415	0.000011	org.heritage
836	20192052	4445	0.000008	org.scala-sbt
837	20191814	3652	0.000010	com.thenationalnews
838	20191408	4179	0.000009	com.rappler
839	20191360	4264	0.000008	com.wusa9
840	20190396	3242	0.000011	org.rferl
841	20189718	2807	0.000013	ru.kommersant
842	20189622	3814	0.000009	org.grist
843	20189254	1485	0.000020	us.imageshack
844	20188822	1391	0.000022	com.freeprivacypolicy
845	20188780	2628	0.000014	org.wbur
846	20188236	5002	0.000007	com.picsart
847	20188030	4827	0.000007	org.frontlinedefenders
848	20187700	3844	0.000009	com.newatlas
849	20185780	536	0.000050	com.wufoo
850	20184996	1401	0.000021	edu.northwestern
851	20183412	2673	0.000014	com.fivethirtyeight
852	20183252	703	0.000039	com.moz
853	20182544	2050	0.000016	to.dev
854	20181860	3799	0.000009	de.wwf
855	20181744	4266	0.000008	com.iconarchive
856	20181366	3775	0.000009	org.pri
857	20179478	943	0.000032	com.redhat
858	20178530	55	0.000603	com.dan
859	20178484	4490	0.000008	tw.blogspot
860	20177610	2876	0.000013	com.infoworld
861	20175736	664	0.000041	com.aliexpress
862	20175656	685	0.000040	com.photobucket
863	20174546	3710	0.000010	int.au
864	20172386	3613	0.000010	org.jenkins-ci
865	20171960	3807	0.000009	com.obsproject
866	20170184	2727	0.000014	com.discogs
867	20170148	4532	0.000008	com.koreaherald
868	20169684	3838	0.000009	ru.forbes
869	20169380	975	0.000031	com.stackexchange
870	20167572	3007	0.000012	com.yougov
871	20167228	3542	0.000010	ly.plot
872	20166876	3086	0.000012	org.panda
873	20166680	3400	0.000011	com.law360
874	20165754	1064	0.000028	com.emarketer
875	20163810	4549	0.000008	org.article19
876	20163770	1109	0.000027	com.merriam-webster
877	20163206	398	0.000067	com.bitly
878	20161958	4227	0.000008	com.prevention
879	20161712	6314	0.000006	org.arkive
880	20161650	1936	0.000016	com.hackerone
881	20161404	3450	0.000010	com.news24
882	20161388	3277	0.000011	com.foreignaffairs
883	20161232	4451	0.000008	fr.huffingtonpost
884	20160526	443	0.000059	com.skype
885	20158390	6653	0.000006	com.booklikes
886	20158282	913	0.000033	com.marketwatch
887	20158006	1199	0.000025	org.webkit
888	20157726	4076	0.000009	au.com.heraldsun
889	20157560	4476	0.000008	org.siggraph
890	20156950	1648	0.000018	com.newrelic
891	20156384	3884	0.000009	gov.fec
892	20155902	3850	0.000009	org.brainpickings
893	20155150	3631	0.000010	de.uni-frankfurt
894	20154848	2339	0.000015	com.w3techs
895	20154428	3267	0.000011	edu.unh
896	20154302	4387	0.000008	br.unicamp
897	20153190	58	0.000586	com.afternic
898	20152876	5742	0.000006	cc.kknews
899	20152610	1046	0.000029	com.pwc
900	20152358	4182	0.000008	com.wallethub
901	20151492	3987	0.000009	com.collinsdictionary
902	20150282	312	0.000090	com.webflow
903	20150194	4569	0.000008	org.firstmonday
904	20150166	1162	0.000026	com.appnexus
905	20149712	4371	0.000008	uk.ac.westminster
906	20148584	4770	0.000007	com.selfridges
907	20148522	3341	0.000011	com.scotsman
908	20148394	1942	0.000016	com.ssllabs
909	20147672	4169	0.000009	com.datacenterknowledge
910	20146582	2953	0.000012	com.washingtonexaminer
911	20146396	417	0.000063	com.force
912	20145716	4739	0.000007	br.ufrgs
913	20145598	2612	0.000014	ru.ria
914	20145070	6018	0.000006	com.armorgames
915	20144414	4478	0.000008	net.middleeasteye
916	20143004	3743	0.000010	com.thediplomat
917	20141930	4340	0.000008	com.the-scientist
918	20141852	3676	0.000010	gov.ornl
919	20141332	2618	0.000014	gov.energystar
920	20140318	2927	0.000013	org.wri
921	20139866	1273	0.000023	org.owasp
922	20137506	3572	0.000010	org.wilsoncenter
923	20137226	4194	0.000008	uk.co.manchestereveningnews
924	20137198	452	0.000058	gov.consumerfinance
925	20137180	1354	0.000022	com.symantec
926	20136936	877	0.000035	com.libsyn
927	20136930	1472	0.000020	com.twilio
928	20136780	1177	0.000025	com.semrush
929	20136756	5763	0.000006	net.postheaven
930	20136742	4107	0.000009	com.crashlytics
931	20136340	1608	0.000019	com.techrepublic
932	20136278	1469	0.000020	com.createjs
933	20136174	911	0.000033	edu.columbia
934	20135526	958	0.000031	com.buzzsprout
935	20135018	591	0.000045	net.azurewebsites
936	20134858	2788	0.000013	org.iucn
937	20134454	4484	0.000008	com.googledrive
938	20134176	3556	0.000010	org.sonatype
939	20134118	1804	0.000017	ly.ow
940	20134104	4522	0.000008	io.meduza
941	20133994	1559	0.000019	net.msecnd
942	20133972	1417	0.000021	com.weather
943	20132796	1375	0.000022	com.rollingstone
944	20132550	4167	0.000009	ru.aif
945	20132488	1649	0.000018	com.upwork
946	20132078	1832	0.000017	com.chrome
947	20131418	445	0.000059	com.dmca
948	20130938	4097	0.000009	org.avaaz
949	20129642	5334	0.000007	cn.edu.sdu
950	20128590	2574	0.000015	ru.rbc
951	20128526	661	0.000041	com.figma
952	20127484	3391	0.000011	nl.rug
953	20126540	5210	0.000007	org.sourcewatch
954	20125866	5148	0.000007	com.wsoctv
955	20125546	5149	0.000007	com.linodeobjects
956	20125462	2724	0.000014	int.reliefweb
957	20124860	3130	0.000012	org.cfr
958	20124542	2859	0.000013	com.springeropen
959	20123970	335	0.000083	com.wistia
960	20122198	990	0.000031	org.json
961	20121982	5840	0.000006	com.grabcad
962	20121118	3551	0.000010	ru.vedomosti
963	20120884	5505	0.000006	org.sfpl
964	20120188	4637	0.000008	ch.qos
965	20119716	3693	0.000010	org.escholarship
966	20119148	3977	0.000009	uk.ac.sussex
967	20118982	344	0.000081	com.automattic
968	20118778	4131	0.000009	com.gannett-cdn
969	20117346	4052	0.000009	edu.scu
970	20117274	4068	0.000009	org.nationalinterest
971	20117084	3510	0.000010	com.tradingeconomics
972	20117052	3702	0.000010	org.thinkprogress
973	20116964	4240	0.000008	com.dawn
974	20116284	4166	0.000009	cc.taplink
975	20115178	3106	0.000012	ca.citizenlab
976	20114828	2652	0.000014	com.bankrate
977	20114682	2030	0.000016	com.tutsplus
978	20114462	1412	0.000021	org.golang
979	20113832	5706	0.000006	com.london2012
980	20113636	2033	0.000016	org.linuxfoundation
981	20113280	1840	0.000017	edu.rutgers
982	20113048	3179	0.000011	org.undocs
983	20112672	4864	0.000007	za.co.dailymaverick
984	20112628	3198	0.000011	com.springernature
985	20112552	3740	0.000010	au.edu.adelaide
986	20111822	4665	0.000008	com.mnn
987	20110280	4312	0.000008	ae.google
988	20110274	3089	0.000012	org.crossref
989	20110262	3545	0.000010	com.vox-cdn
990	20110256	4108	0.000009	com.dailykos
991	20109548	4882	0.000007	uk.ac.lancs
992	20109472	863	0.000036	org.ieee
993	20108050	729	0.000038	ca.canada
994	20106848	3444	0.000010	org.cato
995	20106594	3772	0.000009	gov.ustr
996	20106562	469	0.000056	com.indeed
997	20106554	3479	0.000010	com.cityam
998	20106172	4193	0.000008	de.ebay-kleinanzeigen
999	20105804	1019	0.000030	com.techtarget
1000	20104494	669	0.000040	gov.copyright

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

August 2022 crawl archive now available

August 22, 2022Sebastian Nagel

The crawl archive for August 2022 is now available! The data was crawled August 7 – 20 and contains 2.55 billion web pages or 295 TiB of uncompressed content. Page captures are from 46 million hosts or 37 million registered domains and include 1.3 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The August crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2022-33/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2022-33/segment.paths.gz	100
WARC files	CC-MAIN-2022-33/warc.paths.gz	80000	68.95
WAT files	CC-MAIN-2022-33/wat.paths.gz	80000	17.06
WET files	CC-MAIN-2022-33/wet.paths.gz	80000	7.24
Robots.txt files	CC-MAIN-2022-33/robotstxt.paths.gz	80000	0.15
Non-200 responses files	CC-MAIN-2022-33/non200responses.paths.gz	80000	2.92
URL index files	CC-MAIN-2022-33/cc-index.paths.gz	302	0.2
Columnar URL index files	CC-MAIN-2022-33/cc-index-table.paths.gz	900	0.23

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2022-33/. Also the columnar index has been updated to contain this crawl.

June/July 2022 crawl archive now available

July 13, 2022Sebastian Nagel

The crawl archive for June/July 2022 is now available! The data was crawled June 24 – July 7 and contains 3.1 billion web pages or 370 TiB of uncompressed content. Page captures are from 44 million hosts or 35 million registered domains and include 1.4 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The June/July crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2022-27/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2022-27/segment.paths.gz	100
WARC files	CC-MAIN-2022-27/warc.paths.gz	80000	84.08
WAT files	CC-MAIN-2022-27/wat.paths.gz	80000	20.99
WET files	CC-MAIN-2022-27/wet.paths.gz	80000	8.97
Robots.txt files	CC-MAIN-2022-27/robotstxt.paths.gz	80000	0.15
Non-200 responses files	CC-MAIN-2022-27/non200responses.paths.gz	80000	1.92
URL index files	CC-MAIN-2022-27/cc-index.paths.gz	302	0.23
Columnar URL index files	CC-MAIN-2022-27/cc-index-table.paths.gz	900	0.26

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2022-27/. Also the columnar index has been updated to contain this crawl.

May 2022 crawl archive now available

June 2, 2022Sebastian Nagel

The crawl archive for May 2022 is now available! The data was crawled May 16 – 29 and contains 3.45 billion web pages or 420 TiB of uncompressed content. Page captures are from 45 million hosts or 36 million registered domains and include 1.4 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The May crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2022-21/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2022-21/segment.paths.gz	100
WARC files	CC-MAIN-2022-21/warc.paths.gz	80000	92.81
WAT files	CC-MAIN-2022-21/wat.paths.gz	80000	23.30
WET files	CC-MAIN-2022-21/wet.paths.gz	80000	9.80
Robots.txt files	CC-MAIN-2022-21/robotstxt.paths.gz	80000	0.15
Non-200 responses files	CC-MAIN-2022-21/non200responses.paths.gz	80000	2.26
URL index files	CC-MAIN-2022-21/cc-index.paths.gz	302	0.25
Columnar URL index files	CC-MAIN-2022-21/cc-index-table.paths.gz	900	0.28

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2022-21/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs October, November/December 2021 and January 2022

March 16, 2022Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of October, November/December 2021 and January 2022. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases. You may also visit the projects cc-webgraph and cc-pyspark which include all scripts and tools required to construct the graphs. Instructions to explore the graphs in the webgraph format are given in our collection of webgraph notebooks.

Host-level graph

The graph consists of 384 million nodes and 2.47 billion edges. Both hyperlinks and HTTP redirects and link headers are used as edges to span up the graph. All types of links are included, including pure “technical” ones pointing to images, JavaScript libraries, web fonts, etc. However, only host names with a valid IANA TLD are used. Consequently, URLs with an IP address as host component are not taken into account for building the host-level graph.

There are 326 million dangling nodes (84.6%) and the largest strongly connected component contains 45.2 million (11.7%) nodes. Dangling nodes stem from

hosts that have not been crawled, yet are pointed to from a link on a crawled page
hosts without any links pointing to a different host name
or hosts which did only return an error page (eg. HTTP 404)

Host names in the graph are in reverse domain name notation and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 384 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2021-22-oct-nov-jan/host/ (this requires an account on AWS). Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2021-22-oct-nov-jan/host/ as prefix to access the files from everywhere.

Download files of the Common Crawl Oct/Nov/Jan 2021-2022 host-level webgraph

Size	File	Description
2.66 GB	cc-main-2021-22-oct-nov-jan-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 32 vertices files
11.76 GB	cc-main-2021-22-oct-nov-jan-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 64 edges files

5.32 GB	cc-main-2021-22-oct-nov-jan-host.graph	graph in BVGraph format
2 kB	cc-main-2021-22-oct-nov-jan-host.properties

5.78 GB	cc-main-2021-22-oct-nov-jan-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2021-22-oct-nov-jan-host-t.properties

1 kB	cc-main-2021-22-oct-nov-jan-host.stats	WebGraph statistics

6.38 GB	cc-main-2021-22-oct-nov-jan-host-ranks.txt.gz	harmonic centrality and pagerank

Domain-level graph

The domain graph is built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org. Version (commit) 68b67d3 of the public suffix list was used (commit date 2022-03-04).

The domain-level graph has 90 million nodes and 1.55 billion edges. 50% or 45 million nodes are dangling nodes, the largest strongly connected component covers 36 million or 40% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2021-22-oct-nov-jan/domain/ or on https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2021-22-oct-nov-jan/domain/.

Download files of the Common Crawl Oct/Nov/Jan 2021-2022 domain-level webgraph

Size	File	Description
0.62 GB	cc-main-2021-22-oct-nov-jan-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
6.36 GB	cc-main-2021-22-oct-nov-jan-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

3.65 GB	cc-main-2021-22-oct-nov-jan-domain.graph	graph in BVGraph format
2 kB	cc-main-2021-22-oct-nov-jan-domain.properties

3.53 GB	cc-main-2021-22-oct-nov-jan-domain-t.graph	transpose of the graph
2 kB	cc-main-2021-22-oct-nov-jan-domain-t.properties

1 kB	cc-main-2021-22-oct-nov-jan-domain.stats	WebGraph statistics

1.93 GB	cc-main-2021-22-oct-nov-jan-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 90 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Oct/Nov/Jan 2020-2021)

harmonic centrality rank	hc value	page rank	page rank value	reversed domain name
1	31448418	1	0.017921	com.googleapis
2	30105086	3	0.013006	com.facebook
3	28991726	2	0.014028	com.google
4	26233886	6	0.007154	org.w
5	26230152	4	0.008081	com.twitter
6	25864994	8	0.006261	com.youtube
7	25325250	7	0.006558	com.instagram
8	24665160	5	0.007716	com.googletagmanager
9	24097224	9	0.004716	org.gmpg
10	23219788	11	0.003948	com.gstatic
11	23145196	13	0.003418	com.linkedin
12	22256010	10	0.004364	com.cloudflare
13	21953630	17	0.001942	com.gravatar
14	21841462	21	0.001594	com.pinterest
15	21786278	14	0.003223	org.wordpress
16	21441268	25	0.001417	org.wikipedia
17	21311496	16	0.002057	com.apple
18	21176702	33	0.001077	com.wordpress
19	21045484	32	0.001141	com.vimeo
20	21032900	15	0.002070	com.bootstrapcdn
21	20995456	41	0.000913	be.youtu
22	20796928	18	0.001658	com.jquery
23	20726342	28	0.001191	com.microsoft
24	20673758	45	0.000789	com.blogspot
25	20634464	23	0.001451	io.polyfill
26	20617114	47	0.000775	gl.goo
27	20589400	49	0.000736	com.amazon
28	20486426	64	0.000550	ly.bit
29	20476754	29	0.001178	com.wixstatic
30	20455806	50	0.000729	com.wp
31	20452992	22	0.001551	net.cloudfront
32	20445298	40	0.000962	com.amazonaws
33	20410144	43	0.000863	org.mozilla
34	20403210	31	0.001162	net.jsdelivr
35	20395958	51	0.000721	eu.europa
36	20389728	37	0.000994	com.google-analytics
37	20343648	20	0.001595	com.fontawesome
38	20336692	91	0.000384	com.tumblr
39	20333242	19	0.001648	com.adobe
40	20324398	24	0.001421	com.github
41	20181488	75	0.000480	com.googleusercontent
42	20150384	55	0.000684	com.flickr
43	20122758	104	0.000326	com.yahoo
44	20112952	57	0.000670	com.paypal
45	20106820	48	0.000752	io.github
46	20102750	106	0.000314	com.reddit
47	20060046	117	0.000268	com.soundcloud
48	20049174	38	0.000966	com.googlesyndication
49	20043166	81	0.000425	com.medium
50	20007290	53	0.000703	org.w3
51	19994710	127	0.000231	com.nytimes
52	19974404	62	0.000611	co.t
53	19956144	102	0.000338	com.weebly
54	19955606	114	0.000277	com.spotify
55	19925440	58	0.000656	com.whatsapp
56	19906786	34	0.001038	ru.yandex
57	19901846	85	0.000401	org.creativecommons
58	19894214	136	0.000208	org.archive
59	19870846	183	0.000139	com.cnn
60	19863762	63	0.000608	org.schema
61	19855684	60	0.000645	com.addthis
62	19851344	146	0.000194	com.forbes
63	19836582	203	0.000127	uk.co.bbc
64	19834544	70	0.000513	com.shopify
65	19811758	234	0.000113	com.washingtonpost
66	19808664	69	0.000523	com.vk
67	19804862	185	0.000138	com.bing
68	19803622	147	0.000193	gov.cdc
69	19802914	157	0.000172	int.who
70	19798680	92	0.000383	me.wp
71	19780956	44	0.000809	net.doubleclick
72	19760738	141	0.000201	gov.nih
73	19749646	59	0.000649	com.macromedia
74	19748198	71	0.000506	com.unpkg
75	19747664	260	0.000103	net.researchgate
76	19734110	252	0.000106	com.wsj
77	19733152	293	0.000093	edu.stanford
78	19731674	232	0.000115	com.imdb
79	19724760	188	0.000134	org.wikimedia
80	19712934	39	0.000965	net.fbcdn
81	19700958	206	0.000124	com.businessinsider
82	19690114	143	0.000200	com.dropbox
83	19687820	284	0.000095	edu.mit
84	19683060	100	0.000363	com.list-manage
85	19682338	308	0.000089	com.tinyurl
86	19665976	27	0.001204	org.apache
87	19657446	163	0.000157	com.theguardian
88	19651458	299	0.000092	com.android
89	19647768	445	0.000067	com.quora
90	19643018	165	0.000156	org.doi
91	19636594	328	0.000085	com.go
92	19629826	240	0.000109	com.bloomberg
93	19629762	274	0.000098	edu.harvard
94	19627444	501	0.000060	com.msn
95	19625870	164	0.000157	com.issuu
96	19625646	254	0.000106	com.oracle
97	19624678	344	0.000083	com.springer
98	19621946	149	0.000190	com.wixsite
99	19615182	140	0.000203	us.zoom
100	19612484	131	0.000220	com.npmjs
101	19608812	111	0.000294	me.t
102	19608314	334	0.000084	com.slack
103	19602354	124	0.000241	com.mailchimp
104	19596192	244	0.000108	com.stackoverflow
105	19592312	303	0.000091	com.reuters
106	19587800	321	0.000087	com.techcrunch
107	19586192	505	0.000059	com.myspace
108	19586126	241	0.000109	com.twimg
109	19584238	166	0.000155	com.giphy
110	19583908	292	0.000093	com.example
111	19577384	52	0.000709	com.fb
112	19576726	167	0.000153	com.yelp
113	19576020	169	0.000151	com.office
114	19572502	341	0.000083	com.prnewswire
115	19567966	190	0.000133	com.unsplash
116	19564492	108	0.000309	de.google
117	19561016	300	0.000091	com.wiley
118	19555072	46	0.000785	net.facebook
119	19552510	268	0.000101	org.un
120	19549098	256	0.000105	com.sciencedirect
121	19548162	489	0.000061	com.latimes
122	19547508	594	0.000050	com.livejournal
123	19547452	145	0.000196	gle.forms
124	19544300	466	0.000063	uk.co.telegraph
125	19543690	402	0.000078	com.nature
126	19542384	340	0.000083	org.npr
127	19541420	484	0.000062	com.ted
128	19535332	514	0.000057	edu.berkeley
129	19533106	647	0.000046	com.vice
130	19531328	237	0.000110	org.gnu
131	19529798	198	0.000130	org.ietf
132	19527852	714	0.000042	uk.ac.cam
133	19524938	434	0.000071	com.time
134	19522382	289	0.000094	com.bbc
135	19517460	497	0.000060	com.goodreads
136	19513616	412	0.000076	org.arxiv
137	19510034	306	0.000090	com.cnbc
138	19508184	144	0.000197	com.ytimg
139	19492206	700	0.000043	edu.columbia
140	19483642	433	0.000071	com.sagepub
141	19475908	153	0.000186	com.ft
142	19474592	173	0.000149	org.acm
143	19473316	323	0.000086	com.githubusercontent
144	19473132	438	0.000069	com.cnet
145	19468042	120	0.000254	com.youtube-nocookie
146	19458540	409	0.000077	com.wired
147	19455634	205	0.000125	com.imgur
148	19454210	518	0.000057	uk.co.dailymail
149	19449628	202	0.000127	com.blogger
150	19448116	79	0.000459	com.godaddy
151	19443158	26	0.001330	cn.gov.miit
152	19437758	343	0.000083	com.theverge
153	19435568	509	0.000058	edu.yale
154	19434236	171	0.000150	org.ampproject
155	19426446	472	0.000063	com.nationalgeographic
156	19423252	281	0.000096	com.squarespace
157	19419880	804	0.000037	org.chromium
158	19416530	696	0.000043	uk.ac.ox
159	19413138	488	0.000061	com.googleblog
160	19407848	458	0.000064	gov.whitehouse
161	19407614	304	0.000090	com.usatoday
162	19407228	499	0.000060	com.staticflickr
163	19407066	829	0.000036	com.evernote
164	19404734	192	0.000131	com.hubspot
165	19404414	476	0.000062	org.ieee
166	19402398	587	0.000051	org.worldbank
167	19402004	407	0.000077	com.dribbble
168	19400214	132	0.000219	com.statcounter
169	19399796	512	0.000058	ee.linktr
170	19399358	527	0.000056	edu.cornell
171	19395500	123	0.000243	com.sharethis
172	19388246	519	0.000057	com.theatlantic
173	19387454	408	0.000077	com.docker
174	19385182	673	0.000044	com.git-scm
175	19383304	214	0.000122	com.wpengine
176	19383030	797	0.000037	org.sciencemag
177	19381750	811	0.000037	com.arstechnica
178	19381474	275	0.000098	gg.discord
179	19381222	159	0.000167	com.zendesk
180	19379042	264	0.000101	uk.co.google
181	19376664	138	0.000206	me.line
182	19373526	246	0.000107	uk.co.amazon
183	19373448	602	0.000050	com.zdnet
184	19372154	239	0.000109	net.slideshare
185	19369046	325	0.000086	com.appspot
186	19368866	702	0.000042	com.economist
187	19368620	737	0.000041	org.cambridge
188	19368112	571	0.000052	com.cisco
189	19367144	792	0.000038	edu.washington
190	19366344	648	0.000046	org.weforum
191	19361760	668	0.000044	com.box
192	19361692	624	0.000047	org.pbs
193	19356220	479	0.000062	org.python
194	19355604	436	0.000070	com.huffingtonpost
195	19354284	226	0.000117	com.outlook
196	19353112	542	0.000055	com.typepad
197	19350044	288	0.000095	org.pewresearch
198	19345610	558	0.000054	com.cbsnews
199	19342774	267	0.000101	net.windows
200	19334638	556	0.000054	com.deloitte
201	19333688	1058	0.000028	com.rollingstone
202	19333260	496	0.000060	com.pixabay
203	19333260	554	0.000054	gov.usda
204	19332390	720	0.000042	google.blog
205	19331286	572	0.000052	site.business
206	19330802	544	0.000055	uk.co.independent
207	19325162	1223	0.000025	ly.cutt
208	19323700	42	0.000906	com.qq
209	19323606	760	0.000039	com.apnews
210	19323302	745	0.000040	ca.cbc
211	19322838	639	0.000046	org.unesco
212	19322434	493	0.000061	com.gitlab
213	19307948	704	0.000042	com.mysql
214	19307400	611	0.000049	com.pexels
215	19302186	586	0.000051	gov.loc
216	19297622	728	0.000041	edu.upenn
217	19296262	713	0.000042	edu.wisc
218	19295886	485	0.000062	com.getpocket
219	19295020	553	0.000054	com.nbcnews
220	19294010	451	0.000065	com.fastcompany
221	19292592	1381	0.000022	com.ikea
222	19291294	178	0.000143	com.tripadvisor
223	19286758	1171	0.000026	org.eclipse
224	19285934	676	0.000043	com.scribd
225	19285000	768	0.000039	com.shutterstock
226	19284178	634	0.000047	com.mozilla
227	19282738	1225	0.000025	org.kernel
228	19277868	798	0.000037	uk.co.blogspot
229	19277480	784	0.000038	com.qz
230	19272246	859	0.000034	com.ggpht
231	19271988	318	0.000087	com.live
232	19271080	979	0.000030	uk.co.guardian
233	19268934	301	0.000091	com.w3schools
234	19265398	1924	0.000018	com.lego
235	19264382	498	0.000060	gov.irs
236	19262148	885	0.000034	edu.jhu
237	19260006	672	0.000044	com.buzzfeed
238	19259708	674	0.000044	uk.co.eventbrite
239	19259366	796	0.000038	com.trello
240	19258458	1181	0.000025	com.technologyreview
241	19256192	984	0.000030	com.playstation
242	19255014	1057	0.000028	fr.lemonde
243	19251458	552	0.000054	com.squareup
244	19251266	550	0.000054	com.fortune
245	19249814	507	0.000059	gov.nasa
246	19249644	751	0.000040	me.about
247	19244186	596	0.000050	com.oup
248	19243256	283	0.000095	net.behance
249	19241750	973	0.000031	com.foursquare
250	19240364	2212	0.000017	com.hbo
251	19238806	635	0.000047	fm.anchor
252	19237212	327	0.000085	com.disqus
253	19235880	915	0.000033	com.slate
254	19234328	54	0.000697	co.g
255	19232270	36	0.000996	com.baidu
256	19231094	456	0.000065	com.bigcommerce
257	19229430	175	0.000146	jp.co.google
258	19229270	249	0.000107	com.calendly
259	19229228	756	0.000040	com.vox
260	19227564	557	0.000054	com.dailymotion
261	19226480	654	0.000045	com.investopedia
262	19225524	860	0.000034	com.ubuntu
263	19225146	272	0.000099	com.bandcamp
264	19224258	1564	0.000021	com.hatenablog
265	19222446	1087	0.000027	co.elastic
266	19221444	724	0.000042	com.newyorker
267	19221360	854	0.000035	com.about
268	19220762	460	0.000064	com.arcgis
269	19219440	814	0.000037	com.variety
270	19219024	781	0.000038	au.net.abc
271	19217228	551	0.000054	com.elpais
272	19215612	822	0.000036	edu.ucla
273	19214942	802	0.000037	gov.congress
274	19214422	677	0.000043	org.apa
275	19212402	740	0.000041	com.freepik
276	19211346	1126	0.000026	com.steamcommunity
277	19210898	250	0.000107	gov.ca
278	19209850	773	0.000039	org.pypi
279	19209526	719	0.000042	com.libsyn
280	19208308	865	0.000034	edu.princeton
281	19207924	142	0.000200	com.opera
282	19207284	940	0.000032	com.nypost
283	19205296	813	0.000037	edu.umich
284	19204192	1339	0.000023	com.billboard
285	19204056	312	0.000088	com.typeform
286	19201904	280	0.000097	com.feedburner
287	19200550	871	0.000034	com.ssrn
288	19199714	546	0.000055	com.tandfonline
289	19197908	905	0.000033	com.podbean
290	19197754	224	0.000117	page.g
291	19197604	827	0.000036	org.fao
292	19197538	868	0.000034	com.foxnews
293	19197126	964	0.000031	com.merriam-webster
294	19196242	1093	0.000027	edu.purdue
295	19190330	1579	0.000021	ca.ubc
296	19186772	710	0.000042	org.bitbucket
297	19186418	101	0.000349	com.wix
298	19183972	1075	0.000028	org.owasp
299	19182372	273	0.000099	com.ibm
300	19180986	955	0.000031	com.newsweek
301	19178290	688	0.000043	org.semver
302	19177950	251	0.000106	org.bbb
303	19170374	1403	0.000022	ca.sfu
304	19169910	2155	0.000017	com.discovery
305	19169706	1454	0.000022	uk.co.metro
306	19168948	403	0.000078	org.openstreetmap
307	19168128	928	0.000032	com.webs
308	19164224	277	0.000098	com.eepurl
309	19163312	399	0.000079	com.netdna-ssl
310	19162670	310	0.000089	com.wistia
311	19160768	1301	0.000023	app.netlify
312	19160242	910	0.000033	com.nasdaq
313	19158884	637	0.000047	gov.senate
314	19158734	135	0.000212	com.filesusr
315	19156792	528	0.000056	com.snapchat
316	19156732	317	0.000088	tv.twitch
317	19155472	1352	0.000023	uk.co.standard
318	19153634	977	0.000030	com.uk
319	19150454	669	0.000044	org.eff
320	19148706	1433	0.000022	io.gitlab
321	19146762	1609	0.000021	com.warnerbros
322	19145448	1025	0.000029	com.techradar
323	19144618	1173	0.000026	com.500px
324	19143630	1158	0.000026	com.pastebin
325	19141644	583	0.000051	gov.epa
326	19140574	820	0.000036	com.theconversation
327	19139294	1147	0.000026	org.semanticscholar
328	19139130	125	0.000238	com.rawgit
329	19138842	1228	0.000025	com.sky
330	19136534	1747	0.000019	com.flipboard
331	19134802	454	0.000065	com.ebay
332	19133816	204	0.000125	com.amazon-adsystem
333	19133028	599	0.000050	edu.cmu
334	19132926	1282	0.000024	edu.illinois
335	19132710	1327	0.000023	org.greenpeace
336	19131792	429	0.000073	com.optimizely
337	19130562	1668	0.000020	com.urbandictionary
338	19130312	201	0.000127	org.iana
339	19129546	609	0.000049	gov.house
340	19129270	98	0.000373	com.stripe
341	19127004	463	0.000064	org.opensource
342	19125378	247	0.000107	com.cloudinary
343	19122456	903	0.000033	edu.academia
344	19120070	1182	0.000025	org.mitre
345	19119992	1008	0.000030	gov.usgs
346	19119044	215	0.000122	net.sourceforge
347	19118924	2117	0.000018	com.channel4
348	19118136	1489	0.000022	uk.co.thesun
349	19117516	1558	0.000021	com.deadline
350	19114610	996	0.000030	com.thehill
351	19113576	897	0.000033	edu.umn
352	19113226	758	0.000040	gov.justice
353	19110186	1643	0.000020	org.maven
354	19110016	156	0.000173	com.addtoany
355	19108184	404	0.000077	com.criteo
356	19103220	2356	0.000016	com.freep
357	19101368	128	0.000230	com.paypalobjects
358	19098770	1133	0.000026	com.nikkei
359	19098758	457	0.000065	es.google
360	19096448	678	0.000043	org.oecd
361	19093328	1291	0.000024	org.postgresql
362	19092214	1941	0.000018	com.euronews
363	19091722	772	0.000039	gov.archives
364	19091262	1584	0.000021	com.reverbnation
365	19090376	1138	0.000026	uk.co.mirror
366	19088776	564	0.000053	com.kickstarter
367	19087460	2766	0.000013	edu.byu
368	19087070	1338	0.000023	edu.hbs
369	19085414	1382	0.000022	com.googlesource
370	19084726	2116	0.000018	edu.wustl
371	19084458	1010	0.000030	com.politico
372	19083222	2226	0.000017	org.nobelprize
373	19082584	1111	0.000027	com.dw
374	19082284	1022	0.000029	com.pingdom
375	19082194	548	0.000054	com.walmart
376	19078996	84	0.000405	net.jsfiddle
377	19078568	1284	0.000024	ch.ethz
378	19073766	2299	0.000016	gov.cia
379	19073478	983	0.000030	com.salon
380	19073042	715	0.000042	org.change
381	19072810	1184	0.000025	com.theglobeandmail
382	19071472	462	0.000064	com.elsevier
383	19071222	1837	0.000019	com.storify
384	19066438	137	0.000207	de.bund
385	19066384	121	0.000250	com.jimdo
386	19066344	1429	0.000022	edu.gatech
387	19064084	67	0.000527	net.typekit
388	19062738	1310	0.000023	com.digitaltrends
389	19062478	1286	0.000024	int.unfccc
390	19061312	810	0.000037	au.com.google
391	19059746	1026	0.000029	gov.treasury
392	19059514	2283	0.000016	com.mystrikingly
393	19059110	906	0.000033	com.britannica
394	19058170	1272	0.000024	edu.ucdavis
395	19057558	904	0.000033	uk.parliament
396	19056260	76	0.000468	me.fb
397	19054320	1027	0.000029	com.mdpi
398	19052746	1244	0.000024	com.aljazeera
399	19052144	207	0.000124	com.etsy
400	19052062	269	0.000100	net.azureedge
401	19051812	1063	0.000028	gov.fbi
402	19051720	1608	0.000021	ms.1drv
403	19048836	824	0.000036	com.bmj
404	19048654	1325	0.000023	de.mpg
405	19047424	2538	0.000014	com.virustotal
406	19047122	1106	0.000027	org.nejm
407	19046336	211	0.000123	com.tiktok
408	19045754	194	0.000131	org.nodejs
409	19044892	2566	0.000014	com.diigo
410	19043614	1183	0.000025	com.scmp
411	19042800	1095	0.000027	au.com.smh
412	19042258	718	0.000042	org.d3js
413	19041804	1311	0.000023	com.history
414	19041506	1178	0.000026	org.hrw
415	19039754	1235	0.000025	uk.ac.ucl
416	19038356	1213	0.000025	com.socialmediatoday
417	19035776	1038	0.000029	edu.uchicago
418	19033774	2710	0.000013	com.thecvf
419	19030356	968	0.000031	org.readthedocs
420	19030118	61	0.000612	com.googleadservices
421	19029814	1018	0.000029	org.jstor
422	19029070	833	0.000035	com.pinimg
423	19028696	2486	0.000015	com.oxforddictionaries
424	19028410	2145	0.000017	com.discogs
425	19027476	2568	0.000014	edu.buffalo
426	19023352	1810	0.000019	com.buzzfeednews
427	19023196	957	0.000031	watch.fb
428	19022492	1136	0.000026	org.sphinx-doc
429	19022490	1851	0.000018	com.spreaker
430	19022078	1548	0.000021	com.irishtimes
431	19021850	608	0.000049	com.biomedcentral
432	19019438	1452	0.000022	uk.ac.lse
433	19018308	469	0.000063	org.hbr
434	19018106	396	0.000079	com.statista
435	19017386	1045	0.000029	com.substack
436	19014182	285	0.000095	ru.ok
437	19013374	3509	0.000010	com.quizlet
438	19012470	807	0.000037	com.deviantart
439	19010602	1115	0.000027	org.undp
440	19005532	1959	0.000018	com.rt
441	19004652	1004	0.000030	org.ilo
442	19004416	3352	0.000011	cc.uxdesign
443	19003306	2234	0.000017	org.wto
444	19002690	1673	0.000020	org.rfc-editor
445	19002034	1302	0.000023	com.penguinrandomhouse
446	19001536	920	0.000032	de.spiegel
447	18999940	1318	0.000023	com.producthunt
448	18997582	705	0.000042	gov.sec
449	18997128	520	0.000057	com.meetup
450	18994718	2162	0.000017	com.ibtimes
451	18993898	1061	0.000028	com.sun
452	18992788	2120	0.000018	gov.federalreserve
453	18991624	1760	0.000019	edu.arizona
454	18990732	600	0.000050	edu.utah
455	18990642	1861	0.000018	com.newscientist
456	18989730	537	0.000055	com.gmail
457	18989410	1123	0.000026	net.java
458	18986644	1885	0.000018	com.itv
459	18986450	504	0.000059	com.ssl-images-amazon
460	18986010	243	0.000108	uk.org.ico
461	18985542	1781	0.000019	ca.blogspot
462	18985454	73	0.000503	net.akamaihd
463	18985010	840	0.000035	in.co.google
464	18984802	1391	0.000022	de.zeit
465	18984480	1005	0.000030	uk.co.thetimes
466	18984458	1042	0.000029	com.prweb
467	18983876	2661	0.000013	com.twitpic
468	18983790	1195	0.000025	io.pypa
469	18982648	2956	0.000012	com.openai
470	18980494	447	0.000067	net.imgix
471	18979850	1901	0.000018	com.martinfowler
472	18979062	242	0.000108	org.purl
473	18978596	655	0.000045	de.gesetze-im-internet
474	18978450	397	0.000079	net.themeforest
475	18978408	216	0.000121	jp.co.yahoo
476	18977992	1409	0.000022	edu.ufl
477	18977350	425	0.000073	com.atlassian
478	18975784	1373	0.000023	edu.duke
479	18974486	236	0.000111	to.amzn
480	18974130	2285	0.000016	edu.gmu
481	18974004	1056	0.000028	edu.nyu
482	18973414	575	0.000052	org.debian
483	18973148	1249	0.000024	com.jetbrains
484	18973108	235	0.000111	com.mapbox
485	18973012	296	0.000092	me.telegram
486	18972556	1287	0.000024	com.wikia
487	18972192	82	0.000409	com.oversightboard
488	18971130	342	0.000083	com.proofpoint
489	18970830	994	0.000030	com.jimdofree
490	18970782	2638	0.000014	org.nypl
491	18968834	806	0.000037	edu.brookings
492	18968470	2213	0.000017	org.wfp
493	18968366	2417	0.000015	mp.j
494	18967966	2624	0.000014	app.web
495	18967448	2363	0.000015	com.instructables
496	18966950	1267	0.000024	org.imf
497	18965952	1134	0.000026	org.unhcr
498	18965650	1610	0.000021	edu.virginia
499	18965586	3000	0.000012	ph.telegra
500	18963358	2122	0.000018	org.propublica
501	18963068	1655	0.000020	edu.brown
502	18961020	1430	0.000022	com.seattletimes
503	18960454	313	0.000088	io.shields
504	18960174	2348	0.000016	org.archlinux
505	18959730	355	0.000081	com.surveymonkey
506	18958892	747	0.000040	gov.state
507	18956864	1019	0.000029	com.yarnpkg
508	18956706	2142	0.000017	org.phys
509	18956672	1688	0.000020	org.unwomen
510	18955200	809	0.000037	com.fiverr
511	18954884	2930	0.000012	org.vim
512	18954634	3447	0.000010	com.instapaper
513	18953814	221	0.000119	com.eventbrite
514	18953136	878	0.000034	edu.psu
515	18948514	1599	0.000021	com.asahi
516	18948206	2338	0.000016	ca.ualberta
517	18947706	3130	0.000011	com.rd
518	18947260	748	0.000040	com.intel
519	18947088	2414	0.000015	com.gfycat
520	18946976	2649	0.000013	org.icrc
521	18944700	2467	0.000015	org.biorxiv
522	18941214	1658	0.000020	org.r-project
523	18939874	257	0.000105	com.aliyuncs
524	18936846	139	0.000205	com.weibo
525	18936474	1852	0.000018	com.gettyimages
526	18931798	525	0.000057	com.googlecode
527	18930224	3699	0.000010	com.plurk
528	18929320	2136	0.000017	org.unep
529	18929114	1737	0.000019	com.howstuffworks
530	18928922	2617	0.000014	com.udacity
531	18927534	1830	0.000019	edu.georgetown
532	18926738	2327	0.000016	com.esri
533	18925612	685	0.000043	uk.gov.service
534	18924274	2742	0.000013	jp.co.japantimes
535	18924242	2406	0.000015	com.kobo
536	18921598	555	0.000054	com.samsung
537	18921494	1447	0.000022	fr.gouvernement
538	18920518	2821	0.000013	org.wikibooks
539	18919550	2454	0.000015	it.scoop
540	18917542	3441	0.000010	net.openreview
541	18916950	2137	0.000017	es.abc
542	18913684	2422	0.000015	jp.geocities
543	18913494	2748	0.000013	edu.uoregon
544	18912900	2119	0.000018	google.ai
545	18909392	2655	0.000013	co.carrd
546	18907616	2115	0.000018	uk.co.huffingtonpost
547	18904404	690	0.000043	com.mashable
548	18903570	604	0.000049	com.steampowered
549	18902828	1730	0.000020	org.torproject
550	18902570	449	0.000066	com.netflix
551	18901684	2215	0.000017	google.research
552	18900478	2699	0.000013	at.ac.univie
553	18899428	2143	0.000017	edu.tufts
554	18897774	913	0.000033	com.thelancet
555	18894916	3791	0.000009	goog.translate
556	18893828	982	0.000030	org.ohchr
557	18892464	3880	0.000009	com.bravesites
558	18890432	2887	0.000012	org.rsf
559	18890342	1792	0.000019	gov.usembassy
560	18886662	3016	0.000012	com.architecturaldigest
561	18886098	1103	0.000027	cn.news
562	18885566	2367	0.000015	uk.bl
563	18883186	3407	0.000010	uk.co.walesonline
564	18882344	2461	0.000015	org.accessnow
565	18880856	2291	0.000016	com.france24
566	18879976	3309	0.000011	com.pearltrees
567	18878902	2815	0.000013	org.freedomhouse
568	18875528	219	0.000120	com.salesforce
569	18874868	2575	0.000014	org.scala-lang
570	18874266	1142	0.000026	be.google
571	18874220	2684	0.000013	re.appsto
572	18873246	2400	0.000015	org.ap
573	18873008	3183	0.000011	do.bit
574	18872314	2833	0.000012	com.sputniknews
575	18871926	2165	0.000017	org.americanprogress
576	18871186	1568	0.000021	com.chron
577	18871072	2792	0.000013	org.unaids
578	18869902	2689	0.000013	com.ajc
579	18868558	2251	0.000016	app.vercel
580	18868076	495	0.000060	com.visualstudio
581	18866746	1333	0.000023	net.daringfireball
582	18865464	2654	0.000013	org.csis
583	18863760	2350	0.000016	com.ew
584	18862092	1293	0.000024	link.page
585	18861286	2320	0.000016	fr.gouv.diplomatie
586	18860790	189	0.000133	ru.mail
587	18860636	852	0.000035	org.mediawiki
588	18859930	622	0.000048	com.thinkwithgoogle
589	18858612	2865	0.000012	com.duolingo
590	18855586	2112	0.000018	com.domaintools
591	18855394	291	0.000094	net.secureservercdn
592	18855236	2860	0.000012	com.biography
593	18853964	1648	0.000020	jp.ne.goo
594	18853434	1827	0.000019	com.lifewire
595	18853160	2479	0.000015	ie.independent
596	18850136	2903	0.000012	uk.ac.leeds
597	18849484	2951	0.000012	com.allure
598	18849452	2241	0.000016	com.timeout
599	18848206	3135	0.000011	org.cpj
600	18847016	4137	0.000009	com.bonanza
601	18844324	1640	0.000020	ca.globalnews
602	18843380	1774	0.000019	gov.in
603	18843170	1731	0.000020	com.images-amazon
604	18843116	2782	0.000013	com.depositphotos
605	18843074	1843	0.000018	com.thebalance
606	18842016	86	0.000401	com.livestream
607	18841682	287	0.000095	com.naver
608	18841558	477	0.000062	com.force
609	18840756	1422	0.000022	net.codecanyon
610	18838890	3346	0.000011	io.ghost
611	18838884	2419	0.000015	com.teenvogue
612	18838836	2366	0.000015	nz.co.stuff
613	18837438	2732	0.000013	com.123rf
614	18836408	2508	0.000014	com.motherjones
615	18836354	679	0.000043	int.wipo
616	18836332	2615	0.000014	edu.kit
617	18834684	1706	0.000020	com.routledge
618	18834336	695	0.000043	io.readthedocs
619	18834038	3692	0.000010	com.laweekly
620	18832208	441	0.000069	com.businesswire
621	18831266	2914	0.000012	org.oxfam
622	18829852	486	0.000062	com.adweek
623	18829616	2498	0.000014	edu.hawaii
624	18829216	3543	0.000010	com.udn
625	18828270	726	0.000042	com.canva
626	18828034	2985	0.000012	com.slides
627	18827908	627	0.000047	io.codepen
628	18827552	2664	0.000013	com.googlegroups
629	18826348	3901	0.000009	cn.org.china
630	18825700	2698	0.000013	com.coca-colacompany
631	18825602	888	0.000033	uk.co.pinterest
632	18823058	2551	0.000014	org.fas
633	18822830	1110	0.000027	net.clickbank
634	18822568	3099	0.000011	uk.co.timesonline
635	18822432	363	0.000081	net.php
636	18821410	2612	0.000014	edu.iastate
637	18821392	2144	0.000017	com.refinery29
638	18820314	988	0.000030	gov.dhs
639	18819896	4216	0.000008	com.alamy
640	18819802	991	0.000030	de.t-online
641	18817700	279	0.000097	com.iubenda
642	18817434	1612	0.000021	com.haaretz
643	18816616	1565	0.000021	mil.army
644	18816484	3053	0.000011	com.hm
645	18816156	1605	0.000021	uk.gov.ons
646	18813740	359	0.000081	mp.mailchi
647	18811428	3469	0.000010	org.heritage
648	18808928	1222	0.000025	org.eugdpr
649	18808748	2341	0.000016	za.co.google
650	18808348	563	0.000053	org.unicef
651	18808206	2938	0.000012	com.theonion
652	18807932	345	0.000083	com.akismet
653	18807216	150	0.000190	org.networkadvertising
654	18805384	834	0.000035	com.venturebeat
655	18803670	2671	0.000013	com.timesofisrael
656	18802968	3182	0.000011	com.ogilvy
657	18800978	134	0.000217	info.aboutads
658	18800344	1265	0.000024	tw.com.google
659	18799472	549	0.000054	com.fc2
660	18798170	2715	0.000013	com.theintercept
661	18798140	2377	0.000015	com.foreignpolicy
662	18797944	4009	0.000009	com.zara
663	18796758	3042	0.000012	org.project-syndicate
664	18796410	2790	0.000013	cn.gov.fmprc
665	18795586	541	0.000055	com.patreon
666	18794332	2912	0.000012	org.ballotpedia
667	18793786	3675	0.000010	uk.co.guim
668	18793668	1203	0.000025	com.thenextweb
669	18793250	2460	0.000015	nz.co.nzherald
670	18791956	2329	0.000016	gov.faa
671	18791606	664	0.000045	com.entrepreneur
672	18790298	1699	0.000020	com.nike
673	18788220	2257	0.000016	com.voanews
674	18785682	3646	0.000010	com.podomatic
675	18783652	394	0.000080	jp.ameblo
676	18781732	3665	0.000010	nz.co.scoop
677	18779826	2668	0.000013	com.jpost
678	18779770	755	0.000040	org.js
679	18778874	2589	0.000014	de.tagesspiegel
680	18778436	591	0.000051	com.gofundme
681	18778236	651	0.000046	it.placehold
682	18778130	692	0.000043	gov.nist
683	18777528	3090	0.000011	no.uib
684	18777264	3759	0.000010	com.clustrmaps
685	18776238	2533	0.000014	com.channelnewsasia
686	18775626	2332	0.000016	com.carto
687	18774970	2544	0.000014	edu.usf
688	18774910	4401	0.000008	uk.ac.essex
689	18774474	2203	0.000017	de.br
690	18773102	4256	0.000008	org.marxists
691	18769558	2826	0.000013	br.com.blogspot
692	18769428	680	0.000043	com.photobucket
693	18769216	3395	0.000010	com.parade
694	18768412	3731	0.000010	com.mongabay
695	18768244	742	0.000041	com.moz
696	18768070	3519	0.000010	ar.com.lanacion
697	18767480	1162	0.000026	com.digitaloceanspaces
698	18767252	3833	0.000009	com.scribblelive
699	18766284	3667	0.000010	ru.msk
700	18765148	1707	0.000020	org.oxfordjournals
701	18765018	1846	0.000018	com.speakerdeck
702	18764342	1177	0.000026	com.jekyllrb
703	18764272	1320	0.000023	com.imageshack
704	18763628	698	0.000043	com.withgoogle
705	18763518	2775	0.000013	com.fineartamerica
706	18763306	1627	0.000020	org.amnesty
707	18762512	2572	0.000014	org.unctad
708	18761940	3049	0.000012	int.au
709	18761162	109	0.000306	me.wa
710	18760936	2184	0.000017	org.ncsl
711	18759340	2857	0.000012	uk.org.nationaltrust
712	18758396	4435	0.000008	com.mysanantonio
713	18758278	3154	0.000011	fr.rfi
714	18757674	1300	0.000023	gov.federalregister
715	18756920	5968	0.000006	org.arkive
716	18756858	3385	0.000011	com.nationalreview
717	18756398	2232	0.000017	org.worldcat
718	18756388	4160	0.000009	com.turkishairlines
719	18756100	3077	0.000011	uk.ac.york
720	18755876	3029	0.000012	org.nationalgeographic
721	18755390	3360	0.000011	org.tigris
722	18755228	362	0.000081	com.adnxs
723	18753668	1765	0.000019	com.indianexpress
724	18753482	3293	0.000011	org.neocities
725	18751702	3058	0.000011	ly.genial
726	18750392	3274	0.000011	uk.co.penguin
727	18750242	919	0.000032	com.hootsuite
728	18750002	3494	0.000010	com.nme
729	18747334	2735	0.000013	com.kaggle
730	18746894	193	0.000131	com.discord
731	18746600	3357	0.000011	de.taz
732	18746298	3147	0.000011	edu.bc
733	18746190	3041	0.000012	tr.com.aa
734	18745544	3156	0.000011	com.cgtn
735	18745132	2431	0.000015	org.unodc
736	18744550	282	0.000096	gov.ftc
737	18743682	2344	0.000016	eu.politico
738	18743052	1364	0.000023	com.symantec
739	18740834	2520	0.000014	net.openid
740	18740752	3637	0.000010	il.ac.tau
741	18739398	2453	0.000015	ru.ria
742	18738948	3031	0.000012	com.allafrica
743	18738772	1172	0.000026	jp.ac.keio
744	18738200	2869	0.000012	edu.educause
745	18738156	3402	0.000010	org.firstmonday
746	18738122	2642	0.000014	org.wikidata
747	18737788	491	0.000061	com.placeholder
748	18734980	2900	0.000012	com.simonandschuster
749	18734860	3796	0.000009	org.amnestyusa
750	18734696	2168	0.000017	com.justia
751	18733308	2197	0.000017	ca.on.gov
752	18733088	3707	0.000010	uk.gov.scotland
753	18733026	4563	0.000008	com.flightradar24
754	18732944	4326	0.000008	com.interviewmagazine
755	18732742	3744	0.000010	com.afp
756	18732154	4112	0.000009	org.scala-sbt
757	18730992	2780	0.000013	ae.google
758	18730824	1303	0.000023	org.webkit
759	18729882	2947	0.000012	com.superuser
760	18729678	1014	0.000029	com.highcharts
761	18729020	3802	0.000009	com.wusa9
762	18728900	2550	0.000014	jp.nicovideo
763	18728094	2141	0.000017	gov.pa
764	18727990	3937	0.000009	org.one
765	18726672	2908	0.000012	edu.uky
766	18725034	3052	0.000011	in.businessinsider
767	18724474	3382	0.000011	org.hypotheses
768	18723378	2681	0.000013	org.wbur
769	18723236	569	0.000052	com.inc
770	18721066	1434	0.000022	com.upwork
771	18720908	4232	0.000008	org.sourcewatch
772	18720676	3366	0.000011	com.sciencealert
773	18720222	1316	0.000023	de.rki
774	18719114	3466	0.000010	org.royalsociety
775	18718524	2354	0.000016	ru.rbc
776	18718198	1034	0.000029	com.videojs
777	18717530	3383	0.000011	org.polymer-project
778	18717228	487	0.000062	ee.lin
779	18717008	3202	0.000011	org.texastribune
780	18716598	1054	0.000028	fm.last
781	18716564	3658	0.000010	se.gu
782	18715916	2181	0.000017	it.redd
783	18715552	1219	0.000025	com.smashingmagazine
784	18715536	2743	0.000013	org.undocs
785	18713460	2713	0.000013	org.iucn
786	18713092	3662	0.000010	com.hashicorp
787	18712944	1712	0.000020	scot.gov
788	18712564	976	0.000031	com.jwplayer
789	18712330	3843	0.000009	edu.wayne
790	18711122	576	0.000052	com.booking
791	18710906	803	0.000037	com.fandom
792	18710858	3850	0.000009	com.triplepundit
793	18709690	329	0.000085	com.hackerone
794	18709314	4235	0.000008	com.letterboxd
795	18707654	1151	0.000026	com.alexa
796	18707534	2258	0.000016	com.knightlab
797	18706964	770	0.000039	com.sedo
798	18706750	3394	0.000010	org.iucnredlist
799	18706380	1944	0.000018	com.firebaseapp
800	18705808	4036	0.000009	com.manta
801	18702546	3072	0.000011	au.com.theage
802	18701450	3475	0.000010	org.sierraclub
803	18700424	400	0.000078	com.onesignal
804	18700348	2652	0.000013	ru.kommersant
805	18700250	3379	0.000011	com.hasbro
806	18699356	3821	0.000009	edu.unu
807	18699146	3656	0.000010	com.crashlytics
808	18698894	817	0.000037	com.marketwatch
809	18698494	4071	0.000009	ru.aif
810	18698224	4419	0.000008	com.folkd
811	18698152	1285	0.000024	gov.uspto
812	18697734	3358	0.000011	net.ipsnews
813	18697484	2632	0.000014	org.unfpa
814	18697336	987	0.000030	com.stackexchange
815	18696510	3207	0.000011	ly.plot
816	18696184	532	0.000056	com.indeed
817	18695334	1667	0.000020	fr.blogspot
818	18695246	972	0.000031	com.css-tricks
819	18694382	933	0.000032	org.reactjs
820	18693292	4083	0.000009	com.marinetraffic
821	18692940	2706	0.000013	ru.rg
822	18692900	4365	0.000008	com.balenciaga
823	18692468	2561	0.000014	com.kinstacdn
824	18691710	2586	0.000014	build.bazel
825	18690746	545	0.000055	com.digg
826	18690190	3997	0.000009	jp.co.tepco
827	18690182	1457	0.000022	io.webflow
828	18690148	4685	0.000008	com.gmanetwork
829	18689994	3779	0.000009	org.rferl
830	18689870	4416	0.000008	kr.co.koreatimes
831	18689452	821	0.000036	com.oreilly
832	18689426	953	0.000031	gov.fcc
833	18689024	2957	0.000012	com.articulate
834	18688634	3054	0.000011	site.notion
835	18688256	2441	0.000015	int.reliefweb
836	18688218	2722	0.000013	com.insidehighered
837	18687872	1086	0.000027	so.notion
838	18687810	4466	0.000008	org.sfpl
839	18687672	3580	0.000010	uk.co.spectator
840	18687622	2840	0.000012	com.suntimes
841	18687452	917	0.000032	com.verisign
842	18686880	2967	0.000012	org.cfr
843	18686622	2819	0.000013	org.panda
844	18686298	1016	0.000029	com.mixcloud
845	18686002	83	0.000405	com.messenger
846	18685934	533	0.000056	jp.co.rakuten
847	18685764	4343	0.000008	com.upworthy
848	18685494	2644	0.000014	ru.kremlin
849	18684806	278	0.000097	com.sxsw
850	18684212	3542	0.000010	com.flippa
851	18684006	568	0.000052	com.mckinsey
852	18683956	2534	0.000014	net.convio
853	18683510	1231	0.000025	com.buffer
854	18683006	3100	0.000011	com.yougov
855	18682928	5110	0.000007	com.viki
856	18682436	4674	0.000008	org.birdlife
857	18681966	4413	0.000008	com.itsnicethat
858	18681374	650	0.000046	com.gartner
859	18681372	2927	0.000012	uk.gov.metoffice
860	18681084	494	0.000061	com.dmca
861	18680840	3597	0.000010	org.jenkins-ci
862	18680358	3089	0.000011	int.iom
863	18678670	4082	0.000009	com.iconarchive
864	18677384	4480	0.000008	com.oriflame
865	18676518	4538	0.000008	net.middleeasteye
866	18675854	5041	0.000007	com.waitbutwhy
867	18675534	4316	0.000008	org.pen
868	18675274	2830	0.000013	fm.omny
869	18674282	3902	0.000009	org.icij
870	18674044	4188	0.000008	org.constitutioncenter
871	18673972	3857	0.000009	ch.qos
872	18673474	4070	0.000009	com.9to5google
873	18673432	3526	0.000010	uk.gov.companieshouse
874	18673406	3994	0.000009	uk.ac.sussex
875	18673258	3266	0.000011	com.foreignaffairs
876	18673246	2835	0.000012	com.news24
877	18673204	4132	0.000009	re.cli
878	18672690	4227	0.000008	jp.ac.kobe-u
879	18671714	902	0.000033	br.com.uol
880	18671552	3718	0.000010	com.nybooks
881	18671446	1818	0.000019	com.over-blog
882	18671362	5578	0.000006	com.symbaloo
883	18669612	3170	0.000011	uk.co.bbci
884	18669200	934	0.000032	com.pubmatic
885	18669010	2385	0.000015	com.scene7
886	18668810	3467	0.000010	org.wikileaks
887	18667242	4637	0.000008	org.foodandwaterwatch
888	18666438	3305	0.000011	at.derstandard
889	18666074	876	0.000034	com.zoho
890	18665410	3167	0.000011	org.adb
891	18664362	3518	0.000010	com.benzinga
892	18663988	730	0.000041	com.usnews
893	18663450	5592	0.000006	io.postach
894	18663030	3547	0.000010	com.palgrave
895	18662464	1109	0.000027	net.media
896	18662120	4091	0.000009	net.datasociety
897	18661426	540	0.000055	com.googleoptimize
898	18661044	4116	0.000009	au.com.heraldsun
899	18659068	3440	0.000010	ru.kp
900	18657988	2675	0.000013	com.thenation
901	18657690	663	0.000045	me.zalo
902	18657052	3001	0.000012	com.unity
903	18655796	1635	0.000020	org.altervista
904	18654580	4826	0.000007	it.polito
905	18654490	4482	0.000008	edu.odu
906	18654200	3171	0.000011	org.sonatype
907	18653790	2853	0.000012	net.vnexpress
908	18653284	735	0.000041	com.alibaba
909	18652510	4454	0.000008	com.muckrack
910	18652400	2937	0.000012	com.lexology
911	18652270	4962	0.000007	kr.co.hani
912	18650818	3723	0.000010	com.tradingeconomics
913	18650606	3882	0.000009	com.study
914	18650594	595	0.000050	com.airbnb
915	18649776	3663	0.000010	gov.ustr
916	18649674	4900	0.000007	com.theodysseyonline
917	18649520	3724	0.000010	uk.gov.homeoffice
918	18648278	1031	0.000029	com.pcmag
919	18647134	633	0.000047	org.joomla
920	18645770	3367	0.000011	br.scielo
921	18645448	74	0.000486	com.trustpilot
922	18644890	5552	0.000006	au.edu.vu
923	18644710	3127	0.000011	tw.com.pchome
924	18644260	725	0.000042	com.splashthat
925	18643560	2745	0.000013	ca.citizenlab
926	18642628	4599	0.000008	com.condenast
927	18642354	1676	0.000020	com.techrepublic
928	18641430	2125	0.000018	io.pantheonsite
929	18641322	3281	0.000011	ru.cbr
930	18641240	2866	0.000012	ca.uwaterloo
931	18640912	4384	0.000008	uk.co.belfasttelegraph
932	18640688	597	0.000050	com.wufoo
933	18639194	3231	0.000011	org.ellenmacarthurfoundation
934	18639046	4821	0.000007	com.zimbio
935	18638824	3321	0.000011	com.rabbitmq
936	18638422	547	0.000054	com.herokuapp
937	18636852	5657	0.000006	org.cgsociety
938	18634912	5704	0.000006	in.teletype
939	18634900	531	0.000056	com.aol
940	18634026	3752	0.000010	edu.ucpress
941	18633976	3375	0.000011	com.scotsman
942	18633494	3277	0.000011	com.kroger
943	18632272	405	0.000077	com.constantcontact
944	18631930	870	0.000034	com.emarketer
945	18630874	5643	0.000006	com.dbs
946	18630838	4407	0.000008	au.edu.deakin
947	18630630	3315	0.000011	org.osce
948	18629458	2971	0.000012	com.euractiv
949	18628642	4788	0.000007	com.latercera
950	18626102	3881	0.000009	com.bloombergquint
951	18625580	952	0.000031	com.digitalocean
952	18625102	3638	0.000010	org.ushmm
953	18624888	3751	0.000010	com.lawfareblog
954	18624664	4877	0.000007	ke.co.google
955	18624400	3835	0.000009	com.thenationalnews
956	18624378	4716	0.000007	com.kongregate
957	18624240	5132	0.000007	com.apsense
958	18624020	1342	0.000023	com.nvidia
959	18623838	617	0.000048	gov.copyright
960	18623504	4437	0.000008	com.jacobinmag
961	18623396	2934	0.000012	net.dwcdn
962	18622680	643	0.000046	com.accenture
963	18622320	4529	0.000008	uk.ac.soas
964	18621166	3445	0.000010	de.test
965	18620568	1661	0.000020	com.createjs
966	18620140	3218	0.000011	com.obsproject
967	18619976	2892	0.000012	org.gnupg
968	18619870	4318	0.000008	com.washingtonian
969	18619392	4908	0.000007	uk.co.birminghammail
970	18619154	4548	0.000008	io.meduza
971	18619058	4034	0.000009	ru.mid
972	18618882	1207	0.000025	org.golang
973	18618534	3930	0.000009	org.cgiar
974	18617116	2411	0.000015	co.pcdn
975	18616304	2613	0.000014	com.olark
976	18615562	1007	0.000030	com.gumroad
977	18613652	2755	0.000013	ru.tass
978	18613510	4825	0.000007	com.selfridges
979	18612814	3770	0.000009	fr.capital
980	18612214	4391	0.000008	za.co.mg
981	18612168	911	0.000033	net.atlassian
982	18612044	844	0.000035	com.redhat
983	18611518	1749	0.000019	com.indiegogo
984	18611438	5009	0.000007	edu.utep
985	18610856	1727	0.000020	org.linuxfoundation
986	18610446	956	0.000031	com.att
987	18609186	2890	0.000012	org.transparency
988	18608588	3918	0.000009	com.encyclopedia
989	18606828	72	0.000505	com.oculus
990	18606772	699	0.000043	com.psychologytoday
991	18606698	3091	0.000011	com.sharefile
992	18606504	151	0.000189	org.whatwg
993	18606354	754	0.000040	org.poynter
994	18606268	3386	0.000011	com.alchemer
995	18604648	707	0.000042	co.ibb
996	18604432	286	0.000095	com.caniuse
997	18604402	2738	0.000013	com.springeropen
998	18604386	2490	0.000014	studio.flourish
999	18604138	4373	0.000008	com.googledrive
1000	18604014	4490	0.000008	tw.com.books

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

Important news for users of Common Crawl data: we are introducing CloudFront as a new way to access Common Crawl data as part of Amazon Web Services’ registry of open data

March 1, 2022Sebastian Nagel

Ten years ago(!) Common Crawl joined AWS’s Open Data Sponsorships program, hosted on S3, with free access to everyone. Since then, the dataset has expanded (by petabytes!) and our community of users has seen extraordinary growth. With growth comes change, therefore…

Following are new measures to accommodate the number and volume of requests for Common Crawl data, and to ensure efficient and stable access to the data:

1. Common Crawl data is now available on CloudFront under the URL base prefixes*:

https://data.commoncrawl.org/ or
https://ds5q9oxwqwsfj.cloudfront.net/

* Data is accessible via http:// or https://.

As of Monday, April 4, 2022

2. Access to data from the Amazon cloud using the S3 API will be restricted to authenticated AWS users, and unsigned access to s3://commoncrawl/ will be disabled. See Q&A for further details.

3. To access data from outside the Amazon cloud, via HTTP(S), a new URL prefix – https://data.commoncrawl.org/ – must be used instead of https://commoncrawl.s3.amazonaws.com/.

Q & A

Q: How can I identify whether my code is using unauthenticated S3 access?

A: Authenticated access is the default, but you should verify that your code is not configured otherwise; code fragments requesting unauthenticated access could be (but are not limited to):

AWS CLI with the command-line option --no-sign-request:
```
aws --no-sign-request s3 cp s3://commoncrawl/...
```

Python using boto3 and botocore.UNSIGNED:

import boto3
import botocore

s3client = boto3.client('s3', config=botocore.client.Config(signature_version=botocore.UNSIGNED))

Hadoop or Spark (various programming languages): usage of AnonymousAWSCredentialsProvider

Q: Where can I find up-to-date documentation and examples that are consistent with these new protocols?

A: We will update our documentation and examples to read data either via HTTP(S) using the new CloudFront access, or via S3 and authenticated access. We will also update all data download links. It will take a few days until everything is updated. Thanks for your patience!

Q: Are range requests supported?

A: Yes, range requests are supported by CloudFront same as for S3, see the CloudFront docs about RangeGETs.

Q: What is the recommended access method on AWS but in a different region (not us-east-1)?

A: We recommend that you run your computing workload in the same region (us-east-1) as the Common Crawl dataset whenever possible. If you have a specific, ongoing need to run computing workloads using Common Crawl in another AWS region, the AWS Open Data team would like to hear more about your use case ([email protected]).

Q: What are typical error messages indicating that unauthenticated access is used?

A: The HTTP response status code is a 403 Forbidden. However, also a restrictive IAM policy on the user’s side could deny access to s3://commoncrawl/ using the S3 API.
Two examples for error messages related to unauthenticated access to s3://commoncrawl/:

> curl https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2022-05/warc.paths.gz
<!--?xml version="1.0" encoding="UTF-8"?-->
<error><code>AccessDenied</code><message>Access Denied</message>...</error>

> aws --no-sign-request s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2022-05/warc.paths.gz .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden