Blog – Common Crawl

June/July 2022 crawl archive now available

July 13, 2022Sebastian Nagel

The crawl archive for June/July 2022 is now available! The data was crawled June 24 – July 7 and contains 3.1 billion web pages or 370 TiB of uncompressed content. Page captures are from 44 million hosts or 35 million registered domains and include 1.4 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The June/July crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2022-27/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively, please see accessing the data for detailed instructions.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2022-27/segment.paths.gz	100
WARC files	CC-MAIN-2022-27/warc.paths.gz	80000	84.08
WAT files	CC-MAIN-2022-27/wat.paths.gz	80000	20.99
WET files	CC-MAIN-2022-27/wet.paths.gz	80000	8.97
Robots.txt files	CC-MAIN-2022-27/robotstxt.paths.gz	80000	0.15
Non-200 responses files	CC-MAIN-2022-27/non200responses.paths.gz	80000	1.92
URL index files	CC-MAIN-2022-27/cc-index.paths.gz	302	0.23
Columnar URL index files	CC-MAIN-2022-27/cc-index-table.paths.gz	900	0.26

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2022-27/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

May 2022 crawl archive now available

June 2, 2022Sebastian Nagel

The crawl archive for May 2022 is now available! The data was crawled May 16 – 29 and contains 3.45 billion web pages or 420 TiB of uncompressed content. Page captures are from 45 million hosts or 36 million registered domains and include 1.4 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The May crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2022-21/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2022-21/segment.paths.gz	100
WARC files	CC-MAIN-2022-21/warc.paths.gz	80000	92.81
WAT files	CC-MAIN-2022-21/wat.paths.gz	80000	23.30
WET files	CC-MAIN-2022-21/wet.paths.gz	80000	9.80
Robots.txt files	CC-MAIN-2022-21/robotstxt.paths.gz	80000	0.15
Non-200 responses files	CC-MAIN-2022-21/non200responses.paths.gz	80000	2.26
URL index files	CC-MAIN-2022-21/cc-index.paths.gz	302	0.25
Columnar URL index files	CC-MAIN-2022-21/cc-index-table.paths.gz	900	0.28

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2022-21/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs October, November/December 2021 and January 2022

March 16, 2022Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of October, November/December 2021 and January 2022. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases. You may also visit the projects cc-webgraph and cc-pyspark which include all scripts and tools required to construct the graphs. Instructions to explore the graphs in the webgraph format are given in our collection of webgraph notebooks.

Host-level graph

The graph consists of 384 million nodes and 2.47 billion edges. Both hyperlinks and HTTP redirects and link headers are used as edges to span up the graph. All types of links are included, including pure “technical” ones pointing to images, JavaScript libraries, web fonts, etc. However, only host names with a valid IANA TLD are used. Consequently, URLs with an IP address as host component are not taken into account for building the host-level graph.

There are 326 million dangling nodes (84.6%) and the largest strongly connected component contains 45.2 million (11.7%) nodes. Dangling nodes stem from

hosts that have not been crawled, yet are pointed to from a link on a crawled page
hosts without any links pointing to a different host name
or hosts which did only return an error page (eg. HTTP 404)

Host names in the graph are in reverse domain name notation and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 766 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2021-22-oct-nov-jan/host/ (this requires an account on AWS). Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2021-22-oct-nov-jan/host/ as prefix to access the files from everywhere.

Please note that the text representation of the host-level graph is shipped in 96 gzip-compressed files listed in two path listings – one for the nodes (vertices), one for the edges (arcs). First, download the paths listing and decompress it using “gzip -d” or “gunzip”. By adding the prefix s3://commoncrawl/ or https://data.commoncrawl.org/ to each line in the path listing you get the list of URLs to download the entire graph.

Download files of the Common Crawl Oct/Nov/Jan 2021-2022 host-level webgraph

Size	File	Description
2.66 GB	cc-main-2021-22-oct-nov-jan-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 32 vertices files
11.76 GB	cc-main-2021-22-oct-nov-jan-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 64 edges files

5.32 GB	cc-main-2021-22-oct-nov-jan-host.graph	graph in BVGraph format
2 kB	cc-main-2021-22-oct-nov-jan-host.properties

5.78 GB	cc-main-2021-22-oct-nov-jan-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2021-22-oct-nov-jan-host-t.properties

1 kB	cc-main-2021-22-oct-nov-jan-host.stats	WebGraph statistics

6.38 GB	cc-main-2021-22-oct-nov-jan-host-ranks.txt.gz	harmonic centrality and pagerank

Domain-level graph

The domain graph is built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org. Version (commit) 68b67d3 of the public suffix list was used (commit date 2022-03-04).

The domain-level graph has 90 million nodes and 1.55 billion edges. 50% or 45 million nodes are dangling nodes, the largest strongly connected component covers 36 million or 40% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2021-22-oct-nov-jan/domain/ or on https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2021-22-oct-nov-jan/domain/.

Download files of the Common Crawl Oct/Nov/Jan 2021-2022 domain-level webgraph

Size	File	Description
0.62 GB	cc-main-2021-22-oct-nov-jan-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
6.36 GB	cc-main-2021-22-oct-nov-jan-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

3.65 GB	cc-main-2021-22-oct-nov-jan-domain.graph	graph in BVGraph format
2 kB	cc-main-2021-22-oct-nov-jan-domain.properties

3.53 GB	cc-main-2021-22-oct-nov-jan-domain-t.graph	transpose of the graph
2 kB	cc-main-2021-22-oct-nov-jan-domain-t.properties

1 kB	cc-main-2021-22-oct-nov-jan-domain.stats	WebGraph statistics

1.93 GB	cc-main-2021-22-oct-nov-jan-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 90 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Oct/Nov/Jan 2020-2021)

harmonic centrality rank	hc value	page rank	page rank value	reversed domain name
1	31448418	1	0.017921	com.googleapis
2	30105086	3	0.013006	com.facebook
3	28991726	2	0.014028	com.google
4	26233886	6	0.007154	org.w
5	26230152	4	0.008081	com.twitter
6	25864994	8	0.006261	com.youtube
7	25325250	7	0.006558	com.instagram
8	24665160	5	0.007716	com.googletagmanager
9	24097224	9	0.004716	org.gmpg
10	23219788	11	0.003948	com.gstatic
11	23145196	13	0.003418	com.linkedin
12	22256010	10	0.004364	com.cloudflare
13	21953630	17	0.001942	com.gravatar
14	21841462	21	0.001594	com.pinterest
15	21786278	14	0.003223	org.wordpress
16	21441268	25	0.001417	org.wikipedia
17	21311496	16	0.002057	com.apple
18	21176702	33	0.001077	com.wordpress
19	21045484	32	0.001141	com.vimeo
20	21032900	15	0.002070	com.bootstrapcdn
21	20995456	41	0.000913	be.youtu
22	20796928	18	0.001658	com.jquery
23	20726342	28	0.001191	com.microsoft
24	20673758	45	0.000789	com.blogspot
25	20634464	23	0.001451	io.polyfill
26	20617114	47	0.000775	gl.goo
27	20589400	49	0.000736	com.amazon
28	20486426	64	0.000550	ly.bit
29	20476754	29	0.001178	com.wixstatic
30	20455806	50	0.000729	com.wp
31	20452992	22	0.001551	net.cloudfront
32	20445298	40	0.000962	com.amazonaws
33	20410144	43	0.000863	org.mozilla
34	20403210	31	0.001162	net.jsdelivr
35	20395958	51	0.000721	eu.europa
36	20389728	37	0.000994	com.google-analytics
37	20343648	20	0.001595	com.fontawesome
38	20336692	91	0.000384	com.tumblr
39	20333242	19	0.001648	com.adobe
40	20324398	24	0.001421	com.github
41	20181488	75	0.000480	com.googleusercontent
42	20150384	55	0.000684	com.flickr
43	20122758	104	0.000326	com.yahoo
44	20112952	57	0.000670	com.paypal
45	20106820	48	0.000752	io.github
46	20102750	106	0.000314	com.reddit
47	20060046	117	0.000268	com.soundcloud
48	20049174	38	0.000966	com.googlesyndication
49	20043166	81	0.000425	com.medium
50	20007290	53	0.000703	org.w3
51	19994710	127	0.000231	com.nytimes
52	19974404	62	0.000611	co.t
53	19956144	102	0.000338	com.weebly
54	19955606	114	0.000277	com.spotify
55	19925440	58	0.000656	com.whatsapp
56	19906786	34	0.001038	ru.yandex
57	19901846	85	0.000401	org.creativecommons
58	19894214	136	0.000208	org.archive
59	19870846	183	0.000139	com.cnn
60	19863762	63	0.000608	org.schema
61	19855684	60	0.000645	com.addthis
62	19851344	146	0.000194	com.forbes
63	19836582	203	0.000127	uk.co.bbc
64	19834544	70	0.000513	com.shopify
65	19811758	234	0.000113	com.washingtonpost
66	19808664	69	0.000523	com.vk
67	19804862	185	0.000138	com.bing
68	19803622	147	0.000193	gov.cdc
69	19802914	157	0.000172	int.who
70	19798680	92	0.000383	me.wp
71	19780956	44	0.000809	net.doubleclick
72	19760738	141	0.000201	gov.nih
73	19749646	59	0.000649	com.macromedia
74	19748198	71	0.000506	com.unpkg
75	19747664	260	0.000103	net.researchgate
76	19734110	252	0.000106	com.wsj
77	19733152	293	0.000093	edu.stanford
78	19731674	232	0.000115	com.imdb
79	19724760	188	0.000134	org.wikimedia
80	19712934	39	0.000965	net.fbcdn
81	19700958	206	0.000124	com.businessinsider
82	19690114	143	0.000200	com.dropbox
83	19687820	284	0.000095	edu.mit
84	19683060	100	0.000363	com.list-manage
85	19682338	308	0.000089	com.tinyurl
86	19665976	27	0.001204	org.apache
87	19657446	163	0.000157	com.theguardian
88	19651458	299	0.000092	com.android
89	19647768	445	0.000067	com.quora
90	19643018	165	0.000156	org.doi
91	19636594	328	0.000085	com.go
92	19629826	240	0.000109	com.bloomberg
93	19629762	274	0.000098	edu.harvard
94	19627444	501	0.000060	com.msn
95	19625870	164	0.000157	com.issuu
96	19625646	254	0.000106	com.oracle
97	19624678	344	0.000083	com.springer
98	19621946	149	0.000190	com.wixsite
99	19615182	140	0.000203	us.zoom
100	19612484	131	0.000220	com.npmjs
101	19608812	111	0.000294	me.t
102	19608314	334	0.000084	com.slack
103	19602354	124	0.000241	com.mailchimp
104	19596192	244	0.000108	com.stackoverflow
105	19592312	303	0.000091	com.reuters
106	19587800	321	0.000087	com.techcrunch
107	19586192	505	0.000059	com.myspace
108	19586126	241	0.000109	com.twimg
109	19584238	166	0.000155	com.giphy
110	19583908	292	0.000093	com.example
111	19577384	52	0.000709	com.fb
112	19576726	167	0.000153	com.yelp
113	19576020	169	0.000151	com.office
114	19572502	341	0.000083	com.prnewswire
115	19567966	190	0.000133	com.unsplash
116	19564492	108	0.000309	de.google
117	19561016	300	0.000091	com.wiley
118	19555072	46	0.000785	net.facebook
119	19552510	268	0.000101	org.un
120	19549098	256	0.000105	com.sciencedirect
121	19548162	489	0.000061	com.latimes
122	19547508	594	0.000050	com.livejournal
123	19547452	145	0.000196	gle.forms
124	19544300	466	0.000063	uk.co.telegraph
125	19543690	402	0.000078	com.nature
126	19542384	340	0.000083	org.npr
127	19541420	484	0.000062	com.ted
128	19535332	514	0.000057	edu.berkeley
129	19533106	647	0.000046	com.vice
130	19531328	237	0.000110	org.gnu
131	19529798	198	0.000130	org.ietf
132	19527852	714	0.000042	uk.ac.cam
133	19524938	434	0.000071	com.time
134	19522382	289	0.000094	com.bbc
135	19517460	497	0.000060	com.goodreads
136	19513616	412	0.000076	org.arxiv
137	19510034	306	0.000090	com.cnbc
138	19508184	144	0.000197	com.ytimg
139	19492206	700	0.000043	edu.columbia
140	19483642	433	0.000071	com.sagepub
141	19475908	153	0.000186	com.ft
142	19474592	173	0.000149	org.acm
143	19473316	323	0.000086	com.githubusercontent
144	19473132	438	0.000069	com.cnet
145	19468042	120	0.000254	com.youtube-nocookie
146	19458540	409	0.000077	com.wired
147	19455634	205	0.000125	com.imgur
148	19454210	518	0.000057	uk.co.dailymail
149	19449628	202	0.000127	com.blogger
150	19448116	79	0.000459	com.godaddy
151	19443158	26	0.001330	cn.gov.miit
152	19437758	343	0.000083	com.theverge
153	19435568	509	0.000058	edu.yale
154	19434236	171	0.000150	org.ampproject
155	19426446	472	0.000063	com.nationalgeographic
156	19423252	281	0.000096	com.squarespace
157	19419880	804	0.000037	org.chromium
158	19416530	696	0.000043	uk.ac.ox
159	19413138	488	0.000061	com.googleblog
160	19407848	458	0.000064	gov.whitehouse
161	19407614	304	0.000090	com.usatoday
162	19407228	499	0.000060	com.staticflickr
163	19407066	829	0.000036	com.evernote
164	19404734	192	0.000131	com.hubspot
165	19404414	476	0.000062	org.ieee
166	19402398	587	0.000051	org.worldbank
167	19402004	407	0.000077	com.dribbble
168	19400214	132	0.000219	com.statcounter
169	19399796	512	0.000058	ee.linktr
170	19399358	527	0.000056	edu.cornell
171	19395500	123	0.000243	com.sharethis
172	19388246	519	0.000057	com.theatlantic
173	19387454	408	0.000077	com.docker
174	19385182	673	0.000044	com.git-scm
175	19383304	214	0.000122	com.wpengine
176	19383030	797	0.000037	org.sciencemag
177	19381750	811	0.000037	com.arstechnica
178	19381474	275	0.000098	gg.discord
179	19381222	159	0.000167	com.zendesk
180	19379042	264	0.000101	uk.co.google
181	19376664	138	0.000206	me.line
182	19373526	246	0.000107	uk.co.amazon
183	19373448	602	0.000050	com.zdnet
184	19372154	239	0.000109	net.slideshare
185	19369046	325	0.000086	com.appspot
186	19368866	702	0.000042	com.economist
187	19368620	737	0.000041	org.cambridge
188	19368112	571	0.000052	com.cisco
189	19367144	792	0.000038	edu.washington
190	19366344	648	0.000046	org.weforum
191	19361760	668	0.000044	com.box
192	19361692	624	0.000047	org.pbs
193	19356220	479	0.000062	org.python
194	19355604	436	0.000070	com.huffingtonpost
195	19354284	226	0.000117	com.outlook
196	19353112	542	0.000055	com.typepad
197	19350044	288	0.000095	org.pewresearch
198	19345610	558	0.000054	com.cbsnews
199	19342774	267	0.000101	net.windows
200	19334638	556	0.000054	com.deloitte
201	19333688	1058	0.000028	com.rollingstone
202	19333260	496	0.000060	com.pixabay
203	19333260	554	0.000054	gov.usda
204	19332390	720	0.000042	google.blog
205	19331286	572	0.000052	site.business
206	19330802	544	0.000055	uk.co.independent
207	19325162	1223	0.000025	ly.cutt
208	19323700	42	0.000906	com.qq
209	19323606	760	0.000039	com.apnews
210	19323302	745	0.000040	ca.cbc
211	19322838	639	0.000046	org.unesco
212	19322434	493	0.000061	com.gitlab
213	19307948	704	0.000042	com.mysql
214	19307400	611	0.000049	com.pexels
215	19302186	586	0.000051	gov.loc
216	19297622	728	0.000041	edu.upenn
217	19296262	713	0.000042	edu.wisc
218	19295886	485	0.000062	com.getpocket
219	19295020	553	0.000054	com.nbcnews
220	19294010	451	0.000065	com.fastcompany
221	19292592	1381	0.000022	com.ikea
222	19291294	178	0.000143	com.tripadvisor
223	19286758	1171	0.000026	org.eclipse
224	19285934	676	0.000043	com.scribd
225	19285000	768	0.000039	com.shutterstock
226	19284178	634	0.000047	com.mozilla
227	19282738	1225	0.000025	org.kernel
228	19277868	798	0.000037	uk.co.blogspot
229	19277480	784	0.000038	com.qz
230	19272246	859	0.000034	com.ggpht
231	19271988	318	0.000087	com.live
232	19271080	979	0.000030	uk.co.guardian
233	19268934	301	0.000091	com.w3schools
234	19265398	1924	0.000018	com.lego
235	19264382	498	0.000060	gov.irs
236	19262148	885	0.000034	edu.jhu
237	19260006	672	0.000044	com.buzzfeed
238	19259708	674	0.000044	uk.co.eventbrite
239	19259366	796	0.000038	com.trello
240	19258458	1181	0.000025	com.technologyreview
241	19256192	984	0.000030	com.playstation
242	19255014	1057	0.000028	fr.lemonde
243	19251458	552	0.000054	com.squareup
244	19251266	550	0.000054	com.fortune
245	19249814	507	0.000059	gov.nasa
246	19249644	751	0.000040	me.about
247	19244186	596	0.000050	com.oup
248	19243256	283	0.000095	net.behance
249	19241750	973	0.000031	com.foursquare
250	19240364	2212	0.000017	com.hbo
251	19238806	635	0.000047	fm.anchor
252	19237212	327	0.000085	com.disqus
253	19235880	915	0.000033	com.slate
254	19234328	54	0.000697	co.g
255	19232270	36	0.000996	com.baidu
256	19231094	456	0.000065	com.bigcommerce
257	19229430	175	0.000146	jp.co.google
258	19229270	249	0.000107	com.calendly
259	19229228	756	0.000040	com.vox
260	19227564	557	0.000054	com.dailymotion
261	19226480	654	0.000045	com.investopedia
262	19225524	860	0.000034	com.ubuntu
263	19225146	272	0.000099	com.bandcamp
264	19224258	1564	0.000021	com.hatenablog
265	19222446	1087	0.000027	co.elastic
266	19221444	724	0.000042	com.newyorker
267	19221360	854	0.000035	com.about
268	19220762	460	0.000064	com.arcgis
269	19219440	814	0.000037	com.variety
270	19219024	781	0.000038	au.net.abc
271	19217228	551	0.000054	com.elpais
272	19215612	822	0.000036	edu.ucla
273	19214942	802	0.000037	gov.congress
274	19214422	677	0.000043	org.apa
275	19212402	740	0.000041	com.freepik
276	19211346	1126	0.000026	com.steamcommunity
277	19210898	250	0.000107	gov.ca
278	19209850	773	0.000039	org.pypi
279	19209526	719	0.000042	com.libsyn
280	19208308	865	0.000034	edu.princeton
281	19207924	142	0.000200	com.opera
282	19207284	940	0.000032	com.nypost
283	19205296	813	0.000037	edu.umich
284	19204192	1339	0.000023	com.billboard
285	19204056	312	0.000088	com.typeform
286	19201904	280	0.000097	com.feedburner
287	19200550	871	0.000034	com.ssrn
288	19199714	546	0.000055	com.tandfonline
289	19197908	905	0.000033	com.podbean
290	19197754	224	0.000117	page.g
291	19197604	827	0.000036	org.fao
292	19197538	868	0.000034	com.foxnews
293	19197126	964	0.000031	com.merriam-webster
294	19196242	1093	0.000027	edu.purdue
295	19190330	1579	0.000021	ca.ubc
296	19186772	710	0.000042	org.bitbucket
297	19186418	101	0.000349	com.wix
298	19183972	1075	0.000028	org.owasp
299	19182372	273	0.000099	com.ibm
300	19180986	955	0.000031	com.newsweek
301	19178290	688	0.000043	org.semver
302	19177950	251	0.000106	org.bbb
303	19170374	1403	0.000022	ca.sfu
304	19169910	2155	0.000017	com.discovery
305	19169706	1454	0.000022	uk.co.metro
306	19168948	403	0.000078	org.openstreetmap
307	19168128	928	0.000032	com.webs
308	19164224	277	0.000098	com.eepurl
309	19163312	399	0.000079	com.netdna-ssl
310	19162670	310	0.000089	com.wistia
311	19160768	1301	0.000023	app.netlify
312	19160242	910	0.000033	com.nasdaq
313	19158884	637	0.000047	gov.senate
314	19158734	135	0.000212	com.filesusr
315	19156792	528	0.000056	com.snapchat
316	19156732	317	0.000088	tv.twitch
317	19155472	1352	0.000023	uk.co.standard
318	19153634	977	0.000030	com.uk
319	19150454	669	0.000044	org.eff
320	19148706	1433	0.000022	io.gitlab
321	19146762	1609	0.000021	com.warnerbros
322	19145448	1025	0.000029	com.techradar
323	19144618	1173	0.000026	com.500px
324	19143630	1158	0.000026	com.pastebin
325	19141644	583	0.000051	gov.epa
326	19140574	820	0.000036	com.theconversation
327	19139294	1147	0.000026	org.semanticscholar
328	19139130	125	0.000238	com.rawgit
329	19138842	1228	0.000025	com.sky
330	19136534	1747	0.000019	com.flipboard
331	19134802	454	0.000065	com.ebay
332	19133816	204	0.000125	com.amazon-adsystem
333	19133028	599	0.000050	edu.cmu
334	19132926	1282	0.000024	edu.illinois
335	19132710	1327	0.000023	org.greenpeace
336	19131792	429	0.000073	com.optimizely
337	19130562	1668	0.000020	com.urbandictionary
338	19130312	201	0.000127	org.iana
339	19129546	609	0.000049	gov.house
340	19129270	98	0.000373	com.stripe
341	19127004	463	0.000064	org.opensource
342	19125378	247	0.000107	com.cloudinary
343	19122456	903	0.000033	edu.academia
344	19120070	1182	0.000025	org.mitre
345	19119992	1008	0.000030	gov.usgs
346	19119044	215	0.000122	net.sourceforge
347	19118924	2117	0.000018	com.channel4
348	19118136	1489	0.000022	uk.co.thesun
349	19117516	1558	0.000021	com.deadline
350	19114610	996	0.000030	com.thehill
351	19113576	897	0.000033	edu.umn
352	19113226	758	0.000040	gov.justice
353	19110186	1643	0.000020	org.maven
354	19110016	156	0.000173	com.addtoany
355	19108184	404	0.000077	com.criteo
356	19103220	2356	0.000016	com.freep
357	19101368	128	0.000230	com.paypalobjects
358	19098770	1133	0.000026	com.nikkei
359	19098758	457	0.000065	es.google
360	19096448	678	0.000043	org.oecd
361	19093328	1291	0.000024	org.postgresql
362	19092214	1941	0.000018	com.euronews
363	19091722	772	0.000039	gov.archives
364	19091262	1584	0.000021	com.reverbnation
365	19090376	1138	0.000026	uk.co.mirror
366	19088776	564	0.000053	com.kickstarter
367	19087460	2766	0.000013	edu.byu
368	19087070	1338	0.000023	edu.hbs
369	19085414	1382	0.000022	com.googlesource
370	19084726	2116	0.000018	edu.wustl
371	19084458	1010	0.000030	com.politico
372	19083222	2226	0.000017	org.nobelprize
373	19082584	1111	0.000027	com.dw
374	19082284	1022	0.000029	com.pingdom
375	19082194	548	0.000054	com.walmart
376	19078996	84	0.000405	net.jsfiddle
377	19078568	1284	0.000024	ch.ethz
378	19073766	2299	0.000016	gov.cia
379	19073478	983	0.000030	com.salon
380	19073042	715	0.000042	org.change
381	19072810	1184	0.000025	com.theglobeandmail
382	19071472	462	0.000064	com.elsevier
383	19071222	1837	0.000019	com.storify
384	19066438	137	0.000207	de.bund
385	19066384	121	0.000250	com.jimdo
386	19066344	1429	0.000022	edu.gatech
387	19064084	67	0.000527	net.typekit
388	19062738	1310	0.000023	com.digitaltrends
389	19062478	1286	0.000024	int.unfccc
390	19061312	810	0.000037	au.com.google
391	19059746	1026	0.000029	gov.treasury
392	19059514	2283	0.000016	com.mystrikingly
393	19059110	906	0.000033	com.britannica
394	19058170	1272	0.000024	edu.ucdavis
395	19057558	904	0.000033	uk.parliament
396	19056260	76	0.000468	me.fb
397	19054320	1027	0.000029	com.mdpi
398	19052746	1244	0.000024	com.aljazeera
399	19052144	207	0.000124	com.etsy
400	19052062	269	0.000100	net.azureedge
401	19051812	1063	0.000028	gov.fbi
402	19051720	1608	0.000021	ms.1drv
403	19048836	824	0.000036	com.bmj
404	19048654	1325	0.000023	de.mpg
405	19047424	2538	0.000014	com.virustotal
406	19047122	1106	0.000027	org.nejm
407	19046336	211	0.000123	com.tiktok
408	19045754	194	0.000131	org.nodejs
409	19044892	2566	0.000014	com.diigo
410	19043614	1183	0.000025	com.scmp
411	19042800	1095	0.000027	au.com.smh
412	19042258	718	0.000042	org.d3js
413	19041804	1311	0.000023	com.history
414	19041506	1178	0.000026	org.hrw
415	19039754	1235	0.000025	uk.ac.ucl
416	19038356	1213	0.000025	com.socialmediatoday
417	19035776	1038	0.000029	edu.uchicago
418	19033774	2710	0.000013	com.thecvf
419	19030356	968	0.000031	org.readthedocs
420	19030118	61	0.000612	com.googleadservices
421	19029814	1018	0.000029	org.jstor
422	19029070	833	0.000035	com.pinimg
423	19028696	2486	0.000015	com.oxforddictionaries
424	19028410	2145	0.000017	com.discogs
425	19027476	2568	0.000014	edu.buffalo
426	19023352	1810	0.000019	com.buzzfeednews
427	19023196	957	0.000031	watch.fb
428	19022492	1136	0.000026	org.sphinx-doc
429	19022490	1851	0.000018	com.spreaker
430	19022078	1548	0.000021	com.irishtimes
431	19021850	608	0.000049	com.biomedcentral
432	19019438	1452	0.000022	uk.ac.lse
433	19018308	469	0.000063	org.hbr
434	19018106	396	0.000079	com.statista
435	19017386	1045	0.000029	com.substack
436	19014182	285	0.000095	ru.ok
437	19013374	3509	0.000010	com.quizlet
438	19012470	807	0.000037	com.deviantart
439	19010602	1115	0.000027	org.undp
440	19005532	1959	0.000018	com.rt
441	19004652	1004	0.000030	org.ilo
442	19004416	3352	0.000011	cc.uxdesign
443	19003306	2234	0.000017	org.wto
444	19002690	1673	0.000020	org.rfc-editor
445	19002034	1302	0.000023	com.penguinrandomhouse
446	19001536	920	0.000032	de.spiegel
447	18999940	1318	0.000023	com.producthunt
448	18997582	705	0.000042	gov.sec
449	18997128	520	0.000057	com.meetup
450	18994718	2162	0.000017	com.ibtimes
451	18993898	1061	0.000028	com.sun
452	18992788	2120	0.000018	gov.federalreserve
453	18991624	1760	0.000019	edu.arizona
454	18990732	600	0.000050	edu.utah
455	18990642	1861	0.000018	com.newscientist
456	18989730	537	0.000055	com.gmail
457	18989410	1123	0.000026	net.java
458	18986644	1885	0.000018	com.itv
459	18986450	504	0.000059	com.ssl-images-amazon
460	18986010	243	0.000108	uk.org.ico
461	18985542	1781	0.000019	ca.blogspot
462	18985454	73	0.000503	net.akamaihd
463	18985010	840	0.000035	in.co.google
464	18984802	1391	0.000022	de.zeit
465	18984480	1005	0.000030	uk.co.thetimes
466	18984458	1042	0.000029	com.prweb
467	18983876	2661	0.000013	com.twitpic
468	18983790	1195	0.000025	io.pypa
469	18982648	2956	0.000012	com.openai
470	18980494	447	0.000067	net.imgix
471	18979850	1901	0.000018	com.martinfowler
472	18979062	242	0.000108	org.purl
473	18978596	655	0.000045	de.gesetze-im-internet
474	18978450	397	0.000079	net.themeforest
475	18978408	216	0.000121	jp.co.yahoo
476	18977992	1409	0.000022	edu.ufl
477	18977350	425	0.000073	com.atlassian
478	18975784	1373	0.000023	edu.duke
479	18974486	236	0.000111	to.amzn
480	18974130	2285	0.000016	edu.gmu
481	18974004	1056	0.000028	edu.nyu
482	18973414	575	0.000052	org.debian
483	18973148	1249	0.000024	com.jetbrains
484	18973108	235	0.000111	com.mapbox
485	18973012	296	0.000092	me.telegram
486	18972556	1287	0.000024	com.wikia
487	18972192	82	0.000409	com.oversightboard
488	18971130	342	0.000083	com.proofpoint
489	18970830	994	0.000030	com.jimdofree
490	18970782	2638	0.000014	org.nypl
491	18968834	806	0.000037	edu.brookings
492	18968470	2213	0.000017	org.wfp
493	18968366	2417	0.000015	mp.j
494	18967966	2624	0.000014	app.web
495	18967448	2363	0.000015	com.instructables
496	18966950	1267	0.000024	org.imf
497	18965952	1134	0.000026	org.unhcr
498	18965650	1610	0.000021	edu.virginia
499	18965586	3000	0.000012	ph.telegra
500	18963358	2122	0.000018	org.propublica
501	18963068	1655	0.000020	edu.brown
502	18961020	1430	0.000022	com.seattletimes
503	18960454	313	0.000088	io.shields
504	18960174	2348	0.000016	org.archlinux
505	18959730	355	0.000081	com.surveymonkey
506	18958892	747	0.000040	gov.state
507	18956864	1019	0.000029	com.yarnpkg
508	18956706	2142	0.000017	org.phys
509	18956672	1688	0.000020	org.unwomen
510	18955200	809	0.000037	com.fiverr
511	18954884	2930	0.000012	org.vim
512	18954634	3447	0.000010	com.instapaper
513	18953814	221	0.000119	com.eventbrite
514	18953136	878	0.000034	edu.psu
515	18948514	1599	0.000021	com.asahi
516	18948206	2338	0.000016	ca.ualberta
517	18947706	3130	0.000011	com.rd
518	18947260	748	0.000040	com.intel
519	18947088	2414	0.000015	com.gfycat
520	18946976	2649	0.000013	org.icrc
521	18944700	2467	0.000015	org.biorxiv
522	18941214	1658	0.000020	org.r-project
523	18939874	257	0.000105	com.aliyuncs
524	18936846	139	0.000205	com.weibo
525	18936474	1852	0.000018	com.gettyimages
526	18931798	525	0.000057	com.googlecode
527	18930224	3699	0.000010	com.plurk
528	18929320	2136	0.000017	org.unep
529	18929114	1737	0.000019	com.howstuffworks
530	18928922	2617	0.000014	com.udacity
531	18927534	1830	0.000019	edu.georgetown
532	18926738	2327	0.000016	com.esri
533	18925612	685	0.000043	uk.gov.service
534	18924274	2742	0.000013	jp.co.japantimes
535	18924242	2406	0.000015	com.kobo
536	18921598	555	0.000054	com.samsung
537	18921494	1447	0.000022	fr.gouvernement
538	18920518	2821	0.000013	org.wikibooks
539	18919550	2454	0.000015	it.scoop
540	18917542	3441	0.000010	net.openreview
541	18916950	2137	0.000017	es.abc
542	18913684	2422	0.000015	jp.geocities
543	18913494	2748	0.000013	edu.uoregon
544	18912900	2119	0.000018	google.ai
545	18909392	2655	0.000013	co.carrd
546	18907616	2115	0.000018	uk.co.huffingtonpost
547	18904404	690	0.000043	com.mashable
548	18903570	604	0.000049	com.steampowered
549	18902828	1730	0.000020	org.torproject
550	18902570	449	0.000066	com.netflix
551	18901684	2215	0.000017	google.research
552	18900478	2699	0.000013	at.ac.univie
553	18899428	2143	0.000017	edu.tufts
554	18897774	913	0.000033	com.thelancet
555	18894916	3791	0.000009	goog.translate
556	18893828	982	0.000030	org.ohchr
557	18892464	3880	0.000009	com.bravesites
558	18890432	2887	0.000012	org.rsf
559	18890342	1792	0.000019	gov.usembassy
560	18886662	3016	0.000012	com.architecturaldigest
561	18886098	1103	0.000027	cn.news
562	18885566	2367	0.000015	uk.bl
563	18883186	3407	0.000010	uk.co.walesonline
564	18882344	2461	0.000015	org.accessnow
565	18880856	2291	0.000016	com.france24
566	18879976	3309	0.000011	com.pearltrees
567	18878902	2815	0.000013	org.freedomhouse
568	18875528	219	0.000120	com.salesforce
569	18874868	2575	0.000014	org.scala-lang
570	18874266	1142	0.000026	be.google
571	18874220	2684	0.000013	re.appsto
572	18873246	2400	0.000015	org.ap
573	18873008	3183	0.000011	do.bit
574	18872314	2833	0.000012	com.sputniknews
575	18871926	2165	0.000017	org.americanprogress
576	18871186	1568	0.000021	com.chron
577	18871072	2792	0.000013	org.unaids
578	18869902	2689	0.000013	com.ajc
579	18868558	2251	0.000016	app.vercel
580	18868076	495	0.000060	com.visualstudio
581	18866746	1333	0.000023	net.daringfireball
582	18865464	2654	0.000013	org.csis
583	18863760	2350	0.000016	com.ew
584	18862092	1293	0.000024	link.page
585	18861286	2320	0.000016	fr.gouv.diplomatie
586	18860790	189	0.000133	ru.mail
587	18860636	852	0.000035	org.mediawiki
588	18859930	622	0.000048	com.thinkwithgoogle
589	18858612	2865	0.000012	com.duolingo
590	18855586	2112	0.000018	com.domaintools
591	18855394	291	0.000094	net.secureservercdn
592	18855236	2860	0.000012	com.biography
593	18853964	1648	0.000020	jp.ne.goo
594	18853434	1827	0.000019	com.lifewire
595	18853160	2479	0.000015	ie.independent
596	18850136	2903	0.000012	uk.ac.leeds
597	18849484	2951	0.000012	com.allure
598	18849452	2241	0.000016	com.timeout
599	18848206	3135	0.000011	org.cpj
600	18847016	4137	0.000009	com.bonanza
601	18844324	1640	0.000020	ca.globalnews
602	18843380	1774	0.000019	gov.in
603	18843170	1731	0.000020	com.images-amazon
604	18843116	2782	0.000013	com.depositphotos
605	18843074	1843	0.000018	com.thebalance
606	18842016	86	0.000401	com.livestream
607	18841682	287	0.000095	com.naver
608	18841558	477	0.000062	com.force
609	18840756	1422	0.000022	net.codecanyon
610	18838890	3346	0.000011	io.ghost
611	18838884	2419	0.000015	com.teenvogue
612	18838836	2366	0.000015	nz.co.stuff
613	18837438	2732	0.000013	com.123rf
614	18836408	2508	0.000014	com.motherjones
615	18836354	679	0.000043	int.wipo
616	18836332	2615	0.000014	edu.kit
617	18834684	1706	0.000020	com.routledge
618	18834336	695	0.000043	io.readthedocs
619	18834038	3692	0.000010	com.laweekly
620	18832208	441	0.000069	com.businesswire
621	18831266	2914	0.000012	org.oxfam
622	18829852	486	0.000062	com.adweek
623	18829616	2498	0.000014	edu.hawaii
624	18829216	3543	0.000010	com.udn
625	18828270	726	0.000042	com.canva
626	18828034	2985	0.000012	com.slides
627	18827908	627	0.000047	io.codepen
628	18827552	2664	0.000013	com.googlegroups
629	18826348	3901	0.000009	cn.org.china
630	18825700	2698	0.000013	com.coca-colacompany
631	18825602	888	0.000033	uk.co.pinterest
632	18823058	2551	0.000014	org.fas
633	18822830	1110	0.000027	net.clickbank
634	18822568	3099	0.000011	uk.co.timesonline
635	18822432	363	0.000081	net.php
636	18821410	2612	0.000014	edu.iastate
637	18821392	2144	0.000017	com.refinery29
638	18820314	988	0.000030	gov.dhs
639	18819896	4216	0.000008	com.alamy
640	18819802	991	0.000030	de.t-online
641	18817700	279	0.000097	com.iubenda
642	18817434	1612	0.000021	com.haaretz
643	18816616	1565	0.000021	mil.army
644	18816484	3053	0.000011	com.hm
645	18816156	1605	0.000021	uk.gov.ons
646	18813740	359	0.000081	mp.mailchi
647	18811428	3469	0.000010	org.heritage
648	18808928	1222	0.000025	org.eugdpr
649	18808748	2341	0.000016	za.co.google
650	18808348	563	0.000053	org.unicef
651	18808206	2938	0.000012	com.theonion
652	18807932	345	0.000083	com.akismet
653	18807216	150	0.000190	org.networkadvertising
654	18805384	834	0.000035	com.venturebeat
655	18803670	2671	0.000013	com.timesofisrael
656	18802968	3182	0.000011	com.ogilvy
657	18800978	134	0.000217	info.aboutads
658	18800344	1265	0.000024	tw.com.google
659	18799472	549	0.000054	com.fc2
660	18798170	2715	0.000013	com.theintercept
661	18798140	2377	0.000015	com.foreignpolicy
662	18797944	4009	0.000009	com.zara
663	18796758	3042	0.000012	org.project-syndicate
664	18796410	2790	0.000013	cn.gov.fmprc
665	18795586	541	0.000055	com.patreon
666	18794332	2912	0.000012	org.ballotpedia
667	18793786	3675	0.000010	uk.co.guim
668	18793668	1203	0.000025	com.thenextweb
669	18793250	2460	0.000015	nz.co.nzherald
670	18791956	2329	0.000016	gov.faa
671	18791606	664	0.000045	com.entrepreneur
672	18790298	1699	0.000020	com.nike
673	18788220	2257	0.000016	com.voanews
674	18785682	3646	0.000010	com.podomatic
675	18783652	394	0.000080	jp.ameblo
676	18781732	3665	0.000010	nz.co.scoop
677	18779826	2668	0.000013	com.jpost
678	18779770	755	0.000040	org.js
679	18778874	2589	0.000014	de.tagesspiegel
680	18778436	591	0.000051	com.gofundme
681	18778236	651	0.000046	it.placehold
682	18778130	692	0.000043	gov.nist
683	18777528	3090	0.000011	no.uib
684	18777264	3759	0.000010	com.clustrmaps
685	18776238	2533	0.000014	com.channelnewsasia
686	18775626	2332	0.000016	com.carto
687	18774970	2544	0.000014	edu.usf
688	18774910	4401	0.000008	uk.ac.essex
689	18774474	2203	0.000017	de.br
690	18773102	4256	0.000008	org.marxists
691	18769558	2826	0.000013	br.com.blogspot
692	18769428	680	0.000043	com.photobucket
693	18769216	3395	0.000010	com.parade
694	18768412	3731	0.000010	com.mongabay
695	18768244	742	0.000041	com.moz
696	18768070	3519	0.000010	ar.com.lanacion
697	18767480	1162	0.000026	com.digitaloceanspaces
698	18767252	3833	0.000009	com.scribblelive
699	18766284	3667	0.000010	ru.msk
700	18765148	1707	0.000020	org.oxfordjournals
701	18765018	1846	0.000018	com.speakerdeck
702	18764342	1177	0.000026	com.jekyllrb
703	18764272	1320	0.000023	com.imageshack
704	18763628	698	0.000043	com.withgoogle
705	18763518	2775	0.000013	com.fineartamerica
706	18763306	1627	0.000020	org.amnesty
707	18762512	2572	0.000014	org.unctad
708	18761940	3049	0.000012	int.au
709	18761162	109	0.000306	me.wa
710	18760936	2184	0.000017	org.ncsl
711	18759340	2857	0.000012	uk.org.nationaltrust
712	18758396	4435	0.000008	com.mysanantonio
713	18758278	3154	0.000011	fr.rfi
714	18757674	1300	0.000023	gov.federalregister
715	18756920	5968	0.000006	org.arkive
716	18756858	3385	0.000011	com.nationalreview
717	18756398	2232	0.000017	org.worldcat
718	18756388	4160	0.000009	com.turkishairlines
719	18756100	3077	0.000011	uk.ac.york
720	18755876	3029	0.000012	org.nationalgeographic
721	18755390	3360	0.000011	org.tigris
722	18755228	362	0.000081	com.adnxs
723	18753668	1765	0.000019	com.indianexpress
724	18753482	3293	0.000011	org.neocities
725	18751702	3058	0.000011	ly.genial
726	18750392	3274	0.000011	uk.co.penguin
727	18750242	919	0.000032	com.hootsuite
728	18750002	3494	0.000010	com.nme
729	18747334	2735	0.000013	com.kaggle
730	18746894	193	0.000131	com.discord
731	18746600	3357	0.000011	de.taz
732	18746298	3147	0.000011	edu.bc
733	18746190	3041	0.000012	tr.com.aa
734	18745544	3156	0.000011	com.cgtn
735	18745132	2431	0.000015	org.unodc
736	18744550	282	0.000096	gov.ftc
737	18743682	2344	0.000016	eu.politico
738	18743052	1364	0.000023	com.symantec
739	18740834	2520	0.000014	net.openid
740	18740752	3637	0.000010	il.ac.tau
741	18739398	2453	0.000015	ru.ria
742	18738948	3031	0.000012	com.allafrica
743	18738772	1172	0.000026	jp.ac.keio
744	18738200	2869	0.000012	edu.educause
745	18738156	3402	0.000010	org.firstmonday
746	18738122	2642	0.000014	org.wikidata
747	18737788	491	0.000061	com.placeholder
748	18734980	2900	0.000012	com.simonandschuster
749	18734860	3796	0.000009	org.amnestyusa
750	18734696	2168	0.000017	com.justia
751	18733308	2197	0.000017	ca.on.gov
752	18733088	3707	0.000010	uk.gov.scotland
753	18733026	4563	0.000008	com.flightradar24
754	18732944	4326	0.000008	com.interviewmagazine
755	18732742	3744	0.000010	com.afp
756	18732154	4112	0.000009	org.scala-sbt
757	18730992	2780	0.000013	ae.google
758	18730824	1303	0.000023	org.webkit
759	18729882	2947	0.000012	com.superuser
760	18729678	1014	0.000029	com.highcharts
761	18729020	3802	0.000009	com.wusa9
762	18728900	2550	0.000014	jp.nicovideo
763	18728094	2141	0.000017	gov.pa
764	18727990	3937	0.000009	org.one
765	18726672	2908	0.000012	edu.uky
766	18725034	3052	0.000011	in.businessinsider
767	18724474	3382	0.000011	org.hypotheses
768	18723378	2681	0.000013	org.wbur
769	18723236	569	0.000052	com.inc
770	18721066	1434	0.000022	com.upwork
771	18720908	4232	0.000008	org.sourcewatch
772	18720676	3366	0.000011	com.sciencealert
773	18720222	1316	0.000023	de.rki
774	18719114	3466	0.000010	org.royalsociety
775	18718524	2354	0.000016	ru.rbc
776	18718198	1034	0.000029	com.videojs
777	18717530	3383	0.000011	org.polymer-project
778	18717228	487	0.000062	ee.lin
779	18717008	3202	0.000011	org.texastribune
780	18716598	1054	0.000028	fm.last
781	18716564	3658	0.000010	se.gu
782	18715916	2181	0.000017	it.redd
783	18715552	1219	0.000025	com.smashingmagazine
784	18715536	2743	0.000013	org.undocs
785	18713460	2713	0.000013	org.iucn
786	18713092	3662	0.000010	com.hashicorp
787	18712944	1712	0.000020	scot.gov
788	18712564	976	0.000031	com.jwplayer
789	18712330	3843	0.000009	edu.wayne
790	18711122	576	0.000052	com.booking
791	18710906	803	0.000037	com.fandom
792	18710858	3850	0.000009	com.triplepundit
793	18709690	329	0.000085	com.hackerone
794	18709314	4235	0.000008	com.letterboxd
795	18707654	1151	0.000026	com.alexa
796	18707534	2258	0.000016	com.knightlab
797	18706964	770	0.000039	com.sedo
798	18706750	3394	0.000010	org.iucnredlist
799	18706380	1944	0.000018	com.firebaseapp
800	18705808	4036	0.000009	com.manta
801	18702546	3072	0.000011	au.com.theage
802	18701450	3475	0.000010	org.sierraclub
803	18700424	400	0.000078	com.onesignal
804	18700348	2652	0.000013	ru.kommersant
805	18700250	3379	0.000011	com.hasbro
806	18699356	3821	0.000009	edu.unu
807	18699146	3656	0.000010	com.crashlytics
808	18698894	817	0.000037	com.marketwatch
809	18698494	4071	0.000009	ru.aif
810	18698224	4419	0.000008	com.folkd
811	18698152	1285	0.000024	gov.uspto
812	18697734	3358	0.000011	net.ipsnews
813	18697484	2632	0.000014	org.unfpa
814	18697336	987	0.000030	com.stackexchange
815	18696510	3207	0.000011	ly.plot
816	18696184	532	0.000056	com.indeed
817	18695334	1667	0.000020	fr.blogspot
818	18695246	972	0.000031	com.css-tricks
819	18694382	933	0.000032	org.reactjs
820	18693292	4083	0.000009	com.marinetraffic
821	18692940	2706	0.000013	ru.rg
822	18692900	4365	0.000008	com.balenciaga
823	18692468	2561	0.000014	com.kinstacdn
824	18691710	2586	0.000014	build.bazel
825	18690746	545	0.000055	com.digg
826	18690190	3997	0.000009	jp.co.tepco
827	18690182	1457	0.000022	io.webflow
828	18690148	4685	0.000008	com.gmanetwork
829	18689994	3779	0.000009	org.rferl
830	18689870	4416	0.000008	kr.co.koreatimes
831	18689452	821	0.000036	com.oreilly
832	18689426	953	0.000031	gov.fcc
833	18689024	2957	0.000012	com.articulate
834	18688634	3054	0.000011	site.notion
835	18688256	2441	0.000015	int.reliefweb
836	18688218	2722	0.000013	com.insidehighered
837	18687872	1086	0.000027	so.notion
838	18687810	4466	0.000008	org.sfpl
839	18687672	3580	0.000010	uk.co.spectator
840	18687622	2840	0.000012	com.suntimes
841	18687452	917	0.000032	com.verisign
842	18686880	2967	0.000012	org.cfr
843	18686622	2819	0.000013	org.panda
844	18686298	1016	0.000029	com.mixcloud
845	18686002	83	0.000405	com.messenger
846	18685934	533	0.000056	jp.co.rakuten
847	18685764	4343	0.000008	com.upworthy
848	18685494	2644	0.000014	ru.kremlin
849	18684806	278	0.000097	com.sxsw
850	18684212	3542	0.000010	com.flippa
851	18684006	568	0.000052	com.mckinsey
852	18683956	2534	0.000014	net.convio
853	18683510	1231	0.000025	com.buffer
854	18683006	3100	0.000011	com.yougov
855	18682928	5110	0.000007	com.viki
856	18682436	4674	0.000008	org.birdlife
857	18681966	4413	0.000008	com.itsnicethat
858	18681374	650	0.000046	com.gartner
859	18681372	2927	0.000012	uk.gov.metoffice
860	18681084	494	0.000061	com.dmca
861	18680840	3597	0.000010	org.jenkins-ci
862	18680358	3089	0.000011	int.iom
863	18678670	4082	0.000009	com.iconarchive
864	18677384	4480	0.000008	com.oriflame
865	18676518	4538	0.000008	net.middleeasteye
866	18675854	5041	0.000007	com.waitbutwhy
867	18675534	4316	0.000008	org.pen
868	18675274	2830	0.000013	fm.omny
869	18674282	3902	0.000009	org.icij
870	18674044	4188	0.000008	org.constitutioncenter
871	18673972	3857	0.000009	ch.qos
872	18673474	4070	0.000009	com.9to5google
873	18673432	3526	0.000010	uk.gov.companieshouse
874	18673406	3994	0.000009	uk.ac.sussex
875	18673258	3266	0.000011	com.foreignaffairs
876	18673246	2835	0.000012	com.news24
877	18673204	4132	0.000009	re.cli
878	18672690	4227	0.000008	jp.ac.kobe-u
879	18671714	902	0.000033	br.com.uol
880	18671552	3718	0.000010	com.nybooks
881	18671446	1818	0.000019	com.over-blog
882	18671362	5578	0.000006	com.symbaloo
883	18669612	3170	0.000011	uk.co.bbci
884	18669200	934	0.000032	com.pubmatic
885	18669010	2385	0.000015	com.scene7
886	18668810	3467	0.000010	org.wikileaks
887	18667242	4637	0.000008	org.foodandwaterwatch
888	18666438	3305	0.000011	at.derstandard
889	18666074	876	0.000034	com.zoho
890	18665410	3167	0.000011	org.adb
891	18664362	3518	0.000010	com.benzinga
892	18663988	730	0.000041	com.usnews
893	18663450	5592	0.000006	io.postach
894	18663030	3547	0.000010	com.palgrave
895	18662464	1109	0.000027	net.media
896	18662120	4091	0.000009	net.datasociety
897	18661426	540	0.000055	com.googleoptimize
898	18661044	4116	0.000009	au.com.heraldsun
899	18659068	3440	0.000010	ru.kp
900	18657988	2675	0.000013	com.thenation
901	18657690	663	0.000045	me.zalo
902	18657052	3001	0.000012	com.unity
903	18655796	1635	0.000020	org.altervista
904	18654580	4826	0.000007	it.polito
905	18654490	4482	0.000008	edu.odu
906	18654200	3171	0.000011	org.sonatype
907	18653790	2853	0.000012	net.vnexpress
908	18653284	735	0.000041	com.alibaba
909	18652510	4454	0.000008	com.muckrack
910	18652400	2937	0.000012	com.lexology
911	18652270	4962	0.000007	kr.co.hani
912	18650818	3723	0.000010	com.tradingeconomics
913	18650606	3882	0.000009	com.study
914	18650594	595	0.000050	com.airbnb
915	18649776	3663	0.000010	gov.ustr
916	18649674	4900	0.000007	com.theodysseyonline
917	18649520	3724	0.000010	uk.gov.homeoffice
918	18648278	1031	0.000029	com.pcmag
919	18647134	633	0.000047	org.joomla
920	18645770	3367	0.000011	br.scielo
921	18645448	74	0.000486	com.trustpilot
922	18644890	5552	0.000006	au.edu.vu
923	18644710	3127	0.000011	tw.com.pchome
924	18644260	725	0.000042	com.splashthat
925	18643560	2745	0.000013	ca.citizenlab
926	18642628	4599	0.000008	com.condenast
927	18642354	1676	0.000020	com.techrepublic
928	18641430	2125	0.000018	io.pantheonsite
929	18641322	3281	0.000011	ru.cbr
930	18641240	2866	0.000012	ca.uwaterloo
931	18640912	4384	0.000008	uk.co.belfasttelegraph
932	18640688	597	0.000050	com.wufoo
933	18639194	3231	0.000011	org.ellenmacarthurfoundation
934	18639046	4821	0.000007	com.zimbio
935	18638824	3321	0.000011	com.rabbitmq
936	18638422	547	0.000054	com.herokuapp
937	18636852	5657	0.000006	org.cgsociety
938	18634912	5704	0.000006	in.teletype
939	18634900	531	0.000056	com.aol
940	18634026	3752	0.000010	edu.ucpress
941	18633976	3375	0.000011	com.scotsman
942	18633494	3277	0.000011	com.kroger
943	18632272	405	0.000077	com.constantcontact
944	18631930	870	0.000034	com.emarketer
945	18630874	5643	0.000006	com.dbs
946	18630838	4407	0.000008	au.edu.deakin
947	18630630	3315	0.000011	org.osce
948	18629458	2971	0.000012	com.euractiv
949	18628642	4788	0.000007	com.latercera
950	18626102	3881	0.000009	com.bloombergquint
951	18625580	952	0.000031	com.digitalocean
952	18625102	3638	0.000010	org.ushmm
953	18624888	3751	0.000010	com.lawfareblog
954	18624664	4877	0.000007	ke.co.google
955	18624400	3835	0.000009	com.thenationalnews
956	18624378	4716	0.000007	com.kongregate
957	18624240	5132	0.000007	com.apsense
958	18624020	1342	0.000023	com.nvidia
959	18623838	617	0.000048	gov.copyright
960	18623504	4437	0.000008	com.jacobinmag
961	18623396	2934	0.000012	net.dwcdn
962	18622680	643	0.000046	com.accenture
963	18622320	4529	0.000008	uk.ac.soas
964	18621166	3445	0.000010	de.test
965	18620568	1661	0.000020	com.createjs
966	18620140	3218	0.000011	com.obsproject
967	18619976	2892	0.000012	org.gnupg
968	18619870	4318	0.000008	com.washingtonian
969	18619392	4908	0.000007	uk.co.birminghammail
970	18619154	4548	0.000008	io.meduza
971	18619058	4034	0.000009	ru.mid
972	18618882	1207	0.000025	org.golang
973	18618534	3930	0.000009	org.cgiar
974	18617116	2411	0.000015	co.pcdn
975	18616304	2613	0.000014	com.olark
976	18615562	1007	0.000030	com.gumroad
977	18613652	2755	0.000013	ru.tass
978	18613510	4825	0.000007	com.selfridges
979	18612814	3770	0.000009	fr.capital
980	18612214	4391	0.000008	za.co.mg
981	18612168	911	0.000033	net.atlassian
982	18612044	844	0.000035	com.redhat
983	18611518	1749	0.000019	com.indiegogo
984	18611438	5009	0.000007	edu.utep
985	18610856	1727	0.000020	org.linuxfoundation
986	18610446	956	0.000031	com.att
987	18609186	2890	0.000012	org.transparency
988	18608588	3918	0.000009	com.encyclopedia
989	18606828	72	0.000505	com.oculus
990	18606772	699	0.000043	com.psychologytoday
991	18606698	3091	0.000011	com.sharefile
992	18606504	151	0.000189	org.whatwg
993	18606354	754	0.000040	org.poynter
994	18606268	3386	0.000011	com.alchemer
995	18604648	707	0.000042	co.ibb
996	18604432	286	0.000095	com.caniuse
997	18604402	2738	0.000013	com.springeropen
998	18604386	2490	0.000014	studio.flourish
999	18604138	4373	0.000008	com.googledrive
1000	18604014	4490	0.000008	tw.com.books

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

Important news for users of Common Crawl data: we are introducing CloudFront as a new way to access Common Crawl data as part of Amazon Web Services’ registry of open data

March 1, 2022Sebastian Nagel

Ten years ago(!) Common Crawl joined AWS’s Open Data Sponsorships program, hosted on S3, with free access to everyone. Since then, the dataset has expanded (by petabytes!) and our community of users has seen extraordinary growth. With growth comes change, therefore…

Following are new measures to accommodate the number and volume of requests for Common Crawl data, and to ensure efficient and stable access to the data:

1. Common Crawl data is now available on CloudFront under the URL base prefixes*:

https://data.commoncrawl.org/ or
https://ds5q9oxwqwsfj.cloudfront.net/

* Data is accessible via http:// or https://.

As of Monday, April 4, 2022

2. Access to data from the Amazon cloud using the S3 API will be restricted to authenticated AWS users, and unsigned access to s3://commoncrawl/ will be disabled. See Q&A for further details.

3. To access data from outside the Amazon cloud, via HTTP(S), a new URL prefix – https://data.commoncrawl.org/ – must be used instead of https://commoncrawl.s3.amazonaws.com/.

Q & A

Q: How can I identify whether my code is using unauthenticated S3 access?

A: Authenticated access is the default, but you should verify that your code is not configured otherwise; code fragments requesting unauthenticated access could be (but are not limited to):

AWS CLI with the command-line option --no-sign-request:
```
aws --no-sign-request s3 cp s3://commoncrawl/...
```

Python using boto3 and botocore.UNSIGNED:

import boto3
import botocore

s3client = boto3.client('s3', config=botocore.client.Config(signature_version=botocore.UNSIGNED))

Hadoop or Spark (various programming languages): usage of AnonymousAWSCredentialsProvider

Q: Where can I find up-to-date documentation and examples that are consistent with these new protocols?

A: We will update our documentation and examples to read data either via HTTP(S) using the new CloudFront access, or via S3 and authenticated access. We will also update all data download links. It will take a few days until everything is updated. Thanks for your patience!

Q: Are range requests supported?

A: Yes, range requests are supported by CloudFront same as for S3, see the CloudFront docs about RangeGETs.

Q: What is the recommended access method on AWS but in a different region (not us-east-1)?

A: We recommend that you run your computing workload in the same region (us-east-1) as the Common Crawl dataset whenever possible. If you have a specific, ongoing need to run computing workloads using Common Crawl in another AWS region, the AWS Open Data team would like to hear more about your use case ([email protected]).

Q: What are typical error messages indicating that unauthenticated access is used?

A: The HTTP response status code is a 403 Forbidden. However, also a restrictive IAM policy on the user’s side could deny access to s3://commoncrawl/ using the S3 API.
Two examples for error messages related to unauthenticated access to s3://commoncrawl/:

> curl https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2022-05/warc.paths.gz
<!--?xml version="1.0" encoding="UTF-8"?-->
<error><code>AccessDenied</code><message>Access Denied</message>...</error>

> aws --no-sign-request s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2022-05/warc.paths.gz .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

January 2022 crawl archive now available

February 2, 2022Sebastian Nagel

The crawl archive for January 2022 is now available! The data was crawled January 16 – 29 and contains 2.95 billion web pages or 320 TiB of uncompressed content. It includes page captures of 1.35 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The January crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2022-05/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2022-05/segment.paths.gz	100
WARC files	CC-MAIN-2022-05/warc.paths.gz	72000	73.5
WAT files	CC-MAIN-2022-05/wat.paths.gz	72000	19.85
WET files	CC-MAIN-2022-05/wet.paths.gz	72000	8.63
Robots.txt files	CC-MAIN-2022-05/robotstxt.paths.gz	72000	0.14
Non-200 responses files	CC-MAIN-2022-05/non200responses.paths.gz	72000	1.79
URL index files	CC-MAIN-2022-05/cc-index.paths.gz	302	0.22

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2022-05/. Also the columnar index has been updated to contain this crawl.

November/December 2021 crawl archive now available

December 14, 2021Sebastian Nagel

The crawl archive for November/December 2021 is now available! The data was crawled Nov 26 – Dec 9 and contains 2.5 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.2 billion new URLs, not visited in any of our prior crawls.

Improvements

The column url_host_name_reversed was added to the columnar index. It holds the host name in reverse domain name notation (com.example.www) which is more efficient to query. In order to make use of the new column please use the updated table schema. On Amazon Athena you need to recreate table by running the latest table creation statement. Further details are found in the corresponding pull request.

The columnar index is now built using Spark version 3.2.0 and Parquet MR 1.12.1 – these upgrades allow us to go for further improvements next year. Let us know if you observe any incompatibilities using this index to query data from the Nov/Dec 2021 crawl.

The crawler is now based on Tika 2.1.0 for detection of MIME/content types and character sets used to encode HTML documents.

Archive Location and Download

The November/December crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-49/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-49/segment.paths.gz	100
WARC files	CC-MAIN-2021-49/warc.paths.gz	64000	68.66
WAT files	CC-MAIN-2021-49/wat.paths.gz	64000	16.66
WET files	CC-MAIN-2021-49/wet.paths.gz	64000	7.18
Robots.txt files	CC-MAIN-2021-49/robotstxt.paths.gz	64000	0.15
Non-200 responses files	CC-MAIN-2021-49/non200responses.paths.gz	64000	2.29
URL index files	CC-MAIN-2021-49/cc-index.paths.gz	302	0.2

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-49/. Also the columnar index has been updated to contain this crawl.

October 2021 crawl archive now available

November 1, 2021Sebastian Nagel

The crawl archive for October 2021 is now available! The data was crawled Oct 15 – 28 and contains 3.3 billion web pages or 360 TiB of uncompressed content. It includes page captures of 1.3 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The October crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-43/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-43/segment.paths.gz	100
WARC files	CC-MAIN-2021-43/warc.paths.gz	72000	85.11
WAT files	CC-MAIN-2021-43/wat.paths.gz	72000	22.42
WET files	CC-MAIN-2021-43/wet.paths.gz	72000	9.79
Robots.txt files	CC-MAIN-2021-43/robotstxt.paths.gz	72000	0.15
Non-200 responses files	CC-MAIN-2021-43/non200responses.paths.gz	72000	2.11
URL index files	CC-MAIN-2021-43/cc-index.paths.gz	302	0.25

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-43/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs June, July/August and September 2021

October 8, 2021Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of June, July/August and September 2021. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases. You may also visit the projects cc-webgraph and cc-pyspark which include all scripts and tools required to construct the graphs. Instructions to explore the graphs in the webgraph format are given in our collection of webgraph notebooks.

Host-level graph

The graph consists of 766 million nodes and 4.95 billion edges. Both hyperlinks and HTTP redirects and link headers are used as edges to span up the graph. All types of links are included, including pure “technical” ones pointing to images, JavaScript libraries, web fonts, etc. However, only host names with a valid IANA TLD are used. Consequently, URLs with an IP address as host component are not taken into account for building the host-level graph.

There are 701 million dangling nodes (91.6%) and the largest strongly connected component contains 47.0 million (6.1%) nodes. Dangling nodes stem from

hosts that have not been crawled, yet are pointed to from a link on a crawled page
hosts without any links pointing to a different host name
or hosts which did only return an error page (eg. HTTP 404)

Host names in the graph are in reverse domain name notation and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 766 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2021-jun-jul-sep/host/. Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2021-jun-jul-sep/host/ as prefix to access the files from everywhere.

Please note that the text representation of the host-level graph is shipped in 40 gzip-compressed files listed in two path listings – one for the nodes (vertices), one for the edges (arcs). First, download the paths listing and decompress it using “gzip”. By adding the prefix s3://commoncrawl/ or https://data.commoncrawl.org/ to each line in the path listing you get the list of URLs to download the entire graph.

Size	File	Description
4.50 GB	cc-main-2021-jun-jul-sep-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 16 vertices files
20.76 GB	cc-main-2021-jun-jul-sep-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 24 edges files

8.43 GB	cc-main-2021-jun-jul-sep-host.graph	graph in BVGraph format
2 kB	cc-main-2021-jun-jul-sep-host.properties

9.83 GB	cc-main-2021-jun-jul-sep-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2021-jun-jul-sep-host-t.properties

1 kB	cc-main-2021-jun-jul-sep-host.stats	WebGraph statistics

11.06 GB	cc-main-2021-jun-jul-sep-host-ranks.txt.gz	harmonic centrality and pagerank

Domain-level graph

The domain graph is built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org. Version (commit) a5b046d of the public suffix list was used (commit date 2021-10-06).

The domain-level graph has 88 million nodes and 1.56 billion edges. 49% or 43 million nodes are dangling nodes, the largest strongly connected component covers 35 million or 40% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2021-jun-jul-sep/domain/ resp. https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2021-jun-jul-sep/domain/.

Download files of the Common Crawl Jun/Jul/Sep 2021 domain-level webgraph

Size	File	Description
0.61 GB	cc-main-2021-jun-jul-sep-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
6.34 GB	cc-main-2021-jun-jul-sep-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

3.59 GB	cc-main-2021-jun-jul-sep-domain.graph	graph in BVGraph format
2 kB	cc-main-2021-jun-jul-sep-domain.properties

3.44 GB	cc-main-2021-jun-jul-sep-domain-t.graph	transpose of the graph
2 kB	cc-main-2021-jun-jul-sep-domain-t.properties

1 kB	cc-main-2021-jun-jul-sep-domain.stats	WebGraph statistics

1.89 GB	cc-main-2021-jun-jul-sep-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 88 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Jun/Jul/Sep 2021)

harmonic centrality rank	hc value	page rank	page rank value	reversed domain name
1	31634524	1	0.018526	com.googleapis
2	30590448	3	0.013879	com.facebook
3	29344102	2	0.015082	com.google
4	26579208	5	0.007506	org.w
5	26490886	4	0.008344	com.twitter
6	26194504	6	0.007270	com.youtube
7	25598364	8	0.006802	com.instagram
8	24808618	7	0.006829	com.googletagmanager
9	24314334	9	0.004946	org.gmpg
10	23501518	14	0.003442	com.linkedin
11	23191984	11	0.003661	com.gstatic
12	22552330	10	0.004093	com.cloudflare
13	22306348	17	0.001939	com.gravatar
14	22254410	12	0.003556	org.wordpress
15	22159662	22	0.001575	com.pinterest
16	21706482	23	0.001396	org.wikipedia
17	21593592	16	0.001952	com.apple
18	21537622	37	0.001024	com.wordpress
19	21459498	27	0.001176	com.vimeo
20	21360146	41	0.000898	be.youtu
21	21289640	15	0.002252	com.bootstrapcdn
22	21211768	20	0.001717	com.jquery
23	21069122	28	0.001166	com.microsoft
24	21014760	54	0.000699	com.blogspot
25	20975968	46	0.000793	com.amazon
26	20916000	45	0.000798	gl.goo
27	20909476	50	0.000720	com.wp
28	20878568	62	0.000580	ly.bit
29	20832704	35	0.001053	com.google-analytics
30	20815940	33	0.001068	com.amazonaws
31	20731242	77	0.000417	com.tumblr
32	20705584	42	0.000841	org.mozilla
33	20703870	31	0.001080	net.cloudfront
34	20700046	18	0.001797	com.adobe
35	20687574	49	0.000720	eu.europa
36	20682862	19	0.001772	com.github
37	20663412	34	0.001054	net.jsdelivr
38	20641588	29	0.001134	com.wixstatic
39	20534372	75	0.000442	com.googleusercontent
40	20518204	21	0.001630	com.fontawesome
41	20499484	96	0.000366	com.yahoo
42	20489920	51	0.000711	com.paypal
43	20470068	57	0.000645	co.t
44	20460526	47	0.000784	com.whatsapp
45	20453542	53	0.000704	com.flickr
46	20429610	106	0.000315	com.reddit
47	20405542	72	0.000449	com.medium
48	20396096	38	0.000956	com.googlesyndication
49	20388958	48	0.000748	io.github
50	20364584	125	0.000241	com.nytimes
51	20363658	59	0.000616	org.w3
52	20350454	98	0.000358	com.weebly
53	20330952	97	0.000363	org.creativecommons
54	20324280	65	0.000510	com.shopify
55	20315624	110	0.000281	com.soundcloud
56	20308274	85	0.000402	me.wp
57	20300136	60	0.000610	org.schema
58	20300004	32	0.001072	ru.yandex
59	20272874	132	0.000215	com.forbes
60	20266896	64	0.000527	com.vk
61	20216640	114	0.000268	com.spotify
62	20214818	175	0.000146	com.cnn
63	20172836	52	0.000708	net.doubleclick
64	20156654	56	0.000653	com.addthis
65	20153642	190	0.000132	uk.co.bbc
66	20151882	230	0.000113	com.wsj
67	20149822	135	0.000208	gov.nih
68	20146194	40	0.000924	com.baidu
69	20144952	161	0.000166	com.theguardian
70	20127172	151	0.000179	int.who
71	20121134	240	0.000108	com.bloomberg
72	20115270	130	0.000217	org.archive
73	20111568	163	0.000163	com.giphy
74	20107092	94	0.000388	com.list-manage
75	20100830	192	0.000130	org.wikimedia
76	20095176	55	0.000682	com.macromedia
77	20091806	229	0.000113	com.oracle
78	20081572	145	0.000193	com.imdb
79	20074580	187	0.000134	com.businessinsider
80	20066912	297	0.000091	edu.mit
81	20066436	300	0.000090	edu.stanford
82	20061314	115	0.000267	com.mailchimp
83	20059906	44	0.000800	net.facebook
84	20058050	140	0.000204	us.zoom
85	20056682	426	0.000065	com.googleblog
86	20053210	171	0.000153	com.unsplash
87	20047020	284	0.000095	com.reuters
88	20043722	185	0.000137	com.imgur
89	20035754	142	0.000199	com.wixsite
90	20025300	191	0.000131	com.stackoverflow
91	20022100	128	0.000223	com.weibo
92	20016560	162	0.000163	com.issuu
93	20016026	423	0.000065	gov.nasa
94	20014282	36	0.001043	net.fbcdn
95	20013910	273	0.000098	com.android
96	20012778	119	0.000259	me.t
97	20012366	169	0.000160	org.ietf
98	20004530	139	0.000206	com.ytimg
99	20001182	148	0.000183	org.apache
100	19999464	318	0.000086	com.theverge
101	19993026	325	0.000084	com.slack
102	19992850	258	0.000103	edu.harvard
103	19989870	224	0.000116	com.washingtonpost
104	19988302	268	0.000099	com.bbc
105	19986642	460	0.000060	edu.cornell
106	19984712	155	0.000174	com.ft
107	19979856	117	0.000261	com.npmjs
108	19978744	416	0.000066	com.ted
109	19962738	424	0.000065	com.myspace
110	19957322	352	0.000078	com.wired
111	19956296	526	0.000052	com.livejournal
112	19945484	355	0.000078	com.appspot
113	19944274	263	0.000101	org.un
114	19943458	215	0.000119	org.gnu
115	19933112	390	0.000070	com.goodreads
116	19932600	69	0.000476	com.godaddy
117	19931522	387	0.000071	org.hbr
118	19917440	309	0.000087	org.npr
119	19916612	322	0.000085	com.prnewswire
120	19911628	332	0.000083	net.researchgate
121	19910122	307	0.000088	com.githubusercontent
122	19909276	24	0.001390	io.polyfill
123	19908276	286	0.000095	com.wiley
124	19905960	247	0.000106	com.tiktok
125	19894990	193	0.000130	com.blogger
126	19892346	71	0.000466	com.unpkg
127	19884978	102	0.000322	de.google
128	19869556	432	0.000064	com.gmail
129	19869236	565	0.000049	com.vice
130	19868406	627	0.000045	org.chromium
131	19867146	146	0.000186	gle.forms
132	19866916	113	0.000269	com.youtube-nocookie
133	19866834	179	0.000142	org.ampproject
134	19856608	354	0.000078	com.time
135	19856272	668	0.000043	edu.upenn
136	19856040	370	0.000074	com.example
137	19854818	633	0.000045	com.economist
138	19850884	727	0.000040	com.evernote
139	19842434	500	0.000055	com.steampowered
140	19841380	692	0.000042	google.blog
141	19841354	462	0.000060	com.theatlantic
142	19840036	586	0.000047	org.weforum
143	19837596	628	0.000045	com.deviantart
144	19837072	239	0.000109	uk.co.google
145	19833470	357	0.000077	org.arxiv
146	19833402	395	0.000070	com.scribd
147	19831882	407	0.000067	uk.co.telegraph
148	19829878	368	0.000074	com.huffingtonpost
149	19829876	642	0.000045	com.mysql
150	19821800	555	0.000050	org.worldbank
151	19818100	242	0.000107	com.sciencedirect
152	19816676	347	0.000080	com.nature
153	19814992	228	0.000113	com.twimg
154	19812336	120	0.000259	com.statcounter
155	19810098	246	0.000106	org.acm
156	19805214	554	0.000050	org.ieee
157	19799686	381	0.000071	com.fastcompany
158	19796892	255	0.000104	org.python
159	19794906	701	0.000041	com.apnews
160	19790820	431	0.000064	com.meetup
161	19790276	738	0.000040	com.qz
162	19789858	581	0.000047	com.globenewswire
163	19788416	365	0.000074	com.docker
164	19788402	444	0.000062	com.pixabay
165	19788038	453	0.000061	uk.co.dailymail
166	19784442	321	0.000085	com.springer
167	19783552	251	0.000105	com.bandcamp
168	19779288	256	0.000103	net.behance
169	19778076	361	0.000075	com.gitlab
170	19768948	603	0.000046	com.git-scm
171	19764224	540	0.000051	io.readthedocs
172	19763982	715	0.000041	com.engadget
173	19763034	647	0.000044	com.trello
174	19761140	183	0.000138	com.bing
175	19758120	346	0.000080	com.usatoday
176	19757326	220	0.000117	com.squarespace
177	19752428	158	0.000169	com.yelp
178	19747884	353	0.000078	com.dribbble
179	19747756	417	0.000066	com.digg
180	19747532	143	0.000199	com.dropbox
181	19744818	276	0.000097	com.ibm
182	19734092	464	0.000060	uk.co.independent
183	19733310	398	0.000069	com.w3schools
184	19723800	544	0.000051	ee.linktr
185	19723334	674	0.000043	uk.co.blogspot
186	19723200	412	0.000067	com.staticflickr
187	19721228	551	0.000050	com.pexels
188	19721224	144	0.000194	gov.cdc
189	19718802	539	0.000051	org.pbs
190	19718388	775	0.000038	com.stackexchange
191	19718222	703	0.000041	org.cambridge
192	19717352	1103	0.000029	org.eclipse
193	19716696	43	0.000819	com.fb
194	19712516	678	0.000042	edu.columbia
195	19710308	100	0.000351	com.wix
196	19706256	730	0.000040	edu.washington
197	19705612	290	0.000094	com.tinyurl
198	19705126	367	0.000074	com.sagepub
199	19704258	673	0.000043	me.about
200	19699044	219	0.000117	net.slideshare
201	19694818	653	0.000044	org.sciencemag
202	19689094	294	0.000091	org.pewresearch
203	19685770	601	0.000046	com.withgoogle
204	19684956	458	0.000061	com.herokuapp
205	19675188	440	0.000063	com.quora
206	19673070	123	0.000246	com.sharethis
207	19672408	39	0.000941	com.qq
208	19671982	176	0.000145	org.doi
209	19670088	438	0.000063	co.ibb
210	19667674	614	0.000046	com.newyorker
211	19664724	1183	0.000027	com.nike
212	19664240	316	0.000087	com.typeform
213	19658254	248	0.000106	com.outlook
214	19654436	736	0.000040	com.hp
215	19654104	791	0.000037	com.foxnews
216	19651026	234	0.000112	com.cloudinary
217	19648214	933	0.000035	edu.princeton
218	19646574	572	0.000048	com.moz
219	19642260	437	0.000063	com.getpocket
220	19639898	485	0.000057	com.nbcnews
221	19638836	645	0.000044	org.bitbucket
222	19635872	202	0.000124	page.g
223	19635142	154	0.000176	gov.privacyshield
224	19634924	277	0.000096	com.disqus
225	19624936	141	0.000203	com.opera
226	19623614	463	0.000060	com.airbnb
227	19620318	960	0.000034	com.dropboxusercontent
228	19618624	372	0.000073	com.force
229	19617796	923	0.000035	co.elastic
230	19614006	214	0.000119	com.wpengine
231	19613164	679	0.000042	org.semver
232	19613160	305	0.000089	com.typepad
233	19611254	972	0.000033	com.nypost
234	19610798	726	0.000040	com.ubuntu
235	19610070	1263	0.000025	se.haxx
236	19605234	303	0.000089	com.live
237	19603428	683	0.000042	au.net.abc
238	19603360	465	0.000060	com.mozilla
239	19599764	382	0.000071	com.criteo
240	19593184	1232	0.000026	uk.co.thesun
241	19581354	1434	0.000023	edu.rutgers
242	19581020	278	0.000096	com.feedburner
243	19577288	969	0.000033	com.politico
244	19572414	1054	0.000030	co.g
245	19572372	1849	0.000019	com.instructables
246	19570874	778	0.000038	com.sap
247	19568694	1123	0.000028	org.greenpeace
248	19564134	1127	0.000028	org.kernel
249	19562280	1650	0.000022	com.googlesource
250	19561484	124	0.000245	com.filesusr
251	19559444	1150	0.000028	com.unity3d
252	19558056	682	0.000042	com.freepik
253	19557106	542	0.000051	com.fortune
254	19553954	638	0.000045	uk.ac.ox
255	19553634	209	0.000121	org.iana
256	19551872	241	0.000108	com.eepurl
257	19551398	958	0.000034	com.ssrn
258	19551010	866	0.000035	com.nvidia
259	19549544	1569	0.000022	com.storify
260	19549036	1000	0.000032	com.sun
261	19548514	624	0.000045	uk.co.eventbrite
262	19544466	829	0.000036	edu.jhu
263	19541150	607	0.000046	net.azurewebsites
264	19539076	1291	0.000025	com.reverbnation
265	19538660	439	0.000063	gov.fda
266	19538044	99	0.000351	com.stripe
267	19534860	927	0.000035	com.podbean
268	19532612	341	0.000082	net.windows
269	19530966	1256	0.000025	uk.co.ebay
270	19530706	260	0.000102	com.calendly
271	19526632	804	0.000037	com.chrome
272	19525016	1804	0.000020	com.martinfowler
273	19524922	926	0.000035	edu.academia
274	19523484	568	0.000049	site.business
275	19521568	231	0.000113	com.office
276	19521106	328	0.000084	com.netdna-ssl
277	19518874	253	0.000105	com.newsweek
278	19516880	210	0.000120	tv.twitch
279	19515182	1220	0.000026	com.vogue
280	19514726	2307	0.000017	com.diigo
281	19514662	1214	0.000026	org.postgresql
282	19513292	691	0.000042	com.xinhuanet
283	19508378	1368	0.000024	de.mpg
284	19508326	525	0.000052	com.squareup
285	19507892	408	0.000067	org.debian
286	19502082	121	0.000252	com.paypalobjects
287	19501986	716	0.000041	gov.senate
288	19501788	2392	0.000016	com.pearltrees
289	19500240	1147	0.000028	com.500px
290	19499046	343	0.000081	com.googlecode
291	19497980	712	0.000041	org.change
292	19497516	491	0.000056	com.tandfonline
293	19496848	67	0.000505	net.akamaihd
294	19492076	1265	0.000025	com.aljazeera
295	19492054	706	0.000041	com.qualtrics
296	19490750	783	0.000037	com.theconversation
297	19490388	1078	0.000029	com.theglobeandmail
298	19486228	134	0.000211	de.bund
299	19485788	1299	0.000025	edu.illinois
300	19485786	327	0.000084	com.cnbc
301	19484140	858	0.000036	uk.co.guardian
302	19483768	483	0.000057	com.msn
303	19475416	129	0.000219	com.rawgit
304	19472372	474	0.000059	com.stumbleupon
305	19469278	198	0.000125	net.sourceforge
306	19468982	356	0.000077	com.optimizely
307	19468656	470	0.000059	org.openstreetmap
308	19467526	335	0.000083	com.techcrunch
309	19465998	434	0.000064	com.ssl-images-amazon
310	19464188	1405	0.000023	edu.ufl
311	19462912	1395	0.000023	edu.gatech
312	19462816	165	0.000162	com.hubspot
313	19462736	310	0.000087	com.mapbox
314	19462404	441	0.000063	com.go
315	19459072	608	0.000046	gov.noaa
316	19458846	1470	0.000023	com.channel4
317	19458794	1237	0.000026	ca.sfu
318	19457698	592	0.000047	com.healthline
319	19457660	794	0.000037	org.fao
320	19457528	507	0.000054	ca.google
321	19457170	2431	0.000016	com.wattpad
322	19454268	1280	0.000025	uk.co.standard
323	19452678	667	0.000043	gov.house
324	19451936	1309	0.000025	uk.co.wired
325	19450294	1862	0.000019	com.invisionapp
326	19449754	743	0.000039	com.pinimg
327	19449738	184	0.000138	com.amazon-adsystem
328	19449480	1836	0.000019	org.maven
329	19446242	2507	0.000015	com.openai
330	19443698	803	0.000037	org.pypi
331	19442142	262	0.000101	net.azureedge
332	19441000	490	0.000056	com.kickstarter
333	19439570	1962	0.000019	uk.bl
334	19439402	1236	0.000026	au.com.smh
335	19436904	1339	0.000024	com.vanityfair
336	19435504	261	0.000102	uk.org.ico
337	19434368	153	0.000176	com.addtoany
338	19427034	986	0.000032	com.uk
339	19426114	1097	0.000029	com.scmp
340	19424036	981	0.000033	org.pnas
341	19422474	433	0.000064	com.cnet
342	19422174	348	0.000080	com.statista
343	19421360	149	0.000180	org.nodejs
344	19418836	793	0.000037	us.icio
345	19416290	2916	0.000013	com.instapaper
346	19415752	489	0.000056	gov.epa
347	19405682	1003	0.000032	com.mixcloud
348	19404500	677	0.000042	org.d3js
349	19401872	851	0.000036	com.britannica
350	19400204	593	0.000047	uk.gov.service
351	19399466	177	0.000143	org.allaboutcookies
352	19396954	546	0.000050	edu.berkeley
353	19395162	296	0.000091	me.telegram
354	19393670	1436	0.000023	com.irishtimes
355	19392716	944	0.000034	int.coe
356	19388874	157	0.000170	com.zendesk
357	19388222	1507	0.000022	org.hrc
358	19387040	1251	0.000025	com.history
359	19386872	152	0.000178	io.shields
360	19386294	1686	0.000021	ms.1drv
361	19384766	548	0.000050	com.biomedcentral
362	19384760	473	0.000059	com.latimes
363	19381234	1062	0.000030	org.jstor
364	19381206	1124	0.000028	com.jetbrains
365	19376152	1084	0.000029	org.ilo
366	19376036	777	0.000038	edu.psu
367	19375334	137	0.000207	com.youronlinechoices
368	19374986	895	0.000035	com.ecwid
369	19371930	1067	0.000030	com.brightcove
370	19371460	748	0.000039	it.scoop
371	19369410	329	0.000084	ru.ok
372	19364910	1226	0.000026	com.digitaltrends
373	19362788	1004	0.000032	uk.co.thetimes
374	19359762	1404	0.000023	com.thedailybeast
375	19359122	1604	0.000022	edu.osu
376	19354290	513	0.000054	edu.yale
377	19351764	116	0.000262	com.jimdo
378	19349800	2354	0.000016	com.fastcodesign
379	19349466	988	0.000032	uk.parliament
380	19348128	456	0.000061	org.freecodecamp
381	19343488	2209	0.000018	com.us
382	19340214	573	0.000048	com.deloitte
383	19339976	1010	0.000031	uk.co.huffingtonpost
384	19339738	605	0.000046	com.zdnet
385	19339098	181	0.000139	ru.mail
386	19337572	415	0.000066	com.elsevier
387	19335810	1185	0.000027	org.nejm
388	19335572	2422	0.000016	com.instructure
389	19335456	427	0.000065	net.imgix
390	19334596	1714	0.000021	com.citrix
391	19330958	2590	0.000014	org.aclweb
392	19330886	1964	0.000018	org.haskell
393	19326754	649	0.000044	gov.state
394	19326634	1350	0.000024	app.netlify
395	19326068	801	0.000037	com.venturebeat
396	19323896	217	0.000118	com.eventbrite
397	19323876	622	0.000045	com.seattletimes
398	19323284	1868	0.000019	jp.ac.u-tokyo
399	19322408	3048	0.000012	org.uxplanet
400	19319794	1186	0.000027	com.dw
401	19318404	1248	0.000026	org.undp
402	19317950	211	0.000120	com.etsy
403	19317948	1763	0.000020	com.itv
404	19317548	233	0.000112	net.php
405	19314522	58	0.000638	com.googleadservices
406	19314004	271	0.000098	com.surveymonkey
407	19310004	245	0.000107	org.aboutcookies
408	19309574	1900	0.000019	edu.vt
409	19307434	2477	0.000015	org.wikibooks
410	19305780	687	0.000042	gov.nist
411	19297658	1423	0.000023	com.thehindu
412	19297142	1120	0.000028	org.hrw
413	19295118	600	0.000046	com.thinkwithgoogle
414	19294574	1682	0.000021	gov.usembassy
415	19294274	656	0.000044	com.intel
416	19292884	1725	0.000021	int.unfccc
417	19291806	394	0.000070	com.ebay
418	19289908	2419	0.000016	google.ai
419	19286562	2270	0.000017	com.netvibes
420	19285156	2562	0.000015	io.material
421	19283106	2235	0.000017	ly.rebrand
422	19280870	1954	0.000019	org.archlinux
423	19280114	878	0.000035	uk.co.pinterest
424	19279216	2833	0.000013	org.doctorswithoutborders
425	19277590	2069	0.000018	org.accessnow
426	19272506	1482	0.000023	com.findlaw
427	19271938	800	0.000037	net.clickbank
428	19270876	3536	0.000010	com.viki
429	19269508	980	0.000033	edu.brookings
430	19265638	2363	0.000016	co.carrd
431	19265518	2950	0.000012	org.neocities
432	19261700	1294	0.000025	com.wikia
433	19260608	657	0.000044	com.mashable
434	19260346	798	0.000037	com.thelancet
435	19259074	974	0.000033	uk.ac.cam
436	19258964	2416	0.000016	org.rsf
437	19258786	1841	0.000019	net.daringfireball
438	19255684	708	0.000041	com.canva
439	19255674	495	0.000056	gov.whitehouse
440	19253374	204	0.000122	com.salesforce
441	19252328	1191	0.000027	com.thenextweb
442	19251934	2076	0.000018	com.france24
443	19251914	534	0.000052	io.codepen
444	19250670	2896	0.000013	com.laweekly
445	19250196	1218	0.000026	com.licdn
446	19248894	2839	0.000013	cc.uxdesign
447	19245136	2368	0.000016	edu.kit
448	19240312	1071	0.000030	watch.fb
449	19239224	2583	0.000014	org.scala-lang
450	19238926	2513	0.000015	au.com.theage
451	19234622	2644	0.000014	com.hubpages
452	19232860	1533	0.000022	ch.ipcc
453	19232358	1092	0.000029	com.digitaloceanspaces
454	19231078	2435	0.000015	org.vim
455	19227408	1721	0.000021	com.refinery29
456	19227102	292	0.000093	net.secureservercdn
457	19225996	769	0.000038	com.marketwatch
458	19224690	2608	0.000014	app.web
459	19224666	1681	0.000022	org.unwomen
460	19223622	2447	0.000015	com.fineartamerica
461	19222710	2566	0.000015	nl.blogspot
462	19219678	537	0.000051	edu.cmu
463	19219662	505	0.000054	fr.free
464	19217122	831	0.000036	com.box
465	19216944	1159	0.000027	com.imageshack
466	19215358	2409	0.000016	edu.usf
467	19214138	2396	0.000016	nz.co.nzherald
468	19212598	2758	0.000013	com.smashwords
469	19208544	364	0.000074	net.datatables
470	19207930	419	0.000066	com.nationalgeographic
471	19207832	287	0.000095	com.iubenda
472	19206220	2470	0.000015	re.appsto
473	19205746	235	0.000112	com.adnxs
474	19203826	2509	0.000015	org.gentoo
475	19203684	1982	0.000018	com.voanews
476	19203064	2498	0.000015	com.superuser
477	19201090	379	0.000072	com.businesswire
478	19200162	643	0.000045	int.wipo
479	19199426	2201	0.000018	org.biorxiv
480	19198874	1451	0.000023	org.amnesty
481	19198454	2436	0.000015	com.oregonlive
482	19198448	2033	0.000018	org.nobelprize
483	19197824	81	0.000414	net.jsfiddle
484	19197384	1963	0.000019	com.ew
485	19195648	1008	0.000031	com.arstechnica
486	19194700	1826	0.000020	org.ocks
487	19194656	265	0.000101	com.aliyuncs
488	19192314	2649	0.000014	com.dezeen
489	19191524	2678	0.000014	org.transparency
490	19191474	925	0.000035	org.mediawiki
491	19191020	2853	0.000013	com.scribblelive
492	19190866	1357	0.000024	io.gitlab
493	19190776	1683	0.000021	org.aiga
494	19188994	1985	0.000018	uk.gov.tfl
495	19186250	428	0.000064	com.adweek
496	19185528	1941	0.000019	org.unep
497	19185444	430	0.000064	org.js
498	19184030	371	0.000073	com.atlassian
499	19183274	2282	0.000017	com.foreignpolicy
500	19181288	2696	0.000014	org.democracynow
501	19181160	985	0.000032	com.webs
502	19181124	1198	0.000026	com.wetransfer
503	19179532	1369	0.000024	org.altervista
504	19177814	2501	0.000015	google.research
505	19177102	2959	0.000012	za.co.iol
506	19175572	1028	0.000031	com.slate
507	19175528	2713	0.000014	org.cpj
508	19174456	2016	0.000018	org.example
509	19173782	2390	0.000016	com.googlegroups
510	19167902	243	0.000107	com.naver
511	19164304	2266	0.000017	net.openid
512	19164154	3058	0.000012	com.deepmind
513	19164022	269	0.000099	org.drupal
514	19163716	264	0.000101	gov.ca
515	19163714	414	0.000067	com.livechatinc
516	19155586	2328	0.000016	com.washingtontimes
517	19153082	637	0.000045	com.cbsnews
518	19151824	759	0.000038	com.oreilly
519	19150012	3034	0.000012	com.podomatic
520	19149538	664	0.000043	gov.loc
521	19147634	136	0.000208	org.networkadvertising
522	19146702	718	0.000041	com.buzzfeed
523	19144896	1333	0.000024	link.page
524	19143440	838	0.000036	com.pcmag
525	19142224	956	0.000034	com.verisign
526	19133476	2892	0.000013	com.thoughtworks
527	19132758	2669	0.000014	uk.co.timesonline
528	19131302	274	0.000097	com.getbootstrap
529	19130872	3082	0.000012	com.mariadb
530	19129650	1116	0.000028	com.jekyllrb
531	19128992	938	0.000034	com.vox
532	19127508	127	0.000234	info.aboutads
533	19122362	492	0.000056	com.patreon
534	19122034	2488	0.000015	com.curbed
535	19121736	514	0.000054	it.placehold
536	19121622	1807	0.000020	com.ascentlawfirm
537	19118802	227	0.000114	to.amzn
538	19117462	580	0.000047	com.visualstudio
539	19117426	1281	0.000025	com.smashingmagazine
540	19116926	499	0.000055	com.sxsw
541	19116922	978	0.000033	com.hootsuite
542	19116384	282	0.000095	gov.ftc
543	19113570	2336	0.000016	com.snopes
544	19110698	1392	0.000023	com.upwork
545	19109920	1388	0.000024	com.haaretz
546	19108142	1701	0.000021	com.firebaseapp
547	19104686	920	0.000035	com.zoho
548	19103768	3023	0.000012	org.peta
549	19100220	1174	0.000027	com.att
550	19097502	1687	0.000021	com.techrepublic
551	19097348	1776	0.000020	com.surveygizmo
552	19097050	2394	0.000016	com.treehugger
553	19095294	3359	0.000011	com.letterboxd
554	19094856	3025	0.000012	gov.anl
555	19094436	2568	0.000015	com.kaggle
556	19091430	2479	0.000015	fm.omny
557	19091422	2999	0.000012	com.bangkokpost
558	19089542	478	0.000058	gov.irs
559	19086962	1790	0.000020	ca.bc.gov
560	19086880	823	0.000036	com.emarketer
561	19086854	1173	0.000027	com.mediaplex
562	19086370	259	0.000103	uk.co.amazon
563	19084140	2737	0.000014	int.au
564	19083740	2514	0.000015	no.google
565	19083612	3618	0.000010	com.newgrounds
566	19083596	186	0.000134	jp.co.yahoo
567	19082852	3108	0.000012	org.hypotheses
568	19082332	337	0.000082	mp.mailchi
569	19082228	2808	0.000013	com.usmagazine
570	19080648	1674	0.000022	com.routledge
571	19078904	2881	0.000013	org.polymer-project
572	19077732	2446	0.000015	org.unctad
573	19077060	180	0.000140	com.caniuse
574	19076142	344	0.000080	com.onesignal
575	19073714	2714	0.000014	int.interpol
576	19072522	3248	0.000011	org.elasticsearch
577	19072218	620	0.000045	com.entrepreneur
578	19072108	2550	0.000015	uk.gov.metoffice
579	19071016	2921	0.000013	org.jenkins-ci
580	19070052	632	0.000045	com.samsung
581	19070008	1367	0.000024	org.unicode
582	19069412	3381	0.000011	uk.mod
583	19069358	2715	0.000014	org.mozillazine
584	19067032	3148	0.000011	edu.ucpress
585	19066706	1156	0.000027	com.gizmodo
586	19064356	1901	0.000019	org.americanbar
587	19063358	3593	0.000010	org.scala-sbt
588	19062892	330	0.000084	ai.shortpixel
589	19059914	2912	0.000013	in.indiatoday
590	19058702	301	0.000090	gg.discord
591	19058544	3687	0.000010	jp.riken
592	19056820	2440	0.000015	com.timesofisrael
593	19056518	3064	0.000012	com.manta
594	19056166	785	0.000037	com.fandom
595	19056052	1274	0.000025	com.sfgate
596	19054812	1994	0.000018	com.knightlab
597	19053684	2042	0.000018	org.donorbox
598	19053668	2251	0.000017	eu.politico
599	19051874	2494	0.000015	org.gnupg
600	19051168	84	0.000402	me.ogp
601	19050784	575	0.000048	com.cisco
602	19050146	2757	0.000013	uk.ac.york
603	19048588	1189	0.000027	com.buffer
604	19048534	3065	0.000012	uk.org.wwf
605	19047192	929	0.000035	com.variety
606	19045218	3517	0.000010	com.flightradar24
607	19044416	3387	0.000011	com.flock
608	19044368	579	0.000048	com.sedo
609	19044084	871	0.000035	com.libsyn
610	19043162	2561	0.000015	com.thenation
611	19042992	2569	0.000015	com.monday
612	19042226	482	0.000057	com.arcgis
613	19042098	3223	0.000011	net.inquirer
614	19038824	2699	0.000014	com.real
615	19036762	2195	0.000018	com.secondlife
616	19035414	732	0.000040	org.unesco
617	19035292	932	0.000035	com.wikihow
618	19034196	2609	0.000014	uk.ac.leeds
619	19033676	80	0.000415	com.livestream
620	19031566	3100	0.000012	org.cato
621	19029288	2716	0.000014	org.sonatype
622	19028098	3169	0.000011	com.intensedebate
623	19028006	1089	0.000029	com.symantec
624	19027462	4048	0.000009	org.jw
625	19027208	2793	0.000013	com.wayfair
626	19026864	1922	0.000019	com.scene7
627	19026030	76	0.000421	com.messenger
628	19025656	1689	0.000021	org.coursera
629	19024592	1196	0.000026	edu.umn
630	19024030	3093	0.000012	org.rferl
631	19023590	2512	0.000015	org.wikidata
632	19022464	644	0.000045	com.psychologytoday
633	19021606	3091	0.000012	com.vancouversun
634	19021016	2496	0.000015	uk.org.nationaltrust
635	19020356	1553	0.000022	ly.ow
636	19020300	1264	0.000025	edu.ucsd
637	19020198	2905	0.000013	tr.com.aa
638	19018958	3767	0.000010	it.polito
639	19017674	3389	0.000011	org.sourcewatch
640	19017264	3207	0.000011	ch.qos
641	19017144	3785	0.000010	jp.ac.kobe-u
642	19016500	1659	0.000022	com.speakerdeck
643	19015604	2974	0.000012	com.sciencealert
644	19015034	636	0.000045	com.photobucket
645	19015026	3087	0.000012	com.hsbc
646	19014412	3852	0.000009	edu.uah
647	19013454	188	0.000133	com.jimcdn
648	19012700	1195	0.000027	com.rollingstone
649	19012558	2897	0.000013	org.osce
650	19012150	4205	0.000009	com.gust
651	19009874	1684	0.000021	org.webkit
652	19009396	945	0.000034	com.shutterstock
653	19008674	2945	0.000012	com.townnews
654	19008318	2650	0.000014	org.wri
655	19006196	520	0.000053	com.inc
656	19004672	648	0.000044	com.gartner
657	19004226	2502	0.000015	ru.rg
658	19003942	2648	0.000014	io.bower
659	19003780	3371	0.000011	net.thedailystar
660	19003722	2564	0.000015	net.dwcdn
661	19003574	2770	0.000013	com.articulate
662	19003510	221	0.000117	com.myshopify
663	19002868	172	0.000151	jp.co.google
664	19002548	1354	0.000024	gov.uspto
665	19000824	998	0.000032	edu.ucla
666	18998632	685	0.000042	com.investopedia
667	18998502	3161	0.000011	com.mongabay
668	18997700	532	0.000052	com.aol
669	18994210	2352	0.000016	ca.citizenlab
670	18993878	1284	0.000025	com.today
671	18992626	160	0.000167	org.whatwg
672	18992320	1048	0.000030	com.smartadserver
673	18992054	2995	0.000012	org.pewforum
674	18991266	2997	0.000012	org.sierraclub
675	18991062	2964	0.000012	net.vnexpress
676	18990848	1069	0.000030	com.about
677	18989274	3081	0.000012	uk.co.spectator
678	18988482	480	0.000058	com.dmca
679	18987042	1462	0.000023	ly.cutt
680	18986890	3224	0.000011	ru.interfax
681	18986546	3670	0.000010	uk.co.zoopla
682	18985004	2843	0.000013	org.iucnredlist
683	18984682	194	0.000130	com.tripadvisor
684	18984256	3473	0.000010	fm.audioboo
685	18983654	2762	0.000013	uk.co.bbci
686	18983098	3416	0.000011	edu.sjsu
687	18982290	1444	0.000023	edu.northwestern
688	18982040	543	0.000051	com.googleoptimize
689	18980160	2817	0.000013	int.iom
690	18979598	1386	0.000024	edu.umd
691	18978832	629	0.000045	org.eff
692	18978224	2297	0.000017	uk.org.ofcom
693	18978038	2450	0.000015	int.reliefweb
694	18977708	3502	0.000010	com.torontosun
695	18975348	493	0.000056	com.indeed
696	18973238	1657	0.000022	com.nngroup
697	18972898	351	0.000078	com.constantcontact
698	18972774	1780	0.000020	co.lpages
699	18972434	1329	0.000024	edu.utexas
700	18971302	3386	0.000011	com.iconarchive
701	18971240	312	0.000087	com.pubmatic
702	18971078	1043	0.000030	org.reactjs
703	18969428	975	0.000033	edu.umich
704	18968428	1219	0.000026	com.tableau
705	18968208	1914	0.000019	com.hatenablog
706	18967574	1138	0.000028	com.chicagotribune
707	18967196	3999	0.000009	info.spain
708	18965740	547	0.000050	gov.copyright
709	18965442	4171	0.000009	org.gwtproject
710	18964656	650	0.000044	com.netflix
711	18963714	753	0.000039	net.adform
712	18961462	2847	0.000013	uk.ac.jisc
713	18961172	2865	0.000013	com.ringcentral
714	18960292	976	0.000033	com.redhat
715	18960280	3237	0.000011	com.city-data
716	18959938	2982	0.000012	uk.org.stonewall
717	18958764	3646	0.000010	za.co.timeslive
718	18957560	3989	0.000009	com.programmableweb
719	18957204	403	0.000068	com.bigcommerce
720	18957166	3053	0.000012	com.flippa
721	18955710	3293	0.000011	com.multiscreensite
722	18955294	2719	0.000014	com.bloglines
723	18954560	2629	0.000014	mp.j
724	18952520	2807	0.000013	uk.org.rspb
725	18952358	2949	0.000012	com.foreignaffairs
726	18952314	2206	0.000018	co.pcdn
727	18951462	3549	0.000010	in.theprint
728	18951006	4227	0.000009	com.symbaloo
729	18950992	4478	0.000008	com.algorithmia
730	18950356	1733	0.000021	com.billboard
731	18949998	742	0.000039	com.splashthat
732	18948332	3255	0.000011	com.cleantechnica
733	18947926	3604	0.000010	com.businessdailyafrica
734	18947204	1108	0.000028	com.dell
735	18947190	2825	0.000013	com.yell
736	18947008	443	0.000062	net.hubspot
737	18946944	3838	0.000010	org.rfa
738	18946618	3495	0.000010	za.co.mg
739	18945038	4346	0.000008	com.apsense
740	18944960	1772	0.000020	com.alibabagroup
741	18944660	2267	0.000017	to.dev
742	18944558	3471	0.000010	ru.mid
743	18944274	3498	0.000010	com.itsnicethat
744	18942634	527	0.000052	org.unicef
745	18942358	2364	0.000016	net.noscript
746	18940606	1390	0.000024	com.techradar
747	18938348	1857	0.000019	edu.uci
748	18937064	1118	0.000028	com.windowsphone
749	18936626	2734	0.000014	com.doubleclickbygoogle
750	18936326	3524	0.000010	org.350
751	18935080	3076	0.000012	org.aei
752	18934558	3075	0.000012	gov.arts
753	18934248	671	0.000043	gov.sec
754	18933682	2298	0.000017	com.urbandictionary
755	18933596	3925	0.000009	com.forbesimg
756	18933310	487	0.000056	com.fc2
757	18931436	3352	0.000011	com.brill
758	18931406	2491	0.000015	com.infoworld
759	18930782	1352	0.000024	com.bazaarvoice
760	18930350	3600	0.000010	de.uni-konstanz
761	18930232	1187	0.000027	com.alexa
762	18929982	2240	0.000017	org.linuxfoundation
763	18929738	3580	0.000010	edu.dukeupress
764	18929218	4038	0.000009	com.hotfrog
765	18928884	512	0.000054	com.mckinsey
766	18928710	2539	0.000015	org.crossref
767	18928328	3893	0.000009	com.environmentalleader
768	18927900	2236	0.000017	tv.ustream
769	18927290	1101	0.000029	fm.last
770	18926906	1951	0.000019	com.businessweek
771	18926816	413	0.000067	org.opensource
772	18925308	750	0.000039	org.whatbrowser
773	18925012	1296	0.000025	com.merriam-webster
774	18924400	425	0.000065	com.proofpoint
775	18923302	3196	0.000011	com.alchemer
776	18922680	3684	0.000010	com.arfadia
777	18922120	1692	0.000021	com.kinstacdn
778	18921334	3619	0.000010	com.ecowatch
779	18921302	2215	0.000018	net.leadpages
780	18920188	3448	0.000010	com.total
781	18920112	3878	0.000009	uk.org.npg
782	18919836	3054	0.000012	io.crates
783	18919320	2517	0.000015	com.lego
784	18919150	503	0.000055	com.wufoo
785	18915478	2742	0.000014	io.redis
786	18914956	2289	0.000017	uk.co.metro
787	18914016	4161	0.000009	uk.co.theweek
788	18913930	2204	0.000018	gd.is
789	18913640	4196	0.000009	io.coda
790	18913482	196	0.000128	com.hackerone
791	18912094	1483	0.000023	com.msdn
792	18911500	156	0.000170	org.nginx
793	18911390	3197	0.000011	com.klokantech
794	18911286	1668	0.000022	com.sky
795	18910142	4272	0.000009	de.fernuni-hagen
796	18909064	1832	0.000020	de.hessen
797	18908834	953	0.000034	com.adroll
798	18907960	1986	0.000018	com.windows
799	18906254	4316	0.000008	com.tupalo
800	18904242	218	0.000118	org.icann
801	18904004	1148	0.000028	net.atlassian
802	18903442	4234	0.000009	net.ccm
803	18903372	3808	0.000010	com.oilprice
804	18903128	1956	0.000019	org.khanacademy
805	18902896	4072	0.000009	net.iwpr
806	18902418	324	0.000084	eu.youronlinechoices
807	18902318	3890	0.000009	uk.ac.mmu
808	18901968	1700	0.000021	edu.usc
809	18901674	1146	0.000028	com.playstation
810	18900854	4268	0.000009	uk.ac.ceh
811	18900236	1060	0.000030	com.akamai
812	18897754	2851	0.000013	com.hindustantimes
813	18896694	979	0.000033	gov.fcc
814	18896368	990	0.000032	com.gumroad
815	18896128	4383	0.000008	et.com.google
816	18894652	3952	0.000009	com.theoutline
817	18894586	4348	0.000008	org.cgsociety
818	18892502	4093	0.000009	edu.mtsu
819	18892446	2636	0.000014	com.html5rocks
820	18892222	4578	0.000008	com.blockchair
821	18891572	3603	0.000010	org.spie
822	18891278	1208	0.000026	at.gv.bka
823	18890814	3768	0.000010	uk.co.lrb
824	18888196	410	0.000067	com.heroku
825	18888062	815	0.000036	edu.wisc
826	18887938	1009	0.000031	com.yoast
827	18887842	3807	0.000010	za.co.dailymaverick
828	18885614	1864	0.000019	org.json
829	18885194	2969	0.000012	org.thinkprogress
830	18884928	700	0.000041	com.feedly
831	18883562	4278	0.000009	com.ingress
832	18883012	3706	0.000010	google.design
833	18882952	4666	0.000008	com.bmwblog
834	18881526	3862	0.000009	com.thepetitionsite
835	18881312	3993	0.000009	in.bbc
836	18880860	2237	0.000017	com.w3techs
837	18880532	3257	0.000011	org.carbonbrief
838	18880428	272	0.000098	jp.ne.hatena
839	18880124	2614	0.000014	ru.mk
840	18880064	1924	0.000019	edu.hbs
841	18879486	935	0.000034	com.pingdom
842	18878780	1199	0.000026	com.ycombinator
843	18876422	4065	0.000009	com.gifer
844	18876050	3709	0.000010	uk.org.amnesty
845	18875700	3961	0.000009	com.africanews
846	18875478	4535	0.000008	com.the-dots
847	18875350	992	0.000032	so.notion
848	18869944	3375	0.000011	org.commondreams
849	18869250	4236	0.000009	com.flutterwave
850	18868302	3731	0.000010	org.refworld
851	18866446	3285	0.000011	uk.gov.charitycommission
852	18866244	4396	0.000008	com.newsru
853	18865810	3507	0.000010	uk.org.oxfam
854	18865786	4206	0.000009	uk.org.somersethouse
855	18864590	2484	0.000015	in.scroll
856	18864514	1410	0.000023	com.intuit
857	18864428	4257	0.000009	uk.co.harpercollins
858	18863942	331	0.000084	jp.ameblo
859	18863622	3370	0.000011	ke.co.nation
860	18863580	1091	0.000029	com.insurancejournal
861	18863360	3394	0.000011	com.cbsistatic
862	18863302	2697	0.000014	com.spreaker
863	18862358	2720	0.000014	com.springernature
864	18861824	2228	0.000017	com.firefox
865	18861222	4688	0.000008	co.iglobal
866	18860960	4714	0.000008	io.devdocs
867	18860246	2705	0.000014	com.verywellhealth
868	18860020	538	0.000051	com.booking
869	18859838	535	0.000051	com.gofundme
870	18859798	1234	0.000026	com.indiegogo
871	18859328	4781	0.000008	com.kdpcommunity
872	18858404	2344	0.000016	build.bazel
873	18858016	1119	0.000028	com.foursquare
874	18857452	545	0.000051	com.snapchat
875	18856948	93	0.000390	com.trustpilot
876	18856546	2586	0.000014	com.avast
877	18856160	1813	0.000020	com.pcworld
878	18855278	4149	0.000009	com.hybris
879	18854982	4937	0.000008	com.jetphotos
880	18854014	711	0.000041	com.yandex
881	18853962	1023	0.000031	com.css-tricks
882	18853902	1360	0.000024	org.golang
883	18853900	4168	0.000009	uk.ac.mdx
884	18853596	2673	0.000014	com.flipboard
885	18852988	2848	0.000013	com.discovery
886	18851822	3904	0.000009	at.kleinezeitung
887	18851422	302	0.000090	de.amazon
888	18850596	111	0.000276	me.wa
889	18850558	429	0.000064	com.skype
890	18850294	1134	0.000028	com.scientificamerican
891	18848646	2516	0.000015	org.raspberrypi
892	18847770	4426	0.000008	com.armorgames
893	18847596	1758	0.000020	com.fiverr
894	18847110	721	0.000040	org.iso
895	18846464	2707	0.000014	com.codecademy
896	18843948	3727	0.000010	net.middleeasteye
897	18842876	3057	0.000012	org.man7
898	18841074	4531	0.000008	com.e-estonia
899	18840474	1827	0.000020	fr.blogspot
900	18840282	1047	0.000030	com.huffpost
901	18839944	4725	0.000008	net.gebco
902	18839822	4778	0.000008	com.slite
903	18839776	1799	0.000020	com.visa
904	18839258	722	0.000040	com.newrelic
905	18837214	4163	0.000009	com.cnsnews
906	18836004	807	0.000037	br.com.uol
907	18834494	3388	0.000011	com.lithub
908	18834036	3942	0.000009	net.bostonreview
909	18832052	1022	0.000031	au.com.google
910	18831616	2958	0.000012	com.hackernoon
911	18831490	2802	0.000013	com.unity
912	18831230	249	0.000106	net.2mdn
913	18831090	1224	0.000026	gov.usgs
914	18830714	400	0.000068	com.semrush
915	18829732	3855	0.000009	com.indexmundi
916	18829642	516	0.000053	com.dailymotion
917	18828872	686	0.000042	com.accenture
918	18827314	665	0.000043	org.poynter
919	18826574	2086	0.000018	org.aclu
920	18826336	3974	0.000009	org.jython
921	18825792	1152	0.000027	com.searchengineland
922	18825212	4003	0.000009	com.inthesetimes
923	18824910	1753	0.000020	com.over-blog
924	18824406	518	0.000053	nl.google
925	18824128	3324	0.000011	de.bfarm
926	18823544	1126	0.000028	com.techtarget
927	18823106	4417	0.000008	za.co.ewn
928	18822694	4251	0.000009	uk.co.bristolpost
929	18822100	4799	0.000008	community.studiopress
930	18821800	846	0.000036	gov.justice
931	18820682	1707	0.000021	com.technologyreview
932	18820672	3923	0.000009	com.recyclenow
933	18819750	4293	0.000009	lb.com.dailystar
934	18819328	384	0.000071	com.bitly
935	18819088	4101	0.000009	org.occrp
936	18819058	3786	0.000010	com.theyworkforyou
937	18818798	4022	0.000009	org.ifaw
938	18818648	168	0.000161	com.jimstatic
939	18818242	966	0.000033	sh.brew
940	18818066	4343	0.000008	com.yahoosites
941	18817604	2321	0.000016	com.fool
942	18817360	2281	0.000017	com.pastebin
943	18816928	3607	0.000010	com.gr-assets
944	18815242	3872	0.000009	com.climatechangenews
945	18813886	4771	0.000008	in.ac.iith
946	18813586	702	0.000041	org.plos
947	18812812	4068	0.000009	com.chamberofcommerce
948	18812606	4438	0.000008	us.tuugo
949	18812344	1142	0.000028	com.buzzsprout
950	18812110	1122	0.000028	com.timeanddate
951	18811418	760	0.000038	com.discordapp
952	18811082	1153	0.000027	com.sitepoint
953	18809798	4579	0.000008	com.desmogblog
954	18809638	596	0.000047	com.aliexpress
955	18808906	2522	0.000015	com.sendgrid
956	18807972	4128	0.000009	uk.ac.rcplondon
957	18807934	2004	0.000018	com.ssllabs
958	18807738	3955	0.000009	org.soilassociation
959	18806592	1359	0.000024	com.xkcd
960	18806192	536	0.000051	gov.hhs
961	18805588	3584	0.000010	com.hearstapps
962	18805150	1289	0.000025	com.searchenginejournal
963	18804458	442	0.000062	me.fb
964	18804132	4711	0.000008	net.sott
965	18803628	4594	0.000008	com.gpsvisualizer
966	18803548	213	0.000120	com.discord
967	18802758	1820	0.000020	org.mitre
968	18801916	4744	0.000008	com.natureindex
969	18801374	3499	0.000010	uk.org.rspca
970	18801322	4087	0.000009	org.c2es
971	18801296	3824	0.000010	com.qgiv
972	18801288	3799	0.000010	ug.co.monitor
973	18800492	4596	0.000008	com.lacartes
974	18800384	178	0.000142	com.xing
975	18799966	3973	0.000009	com.svbtle
976	18798944	3875	0.000009	uk.org.savethechildren
977	18798932	4616	0.000008	com.slurl
978	18798332	2466	0.000015	com.sophos
979	18797956	2198	0.000018	com.twilio
980	18797008	4662	0.000008	za.co.moneyweb
981	18796288	4265	0.000009	com.menafn
982	18796048	3821	0.000010	org.usip
983	18795572	4584	0.000008	com.power-technology
984	18795368	4601	0.000008	org.heartland
985	18794834	675	0.000042	com.usnews
986	18794200	2557	0.000015	org.usenix
987	18793962	2899	0.000013	net.privacypolicytemplate
988	18793900	4081	0.000009	org.theecologist
989	18792970	4132	0.000009	org.neweconomics
990	18792680	1290	0.000025	com.netlify
991	18791954	4189	0.000009	com.businessgreen
992	18790752	4611	0.000008	org.monthlyreview
993	18790450	2369	0.000016	uk.ac.ed
994	18790194	2245	0.000017	ch.ethz
995	18789974	267	0.000099	com.nielsen
996	18789302	2674	0.000014	ca.uwaterloo
997	18788810	4284	0.000009	org.unep-wcmc
998	18787702	4042	0.000009	org.ramsar
999	18786952	4384	0.000008	com.googlelabs
1000	18786808	4634	0.000008	org.berkeleyearth

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

September 2021 crawl archive now available

October 4, 2021Sebastian Nagel

The crawl archive for September 2021 is now available! The data was crawled Sept 16 – 29 and contains 2.95 billion web pages or 310 TiB of uncompressed content. It includes page captures of 1.3 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The September crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-39/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-39/segment.paths.gz	100
WARC files	CC-MAIN-2021-39/warc.paths.gz	72000	72.16
WAT files	CC-MAIN-2021-39/wat.paths.gz	72000	19.81
WET files	CC-MAIN-2021-39/wet.paths.gz	72000	8.7
Robots.txt files	CC-MAIN-2021-39/robotstxt.paths.gz	72000	0.15
Non-200 responses files	CC-MAIN-2021-39/non200responses.paths.gz	72000	2.19
URL index files	CC-MAIN-2021-39/cc-index.paths.gz	302	0.22

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-39/. Also the columnar index has been updated to contain this crawl.

July/August 2021 crawl archive now available

August 9, 2021Sebastian Nagel

The crawl archive for July/August 2021 is now available! The data was crawled July 23 – August 6 and contains 3.15 billion web pages or 360 TiB of uncompressed content. It includes page captures of 1 billion new URLs, not visited in any of our prior crawls.

Improvements

Archiving of robots.txt files was improved. Robots.txt files are not archived if

the robots.txt of the target host does not allow it (in case of a HTTP redirect) or
URL filters exclude the entire site, eg. if it’s known ahead that a site does not allow crawling or
the MIME type is not applicable for robots.txt files (eg. HTML, PDF)

More details are found in the corresponding issue report. The change reduces the size of the robots.txt subset (since August 2016) by removing content which should not contained in this dataset.

Archive Location and Download

The July/August crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-31/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-31/segment.paths.gz	100
WARC files	CC-MAIN-2021-31/warc.paths.gz	72000	75.34
WAT files	CC-MAIN-2021-31/wat.paths.gz	72000	21.67
WET files	CC-MAIN-2021-31/wet.paths.gz	72000	9.43
Robots.txt files	CC-MAIN-2021-31/robotstxt.paths.gz	72000	0.14
Non-200 responses files	CC-MAIN-2021-31/non200responses.paths.gz	72000	1.98
URL index files	CC-MAIN-2021-31/cc-index.paths.gz	302	0.23

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-31/. Also the columnar index has been updated to contain this crawl.