Blog – Common Crawl

July/August 2021 crawl archive now available

August 9, 2021Sebastian Nagel

The crawl archive for July/August 2021 is now available! The data was crawled July 23 – August 6 and contains 3.15 billion web pages or 360 TiB of uncompressed content. It includes page captures of 1 billion new URLs, not visited in any of our prior crawls.

Improvements

Archiving of robots.txt files was improved. Robots.txt files are not archived if

the robots.txt of the target host does not allow it (in case of a HTTP redirect) or
URL filters exclude the entire site, eg. if it’s known ahead that a site does not allow crawling or
the MIME type is not applicable for robots.txt files (eg. HTML, PDF)

More details are found in the corresponding issue report. The change reduces the size of the robots.txt subset (since August 2016) by removing content which should not contained in this dataset.

Archive Location and Download

The July/August crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-31/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://commoncrawl.s3.amazonaws.com/ to each line, you end up with the S3 and HTTP paths respectively.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-31/segment.paths.gz	100
WARC files	CC-MAIN-2021-31/warc.paths.gz	72000	75.34
WAT files	CC-MAIN-2021-31/wat.paths.gz	72000	21.67
WET files	CC-MAIN-2021-31/wet.paths.gz	72000	9.43
Robots.txt files	CC-MAIN-2021-31/robotstxt.paths.gz	72000	0.14
Non-200 responses files	CC-MAIN-2021-31/non200responses.paths.gz	72000	1.98
URL index files	CC-MAIN-2021-31/cc-index.paths.gz	302	0.23

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-31/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

June 2021 crawl archive now available

June 28, 2021Sebastian Nagel

The crawl archive for June 2021 is now available! The data was crawled June 12 – 25 and contains 2.45 billion web pages or 260 TiB of uncompressed content. It includes page captures of 1.3 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The June crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-25/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-25/segment.paths.gz	100
WARC files	CC-MAIN-2021-25/warc.paths.gz	64000	57.9
WAT files	CC-MAIN-2021-25/wat.paths.gz	64000	16.54
WET files	CC-MAIN-2021-25/wet.paths.gz	64000	7.24
Robots.txt files	CC-MAIN-2021-25/robotstxt.paths.gz	64000	0.16
Non-200 responses files	CC-MAIN-2021-25/non200responses.paths.gz	64000	1.76
URL index files	CC-MAIN-2021-25/cc-index.paths.gz	302	0.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-25/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs February/March, April and May 2021

May 31, 2021Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of February/March, April and May 2021. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases. You may also visit the projects cc-webgraph and cc-pyspark which include all scripts and tools required to construct the graphs. Instructions to explore the graphs in the webgraph format are given in our collection of webgraph notebooks.

What’s new?

The host-level graph now includes all hosts visited by the crawler even if there is no link pointing to the host and all visited URLs of a host failed (HTTP 404 and other error codes) or the host’s robots.txt does not allow crawling. Note that the links leading to these hosts may have been found in a prior crawl, not in one of the 3 crawls used to build this web graph.

Host-level graph

The graph consists of 515 million nodes and 2.82 billion edges. Both hyperlinks and HTTP redirects and link headers are used as edges to span up the graph. All types of links are included, including pure “technical” ones pointing to images, JavaScript libraries, web fonts, etc. However, only host names with a valid IANA TLD are used. Consequently, URLs with an IP address as host component are not taken into account for building the host-level graph.

There are 452 million dangling nodes (87.9%) and the largest strongly connected component contains 45.2 million (8.8%) nodes. Dangling nodes stem from

hosts that have not been crawled, yet are pointed to from a link on a crawled page
hosts without any links pointing to a different host name
or hosts which did only return an error page (eg. HTTP 404)

Host names in the graph are in reverse domain name notation and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 515 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2021-feb-apr-may/host/. Alternatively, you can use https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2021-feb-apr-may/host/ as prefix to access the files from everywhere.

Please note that the text representation of the host-level graph is shipped in 72 gzip-compressed files listed in two path listings – one for the nodes (vertices), one for the edges (arcs). First, download the paths listing and decompress it using “gzip”. By adding the prefix s3://commoncrawl/ or https://commoncrawl.s3.amazonaws.com/ to each line in the path listing you get the list of URLs to download the entire graph.

Download files of the Common Crawl Feb/Apr/May 2021 host-level webgraph

Size	File	Description
3.31 GB	cc-main-2021-feb-apr-may-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 24 vertices files
12.94 GB	cc-main-2021-feb-apr-may-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 48 edges files

5.57 GB	cc-main-2021-feb-apr-may-host.graph	graph in BVGraph format
2 kB	cc-main-2021-feb-apr-may-host.properties

6.22 GB	cc-main-2021-feb-apr-may-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2021-feb-apr-may-host-t.properties

1 kB	cc-main-2021-feb-apr-may-host.stats	WebGraph statistics

7.69 GB	cc-main-2021-feb-apr-may-host-ranks.txt.gz	harmonic centrality and pagerank

Domain-level graph

The domain graph is built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 88 million nodes and 1.58 billion edges. 50% or 44 million nodes are dangling nodes, the largest strongly connected component covers 34 million or 39% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2021-feb-apr-may/domain/ resp. https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2021-feb-apr-may/domain/.

Download files of the Common Crawl Feb/Apr/May 2021 domain-level webgraph

Size	File	Description
0.61 GB	cc-main-2021-feb-apr-may-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
6.37 GB	cc-main-2021-feb-apr-may-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

3.58 GB	cc-main-2021-feb-apr-may-domain.graph	graph in BVGraph format
2 kB	cc-main-2021-feb-apr-may-domain.properties

3.42 GB	cc-main-2021-feb-apr-may-domain-t.graph	transpose of the graph
2 kB	cc-main-2021-feb-apr-may-domain-t.properties

1 kB	cc-main-2021-feb-apr-may-domain.stats	WebGraph statistics

1.89 GB	cc-main-2021-feb-apr-may-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 88 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Feb/Apr/May 2021)

harmonic centrality rank	hc value	page rank	page rank value	reversed domain name
1	31920934	1	0.017627	com.googleapis
2	31032784	3	0.013762	com.facebook
3	29681304	2	0.013832	com.google
4	27101692	4	0.007844	com.twitter
5	26954660	5	0.007519	org.w
6	26886624	7	0.006967	com.youtube
7	25515850	8	0.005718	com.instagram
8	25031490	6	0.007143	com.googletagmanager
9	24396116	9	0.005506	org.gmpg
10	23807122	12	0.003347	com.linkedin
11	22970992	13	0.003048	com.gstatic
12	22854052	10	0.003951	com.cloudflare
13	22698594	19	0.001914	com.gravatar
14	22504168	14	0.002908	org.wordpress
15	22434542	22	0.001564	com.pinterest
16	22100870	25	0.001270	org.wikipedia
17	21950578	17	0.002031	com.wordpress
18	21940826	18	0.001958	com.apple
19	21766696	15	0.002258	com.bootstrapcdn
20	21762964	30	0.001174	com.vimeo
21	21722198	38	0.000914	be.youtu
22	21556142	21	0.001842	com.jquery
23	21478118	29	0.001182	com.microsoft
24	21432212	53	0.000703	com.blogspot
25	21354260	35	0.001025	com.amazonaws
26	21337432	44	0.000765	com.amazon
27	21320702	43	0.000789	gl.goo
28	21170722	62	0.000600	ly.bit
29	21149628	99	0.000409	com.tumblr
30	21148242	50	0.000739	com.wp
31	21136818	45	0.000758	org.mozilla
32	21110018	57	0.000689	eu.europa
33	21104262	20	0.001894	com.adobe
34	21048760	16	0.002200	com.github
35	21040284	34	0.001026	com.google-analytics
36	21027350	36	0.001015	net.jsdelivr
37	20998320	27	0.001218	com.wixstatic
38	20995232	31	0.001119	net.cloudfront
39	20946148	47	0.000744	com.flickr
40	20913104	107	0.000338	com.yahoo
41	20851316	83	0.000436	com.googleusercontent
42	20843068	37	0.000929	io.github
43	20840670	111	0.000317	com.reddit
44	20834398	58	0.000677	com.paypal
45	20816886	23	0.001554	com.fontawesome
46	20773582	103	0.000368	com.weebly
47	20764576	79	0.000455	com.medium
48	20764512	33	0.001035	com.googlesyndication
49	20757582	32	0.001118	ru.yandex
50	20741944	48	0.000743	com.whatsapp
51	20708152	68	0.000520	org.w3
52	20705826	132	0.000240	com.nytimes
53	20696906	59	0.000673	co.t
54	20678088	102	0.000375	org.creativecommons
55	20675822	115	0.000290	com.soundcloud
56	20644978	60	0.000624	org.schema
57	20627114	74	0.000479	com.shopify
58	20621162	66	0.000543	com.vk
59	20604726	181	0.000149	org.wikimedia
60	20604724	147	0.000204	com.dropbox
61	20579720	55	0.000702	com.addthis
62	20572950	138	0.000211	org.archive
63	20570610	198	0.000133	com.cnn
64	20558114	152	0.000187	gov.cdc
65	20550306	80	0.000446	me.wp
66	20538816	193	0.000136	com.imgur
67	20530078	49	0.000740	net.doubleclick
68	20512294	199	0.000133	uk.co.bbc
69	20505964	200	0.000133	net.slideshare
70	20499864	171	0.000155	com.theguardian
71	20489756	158	0.000175	int.who
72	20482256	120	0.000263	com.spotify
73	20481118	175	0.000151	com.bing
74	20478320	213	0.000124	com.businessinsider
75	20477478	253	0.000104	com.bloomberg
76	20477300	144	0.000206	gov.nih
77	20473648	46	0.000748	com.macromedia
78	20440520	254	0.000103	com.wsj
79	20434320	224	0.000118	edu.stanford
80	20419762	41	0.000847	net.fbcdn
81	20417930	39	0.000885	org.apache
82	20409636	157	0.000175	org.ietf
83	20397792	90	0.000420	com.list-manage
84	20395594	368	0.000071	com.googleblog
85	20395350	217	0.000123	com.stackoverflow
86	20393172	170	0.000155	com.giphy
87	20391226	314	0.000085	edu.mit
88	20381948	223	0.000118	com.washingtonpost
89	20372602	134	0.000232	com.ytimg
90	20363592	362	0.000073	com.appspot
91	20360236	351	0.000076	com.theverge
92	20359610	286	0.000093	com.bbc
93	20358870	396	0.000067	uk.co.telegraph
94	20356036	499	0.000056	edu.berkeley
95	20348048	266	0.000101	edu.harvard
96	20346012	330	0.000080	com.go
97	20341676	237	0.000112	com.office
98	20338710	145	0.000206	us.zoom
99	20335782	247	0.000109	com.android
100	20335366	327	0.000082	com.wired
101	20334160	288	0.000092	com.techcrunch
102	20331782	238	0.000111	com.oracle
103	20323638	547	0.000051	com.livejournal
104	20296670	164	0.000170	com.issuu
105	20295840	296	0.000090	com.cnbc
106	20292146	211	0.000124	gov.ca
107	20291754	402	0.000066	com.ted
108	20288380	379	0.000069	gov.nasa
109	20283426	149	0.000195	com.forbes
110	20283050	148	0.000199	com.wixsite
111	20282972	151	0.000192	com.npmjs
112	20282524	518	0.000054	com.zdnet
113	20279656	447	0.000062	com.msn
114	20277752	292	0.000091	com.reuters
115	20275540	350	0.000076	com.nature
116	20273474	78	0.000459	com.godaddy
117	20271718	371	0.000070	com.myspace
118	20270494	222	0.000119	com.etsy
119	20268832	321	0.000084	com.prnewswire
120	20255726	209	0.000125	org.ampproject
121	20252386	407	0.000065	org.arxiv
122	20252292	312	0.000085	org.npr
123	20252218	263	0.000101	com.sciencedirect
124	20248804	98	0.000410	com.unpkg
125	20246402	265	0.000101	com.example
126	20245616	67	0.000524	net.akamaihd
127	20237056	215	0.000123	com.eventbrite
128	20234532	367	0.000072	org.hbr
129	20232338	176	0.000151	com.blogger
130	20231658	127	0.000247	org.networkadvertising
131	20231552	399	0.000066	com.latimes
132	20228690	268	0.000101	org.acm
133	20223242	338	0.000079	com.statista
134	20209434	389	0.000068	com.fastcompany
135	20205848	660	0.000043	com.economist
136	20202482	343	0.000078	com.time
137	20202452	226	0.000117	com.twimg
138	20201902	679	0.000042	edu.upenn
139	20201530	550	0.000050	edu.yale
140	20200842	258	0.000102	com.githubusercontent
141	20191272	474	0.000060	com.steampowered
142	20189824	143	0.000206	com.opera
143	20188620	444	0.000062	uk.co.dailymail
144	20188486	353	0.000076	com.springer
145	20186806	576	0.000047	com.scribd
146	20184784	780	0.000041	edu.columbia
147	20180100	535	0.000052	org.chromium
148	20175876	591	0.000046	me.about
149	20175732	604	0.000046	google.blog
150	20175284	285	0.000094	com.squarespace
151	20174050	335	0.000079	com.huffingtonpost
152	20171356	431	0.000063	com.nationalgeographic
153	20168788	221	0.000119	uk.co.google
154	20165372	208	0.000125	com.unsplash
155	20163580	388	0.000068	com.w3schools
156	20158956	339	0.000079	com.dribbble
157	20154786	340	0.000079	com.tiktok
158	20153356	293	0.000091	org.un
159	20137924	794	0.000040	com.qz
160	20133814	248	0.000108	com.bandcamp
161	20129598	485	0.000058	edu.cornell
162	20125954	821	0.000039	edu.umich
163	20121120	119	0.000267	com.ft
164	20115342	435	0.000063	com.theatlantic
165	20111028	966	0.000033	edu.princeton
166	20110808	341	0.000078	com.usatoday
167	20105556	786	0.000040	com.evernote
168	20105482	133	0.000235	info.aboutads
169	20104810	408	0.000065	com.meetup
170	20102638	438	0.000062	com.goodreads
171	20100894	625	0.000045	org.ieee
172	20098972	878	0.000036	com.slate
173	20097870	677	0.000042	com.mysql
174	20097656	453	0.000061	com.patreon
175	20097530	137	0.000216	me.t
176	20095600	515	0.000055	com.cbsnews
177	20084204	656	0.000043	com.docker
178	20083336	291	0.000092	com.wiley
179	20082520	480	0.000059	gov.usda
180	20080664	454	0.000061	com.dailymotion
181	20078818	817	0.000039	edu.washington
182	20077160	493	0.000057	com.withgoogle
183	20075064	523	0.000054	io.readthedocs
184	20071014	644	0.000044	com.marketwatch
185	20065010	650	0.000043	uk.co.blogspot
186	20062734	868	0.000037	com.shutterstock
187	20062652	54	0.000703	com.fb
188	20059664	497	0.000056	uk.co.independent
189	20056344	76	0.000467	com.wix
190	20055932	811	0.000039	org.cambridge
191	20051844	559	0.000049	com.pexels
192	20048576	779	0.000041	org.sciencemag
193	20048004	592	0.000046	com.buzzfeed
194	20044248	819	0.000039	com.stackexchange
195	20043466	179	0.000149	ru.mail
196	20043446	844	0.000038	com.webs
197	20043074	573	0.000048	com.git-scm
198	20040208	464	0.000060	com.inc
199	20037354	272	0.000100	net.behance
200	20029744	425	0.000063	gov.whitehouse
201	20025342	832	0.000038	com.apnews
202	20023518	769	0.000041	com.vox
203	20022030	1365	0.000024	uk.co.thesun
204	20018548	274	0.000098	com.outlook
205	20018318	772	0.000041	org.bitbucket
206	20017276	40	0.000871	com.qq
207	20014872	244	0.000110	org.doi
208	20012082	812	0.000039	uk.ac.cam
209	20011998	255	0.000103	com.disqus
210	20007312	236	0.000112	com.feedburner
211	20005630	670	0.000043	org.worldbank
212	20001230	584	0.000047	org.unicef
213	20000932	419	0.000064	com.mozilla
214	19999740	593	0.000046	co.ibb
215	19999080	26	0.001261	io.polyfill
216	19997928	525	0.000054	com.booking
217	19993488	42	0.000808	com.baidu
218	19989784	260	0.000101	com.cloudinary
219	19985856	289	0.000092	com.tinyurl
220	19983980	345	0.000077	com.ibm
221	19983022	1163	0.000027	com.speakerdeck
222	19982506	597	0.000046	gov.noaa
223	19978206	612	0.000045	ee.linktr
224	19977310	569	0.000048	com.psychologytoday
225	19973710	531	0.000053	gov.loc
226	19972920	400	0.000066	com.getpocket
227	19972760	1041	0.000031	edu.utexas
228	19971794	320	0.000084	org.pewresearch
229	19971310	1366	0.000024	edu.rutgers
230	19970894	551	0.000050	com.sagepub
231	19970200	309	0.000087	com.nbcnews
232	19967962	1134	0.000028	org.eclipse
233	19965586	648	0.000043	com.trello
234	19964280	326	0.000082	net.windows
235	19964194	384	0.000068	com.quora
236	19961430	600	0.000046	net.azurewebsites
237	19959910	275	0.000098	gov.ftc
238	19955938	1057	0.000030	edu.uchicago
239	19953308	311	0.000086	com.netdna-ssl
240	19951960	782	0.000041	org.semver
241	19951286	124	0.000252	com.mailchimp
242	19950294	436	0.000063	com.nypost
243	19949296	1195	0.000027	com.hatenablog
244	19947142	652	0.000043	com.newyorker
245	19943938	985	0.000033	uk.co.guardian
246	19943564	590	0.000046	com.usnews
247	19940498	220	0.000119	tv.twitch
248	19939738	784	0.000041	au.net.abc
249	19938820	166	0.000167	com.amazon-adsystem
250	19936308	1278	0.000025	com.vogue
251	19935466	230	0.000113	com.wpengine
252	19934098	106	0.000338	com.stripe
253	19933266	1261	0.000025	org.kernel
254	19929738	941	0.000034	com.politico
255	19926416	1193	0.000027	org.unicode
256	19925602	580	0.000047	org.eff
257	19925174	541	0.000051	br.com.uol
258	19924806	852	0.000037	com.about
259	19923644	1358	0.000024	edu.hbs
260	19923600	954	0.000034	com.dropboxusercontent
261	19923464	911	0.000035	edu.jhu
262	19922062	993	0.000032	co.elastic
263	19921888	913	0.000035	com.steamcommunity
264	19920150	1971	0.000018	com.googlesource
265	19919760	522	0.000054	com.tandfonline
266	19918010	277	0.000097	com.criteo
267	19915708	552	0.000050	org.pbs
268	19912986	1106	0.000029	edu.umd
269	19912224	64	0.000549	co.g
270	19908340	865	0.000037	com.foxnews
271	19907456	123	0.000261	com.sharethis
272	19904178	1027	0.000031	com.rollingstone
273	19903082	228	0.000115	com.imdb
274	19902774	977	0.000033	com.scientificamerican
275	19901940	1392	0.000023	com.urbandictionary
276	19900876	775	0.000041	uk.ac.ox
277	19900406	391	0.000067	com.arcgis
278	19898520	2016	0.000018	com.lego
279	19898420	251	0.000107	page.g
280	19898318	631	0.000044	gov.census
281	19890056	530	0.000053	com.oup
282	19887968	346	0.000077	com.optimizely
283	19887424	582	0.000047	com.indiatimes
284	19887194	376	0.000069	com.cnet
285	19884024	422	0.000064	com.wufoo
286	19882930	704	0.000042	uk.co.eventbrite
287	19882806	421	0.000064	com.bigcommerce
288	19880306	1350	0.000024	ca.blogspot
289	19879016	833	0.000038	org.fao
290	19878732	908	0.000035	com.jetbrains
291	19871044	1467	0.000022	ca.ubc
292	19867650	1938	0.000018	com.warnerbros
293	19866012	446	0.000062	org.d3js
294	19865518	946	0.000034	org.greenpeace
295	19864632	206	0.000127	net.sourceforge
296	19863450	323	0.000083	fr.google
297	19862916	1279	0.000025	com.history
298	19861806	851	0.000038	com.gumroad
299	19861750	919	0.000035	com.chicagotribune
300	19859844	636	0.000044	gov.archives
301	19858902	284	0.000095	com.googlecode
302	19853502	342	0.000078	com.slack
303	19851932	229	0.000114	com.eepurl
304	19845626	114	0.000292	com.paypalobjects
305	19841702	927	0.000035	com.sap
306	19839830	153	0.000180	com.addtoany
307	19837466	290	0.000092	com.typepad
308	19834082	1562	0.000021	de.mpg
309	19830054	664	0.000043	com.pinimg
310	19828148	282	0.000095	com.calendly
311	19827530	491	0.000057	gov.epa
312	19825756	354	0.000076	com.proofpoint
313	19821128	1430	0.000023	ch.ethz
314	19821094	1028	0.000031	com.500px
315	19820554	1732	0.000019	com.diigo
316	19820398	334	0.000079	com.live
317	19820034	1277	0.000025	org.postgresql
318	19818544	1257	0.000025	org.wiktionary
319	19817910	1274	0.000025	org.aclu
320	19817698	981	0.000033	edu.si
321	19816586	1394	0.000023	edu.msu
322	19816210	1029	0.000031	com.thehill
323	19814936	890	0.000036	de.spiegel
324	19813172	916	0.000035	com.huffpost
325	19811282	472	0.000060	gov.hhs
326	19809240	1114	0.000028	com.scmp
327	19806650	73	0.000484	me.fb
328	19806306	764	0.000042	org.change
329	19805070	378	0.000069	com.sohu
330	19804336	1329	0.000024	edu.illinois
331	19804164	185	0.000147	com.xing
332	19801192	1323	0.000024	org.tensorflow
333	19801086	1008	0.000032	com.ssrn
334	19800184	162	0.000171	com.zendesk
335	19798428	904	0.000035	com.netlify
336	19797294	508	0.000056	com.squareup
337	19797020	1352	0.000024	com.sky
338	19794400	196	0.000134	org.iana
339	19792714	1078	0.000029	uk.co.thetimes
340	19792494	847	0.000038	gov.congress
341	19788704	809	0.000039	org.pypi
342	19783878	1422	0.000023	cn.com.chinadaily
343	19781142	972	0.000033	edu.academia
344	19780974	456	0.000061	com.kickstarter
345	19780084	802	0.000040	gov.senate
346	19779128	2415	0.000015	org.pydata
347	19778124	1140	0.000027	org.semanticscholar
348	19775716	620	0.000045	site.business
349	19775012	1275	0.000025	com.over-blog
350	19774866	792	0.000040	org.oecd
351	19774846	1660	0.000020	org.phys
352	19774334	999	0.000032	com.yarnpkg
353	19772248	816	0.000039	com.deviantart
354	19770936	1084	0.000029	uk.co.mirror
355	19770522	187	0.000145	com.rawgit
356	19770114	1315	0.000024	com.axios
357	19769700	623	0.000045	gov.house
358	19768998	894	0.000036	com.discordapp
359	19768866	880	0.000036	com.sciencedaily
360	19766292	511	0.000055	com.gmail
361	19765678	423	0.000064	com.technorati
362	19763944	216	0.000123	com.hubspot
363	19761638	1433	0.000023	com.unity3d
364	19760768	2137	0.000017	org.threejs
365	19760238	1364	0.000024	com.aljazeera
366	19759580	245	0.000109	org.nodejs
367	19758846	896	0.000036	com.bmj
368	19755564	261	0.000101	com.ebay
369	19755198	1197	0.000026	au.com.smh
370	19753628	234	0.000113	org.gnu
371	19751964	1516	0.000021	edu.osu
372	19751362	1025	0.000031	int.coe
373	19750302	994	0.000032	com.britannica
374	19748408	1312	0.000024	edu.gatech
375	19746818	2691	0.000013	com.openai
376	19744370	495	0.000056	org.openstreetmap
377	19743086	437	0.000062	com.ssl-images-amazon
378	19741582	791	0.000040	br.com.google
379	19741030	855	0.000037	ca.cbc
380	19740484	869	0.000037	com.theconversation
381	19739852	2582	0.000014	edu.toronto
382	19738652	1044	0.000031	gov.usgs
383	19738306	1556	0.000021	com.newscientist
384	19736226	301	0.000088	net.themeforest
385	19735698	605	0.000046	com.udacity
386	19735668	473	0.000060	edu.nyu
387	19734084	1716	0.000019	edu.ucsc
388	19723708	1700	0.000020	org.emojipedia
389	19722194	2068	0.000017	it.scoop
390	19722024	2754	0.000013	com.slides
391	19721872	1459	0.000022	ca.sfu
392	19720004	845	0.000038	au.gov.nsw
393	19717908	1903	0.000019	org.propublica
394	19717586	1386	0.000023	com.firebaseapp
395	19716094	2247	0.000016	com.skyrock
396	19710516	776	0.000041	com.freepik
397	19707962	97	0.000412	net.facebook
398	19704900	1454	0.000022	com.penguinrandomhouse
399	19703572	195	0.000135	org.bbb
400	19703432	1934	0.000018	jp.co.japantimes
401	19701030	1762	0.000019	com.itv
402	19700818	82	0.000437	net.jsfiddle
403	19700616	1985	0.000018	org.maven
404	19699746	2370	0.000015	com.deepmind
405	19697844	617	0.000045	com.healthline
406	19695324	506	0.000056	de.gesetze-im-internet
407	19694720	465	0.000060	org.python
408	19694428	2331	0.000015	com.mystrikingly
409	19691536	884	0.000036	gov.dhs
410	19688238	1233	0.000026	com.wikia
411	19685986	2090	0.000017	org.sqlite
412	19682976	1544	0.000021	ms.1drv
413	19682094	178	0.000150	com.salesforce
414	19679914	322	0.000084	net.php
415	19671484	324	0.000083	com.surveymonkey
416	19670962	634	0.000044	com.mashable
417	19670338	1628	0.000020	com.motherjones
418	19668724	139	0.000211	com.weibo
419	19668554	2453	0.000014	com.fastcodesign
420	19667444	1506	0.000021	com.flipboard
421	19666746	2435	0.000015	edu.byu
422	19665482	1748	0.000019	edu.cuny
423	19664886	317	0.000085	ru.ok
424	19662618	287	0.000092	net.azureedge
425	19662108	1339	0.000024	com.thedailybeast
426	19659672	246	0.000109	org.aboutcookies
427	19658838	2283	0.000015	com.shutterfly
428	19656108	1413	0.000023	com.reverbnation
429	19655722	2666	0.000013	io.material
430	19655254	537	0.000052	io.codepen
431	19652776	1296	0.000025	com.dw
432	19651986	125	0.000250	com.youtube-nocookie
433	19650416	1724	0.000019	com.esri
434	19650188	490	0.000057	fr.free
435	19648416	1509	0.000021	com.substack
436	19647438	561	0.000049	com.matterport
437	19646584	1956	0.000018	com.hindustantimes
438	19645830	1909	0.000019	com.insider
439	19642342	2110	0.000017	edu.oregonstate
440	19641814	2390	0.000015	org.wikibooks
441	19640838	891	0.000036	int.wipo
442	19640244	2820	0.000013	org.aclweb
443	19639226	607	0.000045	gov.state
444	19638894	2366	0.000015	com.wattpad
445	19638652	160	0.000172	gle.forms
446	19636692	1052	0.000030	org.jstor
447	19636398	1951	0.000018	com.channel4
448	19636126	1752	0.000019	edu.ucsb
449	19635942	1320	0.000024	gov.supremecourt
450	19633994	56	0.000697	com.googleadservices
451	19631760	2441	0.000015	at.ac.univie
452	19629096	2924	0.000013	com.pbase
453	19626572	278	0.000097	uk.org.ico
454	19624802	639	0.000044	com.licdn
455	19623422	1518	0.000021	ch.ipcc
456	19621874	937	0.000034	com.gallup
457	19621780	496	0.000056	com.herokuapp
458	19618584	1141	0.000027	edu.brookings
459	19617388	963	0.000033	edu.psu
460	19616790	1333	0.000024	mil.army
461	19616626	434	0.000063	com.rackcdn
462	19614484	385	0.000068	com.atlassian
463	19611760	1226	0.000026	com.smashingmagazine
464	19609634	2227	0.000016	blog.home
465	19608450	1362	0.000024	gov.defense
466	19607698	1131	0.000028	com.photoshelter
467	19607464	483	0.000058	net.imgix
468	19607012	182	0.000149	jp.co.yahoo
469	19605316	2284	0.000015	com.contently
470	19602040	826	0.000039	com.oreilly
471	19597708	1174	0.000027	com.mediafire
472	19595596	2117	0.000017	com.thecut
473	19594604	1960	0.000018	google.ai
474	19594568	3151	0.000012	cc.uxdesign
475	19594280	3161	0.000012	edu.uvm
476	19594100	520	0.000054	edu.cmu
477	19593086	3137	0.000012	com.instapaper
478	19591090	1591	0.000020	com.thestar
479	19588378	369	0.000071	net.researchgate
480	19587214	3502	0.000011	com.raywenderlich
481	19587008	527	0.000053	com.thinkwithgoogle
482	19584868	2149	0.000016	fr.liberation
483	19582230	109	0.000336	de.google
484	19581418	1574	0.000021	com.buzzfeednews
485	19577648	767	0.000041	org.worldwildlife
486	19576662	1013	0.000032	com.ecwid
487	19576118	1477	0.000022	com.findlaw
488	19574804	1012	0.000032	com.thelancet
489	19573936	774	0.000041	com.vice
490	19573506	813	0.000039	gov.nist
491	19572872	1964	0.000018	org.google
492	19572508	1531	0.000021	org.hrw
493	19570410	765	0.000042	com.intel
494	19568238	2695	0.000013	uk.co.ibtimes
495	19567790	2372	0.000015	com.oprah
496	19567558	87	0.000428	com.workplace
497	19567194	3329	0.000011	com.pearltrees
498	19567174	2103	0.000017	com.voanews
499	19566762	965	0.000033	com.engadget
500	19566188	126	0.000247	com.statcounter
501	19564772	3365	0.000011	org.edublogs
502	19563980	1260	0.000025	org.aiga
503	19562828	1031	0.000031	de.stern
504	19562068	1583	0.000020	fr.francetvinfo
505	19560196	2620	0.000014	com.hm
506	19559342	315	0.000085	org.drupal
507	19559132	3736	0.000010	fr.unblog
508	19558786	747	0.000042	com.canva
509	19558362	2870	0.000013	edu.ucf
510	19558064	3204	0.000012	ph.telegra
511	19557534	926	0.000035	uk.co.pinterest
512	19557072	2402	0.000015	edu.kit
513	19556358	544	0.000051	it.placehold
514	19555528	2219	0.000016	net.corporate-ir
515	19553910	2768	0.000013	co.ello
516	19553426	881	0.000036	com.arstechnica
517	19553018	1449	0.000022	com.livescience
518	19550968	2150	0.000016	com.gq
519	19550836	1953	0.000018	uk.gov.tfl
520	19550254	210	0.000125	com.iubenda
521	19550042	533	0.000053	com.pixabay
522	19548328	1408	0.000023	org.undp
523	19547668	807	0.000039	ca.amazon
524	19547226	1020	0.000031	it.smarturl
525	19547032	2645	0.000014	org.icrc
526	19546934	2447	0.000015	com.webbyawards
527	19545564	2423	0.000015	uk.ac.kcl
528	19545554	949	0.000034	edu.ucla
529	19544462	1444	0.000022	link.page
530	19543968	2861	0.000013	com.dummies
531	19541366	1581	0.000021	org.ocks
532	19540748	65	0.000544	net.typekit
533	19540022	1122	0.000028	org.ilo
534	19538882	2564	0.000014	com.depositphotos
535	19538866	2502	0.000014	com.unilever
536	19536950	1348	0.000024	org.acs
537	19536262	81	0.000440	com.livestream
538	19535098	2672	0.000013	org.rsf
539	19535076	489	0.000057	com.adweek
540	19534044	2050	0.000017	com.msnbc
541	19530220	2509	0.000014	com.slidesharecdn
542	19530084	2035	0.000018	com.chronicle
543	19529836	3088	0.000012	com.bepress
544	19529580	2571	0.000014	com.biography
545	19529322	3384	0.000011	tl.de
546	19527886	332	0.000079	com.typeform
547	19526428	2185	0.000016	com.newrepublic
548	19525400	2303	0.000015	com.thoughtco
549	19523856	606	0.000045	com.samsung
550	19523112	1100	0.000029	org.ohchr
551	19522668	790	0.000040	com.fiverr
552	19521518	1743	0.000019	io.gitlab
553	19521240	121	0.000262	com.jimdo
554	19520292	1157	0.000027	com.thenextweb
555	19520070	2009	0.000018	fr.orange
556	19519618	3272	0.000012	net.openreview
557	19518936	2294	0.000015	com.channelnewsasia
558	19517090	1283	0.000025	org.aarp
559	19516918	2634	0.000014	org.pewsocialtrends
560	19516476	1998	0.000018	com.straitstimes
561	19513936	2310	0.000015	edu.nd
562	19510956	2099	0.000017	com.dallasnews
563	19510732	2130	0.000017	de.br
564	19508818	2278	0.000015	org.fas
565	19508000	1297	0.000024	org.altervista
566	19507978	256	0.000103	uk.co.amazon
567	19507290	219	0.000121	to.amzn
568	19506624	2835	0.000013	com.thejakartapost
569	19505128	2211	0.000016	gov.lbl
570	19504556	1610	0.000020	de.berlin
571	19504362	1086	0.000029	com.popularmechanics
572	19503706	2743	0.000013	uk.ac.leeds
573	19503644	459	0.000061	com.staticflickr
574	19503210	3397	0.000011	org.neocities
575	19502358	2996	0.000012	org.vim
576	19502186	2883	0.000013	org.globalcitizen
577	19499450	572	0.000048	com.deloitte
578	19499392	922	0.000035	com.zoho
579	19498964	233	0.000113	io.shields
580	19498936	2328	0.000015	com.indianexpress
581	19498902	3889	0.000010	com.stratechery
582	19497728	2819	0.000013	app.web
583	19496358	3386	0.000011	org.zotero
584	19493624	2939	0.000013	uk.gov.scotland
585	19493314	567	0.000048	com.photobucket
586	19491524	3756	0.000010	com.bravesites
587	19490552	1464	0.000022	org.iea
588	19489976	432	0.000063	com.hp
589	19489954	2713	0.000013	uk.co.timesonline
590	19489478	365	0.000073	com.quantserve
591	19489336	404	0.000066	com.digg
592	19486660	560	0.000049	com.cisco
593	19486618	1155	0.000027	uk.parliament
594	19485014	2914	0.000013	com.nwsource
595	19485012	2362	0.000015	com.fineartamerica
596	19484598	267	0.000101	com.onesignal
597	19484238	2234	0.000016	com.foreignpolicy
598	19484200	798	0.000040	org.weforum
599	19483398	2990	0.000012	com.thoughtworks
600	19483202	1548	0.000021	com.treehugger
601	19482398	307	0.000087	com.aliyuncs
602	19482224	602	0.000046	org.js
603	19480232	1527	0.000021	gov.uscis
604	19479040	3256	0.000012	uk.ac.city
605	19477476	2077	0.000017	com.washingtontimes
606	19477198	3504	0.000011	com.mariadb
607	19476316	2565	0.000014	org.oas
608	19475236	417	0.000065	com.gitlab
609	19472258	2584	0.000014	com.mathworks
610	19471752	2830	0.000013	com.dezeen
611	19471284	835	0.000038	com.investopedia
612	19470638	2497	0.000014	uk.co.yougov
613	19469316	2934	0.000013	org.heritage
614	19469308	614	0.000045	com.netflix
615	19466252	3281	0.000011	com.shell
616	19465388	2540	0.000014	fr.paris
617	19464956	448	0.000061	gov.irs
618	19462732	4088	0.000009	tl.page
619	19461330	1361	0.000024	com.upwork
620	19461170	462	0.000061	com.sxsw
621	19460914	1255	0.000025	com.digitaloceanspaces
622	19460548	4091	0.000009	com.jigsy
623	19460066	861	0.000037	com.venturebeat
624	19458418	1215	0.000026	com.dell
625	19457348	1016	0.000031	gov.fcc
626	19456828	3229	0.000012	uk.co.walesonline
627	19456346	2961	0.000013	org.project-syndicate
628	19455696	2024	0.000018	com.fivethirtyeight
629	19455242	920	0.000035	fm.last
630	19455056	2086	0.000017	info.worldometers
631	19454252	931	0.000034	org.mediawiki
632	19453670	2377	0.000015	ly.rebrand
633	19453158	4077	0.000009	net.myanimelist
634	19452824	2075	0.000017	cn.gov.fmprc
635	19452012	1502	0.000021	org.amnesty
636	19450548	349	0.000077	com.adnxs
637	19449350	1945	0.000018	com.justia
638	19448712	4019	0.000009	edu.usfca
639	19448298	2705	0.000013	com.monday
640	19446576	1515	0.000021	ca.bc.gov
641	19446486	943	0.000034	org.reactjs
642	19446126	2285	0.000015	net.openid
643	19445904	383	0.000068	com.newrelic
644	19445366	1363	0.000024	com.imageshack
645	19445144	3568	0.000010	org.globalnetworkinitiative
646	19443940	2549	0.000014	com.kaggle
647	19443562	3693	0.000010	com.doodlekit
648	19439792	259	0.000102	com.getbootstrap
649	19438670	2831	0.000013	uk.co.inews
650	19438312	3129	0.000012	com.bangkokpost
651	19438230	409	0.000065	com.force
652	19437908	2107	0.000017	uk.ac.imperial
653	19435434	4629	0.000008	net.vingle
654	19434150	1982	0.000018	be.kuleuven
655	19434066	3530	0.000011	com.intensedebate
656	19432926	568	0.000048	com.entrepreneur
657	19432350	3518	0.000011	be.blogspot
658	19429740	3166	0.000012	se.blogspot
659	19429712	1318	0.000024	co.lpages
660	19428992	3266	0.000012	org.carnegieendowment
661	19428674	837	0.000038	com.globenewswire
662	19428662	3175	0.000012	is.good
663	19428098	2246	0.000016	com.instructure
664	19427698	2965	0.000012	net.alarabiya
665	19427204	4090	0.000009	com.kongregate
666	19426514	2795	0.000013	com.discovermagazine
667	19425746	2613	0.000014	org.gnupg
668	19425518	556	0.000049	com.visualstudio
669	19424130	191	0.000139	com.atdmt
670	19423528	3773	0.000010	com.openlearning
671	19423230	3794	0.000010	ch.swissinfo
672	19421982	3547	0.000010	com.pixar
673	19420080	2154	0.000016	com.livemint
674	19419708	957	0.000033	com.variety
675	19417142	2816	0.000013	uk.gov.metoffice
676	19414346	2004	0.000018	com.surveygizmo
677	19412994	3337	0.000011	cn.globaltimes
678	19411212	929	0.000035	uk.gov.legislation
679	19411070	2639	0.000014	org.ballotpedia
680	19409736	243	0.000110	org.whatwg
681	19408620	3148	0.000012	com.coca-colacompany
682	19408342	1343	0.000024	uk.gov.nationalarchives
683	19406168	2326	0.000015	com.thebalancesmb
684	19404822	3145	0.000012	uk.gov.companieshouse
685	19403088	3532	0.000011	com.dailykos
686	19401008	165	0.000170	com.yelp
687	19400512	257	0.000103	com.automattic
688	19400270	4169	0.000009	com.penzu
689	19399686	2489	0.000014	com.bloomberglaw
690	19399662	412	0.000065	org.opensource
691	19398126	1547	0.000021	org.khanacademy
692	19397376	3834	0.000010	com.sfweekly
693	19395236	2779	0.000013	com.thumbtack
694	19394202	2880	0.000013	org.royalsociety
695	19393684	1674	0.000020	kr.co.google
696	19393678	2531	0.000014	com.post-gazette
697	19393520	2800	0.000013	org.panda
698	19390648	2421	0.000015	com.thenation
699	19389714	2823	0.000013	io.fabric
700	19388974	4936	0.000008	org.arkive
701	19388756	2689	0.000013	uk.co.bbci
702	19387624	4042	0.000009	hk.edu.cityu
703	19387406	3194	0.000012	com.scribblelive
704	19386352	3553	0.000010	com.gimletmedia
705	19385872	3489	0.000011	com.tweetmeme
706	19384830	2541	0.000014	de.uni-heidelberg
707	19384284	298	0.000089	ai.shortpixel
708	19383872	1920	0.000019	gov.gao
709	19382974	4425	0.000008	com.storeboard
710	19381650	2814	0.000013	com.politifact
711	19380202	3349	0.000011	org.cato
712	19379282	4889	0.000008	com.uberant
713	19377306	3183	0.000012	fr.lepoint
714	19377194	3809	0.000010	edu.depaul
715	19376126	3844	0.000010	net.thedailystar
716	19375590	406	0.000066	com.aol
717	19375570	4046	0.000009	edu.umt
718	19372794	1948	0.000018	tv.ustream
719	19372628	1034	0.000031	com.verisign
720	19369588	3279	0.000011	com.theweek
721	19367934	905	0.000035	com.box
722	19367170	3724	0.000010	com.eklablog
723	19365850	3488	0.000011	com.militarytimes
724	19365832	866	0.000037	gov.uspto
725	19365580	3483	0.000011	com.multiscreensite
726	19364098	3103	0.000012	uk.ac.york
727	19359488	3165	0.000012	org.openweathermap
728	19358574	1526	0.000021	com.techrepublic
729	19358070	3315	0.000011	org.jenkins-ci
730	19357968	2815	0.000013	org.wnyc
731	19357458	638	0.000044	gov.copyright
732	19356834	3433	0.000011	com.lawfareblog
733	19354610	2357	0.000015	co.pcdn
734	19353004	3263	0.000012	com.nyt
735	19352766	3101	0.000012	se.svt
736	19351866	1048	0.000030	net.clickbank
737	19351546	3121	0.000012	com.scotsman
738	19348720	1182	0.000027	com.foursquare
739	19348660	1239	0.000026	com.pingdom
740	19348048	2475	0.000014	com.squarespace-cdn
741	19346678	2323	0.000015	com.natlawreview
742	19346350	2769	0.000013	org.wri
743	19345800	3430	0.000011	com.bigthink
744	19345054	4132	0.000009	com.newgrounds
745	19344692	3862	0.000010	org.sourcewatch
746	19342356	3720	0.000010	re.cli
747	19341788	3156	0.000012	gov.ncjrs
748	19341458	3087	0.000012	my.com.thestar
749	19340698	3307	0.000011	gov.anl
750	19339932	3117	0.000012	com.nationalreview
751	19339132	2597	0.000014	ca.newswire
752	19338090	1603	0.000020	org.webkit
753	19337402	3700	0.000010	org.elasticsearch
754	19335276	928	0.000035	com.hootsuite
755	19334936	300	0.000088	com.caniuse
756	19334252	3236	0.000012	gov.fec
757	19333910	2327	0.000015	ru.rg
758	19333124	3741	0.000010	org.constitutioncenter
759	19332102	1602	0.000020	com.jwplayer
760	19331754	4253	0.000009	com.etymonline
761	19331678	3620	0.000010	it.eventbrite
762	19331510	2960	0.000013	com.madmimi
763	19331460	3491	0.000011	com.afp
764	19330192	1907	0.000019	com.kinstacdn
765	19328136	3163	0.000012	gov.ornl
766	19327042	461	0.000061	com.pubmatic
767	19325866	401	0.000066	gg.discord
768	19325518	1289	0.000025	com.intuit
769	19325482	1168	0.000027	com.ycombinator
770	19325258	3292	0.000011	com.crashlytics
771	19324302	4270	0.000009	com.underconsideration
772	19322856	2599	0.000014	com.articulate
773	19322230	3246	0.000012	de.uni-frankfurt
774	19321496	3692	0.000010	uk.co.spectator
775	19321096	867	0.000037	com.wikihow
776	19321010	4275	0.000009	to.gplus
777	19320802	4920	0.000008	pl.pastebin
778	19320622	3791	0.000010	uk.co.manchestereveningnews
779	19319854	2938	0.000013	edu.unh
780	19318976	2553	0.000014	de.tagesschau
781	19318802	2116	0.000017	gov.energystar
782	19318372	429	0.000063	com.businesswire
783	19318050	829	0.000038	com.moz
784	19314848	3550	0.000010	org.avaaz
785	19314554	3683	0.000010	com.mnn
786	19314476	1172	0.000027	com.alexa
787	19314150	2332	0.000015	net.vnexpress
788	19313268	348	0.000077	com.constantcontact
789	19312732	3600	0.000010	com.heraldscotland
790	19312326	3843	0.000010	fm.audioboo
791	19311750	4481	0.000008	tv.eurovision
792	19311646	974	0.000033	com.fandom
793	19311256	3717	0.000010	uk.ac.uea
794	19311174	3697	0.000010	uk.ac.core
795	19310268	3514	0.000011	com.hsbc
796	19310254	3492	0.000011	org.sciencenews
797	19310242	4916	0.000008	com.blackplanet
798	19310096	3289	0.000011	com.realclearpolitics
799	19309366	1698	0.000020	com.pastebin
800	19309196	3190	0.000012	uk.org.rspb
801	19308322	1377	0.000023	com.techradar
802	19308094	529	0.000053	com.indeed
803	19307548	4985	0.000007	dk.bloggersdelight
804	19307144	4491	0.000008	com.xtgem
805	19306108	2073	0.000017	ca.on.gov
806	19305500	3536	0.000011	uk.co.thisismoney
807	19304908	797	0.000040	gov.sec
808	19302330	1128	0.000028	net.atlassian
809	19302240	3937	0.000009	com.collinsdictionary
810	19299944	1479	0.000022	edu.purdue
811	19299020	3179	0.000012	com.wayfair
812	19298908	3611	0.000010	org.chathamhouse
813	19297900	3218	0.000012	org.rferl
814	19297216	397	0.000066	com.skype
815	19296536	4738	0.000008	edu.ualr
816	19296016	3523	0.000011	org.diva-portal
817	19295672	2785	0.000013	org.cfr
818	19294806	1249	0.000025	com.merriam-webster
819	19292968	4835	0.000008	com.designobserver
820	19292734	3399	0.000011	org.pewforum
821	19292200	270	0.000100	jp.co.amazon
822	19291468	3994	0.000009	uk.co.dailyrecord
823	19290936	3951	0.000009	edu.swarthmore
824	19290570	3339	0.000011	com.ubs
825	19289748	1075	0.000030	so.notion
826	19289742	2847	0.000013	us.govtrack
827	19289236	1256	0.000025	com.udemy
828	19289040	333	0.000079	com.hackerone
829	19288716	3787	0.000010	org.nationalinterest
830	19288626	3138	0.000012	com.doubleclickbygoogle
831	19288000	279	0.000097	de.amazon
832	19287244	2036	0.000018	org.doxygen
833	19286840	1661	0.000020	scot.gov
834	19286652	3933	0.000009	de.berliner-zeitung
835	19285868	1519	0.000021	com.billboard
836	19283910	681	0.000042	com.gartner
837	19283390	4698	0.000008	net.writeablog
838	19282688	2465	0.000014	com.infoworld
839	19282084	823	0.000039	com.sedo
840	19281700	3200	0.000012	org.aei
841	19280820	71	0.000502	com.oculus
842	19280652	1580	0.000021	edu.ucsd
843	19280396	329	0.000081	mp.mailchi
844	19280288	3917	0.000009	edu.umaine
845	19279222	3262	0.000012	org.iucnredlist
846	19279130	2827	0.000013	com.lexology
847	19278304	4851	0.000008	com.nation2
848	19278156	5290	0.000007	com.anotepad
849	19278056	4128	0.000009	za.co.mg
850	19276824	77	0.000467	com.messenger
851	19276460	2083	0.000017	org.dejure
852	19276002	4494	0.000008	net.blogfreely
853	19275630	1302	0.000024	org.owasp
854	19275142	3309	0.000011	com.foreignaffairs
855	19275092	4067	0.000009	tw.com.books
856	19274916	4267	0.000009	ca.nfb
857	19274822	364	0.000073	com.bitly
858	19274560	3225	0.000012	org.osce
859	19274028	3726	0.000010	uk.org.wwf
860	19274006	3971	0.000009	org.truthout
861	19273104	155	0.000178	gov.privacyshield
862	19272708	1981	0.000018	edu.uci
863	19272368	2044	0.000017	se.haxx
864	19272288	897	0.000036	com.emarketer
865	19272110	4532	0.000008	com.symbaloo
866	19271508	1004	0.000032	com.playstation
867	19271338	2196	0.000016	org.sundance
868	19271216	363	0.000073	eu.youronlinechoices
869	19271196	3496	0.000011	com.rev
870	19271080	4071	0.000009	in.thewire
871	19270976	159	0.000174	org.nginx
872	19270528	903	0.000036	com.libsyn
873	19268650	2400	0.000015	us.pa.state
874	19267610	146	0.000205	me.line
875	19267478	5202	0.000007	net.bravejournal
876	19267386	3140	0.000012	ru.kp
877	19267334	4014	0.000009	com.ecowatch
878	19266700	514	0.000055	org.debian
879	19266302	539	0.000052	com.gofundme
880	19266194	976	0.000033	com.pcmag
881	19264914	4151	0.000009	com.theoutline
882	19264512	4316	0.000009	org.icj-cij
883	19263626	1470	0.000022	org.coursera
884	19261610	2076	0.000017	gov.healthcare
885	19260626	3721	0.000010	com.iconarchive
886	19259734	1657	0.000020	net.leadpages
887	19259034	1486	0.000022	com.technologyreview
888	19258032	2367	0.000015	ca.citizenlab
889	19257884	3690	0.000010	com.governing
890	19257782	3322	0.000011	com.wikidot
891	19257260	2385	0.000015	org.raspberrypi
892	19256452	4621	0.000008	jp.ac.kobe-u
893	19255454	1073	0.000030	com.timeanddate
894	19254836	1096	0.000029	com.buffer
895	19254032	3978	0.000009	com.ogilvy
896	19251530	940	0.000034	com.css-tricks
897	19251096	1501	0.000021	com.msdn
898	19250138	3958	0.000009	com.gab
899	19249994	3673	0.000010	com.what3words
900	19249260	1241	0.000026	com.tableau
901	19248316	1319	0.000024	com.xkcd
902	19248224	3695	0.000010	com.nestle
903	19247678	4982	0.000007	net.postheaven
904	19246428	470	0.000060	com.fc2
905	19246238	1795	0.000019	com.pcworld
906	19246028	2589	0.000014	mp.j
907	19245754	4318	0.000009	org.kuow
908	19245300	3906	0.000009	org.migrationpolicy
909	19245282	585	0.000047	com.fortune
910	19244324	3769	0.000010	de.morgenpost
911	19244120	3282	0.000011	uk.gov.data
912	19243558	4952	0.000007	cz.webgarden
913	19243100	2118	0.000017	org.donorbox
914	19242192	3909	0.000009	de.uni-konstanz
915	19241684	4218	0.000009	org.birdlife
916	19240982	3875	0.000010	org.people-press
917	19240778	2132	0.000017	to.dev
918	19239846	906	0.000035	org.golang
919	19238732	2425	0.000015	net.noscript
920	19237742	1223	0.000026	com.podbean
921	19235906	4130	0.000009	com.scienceblogs
922	19235706	4948	0.000007	it.clyp
923	19235498	3355	0.000011	edu.fordham
924	19231696	4076	0.000009	org.oyez
925	19230656	3441	0.000011	com.joebiden
926	19229960	2867	0.000013	com.washingtonexaminer
927	19229728	1115	0.000028	com.gizmodo
928	19229112	2757	0.000013	org.healthaffairs
929	19228910	1232	0.000026	com.searchengineland
930	19228678	854	0.000037	fm.anchor
931	19227412	5084	0.000007	com.zcubes
932	19227258	1995	0.000018	com.ssllabs
933	19225964	1072	0.000030	org.poynter
934	19224436	1644	0.000020	net.java
935	19223632	1514	0.000021	edu.usc
936	19223252	3680	0.000010	org.carbonbrief
937	19221502	5165	0.000007	org.csgrid
938	19221286	308	0.000087	jp.ameblo
939	19220064	1578	0.000021	com.sun
940	19220010	3959	0.000009	org.rfa
941	19218588	2616	0.000014	uk.gov.defra
942	19218556	3912	0.000009	com.exxonmobil
943	19218102	5249	0.000007	com.topsitenet
944	19217732	3012	0.000012	com.html5rocks
945	19217494	3660	0.000010	ca.yelp
946	19216576	2940	0.000013	com.instructables
947	19215582	2212	0.000016	org.linuxfoundation
948	19215410	4069	0.000009	uk.org.woodlandtrust
949	19213854	2058	0.000017	org.json
950	19213790	214	0.000124	com.tripadvisor
951	19212490	5233	0.000007	net.squareblogs
952	19212378	3864	0.000010	ru.mid
953	19212170	231	0.000113	com.myshopify
954	19211108	3310	0.000011	com.flippa
955	19211092	3850	0.000010	com.townandcountrymag
956	19210938	1292	0.000025	build.bazel
957	19210816	5295	0.000007	net.werite
958	19210212	1240	0.000026	com.uk
959	19209742	2354	0.000015	com.storify
960	19209508	3280	0.000011	org.cjr
961	19208854	3158	0.000012	org.acog
962	19208448	3921	0.000009	br.com.sebrae
963	19208380	250	0.000107	org.icann
964	19207876	1635	0.000020	fr.blogspot
965	19207582	122	0.000262	com.bizjournals
966	19207280	3406	0.000011	org.cites
967	19207102	2687	0.000013	com.tutsplus
968	19207058	3409	0.000011	tr.com.aa
969	19206380	1109	0.000028	org.whatbrowser
970	19205750	4680	0.000008	org.learner
971	19205144	3424	0.000011	no.yr
972	19203738	4271	0.000009	com.s-nbcnews
973	19203150	4166	0.000009	org.spie
974	19203082	1335	0.000024	com.indiegogo
975	19202634	708	0.000042	com.airbnb
976	19202288	4217	0.000009	com.revolut
977	19201514	4339	0.000009	org.atsjournals
978	19201386	1033	0.000031	com.redhat
979	19200760	4066	0.000009	uk.co.zoopla
980	19199826	318	0.000084	it.google
981	19199246	1137	0.000028	com.windowsphone
982	19198666	1485	0.000022	edu.unc
983	19198508	466	0.000060	gov.fda
984	19198408	653	0.000043	com.zapier
985	19198272	2161	0.000016	com.gigaom
986	19197316	4457	0.000008	ru.novayagazeta
987	19196504	1936	0.000018	br.com.correios
988	19196468	4101	0.000009	google.design
989	19195350	2194	0.000016	org.eu
990	19192238	3758	0.000010	com.mail-archive
991	19191310	4437	0.000008	com.out
992	19191000	4759	0.000008	tw.focustaiwan
993	19190946	4235	0.000009	org.insideclimatenews
994	19190774	2038	0.000017	com.freeprivacypolicy
995	19190442	4265	0.000009	org.escardio
996	19190354	4663	0.000008	com.theschooloflife
997	19189766	241	0.000111	com.naver
998	19188362	4711	0.000008	edu.uah
999	19188230	1611	0.000020	com.nike
1000	19187360	4319	0.000009	edu.mtsu

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

May 2021 crawl archive now available

May 23, 2021Sebastian Nagel

The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.28 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The May crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-21/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-21/segment.paths.gz	100
WARC files	CC-MAIN-2021-21/warc.paths.gz	64000	66.17
WAT files	CC-MAIN-2021-21/wat.paths.gz	64000	17.61
WET files	CC-MAIN-2021-21/wet.paths.gz	64000	7.65
Robots.txt files	CC-MAIN-2021-21/robotstxt.paths.gz	64000	0.17
Non-200 responses files	CC-MAIN-2021-21/non200responses.paths.gz	64000	1.86
URL index files	CC-MAIN-2021-21/cc-index.paths.gz	302	0.2

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-21/. Also the columnar index has been updated to contain this crawl.

April 2021 crawl archive now available

April 27, 2021Sebastian Nagel

The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content. It includes page captures of 1.35 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The April crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-17/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-17/segment.paths.gz	100
WARC files	CC-MAIN-2021-17/warc.paths.gz	64000	69.78
WAT files	CC-MAIN-2021-17/wat.paths.gz	64000	21.05
WET files	CC-MAIN-2021-17/wet.paths.gz	64000	9.25
Robots.txt files	CC-MAIN-2021-17/robotstxt.paths.gz	64000	0.16
Non-200 responses files	CC-MAIN-2021-17/non200responses.paths.gz	64000	1.76
URL index files	CC-MAIN-2021-17/cc-index.paths.gz	302	0.23

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-17/. Also the columnar index has been updated to contain this crawl.

February/March 2021 crawl archive now available

March 14, 2021Sebastian Nagel

The crawl archive for February/March 2021 is now available! The data was crawled between February 24th and March 9th and contains 2.7 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.2 billion new URLs, not visited in any of our prior crawls.

Improvements and Fixes

The ISO639-3 code for the Hmong language was updated to "hmn" – the code "blu" used so far was already deprecated in 2008. Crawl archives prior to this crawl will still use the code "blu". More details about this update are found here.

Archive Location and Download

The February/March crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-10/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-10/segment.paths.gz	100
WARC files	CC-MAIN-2021-10/warc.paths.gz	64000	62.51
WAT files	CC-MAIN-2021-10/wat.paths.gz	64000	18.44
WET files	CC-MAIN-2021-10/wet.paths.gz	64000	8.06
Robots.txt files	CC-MAIN-2021-10/robotstxt.paths.gz	64000	0.2
Non-200 responses files	CC-MAIN-2021-10/non200responses.paths.gz	64000	1.58
URL index files	CC-MAIN-2021-10/cc-index.paths.gz	302	0.21

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-10/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs October, November/December 2020 and January 2021

February 10, 2021Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of October, November/December 2020 and January 2021. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018 Webgraphs). You may also visit the projects cc-webgraph and cc-pyspark which include all scripts and tools required to construct the graphs. Instructions to explore the graphs in the webgraph format are given in our collection of webgraph notebooks.

Host-level graph

The graph consists of 490 million nodes and 2.57 billion edges and includes dangling nodes i.e. hosts that have not been crawled yet are pointed to from a link on a crawled page. There are 414 million dangling nodes (84.4%) and the largest strongly connected component contains 42.6 million (8.7%) nodes.

Host names in the graph are in reverse domain name notation and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 490 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2020-21-oct-nov-jan/host/. Alternatively, you can use https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2020-21-oct-nov-jan/host/ as prefix to access the files from everywhere.

Please note that the text representation of the host-level graph is shipped in 36 gzip-compressed files listed in two path listings – one for the nodes, one for the edges. First, download the paths listing and uncompress it using “gzip”. By adding the prefix s3://commoncrawl/ or https://commoncrawl.s3.amazonaws.com/ to each line in the path listing you get the list of URLs to download the entire graph.

Download files of the Common Crawl Oct/Nov/Jan 2020-2021 host-level webgraph

Size	File	Description
3.08 GB	cc-main-2020-21-oct-nov-jan-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 12 vertices files
11.76 GB	cc-main-2020-21-oct-nov-jan-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 24 edges files

5.18 GB	cc-main-2020-21-oct-nov-jan-host.graph	graph in BVGraph format
2 kB	cc-main-2020-21-oct-nov-jan-host.properties

5.63 GB	cc-main-2020-21-oct-nov-jan-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2020-21-oct-nov-jan-host-t.properties

1 kB	cc-main-2020-21-oct-nov-jan-host.stats	WebGraph statistics

7.04 GB	cc-main-2020-21-oct-nov-jan-host-ranks.txt.gz	harmonic centrality and pagerank

Domain-level graph

The domain graph was built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 86 million nodes and 1.47 billion edges. 50% or 43 million nodes are dangling nodes, the largest strongly connected component covers 34 million or 39% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2020-21-oct-nov-jan/domain/ resp. https://commoncrawl.s3.amazonaws.com/projects/hyperlinkgraph/cc-main-2020-21-oct-nov-jan/domain/.

Download files of the Common Crawl Oct/Nov/Jan 2020-2021 domain-level webgraph

Size	File	Description
0.59 GB	cc-main-2020-21-oct-nov-jan-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
6.00 GB	cc-main-2020-21-oct-nov-jan-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

3.40 GB	cc-main-2020-21-oct-nov-jan-domain.graph	graph in BVGraph format
2 kB	cc-main-2020-21-oct-nov-jan-domain.properties

3.26 GB	cc-main-2020-21-oct-nov-jan-domain-t.graph	transpose of the graph
2 kB	cc-main-2020-21-oct-nov-jan-domain-t.properties

1 kB	cc-main-2020-21-oct-nov-jan-domain.stats	WebGraph statistics

1.85 GB	cc-main-2020-21-oct-nov-jan-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 86 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Oct/Nov/Jan 2020-2021)

harmonic centrality rank	hc value	page rank	page rank value	reversed domain name
1	30355566	1	0.017956	com.googleapis
2	29427164	3	0.012871	com.facebook
3	28173562	2	0.012899	com.google
4	25702812	5	0.007348	com.twitter
5	25628314	4	0.007628	org.w
6	25297808	6	0.007231	com.youtube
7	24195466	9	0.005352	com.instagram
8	23355356	8	0.005532	org.gmpg
9	23233674	7	0.006500	com.googletagmanager
10	22492432	11	0.003277	com.linkedin
11	21576402	10	0.004076	com.cloudflare
12	21468510	14	0.002649	com.gravatar
13	21395642	13	0.003020	org.wordpress
14	21353798	22	0.001726	com.pinterest
15	20946722	30	0.001242	org.wikipedia
16	20926308	19	0.001834	com.wordpress
17	20877776	16	0.002056	com.gstatic
18	20799472	15	0.002451	com.bootstrapcdn
19	20795402	18	0.001943	com.apple
20	20626472	32	0.001165	com.vimeo
21	20527986	41	0.000886	be.youtu
22	20419038	21	0.001769	com.jquery
23	20391686	28	0.001246	com.microsoft
24	20327544	24	0.001500	com.wp
25	20314602	45	0.000769	com.blogspot
26	20231490	37	0.001025	com.amazonaws
27	20208912	51	0.000691	com.amazon
28	20199388	47	0.000740	gl.goo
29	20093688	71	0.000448	com.tumblr
30	20070176	35	0.001070	com.google-analytics
31	20050256	61	0.000598	ly.bit
32	20030452	20	0.001794	com.adobe
33	19998314	17	0.002005	com.github
34	19989010	50	0.000715	org.mozilla
35	19962834	58	0.000639	eu.europa
36	19945306	34	0.001103	net.cloudfront
37	19849112	52	0.000682	com.flickr
38	19843288	40	0.000909	net.jsdelivr
39	19833032	91	0.000369	com.googleusercontent
40	19823560	105	0.000347	com.yahoo
41	19752300	56	0.000650	co.t
42	19722088	33	0.001114	com.googlesyndication
43	19712406	23	0.001517	com.fontawesome
44	19708354	81	0.000392	com.weebly
45	19706054	55	0.000653	com.paypal
46	19695288	109	0.000308	com.reddit
47	19641534	31	0.001231	me.wp
48	19640398	73	0.000435	com.medium
49	19635162	67	0.000491	io.github
50	19590444	137	0.000225	com.nytimes
51	19587880	121	0.000280	com.soundcloud
52	19585192	27	0.001262	ru.yandex
53	19583494	43	0.000786	com.addthis
54	19582250	44	0.000776	com.macromedia
55	19560416	66	0.000504	org.w3
56	19549714	70	0.000451	com.shopify
57	19518672	146	0.000201	com.forbes
58	19502448	144	0.000205	org.archive
59	19496300	90	0.000371	org.creativecommons
60	19490348	194	0.000131	uk.co.bbc
61	19482926	59	0.000630	org.schema
62	19479528	39	0.000910	com.baidu
63	19464572	36	0.001035	net.doubleclick
64	19459966	200	0.000129	com.cnn
65	19451100	53	0.000677	com.whatsapp
66	19449068	60	0.000611	com.vk
67	19444966	206	0.000126	net.slideshare
68	19443956	158	0.000169	com.bing
69	19419878	174	0.000152	com.imdb
70	19385956	186	0.000140	com.imgur
71	19372520	236	0.000112	com.washingtonpost
72	19371076	176	0.000150	com.theguardian
73	19356952	254	0.000102	com.wsj
74	19356474	210	0.000123	org.wikimedia
75	19352128	219	0.000117	com.businessinsider
76	19347698	209	0.000123	com.stackoverflow
77	19342712	409	0.000065	com.msn
78	19326654	327	0.000079	com.appspot
79	19324334	157	0.000172	int.who
80	19321112	216	0.000119	edu.stanford
81	19316796	179	0.000148	org.apache
82	19310390	333	0.000078	com.ibm
83	19309354	337	0.000077	edu.mit
84	19304938	225	0.000116	net.sourceforge
85	19292932	116	0.000288	com.ytimg
86	19287812	57	0.000649	net.fbcdn
87	19282486	285	0.000091	com.techcrunch
88	19276500	269	0.000094	com.bbc
89	19275480	155	0.000181	com.wixsite
90	19275222	152	0.000189	gov.nih
91	19275200	220	0.000117	com.livejournal
92	19270650	233	0.000113	uk.co.google
93	19270610	440	0.000062	gov.nasa
94	19263354	54	0.000666	com.googleadservices
95	19243404	262	0.000097	edu.harvard
96	19243154	270	0.000094	com.oracle
97	19243126	276	0.000093	org.acm
98	19238650	218	0.000117	org.ietf
99	19238450	185	0.000142	com.blogger
100	19238426	223	0.000116	gov.ca
101	19234630	465	0.000059	fr.free
102	19232058	259	0.000098	com.bloomberg
103	19221844	275	0.000093	com.android
104	19218636	304	0.000085	com.live
105	19210812	126	0.000271	com.jimdo
106	19208896	169	0.000159	com.issuu
107	19205802	166	0.000162	com.giphy
108	19194156	438	0.000062	com.ted
109	19190178	348	0.000075	com.huffingtonpost
110	19187782	130	0.000254	com.weibo
111	19186862	154	0.000186	us.zoom
112	19185794	252	0.000103	org.gnu
113	19176332	403	0.000066	com.myspace
114	19162122	1039	0.000030	com.wikia
115	19152582	373	0.000071	net.researchgate
116	19150058	343	0.000075	com.usatoday
117	19148332	309	0.000084	com.reuters
118	19143988	400	0.000067	uk.co.telegraph
119	19141202	446	0.000061	com.latimes
120	19130976	372	0.000071	com.example
121	19129552	345	0.000075	com.githubusercontent
122	19127344	93	0.000366	com.unpkg
123	19127116	384	0.000069	com.nature
124	19125396	336	0.000077	com.wired
125	19124320	25	0.001485	com.wixstatic
126	19114842	299	0.000087	org.npr
127	19111018	308	0.000084	com.cnbc
128	19107772	328	0.000079	com.ebay
129	19103704	293	0.000088	com.wiley
130	19102814	111	0.000299	de.google
131	19097732	191	0.000135	com.npmjs
132	19095454	344	0.000075	com.hp
133	19088550	539	0.000050	com.cisco
134	19084048	932	0.000034	com.stackexchange
135	19081736	132	0.000251	com.youtube-nocookie
136	19080638	134	0.000250	com.ft
137	19078814	213	0.000120	org.ampproject
138	19077232	532	0.000051	com.steampowered
139	19074638	365	0.000072	com.patreon
140	19072918	455	0.000061	com.theatlantic
141	19072880	476	0.000057	com.gitlab
142	19072344	890	0.000035	com.pcmag
143	19068436	195	0.000131	com.unsplash
144	19065494	877	0.000036	edu.psu
145	19063926	376	0.000070	com.time
146	19061142	208	0.000125	com.twimg
147	19061064	164	0.000165	com.yelp
148	19059332	873	0.000036	edu.washington
149	19057196	533	0.000051	edu.cornell
150	19054152	148	0.000197	com.dropbox
151	19051738	603	0.000046	org.arxiv
152	19047626	379	0.000070	com.statista
153	19043050	324	0.000080	org.un
154	19042602	249	0.000104	com.bandcamp
155	19040914	824	0.000038	com.venturebeat
156	19040684	75	0.000432	me.fb
157	19039882	841	0.000037	org.chromium
158	19033464	65	0.000519	com.wix
159	19026244	284	0.000092	com.sciencedirect
160	19019766	629	0.000045	edu.yale
161	19016326	584	0.000047	com.pexels
162	19015230	826	0.000038	org.bitbucket
163	19010452	832	0.000038	org.ieee
164	19007636	388	0.000068	com.springer
165	19001810	765	0.000041	com.evernote
166	18997506	855	0.000037	edu.upenn
167	18994926	258	0.000098	jp.ameblo
168	18993772	149	0.000195	me.t
169	18992834	416	0.000065	org.hbr
170	18992028	296	0.000088	com.outlook
171	18985954	168	0.000160	jp.co.yahoo
172	18983238	577	0.000048	com.cbsnews
173	18982546	792	0.000040	me.about
174	18981228	891	0.000035	com.git-scm
175	18980336	829	0.000038	com.economist
176	18980328	150	0.000193	com.opera
177	18978056	138	0.000223	me.line
178	18974996	450	0.000061	com.goodreads
179	18973364	645	0.000044	com.mysql
180	18973114	842	0.000037	com.docker
181	18969708	562	0.000048	com.buzzfeed
182	18969566	565	0.000048	com.mashable
183	18968398	587	0.000047	com.mozilla
184	18964540	951	0.000034	com.about
185	18962632	797	0.000040	org.worldbank
186	18956128	815	0.000039	com.newyorker
187	18954668	342	0.000076	com.dribbble
188	18954236	265	0.000096	net.behance
189	18951876	390	0.000068	com.theverge
190	18951838	501	0.000054	gov.whitehouse
191	18950142	456	0.000061	uk.co.dailymail
192	18943890	347	0.000075	com.xinhuanet
193	18942812	320	0.000080	com.w3schools
194	18941124	378	0.000070	com.fc2
195	18936488	1151	0.000027	edu.wisc
196	18935074	764	0.000041	gov.noaa
197	18932396	294	0.000088	com.disqus
198	18931228	1337	0.000023	co.elastic
199	18927646	38	0.000956	com.qq
200	18926694	448	0.000061	com.bigcommerce
201	18926442	624	0.000045	gov.loc
202	18925620	156	0.000179	gov.cdc
203	18924632	929	0.000035	gov.fcc
204	18922816	136	0.000228	info.aboutads
205	18921630	821	0.000039	com.qz
206	18921308	2295	0.000015	com.wikidot
207	18919240	385	0.000069	com.scribd
208	18915104	748	0.000042	org.unesco
209	18914418	959	0.000033	com.apnews
210	18912426	375	0.000070	com.digg
211	18911082	779	0.000040	com.vox
212	18910370	180	0.000147	com.amazon-adsystem
213	18910110	272	0.000094	com.squareup
214	18907410	495	0.000054	uk.co.independent
215	18906224	256	0.000100	org.iana
216	18905608	1251	0.000025	edu.uchicago
217	18901398	420	0.000064	com.force
218	18898702	646	0.000044	com.usnews
219	18898108	647	0.000044	com.gartner
220	18894918	295	0.000088	com.nbcnews
221	18890160	470	0.000058	com.dailymotion
222	18883488	1004	0.000031	com.dropboxusercontent
223	18878276	617	0.000045	org.pbs
224	18876454	181	0.000147	jp.co.google
225	18876164	113	0.000292	com.sharethis
226	18875824	467	0.000059	com.nationalgeographic
227	18874112	811	0.000039	uk.co.blogspot
228	18873340	844	0.000037	au.net.abc
229	18868000	934	0.000034	com.foxnews
230	18865322	1559	0.000020	org.eclipse
231	18859464	399	0.000067	com.getpocket
232	18859228	947	0.000034	com.slate
233	18859062	266	0.000095	org.doi
234	18858866	63	0.000541	com.fb
235	18856638	968	0.000033	com.politico
236	18849992	907	0.000035	com.playstation
237	18849334	600	0.000046	org.semver
238	18848468	1565	0.000020	gd.is
239	18847004	1311	0.000024	edu.unc
240	18846758	1523	0.000021	org.kernel
241	18846310	839	0.000037	org.sciencemag
242	18846038	257	0.000099	com.typepad
243	18844998	1152	0.000027	com.hatenablog
244	18844004	1981	0.000018	com.googlesource
245	18842180	202	0.000128	com.naver
246	18840548	248	0.000104	com.feedburner
247	18839830	1028	0.000030	edu.umn
248	18837518	421	0.000064	com.ecwid
249	18833048	332	0.000078	net.windows
250	18831042	914	0.000035	com.trello
251	18829176	554	0.000049	com.tandfonline
252	18829172	1369	0.000023	cn.com.chinadaily
253	18828382	189	0.000138	org.allaboutcookies
254	18825844	746	0.000042	gov.senate
255	18823946	119	0.000286	com.paypalobjects
256	18819980	1005	0.000031	ly.ow
257	18818724	2014	0.000017	org.tensorflow
258	18818710	901	0.000035	edu.umich
259	18817936	291	0.000089	com.tinyurl
260	18817212	479	0.000056	org.pewresearch
261	18815000	76	0.000423	com.list-manage
262	18811132	239	0.000111	com.wpengine
263	18806908	834	0.000038	ca.cbc
264	18805144	740	0.000043	co.ibb
265	18804044	477	0.000057	gov.fda
266	18802934	222	0.000117	com.eepurl
267	18802462	318	0.000081	it.google
268	18798744	79	0.000413	net.facebook
269	18797046	2019	0.000017	com.instructables
270	18795562	1200	0.000026	edu.northwestern
271	18794710	752	0.000042	org.change
272	18793610	394	0.000068	es.google
273	18793484	893	0.000035	org.cambridge
274	18790202	251	0.000103	com.calendly
275	18784862	962	0.000033	gov.congress
276	18784862	1022	0.000030	uk.co.guardian
277	18782014	555	0.000049	com.bigcartel
278	18777808	1348	0.000023	org.semanticscholar
279	18776340	1006	0.000031	com.gumroad
280	18775690	637	0.000044	org.plos
281	18774956	1341	0.000023	com.nikkei
282	18773712	313	0.000083	com.optimizely
283	18772988	405	0.000066	com.googlecode
284	18766674	896	0.000035	gov.justice
285	18764788	1044	0.000029	com.huffpost
286	18764312	153	0.000186	com.addtoany
287	18763408	398	0.000067	me.m
288	18761658	80	0.000403	com.wsimg
289	18760046	411	0.000065	com.tripod
290	18754884	957	0.000033	ee.linktr
291	18754526	1021	0.000030	gov.usgs
292	18753164	1459	0.000021	uk.co.wired
293	18752728	338	0.000077	fr.google
294	18751846	1059	0.000029	com.500px
295	18751636	452	0.000061	ca.google
296	18749418	1996	0.000017	com.amd
297	18744444	1944	0.000018	com.azure
298	18742964	777	0.000040	au.com.google
299	18742506	481	0.000056	com.163
300	18741292	1091	0.000028	com.ssrn
301	18740758	1065	0.000029	com.newsweek
302	18734910	1688	0.000019	ca.utoronto
303	18734620	139	0.000218	com.spotify
304	18731112	744	0.000042	cn.com.people
305	18730384	334	0.000078	page.g
306	18730074	2751	0.000012	com.nabble
307	18728400	1454	0.000021	com.howstuffworks
308	18722938	2107	0.000016	com.lego
309	18719762	1675	0.000019	com.storify
310	18719332	1140	0.000027	uk.co.thetimes
311	18717930	801	0.000039	site.business
312	18717726	884	0.000036	uk.ac.ox
313	18716206	311	0.000083	com.bitly
314	18715060	1218	0.000026	com.scmp
315	18713618	798	0.000040	com.adage
316	18713552	654	0.000044	com.indiatimes
317	18712564	1908	0.000018	de.mpg
318	18712368	1057	0.000029	com.thehill
319	18705466	519	0.000052	com.criteo
320	18704754	1078	0.000028	org.ohchr
321	18704474	1531	0.000020	com.aljazeera
322	18703348	802	0.000039	uk.gov.service
323	18701482	1545	0.000020	org.greenpeace
324	18699064	331	0.000078	com.netdna-ssl
325	18698378	967	0.000033	ch.google
326	18693994	784	0.000040	us.icio
327	18693690	1153	0.000027	int.coe
328	18692556	933	0.000034	org.d3js
329	18690456	1499	0.000021	com.history
330	18689794	1018	0.000030	com.netlify
331	18688064	1320	0.000023	com.nymag
332	18687064	1363	0.000023	org.wiktionary
333	18684868	287	0.000091	ru.ok
334	18683792	1293	0.000024	com.intuit
335	18682796	1419	0.000022	uk.co.standard
336	18681388	1995	0.000017	edu.arizona
337	18679058	944	0.000034	gov.archives
338	18678794	953	0.000034	ru.google
339	18677084	1054	0.000029	sg.com.google
340	18675890	900	0.000035	br.com.google
341	18674402	85	0.000385	co.g
342	18674068	1975	0.000018	com.wattpad
343	18673754	526	0.000051	ru.gov
344	18673370	1351	0.000023	com.ikea
345	18668598	1461	0.000021	com.reverbnation
346	18668444	2681	0.000013	edu.drexel
347	18668276	1121	0.000027	edu.si
348	18666994	1174	0.000027	uk.co.mirror
349	18666846	2572	0.000013	org.maven
350	18666724	412	0.000065	com.cnet
351	18664542	580	0.000048	org.openstreetmap
352	18663710	1373	0.000023	com.jetbrains
353	18663688	1032	0.000030	com.theconversation
354	18663542	1921	0.000018	com.newscientist
355	18661472	847	0.000037	gov.state
356	18661146	1572	0.000020	ms.1drv
357	18660152	2644	0.000013	com.mystrikingly
358	18655360	973	0.000032	org.fao
359	18654458	590	0.000047	cn.google
360	18653472	235	0.000112	com.etsy
361	18652232	1485	0.000021	com.flipboard
362	18651820	767	0.000041	com.deviantart
363	18651514	1375	0.000023	com.thedailybeast
364	18651404	1220	0.000026	org.jstor
365	18649024	1270	0.000024	com.strikingly
366	18647422	2045	0.000017	blog.home
367	18646812	634	0.000044	com.zdnet
368	18644828	325	0.000079	tv.twitch
369	18642272	2781	0.000012	com.diigo
370	18640482	1123	0.000027	com.britannica
371	18639254	1904	0.000018	ca.ubc
372	18638840	367	0.000072	com.jotform
373	18635188	1959	0.000018	com.gettyimages
374	18634254	1685	0.000019	com.channel4
375	18631278	1494	0.000021	org.pypi
376	18630386	813	0.000039	in.co.google
377	18627814	417	0.000064	com.ssl-images-amazon
378	18626978	161	0.000166	gle.forms
379	18623310	1982	0.000018	org.hrw
380	18623132	281	0.000092	com.cloudinary
381	18618612	1382	0.000022	au.com.smh
382	18617234	1566	0.000020	uk.co.metro
383	18617180	2031	0.000017	hk.com.google
384	18617072	1599	0.000020	edu.ufl
385	18613590	2332	0.000015	ly.rebrand
386	18612786	457	0.000061	net.imgix
387	18609746	418	0.000064	com.webflow
388	18609050	2311	0.000015	com.shutterfly
389	18607782	568	0.000048	com.feedly
390	18603850	538	0.000050	gov.epa
391	18602470	104	0.000348	com.stripe
392	18601118	83	0.000391	net.jsfiddle
393	18599796	3423	0.000010	org.aclweb
394	18597166	2348	0.000014	com.yarnpkg
395	18596278	69	0.000461	net.akamaihd
396	18596202	1907	0.000018	gov.supremecourt
397	18595244	2344	0.000014	com.thefreedictionary
398	18593816	468	0.000058	nl.google
399	18592072	1578	0.000020	com.dw
400	18588294	2955	0.000012	com.upi
401	18587932	981	0.000032	com.thelancet
402	18587926	425	0.000064	com.slack
403	18587680	396	0.000067	com.kickstarter
404	18587378	787	0.000040	com.urldefense
405	18585950	1713	0.000019	ca.sfu
406	18583582	460	0.000060	com.livechatinc
407	18581082	623	0.000045	com.quora
408	18580964	428	0.000063	com.rackcdn
409	18580620	1967	0.000018	com.euronews
410	18580552	451	0.000061	com.go
411	18580130	1368	0.000023	com.tunein
412	18578076	594	0.000046	ru.liveinternet
413	18576712	475	0.000057	com.googleblog
414	18571776	2597	0.000013	pt.sapo
415	18571212	2109	0.000016	com.itv
416	18570630	1945	0.000018	uk.co.huffingtonpost
417	18570542	1286	0.000024	edu.brookings
418	18570528	4423	0.000008	tl.page
419	18570058	2369	0.000014	com.angelfire
420	18568882	2614	0.000013	org.wikibooks
421	18567302	1692	0.000019	com.ifttt
422	18564134	861	0.000036	com.freepik
423	18563246	2244	0.000015	com.netvibes
424	18562602	133	0.000251	com.mailchimp
425	18562564	364	0.000072	me.telegram
426	18562400	561	0.000048	com.microsoftonline
427	18562224	1976	0.000018	uk.co.express
428	18559206	2888	0.000012	sg.edu.nus
429	18559092	1928	0.000018	io.webflow
430	18557292	772	0.000041	pl.google
431	18555900	480	0.000056	com.meetup
432	18555482	4752	0.000007	com.newgrounds
433	18554944	2397	0.000014	google.ai
434	18554512	2439	0.000014	com.yolasite
435	18553912	2124	0.000016	jp.geocities
436	18552986	3394	0.000011	com.instapaper
437	18551338	362	0.000072	com.proofpoint
438	18548844	1358	0.000023	com.people
439	18546296	64	0.000531	net.typekit
440	18543694	2104	0.000016	org.c-span
441	18541918	159	0.000169	ru.mail
442	18541834	2043	0.000017	com.avg
443	18540650	2249	0.000015	app.netlify
444	18539394	3004	0.000011	com.000webhostapp
445	18539316	485	0.000055	com.elsevier
446	18538008	3494	0.000010	cn.edu.pku
447	18536872	1609	0.000020	com.asahi
448	18535422	876	0.000036	org.worldwildlife
449	18535204	1127	0.000027	uk.parliament
450	18534822	1956	0.000018	uk.gov.ons
451	18533694	188	0.000138	com.iubenda
452	18532790	2113	0.000016	org.documentcloud
453	18532338	3074	0.000011	uk.co.timesonline
454	18531118	264	0.000096	com.office
455	18527764	237	0.000112	com.eventbrite
456	18527012	2699	0.000013	com.self
457	18526172	2511	0.000013	com.foreignpolicy
458	18524804	2421	0.000014	org.sundance
459	18524702	214	0.000120	com.aliyuncs
460	18524140	1213	0.000026	be.google
461	18523242	2200	0.000016	ie.google
462	18523000	1432	0.000022	gov.weather
463	18522694	3136	0.000011	com.openai
464	18522588	879	0.000036	org.mediawiki
465	18521124	2806	0.000012	com.pearltrees
466	18520306	1704	0.000019	com.firebaseapp
467	18516520	3620	0.000010	com.dailycaller
468	18514512	498	0.000054	it.placehold
469	18514168	2695	0.000013	com.france24
470	18513026	644	0.000044	edu.berkeley
471	18512138	492	0.000055	cn.360
472	18511428	2296	0.000015	com.msnbc
473	18510986	2089	0.000017	com.thestar
474	18510258	3732	0.000009	me.site123
475	18509392	2133	0.000016	com.gfycat
476	18508906	341	0.000076	com.rawgit
477	18507920	521	0.000052	com.gmail
478	18507686	1952	0.000018	org.ocks
479	18506872	2739	0.000012	org.rsc
480	18504310	2486	0.000014	edu.hawaii
481	18503766	2366	0.000014	de.br
482	18503250	2447	0.000014	edu.colostate
483	18502578	171	0.000154	com.zendesk
484	18501424	2222	0.000015	org.nobelprize
485	18501096	3293	0.000011	net.pixnet
486	18500188	1528	0.000020	net.seesaa
487	18500164	2471	0.000014	com.motherjones
488	18499720	756	0.000042	com.vice
489	18499378	4234	0.000008	com.masslive
490	18496634	2355	0.000014	com.cision
491	18495058	101	0.000361	com.godaddy
492	18492104	886	0.000036	gov.nist
493	18491956	1249	0.000025	org.ilo
494	18490654	2070	0.000017	com.surveygizmo
495	18490628	3378	0.000011	com.minds
496	18490576	635	0.000044	com.matterport
497	18489858	2656	0.000013	ph.com.google
498	18488106	369	0.000071	org.python
499	18487032	980	0.000032	gov.va
500	18485800	1166	0.000027	at.google
501	18485152	1318	0.000023	se.google
502	18483644	1961	0.000018	ru.ucoz
503	18482996	2401	0.000014	com.freep
504	18482190	3874	0.000009	com.wizards
505	18481738	3583	0.000010	edu.uvm
506	18478142	3711	0.000010	org.tvtropes
507	18476988	1506	0.000021	com.cognitoforms
508	18476516	1493	0.000021	gov.uscourts
509	18476024	3530	0.000010	org.oxfam
510	18473992	2235	0.000015	cn.t
511	18473054	4331	0.000008	fm.ask
512	18473034	1708	0.000019	dk.google
513	18470526	3122	0.000011	de.dw
514	18467204	2009	0.000017	ua.com.google
515	18467126	3935	0.000009	com.youdao
516	18464016	128	0.000262	org.networkadvertising
517	18462968	1031	0.000030	com.arstechnica
518	18462674	2310	0.000015	int.unfccc
519	18461844	3323	0.000011	ch.nzz
520	18460156	123	0.000276	com.statcounter
521	18460126	3757	0.000009	net.hinet
522	18460018	2484	0.000014	com.washingtontimes
523	18459778	3391	0.000011	edu.miami
524	18459648	5025	0.000007	tw.com.gamer
525	18459120	4313	0.000008	ch.qos
526	18458774	788	0.000040	com.intel
527	18456584	2220	0.000015	mx.com.google
528	18455734	2241	0.000015	gov.ky
529	18455504	3426	0.000010	com.nwsource
530	18454948	856	0.000037	io.readthedocs
531	18453730	2187	0.000016	gov.cisa
532	18451988	2256	0.000015	com.straitstimes
533	18449466	371	0.000071	io.codepen
534	18447006	361	0.000072	com.prnewswire
535	18446224	4097	0.000009	com.smore
536	18446132	2188	0.000016	pt.google
537	18445920	2719	0.000012	net.bplaced
538	18445802	5349	0.000007	net.wargaming
539	18445232	3272	0.000011	org.csis
540	18444732	1435	0.000022	org.aarp
541	18444080	289	0.000090	net.php
542	18443758	2282	0.000015	no.google
543	18443228	3924	0.000009	com.steemit
544	18443146	1304	0.000024	tw.com.google
545	18442018	314	0.000083	com.squarespace
546	18440872	743	0.000043	com.oreilly
547	18440596	199	0.000130	com.hubspot
548	18439354	4877	0.000007	com.bonanza
549	18438802	2020	0.000017	co.lpages
550	18438606	1079	0.000028	net.ovh
551	18438208	835	0.000037	com.imageshack
552	18437874	4023	0.000009	com.doodlekit
553	18436818	2425	0.000014	com.voanews
554	18436680	358	0.000073	ru.rambler
555	18436048	2805	0.000012	com.nationalpost
556	18435420	4534	0.000008	by.google
557	18435256	614	0.000045	org.nodejs
558	18435200	397	0.000067	com.onesignal
559	18434470	3374	0.000011	fr.rfi
560	18434466	463	0.000060	gov.irs
561	18434444	2584	0.000013	com.snopes
562	18434230	1899	0.000018	link.page
563	18434190	3637	0.000010	org.vim
564	18434018	2240	0.000015	th.co.google
565	18433782	3395	0.000010	org.scala-lang
566	18432434	3142	0.000011	com.inquirer
567	18430898	2887	0.000012	org.ballotpedia
568	18430888	3324	0.000011	com.real
569	18428600	649	0.000044	br.com.uol
570	18428004	513	0.000052	com.pixabay
571	18426658	2142	0.000016	uk.co.which
572	18426634	4070	0.000009	com.viki
573	18425674	1038	0.000030	com.thenextweb
574	18424302	3146	0.000011	org.aps
575	18424050	2764	0.000012	com.post-gazette
576	18423516	2499	0.000014	net.openid
577	18422702	2627	0.000013	edu.usf
578	18421138	82	0.000391	com.livestream
579	18420414	961	0.000033	jp.shinobi
580	18420272	956	0.000033	int.wipo
581	18417146	4450	0.000008	com.bravesites
582	18415542	2881	0.000012	ru.aif
583	18414574	2906	0.000012	io.gitlab
584	18414284	3387	0.000011	org.pri
585	18414276	1932	0.000018	gov.ct
586	18413984	2602	0.000013	il.co.google
587	18413906	1910	0.000018	org.oxfordjournals
588	18413218	4664	0.000008	com.ucoz
589	18412422	566	0.000048	com.photobucket
590	18412344	2191	0.000016	com.xrea
591	18412198	2234	0.000015	nz.co.google
592	18410920	2088	0.000017	net.cnki
593	18410828	2847	0.000012	com.webbyawards
594	18410164	433	0.000063	com.staticflickr
595	18409934	3675	0.000010	org.heritage
596	18408908	1993	0.000018	tr.com.google
597	18408574	2053	0.000017	com.treehugger
598	18406062	1695	0.000019	net.leadpages
599	18405282	2112	0.000016	fi.google
600	18402764	5153	0.000007	kz.google
601	18402708	211	0.000121	to.amzn
602	18402670	569	0.000048	com.deloitte
603	18402662	1100	0.000028	cz.google
604	18402526	4562	0.000008	com.freehostia
605	18402334	2156	0.000016	gov.faa
606	18402326	2724	0.000012	com.detroitnews
607	18402220	2774	0.000012	com.slidesharecdn
608	18402102	346	0.000075	com.adnxs
609	18396726	812	0.000039	com.thinkwithgoogle
610	18392816	1471	0.000021	com.trustwave
611	18392376	2640	0.000013	org.iea
612	18392262	2883	0.000012	jp.blog
613	18391148	4426	0.000008	com.goal
614	18390184	3284	0.000011	com.financialpost
615	18389140	3636	0.000010	net.alarabiya
616	18389082	3570	0.000010	org.neocities
617	18388580	3784	0.000009	co.ello
618	18388256	207	0.000126	com.salesforce
619	18386478	3500	0.000010	com.archdaily
620	18385984	4517	0.000008	com.alamy
621	18385924	2297	0.000015	gr.google
622	18385398	160	0.000168	gov.privacyshield
623	18385020	2569	0.000013	org.kqed
624	18383196	277	0.000093	org.drupal
625	18382110	354	0.000074	com.snapchat
626	18381496	2338	0.000015	ro.google
627	18381392	3367	0.000011	uk.ac.leeds
628	18381316	271	0.000094	com.mapbox
629	18380144	3907	0.000009	uk.gov.scotland
630	18379620	1946	0.000018	hu.google
631	18378224	4399	0.000008	co.aeon
632	18377446	374	0.000070	com.cdninstagram
633	18376062	3545	0.000010	gov.fec
634	18376022	3312	0.000011	com.virgin
635	18375628	2219	0.000015	ar.com.google
636	18375060	4128	0.000009	cn.globaltimes
637	18374688	4333	0.000008	com.corel
638	18374066	464	0.000059	com.herokuapp
639	18373200	4062	0.000009	jp.go.ndl
640	18373110	791	0.000040	google.blog
641	18372316	2208	0.000016	com.justia
642	18372216	2320	0.000015	za.co.google
643	18370616	2216	0.000016	ru.ria
644	18370232	3694	0.000010	com.intensedebate
645	18369794	3793	0.000009	com.visualcapitalist
646	18369094	2722	0.000012	si.google
647	18368512	4182	0.000008	com.rediff
648	18367604	3834	0.000009	ca.uvic
649	18367236	2577	0.000013	ru.rosminzdrav
650	18365918	439	0.000062	com.nypost
651	18365880	4678	0.000008	org.wikimapia
652	18365350	3439	0.000010	com.nationalreview
653	18364962	2134	0.000016	uk.org.asa
654	18364282	3850	0.000009	tw.edu.ntu
655	18363974	598	0.000046	com.samsung
656	18363190	2703	0.000012	is.google
657	18362598	3869	0.000009	com.podomatic
658	18361242	316	0.000082	cn.bshare
659	18360424	3484	0.000010	org.wri
660	18360028	4160	0.000009	uk.co.spectator
661	18359858	1711	0.000019	ly.cutt
662	18358316	4989	0.000007	to.gplus
663	18358086	4908	0.000007	com.atwebpages
664	18357826	177	0.000150	com.tripadvisor
665	18357438	5003	0.000007	org.scala-sbt
666	18356488	4276	0.000008	ru.msu
667	18356450	1161	0.000027	com.udemy
668	18355358	2973	0.000011	com.timesofisrael
669	18352506	5213	0.000007	edu.csulb
670	18351622	4744	0.000007	com.authorstream
671	18350944	4127	0.000009	gy.rb
672	18350110	3204	0.000011	us.ny.state
673	18349876	3644	0.000010	com.linuxquota
674	18349798	3563	0.000010	com.udn
675	18349578	3845	0.000009	org.jenkins-ci
676	18349508	1686	0.000019	com.pcworld
677	18349104	2481	0.000014	uk.ac.imperial
678	18348784	5238	0.000007	com.etymonline
679	18348026	3492	0.000010	eg.com.google
680	18347774	3363	0.000011	uk.co.bbci
681	18347338	2386	0.000014	com.name
682	18346938	3745	0.000009	com.novell
683	18345924	1487	0.000021	com.digitaloceanspaces
684	18345376	6040	0.000006	net.vingle
685	18345350	2615	0.000013	us.pa.state
686	18345040	642	0.000044	com.xiti
687	18345006	2302	0.000015	fr.pagesjaunes
688	18344246	4604	0.000008	by.tut
689	18341982	78	0.000417	com.messenger
690	18341502	1672	0.000019	id.co.google
691	18341492	4012	0.000009	com.donaldjtrump
692	18339724	2359	0.000014	co.pcdn
693	18338674	606	0.000046	com.indeed
694	18338446	459	0.000060	com.sxsw
695	18337870	2379	0.000014	sk.google
696	18337126	246	0.000105	uk.co.amazon
697	18336826	351	0.000074	com.atlassian
698	18336810	1225	0.000025	com.dell
699	18336442	4947	0.000007	fr.online
700	18336226	1933	0.000018	com.law
701	18335648	3783	0.000009	com.wmtransfer
702	18335422	2242	0.000015	kr.co.google
703	18335402	4709	0.000008	edu.odu
704	18335130	2971	0.000011	cl.google
705	18335024	4300	0.000008	il.ac.huji
706	18334782	4271	0.000008	tw.gov.cdc
707	18333794	2886	0.000012	my.com.google
708	18333014	3385	0.000011	com.scotsman
709	18332864	3322	0.000011	com.instructure
710	18332832	4563	0.000008	com.hackaday
711	18332194	2131	0.000016	gov.pa
712	18332054	627	0.000045	com.withgoogle
713	18331108	1997	0.000017	scot.gov
714	18330912	3178	0.000011	com.broadwayworld
715	18330804	858	0.000036	com.canva
716	18330694	4525	0.000008	com.mongabay
717	18329802	4508	0.000008	com.macobserver
718	18329686	3725	0.000010	org.sonatype
719	18328118	2391	0.000014	gov.wi
720	18327736	2683	0.000013	org.usgbc
721	18327662	4113	0.000009	gov.peacecorps
722	18327624	4652	0.000008	cn.tianya
723	18326710	3495	0.000010	pk.com.google
724	18326302	870	0.000036	com.marketwatch
725	18326164	1490	0.000021	com.billboard
726	18324976	107	0.000316	net.gandi
727	18324878	2845	0.000012	com.thecut
728	18324686	89	0.000372	me.ogp
729	18323980	4585	0.000008	io.meduza
730	18323898	2827	0.000012	uk.org.nationaltrust
731	18323758	3911	0.000009	au.edu.adelaide
732	18323398	4766	0.000007	de.uni-erlangen
733	18322482	3759	0.000009	uk.org.rspb
734	18322376	3773	0.000009	cv.google
735	18321256	5135	0.000007	cat.bcn
736	18319736	3728	0.000009	com.ipage
737	18319726	5311	0.000007	com.brother
738	18318148	2410	0.000014	my.com.thestar
739	18317872	3401	0.000010	uk.ac.york
740	18317504	3315	0.000011	com.politifact
741	18317408	3128	0.000011	ee.google
742	18317178	3326	0.000011	org.thinkprogress
743	18317034	2102	0.000016	se.haxx
744	18316764	4554	0.000008	au.edu.rmit
745	18316272	2959	0.000011	hr.google
746	18315296	5212	0.000007	com.selfridges
747	18315244	3772	0.000009	au.com.telstra
748	18313746	1436	0.000022	com.fiverr
749	18313044	3420	0.000010	de.hu-berlin
750	18311516	3572	0.000010	com.nola
751	18311094	3458	0.000010	sa.com.google
752	18310436	4145	0.000009	ca.dal
753	18310126	6237	0.000006	org.arkive
754	18309422	2759	0.000012	bg.google
755	18308696	3429	0.000010	com.monday
756	18308664	4635	0.000008	at.tugraz
757	18308432	3508	0.000010	com.eiseverywhere
758	18308298	3764	0.000009	uk.co.cfdr
759	18308102	3298	0.000011	org.iucn
760	18307444	3571	0.000010	app.web
761	18306932	3702	0.000010	org.iucnredlist
762	18306908	292	0.000088	com.surveymonkey
763	18306390	3806	0.000009	gi.com.google
764	18306038	5056	0.000007	ec.com.google
765	18305962	3875	0.000009	de.uni-freiburg
766	18305528	4244	0.000008	au.com.heraldsun
767	18305222	515	0.000052	io.shields
768	18304914	610	0.000046	org.eff
769	18304878	3829	0.000009	com.psmag
770	18304506	4721	0.000007	ua.at
771	18302798	930	0.000034	gov.uspto
772	18302648	190	0.000137	com.automattic
773	18301286	3948	0.000009	com.mozello
774	18300612	1108	0.000028	com.gizmodo
775	18300418	3596	0.000010	pl.wp
776	18300322	3471	0.000010	org.royalsociety
777	18299622	2819	0.000012	org.unep
778	18299452	3606	0.000010	com.realclearpolitics
779	18298298	3531	0.000010	jp.coocan
780	18298296	2613	0.000013	vn.com.google
781	18298218	4434	0.000008	jp.hatenablog
782	18297896	4281	0.000008	com.waitrose
783	18297876	4676	0.000008	info.webry
784	18297852	4427	0.000008	net.inquirer
785	18297704	4274	0.000008	jp.gree
786	18297178	4611	0.000008	org.nationalinterest
787	18296330	2981	0.000011	edu.uconn
788	18295610	946	0.000034	edu.columbia
789	18295554	5531	0.000006	org.mises
790	18295452	1274	0.000024	com.smashingmagazine
791	18295224	3303	0.000011	uk.gov.companieshouse
792	18294866	4442	0.000008	gov.ourdocuments
793	18294666	3894	0.000009	sl.com.google
794	18292912	6218	0.000006	com.rhino3d
795	18292842	3435	0.000010	org.cfr
796	18292780	790	0.000040	com.airbnb
797	18292712	283	0.000092	jp.co.amazon
798	18291570	413	0.000065	com.pubmatic
799	18290920	878	0.000036	com.box
800	18290426	5610	0.000006	com.coroflot
801	18290346	4348	0.000008	com.thediplomat
802	18286902	4066	0.000009	com.inhabitat
803	18286668	3277	0.000011	com.bp
804	18286522	4592	0.000008	cat.uab
805	18283480	3827	0.000009	uk.co.villiers-london
806	18283014	4140	0.000009	org.grist
807	18282452	4016	0.000009	com.foreignaffairs
808	18281324	1081	0.000028	com.tapad
809	18280378	1347	0.000023	org.altervista
810	18280358	382	0.000069	com.skype
811	18280324	4349	0.000008	com.worldsecuresystems
812	18279680	2409	0.000014	com.volusion
813	18279516	2907	0.000012	ru.nethouse
814	18279480	3527	0.000010	pe.com.google
815	18279438	4779	0.000007	be.lesoir
816	18278874	3288	0.000011	co.com.google
817	18278816	3885	0.000009	de.uni-koeln
818	18278778	2910	0.000012	org.gnupg
819	18278022	4656	0.000008	com.mihanblog
820	18277554	3360	0.000011	org.panda
821	18277186	3440	0.000010	lv.google
822	18276674	5300	0.000007	lu.google
823	18276442	484	0.000055	com.inc
824	18275676	5103	0.000007	cn.com.caijing
825	18275134	3331	0.000011	uk.gov.metoffice
826	18274258	68	0.000471	com.oculus
827	18273732	2364	0.000014	org.donorbox
828	18273312	3038	0.000011	rs.google
829	18273256	1197	0.000026	com.merriam-webster
830	18271448	5051	0.000007	ee.ut
831	18271060	2519	0.000013	com.amebaownd
832	18270922	4482	0.000008	com.marksandspencer
833	18270780	6447	0.000006	su.clan
834	18269948	4096	0.000009	ru.interfax
835	18269620	3852	0.000009	org.rferl
836	18268756	2904	0.000012	gov.nd
837	18267994	548	0.000049	com.fortune
838	18267776	4693	0.000008	it.unitn
839	18267714	5665	0.000006	am.google
840	18266762	3502	0.000010	org.iaea
841	18263748	3893	0.000009	pr.com.google
842	18262158	5045	0.000007	com.tok2
843	18261938	1901	0.000018	ch.ethz
844	18261922	3342	0.000011	gov.la
845	18261182	4507	0.000008	org.democracynow
846	18261176	2593	0.000013	net.noscript
847	18260216	836	0.000037	com.mix
848	18259862	408	0.000066	net.adform
849	18259608	5208	0.000007	tn.google
850	18257978	4212	0.000008	jp.hateblo
851	18257888	6029	0.000006	hk.edu.hkbu
852	18257680	3884	0.000009	nl.wur
853	18257594	5009	0.000007	gr.auth
854	18257406	997	0.000031	com.webs
855	18256760	4512	0.000008	com.mnn
856	18256702	5759	0.000006	ru.nnov
857	18256238	3954	0.000009	com.afp
858	18255744	1365	0.000023	com.format
859	18255662	5209	0.000007	nf.co
860	18253954	329	0.000079	com.getbootstrap
861	18252988	4961	0.000007	jp.hatenadiary
862	18252154	4728	0.000007	hk.com.hkex
863	18251258	1193	0.000026	com.redhat
864	18250974	5600	0.000006	com.gust
865	18250088	1067	0.000029	com.symantec
866	18249466	2562	0.000013	net.ucoz
867	18249320	268	0.000095	com.typeform
868	18248694	6327	0.000006	com.x10host
869	18248332	3547	0.000010	uk.co.saveourschools
870	18247898	2934	0.000012	com.squarespace-cdn
871	18247292	2977	0.000011	lt.google
872	18246872	525	0.000051	com.adweek
873	18246844	4295	0.000008	com.scienceblogs
874	18246472	4848	0.000007	de.uni-konstanz
875	18245562	6362	0.000006	com.ueuo
876	18245048	3856	0.000009	uk.gov.data
877	18244756	4005	0.000009	tr.com.hurriyet
878	18243652	3070	0.000011	ae.google
879	18243570	1891	0.000019	com.speakerdeck
880	18243330	5079	0.000007	com.blogsky
881	18243134	2044	0.000017	tv.ustream
882	18240374	6711	0.000006	su.moy
883	18239298	761	0.000041	gov.copyright
884	18239096	5292	0.000007	ru.novayagazeta
885	18239044	2789	0.000012	gov.nh
886	18238990	4057	0.000009	org.hathitrust
887	18238948	3648	0.000010	org.annualreviews
888	18238932	1154	0.000027	pl.home
889	18238882	3815	0.000009	com.businesscatalyst
890	18237740	472	0.000058	com.ea
891	18237726	3087	0.000011	uk.gov.hmrc
892	18236940	3930	0.000009	cc.uxdesign
893	18236894	6015	0.000006	com.artfire
894	18236704	366	0.000072	org.opensource
895	18236530	3467	0.000010	it.beniculturali
896	18236132	2507	0.000014	gov.mn
897	18236076	1019	0.000030	com.engadget
898	18235902	3682	0.000010	ve.co.google
899	18235452	4973	0.000007	com.teslamotors
900	18234038	7475	0.000005	com.hangame
901	18233966	427	0.000063	com.fastcompany
902	18233600	4263	0.000008	com.hsbc
903	18233074	2462	0.000014	com.netsolhost
904	18232582	5556	0.000006	me.google
905	18232344	5643	0.000006	mu.google
906	18231570	5529	0.000006	com.yam
907	18231242	3969	0.000009	tz.co.google
908	18230998	974	0.000032	com.verisign
909	18230916	3364	0.000011	tw.com.pchome
910	18230662	7293	0.000005	com.addr
911	18230628	2636	0.000013	com.shell
912	18230602	6599	0.000006	com.dropmark
913	18229708	5635	0.000006	li.google
914	18229116	5002	0.000007	com.gab
915	18229106	4493	0.000008	com.tapatalk
916	18228194	1325	0.000023	edu.ucla
917	18227958	3557	0.000010	uk.co.newmedianow
918	18227938	4988	0.000007	edu.whoi
919	18227810	3738	0.000009	ng.com.google
920	18227630	5444	0.000007	ni.com.google
921	18226076	4110	0.000009	uk.co.sainsburys
922	18225458	4412	0.000008	com.iconarchive
923	18225080	5380	0.000007	gr.ntua
924	18224924	6152	0.000006	com.epochtimes
925	18224716	5198	0.000007	org.birdlife
926	18224610	3532	0.000010	uk.co.intersol
927	18224178	5615	0.000006	id.co.kaskus
928	18223762	950	0.000034	com.zoho
929	18223166	5403	0.000007	cr.co.google
930	18223046	5695	0.000006	sv.com.google
931	18222882	4074	0.000009	vn.zing
932	18222714	4537	0.000008	uk.co.zoopla
933	18222480	4039	0.000009	uk.ac.jisc
934	18221034	3836	0.000009	com.prweek
935	18220422	3098	0.000011	int.wmo
936	18220410	5466	0.000006	mz.co.google
937	18220202	4966	0.000007	edu.umb
938	18220196	1290	0.000024	uk.co.freeukbusinessdirectory
939	18220068	1476	0.000021	org.owasp
940	18219726	6669	0.000006	net.comunidades
941	18218976	4141	0.000009	com.scotusblog
942	18218840	5636	0.000006	com.cyberlink
943	18218738	3828	0.000009	do.com.google
944	18218672	2966	0.000011	io.termly
945	18218262	4735	0.000007	com.fatcow
946	18218172	3851	0.000009	mt.com.google
947	18218110	3589	0.000010	uk.org.oxonaa
948	18217958	3774	0.000009	gt.com.google
949	18216908	3737	0.000009	com.solidworks
950	18216782	3641	0.000010	uk.co.profilebusiness
951	18216270	3625	0.000010	uk.co.heatall
952	18216034	4506	0.000008	com.theringer
953	18215388	2558	0.000013	nl.jouwweb
954	18215320	800	0.000039	com.wikihow
955	18215060	5953	0.000006	com.symbaloo
956	18214768	5171	0.000007	pl.cba
957	18214162	5740	0.000006	kg.google
958	18213594	2321	0.000015	com.freeprivacypolicy
959	18212850	1222	0.000026	com.att
960	18212680	5203	0.000007	pl.lublin
961	18212672	1541	0.000020	edu.umd
962	18212174	5485	0.000006	uk.org.labour
963	18212074	4288	0.000008	us.ms.state
964	18211828	3449	0.000010	com.wantedly
965	18211570	4396	0.000008	org.ametsoc
966	18211542	3701	0.000010	uy.com.google
967	18211486	5553	0.000006	jp.ifdef
968	18211438	5218	0.000007	es.usal
969	18211398	769	0.000041	com.netflix
970	18211196	6329	0.000006	org.cgsociety
971	18210854	3897	0.000009	hn.google
972	18210544	5602	0.000006	org.svoboda
973	18207828	4432	0.000008	org.ascd
974	18207784	4500	0.000008	uk.co.dailystar
975	18207712	3651	0.000010	uk.co.articlelistings
976	18207370	503	0.000054	com.dmca
977	18207114	916	0.000035	com.ggpht
978	18207032	5199	0.000007	com.curseforge
979	18206432	5265	0.000007	org.nsidc
980	18206340	1520	0.000021	com.technologyreview
981	18205908	5668	0.000006	ug.co.google
982	18205822	4030	0.000009	org.lacity
983	18205348	4843	0.000007	com.cbn
984	18204716	434	0.000063	com.businesswire
985	18204712	5860	0.000006	mn.google
986	18204394	6868	0.000005	kr.ac.postech
987	18204332	5613	0.000006	it.unige
988	18203526	3314	0.000011	uk.gov.food
989	18203314	6353	0.000006	com.skepticalscience
990	18203052	909	0.000035	org.weforum
991	18202434	4907	0.000007	com.globalpost
992	18202416	5172	0.000007	com.weightwatchers
993	18202000	3403	0.000010	com.lexology
994	18200738	5944	0.000006	tt.google
995	18200210	5282	0.000007	com.betfair
996	18199968	5428	0.000007	py.com.google
997	18198928	4815	0.000007	com.abcnews
998	18198698	763	0.000041	com.psychologytoday
999	18198512	6974	0.000005	org.toile-libre
1000	18198414	3291	0.000011	net.vnexpress

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

January 2021 crawl archive now available

February 2, 2021Sebastian Nagel

The crawl archive for January 2021 is now available! The data was crawled between January 15th and 28th and contains 3.4 billion web pages or 350 TiB of uncompressed content. It includes page captures of 1.15 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The January crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-04/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-04/segment.paths.gz	100
WARC files	CC-MAIN-2021-04/warc.paths.gz	79840	78.98
WAT files	CC-MAIN-2021-04/wat.paths.gz	79840	22.92
WET files	CC-MAIN-2021-04/wet.paths.gz	79840	10.04
Robots.txt files	CC-MAIN-2021-04/robotstxt.paths.gz	79840	0.23
Non-200 responses files	CC-MAIN-2021-04/non200responses.paths.gz	79840	2.11
URL index files	CC-MAIN-2021-04/cc-index.paths.gz	302	0.26

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-04/. Also the columnar index has been updated to contain this crawl.

November/December 2020 crawl archive now available

December 10, 2020Sebastian Nagel

The crawl archive for November/December 2020 is now available! The data was crawled between November 23 and December 6 and contains 2.64 billion web pages or 270 TiB of uncompressed content. It includes page captures of 1.4 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The November/December crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-50/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-50/segment.paths.gz	100
WARC files	CC-MAIN-2020-50/warc.paths.gz	72000	59.95
WAT files	CC-MAIN-2020-50/wat.paths.gz	72000	17.82
WET files	CC-MAIN-2020-50/wet.paths.gz	72000	7.89
Robots.txt files	CC-MAIN-2020-50/robotstxt.paths.gz	72000	0.2
Non-200 responses files	CC-MAIN-2020-50/non200responses.paths.gz	72000	1.71
URL index files	CC-MAIN-2020-50/cc-index.paths.gz	302	0.2

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-50/. Also the columnar index has been updated to contain this crawl.

October 2020 crawl archive now available

November 7, 2020Sebastian Nagel

The crawl archive for October 2020 is now available! The data was crawled between October 19th and November 1st and contains 2.71 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.5 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The October crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2020-45/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2020-45/segment.paths.gz	100
WARC files	CC-MAIN-2020-45/warc.paths.gz	72000	63.79
WAT files	CC-MAIN-2020-45/wat.paths.gz	72000	18.39
WET files	CC-MAIN-2020-45/wet.paths.gz	72000	8.23
Robots.txt files	CC-MAIN-2020-45/robotstxt.paths.gz	72000	0.2
Non-200 responses files	CC-MAIN-2020-45/non200responses.paths.gz	72000	1.75
URL index files	CC-MAIN-2020-45/cc-index.paths.gz	302	0.21

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2020-45/. Also the columnar index has been updated to contain this crawl.