Blog – Page 2 – Common Crawl

Important news for users of Common Crawl data: we are introducing CloudFront as a new way to access Common Crawl data as part of Amazon Web Services’ registry of open data

March 1, 2022Sebastian Nagel

Ten years ago(!) Common Crawl joined AWS’s Open Data Sponsorships program, hosted on S3, with free access to everyone. Since then, the dataset has expanded (by petabytes!) and our community of users has seen extraordinary growth. With growth comes change, therefore…

Following are new measures to accommodate the number and volume of requests for Common Crawl data, and to ensure efficient and stable access to the data:

1. Common Crawl data is now available on CloudFront under the URL base prefixes*:

https://data.commoncrawl.org/ or
https://ds5q9oxwqwsfj.cloudfront.net/

* Data is accessible via http:// or https://.

As of Monday, April 4, 2022

2. Access to data from the Amazon cloud using the S3 API will be restricted to authenticated AWS users, and unsigned access to s3://commoncrawl/ will be disabled. See Q&A for further details.

3. To access data from outside the Amazon cloud, via HTTP(S), a new URL prefix – https://data.commoncrawl.org/ – must be used instead of https://commoncrawl.s3.amazonaws.com/.

Q & A

Q: How can I identify whether my code is using unauthenticated S3 access?

A: Authenticated access is the default, but you should verify that your code is not configured otherwise; code fragments requesting unauthenticated access could be (but are not limited to):

AWS CLI with the command-line option --no-sign-request:
```
aws --no-sign-request s3 cp s3://commoncrawl/...
```

Python using boto3 and botocore.UNSIGNED:

import boto3
import botocore

s3client = boto3.client('s3', config=botocore.client.Config(signature_version=botocore.UNSIGNED))

Hadoop or Spark (various programming languages): usage of AnonymousAWSCredentialsProvider

Q: Where can I find up-to-date documentation and examples that are consistent with these new protocols?

A: We will update our documentation and examples to read data either via HTTP(S) using the new CloudFront access, or via S3 and authenticated access. We will also update all data download links. It will take a few days until everything is updated. Thanks for your patience!

Q: Are range requests supported?

A: Yes, range requests are supported by CloudFront same as for S3, see the CloudFront docs about RangeGETs.

Q: What is the recommended access method on AWS but in a different region (not us-east-1)?

A: We recommend that you run your computing workload in the same region (us-east-1) as the Common Crawl dataset whenever possible. If you have a specific, ongoing need to run computing workloads using Common Crawl in another AWS region, the AWS Open Data team would like to hear more about your use case ([email protected]).

Q: What are typical error messages indicating that unauthenticated access is used?

A: The HTTP response status code is a 403 Forbidden. However, also a restrictive IAM policy on the user’s side could deny access to s3://commoncrawl/ using the S3 API.
Two examples for error messages related to unauthenticated access to s3://commoncrawl/:

> curl https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2022-05/warc.paths.gz
<!--?xml version="1.0" encoding="UTF-8"?-->
<error><code>AccessDenied</code><message>Access Denied</message>...</error>

> aws --no-sign-request s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2022-05/warc.paths.gz .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

January 2022 crawl archive now available

February 2, 2022Sebastian Nagel

The crawl archive for January 2022 is now available! The data was crawled January 16 – 29 and contains 2.95 billion web pages or 320 TiB of uncompressed content. It includes page captures of 1.35 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The January crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2022-05/.

To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2022-05/segment.paths.gz	100
WARC files	CC-MAIN-2022-05/warc.paths.gz	72000	73.5
WAT files	CC-MAIN-2022-05/wat.paths.gz	72000	19.85
WET files	CC-MAIN-2022-05/wet.paths.gz	72000	8.63
Robots.txt files	CC-MAIN-2022-05/robotstxt.paths.gz	72000	0.14
Non-200 responses files	CC-MAIN-2022-05/non200responses.paths.gz	72000	1.79
URL index files	CC-MAIN-2022-05/cc-index.paths.gz	302	0.22

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2022-05/. Also the columnar index has been updated to contain this crawl.

Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.

November/December 2021 crawl archive now available

December 14, 2021Sebastian Nagel

The crawl archive for November/December 2021 is now available! The data was crawled Nov 26 – Dec 9 and contains 2.5 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.2 billion new URLs, not visited in any of our prior crawls.

Improvements

The column url_host_name_reversed was added to the columnar index. It holds the host name in reverse domain name notation (com.example.www) which is more efficient to query. In order to make use of the new column please use the updated table schema. On Amazon Athena you need to recreate table by running the latest table creation statement. Further details are found in the corresponding pull request.

The columnar index is now built using Spark version 3.2.0 and Parquet MR 1.12.1 – these upgrades allow us to go for further improvements next year. Let us know if you observe any incompatibilities using this index to query data from the Nov/Dec 2021 crawl.

The crawler is now based on Tika 2.1.0 for detection of MIME/content types and character sets used to encode HTML documents.

Archive Location and Download

The November/December crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-49/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-49/segment.paths.gz	100
WARC files	CC-MAIN-2021-49/warc.paths.gz	64000	68.66
WAT files	CC-MAIN-2021-49/wat.paths.gz	64000	16.66
WET files	CC-MAIN-2021-49/wet.paths.gz	64000	7.18
Robots.txt files	CC-MAIN-2021-49/robotstxt.paths.gz	64000	0.15
Non-200 responses files	CC-MAIN-2021-49/non200responses.paths.gz	64000	2.29
URL index files	CC-MAIN-2021-49/cc-index.paths.gz	302	0.2

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-49/. Also the columnar index has been updated to contain this crawl.

October 2021 crawl archive now available

November 1, 2021Sebastian Nagel

The crawl archive for October 2021 is now available! The data was crawled Oct 15 – 28 and contains 3.3 billion web pages or 360 TiB of uncompressed content. It includes page captures of 1.3 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The October crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-43/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-43/segment.paths.gz	100
WARC files	CC-MAIN-2021-43/warc.paths.gz	72000	85.11
WAT files	CC-MAIN-2021-43/wat.paths.gz	72000	22.42
WET files	CC-MAIN-2021-43/wet.paths.gz	72000	9.79
Robots.txt files	CC-MAIN-2021-43/robotstxt.paths.gz	72000	0.15
Non-200 responses files	CC-MAIN-2021-43/non200responses.paths.gz	72000	2.11
URL index files	CC-MAIN-2021-43/cc-index.paths.gz	302	0.25

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-43/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs June, July/August and September 2021

October 8, 2021Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of June, July/August and September 2021. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases. You may also visit the projects cc-webgraph and cc-pyspark which include all scripts and tools required to construct the graphs. Instructions to explore the graphs in the webgraph format are given in our collection of webgraph notebooks.

Host-level graph

The graph consists of 766 million nodes and 4.95 billion edges. Both hyperlinks and HTTP redirects and link headers are used as edges to span up the graph. All types of links are included, including pure “technical” ones pointing to images, JavaScript libraries, web fonts, etc. However, only host names with a valid IANA TLD are used. Consequently, URLs with an IP address as host component are not taken into account for building the host-level graph.

There are 701 million dangling nodes (91.6%) and the largest strongly connected component contains 47.0 million (6.1%) nodes. Dangling nodes stem from

hosts that have not been crawled, yet are pointed to from a link on a crawled page
hosts without any links pointing to a different host name
or hosts which did only return an error page (eg. HTTP 404)

Host names in the graph are in reverse domain name notation and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 766 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2021-jun-jul-sep/host/. Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2021-jun-jul-sep/host/ as prefix to access the files from everywhere.

Please note that the text representation of the host-level graph is shipped in 40 gzip-compressed files listed in two path listings – one for the nodes (vertices), one for the edges (arcs). First, download the paths listing and decompress it using “gzip”. By adding the prefix s3://commoncrawl/ or https://data.commoncrawl.org/ to each line in the path listing you get the list of URLs to download the entire graph.

Size	File	Description
4.50 GB	cc-main-2021-jun-jul-sep-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 16 vertices files
20.76 GB	cc-main-2021-jun-jul-sep-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 24 edges files

8.43 GB	cc-main-2021-jun-jul-sep-host.graph	graph in BVGraph format
2 kB	cc-main-2021-jun-jul-sep-host.properties

9.83 GB	cc-main-2021-jun-jul-sep-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2021-jun-jul-sep-host-t.properties

1 kB	cc-main-2021-jun-jul-sep-host.stats	WebGraph statistics

11.06 GB	cc-main-2021-jun-jul-sep-host-ranks.txt.gz	harmonic centrality and pagerank

Domain-level graph

The domain graph is built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org. Version (commit) a5b046d of the public suffix list was used (commit date 2021-10-06).

The domain-level graph has 88 million nodes and 1.56 billion edges. 49% or 43 million nodes are dangling nodes, the largest strongly connected component covers 35 million or 40% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2021-jun-jul-sep/domain/ resp. https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2021-jun-jul-sep/domain/.

Download files of the Common Crawl Jun/Jul/Sep 2021 domain-level webgraph

Size	File	Description
0.61 GB	cc-main-2021-jun-jul-sep-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
6.34 GB	cc-main-2021-jun-jul-sep-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

3.59 GB	cc-main-2021-jun-jul-sep-domain.graph	graph in BVGraph format
2 kB	cc-main-2021-jun-jul-sep-domain.properties

3.44 GB	cc-main-2021-jun-jul-sep-domain-t.graph	transpose of the graph
2 kB	cc-main-2021-jun-jul-sep-domain-t.properties

1 kB	cc-main-2021-jun-jul-sep-domain.stats	WebGraph statistics

1.89 GB	cc-main-2021-jun-jul-sep-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 88 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Jun/Jul/Sep 2021)

harmonic centrality rank	hc value	page rank	page rank value	reversed domain name
1	31634524	1	0.018526	com.googleapis
2	30590448	3	0.013879	com.facebook
3	29344102	2	0.015082	com.google
4	26579208	5	0.007506	org.w
5	26490886	4	0.008344	com.twitter
6	26194504	6	0.007270	com.youtube
7	25598364	8	0.006802	com.instagram
8	24808618	7	0.006829	com.googletagmanager
9	24314334	9	0.004946	org.gmpg
10	23501518	14	0.003442	com.linkedin
11	23191984	11	0.003661	com.gstatic
12	22552330	10	0.004093	com.cloudflare
13	22306348	17	0.001939	com.gravatar
14	22254410	12	0.003556	org.wordpress
15	22159662	22	0.001575	com.pinterest
16	21706482	23	0.001396	org.wikipedia
17	21593592	16	0.001952	com.apple
18	21537622	37	0.001024	com.wordpress
19	21459498	27	0.001176	com.vimeo
20	21360146	41	0.000898	be.youtu
21	21289640	15	0.002252	com.bootstrapcdn
22	21211768	20	0.001717	com.jquery
23	21069122	28	0.001166	com.microsoft
24	21014760	54	0.000699	com.blogspot
25	20975968	46	0.000793	com.amazon
26	20916000	45	0.000798	gl.goo
27	20909476	50	0.000720	com.wp
28	20878568	62	0.000580	ly.bit
29	20832704	35	0.001053	com.google-analytics
30	20815940	33	0.001068	com.amazonaws
31	20731242	77	0.000417	com.tumblr
32	20705584	42	0.000841	org.mozilla
33	20703870	31	0.001080	net.cloudfront
34	20700046	18	0.001797	com.adobe
35	20687574	49	0.000720	eu.europa
36	20682862	19	0.001772	com.github
37	20663412	34	0.001054	net.jsdelivr
38	20641588	29	0.001134	com.wixstatic
39	20534372	75	0.000442	com.googleusercontent
40	20518204	21	0.001630	com.fontawesome
41	20499484	96	0.000366	com.yahoo
42	20489920	51	0.000711	com.paypal
43	20470068	57	0.000645	co.t
44	20460526	47	0.000784	com.whatsapp
45	20453542	53	0.000704	com.flickr
46	20429610	106	0.000315	com.reddit
47	20405542	72	0.000449	com.medium
48	20396096	38	0.000956	com.googlesyndication
49	20388958	48	0.000748	io.github
50	20364584	125	0.000241	com.nytimes
51	20363658	59	0.000616	org.w3
52	20350454	98	0.000358	com.weebly
53	20330952	97	0.000363	org.creativecommons
54	20324280	65	0.000510	com.shopify
55	20315624	110	0.000281	com.soundcloud
56	20308274	85	0.000402	me.wp
57	20300136	60	0.000610	org.schema
58	20300004	32	0.001072	ru.yandex
59	20272874	132	0.000215	com.forbes
60	20266896	64	0.000527	com.vk
61	20216640	114	0.000268	com.spotify
62	20214818	175	0.000146	com.cnn
63	20172836	52	0.000708	net.doubleclick
64	20156654	56	0.000653	com.addthis
65	20153642	190	0.000132	uk.co.bbc
66	20151882	230	0.000113	com.wsj
67	20149822	135	0.000208	gov.nih
68	20146194	40	0.000924	com.baidu
69	20144952	161	0.000166	com.theguardian
70	20127172	151	0.000179	int.who
71	20121134	240	0.000108	com.bloomberg
72	20115270	130	0.000217	org.archive
73	20111568	163	0.000163	com.giphy
74	20107092	94	0.000388	com.list-manage
75	20100830	192	0.000130	org.wikimedia
76	20095176	55	0.000682	com.macromedia
77	20091806	229	0.000113	com.oracle
78	20081572	145	0.000193	com.imdb
79	20074580	187	0.000134	com.businessinsider
80	20066912	297	0.000091	edu.mit
81	20066436	300	0.000090	edu.stanford
82	20061314	115	0.000267	com.mailchimp
83	20059906	44	0.000800	net.facebook
84	20058050	140	0.000204	us.zoom
85	20056682	426	0.000065	com.googleblog
86	20053210	171	0.000153	com.unsplash
87	20047020	284	0.000095	com.reuters
88	20043722	185	0.000137	com.imgur
89	20035754	142	0.000199	com.wixsite
90	20025300	191	0.000131	com.stackoverflow
91	20022100	128	0.000223	com.weibo
92	20016560	162	0.000163	com.issuu
93	20016026	423	0.000065	gov.nasa
94	20014282	36	0.001043	net.fbcdn
95	20013910	273	0.000098	com.android
96	20012778	119	0.000259	me.t
97	20012366	169	0.000160	org.ietf
98	20004530	139	0.000206	com.ytimg
99	20001182	148	0.000183	org.apache
100	19999464	318	0.000086	com.theverge
101	19993026	325	0.000084	com.slack
102	19992850	258	0.000103	edu.harvard
103	19989870	224	0.000116	com.washingtonpost
104	19988302	268	0.000099	com.bbc
105	19986642	460	0.000060	edu.cornell
106	19984712	155	0.000174	com.ft
107	19979856	117	0.000261	com.npmjs
108	19978744	416	0.000066	com.ted
109	19962738	424	0.000065	com.myspace
110	19957322	352	0.000078	com.wired
111	19956296	526	0.000052	com.livejournal
112	19945484	355	0.000078	com.appspot
113	19944274	263	0.000101	org.un
114	19943458	215	0.000119	org.gnu
115	19933112	390	0.000070	com.goodreads
116	19932600	69	0.000476	com.godaddy
117	19931522	387	0.000071	org.hbr
118	19917440	309	0.000087	org.npr
119	19916612	322	0.000085	com.prnewswire
120	19911628	332	0.000083	net.researchgate
121	19910122	307	0.000088	com.githubusercontent
122	19909276	24	0.001390	io.polyfill
123	19908276	286	0.000095	com.wiley
124	19905960	247	0.000106	com.tiktok
125	19894990	193	0.000130	com.blogger
126	19892346	71	0.000466	com.unpkg
127	19884978	102	0.000322	de.google
128	19869556	432	0.000064	com.gmail
129	19869236	565	0.000049	com.vice
130	19868406	627	0.000045	org.chromium
131	19867146	146	0.000186	gle.forms
132	19866916	113	0.000269	com.youtube-nocookie
133	19866834	179	0.000142	org.ampproject
134	19856608	354	0.000078	com.time
135	19856272	668	0.000043	edu.upenn
136	19856040	370	0.000074	com.example
137	19854818	633	0.000045	com.economist
138	19850884	727	0.000040	com.evernote
139	19842434	500	0.000055	com.steampowered
140	19841380	692	0.000042	google.blog
141	19841354	462	0.000060	com.theatlantic
142	19840036	586	0.000047	org.weforum
143	19837596	628	0.000045	com.deviantart
144	19837072	239	0.000109	uk.co.google
145	19833470	357	0.000077	org.arxiv
146	19833402	395	0.000070	com.scribd
147	19831882	407	0.000067	uk.co.telegraph
148	19829878	368	0.000074	com.huffingtonpost
149	19829876	642	0.000045	com.mysql
150	19821800	555	0.000050	org.worldbank
151	19818100	242	0.000107	com.sciencedirect
152	19816676	347	0.000080	com.nature
153	19814992	228	0.000113	com.twimg
154	19812336	120	0.000259	com.statcounter
155	19810098	246	0.000106	org.acm
156	19805214	554	0.000050	org.ieee
157	19799686	381	0.000071	com.fastcompany
158	19796892	255	0.000104	org.python
159	19794906	701	0.000041	com.apnews
160	19790820	431	0.000064	com.meetup
161	19790276	738	0.000040	com.qz
162	19789858	581	0.000047	com.globenewswire
163	19788416	365	0.000074	com.docker
164	19788402	444	0.000062	com.pixabay
165	19788038	453	0.000061	uk.co.dailymail
166	19784442	321	0.000085	com.springer
167	19783552	251	0.000105	com.bandcamp
168	19779288	256	0.000103	net.behance
169	19778076	361	0.000075	com.gitlab
170	19768948	603	0.000046	com.git-scm
171	19764224	540	0.000051	io.readthedocs
172	19763982	715	0.000041	com.engadget
173	19763034	647	0.000044	com.trello
174	19761140	183	0.000138	com.bing
175	19758120	346	0.000080	com.usatoday
176	19757326	220	0.000117	com.squarespace
177	19752428	158	0.000169	com.yelp
178	19747884	353	0.000078	com.dribbble
179	19747756	417	0.000066	com.digg
180	19747532	143	0.000199	com.dropbox
181	19744818	276	0.000097	com.ibm
182	19734092	464	0.000060	uk.co.independent
183	19733310	398	0.000069	com.w3schools
184	19723800	544	0.000051	ee.linktr
185	19723334	674	0.000043	uk.co.blogspot
186	19723200	412	0.000067	com.staticflickr
187	19721228	551	0.000050	com.pexels
188	19721224	144	0.000194	gov.cdc
189	19718802	539	0.000051	org.pbs
190	19718388	775	0.000038	com.stackexchange
191	19718222	703	0.000041	org.cambridge
192	19717352	1103	0.000029	org.eclipse
193	19716696	43	0.000819	com.fb
194	19712516	678	0.000042	edu.columbia
195	19710308	100	0.000351	com.wix
196	19706256	730	0.000040	edu.washington
197	19705612	290	0.000094	com.tinyurl
198	19705126	367	0.000074	com.sagepub
199	19704258	673	0.000043	me.about
200	19699044	219	0.000117	net.slideshare
201	19694818	653	0.000044	org.sciencemag
202	19689094	294	0.000091	org.pewresearch
203	19685770	601	0.000046	com.withgoogle
204	19684956	458	0.000061	com.herokuapp
205	19675188	440	0.000063	com.quora
206	19673070	123	0.000246	com.sharethis
207	19672408	39	0.000941	com.qq
208	19671982	176	0.000145	org.doi
209	19670088	438	0.000063	co.ibb
210	19667674	614	0.000046	com.newyorker
211	19664724	1183	0.000027	com.nike
212	19664240	316	0.000087	com.typeform
213	19658254	248	0.000106	com.outlook
214	19654436	736	0.000040	com.hp
215	19654104	791	0.000037	com.foxnews
216	19651026	234	0.000112	com.cloudinary
217	19648214	933	0.000035	edu.princeton
218	19646574	572	0.000048	com.moz
219	19642260	437	0.000063	com.getpocket
220	19639898	485	0.000057	com.nbcnews
221	19638836	645	0.000044	org.bitbucket
222	19635872	202	0.000124	page.g
223	19635142	154	0.000176	gov.privacyshield
224	19634924	277	0.000096	com.disqus
225	19624936	141	0.000203	com.opera
226	19623614	463	0.000060	com.airbnb
227	19620318	960	0.000034	com.dropboxusercontent
228	19618624	372	0.000073	com.force
229	19617796	923	0.000035	co.elastic
230	19614006	214	0.000119	com.wpengine
231	19613164	679	0.000042	org.semver
232	19613160	305	0.000089	com.typepad
233	19611254	972	0.000033	com.nypost
234	19610798	726	0.000040	com.ubuntu
235	19610070	1263	0.000025	se.haxx
236	19605234	303	0.000089	com.live
237	19603428	683	0.000042	au.net.abc
238	19603360	465	0.000060	com.mozilla
239	19599764	382	0.000071	com.criteo
240	19593184	1232	0.000026	uk.co.thesun
241	19581354	1434	0.000023	edu.rutgers
242	19581020	278	0.000096	com.feedburner
243	19577288	969	0.000033	com.politico
244	19572414	1054	0.000030	co.g
245	19572372	1849	0.000019	com.instructables
246	19570874	778	0.000038	com.sap
247	19568694	1123	0.000028	org.greenpeace
248	19564134	1127	0.000028	org.kernel
249	19562280	1650	0.000022	com.googlesource
250	19561484	124	0.000245	com.filesusr
251	19559444	1150	0.000028	com.unity3d
252	19558056	682	0.000042	com.freepik
253	19557106	542	0.000051	com.fortune
254	19553954	638	0.000045	uk.ac.ox
255	19553634	209	0.000121	org.iana
256	19551872	241	0.000108	com.eepurl
257	19551398	958	0.000034	com.ssrn
258	19551010	866	0.000035	com.nvidia
259	19549544	1569	0.000022	com.storify
260	19549036	1000	0.000032	com.sun
261	19548514	624	0.000045	uk.co.eventbrite
262	19544466	829	0.000036	edu.jhu
263	19541150	607	0.000046	net.azurewebsites
264	19539076	1291	0.000025	com.reverbnation
265	19538660	439	0.000063	gov.fda
266	19538044	99	0.000351	com.stripe
267	19534860	927	0.000035	com.podbean
268	19532612	341	0.000082	net.windows
269	19530966	1256	0.000025	uk.co.ebay
270	19530706	260	0.000102	com.calendly
271	19526632	804	0.000037	com.chrome
272	19525016	1804	0.000020	com.martinfowler
273	19524922	926	0.000035	edu.academia
274	19523484	568	0.000049	site.business
275	19521568	231	0.000113	com.office
276	19521106	328	0.000084	com.netdna-ssl
277	19518874	253	0.000105	com.newsweek
278	19516880	210	0.000120	tv.twitch
279	19515182	1220	0.000026	com.vogue
280	19514726	2307	0.000017	com.diigo
281	19514662	1214	0.000026	org.postgresql
282	19513292	691	0.000042	com.xinhuanet
283	19508378	1368	0.000024	de.mpg
284	19508326	525	0.000052	com.squareup
285	19507892	408	0.000067	org.debian
286	19502082	121	0.000252	com.paypalobjects
287	19501986	716	0.000041	gov.senate
288	19501788	2392	0.000016	com.pearltrees
289	19500240	1147	0.000028	com.500px
290	19499046	343	0.000081	com.googlecode
291	19497980	712	0.000041	org.change
292	19497516	491	0.000056	com.tandfonline
293	19496848	67	0.000505	net.akamaihd
294	19492076	1265	0.000025	com.aljazeera
295	19492054	706	0.000041	com.qualtrics
296	19490750	783	0.000037	com.theconversation
297	19490388	1078	0.000029	com.theglobeandmail
298	19486228	134	0.000211	de.bund
299	19485788	1299	0.000025	edu.illinois
300	19485786	327	0.000084	com.cnbc
301	19484140	858	0.000036	uk.co.guardian
302	19483768	483	0.000057	com.msn
303	19475416	129	0.000219	com.rawgit
304	19472372	474	0.000059	com.stumbleupon
305	19469278	198	0.000125	net.sourceforge
306	19468982	356	0.000077	com.optimizely
307	19468656	470	0.000059	org.openstreetmap
308	19467526	335	0.000083	com.techcrunch
309	19465998	434	0.000064	com.ssl-images-amazon
310	19464188	1405	0.000023	edu.ufl
311	19462912	1395	0.000023	edu.gatech
312	19462816	165	0.000162	com.hubspot
313	19462736	310	0.000087	com.mapbox
314	19462404	441	0.000063	com.go
315	19459072	608	0.000046	gov.noaa
316	19458846	1470	0.000023	com.channel4
317	19458794	1237	0.000026	ca.sfu
318	19457698	592	0.000047	com.healthline
319	19457660	794	0.000037	org.fao
320	19457528	507	0.000054	ca.google
321	19457170	2431	0.000016	com.wattpad
322	19454268	1280	0.000025	uk.co.standard
323	19452678	667	0.000043	gov.house
324	19451936	1309	0.000025	uk.co.wired
325	19450294	1862	0.000019	com.invisionapp
326	19449754	743	0.000039	com.pinimg
327	19449738	184	0.000138	com.amazon-adsystem
328	19449480	1836	0.000019	org.maven
329	19446242	2507	0.000015	com.openai
330	19443698	803	0.000037	org.pypi
331	19442142	262	0.000101	net.azureedge
332	19441000	490	0.000056	com.kickstarter
333	19439570	1962	0.000019	uk.bl
334	19439402	1236	0.000026	au.com.smh
335	19436904	1339	0.000024	com.vanityfair
336	19435504	261	0.000102	uk.org.ico
337	19434368	153	0.000176	com.addtoany
338	19427034	986	0.000032	com.uk
339	19426114	1097	0.000029	com.scmp
340	19424036	981	0.000033	org.pnas
341	19422474	433	0.000064	com.cnet
342	19422174	348	0.000080	com.statista
343	19421360	149	0.000180	org.nodejs
344	19418836	793	0.000037	us.icio
345	19416290	2916	0.000013	com.instapaper
346	19415752	489	0.000056	gov.epa
347	19405682	1003	0.000032	com.mixcloud
348	19404500	677	0.000042	org.d3js
349	19401872	851	0.000036	com.britannica
350	19400204	593	0.000047	uk.gov.service
351	19399466	177	0.000143	org.allaboutcookies
352	19396954	546	0.000050	edu.berkeley
353	19395162	296	0.000091	me.telegram
354	19393670	1436	0.000023	com.irishtimes
355	19392716	944	0.000034	int.coe
356	19388874	157	0.000170	com.zendesk
357	19388222	1507	0.000022	org.hrc
358	19387040	1251	0.000025	com.history
359	19386872	152	0.000178	io.shields
360	19386294	1686	0.000021	ms.1drv
361	19384766	548	0.000050	com.biomedcentral
362	19384760	473	0.000059	com.latimes
363	19381234	1062	0.000030	org.jstor
364	19381206	1124	0.000028	com.jetbrains
365	19376152	1084	0.000029	org.ilo
366	19376036	777	0.000038	edu.psu
367	19375334	137	0.000207	com.youronlinechoices
368	19374986	895	0.000035	com.ecwid
369	19371930	1067	0.000030	com.brightcove
370	19371460	748	0.000039	it.scoop
371	19369410	329	0.000084	ru.ok
372	19364910	1226	0.000026	com.digitaltrends
373	19362788	1004	0.000032	uk.co.thetimes
374	19359762	1404	0.000023	com.thedailybeast
375	19359122	1604	0.000022	edu.osu
376	19354290	513	0.000054	edu.yale
377	19351764	116	0.000262	com.jimdo
378	19349800	2354	0.000016	com.fastcodesign
379	19349466	988	0.000032	uk.parliament
380	19348128	456	0.000061	org.freecodecamp
381	19343488	2209	0.000018	com.us
382	19340214	573	0.000048	com.deloitte
383	19339976	1010	0.000031	uk.co.huffingtonpost
384	19339738	605	0.000046	com.zdnet
385	19339098	181	0.000139	ru.mail
386	19337572	415	0.000066	com.elsevier
387	19335810	1185	0.000027	org.nejm
388	19335572	2422	0.000016	com.instructure
389	19335456	427	0.000065	net.imgix
390	19334596	1714	0.000021	com.citrix
391	19330958	2590	0.000014	org.aclweb
392	19330886	1964	0.000018	org.haskell
393	19326754	649	0.000044	gov.state
394	19326634	1350	0.000024	app.netlify
395	19326068	801	0.000037	com.venturebeat
396	19323896	217	0.000118	com.eventbrite
397	19323876	622	0.000045	com.seattletimes
398	19323284	1868	0.000019	jp.ac.u-tokyo
399	19322408	3048	0.000012	org.uxplanet
400	19319794	1186	0.000027	com.dw
401	19318404	1248	0.000026	org.undp
402	19317950	211	0.000120	com.etsy
403	19317948	1763	0.000020	com.itv
404	19317548	233	0.000112	net.php
405	19314522	58	0.000638	com.googleadservices
406	19314004	271	0.000098	com.surveymonkey
407	19310004	245	0.000107	org.aboutcookies
408	19309574	1900	0.000019	edu.vt
409	19307434	2477	0.000015	org.wikibooks
410	19305780	687	0.000042	gov.nist
411	19297658	1423	0.000023	com.thehindu
412	19297142	1120	0.000028	org.hrw
413	19295118	600	0.000046	com.thinkwithgoogle
414	19294574	1682	0.000021	gov.usembassy
415	19294274	656	0.000044	com.intel
416	19292884	1725	0.000021	int.unfccc
417	19291806	394	0.000070	com.ebay
418	19289908	2419	0.000016	google.ai
419	19286562	2270	0.000017	com.netvibes
420	19285156	2562	0.000015	io.material
421	19283106	2235	0.000017	ly.rebrand
422	19280870	1954	0.000019	org.archlinux
423	19280114	878	0.000035	uk.co.pinterest
424	19279216	2833	0.000013	org.doctorswithoutborders
425	19277590	2069	0.000018	org.accessnow
426	19272506	1482	0.000023	com.findlaw
427	19271938	800	0.000037	net.clickbank
428	19270876	3536	0.000010	com.viki
429	19269508	980	0.000033	edu.brookings
430	19265638	2363	0.000016	co.carrd
431	19265518	2950	0.000012	org.neocities
432	19261700	1294	0.000025	com.wikia
433	19260608	657	0.000044	com.mashable
434	19260346	798	0.000037	com.thelancet
435	19259074	974	0.000033	uk.ac.cam
436	19258964	2416	0.000016	org.rsf
437	19258786	1841	0.000019	net.daringfireball
438	19255684	708	0.000041	com.canva
439	19255674	495	0.000056	gov.whitehouse
440	19253374	204	0.000122	com.salesforce
441	19252328	1191	0.000027	com.thenextweb
442	19251934	2076	0.000018	com.france24
443	19251914	534	0.000052	io.codepen
444	19250670	2896	0.000013	com.laweekly
445	19250196	1218	0.000026	com.licdn
446	19248894	2839	0.000013	cc.uxdesign
447	19245136	2368	0.000016	edu.kit
448	19240312	1071	0.000030	watch.fb
449	19239224	2583	0.000014	org.scala-lang
450	19238926	2513	0.000015	au.com.theage
451	19234622	2644	0.000014	com.hubpages
452	19232860	1533	0.000022	ch.ipcc
453	19232358	1092	0.000029	com.digitaloceanspaces
454	19231078	2435	0.000015	org.vim
455	19227408	1721	0.000021	com.refinery29
456	19227102	292	0.000093	net.secureservercdn
457	19225996	769	0.000038	com.marketwatch
458	19224690	2608	0.000014	app.web
459	19224666	1681	0.000022	org.unwomen
460	19223622	2447	0.000015	com.fineartamerica
461	19222710	2566	0.000015	nl.blogspot
462	19219678	537	0.000051	edu.cmu
463	19219662	505	0.000054	fr.free
464	19217122	831	0.000036	com.box
465	19216944	1159	0.000027	com.imageshack
466	19215358	2409	0.000016	edu.usf
467	19214138	2396	0.000016	nz.co.nzherald
468	19212598	2758	0.000013	com.smashwords
469	19208544	364	0.000074	net.datatables
470	19207930	419	0.000066	com.nationalgeographic
471	19207832	287	0.000095	com.iubenda
472	19206220	2470	0.000015	re.appsto
473	19205746	235	0.000112	com.adnxs
474	19203826	2509	0.000015	org.gentoo
475	19203684	1982	0.000018	com.voanews
476	19203064	2498	0.000015	com.superuser
477	19201090	379	0.000072	com.businesswire
478	19200162	643	0.000045	int.wipo
479	19199426	2201	0.000018	org.biorxiv
480	19198874	1451	0.000023	org.amnesty
481	19198454	2436	0.000015	com.oregonlive
482	19198448	2033	0.000018	org.nobelprize
483	19197824	81	0.000414	net.jsfiddle
484	19197384	1963	0.000019	com.ew
485	19195648	1008	0.000031	com.arstechnica
486	19194700	1826	0.000020	org.ocks
487	19194656	265	0.000101	com.aliyuncs
488	19192314	2649	0.000014	com.dezeen
489	19191524	2678	0.000014	org.transparency
490	19191474	925	0.000035	org.mediawiki
491	19191020	2853	0.000013	com.scribblelive
492	19190866	1357	0.000024	io.gitlab
493	19190776	1683	0.000021	org.aiga
494	19188994	1985	0.000018	uk.gov.tfl
495	19186250	428	0.000064	com.adweek
496	19185528	1941	0.000019	org.unep
497	19185444	430	0.000064	org.js
498	19184030	371	0.000073	com.atlassian
499	19183274	2282	0.000017	com.foreignpolicy
500	19181288	2696	0.000014	org.democracynow
501	19181160	985	0.000032	com.webs
502	19181124	1198	0.000026	com.wetransfer
503	19179532	1369	0.000024	org.altervista
504	19177814	2501	0.000015	google.research
505	19177102	2959	0.000012	za.co.iol
506	19175572	1028	0.000031	com.slate
507	19175528	2713	0.000014	org.cpj
508	19174456	2016	0.000018	org.example
509	19173782	2390	0.000016	com.googlegroups
510	19167902	243	0.000107	com.naver
511	19164304	2266	0.000017	net.openid
512	19164154	3058	0.000012	com.deepmind
513	19164022	269	0.000099	org.drupal
514	19163716	264	0.000101	gov.ca
515	19163714	414	0.000067	com.livechatinc
516	19155586	2328	0.000016	com.washingtontimes
517	19153082	637	0.000045	com.cbsnews
518	19151824	759	0.000038	com.oreilly
519	19150012	3034	0.000012	com.podomatic
520	19149538	664	0.000043	gov.loc
521	19147634	136	0.000208	org.networkadvertising
522	19146702	718	0.000041	com.buzzfeed
523	19144896	1333	0.000024	link.page
524	19143440	838	0.000036	com.pcmag
525	19142224	956	0.000034	com.verisign
526	19133476	2892	0.000013	com.thoughtworks
527	19132758	2669	0.000014	uk.co.timesonline
528	19131302	274	0.000097	com.getbootstrap
529	19130872	3082	0.000012	com.mariadb
530	19129650	1116	0.000028	com.jekyllrb
531	19128992	938	0.000034	com.vox
532	19127508	127	0.000234	info.aboutads
533	19122362	492	0.000056	com.patreon
534	19122034	2488	0.000015	com.curbed
535	19121736	514	0.000054	it.placehold
536	19121622	1807	0.000020	com.ascentlawfirm
537	19118802	227	0.000114	to.amzn
538	19117462	580	0.000047	com.visualstudio
539	19117426	1281	0.000025	com.smashingmagazine
540	19116926	499	0.000055	com.sxsw
541	19116922	978	0.000033	com.hootsuite
542	19116384	282	0.000095	gov.ftc
543	19113570	2336	0.000016	com.snopes
544	19110698	1392	0.000023	com.upwork
545	19109920	1388	0.000024	com.haaretz
546	19108142	1701	0.000021	com.firebaseapp
547	19104686	920	0.000035	com.zoho
548	19103768	3023	0.000012	org.peta
549	19100220	1174	0.000027	com.att
550	19097502	1687	0.000021	com.techrepublic
551	19097348	1776	0.000020	com.surveygizmo
552	19097050	2394	0.000016	com.treehugger
553	19095294	3359	0.000011	com.letterboxd
554	19094856	3025	0.000012	gov.anl
555	19094436	2568	0.000015	com.kaggle
556	19091430	2479	0.000015	fm.omny
557	19091422	2999	0.000012	com.bangkokpost
558	19089542	478	0.000058	gov.irs
559	19086962	1790	0.000020	ca.bc.gov
560	19086880	823	0.000036	com.emarketer
561	19086854	1173	0.000027	com.mediaplex
562	19086370	259	0.000103	uk.co.amazon
563	19084140	2737	0.000014	int.au
564	19083740	2514	0.000015	no.google
565	19083612	3618	0.000010	com.newgrounds
566	19083596	186	0.000134	jp.co.yahoo
567	19082852	3108	0.000012	org.hypotheses
568	19082332	337	0.000082	mp.mailchi
569	19082228	2808	0.000013	com.usmagazine
570	19080648	1674	0.000022	com.routledge
571	19078904	2881	0.000013	org.polymer-project
572	19077732	2446	0.000015	org.unctad
573	19077060	180	0.000140	com.caniuse
574	19076142	344	0.000080	com.onesignal
575	19073714	2714	0.000014	int.interpol
576	19072522	3248	0.000011	org.elasticsearch
577	19072218	620	0.000045	com.entrepreneur
578	19072108	2550	0.000015	uk.gov.metoffice
579	19071016	2921	0.000013	org.jenkins-ci
580	19070052	632	0.000045	com.samsung
581	19070008	1367	0.000024	org.unicode
582	19069412	3381	0.000011	uk.mod
583	19069358	2715	0.000014	org.mozillazine
584	19067032	3148	0.000011	edu.ucpress
585	19066706	1156	0.000027	com.gizmodo
586	19064356	1901	0.000019	org.americanbar
587	19063358	3593	0.000010	org.scala-sbt
588	19062892	330	0.000084	ai.shortpixel
589	19059914	2912	0.000013	in.indiatoday
590	19058702	301	0.000090	gg.discord
591	19058544	3687	0.000010	jp.riken
592	19056820	2440	0.000015	com.timesofisrael
593	19056518	3064	0.000012	com.manta
594	19056166	785	0.000037	com.fandom
595	19056052	1274	0.000025	com.sfgate
596	19054812	1994	0.000018	com.knightlab
597	19053684	2042	0.000018	org.donorbox
598	19053668	2251	0.000017	eu.politico
599	19051874	2494	0.000015	org.gnupg
600	19051168	84	0.000402	me.ogp
601	19050784	575	0.000048	com.cisco
602	19050146	2757	0.000013	uk.ac.york
603	19048588	1189	0.000027	com.buffer
604	19048534	3065	0.000012	uk.org.wwf
605	19047192	929	0.000035	com.variety
606	19045218	3517	0.000010	com.flightradar24
607	19044416	3387	0.000011	com.flock
608	19044368	579	0.000048	com.sedo
609	19044084	871	0.000035	com.libsyn
610	19043162	2561	0.000015	com.thenation
611	19042992	2569	0.000015	com.monday
612	19042226	482	0.000057	com.arcgis
613	19042098	3223	0.000011	net.inquirer
614	19038824	2699	0.000014	com.real
615	19036762	2195	0.000018	com.secondlife
616	19035414	732	0.000040	org.unesco
617	19035292	932	0.000035	com.wikihow
618	19034196	2609	0.000014	uk.ac.leeds
619	19033676	80	0.000415	com.livestream
620	19031566	3100	0.000012	org.cato
621	19029288	2716	0.000014	org.sonatype
622	19028098	3169	0.000011	com.intensedebate
623	19028006	1089	0.000029	com.symantec
624	19027462	4048	0.000009	org.jw
625	19027208	2793	0.000013	com.wayfair
626	19026864	1922	0.000019	com.scene7
627	19026030	76	0.000421	com.messenger
628	19025656	1689	0.000021	org.coursera
629	19024592	1196	0.000026	edu.umn
630	19024030	3093	0.000012	org.rferl
631	19023590	2512	0.000015	org.wikidata
632	19022464	644	0.000045	com.psychologytoday
633	19021606	3091	0.000012	com.vancouversun
634	19021016	2496	0.000015	uk.org.nationaltrust
635	19020356	1553	0.000022	ly.ow
636	19020300	1264	0.000025	edu.ucsd
637	19020198	2905	0.000013	tr.com.aa
638	19018958	3767	0.000010	it.polito
639	19017674	3389	0.000011	org.sourcewatch
640	19017264	3207	0.000011	ch.qos
641	19017144	3785	0.000010	jp.ac.kobe-u
642	19016500	1659	0.000022	com.speakerdeck
643	19015604	2974	0.000012	com.sciencealert
644	19015034	636	0.000045	com.photobucket
645	19015026	3087	0.000012	com.hsbc
646	19014412	3852	0.000009	edu.uah
647	19013454	188	0.000133	com.jimcdn
648	19012700	1195	0.000027	com.rollingstone
649	19012558	2897	0.000013	org.osce
650	19012150	4205	0.000009	com.gust
651	19009874	1684	0.000021	org.webkit
652	19009396	945	0.000034	com.shutterstock
653	19008674	2945	0.000012	com.townnews
654	19008318	2650	0.000014	org.wri
655	19006196	520	0.000053	com.inc
656	19004672	648	0.000044	com.gartner
657	19004226	2502	0.000015	ru.rg
658	19003942	2648	0.000014	io.bower
659	19003780	3371	0.000011	net.thedailystar
660	19003722	2564	0.000015	net.dwcdn
661	19003574	2770	0.000013	com.articulate
662	19003510	221	0.000117	com.myshopify
663	19002868	172	0.000151	jp.co.google
664	19002548	1354	0.000024	gov.uspto
665	19000824	998	0.000032	edu.ucla
666	18998632	685	0.000042	com.investopedia
667	18998502	3161	0.000011	com.mongabay
668	18997700	532	0.000052	com.aol
669	18994210	2352	0.000016	ca.citizenlab
670	18993878	1284	0.000025	com.today
671	18992626	160	0.000167	org.whatwg
672	18992320	1048	0.000030	com.smartadserver
673	18992054	2995	0.000012	org.pewforum
674	18991266	2997	0.000012	org.sierraclub
675	18991062	2964	0.000012	net.vnexpress
676	18990848	1069	0.000030	com.about
677	18989274	3081	0.000012	uk.co.spectator
678	18988482	480	0.000058	com.dmca
679	18987042	1462	0.000023	ly.cutt
680	18986890	3224	0.000011	ru.interfax
681	18986546	3670	0.000010	uk.co.zoopla
682	18985004	2843	0.000013	org.iucnredlist
683	18984682	194	0.000130	com.tripadvisor
684	18984256	3473	0.000010	fm.audioboo
685	18983654	2762	0.000013	uk.co.bbci
686	18983098	3416	0.000011	edu.sjsu
687	18982290	1444	0.000023	edu.northwestern
688	18982040	543	0.000051	com.googleoptimize
689	18980160	2817	0.000013	int.iom
690	18979598	1386	0.000024	edu.umd
691	18978832	629	0.000045	org.eff
692	18978224	2297	0.000017	uk.org.ofcom
693	18978038	2450	0.000015	int.reliefweb
694	18977708	3502	0.000010	com.torontosun
695	18975348	493	0.000056	com.indeed
696	18973238	1657	0.000022	com.nngroup
697	18972898	351	0.000078	com.constantcontact
698	18972774	1780	0.000020	co.lpages
699	18972434	1329	0.000024	edu.utexas
700	18971302	3386	0.000011	com.iconarchive
701	18971240	312	0.000087	com.pubmatic
702	18971078	1043	0.000030	org.reactjs
703	18969428	975	0.000033	edu.umich
704	18968428	1219	0.000026	com.tableau
705	18968208	1914	0.000019	com.hatenablog
706	18967574	1138	0.000028	com.chicagotribune
707	18967196	3999	0.000009	info.spain
708	18965740	547	0.000050	gov.copyright
709	18965442	4171	0.000009	org.gwtproject
710	18964656	650	0.000044	com.netflix
711	18963714	753	0.000039	net.adform
712	18961462	2847	0.000013	uk.ac.jisc
713	18961172	2865	0.000013	com.ringcentral
714	18960292	976	0.000033	com.redhat
715	18960280	3237	0.000011	com.city-data
716	18959938	2982	0.000012	uk.org.stonewall
717	18958764	3646	0.000010	za.co.timeslive
718	18957560	3989	0.000009	com.programmableweb
719	18957204	403	0.000068	com.bigcommerce
720	18957166	3053	0.000012	com.flippa
721	18955710	3293	0.000011	com.multiscreensite
722	18955294	2719	0.000014	com.bloglines
723	18954560	2629	0.000014	mp.j
724	18952520	2807	0.000013	uk.org.rspb
725	18952358	2949	0.000012	com.foreignaffairs
726	18952314	2206	0.000018	co.pcdn
727	18951462	3549	0.000010	in.theprint
728	18951006	4227	0.000009	com.symbaloo
729	18950992	4478	0.000008	com.algorithmia
730	18950356	1733	0.000021	com.billboard
731	18949998	742	0.000039	com.splashthat
732	18948332	3255	0.000011	com.cleantechnica
733	18947926	3604	0.000010	com.businessdailyafrica
734	18947204	1108	0.000028	com.dell
735	18947190	2825	0.000013	com.yell
736	18947008	443	0.000062	net.hubspot
737	18946944	3838	0.000010	org.rfa
738	18946618	3495	0.000010	za.co.mg
739	18945038	4346	0.000008	com.apsense
740	18944960	1772	0.000020	com.alibabagroup
741	18944660	2267	0.000017	to.dev
742	18944558	3471	0.000010	ru.mid
743	18944274	3498	0.000010	com.itsnicethat
744	18942634	527	0.000052	org.unicef
745	18942358	2364	0.000016	net.noscript
746	18940606	1390	0.000024	com.techradar
747	18938348	1857	0.000019	edu.uci
748	18937064	1118	0.000028	com.windowsphone
749	18936626	2734	0.000014	com.doubleclickbygoogle
750	18936326	3524	0.000010	org.350
751	18935080	3076	0.000012	org.aei
752	18934558	3075	0.000012	gov.arts
753	18934248	671	0.000043	gov.sec
754	18933682	2298	0.000017	com.urbandictionary
755	18933596	3925	0.000009	com.forbesimg
756	18933310	487	0.000056	com.fc2
757	18931436	3352	0.000011	com.brill
758	18931406	2491	0.000015	com.infoworld
759	18930782	1352	0.000024	com.bazaarvoice
760	18930350	3600	0.000010	de.uni-konstanz
761	18930232	1187	0.000027	com.alexa
762	18929982	2240	0.000017	org.linuxfoundation
763	18929738	3580	0.000010	edu.dukeupress
764	18929218	4038	0.000009	com.hotfrog
765	18928884	512	0.000054	com.mckinsey
766	18928710	2539	0.000015	org.crossref
767	18928328	3893	0.000009	com.environmentalleader
768	18927900	2236	0.000017	tv.ustream
769	18927290	1101	0.000029	fm.last
770	18926906	1951	0.000019	com.businessweek
771	18926816	413	0.000067	org.opensource
772	18925308	750	0.000039	org.whatbrowser
773	18925012	1296	0.000025	com.merriam-webster
774	18924400	425	0.000065	com.proofpoint
775	18923302	3196	0.000011	com.alchemer
776	18922680	3684	0.000010	com.arfadia
777	18922120	1692	0.000021	com.kinstacdn
778	18921334	3619	0.000010	com.ecowatch
779	18921302	2215	0.000018	net.leadpages
780	18920188	3448	0.000010	com.total
781	18920112	3878	0.000009	uk.org.npg
782	18919836	3054	0.000012	io.crates
783	18919320	2517	0.000015	com.lego
784	18919150	503	0.000055	com.wufoo
785	18915478	2742	0.000014	io.redis
786	18914956	2289	0.000017	uk.co.metro
787	18914016	4161	0.000009	uk.co.theweek
788	18913930	2204	0.000018	gd.is
789	18913640	4196	0.000009	io.coda
790	18913482	196	0.000128	com.hackerone
791	18912094	1483	0.000023	com.msdn
792	18911500	156	0.000170	org.nginx
793	18911390	3197	0.000011	com.klokantech
794	18911286	1668	0.000022	com.sky
795	18910142	4272	0.000009	de.fernuni-hagen
796	18909064	1832	0.000020	de.hessen
797	18908834	953	0.000034	com.adroll
798	18907960	1986	0.000018	com.windows
799	18906254	4316	0.000008	com.tupalo
800	18904242	218	0.000118	org.icann
801	18904004	1148	0.000028	net.atlassian
802	18903442	4234	0.000009	net.ccm
803	18903372	3808	0.000010	com.oilprice
804	18903128	1956	0.000019	org.khanacademy
805	18902896	4072	0.000009	net.iwpr
806	18902418	324	0.000084	eu.youronlinechoices
807	18902318	3890	0.000009	uk.ac.mmu
808	18901968	1700	0.000021	edu.usc
809	18901674	1146	0.000028	com.playstation
810	18900854	4268	0.000009	uk.ac.ceh
811	18900236	1060	0.000030	com.akamai
812	18897754	2851	0.000013	com.hindustantimes
813	18896694	979	0.000033	gov.fcc
814	18896368	990	0.000032	com.gumroad
815	18896128	4383	0.000008	et.com.google
816	18894652	3952	0.000009	com.theoutline
817	18894586	4348	0.000008	org.cgsociety
818	18892502	4093	0.000009	edu.mtsu
819	18892446	2636	0.000014	com.html5rocks
820	18892222	4578	0.000008	com.blockchair
821	18891572	3603	0.000010	org.spie
822	18891278	1208	0.000026	at.gv.bka
823	18890814	3768	0.000010	uk.co.lrb
824	18888196	410	0.000067	com.heroku
825	18888062	815	0.000036	edu.wisc
826	18887938	1009	0.000031	com.yoast
827	18887842	3807	0.000010	za.co.dailymaverick
828	18885614	1864	0.000019	org.json
829	18885194	2969	0.000012	org.thinkprogress
830	18884928	700	0.000041	com.feedly
831	18883562	4278	0.000009	com.ingress
832	18883012	3706	0.000010	google.design
833	18882952	4666	0.000008	com.bmwblog
834	18881526	3862	0.000009	com.thepetitionsite
835	18881312	3993	0.000009	in.bbc
836	18880860	2237	0.000017	com.w3techs
837	18880532	3257	0.000011	org.carbonbrief
838	18880428	272	0.000098	jp.ne.hatena
839	18880124	2614	0.000014	ru.mk
840	18880064	1924	0.000019	edu.hbs
841	18879486	935	0.000034	com.pingdom
842	18878780	1199	0.000026	com.ycombinator
843	18876422	4065	0.000009	com.gifer
844	18876050	3709	0.000010	uk.org.amnesty
845	18875700	3961	0.000009	com.africanews
846	18875478	4535	0.000008	com.the-dots
847	18875350	992	0.000032	so.notion
848	18869944	3375	0.000011	org.commondreams
849	18869250	4236	0.000009	com.flutterwave
850	18868302	3731	0.000010	org.refworld
851	18866446	3285	0.000011	uk.gov.charitycommission
852	18866244	4396	0.000008	com.newsru
853	18865810	3507	0.000010	uk.org.oxfam
854	18865786	4206	0.000009	uk.org.somersethouse
855	18864590	2484	0.000015	in.scroll
856	18864514	1410	0.000023	com.intuit
857	18864428	4257	0.000009	uk.co.harpercollins
858	18863942	331	0.000084	jp.ameblo
859	18863622	3370	0.000011	ke.co.nation
860	18863580	1091	0.000029	com.insurancejournal
861	18863360	3394	0.000011	com.cbsistatic
862	18863302	2697	0.000014	com.spreaker
863	18862358	2720	0.000014	com.springernature
864	18861824	2228	0.000017	com.firefox
865	18861222	4688	0.000008	co.iglobal
866	18860960	4714	0.000008	io.devdocs
867	18860246	2705	0.000014	com.verywellhealth
868	18860020	538	0.000051	com.booking
869	18859838	535	0.000051	com.gofundme
870	18859798	1234	0.000026	com.indiegogo
871	18859328	4781	0.000008	com.kdpcommunity
872	18858404	2344	0.000016	build.bazel
873	18858016	1119	0.000028	com.foursquare
874	18857452	545	0.000051	com.snapchat
875	18856948	93	0.000390	com.trustpilot
876	18856546	2586	0.000014	com.avast
877	18856160	1813	0.000020	com.pcworld
878	18855278	4149	0.000009	com.hybris
879	18854982	4937	0.000008	com.jetphotos
880	18854014	711	0.000041	com.yandex
881	18853962	1023	0.000031	com.css-tricks
882	18853902	1360	0.000024	org.golang
883	18853900	4168	0.000009	uk.ac.mdx
884	18853596	2673	0.000014	com.flipboard
885	18852988	2848	0.000013	com.discovery
886	18851822	3904	0.000009	at.kleinezeitung
887	18851422	302	0.000090	de.amazon
888	18850596	111	0.000276	me.wa
889	18850558	429	0.000064	com.skype
890	18850294	1134	0.000028	com.scientificamerican
891	18848646	2516	0.000015	org.raspberrypi
892	18847770	4426	0.000008	com.armorgames
893	18847596	1758	0.000020	com.fiverr
894	18847110	721	0.000040	org.iso
895	18846464	2707	0.000014	com.codecademy
896	18843948	3727	0.000010	net.middleeasteye
897	18842876	3057	0.000012	org.man7
898	18841074	4531	0.000008	com.e-estonia
899	18840474	1827	0.000020	fr.blogspot
900	18840282	1047	0.000030	com.huffpost
901	18839944	4725	0.000008	net.gebco
902	18839822	4778	0.000008	com.slite
903	18839776	1799	0.000020	com.visa
904	18839258	722	0.000040	com.newrelic
905	18837214	4163	0.000009	com.cnsnews
906	18836004	807	0.000037	br.com.uol
907	18834494	3388	0.000011	com.lithub
908	18834036	3942	0.000009	net.bostonreview
909	18832052	1022	0.000031	au.com.google
910	18831616	2958	0.000012	com.hackernoon
911	18831490	2802	0.000013	com.unity
912	18831230	249	0.000106	net.2mdn
913	18831090	1224	0.000026	gov.usgs
914	18830714	400	0.000068	com.semrush
915	18829732	3855	0.000009	com.indexmundi
916	18829642	516	0.000053	com.dailymotion
917	18828872	686	0.000042	com.accenture
918	18827314	665	0.000043	org.poynter
919	18826574	2086	0.000018	org.aclu
920	18826336	3974	0.000009	org.jython
921	18825792	1152	0.000027	com.searchengineland
922	18825212	4003	0.000009	com.inthesetimes
923	18824910	1753	0.000020	com.over-blog
924	18824406	518	0.000053	nl.google
925	18824128	3324	0.000011	de.bfarm
926	18823544	1126	0.000028	com.techtarget
927	18823106	4417	0.000008	za.co.ewn
928	18822694	4251	0.000009	uk.co.bristolpost
929	18822100	4799	0.000008	community.studiopress
930	18821800	846	0.000036	gov.justice
931	18820682	1707	0.000021	com.technologyreview
932	18820672	3923	0.000009	com.recyclenow
933	18819750	4293	0.000009	lb.com.dailystar
934	18819328	384	0.000071	com.bitly
935	18819088	4101	0.000009	org.occrp
936	18819058	3786	0.000010	com.theyworkforyou
937	18818798	4022	0.000009	org.ifaw
938	18818648	168	0.000161	com.jimstatic
939	18818242	966	0.000033	sh.brew
940	18818066	4343	0.000008	com.yahoosites
941	18817604	2321	0.000016	com.fool
942	18817360	2281	0.000017	com.pastebin
943	18816928	3607	0.000010	com.gr-assets
944	18815242	3872	0.000009	com.climatechangenews
945	18813886	4771	0.000008	in.ac.iith
946	18813586	702	0.000041	org.plos
947	18812812	4068	0.000009	com.chamberofcommerce
948	18812606	4438	0.000008	us.tuugo
949	18812344	1142	0.000028	com.buzzsprout
950	18812110	1122	0.000028	com.timeanddate
951	18811418	760	0.000038	com.discordapp
952	18811082	1153	0.000027	com.sitepoint
953	18809798	4579	0.000008	com.desmogblog
954	18809638	596	0.000047	com.aliexpress
955	18808906	2522	0.000015	com.sendgrid
956	18807972	4128	0.000009	uk.ac.rcplondon
957	18807934	2004	0.000018	com.ssllabs
958	18807738	3955	0.000009	org.soilassociation
959	18806592	1359	0.000024	com.xkcd
960	18806192	536	0.000051	gov.hhs
961	18805588	3584	0.000010	com.hearstapps
962	18805150	1289	0.000025	com.searchenginejournal
963	18804458	442	0.000062	me.fb
964	18804132	4711	0.000008	net.sott
965	18803628	4594	0.000008	com.gpsvisualizer
966	18803548	213	0.000120	com.discord
967	18802758	1820	0.000020	org.mitre
968	18801916	4744	0.000008	com.natureindex
969	18801374	3499	0.000010	uk.org.rspca
970	18801322	4087	0.000009	org.c2es
971	18801296	3824	0.000010	com.qgiv
972	18801288	3799	0.000010	ug.co.monitor
973	18800492	4596	0.000008	com.lacartes
974	18800384	178	0.000142	com.xing
975	18799966	3973	0.000009	com.svbtle
976	18798944	3875	0.000009	uk.org.savethechildren
977	18798932	4616	0.000008	com.slurl
978	18798332	2466	0.000015	com.sophos
979	18797956	2198	0.000018	com.twilio
980	18797008	4662	0.000008	za.co.moneyweb
981	18796288	4265	0.000009	com.menafn
982	18796048	3821	0.000010	org.usip
983	18795572	4584	0.000008	com.power-technology
984	18795368	4601	0.000008	org.heartland
985	18794834	675	0.000042	com.usnews
986	18794200	2557	0.000015	org.usenix
987	18793962	2899	0.000013	net.privacypolicytemplate
988	18793900	4081	0.000009	org.theecologist
989	18792970	4132	0.000009	org.neweconomics
990	18792680	1290	0.000025	com.netlify
991	18791954	4189	0.000009	com.businessgreen
992	18790752	4611	0.000008	org.monthlyreview
993	18790450	2369	0.000016	uk.ac.ed
994	18790194	2245	0.000017	ch.ethz
995	18789974	267	0.000099	com.nielsen
996	18789302	2674	0.000014	ca.uwaterloo
997	18788810	4284	0.000009	org.unep-wcmc
998	18787702	4042	0.000009	org.ramsar
999	18786952	4384	0.000008	com.googlelabs
1000	18786808	4634	0.000008	org.berkeleyearth

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

September 2021 crawl archive now available

October 4, 2021Sebastian Nagel

The crawl archive for September 2021 is now available! The data was crawled Sept 16 – 29 and contains 2.95 billion web pages or 310 TiB of uncompressed content. It includes page captures of 1.3 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The September crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-39/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-39/segment.paths.gz	100
WARC files	CC-MAIN-2021-39/warc.paths.gz	72000	72.16
WAT files	CC-MAIN-2021-39/wat.paths.gz	72000	19.81
WET files	CC-MAIN-2021-39/wet.paths.gz	72000	8.7
Robots.txt files	CC-MAIN-2021-39/robotstxt.paths.gz	72000	0.15
Non-200 responses files	CC-MAIN-2021-39/non200responses.paths.gz	72000	2.19
URL index files	CC-MAIN-2021-39/cc-index.paths.gz	302	0.22

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-39/. Also the columnar index has been updated to contain this crawl.

July/August 2021 crawl archive now available

August 9, 2021Sebastian Nagel

The crawl archive for July/August 2021 is now available! The data was crawled July 23 – August 6 and contains 3.15 billion web pages or 360 TiB of uncompressed content. It includes page captures of 1 billion new URLs, not visited in any of our prior crawls.

Improvements

Archiving of robots.txt files was improved. Robots.txt files are not archived if

the robots.txt of the target host does not allow it (in case of a HTTP redirect) or
URL filters exclude the entire site, eg. if it’s known ahead that a site does not allow crawling or
the MIME type is not applicable for robots.txt files (eg. HTML, PDF)

More details are found in the corresponding issue report. The change reduces the size of the robots.txt subset (since August 2016) by removing content which should not contained in this dataset.

Archive Location and Download

The July/August crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-31/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-31/segment.paths.gz	100
WARC files	CC-MAIN-2021-31/warc.paths.gz	72000	75.34
WAT files	CC-MAIN-2021-31/wat.paths.gz	72000	21.67
WET files	CC-MAIN-2021-31/wet.paths.gz	72000	9.43
Robots.txt files	CC-MAIN-2021-31/robotstxt.paths.gz	72000	0.14
Non-200 responses files	CC-MAIN-2021-31/non200responses.paths.gz	72000	1.98
URL index files	CC-MAIN-2021-31/cc-index.paths.gz	302	0.23

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-31/. Also the columnar index has been updated to contain this crawl.

June 2021 crawl archive now available

June 28, 2021Sebastian Nagel

The crawl archive for June 2021 is now available! The data was crawled June 12 – 25 and contains 2.45 billion web pages or 260 TiB of uncompressed content. It includes page captures of 1.3 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The June crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-25/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-25/segment.paths.gz	100
WARC files	CC-MAIN-2021-25/warc.paths.gz	64000	57.9
WAT files	CC-MAIN-2021-25/wat.paths.gz	64000	16.54
WET files	CC-MAIN-2021-25/wet.paths.gz	64000	7.24
Robots.txt files	CC-MAIN-2021-25/robotstxt.paths.gz	64000	0.16
Non-200 responses files	CC-MAIN-2021-25/non200responses.paths.gz	64000	1.76
URL index files	CC-MAIN-2021-25/cc-index.paths.gz	302	0.19

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-25/. Also the columnar index has been updated to contain this crawl.

Host- and Domain-Level Web Graphs February/March, April and May 2021

May 31, 2021Sebastian Nagel

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of February/March, April and May 2021. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases. You may also visit the projects cc-webgraph and cc-pyspark which include all scripts and tools required to construct the graphs. Instructions to explore the graphs in the webgraph format are given in our collection of webgraph notebooks.

What’s new?

The host-level graph now includes all hosts visited by the crawler even if there is no link pointing to the host and all visited URLs of a host failed (HTTP 404 and other error codes) or the host’s robots.txt does not allow crawling. Note that the links leading to these hosts may have been found in a prior crawl, not in one of the 3 crawls used to build this web graph.

Host-level graph

The graph consists of 515 million nodes and 2.82 billion edges. Both hyperlinks and HTTP redirects and link headers are used as edges to span up the graph. All types of links are included, including pure “technical” ones pointing to images, JavaScript libraries, web fonts, etc. However, only host names with a valid IANA TLD are used. Consequently, URLs with an IP address as host component are not taken into account for building the host-level graph.

There are 452 million dangling nodes (87.9%) and the largest strongly connected component contains 45.2 million (8.8%) nodes. Dangling nodes stem from

hosts that have not been crawled, yet are pointed to from a link on a crawled page
hosts without any links pointing to a different host name
or hosts which did only return an error page (eg. HTTP 404)

Host names in the graph are in reverse domain name notation and a leading www. is stripped: www.subdomain.example.com becomes com.example.subdomain.

You can download the graph and the ranks of all 515 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2021-feb-apr-may/host/. Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2021-feb-apr-may/host/ as prefix to access the files from everywhere.

Please note that the text representation of the host-level graph is shipped in 72 gzip-compressed files listed in two path listings – one for the nodes (vertices), one for the edges (arcs). First, download the paths listing and decompress it using “gzip”. By adding the prefix s3://commoncrawl/ or https://data.commoncrawl.org/ to each line in the path listing you get the list of URLs to download the entire graph.

Download files of the Common Crawl Feb/Apr/May 2021 host-level webgraph

Size	File	Description
3.31 GB	cc-main-2021-feb-apr-may-host-vertices.paths.gz	nodes ⟨id, rev host⟩, paths of 24 vertices files
12.94 GB	cc-main-2021-feb-apr-may-host-edges.paths.gz	edges ⟨from_id, to_id⟩, paths of 48 edges files

5.57 GB	cc-main-2021-feb-apr-may-host.graph	graph in BVGraph format
2 kB	cc-main-2021-feb-apr-may-host.properties

6.22 GB	cc-main-2021-feb-apr-may-host-t.graph	transpose of the graph (outlinks inverted to inlinks)
2 kB	cc-main-2021-feb-apr-may-host-t.properties

1 kB	cc-main-2021-feb-apr-may-host.stats	WebGraph statistics

7.69 GB	cc-main-2021-feb-apr-may-host-ranks.txt.gz	harmonic centrality and pagerank

Domain-level graph

The domain graph is built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.

The domain-level graph has 88 million nodes and 1.58 billion edges. 50% or 44 million nodes are dangling nodes, the largest strongly connected component covers 34 million or 39% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2021-feb-apr-may/domain/ resp. https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2021-feb-apr-may/domain/.

Download files of the Common Crawl Feb/Apr/May 2021 domain-level webgraph

Size	File	Description
0.61 GB	cc-main-2021-feb-apr-may-domain-vertices.txt.gz	nodes ⟨id, rev domain, num hosts⟩
6.37 GB	cc-main-2021-feb-apr-may-domain-edges.txt.gz	edges ⟨from_id, to_id⟩

3.58 GB	cc-main-2021-feb-apr-may-domain.graph	graph in BVGraph format
2 kB	cc-main-2021-feb-apr-may-domain.properties

3.42 GB	cc-main-2021-feb-apr-may-domain-t.graph	transpose of the graph
2 kB	cc-main-2021-feb-apr-may-domain-t.properties

1 kB	cc-main-2021-feb-apr-may-domain.stats	WebGraph statistics

1.89 GB	cc-main-2021-feb-apr-may-domain-ranks.txt.gz	harmonic centrality and pagerank

Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 88 million domain ranks is available for download.

Top 1000 domains ranked by harmonic centrality (Feb/Apr/May 2021)

harmonic centrality rank	hc value	page rank	page rank value	reversed domain name
1	31920934	1	0.017627	com.googleapis
2	31032784	3	0.013762	com.facebook
3	29681304	2	0.013832	com.google
4	27101692	4	0.007844	com.twitter
5	26954660	5	0.007519	org.w
6	26886624	7	0.006967	com.youtube
7	25515850	8	0.005718	com.instagram
8	25031490	6	0.007143	com.googletagmanager
9	24396116	9	0.005506	org.gmpg
10	23807122	12	0.003347	com.linkedin
11	22970992	13	0.003048	com.gstatic
12	22854052	10	0.003951	com.cloudflare
13	22698594	19	0.001914	com.gravatar
14	22504168	14	0.002908	org.wordpress
15	22434542	22	0.001564	com.pinterest
16	22100870	25	0.001270	org.wikipedia
17	21950578	17	0.002031	com.wordpress
18	21940826	18	0.001958	com.apple
19	21766696	15	0.002258	com.bootstrapcdn
20	21762964	30	0.001174	com.vimeo
21	21722198	38	0.000914	be.youtu
22	21556142	21	0.001842	com.jquery
23	21478118	29	0.001182	com.microsoft
24	21432212	53	0.000703	com.blogspot
25	21354260	35	0.001025	com.amazonaws
26	21337432	44	0.000765	com.amazon
27	21320702	43	0.000789	gl.goo
28	21170722	62	0.000600	ly.bit
29	21149628	99	0.000409	com.tumblr
30	21148242	50	0.000739	com.wp
31	21136818	45	0.000758	org.mozilla
32	21110018	57	0.000689	eu.europa
33	21104262	20	0.001894	com.adobe
34	21048760	16	0.002200	com.github
35	21040284	34	0.001026	com.google-analytics
36	21027350	36	0.001015	net.jsdelivr
37	20998320	27	0.001218	com.wixstatic
38	20995232	31	0.001119	net.cloudfront
39	20946148	47	0.000744	com.flickr
40	20913104	107	0.000338	com.yahoo
41	20851316	83	0.000436	com.googleusercontent
42	20843068	37	0.000929	io.github
43	20840670	111	0.000317	com.reddit
44	20834398	58	0.000677	com.paypal
45	20816886	23	0.001554	com.fontawesome
46	20773582	103	0.000368	com.weebly
47	20764576	79	0.000455	com.medium
48	20764512	33	0.001035	com.googlesyndication
49	20757582	32	0.001118	ru.yandex
50	20741944	48	0.000743	com.whatsapp
51	20708152	68	0.000520	org.w3
52	20705826	132	0.000240	com.nytimes
53	20696906	59	0.000673	co.t
54	20678088	102	0.000375	org.creativecommons
55	20675822	115	0.000290	com.soundcloud
56	20644978	60	0.000624	org.schema
57	20627114	74	0.000479	com.shopify
58	20621162	66	0.000543	com.vk
59	20604726	181	0.000149	org.wikimedia
60	20604724	147	0.000204	com.dropbox
61	20579720	55	0.000702	com.addthis
62	20572950	138	0.000211	org.archive
63	20570610	198	0.000133	com.cnn
64	20558114	152	0.000187	gov.cdc
65	20550306	80	0.000446	me.wp
66	20538816	193	0.000136	com.imgur
67	20530078	49	0.000740	net.doubleclick
68	20512294	199	0.000133	uk.co.bbc
69	20505964	200	0.000133	net.slideshare
70	20499864	171	0.000155	com.theguardian
71	20489756	158	0.000175	int.who
72	20482256	120	0.000263	com.spotify
73	20481118	175	0.000151	com.bing
74	20478320	213	0.000124	com.businessinsider
75	20477478	253	0.000104	com.bloomberg
76	20477300	144	0.000206	gov.nih
77	20473648	46	0.000748	com.macromedia
78	20440520	254	0.000103	com.wsj
79	20434320	224	0.000118	edu.stanford
80	20419762	41	0.000847	net.fbcdn
81	20417930	39	0.000885	org.apache
82	20409636	157	0.000175	org.ietf
83	20397792	90	0.000420	com.list-manage
84	20395594	368	0.000071	com.googleblog
85	20395350	217	0.000123	com.stackoverflow
86	20393172	170	0.000155	com.giphy
87	20391226	314	0.000085	edu.mit
88	20381948	223	0.000118	com.washingtonpost
89	20372602	134	0.000232	com.ytimg
90	20363592	362	0.000073	com.appspot
91	20360236	351	0.000076	com.theverge
92	20359610	286	0.000093	com.bbc
93	20358870	396	0.000067	uk.co.telegraph
94	20356036	499	0.000056	edu.berkeley
95	20348048	266	0.000101	edu.harvard
96	20346012	330	0.000080	com.go
97	20341676	237	0.000112	com.office
98	20338710	145	0.000206	us.zoom
99	20335782	247	0.000109	com.android
100	20335366	327	0.000082	com.wired
101	20334160	288	0.000092	com.techcrunch
102	20331782	238	0.000111	com.oracle
103	20323638	547	0.000051	com.livejournal
104	20296670	164	0.000170	com.issuu
105	20295840	296	0.000090	com.cnbc
106	20292146	211	0.000124	gov.ca
107	20291754	402	0.000066	com.ted
108	20288380	379	0.000069	gov.nasa
109	20283426	149	0.000195	com.forbes
110	20283050	148	0.000199	com.wixsite
111	20282972	151	0.000192	com.npmjs
112	20282524	518	0.000054	com.zdnet
113	20279656	447	0.000062	com.msn
114	20277752	292	0.000091	com.reuters
115	20275540	350	0.000076	com.nature
116	20273474	78	0.000459	com.godaddy
117	20271718	371	0.000070	com.myspace
118	20270494	222	0.000119	com.etsy
119	20268832	321	0.000084	com.prnewswire
120	20255726	209	0.000125	org.ampproject
121	20252386	407	0.000065	org.arxiv
122	20252292	312	0.000085	org.npr
123	20252218	263	0.000101	com.sciencedirect
124	20248804	98	0.000410	com.unpkg
125	20246402	265	0.000101	com.example
126	20245616	67	0.000524	net.akamaihd
127	20237056	215	0.000123	com.eventbrite
128	20234532	367	0.000072	org.hbr
129	20232338	176	0.000151	com.blogger
130	20231658	127	0.000247	org.networkadvertising
131	20231552	399	0.000066	com.latimes
132	20228690	268	0.000101	org.acm
133	20223242	338	0.000079	com.statista
134	20209434	389	0.000068	com.fastcompany
135	20205848	660	0.000043	com.economist
136	20202482	343	0.000078	com.time
137	20202452	226	0.000117	com.twimg
138	20201902	679	0.000042	edu.upenn
139	20201530	550	0.000050	edu.yale
140	20200842	258	0.000102	com.githubusercontent
141	20191272	474	0.000060	com.steampowered
142	20189824	143	0.000206	com.opera
143	20188620	444	0.000062	uk.co.dailymail
144	20188486	353	0.000076	com.springer
145	20186806	576	0.000047	com.scribd
146	20184784	780	0.000041	edu.columbia
147	20180100	535	0.000052	org.chromium
148	20175876	591	0.000046	me.about
149	20175732	604	0.000046	google.blog
150	20175284	285	0.000094	com.squarespace
151	20174050	335	0.000079	com.huffingtonpost
152	20171356	431	0.000063	com.nationalgeographic
153	20168788	221	0.000119	uk.co.google
154	20165372	208	0.000125	com.unsplash
155	20163580	388	0.000068	com.w3schools
156	20158956	339	0.000079	com.dribbble
157	20154786	340	0.000079	com.tiktok
158	20153356	293	0.000091	org.un
159	20137924	794	0.000040	com.qz
160	20133814	248	0.000108	com.bandcamp
161	20129598	485	0.000058	edu.cornell
162	20125954	821	0.000039	edu.umich
163	20121120	119	0.000267	com.ft
164	20115342	435	0.000063	com.theatlantic
165	20111028	966	0.000033	edu.princeton
166	20110808	341	0.000078	com.usatoday
167	20105556	786	0.000040	com.evernote
168	20105482	133	0.000235	info.aboutads
169	20104810	408	0.000065	com.meetup
170	20102638	438	0.000062	com.goodreads
171	20100894	625	0.000045	org.ieee
172	20098972	878	0.000036	com.slate
173	20097870	677	0.000042	com.mysql
174	20097656	453	0.000061	com.patreon
175	20097530	137	0.000216	me.t
176	20095600	515	0.000055	com.cbsnews
177	20084204	656	0.000043	com.docker
178	20083336	291	0.000092	com.wiley
179	20082520	480	0.000059	gov.usda
180	20080664	454	0.000061	com.dailymotion
181	20078818	817	0.000039	edu.washington
182	20077160	493	0.000057	com.withgoogle
183	20075064	523	0.000054	io.readthedocs
184	20071014	644	0.000044	com.marketwatch
185	20065010	650	0.000043	uk.co.blogspot
186	20062734	868	0.000037	com.shutterstock
187	20062652	54	0.000703	com.fb
188	20059664	497	0.000056	uk.co.independent
189	20056344	76	0.000467	com.wix
190	20055932	811	0.000039	org.cambridge
191	20051844	559	0.000049	com.pexels
192	20048576	779	0.000041	org.sciencemag
193	20048004	592	0.000046	com.buzzfeed
194	20044248	819	0.000039	com.stackexchange
195	20043466	179	0.000149	ru.mail
196	20043446	844	0.000038	com.webs
197	20043074	573	0.000048	com.git-scm
198	20040208	464	0.000060	com.inc
199	20037354	272	0.000100	net.behance
200	20029744	425	0.000063	gov.whitehouse
201	20025342	832	0.000038	com.apnews
202	20023518	769	0.000041	com.vox
203	20022030	1365	0.000024	uk.co.thesun
204	20018548	274	0.000098	com.outlook
205	20018318	772	0.000041	org.bitbucket
206	20017276	40	0.000871	com.qq
207	20014872	244	0.000110	org.doi
208	20012082	812	0.000039	uk.ac.cam
209	20011998	255	0.000103	com.disqus
210	20007312	236	0.000112	com.feedburner
211	20005630	670	0.000043	org.worldbank
212	20001230	584	0.000047	org.unicef
213	20000932	419	0.000064	com.mozilla
214	19999740	593	0.000046	co.ibb
215	19999080	26	0.001261	io.polyfill
216	19997928	525	0.000054	com.booking
217	19993488	42	0.000808	com.baidu
218	19989784	260	0.000101	com.cloudinary
219	19985856	289	0.000092	com.tinyurl
220	19983980	345	0.000077	com.ibm
221	19983022	1163	0.000027	com.speakerdeck
222	19982506	597	0.000046	gov.noaa
223	19978206	612	0.000045	ee.linktr
224	19977310	569	0.000048	com.psychologytoday
225	19973710	531	0.000053	gov.loc
226	19972920	400	0.000066	com.getpocket
227	19972760	1041	0.000031	edu.utexas
228	19971794	320	0.000084	org.pewresearch
229	19971310	1366	0.000024	edu.rutgers
230	19970894	551	0.000050	com.sagepub
231	19970200	309	0.000087	com.nbcnews
232	19967962	1134	0.000028	org.eclipse
233	19965586	648	0.000043	com.trello
234	19964280	326	0.000082	net.windows
235	19964194	384	0.000068	com.quora
236	19961430	600	0.000046	net.azurewebsites
237	19959910	275	0.000098	gov.ftc
238	19955938	1057	0.000030	edu.uchicago
239	19953308	311	0.000086	com.netdna-ssl
240	19951960	782	0.000041	org.semver
241	19951286	124	0.000252	com.mailchimp
242	19950294	436	0.000063	com.nypost
243	19949296	1195	0.000027	com.hatenablog
244	19947142	652	0.000043	com.newyorker
245	19943938	985	0.000033	uk.co.guardian
246	19943564	590	0.000046	com.usnews
247	19940498	220	0.000119	tv.twitch
248	19939738	784	0.000041	au.net.abc
249	19938820	166	0.000167	com.amazon-adsystem
250	19936308	1278	0.000025	com.vogue
251	19935466	230	0.000113	com.wpengine
252	19934098	106	0.000338	com.stripe
253	19933266	1261	0.000025	org.kernel
254	19929738	941	0.000034	com.politico
255	19926416	1193	0.000027	org.unicode
256	19925602	580	0.000047	org.eff
257	19925174	541	0.000051	br.com.uol
258	19924806	852	0.000037	com.about
259	19923644	1358	0.000024	edu.hbs
260	19923600	954	0.000034	com.dropboxusercontent
261	19923464	911	0.000035	edu.jhu
262	19922062	993	0.000032	co.elastic
263	19921888	913	0.000035	com.steamcommunity
264	19920150	1971	0.000018	com.googlesource
265	19919760	522	0.000054	com.tandfonline
266	19918010	277	0.000097	com.criteo
267	19915708	552	0.000050	org.pbs
268	19912986	1106	0.000029	edu.umd
269	19912224	64	0.000549	co.g
270	19908340	865	0.000037	com.foxnews
271	19907456	123	0.000261	com.sharethis
272	19904178	1027	0.000031	com.rollingstone
273	19903082	228	0.000115	com.imdb
274	19902774	977	0.000033	com.scientificamerican
275	19901940	1392	0.000023	com.urbandictionary
276	19900876	775	0.000041	uk.ac.ox
277	19900406	391	0.000067	com.arcgis
278	19898520	2016	0.000018	com.lego
279	19898420	251	0.000107	page.g
280	19898318	631	0.000044	gov.census
281	19890056	530	0.000053	com.oup
282	19887968	346	0.000077	com.optimizely
283	19887424	582	0.000047	com.indiatimes
284	19887194	376	0.000069	com.cnet
285	19884024	422	0.000064	com.wufoo
286	19882930	704	0.000042	uk.co.eventbrite
287	19882806	421	0.000064	com.bigcommerce
288	19880306	1350	0.000024	ca.blogspot
289	19879016	833	0.000038	org.fao
290	19878732	908	0.000035	com.jetbrains
291	19871044	1467	0.000022	ca.ubc
292	19867650	1938	0.000018	com.warnerbros
293	19866012	446	0.000062	org.d3js
294	19865518	946	0.000034	org.greenpeace
295	19864632	206	0.000127	net.sourceforge
296	19863450	323	0.000083	fr.google
297	19862916	1279	0.000025	com.history
298	19861806	851	0.000038	com.gumroad
299	19861750	919	0.000035	com.chicagotribune
300	19859844	636	0.000044	gov.archives
301	19858902	284	0.000095	com.googlecode
302	19853502	342	0.000078	com.slack
303	19851932	229	0.000114	com.eepurl
304	19845626	114	0.000292	com.paypalobjects
305	19841702	927	0.000035	com.sap
306	19839830	153	0.000180	com.addtoany
307	19837466	290	0.000092	com.typepad
308	19834082	1562	0.000021	de.mpg
309	19830054	664	0.000043	com.pinimg
310	19828148	282	0.000095	com.calendly
311	19827530	491	0.000057	gov.epa
312	19825756	354	0.000076	com.proofpoint
313	19821128	1430	0.000023	ch.ethz
314	19821094	1028	0.000031	com.500px
315	19820554	1732	0.000019	com.diigo
316	19820398	334	0.000079	com.live
317	19820034	1277	0.000025	org.postgresql
318	19818544	1257	0.000025	org.wiktionary
319	19817910	1274	0.000025	org.aclu
320	19817698	981	0.000033	edu.si
321	19816586	1394	0.000023	edu.msu
322	19816210	1029	0.000031	com.thehill
323	19814936	890	0.000036	de.spiegel
324	19813172	916	0.000035	com.huffpost
325	19811282	472	0.000060	gov.hhs
326	19809240	1114	0.000028	com.scmp
327	19806650	73	0.000484	me.fb
328	19806306	764	0.000042	org.change
329	19805070	378	0.000069	com.sohu
330	19804336	1329	0.000024	edu.illinois
331	19804164	185	0.000147	com.xing
332	19801192	1323	0.000024	org.tensorflow
333	19801086	1008	0.000032	com.ssrn
334	19800184	162	0.000171	com.zendesk
335	19798428	904	0.000035	com.netlify
336	19797294	508	0.000056	com.squareup
337	19797020	1352	0.000024	com.sky
338	19794400	196	0.000134	org.iana
339	19792714	1078	0.000029	uk.co.thetimes
340	19792494	847	0.000038	gov.congress
341	19788704	809	0.000039	org.pypi
342	19783878	1422	0.000023	cn.com.chinadaily
343	19781142	972	0.000033	edu.academia
344	19780974	456	0.000061	com.kickstarter
345	19780084	802	0.000040	gov.senate
346	19779128	2415	0.000015	org.pydata
347	19778124	1140	0.000027	org.semanticscholar
348	19775716	620	0.000045	site.business
349	19775012	1275	0.000025	com.over-blog
350	19774866	792	0.000040	org.oecd
351	19774846	1660	0.000020	org.phys
352	19774334	999	0.000032	com.yarnpkg
353	19772248	816	0.000039	com.deviantart
354	19770936	1084	0.000029	uk.co.mirror
355	19770522	187	0.000145	com.rawgit
356	19770114	1315	0.000024	com.axios
357	19769700	623	0.000045	gov.house
358	19768998	894	0.000036	com.discordapp
359	19768866	880	0.000036	com.sciencedaily
360	19766292	511	0.000055	com.gmail
361	19765678	423	0.000064	com.technorati
362	19763944	216	0.000123	com.hubspot
363	19761638	1433	0.000023	com.unity3d
364	19760768	2137	0.000017	org.threejs
365	19760238	1364	0.000024	com.aljazeera
366	19759580	245	0.000109	org.nodejs
367	19758846	896	0.000036	com.bmj
368	19755564	261	0.000101	com.ebay
369	19755198	1197	0.000026	au.com.smh
370	19753628	234	0.000113	org.gnu
371	19751964	1516	0.000021	edu.osu
372	19751362	1025	0.000031	int.coe
373	19750302	994	0.000032	com.britannica
374	19748408	1312	0.000024	edu.gatech
375	19746818	2691	0.000013	com.openai
376	19744370	495	0.000056	org.openstreetmap
377	19743086	437	0.000062	com.ssl-images-amazon
378	19741582	791	0.000040	br.com.google
379	19741030	855	0.000037	ca.cbc
380	19740484	869	0.000037	com.theconversation
381	19739852	2582	0.000014	edu.toronto
382	19738652	1044	0.000031	gov.usgs
383	19738306	1556	0.000021	com.newscientist
384	19736226	301	0.000088	net.themeforest
385	19735698	605	0.000046	com.udacity
386	19735668	473	0.000060	edu.nyu
387	19734084	1716	0.000019	edu.ucsc
388	19723708	1700	0.000020	org.emojipedia
389	19722194	2068	0.000017	it.scoop
390	19722024	2754	0.000013	com.slides
391	19721872	1459	0.000022	ca.sfu
392	19720004	845	0.000038	au.gov.nsw
393	19717908	1903	0.000019	org.propublica
394	19717586	1386	0.000023	com.firebaseapp
395	19716094	2247	0.000016	com.skyrock
396	19710516	776	0.000041	com.freepik
397	19707962	97	0.000412	net.facebook
398	19704900	1454	0.000022	com.penguinrandomhouse
399	19703572	195	0.000135	org.bbb
400	19703432	1934	0.000018	jp.co.japantimes
401	19701030	1762	0.000019	com.itv
402	19700818	82	0.000437	net.jsfiddle
403	19700616	1985	0.000018	org.maven
404	19699746	2370	0.000015	com.deepmind
405	19697844	617	0.000045	com.healthline
406	19695324	506	0.000056	de.gesetze-im-internet
407	19694720	465	0.000060	org.python
408	19694428	2331	0.000015	com.mystrikingly
409	19691536	884	0.000036	gov.dhs
410	19688238	1233	0.000026	com.wikia
411	19685986	2090	0.000017	org.sqlite
412	19682976	1544	0.000021	ms.1drv
413	19682094	178	0.000150	com.salesforce
414	19679914	322	0.000084	net.php
415	19671484	324	0.000083	com.surveymonkey
416	19670962	634	0.000044	com.mashable
417	19670338	1628	0.000020	com.motherjones
418	19668724	139	0.000211	com.weibo
419	19668554	2453	0.000014	com.fastcodesign
420	19667444	1506	0.000021	com.flipboard
421	19666746	2435	0.000015	edu.byu
422	19665482	1748	0.000019	edu.cuny
423	19664886	317	0.000085	ru.ok
424	19662618	287	0.000092	net.azureedge
425	19662108	1339	0.000024	com.thedailybeast
426	19659672	246	0.000109	org.aboutcookies
427	19658838	2283	0.000015	com.shutterfly
428	19656108	1413	0.000023	com.reverbnation
429	19655722	2666	0.000013	io.material
430	19655254	537	0.000052	io.codepen
431	19652776	1296	0.000025	com.dw
432	19651986	125	0.000250	com.youtube-nocookie
433	19650416	1724	0.000019	com.esri
434	19650188	490	0.000057	fr.free
435	19648416	1509	0.000021	com.substack
436	19647438	561	0.000049	com.matterport
437	19646584	1956	0.000018	com.hindustantimes
438	19645830	1909	0.000019	com.insider
439	19642342	2110	0.000017	edu.oregonstate
440	19641814	2390	0.000015	org.wikibooks
441	19640838	891	0.000036	int.wipo
442	19640244	2820	0.000013	org.aclweb
443	19639226	607	0.000045	gov.state
444	19638894	2366	0.000015	com.wattpad
445	19638652	160	0.000172	gle.forms
446	19636692	1052	0.000030	org.jstor
447	19636398	1951	0.000018	com.channel4
448	19636126	1752	0.000019	edu.ucsb
449	19635942	1320	0.000024	gov.supremecourt
450	19633994	56	0.000697	com.googleadservices
451	19631760	2441	0.000015	at.ac.univie
452	19629096	2924	0.000013	com.pbase
453	19626572	278	0.000097	uk.org.ico
454	19624802	639	0.000044	com.licdn
455	19623422	1518	0.000021	ch.ipcc
456	19621874	937	0.000034	com.gallup
457	19621780	496	0.000056	com.herokuapp
458	19618584	1141	0.000027	edu.brookings
459	19617388	963	0.000033	edu.psu
460	19616790	1333	0.000024	mil.army
461	19616626	434	0.000063	com.rackcdn
462	19614484	385	0.000068	com.atlassian
463	19611760	1226	0.000026	com.smashingmagazine
464	19609634	2227	0.000016	blog.home
465	19608450	1362	0.000024	gov.defense
466	19607698	1131	0.000028	com.photoshelter
467	19607464	483	0.000058	net.imgix
468	19607012	182	0.000149	jp.co.yahoo
469	19605316	2284	0.000015	com.contently
470	19602040	826	0.000039	com.oreilly
471	19597708	1174	0.000027	com.mediafire
472	19595596	2117	0.000017	com.thecut
473	19594604	1960	0.000018	google.ai
474	19594568	3151	0.000012	cc.uxdesign
475	19594280	3161	0.000012	edu.uvm
476	19594100	520	0.000054	edu.cmu
477	19593086	3137	0.000012	com.instapaper
478	19591090	1591	0.000020	com.thestar
479	19588378	369	0.000071	net.researchgate
480	19587214	3502	0.000011	com.raywenderlich
481	19587008	527	0.000053	com.thinkwithgoogle
482	19584868	2149	0.000016	fr.liberation
483	19582230	109	0.000336	de.google
484	19581418	1574	0.000021	com.buzzfeednews
485	19577648	767	0.000041	org.worldwildlife
486	19576662	1013	0.000032	com.ecwid
487	19576118	1477	0.000022	com.findlaw
488	19574804	1012	0.000032	com.thelancet
489	19573936	774	0.000041	com.vice
490	19573506	813	0.000039	gov.nist
491	19572872	1964	0.000018	org.google
492	19572508	1531	0.000021	org.hrw
493	19570410	765	0.000042	com.intel
494	19568238	2695	0.000013	uk.co.ibtimes
495	19567790	2372	0.000015	com.oprah
496	19567558	87	0.000428	com.workplace
497	19567194	3329	0.000011	com.pearltrees
498	19567174	2103	0.000017	com.voanews
499	19566762	965	0.000033	com.engadget
500	19566188	126	0.000247	com.statcounter
501	19564772	3365	0.000011	org.edublogs
502	19563980	1260	0.000025	org.aiga
503	19562828	1031	0.000031	de.stern
504	19562068	1583	0.000020	fr.francetvinfo
505	19560196	2620	0.000014	com.hm
506	19559342	315	0.000085	org.drupal
507	19559132	3736	0.000010	fr.unblog
508	19558786	747	0.000042	com.canva
509	19558362	2870	0.000013	edu.ucf
510	19558064	3204	0.000012	ph.telegra
511	19557534	926	0.000035	uk.co.pinterest
512	19557072	2402	0.000015	edu.kit
513	19556358	544	0.000051	it.placehold
514	19555528	2219	0.000016	net.corporate-ir
515	19553910	2768	0.000013	co.ello
516	19553426	881	0.000036	com.arstechnica
517	19553018	1449	0.000022	com.livescience
518	19550968	2150	0.000016	com.gq
519	19550836	1953	0.000018	uk.gov.tfl
520	19550254	210	0.000125	com.iubenda
521	19550042	533	0.000053	com.pixabay
522	19548328	1408	0.000023	org.undp
523	19547668	807	0.000039	ca.amazon
524	19547226	1020	0.000031	it.smarturl
525	19547032	2645	0.000014	org.icrc
526	19546934	2447	0.000015	com.webbyawards
527	19545564	2423	0.000015	uk.ac.kcl
528	19545554	949	0.000034	edu.ucla
529	19544462	1444	0.000022	link.page
530	19543968	2861	0.000013	com.dummies
531	19541366	1581	0.000021	org.ocks
532	19540748	65	0.000544	net.typekit
533	19540022	1122	0.000028	org.ilo
534	19538882	2564	0.000014	com.depositphotos
535	19538866	2502	0.000014	com.unilever
536	19536950	1348	0.000024	org.acs
537	19536262	81	0.000440	com.livestream
538	19535098	2672	0.000013	org.rsf
539	19535076	489	0.000057	com.adweek
540	19534044	2050	0.000017	com.msnbc
541	19530220	2509	0.000014	com.slidesharecdn
542	19530084	2035	0.000018	com.chronicle
543	19529836	3088	0.000012	com.bepress
544	19529580	2571	0.000014	com.biography
545	19529322	3384	0.000011	tl.de
546	19527886	332	0.000079	com.typeform
547	19526428	2185	0.000016	com.newrepublic
548	19525400	2303	0.000015	com.thoughtco
549	19523856	606	0.000045	com.samsung
550	19523112	1100	0.000029	org.ohchr
551	19522668	790	0.000040	com.fiverr
552	19521518	1743	0.000019	io.gitlab
553	19521240	121	0.000262	com.jimdo
554	19520292	1157	0.000027	com.thenextweb
555	19520070	2009	0.000018	fr.orange
556	19519618	3272	0.000012	net.openreview
557	19518936	2294	0.000015	com.channelnewsasia
558	19517090	1283	0.000025	org.aarp
559	19516918	2634	0.000014	org.pewsocialtrends
560	19516476	1998	0.000018	com.straitstimes
561	19513936	2310	0.000015	edu.nd
562	19510956	2099	0.000017	com.dallasnews
563	19510732	2130	0.000017	de.br
564	19508818	2278	0.000015	org.fas
565	19508000	1297	0.000024	org.altervista
566	19507978	256	0.000103	uk.co.amazon
567	19507290	219	0.000121	to.amzn
568	19506624	2835	0.000013	com.thejakartapost
569	19505128	2211	0.000016	gov.lbl
570	19504556	1610	0.000020	de.berlin
571	19504362	1086	0.000029	com.popularmechanics
572	19503706	2743	0.000013	uk.ac.leeds
573	19503644	459	0.000061	com.staticflickr
574	19503210	3397	0.000011	org.neocities
575	19502358	2996	0.000012	org.vim
576	19502186	2883	0.000013	org.globalcitizen
577	19499450	572	0.000048	com.deloitte
578	19499392	922	0.000035	com.zoho
579	19498964	233	0.000113	io.shields
580	19498936	2328	0.000015	com.indianexpress
581	19498902	3889	0.000010	com.stratechery
582	19497728	2819	0.000013	app.web
583	19496358	3386	0.000011	org.zotero
584	19493624	2939	0.000013	uk.gov.scotland
585	19493314	567	0.000048	com.photobucket
586	19491524	3756	0.000010	com.bravesites
587	19490552	1464	0.000022	org.iea
588	19489976	432	0.000063	com.hp
589	19489954	2713	0.000013	uk.co.timesonline
590	19489478	365	0.000073	com.quantserve
591	19489336	404	0.000066	com.digg
592	19486660	560	0.000049	com.cisco
593	19486618	1155	0.000027	uk.parliament
594	19485014	2914	0.000013	com.nwsource
595	19485012	2362	0.000015	com.fineartamerica
596	19484598	267	0.000101	com.onesignal
597	19484238	2234	0.000016	com.foreignpolicy
598	19484200	798	0.000040	org.weforum
599	19483398	2990	0.000012	com.thoughtworks
600	19483202	1548	0.000021	com.treehugger
601	19482398	307	0.000087	com.aliyuncs
602	19482224	602	0.000046	org.js
603	19480232	1527	0.000021	gov.uscis
604	19479040	3256	0.000012	uk.ac.city
605	19477476	2077	0.000017	com.washingtontimes
606	19477198	3504	0.000011	com.mariadb
607	19476316	2565	0.000014	org.oas
608	19475236	417	0.000065	com.gitlab
609	19472258	2584	0.000014	com.mathworks
610	19471752	2830	0.000013	com.dezeen
611	19471284	835	0.000038	com.investopedia
612	19470638	2497	0.000014	uk.co.yougov
613	19469316	2934	0.000013	org.heritage
614	19469308	614	0.000045	com.netflix
615	19466252	3281	0.000011	com.shell
616	19465388	2540	0.000014	fr.paris
617	19464956	448	0.000061	gov.irs
618	19462732	4088	0.000009	tl.page
619	19461330	1361	0.000024	com.upwork
620	19461170	462	0.000061	com.sxsw
621	19460914	1255	0.000025	com.digitaloceanspaces
622	19460548	4091	0.000009	com.jigsy
623	19460066	861	0.000037	com.venturebeat
624	19458418	1215	0.000026	com.dell
625	19457348	1016	0.000031	gov.fcc
626	19456828	3229	0.000012	uk.co.walesonline
627	19456346	2961	0.000013	org.project-syndicate
628	19455696	2024	0.000018	com.fivethirtyeight
629	19455242	920	0.000035	fm.last
630	19455056	2086	0.000017	info.worldometers
631	19454252	931	0.000034	org.mediawiki
632	19453670	2377	0.000015	ly.rebrand
633	19453158	4077	0.000009	net.myanimelist
634	19452824	2075	0.000017	cn.gov.fmprc
635	19452012	1502	0.000021	org.amnesty
636	19450548	349	0.000077	com.adnxs
637	19449350	1945	0.000018	com.justia
638	19448712	4019	0.000009	edu.usfca
639	19448298	2705	0.000013	com.monday
640	19446576	1515	0.000021	ca.bc.gov
641	19446486	943	0.000034	org.reactjs
642	19446126	2285	0.000015	net.openid
643	19445904	383	0.000068	com.newrelic
644	19445366	1363	0.000024	com.imageshack
645	19445144	3568	0.000010	org.globalnetworkinitiative
646	19443940	2549	0.000014	com.kaggle
647	19443562	3693	0.000010	com.doodlekit
648	19439792	259	0.000102	com.getbootstrap
649	19438670	2831	0.000013	uk.co.inews
650	19438312	3129	0.000012	com.bangkokpost
651	19438230	409	0.000065	com.force
652	19437908	2107	0.000017	uk.ac.imperial
653	19435434	4629	0.000008	net.vingle
654	19434150	1982	0.000018	be.kuleuven
655	19434066	3530	0.000011	com.intensedebate
656	19432926	568	0.000048	com.entrepreneur
657	19432350	3518	0.000011	be.blogspot
658	19429740	3166	0.000012	se.blogspot
659	19429712	1318	0.000024	co.lpages
660	19428992	3266	0.000012	org.carnegieendowment
661	19428674	837	0.000038	com.globenewswire
662	19428662	3175	0.000012	is.good
663	19428098	2246	0.000016	com.instructure
664	19427698	2965	0.000012	net.alarabiya
665	19427204	4090	0.000009	com.kongregate
666	19426514	2795	0.000013	com.discovermagazine
667	19425746	2613	0.000014	org.gnupg
668	19425518	556	0.000049	com.visualstudio
669	19424130	191	0.000139	com.atdmt
670	19423528	3773	0.000010	com.openlearning
671	19423230	3794	0.000010	ch.swissinfo
672	19421982	3547	0.000010	com.pixar
673	19420080	2154	0.000016	com.livemint
674	19419708	957	0.000033	com.variety
675	19417142	2816	0.000013	uk.gov.metoffice
676	19414346	2004	0.000018	com.surveygizmo
677	19412994	3337	0.000011	cn.globaltimes
678	19411212	929	0.000035	uk.gov.legislation
679	19411070	2639	0.000014	org.ballotpedia
680	19409736	243	0.000110	org.whatwg
681	19408620	3148	0.000012	com.coca-colacompany
682	19408342	1343	0.000024	uk.gov.nationalarchives
683	19406168	2326	0.000015	com.thebalancesmb
684	19404822	3145	0.000012	uk.gov.companieshouse
685	19403088	3532	0.000011	com.dailykos
686	19401008	165	0.000170	com.yelp
687	19400512	257	0.000103	com.automattic
688	19400270	4169	0.000009	com.penzu
689	19399686	2489	0.000014	com.bloomberglaw
690	19399662	412	0.000065	org.opensource
691	19398126	1547	0.000021	org.khanacademy
692	19397376	3834	0.000010	com.sfweekly
693	19395236	2779	0.000013	com.thumbtack
694	19394202	2880	0.000013	org.royalsociety
695	19393684	1674	0.000020	kr.co.google
696	19393678	2531	0.000014	com.post-gazette
697	19393520	2800	0.000013	org.panda
698	19390648	2421	0.000015	com.thenation
699	19389714	2823	0.000013	io.fabric
700	19388974	4936	0.000008	org.arkive
701	19388756	2689	0.000013	uk.co.bbci
702	19387624	4042	0.000009	hk.edu.cityu
703	19387406	3194	0.000012	com.scribblelive
704	19386352	3553	0.000010	com.gimletmedia
705	19385872	3489	0.000011	com.tweetmeme
706	19384830	2541	0.000014	de.uni-heidelberg
707	19384284	298	0.000089	ai.shortpixel
708	19383872	1920	0.000019	gov.gao
709	19382974	4425	0.000008	com.storeboard
710	19381650	2814	0.000013	com.politifact
711	19380202	3349	0.000011	org.cato
712	19379282	4889	0.000008	com.uberant
713	19377306	3183	0.000012	fr.lepoint
714	19377194	3809	0.000010	edu.depaul
715	19376126	3844	0.000010	net.thedailystar
716	19375590	406	0.000066	com.aol
717	19375570	4046	0.000009	edu.umt
718	19372794	1948	0.000018	tv.ustream
719	19372628	1034	0.000031	com.verisign
720	19369588	3279	0.000011	com.theweek
721	19367934	905	0.000035	com.box
722	19367170	3724	0.000010	com.eklablog
723	19365850	3488	0.000011	com.militarytimes
724	19365832	866	0.000037	gov.uspto
725	19365580	3483	0.000011	com.multiscreensite
726	19364098	3103	0.000012	uk.ac.york
727	19359488	3165	0.000012	org.openweathermap
728	19358574	1526	0.000021	com.techrepublic
729	19358070	3315	0.000011	org.jenkins-ci
730	19357968	2815	0.000013	org.wnyc
731	19357458	638	0.000044	gov.copyright
732	19356834	3433	0.000011	com.lawfareblog
733	19354610	2357	0.000015	co.pcdn
734	19353004	3263	0.000012	com.nyt
735	19352766	3101	0.000012	se.svt
736	19351866	1048	0.000030	net.clickbank
737	19351546	3121	0.000012	com.scotsman
738	19348720	1182	0.000027	com.foursquare
739	19348660	1239	0.000026	com.pingdom
740	19348048	2475	0.000014	com.squarespace-cdn
741	19346678	2323	0.000015	com.natlawreview
742	19346350	2769	0.000013	org.wri
743	19345800	3430	0.000011	com.bigthink
744	19345054	4132	0.000009	com.newgrounds
745	19344692	3862	0.000010	org.sourcewatch
746	19342356	3720	0.000010	re.cli
747	19341788	3156	0.000012	gov.ncjrs
748	19341458	3087	0.000012	my.com.thestar
749	19340698	3307	0.000011	gov.anl
750	19339932	3117	0.000012	com.nationalreview
751	19339132	2597	0.000014	ca.newswire
752	19338090	1603	0.000020	org.webkit
753	19337402	3700	0.000010	org.elasticsearch
754	19335276	928	0.000035	com.hootsuite
755	19334936	300	0.000088	com.caniuse
756	19334252	3236	0.000012	gov.fec
757	19333910	2327	0.000015	ru.rg
758	19333124	3741	0.000010	org.constitutioncenter
759	19332102	1602	0.000020	com.jwplayer
760	19331754	4253	0.000009	com.etymonline
761	19331678	3620	0.000010	it.eventbrite
762	19331510	2960	0.000013	com.madmimi
763	19331460	3491	0.000011	com.afp
764	19330192	1907	0.000019	com.kinstacdn
765	19328136	3163	0.000012	gov.ornl
766	19327042	461	0.000061	com.pubmatic
767	19325866	401	0.000066	gg.discord
768	19325518	1289	0.000025	com.intuit
769	19325482	1168	0.000027	com.ycombinator
770	19325258	3292	0.000011	com.crashlytics
771	19324302	4270	0.000009	com.underconsideration
772	19322856	2599	0.000014	com.articulate
773	19322230	3246	0.000012	de.uni-frankfurt
774	19321496	3692	0.000010	uk.co.spectator
775	19321096	867	0.000037	com.wikihow
776	19321010	4275	0.000009	to.gplus
777	19320802	4920	0.000008	pl.pastebin
778	19320622	3791	0.000010	uk.co.manchestereveningnews
779	19319854	2938	0.000013	edu.unh
780	19318976	2553	0.000014	de.tagesschau
781	19318802	2116	0.000017	gov.energystar
782	19318372	429	0.000063	com.businesswire
783	19318050	829	0.000038	com.moz
784	19314848	3550	0.000010	org.avaaz
785	19314554	3683	0.000010	com.mnn
786	19314476	1172	0.000027	com.alexa
787	19314150	2332	0.000015	net.vnexpress
788	19313268	348	0.000077	com.constantcontact
789	19312732	3600	0.000010	com.heraldscotland
790	19312326	3843	0.000010	fm.audioboo
791	19311750	4481	0.000008	tv.eurovision
792	19311646	974	0.000033	com.fandom
793	19311256	3717	0.000010	uk.ac.uea
794	19311174	3697	0.000010	uk.ac.core
795	19310268	3514	0.000011	com.hsbc
796	19310254	3492	0.000011	org.sciencenews
797	19310242	4916	0.000008	com.blackplanet
798	19310096	3289	0.000011	com.realclearpolitics
799	19309366	1698	0.000020	com.pastebin
800	19309196	3190	0.000012	uk.org.rspb
801	19308322	1377	0.000023	com.techradar
802	19308094	529	0.000053	com.indeed
803	19307548	4985	0.000007	dk.bloggersdelight
804	19307144	4491	0.000008	com.xtgem
805	19306108	2073	0.000017	ca.on.gov
806	19305500	3536	0.000011	uk.co.thisismoney
807	19304908	797	0.000040	gov.sec
808	19302330	1128	0.000028	net.atlassian
809	19302240	3937	0.000009	com.collinsdictionary
810	19299944	1479	0.000022	edu.purdue
811	19299020	3179	0.000012	com.wayfair
812	19298908	3611	0.000010	org.chathamhouse
813	19297900	3218	0.000012	org.rferl
814	19297216	397	0.000066	com.skype
815	19296536	4738	0.000008	edu.ualr
816	19296016	3523	0.000011	org.diva-portal
817	19295672	2785	0.000013	org.cfr
818	19294806	1249	0.000025	com.merriam-webster
819	19292968	4835	0.000008	com.designobserver
820	19292734	3399	0.000011	org.pewforum
821	19292200	270	0.000100	jp.co.amazon
822	19291468	3994	0.000009	uk.co.dailyrecord
823	19290936	3951	0.000009	edu.swarthmore
824	19290570	3339	0.000011	com.ubs
825	19289748	1075	0.000030	so.notion
826	19289742	2847	0.000013	us.govtrack
827	19289236	1256	0.000025	com.udemy
828	19289040	333	0.000079	com.hackerone
829	19288716	3787	0.000010	org.nationalinterest
830	19288626	3138	0.000012	com.doubleclickbygoogle
831	19288000	279	0.000097	de.amazon
832	19287244	2036	0.000018	org.doxygen
833	19286840	1661	0.000020	scot.gov
834	19286652	3933	0.000009	de.berliner-zeitung
835	19285868	1519	0.000021	com.billboard
836	19283910	681	0.000042	com.gartner
837	19283390	4698	0.000008	net.writeablog
838	19282688	2465	0.000014	com.infoworld
839	19282084	823	0.000039	com.sedo
840	19281700	3200	0.000012	org.aei
841	19280820	71	0.000502	com.oculus
842	19280652	1580	0.000021	edu.ucsd
843	19280396	329	0.000081	mp.mailchi
844	19280288	3917	0.000009	edu.umaine
845	19279222	3262	0.000012	org.iucnredlist
846	19279130	2827	0.000013	com.lexology
847	19278304	4851	0.000008	com.nation2
848	19278156	5290	0.000007	com.anotepad
849	19278056	4128	0.000009	za.co.mg
850	19276824	77	0.000467	com.messenger
851	19276460	2083	0.000017	org.dejure
852	19276002	4494	0.000008	net.blogfreely
853	19275630	1302	0.000024	org.owasp
854	19275142	3309	0.000011	com.foreignaffairs
855	19275092	4067	0.000009	tw.com.books
856	19274916	4267	0.000009	ca.nfb
857	19274822	364	0.000073	com.bitly
858	19274560	3225	0.000012	org.osce
859	19274028	3726	0.000010	uk.org.wwf
860	19274006	3971	0.000009	org.truthout
861	19273104	155	0.000178	gov.privacyshield
862	19272708	1981	0.000018	edu.uci
863	19272368	2044	0.000017	se.haxx
864	19272288	897	0.000036	com.emarketer
865	19272110	4532	0.000008	com.symbaloo
866	19271508	1004	0.000032	com.playstation
867	19271338	2196	0.000016	org.sundance
868	19271216	363	0.000073	eu.youronlinechoices
869	19271196	3496	0.000011	com.rev
870	19271080	4071	0.000009	in.thewire
871	19270976	159	0.000174	org.nginx
872	19270528	903	0.000036	com.libsyn
873	19268650	2400	0.000015	us.pa.state
874	19267610	146	0.000205	me.line
875	19267478	5202	0.000007	net.bravejournal
876	19267386	3140	0.000012	ru.kp
877	19267334	4014	0.000009	com.ecowatch
878	19266700	514	0.000055	org.debian
879	19266302	539	0.000052	com.gofundme
880	19266194	976	0.000033	com.pcmag
881	19264914	4151	0.000009	com.theoutline
882	19264512	4316	0.000009	org.icj-cij
883	19263626	1470	0.000022	org.coursera
884	19261610	2076	0.000017	gov.healthcare
885	19260626	3721	0.000010	com.iconarchive
886	19259734	1657	0.000020	net.leadpages
887	19259034	1486	0.000022	com.technologyreview
888	19258032	2367	0.000015	ca.citizenlab
889	19257884	3690	0.000010	com.governing
890	19257782	3322	0.000011	com.wikidot
891	19257260	2385	0.000015	org.raspberrypi
892	19256452	4621	0.000008	jp.ac.kobe-u
893	19255454	1073	0.000030	com.timeanddate
894	19254836	1096	0.000029	com.buffer
895	19254032	3978	0.000009	com.ogilvy
896	19251530	940	0.000034	com.css-tricks
897	19251096	1501	0.000021	com.msdn
898	19250138	3958	0.000009	com.gab
899	19249994	3673	0.000010	com.what3words
900	19249260	1241	0.000026	com.tableau
901	19248316	1319	0.000024	com.xkcd
902	19248224	3695	0.000010	com.nestle
903	19247678	4982	0.000007	net.postheaven
904	19246428	470	0.000060	com.fc2
905	19246238	1795	0.000019	com.pcworld
906	19246028	2589	0.000014	mp.j
907	19245754	4318	0.000009	org.kuow
908	19245300	3906	0.000009	org.migrationpolicy
909	19245282	585	0.000047	com.fortune
910	19244324	3769	0.000010	de.morgenpost
911	19244120	3282	0.000011	uk.gov.data
912	19243558	4952	0.000007	cz.webgarden
913	19243100	2118	0.000017	org.donorbox
914	19242192	3909	0.000009	de.uni-konstanz
915	19241684	4218	0.000009	org.birdlife
916	19240982	3875	0.000010	org.people-press
917	19240778	2132	0.000017	to.dev
918	19239846	906	0.000035	org.golang
919	19238732	2425	0.000015	net.noscript
920	19237742	1223	0.000026	com.podbean
921	19235906	4130	0.000009	com.scienceblogs
922	19235706	4948	0.000007	it.clyp
923	19235498	3355	0.000011	edu.fordham
924	19231696	4076	0.000009	org.oyez
925	19230656	3441	0.000011	com.joebiden
926	19229960	2867	0.000013	com.washingtonexaminer
927	19229728	1115	0.000028	com.gizmodo
928	19229112	2757	0.000013	org.healthaffairs
929	19228910	1232	0.000026	com.searchengineland
930	19228678	854	0.000037	fm.anchor
931	19227412	5084	0.000007	com.zcubes
932	19227258	1995	0.000018	com.ssllabs
933	19225964	1072	0.000030	org.poynter
934	19224436	1644	0.000020	net.java
935	19223632	1514	0.000021	edu.usc
936	19223252	3680	0.000010	org.carbonbrief
937	19221502	5165	0.000007	org.csgrid
938	19221286	308	0.000087	jp.ameblo
939	19220064	1578	0.000021	com.sun
940	19220010	3959	0.000009	org.rfa
941	19218588	2616	0.000014	uk.gov.defra
942	19218556	3912	0.000009	com.exxonmobil
943	19218102	5249	0.000007	com.topsitenet
944	19217732	3012	0.000012	com.html5rocks
945	19217494	3660	0.000010	ca.yelp
946	19216576	2940	0.000013	com.instructables
947	19215582	2212	0.000016	org.linuxfoundation
948	19215410	4069	0.000009	uk.org.woodlandtrust
949	19213854	2058	0.000017	org.json
950	19213790	214	0.000124	com.tripadvisor
951	19212490	5233	0.000007	net.squareblogs
952	19212378	3864	0.000010	ru.mid
953	19212170	231	0.000113	com.myshopify
954	19211108	3310	0.000011	com.flippa
955	19211092	3850	0.000010	com.townandcountrymag
956	19210938	1292	0.000025	build.bazel
957	19210816	5295	0.000007	net.werite
958	19210212	1240	0.000026	com.uk
959	19209742	2354	0.000015	com.storify
960	19209508	3280	0.000011	org.cjr
961	19208854	3158	0.000012	org.acog
962	19208448	3921	0.000009	br.com.sebrae
963	19208380	250	0.000107	org.icann
964	19207876	1635	0.000020	fr.blogspot
965	19207582	122	0.000262	com.bizjournals
966	19207280	3406	0.000011	org.cites
967	19207102	2687	0.000013	com.tutsplus
968	19207058	3409	0.000011	tr.com.aa
969	19206380	1109	0.000028	org.whatbrowser
970	19205750	4680	0.000008	org.learner
971	19205144	3424	0.000011	no.yr
972	19203738	4271	0.000009	com.s-nbcnews
973	19203150	4166	0.000009	org.spie
974	19203082	1335	0.000024	com.indiegogo
975	19202634	708	0.000042	com.airbnb
976	19202288	4217	0.000009	com.revolut
977	19201514	4339	0.000009	org.atsjournals
978	19201386	1033	0.000031	com.redhat
979	19200760	4066	0.000009	uk.co.zoopla
980	19199826	318	0.000084	it.google
981	19199246	1137	0.000028	com.windowsphone
982	19198666	1485	0.000022	edu.unc
983	19198508	466	0.000060	gov.fda
984	19198408	653	0.000043	com.zapier
985	19198272	2161	0.000016	com.gigaom
986	19197316	4457	0.000008	ru.novayagazeta
987	19196504	1936	0.000018	br.com.correios
988	19196468	4101	0.000009	google.design
989	19195350	2194	0.000016	org.eu
990	19192238	3758	0.000010	com.mail-archive
991	19191310	4437	0.000008	com.out
992	19191000	4759	0.000008	tw.focustaiwan
993	19190946	4235	0.000009	org.insideclimatenews
994	19190774	2038	0.000017	com.freeprivacypolicy
995	19190442	4265	0.000009	org.escardio
996	19190354	4663	0.000008	com.theschooloflife
997	19189766	241	0.000111	com.naver
998	19188362	4711	0.000008	edu.uah
999	19188230	1611	0.000020	com.nike
1000	19187360	4319	0.000009	edu.mtsu

Credits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.

We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!

May 2021 crawl archive now available

May 23, 2021Sebastian Nagel

The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.28 billion new URLs, not visited in any of our prior crawls.

Archive Location and Download

The May crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-21/.

	File List	#Files	Total Size Compressed (TiB)
Segments	CC-MAIN-2021-21/segment.paths.gz	100
WARC files	CC-MAIN-2021-21/warc.paths.gz	64000	66.17
WAT files	CC-MAIN-2021-21/wat.paths.gz	64000	17.61
WET files	CC-MAIN-2021-21/wet.paths.gz	64000	7.65
Robots.txt files	CC-MAIN-2021-21/robotstxt.paths.gz	64000	0.17
Non-200 responses files	CC-MAIN-2021-21/non200responses.paths.gz	64000	1.86
URL index files	CC-MAIN-2021-21/cc-index.paths.gz	302	0.2

The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-21/. Also the columnar index has been updated to contain this crawl.