*Important news* for users of Common Crawl data: we are introducing CloudFront as a new way to access Common Crawl data as part of Amazon Web Services’ registry of open data
Ten years ago(!) Common Crawl joined AWS’s Open Data Sponsorships program, hosted on S3, with free access to everyone. Since then, the dataset has expanded (by petabytes!) and our community of users has seen extraordinary growth. With growth comes change, therefore…
Following are new measures to accommodate the number and volume of requests for Common Crawl data, and to ensure efficient and stable access to the data:
1. Common Crawl data is now available on CloudFront under the URL base prefixes*:
https://data.commoncrawl.org/
orhttps://ds5q9oxwqwsfj.cloudfront.net/
* Data is accessible via http:// or https://.
As of Monday, April 4, 2022
2. Access to data from the Amazon cloud using the S3 API will be restricted to authenticated AWS users, and unsigned access to s3://commoncrawl/
will be disabled. See Q&A for further details.
3. To access data from outside the Amazon cloud, via HTTP(S), a new URL prefix – https://data.commoncrawl.org/
– must be used instead of https://commoncrawl.s3.amazonaws.com/
.
Q & A
- Q: How can I identify whether my code is using unauthenticated S3 access?
- A: Authenticated access is the default, but you should verify that your code is not configured otherwise; code fragments requesting unauthenticated access could be (but are not limited to):
- AWS CLI with the command-line option
--no-sign-request
:aws --no-sign-request s3 cp s3://commoncrawl/...
- Python using boto3 and
botocore.UNSIGNED
:import boto3 import botocore s3client = boto3.client('s3', config=botocore.client.Config(signature_version=botocore.UNSIGNED))
- Hadoop or Spark (various programming languages): usage of AnonymousAWSCredentialsProvider
- AWS CLI with the command-line option
- Q: Where can I find up-to-date documentation and examples that are consistent with these new protocols?
- A: We will update our documentation and examples to read data either via HTTP(S) using the new CloudFront access, or via S3 and authenticated access. We will also update all data download links. It will take a few days until everything is updated. Thanks for your patience!
- Q: Are range requests supported?
- A: Yes, range requests are supported by CloudFront same as for S3, see the CloudFront docs about RangeGETs.
- Q: What is the recommended access method on AWS but in a different region (not us-east-1)?
- A: We recommend that you run your computing workload in the same region (us-east-1) as the Common Crawl dataset whenever possible. If you have a specific, ongoing need to run computing workloads using Common Crawl in another AWS region, the AWS Open Data team would like to hear more about your use case ([email protected]).
- Q: What are typical error messages indicating that unauthenticated access is used?
- A: The HTTP response status code is a 403 Forbidden. However, also a restrictive IAM policy on the user’s side could deny access to s3://commoncrawl/ using the S3 API.
Two examples for error messages related to unauthenticated access to s3://commoncrawl/:> curl https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2022-05/warc.paths.gz <!--?xml version="1.0" encoding="UTF-8"?--> <error><code>AccessDenied</code><message>Access Denied</message>...</error> > aws --no-sign-request s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2022-05/warc.paths.gz . fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
January 2022 crawl archive now available
The crawl archive for January 2022 is now available! The data was crawled January 16 – 29 and contains 2.95 billion web pages or 320 TiB of uncompressed content. It includes page captures of 1.35 billion new URLs, not visited in any of our prior crawls.
Archive Location and Download
The January crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2022-05/.
To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.
File List | #Files | Total Size Compressed (TiB) |
|
---|---|---|---|
Segments | CC-MAIN-2022-05/segment.paths.gz | 100 | |
WARC files | CC-MAIN-2022-05/warc.paths.gz | 72000 | 73.5 |
WAT files | CC-MAIN-2022-05/wat.paths.gz | 72000 | 19.85 |
WET files | CC-MAIN-2022-05/wet.paths.gz | 72000 | 8.63 |
Robots.txt files | CC-MAIN-2022-05/robotstxt.paths.gz | 72000 | 0.14 |
Non-200 responses files | CC-MAIN-2022-05/non200responses.paths.gz | 72000 | 1.79 |
URL index files | CC-MAIN-2022-05/cc-index.paths.gz | 302 | 0.22 |
The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2022-05/. Also the columnar index has been updated to contain this crawl.
Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.
November/December 2021 crawl archive now available
The crawl archive for November/December 2021 is now available! The data was crawled Nov 26 – Dec 9 and contains 2.5 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.2 billion new URLs, not visited in any of our prior crawls.
Improvements
The column url_host_name_reversed
was added to the columnar index. It holds the host name in reverse domain name notation (com.example.www
) which is more efficient to query. In order to make use of the new column please use the updated table schema. On Amazon Athena you need to recreate table by running the latest table creation statement. Further details are found in the corresponding pull request.
The columnar index is now built using Spark version 3.2.0 and Parquet MR 1.12.1 – these upgrades allow us to go for further improvements next year. Let us know if you observe any incompatibilities using this index to query data from the Nov/Dec 2021 crawl.
The crawler is now based on Tika 2.1.0 for detection of MIME/content types and character sets used to encode HTML documents.
Archive Location and Download
The November/December crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-49/.
To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.
File List | #Files | Total Size Compressed (TiB) |
|
---|---|---|---|
Segments | CC-MAIN-2021-49/segment.paths.gz | 100 | |
WARC files | CC-MAIN-2021-49/warc.paths.gz | 64000 | 68.66 |
WAT files | CC-MAIN-2021-49/wat.paths.gz | 64000 | 16.66 |
WET files | CC-MAIN-2021-49/wet.paths.gz | 64000 | 7.18 |
Robots.txt files | CC-MAIN-2021-49/robotstxt.paths.gz | 64000 | 0.15 |
Non-200 responses files | CC-MAIN-2021-49/non200responses.paths.gz | 64000 | 2.29 |
URL index files | CC-MAIN-2021-49/cc-index.paths.gz | 302 | 0.2 |
The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-49/. Also the columnar index has been updated to contain this crawl.
Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.
October 2021 crawl archive now available
The crawl archive for October 2021 is now available! The data was crawled Oct 15 – 28 and contains 3.3 billion web pages or 360 TiB of uncompressed content. It includes page captures of 1.3 billion new URLs, not visited in any of our prior crawls.
Archive Location and Download
The October crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-43/.
To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.
File List | #Files | Total Size Compressed (TiB) |
|
---|---|---|---|
Segments | CC-MAIN-2021-43/segment.paths.gz | 100 | |
WARC files | CC-MAIN-2021-43/warc.paths.gz | 72000 | 85.11 |
WAT files | CC-MAIN-2021-43/wat.paths.gz | 72000 | 22.42 |
WET files | CC-MAIN-2021-43/wet.paths.gz | 72000 | 9.79 |
Robots.txt files | CC-MAIN-2021-43/robotstxt.paths.gz | 72000 | 0.15 |
Non-200 responses files | CC-MAIN-2021-43/non200responses.paths.gz | 72000 | 2.11 |
URL index files | CC-MAIN-2021-43/cc-index.paths.gz | 302 | 0.25 |
The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-43/. Also the columnar index has been updated to contain this crawl.
Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.
Host- and Domain-Level Web Graphs June, July/August and September 2021
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of June, July/August and September 2021. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases. You may also visit the projects cc-webgraph and cc-pyspark which include all scripts and tools required to construct the graphs. Instructions to explore the graphs in the webgraph format are given in our collection of webgraph notebooks.
Host-level graph
The graph consists of 766 million nodes and 4.95 billion edges. Both hyperlinks and HTTP redirects and link headers are used as edges to span up the graph. All types of links are included, including pure “technical” ones pointing to images, JavaScript libraries, web fonts, etc. However, only host names with a valid IANA TLD are used. Consequently, URLs with an IP address as host component are not taken into account for building the host-level graph.
There are 701 million dangling nodes (91.6%) and the largest strongly connected component contains 47.0 million (6.1%) nodes. Dangling nodes stem from
- hosts that have not been crawled, yet are pointed to from a link on a crawled page
- hosts without any links pointing to a different host name
- or hosts which did only return an error page (eg. HTTP 404)
Host names in the graph are in reverse domain name notation and a leading www.
is stripped: www.subdomain.example.com
becomes com.example.subdomain
.
You can download the graph and the ranks of all 766 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2021-jun-jul-sep/host/
. Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2021-jun-jul-sep/host/
as prefix to access the files from everywhere.
Please note that the text representation of the host-level graph is shipped in 40 gzip-compressed files listed in two path listings – one for the nodes (vertices), one for the edges (arcs). First, download the paths listing and decompress it using “gzip”. By adding the prefix s3://commoncrawl/
or https://data.commoncrawl.org/
to each line in the path listing you get the list of URLs to download the entire graph.
Size | File | Description |
---|---|---|
4.50 GB | cc-main-2021-jun-jul-sep-host-vertices.paths.gz | nodes 〈id, rev host〉, paths of 16 vertices files |
20.76 GB | cc-main-2021-jun-jul-sep-host-edges.paths.gz | edges 〈from_id, to_id〉, paths of 24 edges files |
8.43 GB | cc-main-2021-jun-jul-sep-host.graph | graph in BVGraph format |
2 kB | cc-main-2021-jun-jul-sep-host.properties | |
9.83 GB | cc-main-2021-jun-jul-sep-host-t.graph | transpose of the graph (outlinks inverted to inlinks) |
2 kB | cc-main-2021-jun-jul-sep-host-t.properties | |
1 kB | cc-main-2021-jun-jul-sep-host.stats | WebGraph statistics |
11.06 GB | cc-main-2021-jun-jul-sep-host-ranks.txt.gz | harmonic centrality and pagerank |
Domain-level graph
The domain graph is built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org. Version (commit) a5b046d of the public suffix list was used (commit date 2021-10-06).
The domain-level graph has 88 million nodes and 1.56 billion edges. 49% or 43 million nodes are dangling nodes, the largest strongly connected component covers 35 million or 40% of the nodes.
All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2021-jun-jul-sep/domain/
resp. https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2021-jun-jul-sep/domain/
.
Download files of the Common Crawl Jun/Jul/Sep 2021 domain-level webgraph
Size | File | Description |
---|---|---|
0.61 GB | cc-main-2021-jun-jul-sep-domain-vertices.txt.gz | nodes 〈id, rev domain, num hosts〉 |
6.34 GB | cc-main-2021-jun-jul-sep-domain-edges.txt.gz | edges 〈from_id, to_id〉 |
3.59 GB | cc-main-2021-jun-jul-sep-domain.graph | graph in BVGraph format |
2 kB | cc-main-2021-jun-jul-sep-domain.properties | |
3.44 GB | cc-main-2021-jun-jul-sep-domain-t.graph | transpose of the graph |
2 kB | cc-main-2021-jun-jul-sep-domain-t.properties | |
1 kB | cc-main-2021-jun-jul-sep-domain.stats | WebGraph statistics |
1.89 GB | cc-main-2021-jun-jul-sep-domain-ranks.txt.gz | harmonic centrality and pagerank |
Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 88 million domain ranks is available for download.
Top 1000 domains ranked by harmonic centrality (Jun/Jul/Sep 2021)
harmonic centrality rank | hc value | page rank | page rank value | reversed domain name |
---|---|---|---|---|
1 | 31634524 | 1 | 0.018526 | com.googleapis |
2 | 30590448 | 3 | 0.013879 | com.facebook |
3 | 29344102 | 2 | 0.015082 | com.google |
4 | 26579208 | 5 | 0.007506 | org.w |
5 | 26490886 | 4 | 0.008344 | com.twitter |
6 | 26194504 | 6 | 0.007270 | com.youtube |
7 | 25598364 | 8 | 0.006802 | com.instagram |
8 | 24808618 | 7 | 0.006829 | com.googletagmanager |
9 | 24314334 | 9 | 0.004946 | org.gmpg |
10 | 23501518 | 14 | 0.003442 | com.linkedin |
11 | 23191984 | 11 | 0.003661 | com.gstatic |
12 | 22552330 | 10 | 0.004093 | com.cloudflare |
13 | 22306348 | 17 | 0.001939 | com.gravatar |
14 | 22254410 | 12 | 0.003556 | org.wordpress |
15 | 22159662 | 22 | 0.001575 | com.pinterest |
16 | 21706482 | 23 | 0.001396 | org.wikipedia |
17 | 21593592 | 16 | 0.001952 | com.apple |
18 | 21537622 | 37 | 0.001024 | com.wordpress |
19 | 21459498 | 27 | 0.001176 | com.vimeo |
20 | 21360146 | 41 | 0.000898 | be.youtu |
21 | 21289640 | 15 | 0.002252 | com.bootstrapcdn |
22 | 21211768 | 20 | 0.001717 | com.jquery |
23 | 21069122 | 28 | 0.001166 | com.microsoft |
24 | 21014760 | 54 | 0.000699 | com.blogspot |
25 | 20975968 | 46 | 0.000793 | com.amazon |
26 | 20916000 | 45 | 0.000798 | gl.goo |
27 | 20909476 | 50 | 0.000720 | com.wp |
28 | 20878568 | 62 | 0.000580 | ly.bit |
29 | 20832704 | 35 | 0.001053 | com.google-analytics |
30 | 20815940 | 33 | 0.001068 | com.amazonaws |
31 | 20731242 | 77 | 0.000417 | com.tumblr |
32 | 20705584 | 42 | 0.000841 | org.mozilla |
33 | 20703870 | 31 | 0.001080 | net.cloudfront |
34 | 20700046 | 18 | 0.001797 | com.adobe |
35 | 20687574 | 49 | 0.000720 | eu.europa |
36 | 20682862 | 19 | 0.001772 | com.github |
37 | 20663412 | 34 | 0.001054 | net.jsdelivr |
38 | 20641588 | 29 | 0.001134 | com.wixstatic |
39 | 20534372 | 75 | 0.000442 | com.googleusercontent |
40 | 20518204 | 21 | 0.001630 | com.fontawesome |
41 | 20499484 | 96 | 0.000366 | com.yahoo |
42 | 20489920 | 51 | 0.000711 | com.paypal |
43 | 20470068 | 57 | 0.000645 | co.t |
44 | 20460526 | 47 | 0.000784 | com.whatsapp |
45 | 20453542 | 53 | 0.000704 | com.flickr |
46 | 20429610 | 106 | 0.000315 | com.reddit |
47 | 20405542 | 72 | 0.000449 | com.medium |
48 | 20396096 | 38 | 0.000956 | com.googlesyndication |
49 | 20388958 | 48 | 0.000748 | io.github |
50 | 20364584 | 125 | 0.000241 | com.nytimes |
51 | 20363658 | 59 | 0.000616 | org.w3 |
52 | 20350454 | 98 | 0.000358 | com.weebly |
53 | 20330952 | 97 | 0.000363 | org.creativecommons |
54 | 20324280 | 65 | 0.000510 | com.shopify |
55 | 20315624 | 110 | 0.000281 | com.soundcloud |
56 | 20308274 | 85 | 0.000402 | me.wp |
57 | 20300136 | 60 | 0.000610 | org.schema |
58 | 20300004 | 32 | 0.001072 | ru.yandex |
59 | 20272874 | 132 | 0.000215 | com.forbes |
60 | 20266896 | 64 | 0.000527 | com.vk |
61 | 20216640 | 114 | 0.000268 | com.spotify |
62 | 20214818 | 175 | 0.000146 | com.cnn |
63 | 20172836 | 52 | 0.000708 | net.doubleclick |
64 | 20156654 | 56 | 0.000653 | com.addthis |
65 | 20153642 | 190 | 0.000132 | uk.co.bbc |
66 | 20151882 | 230 | 0.000113 | com.wsj |
67 | 20149822 | 135 | 0.000208 | gov.nih |
68 | 20146194 | 40 | 0.000924 | com.baidu |
69 | 20144952 | 161 | 0.000166 | com.theguardian |
70 | 20127172 | 151 | 0.000179 | int.who |
71 | 20121134 | 240 | 0.000108 | com.bloomberg |
72 | 20115270 | 130 | 0.000217 | org.archive |
73 | 20111568 | 163 | 0.000163 | com.giphy |
74 | 20107092 | 94 | 0.000388 | com.list-manage |
75 | 20100830 | 192 | 0.000130 | org.wikimedia |
76 | 20095176 | 55 | 0.000682 | com.macromedia |
77 | 20091806 | 229 | 0.000113 | com.oracle |
78 | 20081572 | 145 | 0.000193 | com.imdb |
79 | 20074580 | 187 | 0.000134 | com.businessinsider |
80 | 20066912 | 297 | 0.000091 | edu.mit |
81 | 20066436 | 300 | 0.000090 | edu.stanford |
82 | 20061314 | 115 | 0.000267 | com.mailchimp |
83 | 20059906 | 44 | 0.000800 | net.facebook |
84 | 20058050 | 140 | 0.000204 | us.zoom |
85 | 20056682 | 426 | 0.000065 | com.googleblog |
86 | 20053210 | 171 | 0.000153 | com.unsplash |
87 | 20047020 | 284 | 0.000095 | com.reuters |
88 | 20043722 | 185 | 0.000137 | com.imgur |
89 | 20035754 | 142 | 0.000199 | com.wixsite |
90 | 20025300 | 191 | 0.000131 | com.stackoverflow |
91 | 20022100 | 128 | 0.000223 | com.weibo |
92 | 20016560 | 162 | 0.000163 | com.issuu |
93 | 20016026 | 423 | 0.000065 | gov.nasa |
94 | 20014282 | 36 | 0.001043 | net.fbcdn |
95 | 20013910 | 273 | 0.000098 | com.android |
96 | 20012778 | 119 | 0.000259 | me.t |
97 | 20012366 | 169 | 0.000160 | org.ietf |
98 | 20004530 | 139 | 0.000206 | com.ytimg |
99 | 20001182 | 148 | 0.000183 | org.apache |
100 | 19999464 | 318 | 0.000086 | com.theverge |
101 | 19993026 | 325 | 0.000084 | com.slack |
102 | 19992850 | 258 | 0.000103 | edu.harvard |
103 | 19989870 | 224 | 0.000116 | com.washingtonpost |
104 | 19988302 | 268 | 0.000099 | com.bbc |
105 | 19986642 | 460 | 0.000060 | edu.cornell |
106 | 19984712 | 155 | 0.000174 | com.ft |
107 | 19979856 | 117 | 0.000261 | com.npmjs |
108 | 19978744 | 416 | 0.000066 | com.ted |
109 | 19962738 | 424 | 0.000065 | com.myspace |
110 | 19957322 | 352 | 0.000078 | com.wired |
111 | 19956296 | 526 | 0.000052 | com.livejournal |
112 | 19945484 | 355 | 0.000078 | com.appspot |
113 | 19944274 | 263 | 0.000101 | org.un |
114 | 19943458 | 215 | 0.000119 | org.gnu |
115 | 19933112 | 390 | 0.000070 | com.goodreads |
116 | 19932600 | 69 | 0.000476 | com.godaddy |
117 | 19931522 | 387 | 0.000071 | org.hbr |
118 | 19917440 | 309 | 0.000087 | org.npr |
119 | 19916612 | 322 | 0.000085 | com.prnewswire |
120 | 19911628 | 332 | 0.000083 | net.researchgate |
121 | 19910122 | 307 | 0.000088 | com.githubusercontent |
122 | 19909276 | 24 | 0.001390 | io.polyfill |
123 | 19908276 | 286 | 0.000095 | com.wiley |
124 | 19905960 | 247 | 0.000106 | com.tiktok |
125 | 19894990 | 193 | 0.000130 | com.blogger |
126 | 19892346 | 71 | 0.000466 | com.unpkg |
127 | 19884978 | 102 | 0.000322 | de.google |
128 | 19869556 | 432 | 0.000064 | com.gmail |
129 | 19869236 | 565 | 0.000049 | com.vice |
130 | 19868406 | 627 | 0.000045 | org.chromium |
131 | 19867146 | 146 | 0.000186 | gle.forms |
132 | 19866916 | 113 | 0.000269 | com.youtube-nocookie |
133 | 19866834 | 179 | 0.000142 | org.ampproject |
134 | 19856608 | 354 | 0.000078 | com.time |
135 | 19856272 | 668 | 0.000043 | edu.upenn |
136 | 19856040 | 370 | 0.000074 | com.example |
137 | 19854818 | 633 | 0.000045 | com.economist |
138 | 19850884 | 727 | 0.000040 | com.evernote |
139 | 19842434 | 500 | 0.000055 | com.steampowered |
140 | 19841380 | 692 | 0.000042 | google.blog |
141 | 19841354 | 462 | 0.000060 | com.theatlantic |
142 | 19840036 | 586 | 0.000047 | org.weforum |
143 | 19837596 | 628 | 0.000045 | com.deviantart |
144 | 19837072 | 239 | 0.000109 | uk.co.google |
145 | 19833470 | 357 | 0.000077 | org.arxiv |
146 | 19833402 | 395 | 0.000070 | com.scribd |
147 | 19831882 | 407 | 0.000067 | uk.co.telegraph |
148 | 19829878 | 368 | 0.000074 | com.huffingtonpost |
149 | 19829876 | 642 | 0.000045 | com.mysql |
150 | 19821800 | 555 | 0.000050 | org.worldbank |
151 | 19818100 | 242 | 0.000107 | com.sciencedirect |
152 | 19816676 | 347 | 0.000080 | com.nature |
153 | 19814992 | 228 | 0.000113 | com.twimg |
154 | 19812336 | 120 | 0.000259 | com.statcounter |
155 | 19810098 | 246 | 0.000106 | org.acm |
156 | 19805214 | 554 | 0.000050 | org.ieee |
157 | 19799686 | 381 | 0.000071 | com.fastcompany |
158 | 19796892 | 255 | 0.000104 | org.python |
159 | 19794906 | 701 | 0.000041 | com.apnews |
160 | 19790820 | 431 | 0.000064 | com.meetup |
161 | 19790276 | 738 | 0.000040 | com.qz |
162 | 19789858 | 581 | 0.000047 | com.globenewswire |
163 | 19788416 | 365 | 0.000074 | com.docker |
164 | 19788402 | 444 | 0.000062 | com.pixabay |
165 | 19788038 | 453 | 0.000061 | uk.co.dailymail |
166 | 19784442 | 321 | 0.000085 | com.springer |
167 | 19783552 | 251 | 0.000105 | com.bandcamp |
168 | 19779288 | 256 | 0.000103 | net.behance |
169 | 19778076 | 361 | 0.000075 | com.gitlab |
170 | 19768948 | 603 | 0.000046 | com.git-scm |
171 | 19764224 | 540 | 0.000051 | io.readthedocs |
172 | 19763982 | 715 | 0.000041 | com.engadget |
173 | 19763034 | 647 | 0.000044 | com.trello |
174 | 19761140 | 183 | 0.000138 | com.bing |
175 | 19758120 | 346 | 0.000080 | com.usatoday |
176 | 19757326 | 220 | 0.000117 | com.squarespace |
177 | 19752428 | 158 | 0.000169 | com.yelp |
178 | 19747884 | 353 | 0.000078 | com.dribbble |
179 | 19747756 | 417 | 0.000066 | com.digg |
180 | 19747532 | 143 | 0.000199 | com.dropbox |
181 | 19744818 | 276 | 0.000097 | com.ibm |
182 | 19734092 | 464 | 0.000060 | uk.co.independent |
183 | 19733310 | 398 | 0.000069 | com.w3schools |
184 | 19723800 | 544 | 0.000051 | ee.linktr |
185 | 19723334 | 674 | 0.000043 | uk.co.blogspot |
186 | 19723200 | 412 | 0.000067 | com.staticflickr |
187 | 19721228 | 551 | 0.000050 | com.pexels |
188 | 19721224 | 144 | 0.000194 | gov.cdc |
189 | 19718802 | 539 | 0.000051 | org.pbs |
190 | 19718388 | 775 | 0.000038 | com.stackexchange |
191 | 19718222 | 703 | 0.000041 | org.cambridge |
192 | 19717352 | 1103 | 0.000029 | org.eclipse |
193 | 19716696 | 43 | 0.000819 | com.fb |
194 | 19712516 | 678 | 0.000042 | edu.columbia |
195 | 19710308 | 100 | 0.000351 | com.wix |
196 | 19706256 | 730 | 0.000040 | edu.washington |
197 | 19705612 | 290 | 0.000094 | com.tinyurl |
198 | 19705126 | 367 | 0.000074 | com.sagepub |
199 | 19704258 | 673 | 0.000043 | me.about |
200 | 19699044 | 219 | 0.000117 | net.slideshare |
201 | 19694818 | 653 | 0.000044 | org.sciencemag |
202 | 19689094 | 294 | 0.000091 | org.pewresearch |
203 | 19685770 | 601 | 0.000046 | com.withgoogle |
204 | 19684956 | 458 | 0.000061 | com.herokuapp |
205 | 19675188 | 440 | 0.000063 | com.quora |
206 | 19673070 | 123 | 0.000246 | com.sharethis |
207 | 19672408 | 39 | 0.000941 | com.qq |
208 | 19671982 | 176 | 0.000145 | org.doi |
209 | 19670088 | 438 | 0.000063 | co.ibb |
210 | 19667674 | 614 | 0.000046 | com.newyorker |
211 | 19664724 | 1183 | 0.000027 | com.nike |
212 | 19664240 | 316 | 0.000087 | com.typeform |
213 | 19658254 | 248 | 0.000106 | com.outlook |
214 | 19654436 | 736 | 0.000040 | com.hp |
215 | 19654104 | 791 | 0.000037 | com.foxnews |
216 | 19651026 | 234 | 0.000112 | com.cloudinary |
217 | 19648214 | 933 | 0.000035 | edu.princeton |
218 | 19646574 | 572 | 0.000048 | com.moz |
219 | 19642260 | 437 | 0.000063 | com.getpocket |
220 | 19639898 | 485 | 0.000057 | com.nbcnews |
221 | 19638836 | 645 | 0.000044 | org.bitbucket |
222 | 19635872 | 202 | 0.000124 | page.g |
223 | 19635142 | 154 | 0.000176 | gov.privacyshield |
224 | 19634924 | 277 | 0.000096 | com.disqus |
225 | 19624936 | 141 | 0.000203 | com.opera |
226 | 19623614 | 463 | 0.000060 | com.airbnb |
227 | 19620318 | 960 | 0.000034 | com.dropboxusercontent |
228 | 19618624 | 372 | 0.000073 | com.force |
229 | 19617796 | 923 | 0.000035 | co.elastic |
230 | 19614006 | 214 | 0.000119 | com.wpengine |
231 | 19613164 | 679 | 0.000042 | org.semver |
232 | 19613160 | 305 | 0.000089 | com.typepad |
233 | 19611254 | 972 | 0.000033 | com.nypost |
234 | 19610798 | 726 | 0.000040 | com.ubuntu |
235 | 19610070 | 1263 | 0.000025 | se.haxx |
236 | 19605234 | 303 | 0.000089 | com.live |
237 | 19603428 | 683 | 0.000042 | au.net.abc |
238 | 19603360 | 465 | 0.000060 | com.mozilla |
239 | 19599764 | 382 | 0.000071 | com.criteo |
240 | 19593184 | 1232 | 0.000026 | uk.co.thesun |
241 | 19581354 | 1434 | 0.000023 | edu.rutgers |
242 | 19581020 | 278 | 0.000096 | com.feedburner |
243 | 19577288 | 969 | 0.000033 | com.politico |
244 | 19572414 | 1054 | 0.000030 | co.g |
245 | 19572372 | 1849 | 0.000019 | com.instructables |
246 | 19570874 | 778 | 0.000038 | com.sap |
247 | 19568694 | 1123 | 0.000028 | org.greenpeace |
248 | 19564134 | 1127 | 0.000028 | org.kernel |
249 | 19562280 | 1650 | 0.000022 | com.googlesource |
250 | 19561484 | 124 | 0.000245 | com.filesusr |
251 | 19559444 | 1150 | 0.000028 | com.unity3d |
252 | 19558056 | 682 | 0.000042 | com.freepik |
253 | 19557106 | 542 | 0.000051 | com.fortune |
254 | 19553954 | 638 | 0.000045 | uk.ac.ox |
255 | 19553634 | 209 | 0.000121 | org.iana |
256 | 19551872 | 241 | 0.000108 | com.eepurl |
257 | 19551398 | 958 | 0.000034 | com.ssrn |
258 | 19551010 | 866 | 0.000035 | com.nvidia |
259 | 19549544 | 1569 | 0.000022 | com.storify |
260 | 19549036 | 1000 | 0.000032 | com.sun |
261 | 19548514 | 624 | 0.000045 | uk.co.eventbrite |
262 | 19544466 | 829 | 0.000036 | edu.jhu |
263 | 19541150 | 607 | 0.000046 | net.azurewebsites |
264 | 19539076 | 1291 | 0.000025 | com.reverbnation |
265 | 19538660 | 439 | 0.000063 | gov.fda |
266 | 19538044 | 99 | 0.000351 | com.stripe |
267 | 19534860 | 927 | 0.000035 | com.podbean |
268 | 19532612 | 341 | 0.000082 | net.windows |
269 | 19530966 | 1256 | 0.000025 | uk.co.ebay |
270 | 19530706 | 260 | 0.000102 | com.calendly |
271 | 19526632 | 804 | 0.000037 | com.chrome |
272 | 19525016 | 1804 | 0.000020 | com.martinfowler |
273 | 19524922 | 926 | 0.000035 | edu.academia |
274 | 19523484 | 568 | 0.000049 | site.business |
275 | 19521568 | 231 | 0.000113 | com.office |
276 | 19521106 | 328 | 0.000084 | com.netdna-ssl |
277 | 19518874 | 253 | 0.000105 | com.newsweek |
278 | 19516880 | 210 | 0.000120 | tv.twitch |
279 | 19515182 | 1220 | 0.000026 | com.vogue |
280 | 19514726 | 2307 | 0.000017 | com.diigo |
281 | 19514662 | 1214 | 0.000026 | org.postgresql |
282 | 19513292 | 691 | 0.000042 | com.xinhuanet |
283 | 19508378 | 1368 | 0.000024 | de.mpg |
284 | 19508326 | 525 | 0.000052 | com.squareup |
285 | 19507892 | 408 | 0.000067 | org.debian |
286 | 19502082 | 121 | 0.000252 | com.paypalobjects |
287 | 19501986 | 716 | 0.000041 | gov.senate |
288 | 19501788 | 2392 | 0.000016 | com.pearltrees |
289 | 19500240 | 1147 | 0.000028 | com.500px |
290 | 19499046 | 343 | 0.000081 | com.googlecode |
291 | 19497980 | 712 | 0.000041 | org.change |
292 | 19497516 | 491 | 0.000056 | com.tandfonline |
293 | 19496848 | 67 | 0.000505 | net.akamaihd |
294 | 19492076 | 1265 | 0.000025 | com.aljazeera |
295 | 19492054 | 706 | 0.000041 | com.qualtrics |
296 | 19490750 | 783 | 0.000037 | com.theconversation |
297 | 19490388 | 1078 | 0.000029 | com.theglobeandmail |
298 | 19486228 | 134 | 0.000211 | de.bund |
299 | 19485788 | 1299 | 0.000025 | edu.illinois |
300 | 19485786 | 327 | 0.000084 | com.cnbc |
301 | 19484140 | 858 | 0.000036 | uk.co.guardian |
302 | 19483768 | 483 | 0.000057 | com.msn |
303 | 19475416 | 129 | 0.000219 | com.rawgit |
304 | 19472372 | 474 | 0.000059 | com.stumbleupon |
305 | 19469278 | 198 | 0.000125 | net.sourceforge |
306 | 19468982 | 356 | 0.000077 | com.optimizely |
307 | 19468656 | 470 | 0.000059 | org.openstreetmap |
308 | 19467526 | 335 | 0.000083 | com.techcrunch |
309 | 19465998 | 434 | 0.000064 | com.ssl-images-amazon |
310 | 19464188 | 1405 | 0.000023 | edu.ufl |
311 | 19462912 | 1395 | 0.000023 | edu.gatech |
312 | 19462816 | 165 | 0.000162 | com.hubspot |
313 | 19462736 | 310 | 0.000087 | com.mapbox |
314 | 19462404 | 441 | 0.000063 | com.go |
315 | 19459072 | 608 | 0.000046 | gov.noaa |
316 | 19458846 | 1470 | 0.000023 | com.channel4 |
317 | 19458794 | 1237 | 0.000026 | ca.sfu |
318 | 19457698 | 592 | 0.000047 | com.healthline |
319 | 19457660 | 794 | 0.000037 | org.fao |
320 | 19457528 | 507 | 0.000054 | ca.google |
321 | 19457170 | 2431 | 0.000016 | com.wattpad |
322 | 19454268 | 1280 | 0.000025 | uk.co.standard |
323 | 19452678 | 667 | 0.000043 | gov.house |
324 | 19451936 | 1309 | 0.000025 | uk.co.wired |
325 | 19450294 | 1862 | 0.000019 | com.invisionapp |
326 | 19449754 | 743 | 0.000039 | com.pinimg |
327 | 19449738 | 184 | 0.000138 | com.amazon-adsystem |
328 | 19449480 | 1836 | 0.000019 | org.maven |
329 | 19446242 | 2507 | 0.000015 | com.openai |
330 | 19443698 | 803 | 0.000037 | org.pypi |
331 | 19442142 | 262 | 0.000101 | net.azureedge |
332 | 19441000 | 490 | 0.000056 | com.kickstarter |
333 | 19439570 | 1962 | 0.000019 | uk.bl |
334 | 19439402 | 1236 | 0.000026 | au.com.smh |
335 | 19436904 | 1339 | 0.000024 | com.vanityfair |
336 | 19435504 | 261 | 0.000102 | uk.org.ico |
337 | 19434368 | 153 | 0.000176 | com.addtoany |
338 | 19427034 | 986 | 0.000032 | com.uk |
339 | 19426114 | 1097 | 0.000029 | com.scmp |
340 | 19424036 | 981 | 0.000033 | org.pnas |
341 | 19422474 | 433 | 0.000064 | com.cnet |
342 | 19422174 | 348 | 0.000080 | com.statista |
343 | 19421360 | 149 | 0.000180 | org.nodejs |
344 | 19418836 | 793 | 0.000037 | us.icio |
345 | 19416290 | 2916 | 0.000013 | com.instapaper |
346 | 19415752 | 489 | 0.000056 | gov.epa |
347 | 19405682 | 1003 | 0.000032 | com.mixcloud |
348 | 19404500 | 677 | 0.000042 | org.d3js |
349 | 19401872 | 851 | 0.000036 | com.britannica |
350 | 19400204 | 593 | 0.000047 | uk.gov.service |
351 | 19399466 | 177 | 0.000143 | org.allaboutcookies |
352 | 19396954 | 546 | 0.000050 | edu.berkeley |
353 | 19395162 | 296 | 0.000091 | me.telegram |
354 | 19393670 | 1436 | 0.000023 | com.irishtimes |
355 | 19392716 | 944 | 0.000034 | int.coe |
356 | 19388874 | 157 | 0.000170 | com.zendesk |
357 | 19388222 | 1507 | 0.000022 | org.hrc |
358 | 19387040 | 1251 | 0.000025 | com.history |
359 | 19386872 | 152 | 0.000178 | io.shields |
360 | 19386294 | 1686 | 0.000021 | ms.1drv |
361 | 19384766 | 548 | 0.000050 | com.biomedcentral |
362 | 19384760 | 473 | 0.000059 | com.latimes |
363 | 19381234 | 1062 | 0.000030 | org.jstor |
364 | 19381206 | 1124 | 0.000028 | com.jetbrains |
365 | 19376152 | 1084 | 0.000029 | org.ilo |
366 | 19376036 | 777 | 0.000038 | edu.psu |
367 | 19375334 | 137 | 0.000207 | com.youronlinechoices |
368 | 19374986 | 895 | 0.000035 | com.ecwid |
369 | 19371930 | 1067 | 0.000030 | com.brightcove |
370 | 19371460 | 748 | 0.000039 | it.scoop |
371 | 19369410 | 329 | 0.000084 | ru.ok |
372 | 19364910 | 1226 | 0.000026 | com.digitaltrends |
373 | 19362788 | 1004 | 0.000032 | uk.co.thetimes |
374 | 19359762 | 1404 | 0.000023 | com.thedailybeast |
375 | 19359122 | 1604 | 0.000022 | edu.osu |
376 | 19354290 | 513 | 0.000054 | edu.yale |
377 | 19351764 | 116 | 0.000262 | com.jimdo |
378 | 19349800 | 2354 | 0.000016 | com.fastcodesign |
379 | 19349466 | 988 | 0.000032 | uk.parliament |
380 | 19348128 | 456 | 0.000061 | org.freecodecamp |
381 | 19343488 | 2209 | 0.000018 | com.us |
382 | 19340214 | 573 | 0.000048 | com.deloitte |
383 | 19339976 | 1010 | 0.000031 | uk.co.huffingtonpost |
384 | 19339738 | 605 | 0.000046 | com.zdnet |
385 | 19339098 | 181 | 0.000139 | ru.mail |
386 | 19337572 | 415 | 0.000066 | com.elsevier |
387 | 19335810 | 1185 | 0.000027 | org.nejm |
388 | 19335572 | 2422 | 0.000016 | com.instructure |
389 | 19335456 | 427 | 0.000065 | net.imgix |
390 | 19334596 | 1714 | 0.000021 | com.citrix |
391 | 19330958 | 2590 | 0.000014 | org.aclweb |
392 | 19330886 | 1964 | 0.000018 | org.haskell |
393 | 19326754 | 649 | 0.000044 | gov.state |
394 | 19326634 | 1350 | 0.000024 | app.netlify |
395 | 19326068 | 801 | 0.000037 | com.venturebeat |
396 | 19323896 | 217 | 0.000118 | com.eventbrite |
397 | 19323876 | 622 | 0.000045 | com.seattletimes |
398 | 19323284 | 1868 | 0.000019 | jp.ac.u-tokyo |
399 | 19322408 | 3048 | 0.000012 | org.uxplanet |
400 | 19319794 | 1186 | 0.000027 | com.dw |
401 | 19318404 | 1248 | 0.000026 | org.undp |
402 | 19317950 | 211 | 0.000120 | com.etsy |
403 | 19317948 | 1763 | 0.000020 | com.itv |
404 | 19317548 | 233 | 0.000112 | net.php |
405 | 19314522 | 58 | 0.000638 | com.googleadservices |
406 | 19314004 | 271 | 0.000098 | com.surveymonkey |
407 | 19310004 | 245 | 0.000107 | org.aboutcookies |
408 | 19309574 | 1900 | 0.000019 | edu.vt |
409 | 19307434 | 2477 | 0.000015 | org.wikibooks |
410 | 19305780 | 687 | 0.000042 | gov.nist |
411 | 19297658 | 1423 | 0.000023 | com.thehindu |
412 | 19297142 | 1120 | 0.000028 | org.hrw |
413 | 19295118 | 600 | 0.000046 | com.thinkwithgoogle |
414 | 19294574 | 1682 | 0.000021 | gov.usembassy |
415 | 19294274 | 656 | 0.000044 | com.intel |
416 | 19292884 | 1725 | 0.000021 | int.unfccc |
417 | 19291806 | 394 | 0.000070 | com.ebay |
418 | 19289908 | 2419 | 0.000016 | google.ai |
419 | 19286562 | 2270 | 0.000017 | com.netvibes |
420 | 19285156 | 2562 | 0.000015 | io.material |
421 | 19283106 | 2235 | 0.000017 | ly.rebrand |
422 | 19280870 | 1954 | 0.000019 | org.archlinux |
423 | 19280114 | 878 | 0.000035 | uk.co.pinterest |
424 | 19279216 | 2833 | 0.000013 | org.doctorswithoutborders |
425 | 19277590 | 2069 | 0.000018 | org.accessnow |
426 | 19272506 | 1482 | 0.000023 | com.findlaw |
427 | 19271938 | 800 | 0.000037 | net.clickbank |
428 | 19270876 | 3536 | 0.000010 | com.viki |
429 | 19269508 | 980 | 0.000033 | edu.brookings |
430 | 19265638 | 2363 | 0.000016 | co.carrd |
431 | 19265518 | 2950 | 0.000012 | org.neocities |
432 | 19261700 | 1294 | 0.000025 | com.wikia |
433 | 19260608 | 657 | 0.000044 | com.mashable |
434 | 19260346 | 798 | 0.000037 | com.thelancet |
435 | 19259074 | 974 | 0.000033 | uk.ac.cam |
436 | 19258964 | 2416 | 0.000016 | org.rsf |
437 | 19258786 | 1841 | 0.000019 | net.daringfireball |
438 | 19255684 | 708 | 0.000041 | com.canva |
439 | 19255674 | 495 | 0.000056 | gov.whitehouse |
440 | 19253374 | 204 | 0.000122 | com.salesforce |
441 | 19252328 | 1191 | 0.000027 | com.thenextweb |
442 | 19251934 | 2076 | 0.000018 | com.france24 |
443 | 19251914 | 534 | 0.000052 | io.codepen |
444 | 19250670 | 2896 | 0.000013 | com.laweekly |
445 | 19250196 | 1218 | 0.000026 | com.licdn |
446 | 19248894 | 2839 | 0.000013 | cc.uxdesign |
447 | 19245136 | 2368 | 0.000016 | edu.kit |
448 | 19240312 | 1071 | 0.000030 | watch.fb |
449 | 19239224 | 2583 | 0.000014 | org.scala-lang |
450 | 19238926 | 2513 | 0.000015 | au.com.theage |
451 | 19234622 | 2644 | 0.000014 | com.hubpages |
452 | 19232860 | 1533 | 0.000022 | ch.ipcc |
453 | 19232358 | 1092 | 0.000029 | com.digitaloceanspaces |
454 | 19231078 | 2435 | 0.000015 | org.vim |
455 | 19227408 | 1721 | 0.000021 | com.refinery29 |
456 | 19227102 | 292 | 0.000093 | net.secureservercdn |
457 | 19225996 | 769 | 0.000038 | com.marketwatch |
458 | 19224690 | 2608 | 0.000014 | app.web |
459 | 19224666 | 1681 | 0.000022 | org.unwomen |
460 | 19223622 | 2447 | 0.000015 | com.fineartamerica |
461 | 19222710 | 2566 | 0.000015 | nl.blogspot |
462 | 19219678 | 537 | 0.000051 | edu.cmu |
463 | 19219662 | 505 | 0.000054 | fr.free |
464 | 19217122 | 831 | 0.000036 | com.box |
465 | 19216944 | 1159 | 0.000027 | com.imageshack |
466 | 19215358 | 2409 | 0.000016 | edu.usf |
467 | 19214138 | 2396 | 0.000016 | nz.co.nzherald |
468 | 19212598 | 2758 | 0.000013 | com.smashwords |
469 | 19208544 | 364 | 0.000074 | net.datatables |
470 | 19207930 | 419 | 0.000066 | com.nationalgeographic |
471 | 19207832 | 287 | 0.000095 | com.iubenda |
472 | 19206220 | 2470 | 0.000015 | re.appsto |
473 | 19205746 | 235 | 0.000112 | com.adnxs |
474 | 19203826 | 2509 | 0.000015 | org.gentoo |
475 | 19203684 | 1982 | 0.000018 | com.voanews |
476 | 19203064 | 2498 | 0.000015 | com.superuser |
477 | 19201090 | 379 | 0.000072 | com.businesswire |
478 | 19200162 | 643 | 0.000045 | int.wipo |
479 | 19199426 | 2201 | 0.000018 | org.biorxiv |
480 | 19198874 | 1451 | 0.000023 | org.amnesty |
481 | 19198454 | 2436 | 0.000015 | com.oregonlive |
482 | 19198448 | 2033 | 0.000018 | org.nobelprize |
483 | 19197824 | 81 | 0.000414 | net.jsfiddle |
484 | 19197384 | 1963 | 0.000019 | com.ew |
485 | 19195648 | 1008 | 0.000031 | com.arstechnica |
486 | 19194700 | 1826 | 0.000020 | org.ocks |
487 | 19194656 | 265 | 0.000101 | com.aliyuncs |
488 | 19192314 | 2649 | 0.000014 | com.dezeen |
489 | 19191524 | 2678 | 0.000014 | org.transparency |
490 | 19191474 | 925 | 0.000035 | org.mediawiki |
491 | 19191020 | 2853 | 0.000013 | com.scribblelive |
492 | 19190866 | 1357 | 0.000024 | io.gitlab |
493 | 19190776 | 1683 | 0.000021 | org.aiga |
494 | 19188994 | 1985 | 0.000018 | uk.gov.tfl |
495 | 19186250 | 428 | 0.000064 | com.adweek |
496 | 19185528 | 1941 | 0.000019 | org.unep |
497 | 19185444 | 430 | 0.000064 | org.js |
498 | 19184030 | 371 | 0.000073 | com.atlassian |
499 | 19183274 | 2282 | 0.000017 | com.foreignpolicy |
500 | 19181288 | 2696 | 0.000014 | org.democracynow |
501 | 19181160 | 985 | 0.000032 | com.webs |
502 | 19181124 | 1198 | 0.000026 | com.wetransfer |
503 | 19179532 | 1369 | 0.000024 | org.altervista |
504 | 19177814 | 2501 | 0.000015 | google.research |
505 | 19177102 | 2959 | 0.000012 | za.co.iol |
506 | 19175572 | 1028 | 0.000031 | com.slate |
507 | 19175528 | 2713 | 0.000014 | org.cpj |
508 | 19174456 | 2016 | 0.000018 | org.example |
509 | 19173782 | 2390 | 0.000016 | com.googlegroups |
510 | 19167902 | 243 | 0.000107 | com.naver |
511 | 19164304 | 2266 | 0.000017 | net.openid |
512 | 19164154 | 3058 | 0.000012 | com.deepmind |
513 | 19164022 | 269 | 0.000099 | org.drupal |
514 | 19163716 | 264 | 0.000101 | gov.ca |
515 | 19163714 | 414 | 0.000067 | com.livechatinc |
516 | 19155586 | 2328 | 0.000016 | com.washingtontimes |
517 | 19153082 | 637 | 0.000045 | com.cbsnews |
518 | 19151824 | 759 | 0.000038 | com.oreilly |
519 | 19150012 | 3034 | 0.000012 | com.podomatic |
520 | 19149538 | 664 | 0.000043 | gov.loc |
521 | 19147634 | 136 | 0.000208 | org.networkadvertising |
522 | 19146702 | 718 | 0.000041 | com.buzzfeed |
523 | 19144896 | 1333 | 0.000024 | link.page |
524 | 19143440 | 838 | 0.000036 | com.pcmag |
525 | 19142224 | 956 | 0.000034 | com.verisign |
526 | 19133476 | 2892 | 0.000013 | com.thoughtworks |
527 | 19132758 | 2669 | 0.000014 | uk.co.timesonline |
528 | 19131302 | 274 | 0.000097 | com.getbootstrap |
529 | 19130872 | 3082 | 0.000012 | com.mariadb |
530 | 19129650 | 1116 | 0.000028 | com.jekyllrb |
531 | 19128992 | 938 | 0.000034 | com.vox |
532 | 19127508 | 127 | 0.000234 | info.aboutads |
533 | 19122362 | 492 | 0.000056 | com.patreon |
534 | 19122034 | 2488 | 0.000015 | com.curbed |
535 | 19121736 | 514 | 0.000054 | it.placehold |
536 | 19121622 | 1807 | 0.000020 | com.ascentlawfirm |
537 | 19118802 | 227 | 0.000114 | to.amzn |
538 | 19117462 | 580 | 0.000047 | com.visualstudio |
539 | 19117426 | 1281 | 0.000025 | com.smashingmagazine |
540 | 19116926 | 499 | 0.000055 | com.sxsw |
541 | 19116922 | 978 | 0.000033 | com.hootsuite |
542 | 19116384 | 282 | 0.000095 | gov.ftc |
543 | 19113570 | 2336 | 0.000016 | com.snopes |
544 | 19110698 | 1392 | 0.000023 | com.upwork |
545 | 19109920 | 1388 | 0.000024 | com.haaretz |
546 | 19108142 | 1701 | 0.000021 | com.firebaseapp |
547 | 19104686 | 920 | 0.000035 | com.zoho |
548 | 19103768 | 3023 | 0.000012 | org.peta |
549 | 19100220 | 1174 | 0.000027 | com.att |
550 | 19097502 | 1687 | 0.000021 | com.techrepublic |
551 | 19097348 | 1776 | 0.000020 | com.surveygizmo |
552 | 19097050 | 2394 | 0.000016 | com.treehugger |
553 | 19095294 | 3359 | 0.000011 | com.letterboxd |
554 | 19094856 | 3025 | 0.000012 | gov.anl |
555 | 19094436 | 2568 | 0.000015 | com.kaggle |
556 | 19091430 | 2479 | 0.000015 | fm.omny |
557 | 19091422 | 2999 | 0.000012 | com.bangkokpost |
558 | 19089542 | 478 | 0.000058 | gov.irs |
559 | 19086962 | 1790 | 0.000020 | ca.bc.gov |
560 | 19086880 | 823 | 0.000036 | com.emarketer |
561 | 19086854 | 1173 | 0.000027 | com.mediaplex |
562 | 19086370 | 259 | 0.000103 | uk.co.amazon |
563 | 19084140 | 2737 | 0.000014 | int.au |
564 | 19083740 | 2514 | 0.000015 | no.google |
565 | 19083612 | 3618 | 0.000010 | com.newgrounds |
566 | 19083596 | 186 | 0.000134 | jp.co.yahoo |
567 | 19082852 | 3108 | 0.000012 | org.hypotheses |
568 | 19082332 | 337 | 0.000082 | mp.mailchi |
569 | 19082228 | 2808 | 0.000013 | com.usmagazine |
570 | 19080648 | 1674 | 0.000022 | com.routledge |
571 | 19078904 | 2881 | 0.000013 | org.polymer-project |
572 | 19077732 | 2446 | 0.000015 | org.unctad |
573 | 19077060 | 180 | 0.000140 | com.caniuse |
574 | 19076142 | 344 | 0.000080 | com.onesignal |
575 | 19073714 | 2714 | 0.000014 | int.interpol |
576 | 19072522 | 3248 | 0.000011 | org.elasticsearch |
577 | 19072218 | 620 | 0.000045 | com.entrepreneur |
578 | 19072108 | 2550 | 0.000015 | uk.gov.metoffice |
579 | 19071016 | 2921 | 0.000013 | org.jenkins-ci |
580 | 19070052 | 632 | 0.000045 | com.samsung |
581 | 19070008 | 1367 | 0.000024 | org.unicode |
582 | 19069412 | 3381 | 0.000011 | uk.mod |
583 | 19069358 | 2715 | 0.000014 | org.mozillazine |
584 | 19067032 | 3148 | 0.000011 | edu.ucpress |
585 | 19066706 | 1156 | 0.000027 | com.gizmodo |
586 | 19064356 | 1901 | 0.000019 | org.americanbar |
587 | 19063358 | 3593 | 0.000010 | org.scala-sbt |
588 | 19062892 | 330 | 0.000084 | ai.shortpixel |
589 | 19059914 | 2912 | 0.000013 | in.indiatoday |
590 | 19058702 | 301 | 0.000090 | gg.discord |
591 | 19058544 | 3687 | 0.000010 | jp.riken |
592 | 19056820 | 2440 | 0.000015 | com.timesofisrael |
593 | 19056518 | 3064 | 0.000012 | com.manta |
594 | 19056166 | 785 | 0.000037 | com.fandom |
595 | 19056052 | 1274 | 0.000025 | com.sfgate |
596 | 19054812 | 1994 | 0.000018 | com.knightlab |
597 | 19053684 | 2042 | 0.000018 | org.donorbox |
598 | 19053668 | 2251 | 0.000017 | eu.politico |
599 | 19051874 | 2494 | 0.000015 | org.gnupg |
600 | 19051168 | 84 | 0.000402 | me.ogp |
601 | 19050784 | 575 | 0.000048 | com.cisco |
602 | 19050146 | 2757 | 0.000013 | uk.ac.york |
603 | 19048588 | 1189 | 0.000027 | com.buffer |
604 | 19048534 | 3065 | 0.000012 | uk.org.wwf |
605 | 19047192 | 929 | 0.000035 | com.variety |
606 | 19045218 | 3517 | 0.000010 | com.flightradar24 |
607 | 19044416 | 3387 | 0.000011 | com.flock |
608 | 19044368 | 579 | 0.000048 | com.sedo |
609 | 19044084 | 871 | 0.000035 | com.libsyn |
610 | 19043162 | 2561 | 0.000015 | com.thenation |
611 | 19042992 | 2569 | 0.000015 | com.monday |
612 | 19042226 | 482 | 0.000057 | com.arcgis |
613 | 19042098 | 3223 | 0.000011 | net.inquirer |
614 | 19038824 | 2699 | 0.000014 | com.real |
615 | 19036762 | 2195 | 0.000018 | com.secondlife |
616 | 19035414 | 732 | 0.000040 | org.unesco |
617 | 19035292 | 932 | 0.000035 | com.wikihow |
618 | 19034196 | 2609 | 0.000014 | uk.ac.leeds |
619 | 19033676 | 80 | 0.000415 | com.livestream |
620 | 19031566 | 3100 | 0.000012 | org.cato |
621 | 19029288 | 2716 | 0.000014 | org.sonatype |
622 | 19028098 | 3169 | 0.000011 | com.intensedebate |
623 | 19028006 | 1089 | 0.000029 | com.symantec |
624 | 19027462 | 4048 | 0.000009 | org.jw |
625 | 19027208 | 2793 | 0.000013 | com.wayfair |
626 | 19026864 | 1922 | 0.000019 | com.scene7 |
627 | 19026030 | 76 | 0.000421 | com.messenger |
628 | 19025656 | 1689 | 0.000021 | org.coursera |
629 | 19024592 | 1196 | 0.000026 | edu.umn |
630 | 19024030 | 3093 | 0.000012 | org.rferl |
631 | 19023590 | 2512 | 0.000015 | org.wikidata |
632 | 19022464 | 644 | 0.000045 | com.psychologytoday |
633 | 19021606 | 3091 | 0.000012 | com.vancouversun |
634 | 19021016 | 2496 | 0.000015 | uk.org.nationaltrust |
635 | 19020356 | 1553 | 0.000022 | ly.ow |
636 | 19020300 | 1264 | 0.000025 | edu.ucsd |
637 | 19020198 | 2905 | 0.000013 | tr.com.aa |
638 | 19018958 | 3767 | 0.000010 | it.polito |
639 | 19017674 | 3389 | 0.000011 | org.sourcewatch |
640 | 19017264 | 3207 | 0.000011 | ch.qos |
641 | 19017144 | 3785 | 0.000010 | jp.ac.kobe-u |
642 | 19016500 | 1659 | 0.000022 | com.speakerdeck |
643 | 19015604 | 2974 | 0.000012 | com.sciencealert |
644 | 19015034 | 636 | 0.000045 | com.photobucket |
645 | 19015026 | 3087 | 0.000012 | com.hsbc |
646 | 19014412 | 3852 | 0.000009 | edu.uah |
647 | 19013454 | 188 | 0.000133 | com.jimcdn |
648 | 19012700 | 1195 | 0.000027 | com.rollingstone |
649 | 19012558 | 2897 | 0.000013 | org.osce |
650 | 19012150 | 4205 | 0.000009 | com.gust |
651 | 19009874 | 1684 | 0.000021 | org.webkit |
652 | 19009396 | 945 | 0.000034 | com.shutterstock |
653 | 19008674 | 2945 | 0.000012 | com.townnews |
654 | 19008318 | 2650 | 0.000014 | org.wri |
655 | 19006196 | 520 | 0.000053 | com.inc |
656 | 19004672 | 648 | 0.000044 | com.gartner |
657 | 19004226 | 2502 | 0.000015 | ru.rg |
658 | 19003942 | 2648 | 0.000014 | io.bower |
659 | 19003780 | 3371 | 0.000011 | net.thedailystar |
660 | 19003722 | 2564 | 0.000015 | net.dwcdn |
661 | 19003574 | 2770 | 0.000013 | com.articulate |
662 | 19003510 | 221 | 0.000117 | com.myshopify |
663 | 19002868 | 172 | 0.000151 | jp.co.google |
664 | 19002548 | 1354 | 0.000024 | gov.uspto |
665 | 19000824 | 998 | 0.000032 | edu.ucla |
666 | 18998632 | 685 | 0.000042 | com.investopedia |
667 | 18998502 | 3161 | 0.000011 | com.mongabay |
668 | 18997700 | 532 | 0.000052 | com.aol |
669 | 18994210 | 2352 | 0.000016 | ca.citizenlab |
670 | 18993878 | 1284 | 0.000025 | com.today |
671 | 18992626 | 160 | 0.000167 | org.whatwg |
672 | 18992320 | 1048 | 0.000030 | com.smartadserver |
673 | 18992054 | 2995 | 0.000012 | org.pewforum |
674 | 18991266 | 2997 | 0.000012 | org.sierraclub |
675 | 18991062 | 2964 | 0.000012 | net.vnexpress |
676 | 18990848 | 1069 | 0.000030 | com.about |
677 | 18989274 | 3081 | 0.000012 | uk.co.spectator |
678 | 18988482 | 480 | 0.000058 | com.dmca |
679 | 18987042 | 1462 | 0.000023 | ly.cutt |
680 | 18986890 | 3224 | 0.000011 | ru.interfax |
681 | 18986546 | 3670 | 0.000010 | uk.co.zoopla |
682 | 18985004 | 2843 | 0.000013 | org.iucnredlist |
683 | 18984682 | 194 | 0.000130 | com.tripadvisor |
684 | 18984256 | 3473 | 0.000010 | fm.audioboo |
685 | 18983654 | 2762 | 0.000013 | uk.co.bbci |
686 | 18983098 | 3416 | 0.000011 | edu.sjsu |
687 | 18982290 | 1444 | 0.000023 | edu.northwestern |
688 | 18982040 | 543 | 0.000051 | com.googleoptimize |
689 | 18980160 | 2817 | 0.000013 | int.iom |
690 | 18979598 | 1386 | 0.000024 | edu.umd |
691 | 18978832 | 629 | 0.000045 | org.eff |
692 | 18978224 | 2297 | 0.000017 | uk.org.ofcom |
693 | 18978038 | 2450 | 0.000015 | int.reliefweb |
694 | 18977708 | 3502 | 0.000010 | com.torontosun |
695 | 18975348 | 493 | 0.000056 | com.indeed |
696 | 18973238 | 1657 | 0.000022 | com.nngroup |
697 | 18972898 | 351 | 0.000078 | com.constantcontact |
698 | 18972774 | 1780 | 0.000020 | co.lpages |
699 | 18972434 | 1329 | 0.000024 | edu.utexas |
700 | 18971302 | 3386 | 0.000011 | com.iconarchive |
701 | 18971240 | 312 | 0.000087 | com.pubmatic |
702 | 18971078 | 1043 | 0.000030 | org.reactjs |
703 | 18969428 | 975 | 0.000033 | edu.umich |
704 | 18968428 | 1219 | 0.000026 | com.tableau |
705 | 18968208 | 1914 | 0.000019 | com.hatenablog |
706 | 18967574 | 1138 | 0.000028 | com.chicagotribune |
707 | 18967196 | 3999 | 0.000009 | info.spain |
708 | 18965740 | 547 | 0.000050 | gov.copyright |
709 | 18965442 | 4171 | 0.000009 | org.gwtproject |
710 | 18964656 | 650 | 0.000044 | com.netflix |
711 | 18963714 | 753 | 0.000039 | net.adform |
712 | 18961462 | 2847 | 0.000013 | uk.ac.jisc |
713 | 18961172 | 2865 | 0.000013 | com.ringcentral |
714 | 18960292 | 976 | 0.000033 | com.redhat |
715 | 18960280 | 3237 | 0.000011 | com.city-data |
716 | 18959938 | 2982 | 0.000012 | uk.org.stonewall |
717 | 18958764 | 3646 | 0.000010 | za.co.timeslive |
718 | 18957560 | 3989 | 0.000009 | com.programmableweb |
719 | 18957204 | 403 | 0.000068 | com.bigcommerce |
720 | 18957166 | 3053 | 0.000012 | com.flippa |
721 | 18955710 | 3293 | 0.000011 | com.multiscreensite |
722 | 18955294 | 2719 | 0.000014 | com.bloglines |
723 | 18954560 | 2629 | 0.000014 | mp.j |
724 | 18952520 | 2807 | 0.000013 | uk.org.rspb |
725 | 18952358 | 2949 | 0.000012 | com.foreignaffairs |
726 | 18952314 | 2206 | 0.000018 | co.pcdn |
727 | 18951462 | 3549 | 0.000010 | in.theprint |
728 | 18951006 | 4227 | 0.000009 | com.symbaloo |
729 | 18950992 | 4478 | 0.000008 | com.algorithmia |
730 | 18950356 | 1733 | 0.000021 | com.billboard |
731 | 18949998 | 742 | 0.000039 | com.splashthat |
732 | 18948332 | 3255 | 0.000011 | com.cleantechnica |
733 | 18947926 | 3604 | 0.000010 | com.businessdailyafrica |
734 | 18947204 | 1108 | 0.000028 | com.dell |
735 | 18947190 | 2825 | 0.000013 | com.yell |
736 | 18947008 | 443 | 0.000062 | net.hubspot |
737 | 18946944 | 3838 | 0.000010 | org.rfa |
738 | 18946618 | 3495 | 0.000010 | za.co.mg |
739 | 18945038 | 4346 | 0.000008 | com.apsense |
740 | 18944960 | 1772 | 0.000020 | com.alibabagroup |
741 | 18944660 | 2267 | 0.000017 | to.dev |
742 | 18944558 | 3471 | 0.000010 | ru.mid |
743 | 18944274 | 3498 | 0.000010 | com.itsnicethat |
744 | 18942634 | 527 | 0.000052 | org.unicef |
745 | 18942358 | 2364 | 0.000016 | net.noscript |
746 | 18940606 | 1390 | 0.000024 | com.techradar |
747 | 18938348 | 1857 | 0.000019 | edu.uci |
748 | 18937064 | 1118 | 0.000028 | com.windowsphone |
749 | 18936626 | 2734 | 0.000014 | com.doubleclickbygoogle |
750 | 18936326 | 3524 | 0.000010 | org.350 |
751 | 18935080 | 3076 | 0.000012 | org.aei |
752 | 18934558 | 3075 | 0.000012 | gov.arts |
753 | 18934248 | 671 | 0.000043 | gov.sec |
754 | 18933682 | 2298 | 0.000017 | com.urbandictionary |
755 | 18933596 | 3925 | 0.000009 | com.forbesimg |
756 | 18933310 | 487 | 0.000056 | com.fc2 |
757 | 18931436 | 3352 | 0.000011 | com.brill |
758 | 18931406 | 2491 | 0.000015 | com.infoworld |
759 | 18930782 | 1352 | 0.000024 | com.bazaarvoice |
760 | 18930350 | 3600 | 0.000010 | de.uni-konstanz |
761 | 18930232 | 1187 | 0.000027 | com.alexa |
762 | 18929982 | 2240 | 0.000017 | org.linuxfoundation |
763 | 18929738 | 3580 | 0.000010 | edu.dukeupress |
764 | 18929218 | 4038 | 0.000009 | com.hotfrog |
765 | 18928884 | 512 | 0.000054 | com.mckinsey |
766 | 18928710 | 2539 | 0.000015 | org.crossref |
767 | 18928328 | 3893 | 0.000009 | com.environmentalleader |
768 | 18927900 | 2236 | 0.000017 | tv.ustream |
769 | 18927290 | 1101 | 0.000029 | fm.last |
770 | 18926906 | 1951 | 0.000019 | com.businessweek |
771 | 18926816 | 413 | 0.000067 | org.opensource |
772 | 18925308 | 750 | 0.000039 | org.whatbrowser |
773 | 18925012 | 1296 | 0.000025 | com.merriam-webster |
774 | 18924400 | 425 | 0.000065 | com.proofpoint |
775 | 18923302 | 3196 | 0.000011 | com.alchemer |
776 | 18922680 | 3684 | 0.000010 | com.arfadia |
777 | 18922120 | 1692 | 0.000021 | com.kinstacdn |
778 | 18921334 | 3619 | 0.000010 | com.ecowatch |
779 | 18921302 | 2215 | 0.000018 | net.leadpages |
780 | 18920188 | 3448 | 0.000010 | com.total |
781 | 18920112 | 3878 | 0.000009 | uk.org.npg |
782 | 18919836 | 3054 | 0.000012 | io.crates |
783 | 18919320 | 2517 | 0.000015 | com.lego |
784 | 18919150 | 503 | 0.000055 | com.wufoo |
785 | 18915478 | 2742 | 0.000014 | io.redis |
786 | 18914956 | 2289 | 0.000017 | uk.co.metro |
787 | 18914016 | 4161 | 0.000009 | uk.co.theweek |
788 | 18913930 | 2204 | 0.000018 | gd.is |
789 | 18913640 | 4196 | 0.000009 | io.coda |
790 | 18913482 | 196 | 0.000128 | com.hackerone |
791 | 18912094 | 1483 | 0.000023 | com.msdn |
792 | 18911500 | 156 | 0.000170 | org.nginx |
793 | 18911390 | 3197 | 0.000011 | com.klokantech |
794 | 18911286 | 1668 | 0.000022 | com.sky |
795 | 18910142 | 4272 | 0.000009 | de.fernuni-hagen |
796 | 18909064 | 1832 | 0.000020 | de.hessen |
797 | 18908834 | 953 | 0.000034 | com.adroll |
798 | 18907960 | 1986 | 0.000018 | com.windows |
799 | 18906254 | 4316 | 0.000008 | com.tupalo |
800 | 18904242 | 218 | 0.000118 | org.icann |
801 | 18904004 | 1148 | 0.000028 | net.atlassian |
802 | 18903442 | 4234 | 0.000009 | net.ccm |
803 | 18903372 | 3808 | 0.000010 | com.oilprice |
804 | 18903128 | 1956 | 0.000019 | org.khanacademy |
805 | 18902896 | 4072 | 0.000009 | net.iwpr |
806 | 18902418 | 324 | 0.000084 | eu.youronlinechoices |
807 | 18902318 | 3890 | 0.000009 | uk.ac.mmu |
808 | 18901968 | 1700 | 0.000021 | edu.usc |
809 | 18901674 | 1146 | 0.000028 | com.playstation |
810 | 18900854 | 4268 | 0.000009 | uk.ac.ceh |
811 | 18900236 | 1060 | 0.000030 | com.akamai |
812 | 18897754 | 2851 | 0.000013 | com.hindustantimes |
813 | 18896694 | 979 | 0.000033 | gov.fcc |
814 | 18896368 | 990 | 0.000032 | com.gumroad |
815 | 18896128 | 4383 | 0.000008 | et.com.google |
816 | 18894652 | 3952 | 0.000009 | com.theoutline |
817 | 18894586 | 4348 | 0.000008 | org.cgsociety |
818 | 18892502 | 4093 | 0.000009 | edu.mtsu |
819 | 18892446 | 2636 | 0.000014 | com.html5rocks |
820 | 18892222 | 4578 | 0.000008 | com.blockchair |
821 | 18891572 | 3603 | 0.000010 | org.spie |
822 | 18891278 | 1208 | 0.000026 | at.gv.bka |
823 | 18890814 | 3768 | 0.000010 | uk.co.lrb |
824 | 18888196 | 410 | 0.000067 | com.heroku |
825 | 18888062 | 815 | 0.000036 | edu.wisc |
826 | 18887938 | 1009 | 0.000031 | com.yoast |
827 | 18887842 | 3807 | 0.000010 | za.co.dailymaverick |
828 | 18885614 | 1864 | 0.000019 | org.json |
829 | 18885194 | 2969 | 0.000012 | org.thinkprogress |
830 | 18884928 | 700 | 0.000041 | com.feedly |
831 | 18883562 | 4278 | 0.000009 | com.ingress |
832 | 18883012 | 3706 | 0.000010 | google.design |
833 | 18882952 | 4666 | 0.000008 | com.bmwblog |
834 | 18881526 | 3862 | 0.000009 | com.thepetitionsite |
835 | 18881312 | 3993 | 0.000009 | in.bbc |
836 | 18880860 | 2237 | 0.000017 | com.w3techs |
837 | 18880532 | 3257 | 0.000011 | org.carbonbrief |
838 | 18880428 | 272 | 0.000098 | jp.ne.hatena |
839 | 18880124 | 2614 | 0.000014 | ru.mk |
840 | 18880064 | 1924 | 0.000019 | edu.hbs |
841 | 18879486 | 935 | 0.000034 | com.pingdom |
842 | 18878780 | 1199 | 0.000026 | com.ycombinator |
843 | 18876422 | 4065 | 0.000009 | com.gifer |
844 | 18876050 | 3709 | 0.000010 | uk.org.amnesty |
845 | 18875700 | 3961 | 0.000009 | com.africanews |
846 | 18875478 | 4535 | 0.000008 | com.the-dots |
847 | 18875350 | 992 | 0.000032 | so.notion |
848 | 18869944 | 3375 | 0.000011 | org.commondreams |
849 | 18869250 | 4236 | 0.000009 | com.flutterwave |
850 | 18868302 | 3731 | 0.000010 | org.refworld |
851 | 18866446 | 3285 | 0.000011 | uk.gov.charitycommission |
852 | 18866244 | 4396 | 0.000008 | com.newsru |
853 | 18865810 | 3507 | 0.000010 | uk.org.oxfam |
854 | 18865786 | 4206 | 0.000009 | uk.org.somersethouse |
855 | 18864590 | 2484 | 0.000015 | in.scroll |
856 | 18864514 | 1410 | 0.000023 | com.intuit |
857 | 18864428 | 4257 | 0.000009 | uk.co.harpercollins |
858 | 18863942 | 331 | 0.000084 | jp.ameblo |
859 | 18863622 | 3370 | 0.000011 | ke.co.nation |
860 | 18863580 | 1091 | 0.000029 | com.insurancejournal |
861 | 18863360 | 3394 | 0.000011 | com.cbsistatic |
862 | 18863302 | 2697 | 0.000014 | com.spreaker |
863 | 18862358 | 2720 | 0.000014 | com.springernature |
864 | 18861824 | 2228 | 0.000017 | com.firefox |
865 | 18861222 | 4688 | 0.000008 | co.iglobal |
866 | 18860960 | 4714 | 0.000008 | io.devdocs |
867 | 18860246 | 2705 | 0.000014 | com.verywellhealth |
868 | 18860020 | 538 | 0.000051 | com.booking |
869 | 18859838 | 535 | 0.000051 | com.gofundme |
870 | 18859798 | 1234 | 0.000026 | com.indiegogo |
871 | 18859328 | 4781 | 0.000008 | com.kdpcommunity |
872 | 18858404 | 2344 | 0.000016 | build.bazel |
873 | 18858016 | 1119 | 0.000028 | com.foursquare |
874 | 18857452 | 545 | 0.000051 | com.snapchat |
875 | 18856948 | 93 | 0.000390 | com.trustpilot |
876 | 18856546 | 2586 | 0.000014 | com.avast |
877 | 18856160 | 1813 | 0.000020 | com.pcworld |
878 | 18855278 | 4149 | 0.000009 | com.hybris |
879 | 18854982 | 4937 | 0.000008 | com.jetphotos |
880 | 18854014 | 711 | 0.000041 | com.yandex |
881 | 18853962 | 1023 | 0.000031 | com.css-tricks |
882 | 18853902 | 1360 | 0.000024 | org.golang |
883 | 18853900 | 4168 | 0.000009 | uk.ac.mdx |
884 | 18853596 | 2673 | 0.000014 | com.flipboard |
885 | 18852988 | 2848 | 0.000013 | com.discovery |
886 | 18851822 | 3904 | 0.000009 | at.kleinezeitung |
887 | 18851422 | 302 | 0.000090 | de.amazon |
888 | 18850596 | 111 | 0.000276 | me.wa |
889 | 18850558 | 429 | 0.000064 | com.skype |
890 | 18850294 | 1134 | 0.000028 | com.scientificamerican |
891 | 18848646 | 2516 | 0.000015 | org.raspberrypi |
892 | 18847770 | 4426 | 0.000008 | com.armorgames |
893 | 18847596 | 1758 | 0.000020 | com.fiverr |
894 | 18847110 | 721 | 0.000040 | org.iso |
895 | 18846464 | 2707 | 0.000014 | com.codecademy |
896 | 18843948 | 3727 | 0.000010 | net.middleeasteye |
897 | 18842876 | 3057 | 0.000012 | org.man7 |
898 | 18841074 | 4531 | 0.000008 | com.e-estonia |
899 | 18840474 | 1827 | 0.000020 | fr.blogspot |
900 | 18840282 | 1047 | 0.000030 | com.huffpost |
901 | 18839944 | 4725 | 0.000008 | net.gebco |
902 | 18839822 | 4778 | 0.000008 | com.slite |
903 | 18839776 | 1799 | 0.000020 | com.visa |
904 | 18839258 | 722 | 0.000040 | com.newrelic |
905 | 18837214 | 4163 | 0.000009 | com.cnsnews |
906 | 18836004 | 807 | 0.000037 | br.com.uol |
907 | 18834494 | 3388 | 0.000011 | com.lithub |
908 | 18834036 | 3942 | 0.000009 | net.bostonreview |
909 | 18832052 | 1022 | 0.000031 | au.com.google |
910 | 18831616 | 2958 | 0.000012 | com.hackernoon |
911 | 18831490 | 2802 | 0.000013 | com.unity |
912 | 18831230 | 249 | 0.000106 | net.2mdn |
913 | 18831090 | 1224 | 0.000026 | gov.usgs |
914 | 18830714 | 400 | 0.000068 | com.semrush |
915 | 18829732 | 3855 | 0.000009 | com.indexmundi |
916 | 18829642 | 516 | 0.000053 | com.dailymotion |
917 | 18828872 | 686 | 0.000042 | com.accenture |
918 | 18827314 | 665 | 0.000043 | org.poynter |
919 | 18826574 | 2086 | 0.000018 | org.aclu |
920 | 18826336 | 3974 | 0.000009 | org.jython |
921 | 18825792 | 1152 | 0.000027 | com.searchengineland |
922 | 18825212 | 4003 | 0.000009 | com.inthesetimes |
923 | 18824910 | 1753 | 0.000020 | com.over-blog |
924 | 18824406 | 518 | 0.000053 | nl.google |
925 | 18824128 | 3324 | 0.000011 | de.bfarm |
926 | 18823544 | 1126 | 0.000028 | com.techtarget |
927 | 18823106 | 4417 | 0.000008 | za.co.ewn |
928 | 18822694 | 4251 | 0.000009 | uk.co.bristolpost |
929 | 18822100 | 4799 | 0.000008 | community.studiopress |
930 | 18821800 | 846 | 0.000036 | gov.justice |
931 | 18820682 | 1707 | 0.000021 | com.technologyreview |
932 | 18820672 | 3923 | 0.000009 | com.recyclenow |
933 | 18819750 | 4293 | 0.000009 | lb.com.dailystar |
934 | 18819328 | 384 | 0.000071 | com.bitly |
935 | 18819088 | 4101 | 0.000009 | org.occrp |
936 | 18819058 | 3786 | 0.000010 | com.theyworkforyou |
937 | 18818798 | 4022 | 0.000009 | org.ifaw |
938 | 18818648 | 168 | 0.000161 | com.jimstatic |
939 | 18818242 | 966 | 0.000033 | sh.brew |
940 | 18818066 | 4343 | 0.000008 | com.yahoosites |
941 | 18817604 | 2321 | 0.000016 | com.fool |
942 | 18817360 | 2281 | 0.000017 | com.pastebin |
943 | 18816928 | 3607 | 0.000010 | com.gr-assets |
944 | 18815242 | 3872 | 0.000009 | com.climatechangenews |
945 | 18813886 | 4771 | 0.000008 | in.ac.iith |
946 | 18813586 | 702 | 0.000041 | org.plos |
947 | 18812812 | 4068 | 0.000009 | com.chamberofcommerce |
948 | 18812606 | 4438 | 0.000008 | us.tuugo |
949 | 18812344 | 1142 | 0.000028 | com.buzzsprout |
950 | 18812110 | 1122 | 0.000028 | com.timeanddate |
951 | 18811418 | 760 | 0.000038 | com.discordapp |
952 | 18811082 | 1153 | 0.000027 | com.sitepoint |
953 | 18809798 | 4579 | 0.000008 | com.desmogblog |
954 | 18809638 | 596 | 0.000047 | com.aliexpress |
955 | 18808906 | 2522 | 0.000015 | com.sendgrid |
956 | 18807972 | 4128 | 0.000009 | uk.ac.rcplondon |
957 | 18807934 | 2004 | 0.000018 | com.ssllabs |
958 | 18807738 | 3955 | 0.000009 | org.soilassociation |
959 | 18806592 | 1359 | 0.000024 | com.xkcd |
960 | 18806192 | 536 | 0.000051 | gov.hhs |
961 | 18805588 | 3584 | 0.000010 | com.hearstapps |
962 | 18805150 | 1289 | 0.000025 | com.searchenginejournal |
963 | 18804458 | 442 | 0.000062 | me.fb |
964 | 18804132 | 4711 | 0.000008 | net.sott |
965 | 18803628 | 4594 | 0.000008 | com.gpsvisualizer |
966 | 18803548 | 213 | 0.000120 | com.discord |
967 | 18802758 | 1820 | 0.000020 | org.mitre |
968 | 18801916 | 4744 | 0.000008 | com.natureindex |
969 | 18801374 | 3499 | 0.000010 | uk.org.rspca |
970 | 18801322 | 4087 | 0.000009 | org.c2es |
971 | 18801296 | 3824 | 0.000010 | com.qgiv |
972 | 18801288 | 3799 | 0.000010 | ug.co.monitor |
973 | 18800492 | 4596 | 0.000008 | com.lacartes |
974 | 18800384 | 178 | 0.000142 | com.xing |
975 | 18799966 | 3973 | 0.000009 | com.svbtle |
976 | 18798944 | 3875 | 0.000009 | uk.org.savethechildren |
977 | 18798932 | 4616 | 0.000008 | com.slurl |
978 | 18798332 | 2466 | 0.000015 | com.sophos |
979 | 18797956 | 2198 | 0.000018 | com.twilio |
980 | 18797008 | 4662 | 0.000008 | za.co.moneyweb |
981 | 18796288 | 4265 | 0.000009 | com.menafn |
982 | 18796048 | 3821 | 0.000010 | org.usip |
983 | 18795572 | 4584 | 0.000008 | com.power-technology |
984 | 18795368 | 4601 | 0.000008 | org.heartland |
985 | 18794834 | 675 | 0.000042 | com.usnews |
986 | 18794200 | 2557 | 0.000015 | org.usenix |
987 | 18793962 | 2899 | 0.000013 | net.privacypolicytemplate |
988 | 18793900 | 4081 | 0.000009 | org.theecologist |
989 | 18792970 | 4132 | 0.000009 | org.neweconomics |
990 | 18792680 | 1290 | 0.000025 | com.netlify |
991 | 18791954 | 4189 | 0.000009 | com.businessgreen |
992 | 18790752 | 4611 | 0.000008 | org.monthlyreview |
993 | 18790450 | 2369 | 0.000016 | uk.ac.ed |
994 | 18790194 | 2245 | 0.000017 | ch.ethz |
995 | 18789974 | 267 | 0.000099 | com.nielsen |
996 | 18789302 | 2674 | 0.000014 | ca.uwaterloo |
997 | 18788810 | 4284 | 0.000009 | org.unep-wcmc |
998 | 18787702 | 4042 | 0.000009 | org.ramsar |
999 | 18786952 | 4384 | 0.000008 | com.googlelabs |
1000 | 18786808 | 4634 | 0.000008 | org.berkeleyearth |
Credits
Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.
We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!
September 2021 crawl archive now available
The crawl archive for September 2021 is now available! The data was crawled Sept 16 – 29 and contains 2.95 billion web pages or 310 TiB of uncompressed content. It includes page captures of 1.3 billion new URLs, not visited in any of our prior crawls.
Archive Location and Download
The September crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-39/.
To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.
File List | #Files | Total Size Compressed (TiB) |
|
---|---|---|---|
Segments | CC-MAIN-2021-39/segment.paths.gz | 100 | |
WARC files | CC-MAIN-2021-39/warc.paths.gz | 72000 | 72.16 |
WAT files | CC-MAIN-2021-39/wat.paths.gz | 72000 | 19.81 |
WET files | CC-MAIN-2021-39/wet.paths.gz | 72000 | 8.7 |
Robots.txt files | CC-MAIN-2021-39/robotstxt.paths.gz | 72000 | 0.15 |
Non-200 responses files | CC-MAIN-2021-39/non200responses.paths.gz | 72000 | 2.19 |
URL index files | CC-MAIN-2021-39/cc-index.paths.gz | 302 | 0.22 |
The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-39/. Also the columnar index has been updated to contain this crawl.
Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.
July/August 2021 crawl archive now available
The crawl archive for July/August 2021 is now available! The data was crawled July 23 – August 6 and contains 3.15 billion web pages or 360 TiB of uncompressed content. It includes page captures of 1 billion new URLs, not visited in any of our prior crawls.
Improvements
Archiving of robots.txt files was improved. Robots.txt files are not archived if
- the robots.txt of the target host does not allow it (in case of a HTTP redirect) or
- URL filters exclude the entire site, eg. if it’s known ahead that a site does not allow crawling or
- the MIME type is not applicable for robots.txt files (eg. HTML, PDF)
More details are found in the corresponding issue report. The change reduces the size of the robots.txt subset (since August 2016) by removing content which should not contained in this dataset.
Archive Location and Download
The July/August crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-31/.
To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.
File List | #Files | Total Size Compressed (TiB) |
|
---|---|---|---|
Segments | CC-MAIN-2021-31/segment.paths.gz | 100 | |
WARC files | CC-MAIN-2021-31/warc.paths.gz | 72000 | 75.34 |
WAT files | CC-MAIN-2021-31/wat.paths.gz | 72000 | 21.67 |
WET files | CC-MAIN-2021-31/wet.paths.gz | 72000 | 9.43 |
Robots.txt files | CC-MAIN-2021-31/robotstxt.paths.gz | 72000 | 0.14 |
Non-200 responses files | CC-MAIN-2021-31/non200responses.paths.gz | 72000 | 1.98 |
URL index files | CC-MAIN-2021-31/cc-index.paths.gz | 302 | 0.23 |
The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-31/. Also the columnar index has been updated to contain this crawl.
Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.
June 2021 crawl archive now available
The crawl archive for June 2021 is now available! The data was crawled June 12 – 25 and contains 2.45 billion web pages or 260 TiB of uncompressed content. It includes page captures of 1.3 billion new URLs, not visited in any of our prior crawls.
Archive Location and Download
The June crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-25/.
To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.
File List | #Files | Total Size Compressed (TiB) |
|
---|---|---|---|
Segments | CC-MAIN-2021-25/segment.paths.gz | 100 | |
WARC files | CC-MAIN-2021-25/warc.paths.gz | 64000 | 57.9 |
WAT files | CC-MAIN-2021-25/wat.paths.gz | 64000 | 16.54 |
WET files | CC-MAIN-2021-25/wet.paths.gz | 64000 | 7.24 |
Robots.txt files | CC-MAIN-2021-25/robotstxt.paths.gz | 64000 | 0.16 |
Non-200 responses files | CC-MAIN-2021-25/non200responses.paths.gz | 64000 | 1.76 |
URL index files | CC-MAIN-2021-25/cc-index.paths.gz | 302 | 0.19 |
The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-25/. Also the columnar index has been updated to contain this crawl.
Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.
Host- and Domain-Level Web Graphs February/March, April and May 2021
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of February/March, April and May 2021. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases. You may also visit the projects cc-webgraph and cc-pyspark which include all scripts and tools required to construct the graphs. Instructions to explore the graphs in the webgraph format are given in our collection of webgraph notebooks.
What’s new?
The host-level graph now includes all hosts visited by the crawler even if there is no link pointing to the host and all visited URLs of a host failed (HTTP 404 and other error codes) or the host’s robots.txt does not allow crawling. Note that the links leading to these hosts may have been found in a prior crawl, not in one of the 3 crawls used to build this web graph.
Host-level graph
The graph consists of 515 million nodes and 2.82 billion edges. Both hyperlinks and HTTP redirects and link headers are used as edges to span up the graph. All types of links are included, including pure “technical” ones pointing to images, JavaScript libraries, web fonts, etc. However, only host names with a valid IANA TLD are used. Consequently, URLs with an IP address as host component are not taken into account for building the host-level graph.
There are 452 million dangling nodes (87.9%) and the largest strongly connected component contains 45.2 million (8.8%) nodes. Dangling nodes stem from
- hosts that have not been crawled, yet are pointed to from a link on a crawled page
- hosts without any links pointing to a different host name
- or hosts which did only return an error page (eg. HTTP 404)
Host names in the graph are in reverse domain name notation and a leading www.
is stripped: www.subdomain.example.com
becomes com.example.subdomain
.
You can download the graph and the ranks of all 515 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2021-feb-apr-may/host/
. Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2021-feb-apr-may/host/
as prefix to access the files from everywhere.
Please note that the text representation of the host-level graph is shipped in 72 gzip-compressed files listed in two path listings – one for the nodes (vertices), one for the edges (arcs). First, download the paths listing and decompress it using “gzip”. By adding the prefix s3://commoncrawl/
or https://data.commoncrawl.org/
to each line in the path listing you get the list of URLs to download the entire graph.
Download files of the Common Crawl Feb/Apr/May 2021 host-level webgraph
Size | File | Description |
---|---|---|
3.31 GB | cc-main-2021-feb-apr-may-host-vertices.paths.gz | nodes 〈id, rev host〉, paths of 24 vertices files |
12.94 GB | cc-main-2021-feb-apr-may-host-edges.paths.gz | edges 〈from_id, to_id〉, paths of 48 edges files |
5.57 GB | cc-main-2021-feb-apr-may-host.graph | graph in BVGraph format |
2 kB | cc-main-2021-feb-apr-may-host.properties | |
6.22 GB | cc-main-2021-feb-apr-may-host-t.graph | transpose of the graph (outlinks inverted to inlinks) |
2 kB | cc-main-2021-feb-apr-may-host-t.properties | |
1 kB | cc-main-2021-feb-apr-may-host.stats | WebGraph statistics |
7.69 GB | cc-main-2021-feb-apr-may-host-ranks.txt.gz | harmonic centrality and pagerank |
Domain-level graph
The domain graph is built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org.
The domain-level graph has 88 million nodes and 1.58 billion edges. 50% or 44 million nodes are dangling nodes, the largest strongly connected component covers 34 million or 39% of the nodes.
All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2021-feb-apr-may/domain/
resp. https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2021-feb-apr-may/domain/
.
Download files of the Common Crawl Feb/Apr/May 2021 domain-level webgraph
Size | File | Description |
---|---|---|
0.61 GB | cc-main-2021-feb-apr-may-domain-vertices.txt.gz | nodes 〈id, rev domain, num hosts〉 |
6.37 GB | cc-main-2021-feb-apr-may-domain-edges.txt.gz | edges 〈from_id, to_id〉 |
3.58 GB | cc-main-2021-feb-apr-may-domain.graph | graph in BVGraph format |
2 kB | cc-main-2021-feb-apr-may-domain.properties | |
3.42 GB | cc-main-2021-feb-apr-may-domain-t.graph | transpose of the graph |
2 kB | cc-main-2021-feb-apr-may-domain-t.properties | |
1 kB | cc-main-2021-feb-apr-may-domain.stats | WebGraph statistics |
1.89 GB | cc-main-2021-feb-apr-may-domain-ranks.txt.gz | harmonic centrality and pagerank |
Below you’ll find the top 1000 domains ranked by Harmonic Centrality or PageRank. The full list of all 88 million domain ranks is available for download.
Top 1000 domains ranked by harmonic centrality (Feb/Apr/May 2021)
harmonic centrality rank | hc value | page rank | page rank value | reversed domain name |
---|---|---|---|---|
1 | 31920934 | 1 | 0.017627 | com.googleapis |
2 | 31032784 | 3 | 0.013762 | com.facebook |
3 | 29681304 | 2 | 0.013832 | com.google |
4 | 27101692 | 4 | 0.007844 | com.twitter |
5 | 26954660 | 5 | 0.007519 | org.w |
6 | 26886624 | 7 | 0.006967 | com.youtube |
7 | 25515850 | 8 | 0.005718 | com.instagram |
8 | 25031490 | 6 | 0.007143 | com.googletagmanager |
9 | 24396116 | 9 | 0.005506 | org.gmpg |
10 | 23807122 | 12 | 0.003347 | com.linkedin |
11 | 22970992 | 13 | 0.003048 | com.gstatic |
12 | 22854052 | 10 | 0.003951 | com.cloudflare |
13 | 22698594 | 19 | 0.001914 | com.gravatar |
14 | 22504168 | 14 | 0.002908 | org.wordpress |
15 | 22434542 | 22 | 0.001564 | com.pinterest |
16 | 22100870 | 25 | 0.001270 | org.wikipedia |
17 | 21950578 | 17 | 0.002031 | com.wordpress |
18 | 21940826 | 18 | 0.001958 | com.apple |
19 | 21766696 | 15 | 0.002258 | com.bootstrapcdn |
20 | 21762964 | 30 | 0.001174 | com.vimeo |
21 | 21722198 | 38 | 0.000914 | be.youtu |
22 | 21556142 | 21 | 0.001842 | com.jquery |
23 | 21478118 | 29 | 0.001182 | com.microsoft |
24 | 21432212 | 53 | 0.000703 | com.blogspot |
25 | 21354260 | 35 | 0.001025 | com.amazonaws |
26 | 21337432 | 44 | 0.000765 | com.amazon |
27 | 21320702 | 43 | 0.000789 | gl.goo |
28 | 21170722 | 62 | 0.000600 | ly.bit |
29 | 21149628 | 99 | 0.000409 | com.tumblr |
30 | 21148242 | 50 | 0.000739 | com.wp |
31 | 21136818 | 45 | 0.000758 | org.mozilla |
32 | 21110018 | 57 | 0.000689 | eu.europa |
33 | 21104262 | 20 | 0.001894 | com.adobe |
34 | 21048760 | 16 | 0.002200 | com.github |
35 | 21040284 | 34 | 0.001026 | com.google-analytics |
36 | 21027350 | 36 | 0.001015 | net.jsdelivr |
37 | 20998320 | 27 | 0.001218 | com.wixstatic |
38 | 20995232 | 31 | 0.001119 | net.cloudfront |
39 | 20946148 | 47 | 0.000744 | com.flickr |
40 | 20913104 | 107 | 0.000338 | com.yahoo |
41 | 20851316 | 83 | 0.000436 | com.googleusercontent |
42 | 20843068 | 37 | 0.000929 | io.github |
43 | 20840670 | 111 | 0.000317 | com.reddit |
44 | 20834398 | 58 | 0.000677 | com.paypal |
45 | 20816886 | 23 | 0.001554 | com.fontawesome |
46 | 20773582 | 103 | 0.000368 | com.weebly |
47 | 20764576 | 79 | 0.000455 | com.medium |
48 | 20764512 | 33 | 0.001035 | com.googlesyndication |
49 | 20757582 | 32 | 0.001118 | ru.yandex |
50 | 20741944 | 48 | 0.000743 | com.whatsapp |
51 | 20708152 | 68 | 0.000520 | org.w3 |
52 | 20705826 | 132 | 0.000240 | com.nytimes |
53 | 20696906 | 59 | 0.000673 | co.t |
54 | 20678088 | 102 | 0.000375 | org.creativecommons |
55 | 20675822 | 115 | 0.000290 | com.soundcloud |
56 | 20644978 | 60 | 0.000624 | org.schema |
57 | 20627114 | 74 | 0.000479 | com.shopify |
58 | 20621162 | 66 | 0.000543 | com.vk |
59 | 20604726 | 181 | 0.000149 | org.wikimedia |
60 | 20604724 | 147 | 0.000204 | com.dropbox |
61 | 20579720 | 55 | 0.000702 | com.addthis |
62 | 20572950 | 138 | 0.000211 | org.archive |
63 | 20570610 | 198 | 0.000133 | com.cnn |
64 | 20558114 | 152 | 0.000187 | gov.cdc |
65 | 20550306 | 80 | 0.000446 | me.wp |
66 | 20538816 | 193 | 0.000136 | com.imgur |
67 | 20530078 | 49 | 0.000740 | net.doubleclick |
68 | 20512294 | 199 | 0.000133 | uk.co.bbc |
69 | 20505964 | 200 | 0.000133 | net.slideshare |
70 | 20499864 | 171 | 0.000155 | com.theguardian |
71 | 20489756 | 158 | 0.000175 | int.who |
72 | 20482256 | 120 | 0.000263 | com.spotify |
73 | 20481118 | 175 | 0.000151 | com.bing |
74 | 20478320 | 213 | 0.000124 | com.businessinsider |
75 | 20477478 | 253 | 0.000104 | com.bloomberg |
76 | 20477300 | 144 | 0.000206 | gov.nih |
77 | 20473648 | 46 | 0.000748 | com.macromedia |
78 | 20440520 | 254 | 0.000103 | com.wsj |
79 | 20434320 | 224 | 0.000118 | edu.stanford |
80 | 20419762 | 41 | 0.000847 | net.fbcdn |
81 | 20417930 | 39 | 0.000885 | org.apache |
82 | 20409636 | 157 | 0.000175 | org.ietf |
83 | 20397792 | 90 | 0.000420 | com.list-manage |
84 | 20395594 | 368 | 0.000071 | com.googleblog |
85 | 20395350 | 217 | 0.000123 | com.stackoverflow |
86 | 20393172 | 170 | 0.000155 | com.giphy |
87 | 20391226 | 314 | 0.000085 | edu.mit |
88 | 20381948 | 223 | 0.000118 | com.washingtonpost |
89 | 20372602 | 134 | 0.000232 | com.ytimg |
90 | 20363592 | 362 | 0.000073 | com.appspot |
91 | 20360236 | 351 | 0.000076 | com.theverge |
92 | 20359610 | 286 | 0.000093 | com.bbc |
93 | 20358870 | 396 | 0.000067 | uk.co.telegraph |
94 | 20356036 | 499 | 0.000056 | edu.berkeley |
95 | 20348048 | 266 | 0.000101 | edu.harvard |
96 | 20346012 | 330 | 0.000080 | com.go |
97 | 20341676 | 237 | 0.000112 | com.office |
98 | 20338710 | 145 | 0.000206 | us.zoom |
99 | 20335782 | 247 | 0.000109 | com.android |
100 | 20335366 | 327 | 0.000082 | com.wired |
101 | 20334160 | 288 | 0.000092 | com.techcrunch |
102 | 20331782 | 238 | 0.000111 | com.oracle |
103 | 20323638 | 547 | 0.000051 | com.livejournal |
104 | 20296670 | 164 | 0.000170 | com.issuu |
105 | 20295840 | 296 | 0.000090 | com.cnbc |
106 | 20292146 | 211 | 0.000124 | gov.ca |
107 | 20291754 | 402 | 0.000066 | com.ted |
108 | 20288380 | 379 | 0.000069 | gov.nasa |
109 | 20283426 | 149 | 0.000195 | com.forbes |
110 | 20283050 | 148 | 0.000199 | com.wixsite |
111 | 20282972 | 151 | 0.000192 | com.npmjs |
112 | 20282524 | 518 | 0.000054 | com.zdnet |
113 | 20279656 | 447 | 0.000062 | com.msn |
114 | 20277752 | 292 | 0.000091 | com.reuters |
115 | 20275540 | 350 | 0.000076 | com.nature |
116 | 20273474 | 78 | 0.000459 | com.godaddy |
117 | 20271718 | 371 | 0.000070 | com.myspace |
118 | 20270494 | 222 | 0.000119 | com.etsy |
119 | 20268832 | 321 | 0.000084 | com.prnewswire |
120 | 20255726 | 209 | 0.000125 | org.ampproject |
121 | 20252386 | 407 | 0.000065 | org.arxiv |
122 | 20252292 | 312 | 0.000085 | org.npr |
123 | 20252218 | 263 | 0.000101 | com.sciencedirect |
124 | 20248804 | 98 | 0.000410 | com.unpkg |
125 | 20246402 | 265 | 0.000101 | com.example |
126 | 20245616 | 67 | 0.000524 | net.akamaihd |
127 | 20237056 | 215 | 0.000123 | com.eventbrite |
128 | 20234532 | 367 | 0.000072 | org.hbr |
129 | 20232338 | 176 | 0.000151 | com.blogger |
130 | 20231658 | 127 | 0.000247 | org.networkadvertising |
131 | 20231552 | 399 | 0.000066 | com.latimes |
132 | 20228690 | 268 | 0.000101 | org.acm |
133 | 20223242 | 338 | 0.000079 | com.statista |
134 | 20209434 | 389 | 0.000068 | com.fastcompany |
135 | 20205848 | 660 | 0.000043 | com.economist |
136 | 20202482 | 343 | 0.000078 | com.time |
137 | 20202452 | 226 | 0.000117 | com.twimg |
138 | 20201902 | 679 | 0.000042 | edu.upenn |
139 | 20201530 | 550 | 0.000050 | edu.yale |
140 | 20200842 | 258 | 0.000102 | com.githubusercontent |
141 | 20191272 | 474 | 0.000060 | com.steampowered |
142 | 20189824 | 143 | 0.000206 | com.opera |
143 | 20188620 | 444 | 0.000062 | uk.co.dailymail |
144 | 20188486 | 353 | 0.000076 | com.springer |
145 | 20186806 | 576 | 0.000047 | com.scribd |
146 | 20184784 | 780 | 0.000041 | edu.columbia |
147 | 20180100 | 535 | 0.000052 | org.chromium |
148 | 20175876 | 591 | 0.000046 | me.about |
149 | 20175732 | 604 | 0.000046 | google.blog |
150 | 20175284 | 285 | 0.000094 | com.squarespace |
151 | 20174050 | 335 | 0.000079 | com.huffingtonpost |
152 | 20171356 | 431 | 0.000063 | com.nationalgeographic |
153 | 20168788 | 221 | 0.000119 | uk.co.google |
154 | 20165372 | 208 | 0.000125 | com.unsplash |
155 | 20163580 | 388 | 0.000068 | com.w3schools |
156 | 20158956 | 339 | 0.000079 | com.dribbble |
157 | 20154786 | 340 | 0.000079 | com.tiktok |
158 | 20153356 | 293 | 0.000091 | org.un |
159 | 20137924 | 794 | 0.000040 | com.qz |
160 | 20133814 | 248 | 0.000108 | com.bandcamp |
161 | 20129598 | 485 | 0.000058 | edu.cornell |
162 | 20125954 | 821 | 0.000039 | edu.umich |
163 | 20121120 | 119 | 0.000267 | com.ft |
164 | 20115342 | 435 | 0.000063 | com.theatlantic |
165 | 20111028 | 966 | 0.000033 | edu.princeton |
166 | 20110808 | 341 | 0.000078 | com.usatoday |
167 | 20105556 | 786 | 0.000040 | com.evernote |
168 | 20105482 | 133 | 0.000235 | info.aboutads |
169 | 20104810 | 408 | 0.000065 | com.meetup |
170 | 20102638 | 438 | 0.000062 | com.goodreads |
171 | 20100894 | 625 | 0.000045 | org.ieee |
172 | 20098972 | 878 | 0.000036 | com.slate |
173 | 20097870 | 677 | 0.000042 | com.mysql |
174 | 20097656 | 453 | 0.000061 | com.patreon |
175 | 20097530 | 137 | 0.000216 | me.t |
176 | 20095600 | 515 | 0.000055 | com.cbsnews |
177 | 20084204 | 656 | 0.000043 | com.docker |
178 | 20083336 | 291 | 0.000092 | com.wiley |
179 | 20082520 | 480 | 0.000059 | gov.usda |
180 | 20080664 | 454 | 0.000061 | com.dailymotion |
181 | 20078818 | 817 | 0.000039 | edu.washington |
182 | 20077160 | 493 | 0.000057 | com.withgoogle |
183 | 20075064 | 523 | 0.000054 | io.readthedocs |
184 | 20071014 | 644 | 0.000044 | com.marketwatch |
185 | 20065010 | 650 | 0.000043 | uk.co.blogspot |
186 | 20062734 | 868 | 0.000037 | com.shutterstock |
187 | 20062652 | 54 | 0.000703 | com.fb |
188 | 20059664 | 497 | 0.000056 | uk.co.independent |
189 | 20056344 | 76 | 0.000467 | com.wix |
190 | 20055932 | 811 | 0.000039 | org.cambridge |
191 | 20051844 | 559 | 0.000049 | com.pexels |
192 | 20048576 | 779 | 0.000041 | org.sciencemag |
193 | 20048004 | 592 | 0.000046 | com.buzzfeed |
194 | 20044248 | 819 | 0.000039 | com.stackexchange |
195 | 20043466 | 179 | 0.000149 | ru.mail |
196 | 20043446 | 844 | 0.000038 | com.webs |
197 | 20043074 | 573 | 0.000048 | com.git-scm |
198 | 20040208 | 464 | 0.000060 | com.inc |
199 | 20037354 | 272 | 0.000100 | net.behance |
200 | 20029744 | 425 | 0.000063 | gov.whitehouse |
201 | 20025342 | 832 | 0.000038 | com.apnews |
202 | 20023518 | 769 | 0.000041 | com.vox |
203 | 20022030 | 1365 | 0.000024 | uk.co.thesun |
204 | 20018548 | 274 | 0.000098 | com.outlook |
205 | 20018318 | 772 | 0.000041 | org.bitbucket |
206 | 20017276 | 40 | 0.000871 | com.qq |
207 | 20014872 | 244 | 0.000110 | org.doi |
208 | 20012082 | 812 | 0.000039 | uk.ac.cam |
209 | 20011998 | 255 | 0.000103 | com.disqus |
210 | 20007312 | 236 | 0.000112 | com.feedburner |
211 | 20005630 | 670 | 0.000043 | org.worldbank |
212 | 20001230 | 584 | 0.000047 | org.unicef |
213 | 20000932 | 419 | 0.000064 | com.mozilla |
214 | 19999740 | 593 | 0.000046 | co.ibb |
215 | 19999080 | 26 | 0.001261 | io.polyfill |
216 | 19997928 | 525 | 0.000054 | com.booking |
217 | 19993488 | 42 | 0.000808 | com.baidu |
218 | 19989784 | 260 | 0.000101 | com.cloudinary |
219 | 19985856 | 289 | 0.000092 | com.tinyurl |
220 | 19983980 | 345 | 0.000077 | com.ibm |
221 | 19983022 | 1163 | 0.000027 | com.speakerdeck |
222 | 19982506 | 597 | 0.000046 | gov.noaa |
223 | 19978206 | 612 | 0.000045 | ee.linktr |
224 | 19977310 | 569 | 0.000048 | com.psychologytoday |
225 | 19973710 | 531 | 0.000053 | gov.loc |
226 | 19972920 | 400 | 0.000066 | com.getpocket |
227 | 19972760 | 1041 | 0.000031 | edu.utexas |
228 | 19971794 | 320 | 0.000084 | org.pewresearch |
229 | 19971310 | 1366 | 0.000024 | edu.rutgers |
230 | 19970894 | 551 | 0.000050 | com.sagepub |
231 | 19970200 | 309 | 0.000087 | com.nbcnews |
232 | 19967962 | 1134 | 0.000028 | org.eclipse |
233 | 19965586 | 648 | 0.000043 | com.trello |
234 | 19964280 | 326 | 0.000082 | net.windows |
235 | 19964194 | 384 | 0.000068 | com.quora |
236 | 19961430 | 600 | 0.000046 | net.azurewebsites |
237 | 19959910 | 275 | 0.000098 | gov.ftc |
238 | 19955938 | 1057 | 0.000030 | edu.uchicago |
239 | 19953308 | 311 | 0.000086 | com.netdna-ssl |
240 | 19951960 | 782 | 0.000041 | org.semver |
241 | 19951286 | 124 | 0.000252 | com.mailchimp |
242 | 19950294 | 436 | 0.000063 | com.nypost |
243 | 19949296 | 1195 | 0.000027 | com.hatenablog |
244 | 19947142 | 652 | 0.000043 | com.newyorker |
245 | 19943938 | 985 | 0.000033 | uk.co.guardian |
246 | 19943564 | 590 | 0.000046 | com.usnews |
247 | 19940498 | 220 | 0.000119 | tv.twitch |
248 | 19939738 | 784 | 0.000041 | au.net.abc |
249 | 19938820 | 166 | 0.000167 | com.amazon-adsystem |
250 | 19936308 | 1278 | 0.000025 | com.vogue |
251 | 19935466 | 230 | 0.000113 | com.wpengine |
252 | 19934098 | 106 | 0.000338 | com.stripe |
253 | 19933266 | 1261 | 0.000025 | org.kernel |
254 | 19929738 | 941 | 0.000034 | com.politico |
255 | 19926416 | 1193 | 0.000027 | org.unicode |
256 | 19925602 | 580 | 0.000047 | org.eff |
257 | 19925174 | 541 | 0.000051 | br.com.uol |
258 | 19924806 | 852 | 0.000037 | com.about |
259 | 19923644 | 1358 | 0.000024 | edu.hbs |
260 | 19923600 | 954 | 0.000034 | com.dropboxusercontent |
261 | 19923464 | 911 | 0.000035 | edu.jhu |
262 | 19922062 | 993 | 0.000032 | co.elastic |
263 | 19921888 | 913 | 0.000035 | com.steamcommunity |
264 | 19920150 | 1971 | 0.000018 | com.googlesource |
265 | 19919760 | 522 | 0.000054 | com.tandfonline |
266 | 19918010 | 277 | 0.000097 | com.criteo |
267 | 19915708 | 552 | 0.000050 | org.pbs |
268 | 19912986 | 1106 | 0.000029 | edu.umd |
269 | 19912224 | 64 | 0.000549 | co.g |
270 | 19908340 | 865 | 0.000037 | com.foxnews |
271 | 19907456 | 123 | 0.000261 | com.sharethis |
272 | 19904178 | 1027 | 0.000031 | com.rollingstone |
273 | 19903082 | 228 | 0.000115 | com.imdb |
274 | 19902774 | 977 | 0.000033 | com.scientificamerican |
275 | 19901940 | 1392 | 0.000023 | com.urbandictionary |
276 | 19900876 | 775 | 0.000041 | uk.ac.ox |
277 | 19900406 | 391 | 0.000067 | com.arcgis |
278 | 19898520 | 2016 | 0.000018 | com.lego |
279 | 19898420 | 251 | 0.000107 | page.g |
280 | 19898318 | 631 | 0.000044 | gov.census |
281 | 19890056 | 530 | 0.000053 | com.oup |
282 | 19887968 | 346 | 0.000077 | com.optimizely |
283 | 19887424 | 582 | 0.000047 | com.indiatimes |
284 | 19887194 | 376 | 0.000069 | com.cnet |
285 | 19884024 | 422 | 0.000064 | com.wufoo |
286 | 19882930 | 704 | 0.000042 | uk.co.eventbrite |
287 | 19882806 | 421 | 0.000064 | com.bigcommerce |
288 | 19880306 | 1350 | 0.000024 | ca.blogspot |
289 | 19879016 | 833 | 0.000038 | org.fao |
290 | 19878732 | 908 | 0.000035 | com.jetbrains |
291 | 19871044 | 1467 | 0.000022 | ca.ubc |
292 | 19867650 | 1938 | 0.000018 | com.warnerbros |
293 | 19866012 | 446 | 0.000062 | org.d3js |
294 | 19865518 | 946 | 0.000034 | org.greenpeace |
295 | 19864632 | 206 | 0.000127 | net.sourceforge |
296 | 19863450 | 323 | 0.000083 | fr.google |
297 | 19862916 | 1279 | 0.000025 | com.history |
298 | 19861806 | 851 | 0.000038 | com.gumroad |
299 | 19861750 | 919 | 0.000035 | com.chicagotribune |
300 | 19859844 | 636 | 0.000044 | gov.archives |
301 | 19858902 | 284 | 0.000095 | com.googlecode |
302 | 19853502 | 342 | 0.000078 | com.slack |
303 | 19851932 | 229 | 0.000114 | com.eepurl |
304 | 19845626 | 114 | 0.000292 | com.paypalobjects |
305 | 19841702 | 927 | 0.000035 | com.sap |
306 | 19839830 | 153 | 0.000180 | com.addtoany |
307 | 19837466 | 290 | 0.000092 | com.typepad |
308 | 19834082 | 1562 | 0.000021 | de.mpg |
309 | 19830054 | 664 | 0.000043 | com.pinimg |
310 | 19828148 | 282 | 0.000095 | com.calendly |
311 | 19827530 | 491 | 0.000057 | gov.epa |
312 | 19825756 | 354 | 0.000076 | com.proofpoint |
313 | 19821128 | 1430 | 0.000023 | ch.ethz |
314 | 19821094 | 1028 | 0.000031 | com.500px |
315 | 19820554 | 1732 | 0.000019 | com.diigo |
316 | 19820398 | 334 | 0.000079 | com.live |
317 | 19820034 | 1277 | 0.000025 | org.postgresql |
318 | 19818544 | 1257 | 0.000025 | org.wiktionary |
319 | 19817910 | 1274 | 0.000025 | org.aclu |
320 | 19817698 | 981 | 0.000033 | edu.si |
321 | 19816586 | 1394 | 0.000023 | edu.msu |
322 | 19816210 | 1029 | 0.000031 | com.thehill |
323 | 19814936 | 890 | 0.000036 | de.spiegel |
324 | 19813172 | 916 | 0.000035 | com.huffpost |
325 | 19811282 | 472 | 0.000060 | gov.hhs |
326 | 19809240 | 1114 | 0.000028 | com.scmp |
327 | 19806650 | 73 | 0.000484 | me.fb |
328 | 19806306 | 764 | 0.000042 | org.change |
329 | 19805070 | 378 | 0.000069 | com.sohu |
330 | 19804336 | 1329 | 0.000024 | edu.illinois |
331 | 19804164 | 185 | 0.000147 | com.xing |
332 | 19801192 | 1323 | 0.000024 | org.tensorflow |
333 | 19801086 | 1008 | 0.000032 | com.ssrn |
334 | 19800184 | 162 | 0.000171 | com.zendesk |
335 | 19798428 | 904 | 0.000035 | com.netlify |
336 | 19797294 | 508 | 0.000056 | com.squareup |
337 | 19797020 | 1352 | 0.000024 | com.sky |
338 | 19794400 | 196 | 0.000134 | org.iana |
339 | 19792714 | 1078 | 0.000029 | uk.co.thetimes |
340 | 19792494 | 847 | 0.000038 | gov.congress |
341 | 19788704 | 809 | 0.000039 | org.pypi |
342 | 19783878 | 1422 | 0.000023 | cn.com.chinadaily |
343 | 19781142 | 972 | 0.000033 | edu.academia |
344 | 19780974 | 456 | 0.000061 | com.kickstarter |
345 | 19780084 | 802 | 0.000040 | gov.senate |
346 | 19779128 | 2415 | 0.000015 | org.pydata |
347 | 19778124 | 1140 | 0.000027 | org.semanticscholar |
348 | 19775716 | 620 | 0.000045 | site.business |
349 | 19775012 | 1275 | 0.000025 | com.over-blog |
350 | 19774866 | 792 | 0.000040 | org.oecd |
351 | 19774846 | 1660 | 0.000020 | org.phys |
352 | 19774334 | 999 | 0.000032 | com.yarnpkg |
353 | 19772248 | 816 | 0.000039 | com.deviantart |
354 | 19770936 | 1084 | 0.000029 | uk.co.mirror |
355 | 19770522 | 187 | 0.000145 | com.rawgit |
356 | 19770114 | 1315 | 0.000024 | com.axios |
357 | 19769700 | 623 | 0.000045 | gov.house |
358 | 19768998 | 894 | 0.000036 | com.discordapp |
359 | 19768866 | 880 | 0.000036 | com.sciencedaily |
360 | 19766292 | 511 | 0.000055 | com.gmail |
361 | 19765678 | 423 | 0.000064 | com.technorati |
362 | 19763944 | 216 | 0.000123 | com.hubspot |
363 | 19761638 | 1433 | 0.000023 | com.unity3d |
364 | 19760768 | 2137 | 0.000017 | org.threejs |
365 | 19760238 | 1364 | 0.000024 | com.aljazeera |
366 | 19759580 | 245 | 0.000109 | org.nodejs |
367 | 19758846 | 896 | 0.000036 | com.bmj |
368 | 19755564 | 261 | 0.000101 | com.ebay |
369 | 19755198 | 1197 | 0.000026 | au.com.smh |
370 | 19753628 | 234 | 0.000113 | org.gnu |
371 | 19751964 | 1516 | 0.000021 | edu.osu |
372 | 19751362 | 1025 | 0.000031 | int.coe |
373 | 19750302 | 994 | 0.000032 | com.britannica |
374 | 19748408 | 1312 | 0.000024 | edu.gatech |
375 | 19746818 | 2691 | 0.000013 | com.openai |
376 | 19744370 | 495 | 0.000056 | org.openstreetmap |
377 | 19743086 | 437 | 0.000062 | com.ssl-images-amazon |
378 | 19741582 | 791 | 0.000040 | br.com.google |
379 | 19741030 | 855 | 0.000037 | ca.cbc |
380 | 19740484 | 869 | 0.000037 | com.theconversation |
381 | 19739852 | 2582 | 0.000014 | edu.toronto |
382 | 19738652 | 1044 | 0.000031 | gov.usgs |
383 | 19738306 | 1556 | 0.000021 | com.newscientist |
384 | 19736226 | 301 | 0.000088 | net.themeforest |
385 | 19735698 | 605 | 0.000046 | com.udacity |
386 | 19735668 | 473 | 0.000060 | edu.nyu |
387 | 19734084 | 1716 | 0.000019 | edu.ucsc |
388 | 19723708 | 1700 | 0.000020 | org.emojipedia |
389 | 19722194 | 2068 | 0.000017 | it.scoop |
390 | 19722024 | 2754 | 0.000013 | com.slides |
391 | 19721872 | 1459 | 0.000022 | ca.sfu |
392 | 19720004 | 845 | 0.000038 | au.gov.nsw |
393 | 19717908 | 1903 | 0.000019 | org.propublica |
394 | 19717586 | 1386 | 0.000023 | com.firebaseapp |
395 | 19716094 | 2247 | 0.000016 | com.skyrock |
396 | 19710516 | 776 | 0.000041 | com.freepik |
397 | 19707962 | 97 | 0.000412 | net.facebook |
398 | 19704900 | 1454 | 0.000022 | com.penguinrandomhouse |
399 | 19703572 | 195 | 0.000135 | org.bbb |
400 | 19703432 | 1934 | 0.000018 | jp.co.japantimes |
401 | 19701030 | 1762 | 0.000019 | com.itv |
402 | 19700818 | 82 | 0.000437 | net.jsfiddle |
403 | 19700616 | 1985 | 0.000018 | org.maven |
404 | 19699746 | 2370 | 0.000015 | com.deepmind |
405 | 19697844 | 617 | 0.000045 | com.healthline |
406 | 19695324 | 506 | 0.000056 | de.gesetze-im-internet |
407 | 19694720 | 465 | 0.000060 | org.python |
408 | 19694428 | 2331 | 0.000015 | com.mystrikingly |
409 | 19691536 | 884 | 0.000036 | gov.dhs |
410 | 19688238 | 1233 | 0.000026 | com.wikia |
411 | 19685986 | 2090 | 0.000017 | org.sqlite |
412 | 19682976 | 1544 | 0.000021 | ms.1drv |
413 | 19682094 | 178 | 0.000150 | com.salesforce |
414 | 19679914 | 322 | 0.000084 | net.php |
415 | 19671484 | 324 | 0.000083 | com.surveymonkey |
416 | 19670962 | 634 | 0.000044 | com.mashable |
417 | 19670338 | 1628 | 0.000020 | com.motherjones |
418 | 19668724 | 139 | 0.000211 | com.weibo |
419 | 19668554 | 2453 | 0.000014 | com.fastcodesign |
420 | 19667444 | 1506 | 0.000021 | com.flipboard |
421 | 19666746 | 2435 | 0.000015 | edu.byu |
422 | 19665482 | 1748 | 0.000019 | edu.cuny |
423 | 19664886 | 317 | 0.000085 | ru.ok |
424 | 19662618 | 287 | 0.000092 | net.azureedge |
425 | 19662108 | 1339 | 0.000024 | com.thedailybeast |
426 | 19659672 | 246 | 0.000109 | org.aboutcookies |
427 | 19658838 | 2283 | 0.000015 | com.shutterfly |
428 | 19656108 | 1413 | 0.000023 | com.reverbnation |
429 | 19655722 | 2666 | 0.000013 | io.material |
430 | 19655254 | 537 | 0.000052 | io.codepen |
431 | 19652776 | 1296 | 0.000025 | com.dw |
432 | 19651986 | 125 | 0.000250 | com.youtube-nocookie |
433 | 19650416 | 1724 | 0.000019 | com.esri |
434 | 19650188 | 490 | 0.000057 | fr.free |
435 | 19648416 | 1509 | 0.000021 | com.substack |
436 | 19647438 | 561 | 0.000049 | com.matterport |
437 | 19646584 | 1956 | 0.000018 | com.hindustantimes |
438 | 19645830 | 1909 | 0.000019 | com.insider |
439 | 19642342 | 2110 | 0.000017 | edu.oregonstate |
440 | 19641814 | 2390 | 0.000015 | org.wikibooks |
441 | 19640838 | 891 | 0.000036 | int.wipo |
442 | 19640244 | 2820 | 0.000013 | org.aclweb |
443 | 19639226 | 607 | 0.000045 | gov.state |
444 | 19638894 | 2366 | 0.000015 | com.wattpad |
445 | 19638652 | 160 | 0.000172 | gle.forms |
446 | 19636692 | 1052 | 0.000030 | org.jstor |
447 | 19636398 | 1951 | 0.000018 | com.channel4 |
448 | 19636126 | 1752 | 0.000019 | edu.ucsb |
449 | 19635942 | 1320 | 0.000024 | gov.supremecourt |
450 | 19633994 | 56 | 0.000697 | com.googleadservices |
451 | 19631760 | 2441 | 0.000015 | at.ac.univie |
452 | 19629096 | 2924 | 0.000013 | com.pbase |
453 | 19626572 | 278 | 0.000097 | uk.org.ico |
454 | 19624802 | 639 | 0.000044 | com.licdn |
455 | 19623422 | 1518 | 0.000021 | ch.ipcc |
456 | 19621874 | 937 | 0.000034 | com.gallup |
457 | 19621780 | 496 | 0.000056 | com.herokuapp |
458 | 19618584 | 1141 | 0.000027 | edu.brookings |
459 | 19617388 | 963 | 0.000033 | edu.psu |
460 | 19616790 | 1333 | 0.000024 | mil.army |
461 | 19616626 | 434 | 0.000063 | com.rackcdn |
462 | 19614484 | 385 | 0.000068 | com.atlassian |
463 | 19611760 | 1226 | 0.000026 | com.smashingmagazine |
464 | 19609634 | 2227 | 0.000016 | blog.home |
465 | 19608450 | 1362 | 0.000024 | gov.defense |
466 | 19607698 | 1131 | 0.000028 | com.photoshelter |
467 | 19607464 | 483 | 0.000058 | net.imgix |
468 | 19607012 | 182 | 0.000149 | jp.co.yahoo |
469 | 19605316 | 2284 | 0.000015 | com.contently |
470 | 19602040 | 826 | 0.000039 | com.oreilly |
471 | 19597708 | 1174 | 0.000027 | com.mediafire |
472 | 19595596 | 2117 | 0.000017 | com.thecut |
473 | 19594604 | 1960 | 0.000018 | google.ai |
474 | 19594568 | 3151 | 0.000012 | cc.uxdesign |
475 | 19594280 | 3161 | 0.000012 | edu.uvm |
476 | 19594100 | 520 | 0.000054 | edu.cmu |
477 | 19593086 | 3137 | 0.000012 | com.instapaper |
478 | 19591090 | 1591 | 0.000020 | com.thestar |
479 | 19588378 | 369 | 0.000071 | net.researchgate |
480 | 19587214 | 3502 | 0.000011 | com.raywenderlich |
481 | 19587008 | 527 | 0.000053 | com.thinkwithgoogle |
482 | 19584868 | 2149 | 0.000016 | fr.liberation |
483 | 19582230 | 109 | 0.000336 | de.google |
484 | 19581418 | 1574 | 0.000021 | com.buzzfeednews |
485 | 19577648 | 767 | 0.000041 | org.worldwildlife |
486 | 19576662 | 1013 | 0.000032 | com.ecwid |
487 | 19576118 | 1477 | 0.000022 | com.findlaw |
488 | 19574804 | 1012 | 0.000032 | com.thelancet |
489 | 19573936 | 774 | 0.000041 | com.vice |
490 | 19573506 | 813 | 0.000039 | gov.nist |
491 | 19572872 | 1964 | 0.000018 | org.google |
492 | 19572508 | 1531 | 0.000021 | org.hrw |
493 | 19570410 | 765 | 0.000042 | com.intel |
494 | 19568238 | 2695 | 0.000013 | uk.co.ibtimes |
495 | 19567790 | 2372 | 0.000015 | com.oprah |
496 | 19567558 | 87 | 0.000428 | com.workplace |
497 | 19567194 | 3329 | 0.000011 | com.pearltrees |
498 | 19567174 | 2103 | 0.000017 | com.voanews |
499 | 19566762 | 965 | 0.000033 | com.engadget |
500 | 19566188 | 126 | 0.000247 | com.statcounter |
501 | 19564772 | 3365 | 0.000011 | org.edublogs |
502 | 19563980 | 1260 | 0.000025 | org.aiga |
503 | 19562828 | 1031 | 0.000031 | de.stern |
504 | 19562068 | 1583 | 0.000020 | fr.francetvinfo |
505 | 19560196 | 2620 | 0.000014 | com.hm |
506 | 19559342 | 315 | 0.000085 | org.drupal |
507 | 19559132 | 3736 | 0.000010 | fr.unblog |
508 | 19558786 | 747 | 0.000042 | com.canva |
509 | 19558362 | 2870 | 0.000013 | edu.ucf |
510 | 19558064 | 3204 | 0.000012 | ph.telegra |
511 | 19557534 | 926 | 0.000035 | uk.co.pinterest |
512 | 19557072 | 2402 | 0.000015 | edu.kit |
513 | 19556358 | 544 | 0.000051 | it.placehold |
514 | 19555528 | 2219 | 0.000016 | net.corporate-ir |
515 | 19553910 | 2768 | 0.000013 | co.ello |
516 | 19553426 | 881 | 0.000036 | com.arstechnica |
517 | 19553018 | 1449 | 0.000022 | com.livescience |
518 | 19550968 | 2150 | 0.000016 | com.gq |
519 | 19550836 | 1953 | 0.000018 | uk.gov.tfl |
520 | 19550254 | 210 | 0.000125 | com.iubenda |
521 | 19550042 | 533 | 0.000053 | com.pixabay |
522 | 19548328 | 1408 | 0.000023 | org.undp |
523 | 19547668 | 807 | 0.000039 | ca.amazon |
524 | 19547226 | 1020 | 0.000031 | it.smarturl |
525 | 19547032 | 2645 | 0.000014 | org.icrc |
526 | 19546934 | 2447 | 0.000015 | com.webbyawards |
527 | 19545564 | 2423 | 0.000015 | uk.ac.kcl |
528 | 19545554 | 949 | 0.000034 | edu.ucla |
529 | 19544462 | 1444 | 0.000022 | link.page |
530 | 19543968 | 2861 | 0.000013 | com.dummies |
531 | 19541366 | 1581 | 0.000021 | org.ocks |
532 | 19540748 | 65 | 0.000544 | net.typekit |
533 | 19540022 | 1122 | 0.000028 | org.ilo |
534 | 19538882 | 2564 | 0.000014 | com.depositphotos |
535 | 19538866 | 2502 | 0.000014 | com.unilever |
536 | 19536950 | 1348 | 0.000024 | org.acs |
537 | 19536262 | 81 | 0.000440 | com.livestream |
538 | 19535098 | 2672 | 0.000013 | org.rsf |
539 | 19535076 | 489 | 0.000057 | com.adweek |
540 | 19534044 | 2050 | 0.000017 | com.msnbc |
541 | 19530220 | 2509 | 0.000014 | com.slidesharecdn |
542 | 19530084 | 2035 | 0.000018 | com.chronicle |
543 | 19529836 | 3088 | 0.000012 | com.bepress |
544 | 19529580 | 2571 | 0.000014 | com.biography |
545 | 19529322 | 3384 | 0.000011 | tl.de |
546 | 19527886 | 332 | 0.000079 | com.typeform |
547 | 19526428 | 2185 | 0.000016 | com.newrepublic |
548 | 19525400 | 2303 | 0.000015 | com.thoughtco |
549 | 19523856 | 606 | 0.000045 | com.samsung |
550 | 19523112 | 1100 | 0.000029 | org.ohchr |
551 | 19522668 | 790 | 0.000040 | com.fiverr |
552 | 19521518 | 1743 | 0.000019 | io.gitlab |
553 | 19521240 | 121 | 0.000262 | com.jimdo |
554 | 19520292 | 1157 | 0.000027 | com.thenextweb |
555 | 19520070 | 2009 | 0.000018 | fr.orange |
556 | 19519618 | 3272 | 0.000012 | net.openreview |
557 | 19518936 | 2294 | 0.000015 | com.channelnewsasia |
558 | 19517090 | 1283 | 0.000025 | org.aarp |
559 | 19516918 | 2634 | 0.000014 | org.pewsocialtrends |
560 | 19516476 | 1998 | 0.000018 | com.straitstimes |
561 | 19513936 | 2310 | 0.000015 | edu.nd |
562 | 19510956 | 2099 | 0.000017 | com.dallasnews |
563 | 19510732 | 2130 | 0.000017 | de.br |
564 | 19508818 | 2278 | 0.000015 | org.fas |
565 | 19508000 | 1297 | 0.000024 | org.altervista |
566 | 19507978 | 256 | 0.000103 | uk.co.amazon |
567 | 19507290 | 219 | 0.000121 | to.amzn |
568 | 19506624 | 2835 | 0.000013 | com.thejakartapost |
569 | 19505128 | 2211 | 0.000016 | gov.lbl |
570 | 19504556 | 1610 | 0.000020 | de.berlin |
571 | 19504362 | 1086 | 0.000029 | com.popularmechanics |
572 | 19503706 | 2743 | 0.000013 | uk.ac.leeds |
573 | 19503644 | 459 | 0.000061 | com.staticflickr |
574 | 19503210 | 3397 | 0.000011 | org.neocities |
575 | 19502358 | 2996 | 0.000012 | org.vim |
576 | 19502186 | 2883 | 0.000013 | org.globalcitizen |
577 | 19499450 | 572 | 0.000048 | com.deloitte |
578 | 19499392 | 922 | 0.000035 | com.zoho |
579 | 19498964 | 233 | 0.000113 | io.shields |
580 | 19498936 | 2328 | 0.000015 | com.indianexpress |
581 | 19498902 | 3889 | 0.000010 | com.stratechery |
582 | 19497728 | 2819 | 0.000013 | app.web |
583 | 19496358 | 3386 | 0.000011 | org.zotero |
584 | 19493624 | 2939 | 0.000013 | uk.gov.scotland |
585 | 19493314 | 567 | 0.000048 | com.photobucket |
586 | 19491524 | 3756 | 0.000010 | com.bravesites |
587 | 19490552 | 1464 | 0.000022 | org.iea |
588 | 19489976 | 432 | 0.000063 | com.hp |
589 | 19489954 | 2713 | 0.000013 | uk.co.timesonline |
590 | 19489478 | 365 | 0.000073 | com.quantserve |
591 | 19489336 | 404 | 0.000066 | com.digg |
592 | 19486660 | 560 | 0.000049 | com.cisco |
593 | 19486618 | 1155 | 0.000027 | uk.parliament |
594 | 19485014 | 2914 | 0.000013 | com.nwsource |
595 | 19485012 | 2362 | 0.000015 | com.fineartamerica |
596 | 19484598 | 267 | 0.000101 | com.onesignal |
597 | 19484238 | 2234 | 0.000016 | com.foreignpolicy |
598 | 19484200 | 798 | 0.000040 | org.weforum |
599 | 19483398 | 2990 | 0.000012 | com.thoughtworks |
600 | 19483202 | 1548 | 0.000021 | com.treehugger |
601 | 19482398 | 307 | 0.000087 | com.aliyuncs |
602 | 19482224 | 602 | 0.000046 | org.js |
603 | 19480232 | 1527 | 0.000021 | gov.uscis |
604 | 19479040 | 3256 | 0.000012 | uk.ac.city |
605 | 19477476 | 2077 | 0.000017 | com.washingtontimes |
606 | 19477198 | 3504 | 0.000011 | com.mariadb |
607 | 19476316 | 2565 | 0.000014 | org.oas |
608 | 19475236 | 417 | 0.000065 | com.gitlab |
609 | 19472258 | 2584 | 0.000014 | com.mathworks |
610 | 19471752 | 2830 | 0.000013 | com.dezeen |
611 | 19471284 | 835 | 0.000038 | com.investopedia |
612 | 19470638 | 2497 | 0.000014 | uk.co.yougov |
613 | 19469316 | 2934 | 0.000013 | org.heritage |
614 | 19469308 | 614 | 0.000045 | com.netflix |
615 | 19466252 | 3281 | 0.000011 | com.shell |
616 | 19465388 | 2540 | 0.000014 | fr.paris |
617 | 19464956 | 448 | 0.000061 | gov.irs |
618 | 19462732 | 4088 | 0.000009 | tl.page |
619 | 19461330 | 1361 | 0.000024 | com.upwork |
620 | 19461170 | 462 | 0.000061 | com.sxsw |
621 | 19460914 | 1255 | 0.000025 | com.digitaloceanspaces |
622 | 19460548 | 4091 | 0.000009 | com.jigsy |
623 | 19460066 | 861 | 0.000037 | com.venturebeat |
624 | 19458418 | 1215 | 0.000026 | com.dell |
625 | 19457348 | 1016 | 0.000031 | gov.fcc |
626 | 19456828 | 3229 | 0.000012 | uk.co.walesonline |
627 | 19456346 | 2961 | 0.000013 | org.project-syndicate |
628 | 19455696 | 2024 | 0.000018 | com.fivethirtyeight |
629 | 19455242 | 920 | 0.000035 | fm.last |
630 | 19455056 | 2086 | 0.000017 | info.worldometers |
631 | 19454252 | 931 | 0.000034 | org.mediawiki |
632 | 19453670 | 2377 | 0.000015 | ly.rebrand |
633 | 19453158 | 4077 | 0.000009 | net.myanimelist |
634 | 19452824 | 2075 | 0.000017 | cn.gov.fmprc |
635 | 19452012 | 1502 | 0.000021 | org.amnesty |
636 | 19450548 | 349 | 0.000077 | com.adnxs |
637 | 19449350 | 1945 | 0.000018 | com.justia |
638 | 19448712 | 4019 | 0.000009 | edu.usfca |
639 | 19448298 | 2705 | 0.000013 | com.monday |
640 | 19446576 | 1515 | 0.000021 | ca.bc.gov |
641 | 19446486 | 943 | 0.000034 | org.reactjs |
642 | 19446126 | 2285 | 0.000015 | net.openid |
643 | 19445904 | 383 | 0.000068 | com.newrelic |
644 | 19445366 | 1363 | 0.000024 | com.imageshack |
645 | 19445144 | 3568 | 0.000010 | org.globalnetworkinitiative |
646 | 19443940 | 2549 | 0.000014 | com.kaggle |
647 | 19443562 | 3693 | 0.000010 | com.doodlekit |
648 | 19439792 | 259 | 0.000102 | com.getbootstrap |
649 | 19438670 | 2831 | 0.000013 | uk.co.inews |
650 | 19438312 | 3129 | 0.000012 | com.bangkokpost |
651 | 19438230 | 409 | 0.000065 | com.force |
652 | 19437908 | 2107 | 0.000017 | uk.ac.imperial |
653 | 19435434 | 4629 | 0.000008 | net.vingle |
654 | 19434150 | 1982 | 0.000018 | be.kuleuven |
655 | 19434066 | 3530 | 0.000011 | com.intensedebate |
656 | 19432926 | 568 | 0.000048 | com.entrepreneur |
657 | 19432350 | 3518 | 0.000011 | be.blogspot |
658 | 19429740 | 3166 | 0.000012 | se.blogspot |
659 | 19429712 | 1318 | 0.000024 | co.lpages |
660 | 19428992 | 3266 | 0.000012 | org.carnegieendowment |
661 | 19428674 | 837 | 0.000038 | com.globenewswire |
662 | 19428662 | 3175 | 0.000012 | is.good |
663 | 19428098 | 2246 | 0.000016 | com.instructure |
664 | 19427698 | 2965 | 0.000012 | net.alarabiya |
665 | 19427204 | 4090 | 0.000009 | com.kongregate |
666 | 19426514 | 2795 | 0.000013 | com.discovermagazine |
667 | 19425746 | 2613 | 0.000014 | org.gnupg |
668 | 19425518 | 556 | 0.000049 | com.visualstudio |
669 | 19424130 | 191 | 0.000139 | com.atdmt |
670 | 19423528 | 3773 | 0.000010 | com.openlearning |
671 | 19423230 | 3794 | 0.000010 | ch.swissinfo |
672 | 19421982 | 3547 | 0.000010 | com.pixar |
673 | 19420080 | 2154 | 0.000016 | com.livemint |
674 | 19419708 | 957 | 0.000033 | com.variety |
675 | 19417142 | 2816 | 0.000013 | uk.gov.metoffice |
676 | 19414346 | 2004 | 0.000018 | com.surveygizmo |
677 | 19412994 | 3337 | 0.000011 | cn.globaltimes |
678 | 19411212 | 929 | 0.000035 | uk.gov.legislation |
679 | 19411070 | 2639 | 0.000014 | org.ballotpedia |
680 | 19409736 | 243 | 0.000110 | org.whatwg |
681 | 19408620 | 3148 | 0.000012 | com.coca-colacompany |
682 | 19408342 | 1343 | 0.000024 | uk.gov.nationalarchives |
683 | 19406168 | 2326 | 0.000015 | com.thebalancesmb |
684 | 19404822 | 3145 | 0.000012 | uk.gov.companieshouse |
685 | 19403088 | 3532 | 0.000011 | com.dailykos |
686 | 19401008 | 165 | 0.000170 | com.yelp |
687 | 19400512 | 257 | 0.000103 | com.automattic |
688 | 19400270 | 4169 | 0.000009 | com.penzu |
689 | 19399686 | 2489 | 0.000014 | com.bloomberglaw |
690 | 19399662 | 412 | 0.000065 | org.opensource |
691 | 19398126 | 1547 | 0.000021 | org.khanacademy |
692 | 19397376 | 3834 | 0.000010 | com.sfweekly |
693 | 19395236 | 2779 | 0.000013 | com.thumbtack |
694 | 19394202 | 2880 | 0.000013 | org.royalsociety |
695 | 19393684 | 1674 | 0.000020 | kr.co.google |
696 | 19393678 | 2531 | 0.000014 | com.post-gazette |
697 | 19393520 | 2800 | 0.000013 | org.panda |
698 | 19390648 | 2421 | 0.000015 | com.thenation |
699 | 19389714 | 2823 | 0.000013 | io.fabric |
700 | 19388974 | 4936 | 0.000008 | org.arkive |
701 | 19388756 | 2689 | 0.000013 | uk.co.bbci |
702 | 19387624 | 4042 | 0.000009 | hk.edu.cityu |
703 | 19387406 | 3194 | 0.000012 | com.scribblelive |
704 | 19386352 | 3553 | 0.000010 | com.gimletmedia |
705 | 19385872 | 3489 | 0.000011 | com.tweetmeme |
706 | 19384830 | 2541 | 0.000014 | de.uni-heidelberg |
707 | 19384284 | 298 | 0.000089 | ai.shortpixel |
708 | 19383872 | 1920 | 0.000019 | gov.gao |
709 | 19382974 | 4425 | 0.000008 | com.storeboard |
710 | 19381650 | 2814 | 0.000013 | com.politifact |
711 | 19380202 | 3349 | 0.000011 | org.cato |
712 | 19379282 | 4889 | 0.000008 | com.uberant |
713 | 19377306 | 3183 | 0.000012 | fr.lepoint |
714 | 19377194 | 3809 | 0.000010 | edu.depaul |
715 | 19376126 | 3844 | 0.000010 | net.thedailystar |
716 | 19375590 | 406 | 0.000066 | com.aol |
717 | 19375570 | 4046 | 0.000009 | edu.umt |
718 | 19372794 | 1948 | 0.000018 | tv.ustream |
719 | 19372628 | 1034 | 0.000031 | com.verisign |
720 | 19369588 | 3279 | 0.000011 | com.theweek |
721 | 19367934 | 905 | 0.000035 | com.box |
722 | 19367170 | 3724 | 0.000010 | com.eklablog |
723 | 19365850 | 3488 | 0.000011 | com.militarytimes |
724 | 19365832 | 866 | 0.000037 | gov.uspto |
725 | 19365580 | 3483 | 0.000011 | com.multiscreensite |
726 | 19364098 | 3103 | 0.000012 | uk.ac.york |
727 | 19359488 | 3165 | 0.000012 | org.openweathermap |
728 | 19358574 | 1526 | 0.000021 | com.techrepublic |
729 | 19358070 | 3315 | 0.000011 | org.jenkins-ci |
730 | 19357968 | 2815 | 0.000013 | org.wnyc |
731 | 19357458 | 638 | 0.000044 | gov.copyright |
732 | 19356834 | 3433 | 0.000011 | com.lawfareblog |
733 | 19354610 | 2357 | 0.000015 | co.pcdn |
734 | 19353004 | 3263 | 0.000012 | com.nyt |
735 | 19352766 | 3101 | 0.000012 | se.svt |
736 | 19351866 | 1048 | 0.000030 | net.clickbank |
737 | 19351546 | 3121 | 0.000012 | com.scotsman |
738 | 19348720 | 1182 | 0.000027 | com.foursquare |
739 | 19348660 | 1239 | 0.000026 | com.pingdom |
740 | 19348048 | 2475 | 0.000014 | com.squarespace-cdn |
741 | 19346678 | 2323 | 0.000015 | com.natlawreview |
742 | 19346350 | 2769 | 0.000013 | org.wri |
743 | 19345800 | 3430 | 0.000011 | com.bigthink |
744 | 19345054 | 4132 | 0.000009 | com.newgrounds |
745 | 19344692 | 3862 | 0.000010 | org.sourcewatch |
746 | 19342356 | 3720 | 0.000010 | re.cli |
747 | 19341788 | 3156 | 0.000012 | gov.ncjrs |
748 | 19341458 | 3087 | 0.000012 | my.com.thestar |
749 | 19340698 | 3307 | 0.000011 | gov.anl |
750 | 19339932 | 3117 | 0.000012 | com.nationalreview |
751 | 19339132 | 2597 | 0.000014 | ca.newswire |
752 | 19338090 | 1603 | 0.000020 | org.webkit |
753 | 19337402 | 3700 | 0.000010 | org.elasticsearch |
754 | 19335276 | 928 | 0.000035 | com.hootsuite |
755 | 19334936 | 300 | 0.000088 | com.caniuse |
756 | 19334252 | 3236 | 0.000012 | gov.fec |
757 | 19333910 | 2327 | 0.000015 | ru.rg |
758 | 19333124 | 3741 | 0.000010 | org.constitutioncenter |
759 | 19332102 | 1602 | 0.000020 | com.jwplayer |
760 | 19331754 | 4253 | 0.000009 | com.etymonline |
761 | 19331678 | 3620 | 0.000010 | it.eventbrite |
762 | 19331510 | 2960 | 0.000013 | com.madmimi |
763 | 19331460 | 3491 | 0.000011 | com.afp |
764 | 19330192 | 1907 | 0.000019 | com.kinstacdn |
765 | 19328136 | 3163 | 0.000012 | gov.ornl |
766 | 19327042 | 461 | 0.000061 | com.pubmatic |
767 | 19325866 | 401 | 0.000066 | gg.discord |
768 | 19325518 | 1289 | 0.000025 | com.intuit |
769 | 19325482 | 1168 | 0.000027 | com.ycombinator |
770 | 19325258 | 3292 | 0.000011 | com.crashlytics |
771 | 19324302 | 4270 | 0.000009 | com.underconsideration |
772 | 19322856 | 2599 | 0.000014 | com.articulate |
773 | 19322230 | 3246 | 0.000012 | de.uni-frankfurt |
774 | 19321496 | 3692 | 0.000010 | uk.co.spectator |
775 | 19321096 | 867 | 0.000037 | com.wikihow |
776 | 19321010 | 4275 | 0.000009 | to.gplus |
777 | 19320802 | 4920 | 0.000008 | pl.pastebin |
778 | 19320622 | 3791 | 0.000010 | uk.co.manchestereveningnews |
779 | 19319854 | 2938 | 0.000013 | edu.unh |
780 | 19318976 | 2553 | 0.000014 | de.tagesschau |
781 | 19318802 | 2116 | 0.000017 | gov.energystar |
782 | 19318372 | 429 | 0.000063 | com.businesswire |
783 | 19318050 | 829 | 0.000038 | com.moz |
784 | 19314848 | 3550 | 0.000010 | org.avaaz |
785 | 19314554 | 3683 | 0.000010 | com.mnn |
786 | 19314476 | 1172 | 0.000027 | com.alexa |
787 | 19314150 | 2332 | 0.000015 | net.vnexpress |
788 | 19313268 | 348 | 0.000077 | com.constantcontact |
789 | 19312732 | 3600 | 0.000010 | com.heraldscotland |
790 | 19312326 | 3843 | 0.000010 | fm.audioboo |
791 | 19311750 | 4481 | 0.000008 | tv.eurovision |
792 | 19311646 | 974 | 0.000033 | com.fandom |
793 | 19311256 | 3717 | 0.000010 | uk.ac.uea |
794 | 19311174 | 3697 | 0.000010 | uk.ac.core |
795 | 19310268 | 3514 | 0.000011 | com.hsbc |
796 | 19310254 | 3492 | 0.000011 | org.sciencenews |
797 | 19310242 | 4916 | 0.000008 | com.blackplanet |
798 | 19310096 | 3289 | 0.000011 | com.realclearpolitics |
799 | 19309366 | 1698 | 0.000020 | com.pastebin |
800 | 19309196 | 3190 | 0.000012 | uk.org.rspb |
801 | 19308322 | 1377 | 0.000023 | com.techradar |
802 | 19308094 | 529 | 0.000053 | com.indeed |
803 | 19307548 | 4985 | 0.000007 | dk.bloggersdelight |
804 | 19307144 | 4491 | 0.000008 | com.xtgem |
805 | 19306108 | 2073 | 0.000017 | ca.on.gov |
806 | 19305500 | 3536 | 0.000011 | uk.co.thisismoney |
807 | 19304908 | 797 | 0.000040 | gov.sec |
808 | 19302330 | 1128 | 0.000028 | net.atlassian |
809 | 19302240 | 3937 | 0.000009 | com.collinsdictionary |
810 | 19299944 | 1479 | 0.000022 | edu.purdue |
811 | 19299020 | 3179 | 0.000012 | com.wayfair |
812 | 19298908 | 3611 | 0.000010 | org.chathamhouse |
813 | 19297900 | 3218 | 0.000012 | org.rferl |
814 | 19297216 | 397 | 0.000066 | com.skype |
815 | 19296536 | 4738 | 0.000008 | edu.ualr |
816 | 19296016 | 3523 | 0.000011 | org.diva-portal |
817 | 19295672 | 2785 | 0.000013 | org.cfr |
818 | 19294806 | 1249 | 0.000025 | com.merriam-webster |
819 | 19292968 | 4835 | 0.000008 | com.designobserver |
820 | 19292734 | 3399 | 0.000011 | org.pewforum |
821 | 19292200 | 270 | 0.000100 | jp.co.amazon |
822 | 19291468 | 3994 | 0.000009 | uk.co.dailyrecord |
823 | 19290936 | 3951 | 0.000009 | edu.swarthmore |
824 | 19290570 | 3339 | 0.000011 | com.ubs |
825 | 19289748 | 1075 | 0.000030 | so.notion |
826 | 19289742 | 2847 | 0.000013 | us.govtrack |
827 | 19289236 | 1256 | 0.000025 | com.udemy |
828 | 19289040 | 333 | 0.000079 | com.hackerone |
829 | 19288716 | 3787 | 0.000010 | org.nationalinterest |
830 | 19288626 | 3138 | 0.000012 | com.doubleclickbygoogle |
831 | 19288000 | 279 | 0.000097 | de.amazon |
832 | 19287244 | 2036 | 0.000018 | org.doxygen |
833 | 19286840 | 1661 | 0.000020 | scot.gov |
834 | 19286652 | 3933 | 0.000009 | de.berliner-zeitung |
835 | 19285868 | 1519 | 0.000021 | com.billboard |
836 | 19283910 | 681 | 0.000042 | com.gartner |
837 | 19283390 | 4698 | 0.000008 | net.writeablog |
838 | 19282688 | 2465 | 0.000014 | com.infoworld |
839 | 19282084 | 823 | 0.000039 | com.sedo |
840 | 19281700 | 3200 | 0.000012 | org.aei |
841 | 19280820 | 71 | 0.000502 | com.oculus |
842 | 19280652 | 1580 | 0.000021 | edu.ucsd |
843 | 19280396 | 329 | 0.000081 | mp.mailchi |
844 | 19280288 | 3917 | 0.000009 | edu.umaine |
845 | 19279222 | 3262 | 0.000012 | org.iucnredlist |
846 | 19279130 | 2827 | 0.000013 | com.lexology |
847 | 19278304 | 4851 | 0.000008 | com.nation2 |
848 | 19278156 | 5290 | 0.000007 | com.anotepad |
849 | 19278056 | 4128 | 0.000009 | za.co.mg |
850 | 19276824 | 77 | 0.000467 | com.messenger |
851 | 19276460 | 2083 | 0.000017 | org.dejure |
852 | 19276002 | 4494 | 0.000008 | net.blogfreely |
853 | 19275630 | 1302 | 0.000024 | org.owasp |
854 | 19275142 | 3309 | 0.000011 | com.foreignaffairs |
855 | 19275092 | 4067 | 0.000009 | tw.com.books |
856 | 19274916 | 4267 | 0.000009 | ca.nfb |
857 | 19274822 | 364 | 0.000073 | com.bitly |
858 | 19274560 | 3225 | 0.000012 | org.osce |
859 | 19274028 | 3726 | 0.000010 | uk.org.wwf |
860 | 19274006 | 3971 | 0.000009 | org.truthout |
861 | 19273104 | 155 | 0.000178 | gov.privacyshield |
862 | 19272708 | 1981 | 0.000018 | edu.uci |
863 | 19272368 | 2044 | 0.000017 | se.haxx |
864 | 19272288 | 897 | 0.000036 | com.emarketer |
865 | 19272110 | 4532 | 0.000008 | com.symbaloo |
866 | 19271508 | 1004 | 0.000032 | com.playstation |
867 | 19271338 | 2196 | 0.000016 | org.sundance |
868 | 19271216 | 363 | 0.000073 | eu.youronlinechoices |
869 | 19271196 | 3496 | 0.000011 | com.rev |
870 | 19271080 | 4071 | 0.000009 | in.thewire |
871 | 19270976 | 159 | 0.000174 | org.nginx |
872 | 19270528 | 903 | 0.000036 | com.libsyn |
873 | 19268650 | 2400 | 0.000015 | us.pa.state |
874 | 19267610 | 146 | 0.000205 | me.line |
875 | 19267478 | 5202 | 0.000007 | net.bravejournal |
876 | 19267386 | 3140 | 0.000012 | ru.kp |
877 | 19267334 | 4014 | 0.000009 | com.ecowatch |
878 | 19266700 | 514 | 0.000055 | org.debian |
879 | 19266302 | 539 | 0.000052 | com.gofundme |
880 | 19266194 | 976 | 0.000033 | com.pcmag |
881 | 19264914 | 4151 | 0.000009 | com.theoutline |
882 | 19264512 | 4316 | 0.000009 | org.icj-cij |
883 | 19263626 | 1470 | 0.000022 | org.coursera |
884 | 19261610 | 2076 | 0.000017 | gov.healthcare |
885 | 19260626 | 3721 | 0.000010 | com.iconarchive |
886 | 19259734 | 1657 | 0.000020 | net.leadpages |
887 | 19259034 | 1486 | 0.000022 | com.technologyreview |
888 | 19258032 | 2367 | 0.000015 | ca.citizenlab |
889 | 19257884 | 3690 | 0.000010 | com.governing |
890 | 19257782 | 3322 | 0.000011 | com.wikidot |
891 | 19257260 | 2385 | 0.000015 | org.raspberrypi |
892 | 19256452 | 4621 | 0.000008 | jp.ac.kobe-u |
893 | 19255454 | 1073 | 0.000030 | com.timeanddate |
894 | 19254836 | 1096 | 0.000029 | com.buffer |
895 | 19254032 | 3978 | 0.000009 | com.ogilvy |
896 | 19251530 | 940 | 0.000034 | com.css-tricks |
897 | 19251096 | 1501 | 0.000021 | com.msdn |
898 | 19250138 | 3958 | 0.000009 | com.gab |
899 | 19249994 | 3673 | 0.000010 | com.what3words |
900 | 19249260 | 1241 | 0.000026 | com.tableau |
901 | 19248316 | 1319 | 0.000024 | com.xkcd |
902 | 19248224 | 3695 | 0.000010 | com.nestle |
903 | 19247678 | 4982 | 0.000007 | net.postheaven |
904 | 19246428 | 470 | 0.000060 | com.fc2 |
905 | 19246238 | 1795 | 0.000019 | com.pcworld |
906 | 19246028 | 2589 | 0.000014 | mp.j |
907 | 19245754 | 4318 | 0.000009 | org.kuow |
908 | 19245300 | 3906 | 0.000009 | org.migrationpolicy |
909 | 19245282 | 585 | 0.000047 | com.fortune |
910 | 19244324 | 3769 | 0.000010 | de.morgenpost |
911 | 19244120 | 3282 | 0.000011 | uk.gov.data |
912 | 19243558 | 4952 | 0.000007 | cz.webgarden |
913 | 19243100 | 2118 | 0.000017 | org.donorbox |
914 | 19242192 | 3909 | 0.000009 | de.uni-konstanz |
915 | 19241684 | 4218 | 0.000009 | org.birdlife |
916 | 19240982 | 3875 | 0.000010 | org.people-press |
917 | 19240778 | 2132 | 0.000017 | to.dev |
918 | 19239846 | 906 | 0.000035 | org.golang |
919 | 19238732 | 2425 | 0.000015 | net.noscript |
920 | 19237742 | 1223 | 0.000026 | com.podbean |
921 | 19235906 | 4130 | 0.000009 | com.scienceblogs |
922 | 19235706 | 4948 | 0.000007 | it.clyp |
923 | 19235498 | 3355 | 0.000011 | edu.fordham |
924 | 19231696 | 4076 | 0.000009 | org.oyez |
925 | 19230656 | 3441 | 0.000011 | com.joebiden |
926 | 19229960 | 2867 | 0.000013 | com.washingtonexaminer |
927 | 19229728 | 1115 | 0.000028 | com.gizmodo |
928 | 19229112 | 2757 | 0.000013 | org.healthaffairs |
929 | 19228910 | 1232 | 0.000026 | com.searchengineland |
930 | 19228678 | 854 | 0.000037 | fm.anchor |
931 | 19227412 | 5084 | 0.000007 | com.zcubes |
932 | 19227258 | 1995 | 0.000018 | com.ssllabs |
933 | 19225964 | 1072 | 0.000030 | org.poynter |
934 | 19224436 | 1644 | 0.000020 | net.java |
935 | 19223632 | 1514 | 0.000021 | edu.usc |
936 | 19223252 | 3680 | 0.000010 | org.carbonbrief |
937 | 19221502 | 5165 | 0.000007 | org.csgrid |
938 | 19221286 | 308 | 0.000087 | jp.ameblo |
939 | 19220064 | 1578 | 0.000021 | com.sun |
940 | 19220010 | 3959 | 0.000009 | org.rfa |
941 | 19218588 | 2616 | 0.000014 | uk.gov.defra |
942 | 19218556 | 3912 | 0.000009 | com.exxonmobil |
943 | 19218102 | 5249 | 0.000007 | com.topsitenet |
944 | 19217732 | 3012 | 0.000012 | com.html5rocks |
945 | 19217494 | 3660 | 0.000010 | ca.yelp |
946 | 19216576 | 2940 | 0.000013 | com.instructables |
947 | 19215582 | 2212 | 0.000016 | org.linuxfoundation |
948 | 19215410 | 4069 | 0.000009 | uk.org.woodlandtrust |
949 | 19213854 | 2058 | 0.000017 | org.json |
950 | 19213790 | 214 | 0.000124 | com.tripadvisor |
951 | 19212490 | 5233 | 0.000007 | net.squareblogs |
952 | 19212378 | 3864 | 0.000010 | ru.mid |
953 | 19212170 | 231 | 0.000113 | com.myshopify |
954 | 19211108 | 3310 | 0.000011 | com.flippa |
955 | 19211092 | 3850 | 0.000010 | com.townandcountrymag |
956 | 19210938 | 1292 | 0.000025 | build.bazel |
957 | 19210816 | 5295 | 0.000007 | net.werite |
958 | 19210212 | 1240 | 0.000026 | com.uk |
959 | 19209742 | 2354 | 0.000015 | com.storify |
960 | 19209508 | 3280 | 0.000011 | org.cjr |
961 | 19208854 | 3158 | 0.000012 | org.acog |
962 | 19208448 | 3921 | 0.000009 | br.com.sebrae |
963 | 19208380 | 250 | 0.000107 | org.icann |
964 | 19207876 | 1635 | 0.000020 | fr.blogspot |
965 | 19207582 | 122 | 0.000262 | com.bizjournals |
966 | 19207280 | 3406 | 0.000011 | org.cites |
967 | 19207102 | 2687 | 0.000013 | com.tutsplus |
968 | 19207058 | 3409 | 0.000011 | tr.com.aa |
969 | 19206380 | 1109 | 0.000028 | org.whatbrowser |
970 | 19205750 | 4680 | 0.000008 | org.learner |
971 | 19205144 | 3424 | 0.000011 | no.yr |
972 | 19203738 | 4271 | 0.000009 | com.s-nbcnews |
973 | 19203150 | 4166 | 0.000009 | org.spie |
974 | 19203082 | 1335 | 0.000024 | com.indiegogo |
975 | 19202634 | 708 | 0.000042 | com.airbnb |
976 | 19202288 | 4217 | 0.000009 | com.revolut |
977 | 19201514 | 4339 | 0.000009 | org.atsjournals |
978 | 19201386 | 1033 | 0.000031 | com.redhat |
979 | 19200760 | 4066 | 0.000009 | uk.co.zoopla |
980 | 19199826 | 318 | 0.000084 | it.google |
981 | 19199246 | 1137 | 0.000028 | com.windowsphone |
982 | 19198666 | 1485 | 0.000022 | edu.unc |
983 | 19198508 | 466 | 0.000060 | gov.fda |
984 | 19198408 | 653 | 0.000043 | com.zapier |
985 | 19198272 | 2161 | 0.000016 | com.gigaom |
986 | 19197316 | 4457 | 0.000008 | ru.novayagazeta |
987 | 19196504 | 1936 | 0.000018 | br.com.correios |
988 | 19196468 | 4101 | 0.000009 | google.design |
989 | 19195350 | 2194 | 0.000016 | org.eu |
990 | 19192238 | 3758 | 0.000010 | com.mail-archive |
991 | 19191310 | 4437 | 0.000008 | com.out |
992 | 19191000 | 4759 | 0.000008 | tw.focustaiwan |
993 | 19190946 | 4235 | 0.000009 | org.insideclimatenews |
994 | 19190774 | 2038 | 0.000017 | com.freeprivacypolicy |
995 | 19190442 | 4265 | 0.000009 | org.escardio |
996 | 19190354 | 4663 | 0.000008 | com.theschooloflife |
997 | 19189766 | 241 | 0.000111 | com.naver |
998 | 19188362 | 4711 | 0.000008 | edu.uah |
999 | 19188230 | 1611 | 0.000020 | com.nike |
1000 | 19187360 | 4319 | 0.000009 | edu.mtsu |
Credits
Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible.
We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc. Let us know about your results via Common Crawl’s Google Group!
May 2021 crawl archive now available
The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.28 billion new URLs, not visited in any of our prior crawls.
Archive Location and Download
The May crawl archive is located in the commoncrawl bucket at crawl-data/CC-MAIN-2021-21/.
To assist with exploring and using the dataset, we provide gzipped files which list all segments, WARC, WAT and WET files.
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.
File List | #Files | Total Size Compressed (TiB) |
|
---|---|---|---|
Segments | CC-MAIN-2021-21/segment.paths.gz | 100 | |
WARC files | CC-MAIN-2021-21/warc.paths.gz | 64000 | 66.17 |
WAT files | CC-MAIN-2021-21/wat.paths.gz | 64000 | 17.61 |
WET files | CC-MAIN-2021-21/wet.paths.gz | 64000 | 7.65 |
Robots.txt files | CC-MAIN-2021-21/robotstxt.paths.gz | 64000 | 0.17 |
Non-200 responses files | CC-MAIN-2021-21/non200responses.paths.gz | 64000 | 1.86 |
URL index files | CC-MAIN-2021-21/cc-index.paths.gz | 302 | 0.2 |
The Common Crawl URL Index for this crawl is available at: https://index.commoncrawl.org/CC-MAIN-2021-21/. Also the columnar index has been updated to contain this crawl.
Please donate to Common Crawl if you appreciate our free datasets! We’re also seeking corporate sponsors to partner with Common Crawl for our non-profit work in open data. Please contact [email protected] for sponsorship information.