Beautiful. Unwelcome Gaze is a triptych visualizing the publicly reachable web server infrastructure of Google, Facebook, Amazon and the routing graph(s) leading to them.

Do you like this sort of Stuff? Pleasesupport me on Patreon. I'd really appreciate it. Know anyone looking for a simple book explaining the cloud? Then please recommend my well reviewed (30 reviews on Amazon and 72 on Goodreads!) book: Explain the Cloud Like I'm 10. They'll love it and you'll be their hero forever.

Larry Ellison: It’s kind of embarrassing when Amazon uses Oracle but they want you to use Aurora and Redshift. They’ve had 10 years to get off Oracle, and they’re still on Oracle.

@Werner: Amazon's Oracle data warehouse was one of the largest (if not THE largest) in the world. RIP. We have moved on to newer, faster, more reliable, more agile, more versatile technology at more lower cost and higher scale. #AWS Redshift FTW!

Sheera Frenkel: “We’re not going to traffic in your personal life,” Tim Cook, Apple’s chief executive, said in an MSNBC interview. “Privacy to us is a human right. It’s a civil liberty.” (Mr. Cook’s criticisms infuriated Mr. Zuckerberg, who later ordered his management team to use only Android phones — arguing that the operating system had far more users than Apple’s.)

@copyconstruct: This is a fantastic companion to the first Aurora paper from Amazon.

The main takeaway is the distributed consensus kills performance, that local state can actually be a good thing, use an immutable log as the source of truth (basically what the Kafka folks have been preaching)

Daniel Lemir: Another way to look at the problem is to measure the “speedup” due to the memory-level parallelism: we divide the time it takes to traverse the array using 1 lane by the time it takes to do so using X lane. We see that the Intel Skylake processor is limited to about a 10x or 11x speedup whereas the Apple processors go much higher.

Shaun Raviv: Friston’s free energy principle says that all life, at every scale of organization—from single cells to the human brain, with its billions of neurons—is driven by the same universal imperative, which can be reduced to a mathematical function. To be alive, he says, is to act in ways that reduce the gulf between your expectations and your sensory inputs. Or, in Fristonian terms, it is to minimize free energy.

Max J. Krause: with the exception of aluminium, cryptomining consumed more energy than mineral mining to produce an equivalent market value. While the market prices of the coins are quite volatile, the network hashrates for three of the four cryptocurrencies have trended consistently upward, suggesting that energy requirements will continue to increase. During this period, we estimate mining for all 4 cryptocurrencies was responsible for 3–15 million tonnes of CO2 emissions.

Doubleslash: Back in 2014 OpenStack may have been a thing but nowadays, 4 years later, people learned a lesson. There is no value in being a full-stack operator of infrastructure for systems. The idea of running a complex, distributed architecture like a private cloud at the same or higher stability and comparable or even lower cost than public clouds eventually proved futile. That's why in 2018, consensus is, infrastructure for systems has been replaced by infrastructure for applications (k8s). The infrastructure is typically consumed on public clouds.

How many quotes are there do you think?

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...]]>http://highscalability.com/blog/rss-comments-entry-36128735.xmlNew: The Datacenter as a Computer: Designing Warehouse-Scale Machines, Third EditionTodd HoffWed, 14 Nov 2018 18:18:04 +0000http://highscalability.com/blog/2018/11/14/new-the-datacenter-as-a-computer-designing-warehouse-scale-m.html426227:4867632:36129592

The third edition reflects four years of advancements since the previous edition and nearly doubles the number of pictures and figures. New topics range from additional workloads like video streaming, machine learning, and public cloud to specialized silicon accelerators, storage and network building blocks, and a revised discussion of data center power and cooling, and uptime. Further discussions of emerging trends and opportunities ensure that this revised edition will remain an essential resource for educators and professionals working on the next generation of WSCs.

The abstract:

This book describes warehouse-scale computers (WSCs), the computing platforms that power cloud computing and all the great web services we use every day. It discusses how these new systems treat the datacenter itself as one massive computer designed at warehouse scale, with hardware and software working in concert to deliver good levels of internet service performance. The book details the architecture of WSCs and covers the main factors influencing their design, operation, and cost structure, and the characteristics of their software base. Each chapter contains multiple real-world examples, including detailed case studies and previously unpublished details of the infrastructure used to power Google’s online services. Targeted at the architects and programmers of today’s WSCs, this book provides a great foundation for those looking to innovate in this fascinating and important area, but the material will also be broadly interesting to those who just want to under- stand the infrastructure powering the internet.

Who's Hiring?

Twitch's commerce team in San Francisco is looking to hire senior developers to keep up with rapidly increasing demand for our Subscriptions and Payment platform. Engineers will be tasked with building new products and features to solve business and ecommerce challenges as we're dealing with engaging problems at a massive scale and will create solutions that impact millions of people around the world. Apply here.

Fun and Informative Events

Advertise your event here!

Cool Products and Services

InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.

Scalyr is a lightning-fast log management and operational data platform. It's a tool (actually, multiple tools) that your entire team will love. Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. . Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Make Your Job Search O(1) — not O(n)

Triplebyte is unique because they're a team of engineers running their own centralized technical assessment. Companies like Apple, Dropbox, Mixpanel, and Instacart now let Triplebyte-recommended engineers skip their own screening steps.

We found that High Scalability readers are about 80% more likely to be in the top bracket of engineering skill.

The Solution to Your Operational Diagnostics Woes

Scalyr gives you instant visibility of your production systems, helping you turn chaotic logs and system metrics into actionable data at interactive speeds. Don't be limited by the slow and narrow capabilities of traditional log monitoring tools. View and analyze all your logs and system metrics from multiple sources in one place. Get enterprise-grade functionality with sane pricing and insane performance. Learn more today.

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Do you like this sort of Stuff? Pleasesupport me on Patreon. I'd really appreciate it. Know anyone looking for a simple book explaining the cloud? Then please recommend my well reviewed (30 reviews on Amazon and 72 on Goodreads!) book: Explain the Cloud Like I'm 10. They'll love it and you'll be their hero forever.

@davidgerard: It really won't, because it can't possibly scale. This is the key problem with every musical blockchain initiative I've ever seen. Proposals whose "blockchain" would need to add 1GB/hour, that sort of thing.

@matei_zaharia: Since we opened #DAWNBench deep learning benchmark rolling submissions, there have been some cool entires. You can now train CIFAR10 for just $0.06 and ImageNet for $12.60 (4x less than in April!): https://dawn.cs.stanford.edu/benchmark/ . MLPerf's deadline is also soon (http://mlperf.org ).

Paul Alcorn~ The 7nm Rome CPUs come with 64 physical Zen 2 cores, which equates to 128 threads per processor, double that of the first-gen Naples chips. In a two socket server, that equates to 128 physical cores and 256 threads in a single box. Rome is also the first PCIe 4.0 CPU, which offers double the bandwidth per channel. AMD has improved the Infinity Fabric. AMD is now using the second-gen Infinity Fabric to connect a multi-chip design with a 14nm I/O die serving as the linchpin of the design. That central chip ties together the 7nm CPU chiplets, creating a massively scalable architecture. Amazon Web Services, one of the world's largest cloud service providers, announced that, beginning today, it is offering new EPYC-powered cloud instances. The R5a, M5a and T3a instances purportedly will offer a 10% price-to-performance advantage over AWS's other cloud instances.

@tony_goodfellow: He's World of Warcraft's first and only conscientious objector—a neutral pandaren shaman who spends his time picking flowers and mining rocks instead of fighting and killing. ...after roughly 8000 .... Doubleagent has reached the maximum level of 110."

...

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...]]>http://highscalability.com/blog/rss-comments-entry-36127664.xmlFighting the Ecosystem Wars in the Proactive CloudTodd HoffWed, 07 Nov 2018 16:59:18 +0000http://highscalability.com/blog/2018/11/7/fighting-the-ecosystem-wars-in-the-proactive-cloud.html426227:4867632:36126767This article is a chapter from my book Explain the Cloud Like I'm 10. It has 30 reviews on Amazon! If you like this chapter then you'll love the book.

The cloud is always busy proactively working for you in the background. That’s how cloud services compete with each other to keep you in their ecosystem.

Most of the cloud services we’ve talked about so far have been request driven. You initiate a request and the cloud does something for you. You read a book. You search for the nearest coffee shop. You navigate to a destination. You send a message. You play a movie.

Handling direct requests is not all a cloud is good for. In fact, the biggest potential of the cloud is how it can proactively perform jobs for you in the background, without you asking or even knowing that it can be done.

"The future is already here – it's just not evenly distributed." — William Gibson

Do you like this sort of Stuff? Pleasesupport me on Patreon. I'd really appreciate it. Know anyone looking for a simple book explaining the cloud? Then please recommend my well reviewed (30 reviews on Amazon and 72 on Goodreads!) book: Explain the Cloud Like I'm 10. They'll love it and you'll be their hero forever.

@cdixon: 1/ There have been many great software-related forum posts over the years. Some favorites…2/ Tim Berners Lee, proposing the World Wide Web in 1991...3/ Linus Torvalds proposing Linux, also in 1991...4/ Marc Andreessen proposing images for web browsers, in 1993...5/ And, of course, Satoshi proposing Bitcoin, 10 years ago today.

Federal Register: In this final rule, the Librarian of Congress adopts exemptions to the provision of the Digital Millennium Copyright Act (“DMCA”) that prohibits circumvention of technological measures that control access to copyrighted works, codified in the United States Code.

@johncutlefish: Them: “Why can’t the teams just focus on shipping!?Maybe it is a generational thing, but these people just keep asking questions....” Me: “But you hired people who solve puzzles / problem solve for a living, what did you expect?”

David Rosenthal: The big successes in the field haven't come from consensus building around a roadmap, they have come from idiosyncratic individuals such as Brewster Kahle, Roberto di Cosmo and Jason Scott identifying a need and building a system to address it no matter what "the community" thinks. We have a couple of decades of experience showing that "the community" is incapable of coming to a coherent consensus that leads to action on a scale appropriate to the problem. In any case, describing road-mapping as "research" is a stretch.

Simon Wistow: Observability goes beyond monitoring, enabling the proactive introspection of distributed systems for greater operational visibility. Observability allows you to ask open-ended questions and have the data you need in order to explore the data to find answers. In short, observability gives you the information you need to make better decisions using real data.

@keithchambers: We use Mesos + Marathon in Yammer at Microsoft. It works fine for us. But long term we don’t want to run any of our own infrastructure (in Yammer) so we will look to move to Service Fabric or Azure Kubernetes Service. We have bigger fish to fry so we’ll be on Mesos for years.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...]]>http://highscalability.com/blog/rss-comments-entry-36125270.xmlSponsored Post: Twitch, InMemory.Net, Triplebyte, Etleap, Stream, Scalyr, MemSQLsponsored postTodd HoffTue, 30 Oct 2018 18:54:47 +0000http://highscalability.com/blog/2018/10/30/sponsored-post-twitch-inmemorynet-triplebyte-etleap-stream-s.html426227:4867632:36124562

Who's Hiring?

Twitch's commerce team in San Francisco is looking to hire senior developers to keep up with rapidly increasing demand for our Subscriptions and Payment platform. Engineers will be tasked with building new products and features to solve business and ecommerce challenges as we're dealing with engaging problems at a massive scale and will create solutions that impact millions of people around the world. Apply here.

Fun and Informative Events

Advertise your event here!

Cool Products and Services

InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.

Scalyr is a lightning-fast log management and operational data platform. It's a tool (actually, multiple tools) that your entire team will love. Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. . Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

Advertise your product or service here!

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Make Your Job Search O(1) — not O(n)

Triplebyte is unique because they're a team of engineers running their own centralized technical assessment. Companies like Apple, Dropbox, Mixpanel, and Instacart now let Triplebyte-recommended engineers skip their own screening steps.

We found that High Scalability readers are about 80% more likely to be in the top bracket of engineering skill.

The Solution to Your Operational Diagnostics Woes

Scalyr gives you instant visibility of your production systems, helping you turn chaotic logs and system metrics into actionable data at interactive speeds. Don't be limited by the slow and narrow capabilities of traditional log monitoring tools. View and analyze all your logs and system metrics from multiple sources in one place. Get enterprise-grade functionality with sane pricing and insane performance. Learn more today.

If you are interested in a sponsored post for an event, job, or product, please contact us for more information.

Do you like this sort of Stuff? Pleasesupport me on Patreon. I'd really appreciate it. Know anyone looking for a simple book explaining the cloud? Then please recommend my well reviewed book: Explain the Cloud Like I'm 10 (30 reviews!). They'll love it and you'll be their hero forever.

Tim Cook: We at Apple are in full support of a comprehensive federal privacy law in the United States. There, and everywhere, it should be rooted in four essential rights: First, the right to have personal data minimized. Companies should challenge themselves to de-identify customer data—or not to collect it in the first place. Second, the right to knowledge. Users should always know what data is being collected and what it is being collected for. This is the only way to empower users to decide what collection is legitimate and what isn't. Anything less is a sham. Third, the right to access. Companies should recognize that data belongs to users, and we should all make it easy for users to get a copy of, correct, and delete their personal data. And fourth, the right to security. Security is foundational to trust and all other privacy rights.

Eric Feng: there’s now 1 venture capital investment being made every hour of every day, 7 days a week, 365 days a year.

@Werner: Never let facts interrupt a "good story.” Tried to help reporter get it right, but clickbait won. Our Fulfillment Centers have migrated 92% of DBs from Oracle to Aurora with better avail, less bugs and patches, less troubleshooting, less hw cost. More:

@swardley: I occasionally hear companies announcing they are kicking off a "DevOps" program and I can't but feel sympathy for them. 7 years from now, when they finally finish, they will be in a world of capital flow / conversational programming and well ... it's a bit sad really.

@cornazano: Building the wrong thing is a nightmare. Contraining the engineers tends to lead to poorer results; giving them choices produces a better chance of success. #DOES18

@markimbriaco: The greatest trick the devil ever pulled was convincing the world that the network is reliable.

There are a lot more quotes. Can you handle the truth? If so, read on bold adventurer...

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...]]>http://highscalability.com/blog/rss-comments-entry-36122593.xmlStuff The Internet Says On Scalability For October 19th, 2018hot linksTodd HoffFri, 19 Oct 2018 15:59:45 +0000http://highscalability.com/blog/2018/10/19/stuff-the-internet-says-on-scalability-for-october-19th-2018.html426227:4867632:36119183Hey, wake up! It's HighScalability time:

Now that's a cloud! The largest structure ever found in the early universe. The proto-supercluster Hyperion may contain thousands of galaxies or more. (Science)

Do you like this sort of Stuff? Pleasesupport me on Patreon. I'd really appreciate it. Know anyone looking for a simple book explaining the cloud? Then please recommend my well reviewed book: Explain the Cloud Like I'm 10. They'll love it and you'll be their hero forever.

@cpeterso: Your quote reminds me of cybernetics' Law of Requisite Variety: "If a system is to be stable, the number of states of its control mechanism must be greater than or equal to the number of states in the system being controlled."

Mark Graham: I love Google, but their job isn’t to make copies of the homepage every 10 minutes. Ours [Wayback Machine] is.

@gmiranda23:That is legit my favorite quote in IT: “Every application has an inherent amount of irreducible complexity. The only question is: Who will have to deal with it—the user, the application developer, or the platform developer?” -Larry Tesler

dweis: I'm an actual author of Protocol Buffers :) I think Sandy's analysis would benefit from considering why Protocol Buffers behave the way they do rather than outright attacking the design because it doesn't appear to make sense from a PL-centric perspective. As with all software systems, there are a number of competing constraints that have been weighed that have led to compromises.

atombender: I don't think GraphQL is over-hyped at all. Maybe it's flawed, but the design is absolutely on the right traack. GraphQL completely changes how you work with APIs in a front end.

@adrianco: The AWS EC2 NTP service has been backed by atomic clocks for the last few years...

@BrianRoemmele: “Amazon has more job openings in their voice group than Google has in the entire company right now"—@profgalloway This is a #VoiceFirst revolution. Only few astute folks take seriously. This was foolish on multiple dimensions...

@kellabyte: We created TV’s without visible scan lines so artists created simulated scan lines. We moved to digital and artists create simulated analog noise. We created high resolution displays and sometimes artists create simulated pixelation. We create 4K HDR and artists simulate banding.

Steven Acreman: My recommendation is go with Google GKE whenever possible. If you’re already on AWS then trial EKS but it doesn’t really give you that much currently.

Nick Farrell: Music piracy is falling out of favour as streaming services become more widespread, new figures show. One in 10 people in the UK use illegal downloads, down from 18% in 2013, according to YouGov's Music Report.

You can't handle all the quotes that are coming your way...

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...]]>http://highscalability.com/blog/rss-comments-entry-36119183.xmlStuff The Internet Says On Scalability For October 12th, 2018Todd HoffFri, 12 Oct 2018 15:31:37 +0000http://highscalability.com/blog/2018/10/12/stuff-the-internet-says-on-scalability-for-october-12th-2018.html426227:4867632:36119169

There's good news and bad news. The bad news is I'm out sick so there's won't be a post this week. The good news? There will be that much more for next week!