Tuesday, March 28, 2017

The insidious thing about research debt is that it’s normal. Everyone takes it for granted, and doesn’t realize that things could be different. For example, it’s normal to give very mediocre explanations of research, and people perceive that to be the ceiling of explanation quality. On the rare occasions that truly excellent explanations come along, people see them as one-off miracles rather than a sign that we could systematically be doing better.

This book helps you gain the foundational knowledge required to write an operating system from scratch. Hence the title, 0 to 1.

After completing this book, at the very least you will learn:

How to write an operating system from scratch by reading hardware datasheets. In the real world, it works like that. You won’t be able to consult Google for a quick answer.

A big picture of how each layer of a computer is related to the other, from hardware to software.

Write code independently. It’s pointless to copy and paste code. Real learning happens when you solve problems on your own. Some examples are given to kick start, but most problems are yours to conquer. However, the solutions are available online for you to examine after giving it a good try.

Linux as a development environment and how to use common tools for low-level programming.

x86 assembly in-depth.

How a program is structured so that an operating system can run.

How to debug a program running directly on hardware with gdb and QEMU.

Linking and loading on bare metal x86_64, with pure C. No standard library. No runtime overhead.

In this series of articles, I’d like to walk you through the default diff algorithm used by Git. It was developed by Eugene W. Myers, and the original paper is available online. While the paper is quite short, it is quite mathematically dense and is focussed on proving that it works. The explanations here will be less rigorous, but will hopefully be more intuitive, giving a detailed walk-through of what the algorithm actually does and how it works.

A collision attack is therefore well within the range of what an organized crime syndicate can practically budget by 2018, and a university research project by 2021.

Pretty good estimate, I'd say, Mr. Walker.

But what does this mean, in practice?

Perhaps the most visible impact is in the area of network security, where Google has been warning about problems for quite some time, and started putting those warnings into action last fall: SHA-1 Certificates in Chrome

To protect users from such attacks, Chrome will stop trusting certificates that use the SHA-1 algorithm, and visiting a site using such a certificate will result in an interstitial warning.

Other large internet sites have followed suit; kudos to them for doing so quickly and responsibly.

Another very interesting aspect of this signature collision arises in what are known as "content-addressible file systems", of which git is the best known. This is a very significant issue, as the Shattered web site points out:

It is essentially possible to create two GIT repositories with the same head commit hash and different contents, say a benign source code and a backdoored one. An attacker could potentially selectively serve either repository to targeted users. This will require attackers to compute their own collision.

However, when it comes to the SCM issue, I think that the issue isn't completely cut-and-dried, for several reasons:

Firstly, we're talking about an issue in which an attacker deliberately constructs a collision, as opposed to an accidental collision. The use of SHA-1 identifiers for git objects remains a useful, practical, and trouble-free technique for allowing people to collaborate independently on common computer files without sharing a central server (the so-called DVCS paradigm). In the 12 years that git has been in use, and the trillions of git object SHAs that have been computed, nobody anywhere in the world has reported an accidental collision in practice.

This strength of accidental collision detection is strengthened by the fact that git encodes certain other information into the computed SHA-1 value besides just the file's content: namely, the object type (blob/tree/commit/tag), and the object length, for blob shas, and other ancillary data such as timestamps, etc. for commit shas. I'm not saying this makes git any safer from a security point of view; after all Google arranged to have their two colliding PDF files be both exactly 422,435 bytes long. But it does mean that the accidental collision risk is clearly quite small.

And, of course, for the attacker to actually supplant "a benign source code" with "a backdoored one," not only does the attacker have to construct the alternate file (of identical length and identical SHA-1, but with evil content), but that backdoored file has to still be valid source code. It is no easy task to add in this additional constraint, even if you are the wealthy-enough attacker to be willing to spend "9,223,372,036,854,775,808 SHA1 computations". I'd imagine that this task gets easier, somewhat, as the size of that source file gets larger; that is, given that a certain amount of the backdoored evil source file is necessarily consumed by the source code of the evil payload itself, the attacker is forced to use the remainder of the file size for containing the rubbish that is necessary to make the SHA-1 values line up, and the smaller that remainder is, the harder it will be to generate that matching SHA-1, right? So it's one more reason to keep your individual source files small?

The above was too many words: what I'm trying to point out is:

With SSH, people use SHA-1 to provide security

With git/Mercurial, people use SHA-1 to provide decentralized object identification workflows, for easier collaboration among trusted teams.

That is, when you connect to a valuable web site using SSH, you are depending on that SSH signature to establish trust in your mind between yourself and some remote network entity.

But when you share source code with your team, with whom you are collaborating using a tool like Mercurial, Subversion, or git, there are, crucially, other trust relationships in effect between you and the other human beings with whom you are a collaborator.

So, yes, be careful from whom you download a git repo full of source code that you intend to compile and run on your computer.

Thursday, March 23, 2017

Various estimates of the scale of need for basic skills services in the region
convey a crisis-level order of magnitude.

The National Institute for Literacy estimates that 47% of adults (more
than 200,000 individuals) in the City of Detroit are functionally illiterate,
referring to the inability of an individual to use reading, speaking, writing,
and computational skills in everyday life situations.

We also know that of the 200,000 adults who are functionally illiterate,
approximately half have a high school diploma or GED, so this issue cannot
be solely addressed by a focus on adult high-school completion.

The remaining 100,000 of these functionally illiterate adults (age 25 and
older) lack a high school diploma or GED, another prerequisite for
employment success.

I'm not sure how this institute made this estimate.

Later, the report expands somewhat on the topic:

Generally, those adults who score at Level 1 (on a scale of 1 to 5,
lowest to highest) have difficulty performing such everyday tasks as
locating an intersection on a street map, reading and comprehending
a short newspaper article, or calculating total costs on an order form.

It isn't clear whether their estimate was that all 47% were at "Level 1", or whether those were five levels of illiteracy (versus five levels of literacy), but no matter how you slice it, those are some astonishing claims about the literacy problem in the greater Detroit region.

Tuesday, March 21, 2017

There is something pleasing about the fact that the Charging Bull, a global symbol of rapacious financial capitalism, is a piece of guerrilla art installed without payment or permission -- while the Fearless Girl, an egalitarian symbol meant to challenge the bull's soulless greed, is a piece of corporate advertising commissioned by an asset-management company.

I've now read Sapiens, which is both readable and thought-provoking, no easy accomplishment.

Harari is certainly ambitious. As I read Sapiens, I amused myself by pretending to be a library cataloger, faced with the task of trying to assign appropriate subject categories under which Sapiens should be listed.

But that's not adequate either, for you'd want to be more precise that just saying "history", rather: world history; cultural history; ancient history; history of language; military history; world exploration; religious history; history of science; literary history; etc.

Oh, you could go on for hours and hours.

So, Sapiens is very much a book written by an intellectual omnivore, which will most likely appeal to omnivorous readers, by which I mean those who don't want to spend their time reading history books that get trapped for many pages on the individual details of precisely what happened on such-and-such a day, but instead feel like it's reasonable to try to cover the 100,000 year history of mankind on earth in, say, 400 pages or so.

It actually works out better than the previous sentence makes it sound, for Harari is a fine writer and he moves things along briskly.

I think that the strongest and most interesting argument that Sapiens makes is a linguistic one, rooted in the power of the concept of abstraction.

Discussing the evolution of language itself, Harari observes that many species of animal have languages and can communicate, typically using their language abilities to communicate information about food, danger, reproduction, and other universal topics. However:

the truly unique feature of our language is not its ability to transmit information about men and lions. Rather, it's the ability to transmit information about things that do not exist at all. As far as we know, only Sapiens can talk about entire kinds of entities that they have never seen, touched or smelled.

Legends, myths, gods and religions appeared for the first time with the Cognitive Revolution. Many animals and human species could previously say, 'Careful! A lion!' Thanks to the Cognitive Revolution, Homo Sapiens acquired the ability to say, 'The lion is the guardian spirit of our tribe.' This ability to speak about fictions is the most unique feature of Sapiens language.

Although, superficially, this seems to be a discussion about telling entertaining stories around the campfire, or fabricating super-natural explanations as the basis for the founding of religions, Harari quickly re-orients this discussion in a much more practical direction:

fiction has enabled us not merely to imagine things, but to do so collectively.

...

Such myths give Sapiens the unprecedented ability to cooperate flexibly in large numbers [...] with countless numbers of strangers.

It's that "with ... strangers" part that is so important, as Harari proceeds to demonstrate how this ability to discuss hypothetical scenarios with people who aren't part of your immediate circle of family and friends is what gives rise to things like corporate finance, systems of justice, the scientific method, etc. All of these things are built on the ability to have abstractions:

In what sense can we say that Peugeot SA (the company's official name) exists? There are many Peugeot vehicles, but these are obviously not the company. Even if every Peugeot in the world were simultaneously junked and sold for scrap metal, Peugeot SA would not disappear.

...

Peugeot is a figment of our collective imagination. Lawyers call this a 'legal fiction.' It can't be pointed at; it is not a physical object. But it exists as a legal entity. Just like you or me, it is bound by the laws of the countries in which it operates. It can open a bank account and own property. It pays taxes, and it can be sued and even prosecuted separately from any of the people who own or work for it.

Ostensibly, Sapiens is a history; that is, it is a book about the past, helping us understand what came before, and how it led us to what is now.

But, as is perhaps universally true, Harari is not actually that terribly interested in what happened in the past, often breezily sweeping whole questions aside with a sort of "it's gone; it's forgotten; we have no accurate evidence; we cannot know for sure" superficiality that is startling.

Rather, as Harari reveals near the end of his book, he is principally interested in the future, and it's here where Sapiens takes a rather unexpected turn.

I must admit, I was wholly unprepared when, just pages before the end of Sapiens, Harari suddenly introduces the topic of "Intelligent Design".

However, it turns out that Harari doesn't mean the term in the sense in which it is typically used; he is firmly in the Darwin/Russell camp.

Rather, Harari is fascinated by the idea that scientific methods may have arrived at the point where humans will soon be capable of intelligent design in the future:

After 4 billion years of natural selection, Alba stands at the dawn of a new cosmic era, in which life will be ruled by intelligent design.

...

Biologists the world over are locked in battle with the intelligent-design movement, which opposes the teaching of Darwinian evolution in schools and claims that biological complexity proves there must be a creator who thought out all biological details in advance. The biologists are right about the past, but the proponents of intelligent design might, ironically, be right about the future.

At the time of writing, the replacement of natural selection by intelligent design could happen in any of three ways: through biological engineering, cyborg engineering (cyborgs are beings who combine organic with non-organic parts) or the engineering of in-organic life.

If Harari painted with a broad brush when discussing the past, his descriptions of our near-term future are equally vague and loosely-grounded, and those final 25 pages of Sapiens are a rather bewildering peek into "what might be."

But, as Yogi Berra pointed out, "predictions are hard, especially about the future," so I can't fault Harari too much for wanting to have a go at what might come next.

I imagine that, eventually, I will read more of Harari's work, as it's clear he has a lot of interesting things to say.

What I yearned for at Oracle was clarity on our vision and the goals we wanted to achieve. As I started to manage my own divisions, I found that I personally lacked the tools to spell out what we needed to do and a simple a process to communicate it. The problem only increased as the teams that I was managing increased.

...

At salesforce.com, everything we do in terms of organiza­tional management is based on our V2MOM. It is the core way we run our business; it allows us to define our goals and organize a principled way to execute them; and it takes into consideration our constant drive to evolve. The collaborative construct works especially well for a fast-paced environment.

I can greatly sympathize. It is not a great exaggeration to say that the reason I changed jobs this winter was because I realized I was no longer in alignment with my (former) company. In fact, we hadn't been aligned for nearly a year. I wanted to take the technology, and the products, and the customer base, in a certain direction, but the company had entirely different plans, and goals, and intentions.

That's fine. But what's NOT fine, is that I didn't know that at the time. Horribly, I didn't know it for nearly a year. Which is a shame, both for me, and for the company, as neither of us were well-served by that disconnect.

before the dinner was over, Harris walked up to Benioff and gave him a gift: a framed American Express envelope.

It was the envelope Benioff had used to scribble down Salesforce’s first-ever V2MOM — a list of management guidelines that stands for vision, values, methods, obstacles, and measures — when launching the company in 1999.

The goal of the V2MOM is create complete alignment. Immediately after writing it, share it with your top officers for input (for a startup, this is probably everyone). The brevity ensures a simplicity that is easy to digest. Clarified direction focuses collective attention on the desired outcome and eliminates anxiety in times of change. It is easy for people to connect with and scan quickly for alignment. The V2MOM is flexible enough for startups as well as public companies.

A few months ago I had a great meeting with a good friend and one of my mentors, Mariusz, who is already running a very successful Internet company (and a lot bigger than mine). We talked about team-building and how to maintain focus and make sure the team feels like "one vehicle driving in one direction" and everyone knows they have a big role to play and depend on each other. He suggested I read the "Behind the Cloud" book by Marc Benioff and implemented the V2MOM system Marc invented. I was like "V2what?" and he explained

So, anyway, we're now nearly done with the big annual V2MOm process for this year. The process proceeds top-down:

Marc writes his V2MOM, which is the V2MOM for the entire company, and publishes it

Then each level down the org chart writes, and publishes, their own V2MOM, extracting, selecting, refining, and elaborating on the V2MOMs already published

Eventually, we get down to people like me, and once we've published our V2MOMs, the annual publication event completes.

This is, obviously, the first time I've been through this process, so it's not clear what standing I have to comment.

But it's been interesting enough that I'd like to share a few thoughts.

EVERYONE participates. This is not an optional activity. Some people put more time into it, others less, but nobody sits out entirely. That fact, by itself, creates a curious sense of "belonging," all by itself.

This is not just an exercise for show. The company takes this process VERY seriously. People devote substantial amounts of time to drafting, discussing, revising, and editing their V2MOMs.

The plans are interesting, but much more interesting and much more important is the fact that we are PLANNING. Recall Eisenhower:

Plans are nothing; planning is everything.

At the middle levels of the organization, the V2MOMs describe, collectively, the work of teams of dozens or even hundreds of people, and they can be impressively detailed and robust. I participated in a 3 hour "readout" (a bit of business jargon which I'm told has Microsoft-heritage), in which my 50+ person team collectively reviewed a 35-page detailed description of our goals, aspirations, and worries for the year.

These are not private documents. Everyone's V2MOM is made available to the entire company (though obviously I'm not going to sit down and read 28,000 V2MOM documents).

In fact, you could say that this is perhaps the entire point, as the openness of the V2MOM process is a great example of what people mean when they talk about "transparency."

A crucial part of the V2MOM process involves ORDERING. When you choose your methods, you have to place them in a certain order, and this order conveys your priorities. Your top methods are crucial; these are the things you will fight to accomplish this year. Farther down the list, are things that you believe in, and want to do, but may not be able to achieve.

A famous cliche goes: "if everything's important, nothing's important." Placing your methods in a definite order forces you to stop and think about what REALLY matters.

And people pay attention to this order. They think about it; they arrange their own work around it; it structures the entire conversation. There is an often-retold story inside Salesforce about a very public meeting that occurred not too long after Keith Block had joined. It happened to be V2MOM time, and so Block was producing his V2MOM, and, as part of that process, it was being presented to the team, which meant that it was being presented to, more-or-less, the entire company (Block is maybe the 2nd or 3rd-most important person at the company). During this (open to all, broadcast, widely-watched) event, Block is stepping through his methods, one at a time, when a voice from the audience interrupts: Benioff himself:

Keith, here, I'm a bit puzzled: why did you prioritize this as Method 4? What makes it less important than numbers 2 and 3?

The message: this is important; this is open; nobody gets a free pass; we are all agreeing on this together.

But after all of this, I'd say that the single thing that startled me the most about the entire V2MOM process is: everybody does it!

Even in a small company, it's rare to find anything that everyone does. Corporate activities like this tend to be the sorts of thing that see 20% participation, at least in the corporate settings that I've been part of.

So the simple fact of saying that we ALL have a V2MOM is marvelously compelling.

I’ll tell you what: first of all, stop calling him that. Brilliant is just how we excuse Jerk. It’s how he gets away with bad behavior, like sexual harassment and insubordination (and yes, he is usually a he). Let me get all third grade on you for a second here: If you’re so brilliant, then why haven’t you figured out how to be good at your job? The jury is long out on the jerks’ performance: they may be highly productive, “but they are not, however brilliant business people,” wrote Cliff Oxford in the Times. This is because good business people, strangely enough, are good people.

We are making several changes as a result of this operational event. While removal of capacity is a key operational practice, in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove capacity more slowly and added safeguards to prevent capacity from being removed when it will take any subsystem below its minimum required capacity level. This will prevent an incorrect input from triggering a similar event in the future. We are also auditing our other operational tools to ensure we have similar safety checks.

At Spire, we’ve been using Kubernetes for a little over 9 months at this point, the last 6 of which were in production. It’s transformed our workflow and provided us with a significantly more reliable product. If you’re considering a move to Kubernetes, I highly recommend it. It’s an incredibly powerful tool that is guaranteed to leave you in awe at least a few times.

Cloudflare points out that the flaw meant that its servers leaked private information just once in every 3.3 million Web requests it dealt with. But such is the scale of Cloudflare’s operations that those numbers add up—and quickly. Among its clients are the likes of Uber, Fitbit, OKCupid, 4chan, and 1Password. All told, as many as 120,000 pages per day from 3,438 domains could have leaked data, and the bug remained undiscovered for over five months.

the development includes the construction of two high rise towers and the renovation of two existing buildings, 78 and 88 1st Street. The 910 foot tall, 61-story 1st Street Tower will include a 4 story basement, a 7-story tall open public space on the ground level, 33 stories of office space (1.35 million square feet) and 109 ultra-luxury condos. The steel 1st Street Tower will be the second tallest building in San Francisco and will be targeting Platinum LEED Certification.

I have long had an image in my head of what Margaret Wise Brown must have been like and Amy Gary's new book, In the Great Green Room totally blew that away. Brown was a firecracker. She was an innovator. She was amazing.

Schafer: The thing about using clairvoyance in Psychonauts is that it didn’t just let you teleport around, but also let you feel like you’re seeing the world through someone else’s eyes. So if you’re looking through the world through the POV of a big person or a small person or a tiny crab sandwich, or some insects, or giant creatures — you want to feel big, or you want to feel small. We want to represent that altered mental state as you see through their eyes, so it’s teleportation but also an empathy device.

Wednesday, March 1, 2017

At a GDC talk on Monday, CD Projekt animation director Sebastian Kalemba put some impressive numbers to those scenes

...

Kalemba's talk was mostly about the obstacles his animation team faced in tackling so much work for the DLC in less time, and how they improved their workflow to make it all happen. Some of that best practices talk is more interesting for developers than us, but other parts felt like glimpsing a tiny piece of the formula that made The Witcher 3's cinematics a cut above.