Why do we hash user passwords in our databases? So that if bad actors somehow gain access to our database, they will have a hard time stealing our secrets. But suppose we need to securely store other sensitive data in our database? Encrypt it. But how do we go about that? Assymetric, public/private key encryption with PGP would undoubtedly be the best way to go, but in the case at hand, it’s too impractical. Faced with this situation, I reasoned that symmetrical encryption was the next best option: use one key for both encryption and decryption. But how to store the encryption key securely? As a plain text file sitting on the same server as the database username and password? Which, by the way, are also sitting there in plain text, if not in plain sight?

Disclosures, Disclaimers, Excuses

Now that I have your attention, a quick digression. I am not a security expert — I just try to pay attention to those who are. So please, use your own judgment at your own risk yadda yadda yadda. Moreover, I’m not even officially a web application developer. My job title has nothing to do with that. Long story short: I stumbled into coding some 20 years ago because in my workplace we could find no software, commerical or otherwise, that met our needs in managing a busy federal court interpreters office. So we (OK, I) rolled our own, and have been rolling ever since. Now I’m working on the next iteration of our great project, using Zend Framework 3 and Doctrine, and that’s the context in which this security problem arises.

Wandering back to our main topic: we need to store some sensitive data of contract court interpreters in a database, we want to encrypt it symmetrically, and we need to keep the secret encryption key secret.

Hashicorp Vault to the rescue

The very clever people at Hashicorp generously provide, among other things, a secret-management tool aptly named Vault. As their website puts it,

Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, and more. Vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log.

The use of Vault is a complex subject, and the following discussion is no substitute for the documentation. We should just point out that secrets (yes, that’s the technical term) can be stored in several different types of storage backends, and authentication can happen by way of a number of different authentication backends. These things are identified and accessed by way of paths similar to a filesystem. And that unified interface mentioned above is an HTTP API. You talk to Vault by sending HTTP requests with an authentication token in your request headers, and get JSON responses. Of course, this makes it a snap for applications to communicate with Vault with the language of your choosing.

The challenging part is devising a good security model, and there is a vast — if not to say baffling — array of choices and decisions to be made. I’m still not convinced my current solution is the best, so it’s subject to change. But let’s step through it as it is at the moment. Of course, you’ll first need to install and configure and run Vault, and they make that pretty damn easy. (Though you do have to decide what machine(s) to put it on. There’s replication and high availability and all that sexy stuff for large-scale deployments. Ours happens to be a modest configuration, involving a small set of users behind our organization’s firewall.)

Creating Policies

In Vault, you create authentication tokens that have policies attached to them, determining what the bearer of the token is allowed to do. I want our authenticated application not to be authorized to read our ultimate secret, but to grant an auth token bound to a policy that does allow it. So we need a policy for reading the secret. We also need a policy for creating the token that’s attached to the policy for reading the secret. Sound complicated? Don’t worry: it is! But the underlying principle is a basic one: grant access that is as permissive as necessary, and no more. So I took out my quill and wrote policies, thus:

read-secret.hcl

path "secret/data" {
policy = "read"
}

create-token.hcl

path "auth/token/create/read-secret" {
policy = "write"
}

Incidentally, that .hcl extension you see refers to Hashicorp Configuration Language. It’s a lot like JSON and you’ll pick it up immediately. Then, after logging into Vault with sufficient privileges, we wrote our policies to Vault via its CLI:

vault policy-write read-secret read-secret.hcl

and

vault policy-write create-token create-token.hcl

and finally, the rather more esoteric one:

vault write auth/token/roles/read-secret allowed_policies=read-secret

which magically allows the “role” to bestow access to something that the role itself cannot access.

then with the HTTP API using our good friend curl. By the way, before thinking too much about integration with ZF, I followed the same procedure with all this token stuff until I was able to go from zero to reading the secret (encryption key) from the command line. I predicted (correctly!) that the heavy lifing was setting up the Vault stuff, and getting it to play well with ZF would be more like fun than work.

Getting a token for getting the secret

After the previous command, now that we’re authenticated via TLS, we can go

and that “cipher” you see is the encryption key for encrypting/decrypting our sensitive database fields, a step we will perform in our PHP application.

In all honesty, it’s a bit more complicated than this because we are going to be using response wrapping. When a client sends the header so indicating, the response is not the thing requested, but another auth token with which you can get the thing requested. And that token is a single-use token: use it once and then it’s revoked. You can also limit the time-to-live on this token, so that whether it gets used or expires, it is not going to be hanging around for long as a valid token. This exemplifies the Vault (and general security) principle of limiting the exposure of a secret.

Plugging it into ZF

The front end

At last, the fun part. (Not to say that Vault isn’t loads of fun — it absolutely is! But after dealing and struggling with something unfamiliar, it’s comforting like a warm bath to return to the familiar, isn’t it?) I decided that this application would have its Vault capabilities in a separate module to make it easy to enable or disable. If it’s disabled, they simply don’t get to work with sensitive data. Let’s work from the front end to the back. The form has a good number of fields, but the ones we’re concerned with look like this:

because there’s no reason to expose this data if they don’t ask. The encrypted values are tucked away as hidden fields. When they click those lock thingies, we display a modal dialog inviting the already-authenticated user to re-authenticate (in case they load the form, wander off to the bathroom, and just then an identity thief comes up to their workstation). A couple of Javascript xhr calls later, if all goes well, the dialog goes away and these fields are re-populated with the decrypted values.

The Controller

And the above-mentioned xhr sends a POST request containing the encrypted values to /vault/decrypt, a route mapped to this action method in our module/Vault/src/Controller/VaultController.php:

The part involving Vault gets our encryption key, and it’s a one-liner. The next bit simply makes use of zend-crypt to take care of decryption.

You notice $this->vaultService. That’s a dependency injected via the constructor, so there’s a factory which pulls it from the container. Boring, so we’ll skip that part. Let’s have a look at the module’s module.config.php (ooh, exciting!)

Configuration

<?phpnamespace SDNY\Vault;use Zend\Router\Http\Literal;return['vault'=>[// override these with a local configuration'vault_address'=>'https://vault.example.org:8200','sslcafile'=>'/usr/share/ca-certificates/ca-chain.cert.pem',// these settings must match the configuration set in Vault'ssl_key'=>'/path/to/your-private-key.key.pem','ssl_cert'=>'/path/to/your-cert.pem','path_to_secret'=>'/path/to/your/secret',// including leading slash// but do not change this adapter'adapter'=>'Zend\Http\Client\Adapter\Curl',],'service_manager'=>['factories'=>[
Service\Vault::class=> Service\Factory\VaultServiceFactory::class,]],'controllers'=>['factories'=>[
Controller\VaultController::class=> Controller\Factory\VaultControllerFactory::class,]],];

So the module’s config sets some default/dummy values, with the expectation that they will be overridden by a local configuration file called something like config/autoload/vault.local.php, consistent with the ZF convention. No reason to put this in a public repository.

The Vault service

We have a VaultServiceFactory that injects all that config for us when it instantiates our VaultService, which is what does the talking to Vault.

This Vault service extends Zend\Http\Client because it basically is a specialized http client. The general pattern is that most of these methods attempt to get the token we need, store it in $this->token, and return $this so the lazy coder using the class can save some keystrokes.

We make use of the handy Zend\EventManager\EventManagerAwareInterface and Zend\EventManager\EventManagerAwareTrait and do a fair amount of $this->getEventManager()->trigger(). It’s most definitely a work in progress, but by the time this project is done (or pretty close) there are going to be event listeners responsible for logging things and generally paying attention. On the Vault side, I plan to (figure out a way to) monitor the audit log with the ultimate goal of trying to detect a possible breach and hit the panic button.

So we’ve demonstrated decryption, how about encryption? Coming soon! But as you can see, it will be straightforward.

Further thoughts on Vault and PHP Applications

Remember we started out with a reference to the practice of storing database credentials in plain text? Vault provides a database secret backend so you can avoid doing that. Vault also has a very straightforward user/password authentication backend that makes it tempting to consider implementing a Zend\Authentication\Adapter\AdapterInterface that uses it.

All of this is to say that in terms of security, I think Vault offers us ways to take Zend Framework applications, and PHP applications generally, to the next level.

Conclusion

As noted at the beginning, this solution to my challenge of keeping our encryption key secure may not be the best possible. But it’s a start. The first objective was to avoid leaving our secret lying around in plain text, and this accomplishes that. My holy grail is to make it so that even in a worst case scenario, where an attacker gains root access to the machine where this application is installed, my precious social security numbers and dates of birth would still not be compromised. Like anything, this model is not guaranteed bullet-proof against a breach that severe, but again, it’s a start.

Hopefully this demonstrates that once you have Vault going, it’s quite easy to integrate with your ZF application. As a personal note, I would add that this is actually my debut blogging about Zend Framework, and I do feel a little self-conscious about presuming to address an audience with so many badasses in it. Stage fright notwithstanding, I am truly interested in hearing your comments, so please feel free.

The objective connection between Danny and me is that we are stepbrothers: his father and my mother were married to each other for over 40 years. They met when both were employed at the U.S. Naval Observatory in Washington, D.C.

When I first met him, I was 13 and Dan was 17. Those days were very much the latter phase of the 1960s, culturally speaking. I was then (already) keenly interested in doing drugs and being cool. Dan was the embodiment of all that I aspired to: a cool hippie dude. Complete with long hair, beard, and musical abilities, he was what we used to call “far out” without irony. He was like an older brother and I attached to him immediately.

Our relationship was episodic. For a time in the 1970s we lived together — indeed, in the same room — in La Serena, Chile, where his father directed the Cerro Tololo observatory. In 1982-1983, we happened to overlap again in Tucson when I was graduate student in guitar at the University. After that phase, we would cross paths from time to time owing to the family connection. He showed up for the wedding festivities when I got married (the first of two times) in Puerto Rico in 1994. A few times we coincided while visiting his dad’s/my mom’s home in La Serena and later in Vero Beach, Florida.

Danny was what’s known as a survivor. Of course the past tense is appropriate here — he survived until he didn’t, and if he were here now he’d probably laugh at some grim joke along those lines. But my point is that he overcame some fantastically bad breaks. He was dealt a shitty hand, with mother and brother both suffering from severe mental illnesses and a sister that was… kind of strange, if I may be forgiven for saying so. His dad, my stepdad, was remarkable: an accomplished astronomer and unfailingly loving and generous person. Except that he didn’t much care for dealing straight on with the interpersonal — not an uncommon characteristic, especially among men, especially men of his generation. So he took off for Washington D.C. to take the job at the Naval Observatory, and left Danny at age 15 to deal alone with his profoundly dysfunctional family. And deal with it he did. He got through all that and more, and led a life doing a great many interesting things that he wanted to do, rich in relationships with interesting people.

Truth be told, most of my memories of the times I spent with him are pretty banal, but entirely pleasant. There was good chemistry among our parents and us, and we enjoyed many a happy evening as a merry quartet, eating and drinking well and enjoying lots of long hard laughs. I have a mental picture from one afternoon at a beach in Chile. We waded into the surf and threw a frisbee back and forth, making heroic diving catches of errant throws — the sort of maneuver you might expect from a serious baseball player — but falling painlessly into the water instead of hitting the ground. It was funny, really funny in a physical-comedic way that’s impossible to describe. I have a vivid image of him emerging from the water after one those moves, laughing, laughing…

Now this is part where I make my presentation more compelling by choking up and shedding a tear or two. Note to Danny: that’s another laugh line I think you’d have found amusing.

And now I recover and continue with another fun memory, probably from that same period of time when we spent a couple of weeks in La Serena. We played several games of chess, winning and losing more or less evenly. It was the last night — one of us, I forget who, was traveling back to the U.S. the next morning. So we sat down for a final chess game and abused a bottle of Bailey’s Irish Creme, of all things. That isn’t a very high-alcohol drink and we were both fairly robust drinkers, so we got a little silly but not excessively drunk. The laughter, conversation and chess lasted through the night. The game finally ended in a draw, an outcome that pleased us both.

It is common to canonize people when they die, but there’s no need for that here. Dan definitely had his peculiarities, as do we all in one form or another. He was also a truly extraordinary person, talented and intelligent in a multiplicity of ways, an independent and original thinker, with a sense of humor surpassed by no one. Cliché though it sounds, he lived a full life. But he also did an exceptionally courageous job of dying, from what I can tell. One thing I regret about not attending his memorial in person is not being able to hear more about this from those who were with him at the end. He went out with equanimity, dignity and grace. At times like these I like to quip that dying is so very fashionable. Everyone does it! Those who have not done it yet will get their chance in due course. When my great moment comes I hope to emulate his example.

There is a powerful line in a novel by the writer Haruki Murakami, where two of his characters are in the presence of another who has just died. It reads: he had just accomplished the profound, personal feat of dying.

Good bye, brother Dan. You have accomplished the profound, personal feat of dying.

This little ditty, to the tune of I’ve Got You Under My Skin, was created for the 2013 Courthouse Follies. It is disrespectfully dedicated to the NSA.

They’ve got us under their skin
They’ve got us deep in the heart of them
So deep in their heart, we’re really a part of them
They’ve got us under their skin

We’re always listening in
Their emails and phone calls and texts always go down so well
To our data centers where they’re lovingly stored so well[addressing audience from here on]
You’ve got us under your skin

We’ll sacrifice anything, come what might
For the sake of stoking up fear
We violate every constitutional right
That we still claim to hold dear

Don’t you know little fools, you can never win?
Forget legality
Wake up to reality
If you really do value privacy
You should stop before you log in
‘Cause you’ve got us under your skin

“Capitalism doesn’t create inequality, if it did there wouldn’t be inequality existent in other economic systems which there most certainly is. Inequality is inherent to the human condition. We are not all equally capable or equally industrious so we will not enjoy equal results. There is no system that creates equality because equality doesn’t exist. It’s a fiction people choose to believe. Believe it if you will, but don’t tear down an economic system that provides you with the free time to indulge such fantasies. I’ll grant you that capitalism isn’t fair, but show me an economic model that produces more.
I’ll take freedom over equality any day. Right now, we have neither.”

Hereafter I’m using the second person to address the commenter.

Your remarks reflect such a strong attachment to capitalist orthodoxies that it is unlikely I can say anything to convince you, and there are so many misconceptions and fallacies that it’s hard to know where to begin. But I should try to make a couple points just for the record.

The fact that “other economic systems” — i.e., other than capitalism — have resulted in social inequality (or were likewise based on it, as with slavery and feudalism) in no way proves that capitalism does not also produce inequality. More to the point, the data and the historical record are overwhelming. Few serious scholars, economists, etc. dispute that the global capitalist order has produced massive social inequality – even capitalists acknowledge this. The world’s most powerful, elite government officials, corporate executives and academics gather for their conference in Davos to wring their hands over the global social, economic and environmental crisis resulting from capitalism’s excesses. (And rightly so, because such a system not only morally reprehensible, but also unsustainable.). No, the great dispute is over the way forward.

Further, there really is no other economic system of consequence in the modern global economy, other than capitalism, hence nothing to compare except in historical terms. If you look at history, you can say slave-based and feudal societies were unequal, so that must be the human condition. But this too is fallacious; there is no basis in evolutionary biology, history, anthropology, or what have you, to conclude that capitalist, market-based forms of social organization are somehow genetically embedded in homo sapiens.

You seem to be confusing the Enlightenment-inspired ideal of equality, the notion of human and social rights, with the obvious fact of variation from one individual to another. Of course, some people excel more than others at sports, math, language, music — and some are just plain smarter than most of the rest of us. Some are “industrious,” some are boneheads and slackers. Is that any excuse for a system in which countless millions toil at slave wages while a handful of billionaires control the world’s resources?

Capitalism subordinates social need to private accumulation. Inequality is not just an unfortunate side effect; it’s an essential feature. Marx told you a long time ago that under capitalism, wealth tends to concentrate at the top at the expense of the many, and history has proven it. Yes, there have been booms, as in the post-Word War II period, when living standards in the U.S. rose generally —thanks to the fearless struggles of socialist-minded workers who achieved a few significant social reforms. The data is incontrovertible, however, that for the past three decades or so, wages have stagnated even while productivity has increased. Since the financial collapse 2008, of course, matters have gotten much worse for broad swaths of the population, while the tiny layer at the top enjoys a share of the national income not seen since the robber baron days.
These levels of inequality are wholly incompatible with democracy. You correctly state that right now we have neither freedom nor equality. But that is no mere coincidence. With these levels of inequality, social explosion is inevitable. The ruling class cannot permit freedom in the face of such instability; in order to maintain control, there has to be mass surveillance of the population (brought to you by the NSA), ruthless police violence, and an exaggerated threat of terrorism to incite fear.

You say don’t tear down an economic system that provides me with the free time to indulge my fantasies. The implication here is that if capitalism has been OK for me, and I enjoy a reasonable level of material comfort, then it’s fine, and for those innumerable millions all over the world living in misery, sorry but life is unfair. This view is self-centered and short-sighted. You probably would agree that we should deplore racism even if we are not a victim of it. This is ordinary moral decency. Why is social inequality any different? Because you start from the mistaken assumption that inequality is somehow an immutable characteristic of human social organization, a pessimistic and defeatist worldview with no objective, scientific basis.

The good news is that there is considerable evidence to the contrary. Of course, there are both selfish and altruistic tendencies in human behavior. The solution is to create social structures that make it difficult for the selfish and greedy impulses of a few to screw everyone else. The hope — and there is reason for hope — is that when material want is a thing of the past, the need for extreme greed will diminish and eventually fall away.

A few hundred years ago, slavery and genocide were far more common than they are today. Those who advocated for the abolition of the slave trade were derided as dreamers, or worse. Nowadays, world opinion is virtually unanimous that genocide and slavery are wrong. This is progress, and there are other examples. Human society has progressed from slavocracy to feudalism to bourgeois democracy and capitalism. Fortunately, there is no reason to believe that capitalism is the endpoint of human social evolution. Quite the opposite is true: capitalism gives every indication of coming apart at the seams. What will follow? That is the big question.

My view is that world socialist revolution is the only way forward. The resources of the planet must be brought under the democratic control of the working class — meaning the overwhelming majority of humankind — and utilized rationally to meet the needs of people and the environment, rather than pillaged for the benefit of a tiny minority at the top. If you think this is an unrealistic goal, consider the alternative.

People on Facebook have inviting their FB friends to list the top ten books that have had the greatest impact in our lives. Some people start naming big-name classics like Cervantes, Tolstoy, Shakespeare, Joyce. This strikes me as rather uninteresting, but maybe I am just envious because I am not well-read in the classics. Others are surprisingly candid — or perhaps, naive — in listing some real crap, self-help junk, various pop-schlock titles. I guess I am somewhere between an intellectual and a moron; ignorant, but a snob.
Problem is, I can’t bring myself to do this top ten list. I have been living 50+ years and reading for so long that I no longer remember very well the books that had a great impact at the time I read them, even where their impact was indeed great. The more recently read books tend to displace the old ones. So most of my top ten would be things from the past five years or so. All right, let’s give it a try anyway:
(1) Nietszche – Thus Spoke Zarathustra. Made a huge impression on me when I was 16 years old, so it has to stay on the list. I may not have understood it deeply, but nevertheless.
(2) Juliet Schor – The Overworked American. I read it in the early 1990s and keep remembering it time and again, so it makes the list.
(3) Duke and Gross – America’s Longest War. Greatly informed my thinking about the so-called War on Drugs, which I have been observing from the perspective of a judiciary employee for the past 20 years.
(4) Mathieu Ricard – Happiness. Picked up a copy while looking for something to do at an airport in December 2006. The timing was perfect. It actually changed my life for the better, permanently.
(5) Michael Pollan – The Omnivore’s Dilemma. I was already leaning in the direction of a vegetarian diet, but after reading this, I changed the way I eat.
(6) The Gateless Barrier, a/k/a The Gateless Gate. I was a student at a zendo for about two and a half years, studying with a teacher. Maybe some of it was bullshit. But we went through this koan collection, and I did a lot of sitting (still do hit the mat every day). I know the exercise had a profound effect.
(7) Kurt Vonnegut – Slaughterhouse 5. I never read it until recently — a couple years ago. I think it’s one of the finest novels I have ever read.
(8) Haruki Murakami – 1Q84. It isn’t just this novel, but that it introduced me to this writer and I went on to read several more of his books. You talk about a work of fiction grabbing you in the first few pages. This one grabbed me and did not let go for the next 900+ pages. I don’t know what it is about this guy. He sees the world in a weird way that is peculiar to him, and yet… universal? “Remember: there is always only one reality.” Really?
(9) Don deLillo – Underground. The one that begins at the famous Dodgers-Giants game in 1959. Man, that was one fucking good book.
(10) Terry Eagleton – Why Marx Was Right. My father and I have a decades-long history of talking about politics, about which we generally agree, although I have moved to the left of him. He lent me this book, and it had an enormous impact. Hitherto, I had often said I would consider myself a socialist but for the fact that I had not read any Marx or Engels, much less Lenin or Trotsky. I went on to establish contact with some real, practicing Marxists. In a conversation with one of them — a particularly feisty and erudite old bastard whom I’ll call Fred — he scoffed at Why Marx Was Right, saying Eagleton was a “Catholic Marxist,” i.e., something of a joke. But this book got me started reading some of the works of Trotsky, Lenin, Marx, Engels, and finding out for myself what the political theory is. Combining that with readings of countless contemporary articles that use Marxist methods of analysis, attending some lectures, and observing world events unfold through my own eyes, my political education has advanced greatly in the past two or three years. I am a Marxist-Trotskyist. In this capitalist culture, much of what my generation has been taught about history and socialism is utter nonsense. I have developed a reasonable level of confidence in my ability to sort out the truth from the bullshit.

I never expected this. For over 20 years I was a typical jogger, going out two or three times a week for about four miles at a leisurely pace. With no goal other than fitness and enjoyment, I was unconcerned about going farther or faster. Suddenly I was 55 years old and lacing up my shoes for my fifth marathon.
It started in the winter of 2008, when a friend with marathon experience invited me to join him for a half marathon in Central Park, and I discovered that I enjoyed running nonstop for two hours. The idea of a marathon, hitherto almost inconceivable, became attractive. In the fall of that same year I ran Philly in 3:51, and loved it. I had the appropriate attitude of humility and respect for the distance, running a disciplined, well-regulated race the likes of which, ironically, I haven’t been able to match ever since.
Next was New York in 2009. By now I had moved to South Orange and fell in with a delightful gang of experienced, enthusiastic, and talented distance runners known as The South Mountain Running Collective. I became ambitious, and wanted to meet the Boston qualifying standard of 3:35. Like a fool, I went out too fast, and had to struggle and suffer to hang on over the last 10K, attaining the goal with only seconds to spare. Going out too fast is the classic mistake in distance running, though you might think it would not be so difficult to avoid.
But this performance got me to Boston in 2011, where I repeated the same error, albeit not as drastically: another positive split, but I beat my goal of 3:20 by five seconds, and easily qualified again for Boston in 2012. (Runners call it a negative split where the second half of a race is faster than the first, and it is well established that it is far more effective than the reverse.)
And how could I resist going back to Boston again? I could not, and trained with a view to another personal record (or PR, which is also a verb in runnerspeak) of about 3:17. But the temperatures were in the high 80s that day in April. We were forced to revise our plans and focus on simple survival. Hence my disappointing 3:43, 13 minutes short of Boston qualification (BQ) for my age group.
By now I was thinking it might be nice to get off the crazy train. Marathon training is a time- and energy-consuming pain in the ass, and try though we might to keep it tucked away in its own discreet little compartment, it inevitably has an impact on other people in our lives. And sometimes those people don’t like it. But could I really go out like that, credible excuse notwithstanding? With a 3:43?
No. In November 2013 I was back in Philadelphia again to settle accounts with the marathon gods. This time, however, the training cycle had been more challenging. It is not unusual to encounter setbacks of some kind during 20 weeks of training. Sickness, injuries, family dramas, the demands of work — in short, life — sometimes interfere. I had successfully navigated through bronchitis, a death in the family, and other challenges in previous training cycles. But this time injuries cost me a week and a half in August and several more days in October, and I was not able to get all the mileage I would have liked.
Even so, when I walked up to the line, I felt fit to run a successful marathon. The key workouts in the final stage of training had gone well. My injuries had subsided, the weather was fine, even my pre-race jitters seemed noticeably less severe than in the past. This time I did not have a fixed, specific goal. I thought 3:15 was conceivable, but decided anything up to 3:17:30 would be acceptable. And I had a plan, known as 10 + 10 + 10: go out relatively conservatively for the first 10 miles, i.e., at a 3:17:30 pace; pick it by a few seconds per mile for the next 10 miles; and for the last 10K, be in a position to pick up the pace even more, possibly enough to get me there in 3:15.
Once we got underway, embarrassing though it is to admit, I am not really sure what I was thinking. I do recall making a conscious effort to hold back initially. Over the first mile, traffic was congested, and I decided to go with it rather than fight. My first mile was a 7:42, but that was perfectly OK. A slow start was desirable; there was plenty of time to make it up.
For the next 18 miles or so, it seemed that nobody was in charge. The data on my GPS watch says that mile 2 was 7:18; mile 3, 7:08. For miles 4 and 5 I dialed it back to 7:24, but that was still about 8 seconds per mile too fast. The next few miles included a couple of moderate ones, and my average pace for the first 10 was around 7:30. The pushing and pulling continued over the second 10 miles, but the average overall was around 7:25 — consistent with the overall plan, yes, but too late because I had already blown too much energy in the first 10. By around mile 17 or 18 I was still feeling OK, but tired enough to predict that after 20 miles it was going to be hard to maintain the pace. Miles 22 to 25 were 7:40, 7:44, 7:48, 7:56, 7:55.
Crossing 26 with the crowds screaming encouragement, I was able to pick it up to a 6:53 pace, but over the last 10K I averaged about 7:48 and came in at 3:17:23. (The gory details are published at http://connect.garmin.com/activity/406675139.)
This marathon experience was different from the previous two in which I was seriously trying to reach a goal. In New York, the last 10K were hellish, but my addled mind had enough command of the numbers to understand that I would meet the BQ threshold if and only if I ran like hell. This motivated me to fight hard against the fatigue. The same was true in Boston in 2011: I knew the 3:20 mark was still within reach, but I had to dig. In this race, the goal was comparatively vague: sub-3:17:30, hopefully something closer to 3:15. For the last 10K I felt fatigue, but it wasn’t especially painful — I simply couldn’t convince myself to run faster. But I also felt pretty sure that the 3:17:30, and certainly the PR, was in the bag. That complacency probably hurt my cause; a bit of drama, pressure and anxiety might have provided helpful motivation.
3:17:23 represents a PR by over two minutes, and a Boston qualifier with over 22 minutes to spare; it was also good enough for 16th place among 319 men in the 55-59 age group. Maybe I ought to be happy with that. And although I am not completely disappointed with the bottom line, I am pretty disgusted with myself for not having better discipline. The objective numbers demonstrate that I had the fitness; the subjective experience of how smoothly the whole 26.2 went by, weak finish nothwithstanding, confirms it. I am almost certain I was physically prepared to run at least one, maybe even as much as two minutes faster. But I positive-split it by 1:22 and squandered this splendid opportunity.
Why is it so hard to slow down in those early miles? For me, I think one problem — if it can be considered a problem — is that proper marathon training really works. On race day you are fit, tuned up, but also tapered and rested — not to mention jacked up with the excitement. When the gun goes off and you start running, it is extremely difficult to believe that running can feel this easy and still be fast enough. It seems like you should be exerting at least a little bit. Even the minutes and seconds your watch displays at the mile markers do not convince. So, for me, the lesson is: believe it, bitch. It may even be that I could run a faster marathon next time by setting out to run it slower. Then there might be enough gas in the tank at the end to finish strong enough to get a better result.
Wait — did I just say next time? Uh, yeah, I guess I did. Which brings us back to where we started. I said I never expected to be running marathons, much less running them this fast. I said I would like to get off the crazy train, and maybe I will. But it isn’t easy. When you’re an athlete, you enjoy the challenge of seeing how well you can do, so you keep trying. Eventually the inevitable effects of age overcome the positive effects of training, but that’s no reason to give up prematurely. Another factor is that running marathons (and shorter races as well) is really cool, and fun, and profoundly rewarding. It gets into your bones and becomes part of your identity: you’re a runner. What’s the use of trying to be anybody else?

It has been just about a year since a good friend died at a mere 57 years of age. I observe this anniversary by publishing the eulogy that my evil twin gave at her memorial, with just a couple of minor edits. Whereas he spoke of the legendary SDNY Courthouse Follies in the past tense, I have changed it to the present tense. That’s because at the time, Nancy’s colleagues were too stupified with grief to think seriously about doing the Follies without her. Not long thereafter, they came to the realization that the Courthouse Follies is too good not to continue, and that there is absolutely no better and more appropriate way to honor Nancy’s memory than by putting on the show.

Good evening.

We are all going to die. That simple, self-evident fact is perfectly easy to accept as an intellectual, objective proposition; it is much harder to come to grips with on a deeper, emotional and psychological level.

I’ve come to believe that one of our most important tasks in this thrilling, unpredictable, supremely delightful, horribly painful game known as life is not to obliterate our fear of death, but rather to learn to live with that fear — truly live with it — and be free at last from that ever-present undercurrent of anxiety and fear of our inevitable death. If we accomplish that, we have a chance at genuine happiness, and liberation: the freedom to live our life exactly as it is right now, rather than worrying about losing it.

This is much more easily said than done. In Buddhism they have a word for it: they call it Practice.
Nancy was not a Buddhist — not any kind of -ist as far as I could tell. But she surely had a Practice. I am convinced that on some level she understood and embraced the kind of worldview I’m describing — especially as she approached the end of her life, in which she was short-changed by who knows how many decades. Rather than trying to mention her many wonderful achievements and attributes, I will emphasize this: she had a fabulous sense of humor. Underlying that splendid humor was deep wisdom: though she was no stranger to pain and loss, she knew how to live and enjoy life. She knew how to appreciate life’s offerings. She knew how to have fun.

Over the 20 years I worked as one of the staff interpreters in the office she managed so ably, I must have had hundreds of conversations with her that followed a common pattern: they would begin with me poking my head into her office to discuss something or other, usually office business. She would always look up from her monitor and keyboard to greet me with her characteristic warmth. “Yes, mon cher?” Then one or the other of us would say something funny. And the conversation would end with both of us laughing. You know the way she laughed, with lots of nasal involvement.

We’re talking a lot this evening about Courthouse Follies because the deservedly legendary show was one of her signature achievements, possibly the most visible and well-loved. Entertaining as it is, the show is about so much more than just amusement. No, it is a matter of utmost importance to us all. As Nancy sometimes liked to quip, comedy is serious business. (Not sure whom she stole that line from.) Our job can be brutally stressful. We witness suffering and pain every day. We are all duty-bound to remain stuck in our rigidly defined roles, even at those times when our humanity cries out for us to do otherwise. Nothing is more important to our collective mental health than the Follies. At least once a year we need to sit in a room together and just laugh. Nancy was the leader of that noble project. It was she who put it all together and made it happen. (If this were a federal Sentencing Guidelines analysis, she would get an enhancement for being a leader, supervisor, manager…)
I remember one time when she addressed the cast just before a Follies performance, back stage. She gave us a pep talk in which she reminded us to pay attention and be where we were supposed to be at the appropriate time (incorrigibly ragged band of amateurs that we are, drinking our wine backstage before going on, she had to remind us of that!). She closed these remarks by saying something like: last but not least, have fun. Have fun.

Thank you, Nancy. Thank you for making such a positive contribution to the world and enriching our lives.

You know how she loved comedy. I’ll share this with you on condition of the strictest confidentiality, all 200 of you my closest friends, lest I incriminate myself: I sometimes do judge imitations. Especially when I’m just back from a courtroom assignment with a judge’s idiosyncracies still fresh in memory. Nancy absolutely loved laughing at these imitations. She would sometimes come up to me, eyes ablaze with that childlike, gleeful enthusiasm of hers — and urge me to to entertain the assembly with my imitation of Judge So-and-So. I would comply, and she would laugh and laugh, hoot with laughter.

I usually prefered to launch into an imitation spontaneously, that is, whenever the Muses moved me to it. So there were times when she would ask me to do an imitation, and I would decline on the grounds that I wasn’t in the mood, or some such nonsense. I now understand, and have learned this valuable lesson from Nancy: life is too short not to “do” one more judge. Even on demand.

One of the things I do at my job is maintain a couple of LAMP servers for our office’s use. We recently upgraded them from a long-outdated Ubuntu server edition to Debian 7 and PHP 5.4.4. Among the duties of these boxes is running some archaic and crudely written PHP code, so I expected some compatibility nightmares. I was able to resolve most of the errors, warnings, and weird behaviors one way or another, but the session garbage collection would not work, and I spent too long trying to figure out why. I don’t generally blog about technical stuff, but this might save someone some pain in the future — possibly even myself, when I forget the lesson learned here.

If you’re familiar with Debian and progeny (like Ubuntu), then you know Debian likes to store PHP session data files in /var/lib/php5 and put permissions on it such that if you let PHP try to do garbage collection, it fails with an error like this:

And if your php.ini settings for session.gc_probability and session.gc_divisor are the defaults, this error will only occur 1% of the time, making it fun to debug. But you did your Google diligence and RTFM, and learned that out of the box, Debian sets up a cron job that once per 30 minutes removes session data files whose modification times are older than session.gc_maxlifetime/60 minutes old. The Debian way seems to be to disable PHP garbage collection by setting session.gc_probability to 0 and letting cron do its thing, as
we can see in /etc/cron.d/php5:

Fine. The Debian dudes know what they are doing, so I went with it. Before long, however, I started wondering why none of my session files were getting deleted ever. I am not a black belt in system administration, but I really scoured the logs and tried to figure it out, and Google was no help. So I took a closer look at that cron entry. What’s it doing? In essence, it is using the output of /usr/lib/php5/maxlifetime to determine how many minutes old a session data file has to be to be eligible for deletion. OK, then let’s have a look at /usr/lib/php5/maxlifetime. Not being fluent in shell scripting, I had to squint and scratch my head a bit to follow this.

If you are new to shell scripting and would like to understand this, pay attention. Otherwise, skip ahead. We start by setting max=1440 — that means it will be our default if we don’t overwrite it in the code that follows. That first if condition tests whether there is a php5 binary installed someplace on this system. If so, then we iterate through the standard places where php.ini files are found, and use that php5 to load that configuration file, print out its session.gc_maxlifetime setting, and assign it to the variable $cur. Then we check to make sure we actually got a non-empty $cur before comparing it to $max. By the way, it appears that

[ -z "$cur" ] && cur=0

is a popular shell scripting idiom that is faster to type than the more verbose if construct. The stuff to the right of && only executes if the test on the left of it is true. In this case, the -z tests whether $cur is a zero-length string. This is kind of counter-intuitive to the uninitiated, because one might think the && really means and rather than if. It does mean and, but only if the expression to its left evaluates to true.

If $cur is greater than $max, we assign it to $max, and the result of this iterative processing is that the highest $cur we ever see will become our $max, as long as it’s greater than the initial 1440.

OK, and what’s with the next big else block? If we don’t find a php5 executable, we use sed — the Stream EDitor — to try to parse out the session.gc_maxlifetime value from the .ini file, and otherwise do the same thing as the preceding if.

Well, you may wonder, if they think sed is guaranteed to work, why not just do it that way in the first place? Frankly, I’m not sure. It turns out, as we’ll see, the use of php5 eventually alerted me to a misconfiguration that would not have been picked up had we used sed

So, what is the output of this damn thing, which the cron job is relying on? I ran it from the command line and found that it printed exactly nothing. So I made a copy, which I loaded into a text editor, and started applying my primitive debugging techniques, i.e., using echo statements just to figure out what was getting run and to see the values of things like $max and $cur. Pretty soon I determined that the script was choking on

Then I started looking at the exit status. When I ran /usr/lib/php5/maxlifetime; echo $? from the command line, behold, our exit status is 127. Another consultation with Mr. Google reveals that 127 means “command not found.” What the hell is the command that isn’t being found? I was baffled.

Going back to the very top of this shell script, I wondered what the -e switch means at the shebang line. Back to Google to learn that it means “if not interactive, exit immediately if any untested command fails.” OK, then let’s remove the -e and see what happens:

10: [: Illegal number:
Fatal error: Directive 'allow_call_time_pass_reference' is no longer available in PHP in Unknown on line 0
24

Maybe I am a little slow, but I had to think on this for a while. Who said anything about allow_call_time_pass_reference? Look again at the first thing that php5 command is doing: loading the configuration file indicated by the -c switch. It is puking because there is an invalid configuration directive in php.ini! The above output is telling us that php5 is complaining about that, and the shell script in turn is complaining because it expects a number in the comparison at line 10.

Why is allow_call_time_pass_reference even there in the first place? Because it’s my habit to recycle php.ini across PHP upgrades so I don’t have to hand-edit all my settings again. I fixed that setting and tried again, then got another complaint about register_long_arrays, another configuration setting that is no longer valid in PHP 5.4. Both of these really are things I can do without, so good riddance. The lesson is that if you are going to keep an old php.ini, you should test it after an upgrade with a php5 command that does what this does: load the config, set error reporting to the max, and see what happens.

Now it’s clear that -cmin was getting an empty string, which will make find choke. But cron doesn’t email me the error output because 2>/dev/null sends it into the void.

The thing that made this adventure so much fun is that the cron job was failing because the shell script was failing because the php5 command therein was failing because of an obsolete php.ini setting that had no relevance to sessions — several levels of indirection — and error suppression was making it all the harder to track down.

A final weird note: I never did figure out why I was getting exit status 127. I tried to recreate the scenario while writing this entry, and couldn’t get it to happen again; instead, I got exit status 1. Something was different. That’s one mystery I can live without solving.

I can’t help but respond to this blog post that appeared on the website of the National Association of Judiciary Interpreters and Translators. Jennifer de la Cruz’ thoughtfulness and good intentions are commendable, but underneath all this positive attitude and cheer I detect the sickly spirit of pessimism and defeatism so deeply internalized that it is all but unconscious — hence the analogy between austerity economics and tornadoes. The latter may be partly anthropogenic, thanks to global warming; but the former is entirely so, and there is a good deal more we can do about it. Employment security, education, housing, health care, retirement security, and even the enjoyment of culture and leisure are — guess what! — human rights. The fact that those who point this out are regarded as Bolsheviks attests to how far our entire political culture has shifted to the reactionary right. We are expected to bow down and be grateful to The Man for our crust of bread. The trade unions, by and large, are collaborationists who clamor for a seat at the table where they can participate in the slashing of wages and the gutting of benefits.
So we are told to embrace change and be positive as this attack is going on. No. We need to work on our political consciousness, clear the clutter from our minds and see the big picture. You know the facts. Income inequality has gone through the roof even as we are told there is no money for frills like education and health care, even as Wall Street and the war machine are still lavishly funded courtesy of the taxpayer.
As we speak, the Federal Defenders in Manhattan — a first-rate squad of extremely hard-working, committed lawyers who represent indigent defendants in federal cases — are subjected to furloughs tantamount to a 20% pay cut with no reduction in work load. As we speak, millions of federal workers like your humble servant are in their third year with no cost of living adjustment, even as the cost of living itself has risen — a de facto pay cut. As we speak, public sector workers in the judiciary, and elsewhere, are being furloughed or laid off. There are, of course, innumerable other examples, many of them even worse.
Economic inequality of fantastic proportions, hideous social problems (including mass incarceration, of which court interpreting is but one of many spinoff industries), environmental destruction: the word for this grotesque condition is capitalism, a system where virtually by definition, the needs of people are subordinated to private gain for the few. What we need is a worldwide socialist revolution whose aim is to invert these priorities and place the resources of the planet under the democratic, rational control of the working class, by which I mean at least 90% of the world population. No one is saying this is going to be easy. But the alternative is far more dire: quite possibly, species extinction; at a minimum, further deterioration of your material conditions and a shitty future for you and your children.
Dear reader, if you are snorting “good luck with that revolution thing,” then let us return to where we started. Defeatism and pessimism will get us nowhere. Rather than bowing down and “embracing change” with pop-psychological cheerfulness, we should be planning ways to resist and fight back. One idea that comes to mind is what I call the counter-furlough. For every furlough day that is meted out to workers, every worker strikes for one day, furloughed or not, and not just in the industry of the afflicted workers, but as far across the economy as we can manage. This action should be organized not under the aegis of a union, but by ad hoc rank-and-file committees of workers themselves. Do not wait to be saved by reforms implemented within the Republican/Democrat political framework because that, truly, will never happen.
Yes there is enough to go around. Please think about this and use your imagination to envision a world in which things like employment insecurity, hunger and poverty are all but eliminated — change of the sort we can truly embrace. If you are curious to learn more, I invite you to check out the Socialist Equality Party. You have nothing to lose but your chains.

I grew up as the child of liberal parents in the 1960s. One was an astronomer who was active in the women’s movement against the Vietnam war; the other, a musicologist then working as a journalist and book critic whose scholarly reviews were sympathetic to lefty and liberal ideas. Both were committed supporters of the civil rights movement. My father insists that our phone was tapped during those turbulent years. I think that’s uncertain, but by no means implausible.
After going through a more or less apolitical period in my teens and 20s, I gradually became a political animal, watching public affairs ever more closely. And the more I paid attention, the more I understood that notwithstanding the occasional battle won by progressive elements, the United States is fundamentally not a participatory democracy, but rather is ruled by the wealthy and corporations. Not coincidentally, wages for the many have stagnated over the past few decades even while total productivity has increased, and of course economic inequality has skyrocketed. These propositions are not controversial, but well established by objective data.
Meanwhile, in the realm of foreign policy, we’ve seen such monstrous criminal misadventures as the Iraq war, with complete impunity for its perpetrators. Now, under Obama — the darling of so many liberals — mass surveillance of the population, indefinite detention without charges, and assassination have become institutionalized in the name of the so-called War on Terror. On the domestic front we have rampant unemployment and underemployment; millions in debt servitude for the sin of attending college; mass incarceration on a scale unparalleled anywhere else in the world; no sign of serious response from the Obama administration in the face of catastrophic climate change; and the list goes on.

Financial deregulation, and the spectacular display of greed and corruption that ensued, resulted in the meltdown of 2008, coming around the same time that candidate Obama was absurdly being called a socialist by the far right. I began to think, would that it were so. Capitalism is a disaster for most of the world’s population. True, the middle class generally has done pretty well during boom times when there’s enough to go around, and with a labor movement driving some reforms. But with the hegemonic power of the USA in decline, those post-World War II days are gone. So long gone, in fact, that millions belonging to the generations born in the 80s and 90s have never experienced those Leave It To Beaver days of prosperity, as my friends at the Socialist Equality Party point out in lucid detail. Hereagain, this should not be too controversial a proposition. Even the cream of global elites get together for their conference in Davos to worry about inequality getting out of hand and causing severe unrest. In a sense, capitalism is the victim of its own “success.” The more rich and powerful the top layer gets, the more its rigs the system so it can grab more wealth and power — in a vicious circle that has long since gotten out of control. Hence the crisis of capitalism.

But I still didn’t quite get it. I was in what I now think of as my middle-class protest politics period. For years I gave what money I could to progressive causes; emailed and called my elected officials to urge them to this or that; went to street demonstrations. Weary from the Bush years, I dropped to my knees and voted for Obama in 2008 even though I well knew both parties were owned by big business. In 2009, the uproar broke out around Obama’s much-acclaimed, much-maligned healthcare “reform” legislation. I got involved in Single Payer activism, and even suffered the inconvenience of spending a night in jail for doing civil disobedience — sitting in at the offices of a large health insurance company.

Of course, all that effort came to naught. The healthcare fiasco provided an instructive example. There were four main guys in the House and Senate who advocated for Single Payer during the Obamacare debate: Bernie Sanders, Dennis Kucinich, John Conyers, and Anthony Weiner (before he was disgraced for emailing pictures of his dick). When the time came, each and every one of them sold out and voted for Obama’s massive bailout to the profit-driven private health insurance industry, a piece of legislation substantially written by its lobbyists. This is just one case, but it illustrates an essential point that I have since come to understand: reformism doesn’t work. Capitalism subjugates social need to private profit, and it requires inequality — that’s how it works. To paraphrase Leon Trotsky, capitalism itself has to be abolished, not reformed.

I wondered: what’s the alternative? I knew about socialism approximately as much as the average reasonably educated US citizen who grew up in the Cold War: not much, really. I had never actually read anything by Marx or Engels. I stumbled across the World Socialist Web Site, started reading it regularly, and became even more curious. I had never seen anyone state the political truth with such unrelenting bluntness. Who were these guys? Just as my theoretical curiosity was thus aroused, my father happened to lend me Why Marx Was Right by Terry Eagleton, in which he assures us that if you haven’t read anything by Karl Marx, no worries, this book makes a good introduction. (I have since talked with serious Marxists who scorn Eagleton as a “Catholic Marxist,” but the book was nevertheless a useful introduction.) Then I took the trouble to read some texts by Marx, Engels, Lenin, and Trotsky to see for myself what they actually say, and was struck by their prescience and continuing relevance. I followed up with some more current writings, such as David North’s In Defense of Leon Trotsky, The Historical and International Foundations of the Socialist Equality Party, and the SEP Statement of Principles.

Once exposed to a bit of Marxist theory, I began to get it. The history of human social organization goes from slavery-based societies to feudalism to capitalism to… what? It is by no means a foregone conclusion that capitalism is the end point of the evolution of human society (Francis Fukuyama’s famous, now discredited, assertion to the contrary notwithstanding). If it turns out that capitalism is the final word, it will be because the human species extinguishes itself under capitalism, most likely by way of environmental catastrophe, before it gets its shit together.

So what sort of world socialist society do I envision? One in which the wealth of the planet is utilized rationally and democratically, i.e., with the priority on social need over private profit and accumulation — the inverse of capitalism. If that sounds somewhat vague, it is. I haven’t mastered all the implementation details. If it sounds ambitious, it is; history teaches that the struggle has been and will likely continue to be long and bitter. If it sounds like so much dreaming, it isn’t. Dreamers are those who think the human species has any chance of survival under capitalism.

Martin Luther King once said “the arc of the moral universe is long, but it bends toward justice.” Although his statement may have been partly rooted in religious faith, there is some objective historical evidence that the principle is correct. A few centuries ago, genocide and enslavement were commonplace. Now world opinion is in general agreement that genocide and enslavement are wrong, and people get upset when they happen. Some day people will likewise look back at history to an economic system where the many — i.e., the working class — suffered and were exploited so that a few could become fabulously wealthy, and they will find it appalling and unacceptable, just as we consider slavery appalling and unacceptable today. In the future, the revolutionary Marxists of today will be recognized as having been ahead of their time.