Friday Philosophy – Whatever Happened to Run Books? July 27, 2012

I realised recently that it is many years since I saw what used to be called a Run Book or System Log Book. This was a file – as in a plastic binder – with sheets of paper or printouts in it about a given system. Yes, this was a while back. It would often also have diagrams {occasionally drawn by hand on scraps of paper – so that would be the database ERD then}, hand-written notes and often the printed stuff would have scribbles against it.

{BTW I asked a colleague if he remembered these and when he said he did, what he used to call them – “err, documentation???”. Lol}

There was one book per key system and you could tell if a system was key (that is, Production, or a development system where a large development manager would punch you in the eye for losing anything, or any system the DBAs wanted) as it had a run book. It held information that was important about the system and, although you could look up most of it when logged onto the system itself, was useful to grab and just check something. However, it was vital if you had to recover the system.

Being a DBA-type, the run books I used to see and use were database focused. The front page would have the SID, name, host name (and even the spec of the host), version, tnsnames info, block size, backup strategy and schedule and, very importantly, the system owner. Yes, the big guy who would be upset if you lost the system. In there you would have printouts of the tablespaces, datafiles and sizes, the backup script, users (and passwords, very often), reference data tables, filesystem layout, OS user details and anything else
needed to recover the system.

This was an evolving and historical set of data. I mentioned above that you would have maybe scraps of paper from when a design session had come up with an alteration to the system. Corrections would often be done by hand. When you printed off the tablespace sizes on Monday, you did not throw the old one away but just added the new one, so you had information about the growth of the DB going back in time. Once in a while you might thin out the set but you kept say one a month.

It was actually that which got me to thinking about runbooks. At a site recently one of the DBAs was asking me if I knew of a screen in OEM that showed the growth of space used over time and my immediate thought was “well look in the run book” {I was very tired that day and losing my grip on reality}. Not being able to find a screen for what he wanted and knowing the data in OEM/AWR was only going back a month anyway, I suggested a simple spreadsheet that he could maintain. With the run book you could flip to the printouts of tablespace sizes, grab a piece of paper and do something lo-tech like this:

This would take less time than firing up Excel, typing the figures in, getting the graph wrong 3 times and then printing it out. Though if you had to go show Managers how the data was growing, you invested that time in making it pretty {why do high level managers insist on “pretty” when what they really want is “informative”?}

So why have Run Books gone {and does anyone out there still use them, in physical or electronic format}? It certainly seemed standard practice across IT in the 80’s and 90’s. I suspect that the reason is that most of the information that used to go into them is now available via online GUI admin tools and looking at them is actually faster than going and grabbing a physical book. Besides, if your DBA or Sys Admin team is split between UK, India and Australia, where do you keep a physical book and allow everyone to check it? I have vague memories of electronic Run Book applications appearing but they never seemed to get traction.

That is one of the drawbacks of using GUI admin tools. No, this is not just some tirad by a bitter old lag against GUI tools – they are generally a massive improvement on the old ways – but they are not perfect. Most of them only hold a short history and printing out the data is often tricky or impossible. All you can really do is screen dumps. No one has those little scripts for listing out basic information anymore {except us bitter old lags} as they have GUIs to do all that and, heck, I can’t go printing off a load of stuff on paper and sticking it in a binder – that is so 20th century!

Maybe I’m being unfair and OEM has a “run book” section I have simply never seen – but I’ve never seen it. If it is/was there, how many people would use it?
I do miss the Run Book though. Especially the ease with which I could look up all those passwords…

SID – people tend to remember this, and it is usually distributed all over the placename – people tend to not care or see SIDhost name – see SIDthe spec of the host – network admins usually have this since they have to keep a spreadsheet or facility tracking system for a gazillion PC’sversion – DBA’s tend to keep this for all their db’s. One time when I was doing customer dba support, one upset customer got ahold of me and demanded to know what version I was running – I couldn’t come up with a quick answer, because I had already been on half a dozen systems that morning.tnsnames info – see SIDblock size – does anyone not use 8?backup strategy and schedule – now, here is a set of procedures that is still, and ought to be, in a book.system owner – it’s so important, it tends to be common institutional knowledge.

I keep a text file with the SID in the name, to track dba information, in the oracle user home directory. So it gets backed up just like all other user data.

(by the way, this says “click an icon to log in:” but shows no icons [ possibly if there was a previous email entered] – but hovering over where they should be shows something there. clicking on the gravatar icon to the left of the email goes to a signup page, logging in there gets a redirect problem, but logs you in anyways, but this page doesn’t know about it. firefox 14.01)