Sometimes I can become slightly annoyed by the silly way the media puts out total tosh and twaddle(*) that over-states the impact or drawbacks about technology (and science ( and especially medicine (and pretty much anything the media decides to talk about)))). Occasionally I get very vexed indeed.

My attention was drawn to some such thing about SSDs (solid State Discs) via a tweet by Gwen Shapira yesterday {I make no statement about her opinion in this in any way, I’m just thanking her for the tweet}. According to Computerworld

SSDs have a ‘bleak’ future, researchers say

So are SSDs somehow going to stop working or no longer be useful? No, absolutely not. Are SSDs not actually going to be more and more significant in computing over the next decade or so? No, they are and will continue to have a massive impact. What this is, is a case of a stupidly exaggerated title over not a lot. {I’m ignoring the fact that SSDs can’t have any sort of emotional future as they are not sentient and cannot perceive – the title should be something like “the future usefulness of SSDs looks bleak”}.

The key argument is that by 2024 we will be using something like 6.4nm dies and at that size, the physics of it all means everything becomes a little more flaky. After all, Silicon atoms are around 0.28nm wide (most atoms of things solid at room temperature are between 0.2nm and 0.5nm wide), at that size we are building structures with things only an order of magnitude or so smaller. We have all heard of quantum effects and tunneling, which means that at such scales and below odd things can happen. So error correction becomes more significant.

But taking a reality check, is this really an issue:

I look at my now 4-year-old 8GB micro-USB stick (90nm die?) and it is 2*12*30mm, including packaging. The 1 TB disc on my desk next to it is 24*98*145mm. I can get 470 of those chips in the same space as the disc, so that’s 3.8TB based on now-old technology.

Even if the NAND materials stay the same and the SSD layout stays the same and the packaging design stays the same, we can expect about 10-50 times the current density before we hit any problems

The alternative of spinning platers of metal oxides is pretty much a stagnant technology now, the seek time and per-spindle data transfer rate is hardly changing. We’ve even exceeded the interface bottleneck that was kind-of hiding the non-progress of spinning disk technology

The future of SSD technology is not bleak. There are some interesting challenges ahead, but things are certainly going to continue to improve in SSD technology between now and when I hang up my keyboard. I’m particularly interested to see how the technologists can improve write times and overall throughput to something closer to SDRAM speeds.

I’m willing to lay bets that a major change is going to be in form factor, for both processing chips and memory-based storage. We don’t need smaller dies, we need lower power consumption and a way to stack the silicon slices and package them (for processing chips we also need a way to make thousands of connections between the silicon slices too). What might also work is simply wider chips, though that scales less well. What we see as chips on a circuit board is mostly the plastic wrapper. If part of that plastic wrapper was either a porous honeycomb air could move through or a heat-conducting strip, the current technology used for SSD storage could be stacked on top of each other into blocks of storage, rather then the in-effect 2D sheets we have at present.

What could really be a cause of technical issues? The bl00dy journalists and marketing. Look at digital cameras. Do you really need 12, 16 mega-pixels in your compact point-and-shoot camera? No, you don’t, you really don’t, as the optics on the thing are probably not up to the level of clarity those megapixels can theoretically give you, the lens is almost certainly not clean any more and, most significantly, the chip is using smaller and smaller areas to collect photons (the sensor is not getting bigger with more mega-pixels you know – though the sensor size is larger in proper digital SLRs which is a large part of why they are better). This less-photons-per-pixel means less sensitivity and more artefacts. What we really need is maybe staying with 8MP and more light sensitivity. But the mega-pixel count is what is used to market the camera at you and I. As a result, most people go for the higher figures and buy something technically worse, so we are all sold something worse. No one really makes domestic-market cameras where the mega-pixel count stays enough and the rest of the camera improves.

And don’t forget. IT procurement managers are just like us idiots buying compact cameras.

(*) For any readers where UK English is not a first language, “twaddle” and “tosh” both mean statements or arguments that are silly, wrong, pointless or just asinine. oh, Asinine means talk like an ass :-) {and I mean the four-legged animal, not one’s bottom, Mr Brooks}

For many, today is the last working day before Christmas and the festive season – So I sincerely wish upon everyone a Merry Christmas.. If you don’t celebrate Christmas, well the intent of my wishes still holds – I hope everyone; whether working or not; religious leanings for, against or indifferent; has an enjoyable few days during whatever end-of-year festives you have.

I’m going to be miserably now. You might want to stop reading here and maybe go to the shops for that last spell of retail hell or some other Christmas tradition. It’s probably best if you do…

You see, despite the best wishes above, generally speaking I am not a big fan of Christmas and have not been for as long as I can remember. It is not the principle of Christmas I am not keen on {I rather like both the religious and secular aspects of the whole thing, especially the seeing-people part like Di and Bri and ringing up old friends}, it is what Business does to it. Like many people, I really object to the bombarding we endure of advertising, selling and down-right commercialist bullying for what seems to be 3 months on the run-up to Christmas. I know, I know, many people make this very same point ad nauseum around this time. What ticks me off the most is that I don’t think it would be an easy thing to change, for the fundamental reason that the businesses that are so set on telling us that Christmas will not be as good as it could be if we don’t buy their food to make us fat/get expensive presents for the kids to break/buy this bottle of smelly stuff so we get more sex/buy this booze cheap, probably for the same reason as the smelly stuff {or to help ignore the lack of sex}/take out a loan to make this Christmas REALLY “special” and you can pay it off for the whole of the rest of the year and be miserable as a result, {pause to catch breath…} as I was saying, any business that sells more stuff as a result of their advertising, no matter how much it annoys other people or adds to the degrading of the whole Christmas experience, will do better than a company that does not. And so will out-compete less tacky, crass and manipulative businesses.

That’s the huge problem with Christmas and other celebratory times. We live in a commercial society and commercial selection pressure means those companies that can squeeze the most out of a situation to sell tat will win. They give not a hoot about if we enjoy ourselves really {we are back to the smelly stuff and booze again, aren’t we?}, it’s profit. Oh, if enjoying ourselves in some way aids them in getting more profit then they won’t object, but it is not in the company mission statement of 99% of companies – and any that it is in are doing it for cynical, commercialist reasons.

So, all successful businesses are Evil and are ruining Christmas for us all {OK, so that’s a bit of a big leap, stay with me….} So, have your revenge!!!

Next year:

Don’t buy stuff people probably don’t want. No adult wants 95% of what they get so….get nothing.

Tell everyone “I have all the stuff I need, buy yourself something instead – treat yourself on me”. You can buy the stuff you really want from the savings from point 1.

Having established the principle of reciprocal meanness above, that’s all that shopping hassle ditched.

Get normal food you like {and that does not play merry hell with your digestive system}. Preferably stuff you can freeze or keep a while, so you don’t need to go into the supermarket after Dec 20th.

Turn off the TV in December {or at least record everything and skip the adverts}. There is no decent TV in December anyway, it is all being saved up for the end of the month and, heck, even that is pretty awful.

Don’t read the paper. Or if you do, if you must, first four pages and last four pages only and scribble over adverts with a felt-tip pen. You’ll get the gist of world events and if your team is winning or losing.

That company you work for, that thinks paying you a wage means it owns your soul? It’s Evil, you owe them nothing they are not getting out of you already, so have a nice break at Christmas. {Unless you work at the same place as me, then they will need you to fill in for me as I will be on holiday}.

You will now be more relaxed, less stressed, have more time and generally be a nicer person. Take people to the pub, spend more time with people who like you being around (and this will be easier due to the people who no longer like you as you did not buy them any socks or a rubbish “humorous” golf book). Do things you actually enjoy. This year it is just going to be me, my gorgeous wife and the cat over Christmas and Boxing Day. The cat is really happy about this as we both like scratching the cat’s ears.

I might invite some neighbours over. They won’t come as they have to fulfil their awful Christmas Obligations – but they will like the fact they were invited. Heck, if they do turn up I’ll be in such a fine, happy mood I will even be nice to them.

A while back I was asked by a friend to blog about being a contractor. In the pub last week my friend reminded me of this and that I had not obliged him. I will – think of this as instalment one Jason…

I’ve been a contractor on and off for 18 years. For anyone not familiar with the concept, it is where you are self-employed and you simply hire yourself out to a company for a period of time or to do a specific job. You generally have less job security than an employee and less rights and benefits – No holiday pay, no paid sick leave, no annual pay increase {OK, so that one is rare for employees too these days}, no training and generally the first out the door when the money gets tight. In return you get more money when working and a lot, lot less to do with office politics, HR, annual reviews and the like.

It is not for everyone but I like being a contractor. It gives me a broader degree of experience.

I like it apart from one main thing.

Recruitment Consultants. For every good one there are 3 bad ones. And for each bad one there are 5 absolutely terrible ones.

There are good recruitment consultants out there, some absolutely fantastic ones who do things like actually read CV’s, understand the business they are hiring into and can be bothered responding to emails and telephone calls. You might even find one who has a mental list of their clients and their requirements and will actively look to place a good candidate in front of those clients. Claire Green at GT-Consulting is one. There are others of course.

However, most do little more than scan the database of candidate CVs for keywords and send the first three found off to the client for them to do the actual work of seeing if they actually have the skills and experience required. It would seem most have no ability or interest in trying to work out who would be a good or bad candidate themselves, like it being the service they are supposed to supply. If you try and get in touch directly to discuss a role, to maybe ask some questions to save both you and the consultant’s client a wasted interview, many will not take your call {“Can I ask who’s calling?” Brief pause whilst they realise you are a candidate not a client company “Ahh, sorry they are out of the office today, they’ll call you back. Who were you again?”}. Only the good ones call you back. You will hardly ever be called back.

If you do speak to them, some will be your best mate – but can’t quite fake sincerity… Sadly, it is often obvious that they have no idea about the business. I had a chap a week or two back telling me I needed PL or SQL to do the role and when I queried if they meant PL/SQL they got tetchy with me. Another a while back was insisting I was not suitable as I did not have 10 years of Oracle 10. As I beta tested Oracle 10 for over a year and thus, with around 8 years’ experience at that time, was well ahead of the pack I suggested that maybe they needed to alter that requirement – or find someone who helped develop it at Oracle Corp…Again, some kindly advice was poorly received. OK, I was not kindly, I was tetchy too. He had stared off being my insincere best mate.

I could just be having a self-centred moan of course, in that the recruitment consultants don’t realise how great I am ( :-) ) and find me lucrative jobs – but I’ve also been the client and had to wade through dozens of utterly unsuitable CVs sent in from them. The last time was particularly awful as we were not able to offer a great wage (but we were happy to take people with experience of prior versions and train them up to the latest-greatest). Most CVs sent in had the words Oracle, database and administration on them but not together. Several lacked any Oracle at all. Every recruitment consultant I dealt with that time gave me the same spiel about having the best candidates on their books, how they vetted everyone and sent only the ones with the best match of skills. They must have been telling a miss-truth about at least one of those claims as there was little match with our requirements for an Oracle DBA.

So, I really like contracting but not the dealing-with-agents bit. Oddly enough, any discussion with other contractors or managers who hire nearly always shows that my feelings are widely shared…

I’ve been thinking about doing this post ever since I started blogging but I didn’t – because many jobs are only available via recruitment consultants. Insulting them is not going to help me get put forward for jobs. However, last time I was mouthing off about Satan’s little Imps in the pub and how I had never done a Friday Philosophy on the topic, due to the fear of the consequences, one of the guys pointed out I was an idiot. Most recruitment consultants can’t even be bothered reading your CV so they are not going to go check out someone’s technical blog! {and Neil has just beaten me to posting about it and how they always ask for mostly irrelevant industry experience}. Any who do are going to be firmly in that rare Good category. I’d go as far as to say that any recruitment consultant who is reading this is in the top 5% of their field. Nice to talk to you again, Claire…

How many people under the age of {Martin checks his age and takes a decade or so off} ohh, mid 30’s does any database design these days? You know, asks the business community what they want the system to do, how the information flows through their business, what information they need to report on. And then construct a logical model of that information? Judging by some of the comments I’ve had on my blog in the last couple of years and also the meandering diatribes of bitter, vitriolic complaints uttered by fellow old(er) hacks in the pub in the evening, it seems to be coming a very uncommon practice – and thus a rare and possibly dying skill.

{update – this topic has obviously been eating at my soul for many years. Andrew Clark and I had a discussion about it in 2008 and he posted a really good article on it and many, many good comments followed}

Everything seems to have turned into “Ready, Fire, Aim”. Ie, you get the guys doing the work in a room, develop some rough idea of what you want to develop (like, look at the system you are replacing), start knocking together the application and then {on more enlightened projects} ask the users what they think. The key points are the that development kicks off before you really know what you need to produce, there is no clear idea of how the stored data will be structured and you steer the ongoing development towards the final, undefined, target. I keep coming across applications where the screen layouts for the end users seem to almost be the design document and then someone comes up with the database – as the database is just this bucket to chuck the data into and scrape it out of again.

The functionality is the important thing, “we can get ‘someone’ to make the database run faster if and when we have a problem”.

Maybe I should not complain as sometimes I am that ‘someone’ making the database run faster. But I am complaining – I’m mad as hell and I ain’t gonna take it anymore! Oh, OK, in reality I’m mildly peeved and I’m going to let off steam about it. But it’s just wrong, it’s wasting people’s time and it results in poorer systems.

Now, if you have to develop a simple system with a couple of screens and a handful of reports, it might be a waste of time doing formal design. You and Dave can whack it together in a week or two, Chi will make the screens nice, it will be used by a handful of happy people and the job is done. It’s like building a wall around a flower bed. Go to the local builders merchants, get a pallet of bricks, some cement and sand (Ready), dig a bit of a trench where you want to start(Aim) and put the wall up, extending it as you see fit (Fire). This approach won’t work when you decide to build an office block and only a fool from the school of stupid would attempt it that way.

You see, as far as I am concerned, most IT systems are all about managing data. Think about it. You want to get your initial information (like the products you sell), present it to the users (those customers), get the new (orders) data, pass it to the next business process (warehouse team) and then mine the data for extra knowledge (sales patterns). It’s a hospital system? You want information about the patients, the staff, the beds and departments, tests that need doing, results, diagnoses, 15,000 reports for the regulators… It’s all moving data. Yes, a well design front end is important (sometimes very important) but the data is everything. If the database can’t represent the data you need, you are going to have to patch an alteration in. If you can’t get the data in quick enough or out quick enough, your screens and reports are not going to be any use. If you can’t link the data together as needed you may well not be able to DO your reports and screens. If the data is wrong (loses integrity) you will make mistakes. Faster CPUS are not going to help either, data at some point has to flow onto and off disks. Those slow spinning chunks of rust. CPUS have got faster and faster, rust-busting has not. So data flow is even more important than it was.

Also, once you have built your application on top of an inadequate database design, you not only have to redesign it, you have to:

do some quick, hacky fixes to get by for now

migrate the existing data

transform some of it (do some data duplication or splitting maybe)

alter the application to cope

schedule all of the above to be done together

tie it in with the ongoing development of the system as hey, if you are not going to take time to design you are not going to take time to assess things before promising phase 2.

I’m utterly convinced, and experience backs this up, that when you take X weeks up front doing the database design, you save 5*X weeks later on in trying to rework the system, applying emergency hacks and having meetings about what went wrong. I know this is an idea out of the 80’s guys, but database design worked.

*sigh* I’m off to the pub for a pint and to reminisce about the good-old-days.

A while ago whilst working on one project, a colleague came back to his desk next to mine and exclaimed “I hate working with that team! – they are so bad that it makes everyone who works with them look incompetent!”

Now there is often an argument to be made that working with people who are not good at their job can be great for you, as you always looks good in comparison {it’s like the old adage about hanging around with someone less attractive than you – but I’ve never found anyone I can do that with…}. It is to an extent true of course, and though it can seem a negative attitude, it is also an opportunity to teach these people and help them improve, so everyone potentially is a winner. I actually enjoy working with people who are clueless, so long as they will accept the clues. You leave them in a better state than when you joined them.

However, my friend was in the situation where the team he was dealing with was so lacking in the skills required that if you provided them with code that worked as specified, which passed back the values stated in the correct format derived from the database with the right logic… their application code would still fall over with exceptions – because it was written to a very, very “strict” interpretation of the spec.

In one example, the specification for a module included a “screen shot” showing 3 detail items being displayed for the parent object. So the application team had written code to accept only up to 3 detail items. Any more and it would crash. Not error, crash. The other part of the application, which the same people in the application team had also written, would let you create as many detail items for the parent as you liked. The data model stated there could be many more than 3 detail items. I suppose you could argue that the specification for the module failed to state “allow more than three items” – but there was a gap in the screen to allow more data, there was the data model and there was the wider concept of the application. In a second example, the same PL/SQL package was used to populate a screen in several modes. Depending on the mode, certain fields were populated or not. The application however would fail if the variables for these unused fields were null. Or it would fail if they were populated. The decision for each one depended on the day that bit of the module had been written, it would seem. *sigh*

The situation was made worse by the team manager being a skilled political animal, who would always try to shift any blame to any and all other teams as his first reaction. In the above examples he tried to immediately lay the blame with my colleague and then with the specification, but my colleague had managed to interpret the spec fine (he did the outrageous thing of asking questions if he was not sure or checked the data model). Further, this manager did not seem to like his people asking us questions, as he felt it would make it look like they did not know what they were doing. Oddly enough they did NOT know what they were doing. Anyway, as a consequence of the manager’s hostile attitude, the opportunity to actually teach the poor staff was strictly limited.

That was really the root of the problem, the manager. It was not the fault of the team members that they could not do the job – they had not had proper training, were unpracticed with the skills, siloed into their team, not encouraged to think beyond the single task in front of them and there was no one available to show them any better. The issue was that they were being made to do work they were not able to do. The problem, to my mind, was with the manager and with the culture of that part of the organisation that did not deal with that manager. He obviously did not believe that rule one of a good manager is to look after the best interests of your team. It was to protect his own backside.

But the bottom line was that this team was so bad that anything they were involved in was a disaster and no one wants to be part of a disaster. If you worked with them, you were part of the disaster. So we took the pragmatic approach. When they had the spec wrong, if we would alter our code to cope, we would alter our code. And document that. It gave us a lot of work and we ended up having a lot of “bugs” allocated to our team. But it got the app out almost on time. On-going maintencance could be a bit of an issue but we did what we could on our side to spell out the odditites.

I still know my friend from above and he still can’t talk about it in the pub without getting really quite agitated :-)

A couple of weeks ago I extolled the virtues of someone I felt was a great person to work with. This week I’m going to do the opposite (and it will be interesting to see which posting gets more hits).

The worst person I have worked with in IT is Mick. I’ve only known a couple of Micks {and if you are one of them, but you don’t know Barry, you are not the Mick}. In an ironic twist of fate I met Mick at the same time I met the best person I have worked with, Barry. We were all in the same team you see, a UNIX sys admin team I got parachuted into. Maybe the vast difference between the two help make them so distinct in my mind.

Mick was very knowledgable and technically very capable. No, that is not fair, he was extremely good. He actually knew all this system admin stuff and several variations of shell programming, perl, C and a few other two-steps-from-assembler type languages. And he was an absolute and utter pain in the behind.

Barry and I did not know much (or in some cases, any) of this sys admin stuff. If we needed to do something and did not know how, Mick was supposed to show us. It worked something like this:

“Mick, I need to copy all the files that were changed last week from this directory on box X to box Y, keeping the directory structure – Can you help?”. Mick would not hear. He suffered from “intermittent deafness” – though he never missed any announcements about free food. You had to go and stand by Mick and wait for him to deem to notice you. If you actually interrupted him he would swear at you and utterly refuse to help, you had to wait quietly. If it was a good day he would deem this acceptable after a minute or two, but he would do his utmost to convey the impression he despised your lack of knowledge and your concerns were beneath his talents… but he would stoop to help.

You would repeat the task you were trying to do and, pausing only briefly to pour scorn on such a trivial thing, he would turn his back and start typing. He’d write a script to do it. “no, no, don’t write it, just tell me the basic commands and I’ll work it out!” No, he insisted on writing the script.

The script would be a thing to behold. Mick would write it in as few lines as possible and the least number of letters. For ages. Oh, he would have a working version in about the time it took Barry or I to explain the task, but he would not give you that version, oh no. He would ignore you until he had made all variables 1 character, took out all whitespace, replaced anything obvious with something obtuse, replaced a small chain of simple commands with one or two arcane commands. Every script was an attempt to win an “obfuscated code” competition. If we waited for the end result, it was impossible for Barry or I to decipher. The only benefit to the process was you would see the commands he was using and you could wander off and start with the unix Manuals yourself and get the job done.

He had other methods with which to demonstrate his greater worth.
Mick would agree to help (under duress of the boss telling him to do so) with an urgent task, but keep asking you to wait all day – then go home without doing his bit.
He seemed to love to intercept anyone coming to you for help, tell them he would sort out the problem for them – only to not. And then tell the user the next day that it was Barry or My problem to sort out. Correct, Mick would not have mentioned this to us.

Mick was fair though, he would treat everyone the same. With scorn. Any expertise in a field he did not know was unimportant and anyone with skills in his field was just competition to be shown who was best. Sadly, he usually was best, if best means biggest smartass.

Over time, as Barry and I learnt stuff (almost never from him), Mick became redundant. Not because we caught him up, not by a long way, but because no one else in the department would ask him anything. They would come to Barry and I. We might be slow and we sometimes screwed it up but we did not sneer and we fixed the problem in a way they could understand.

The reason Mick is the worst person I ever worked with is, unlike people who simply break stuff or lie about their skills or are stupid, he was actually very talented and capable – and yet took a perverse pleasure in not doing so. Mick would put effort into the art of maximizing his unhelpfulness. It was the difference between his potential to help and his drive to not do so that made it so hard for me to deal with him. I’d rather work with a talentless, idiot liar because at least you don’t need or expect much from them.

Do you remember the last time you signed a contract for a job? Did you read all the terms, conditions and clauses? How angry did it make you? If you did not read it, dig it out and do so. It will ruin your whole day.

I do a mixture of contracting and consulting to provide bread on the table and catfood in the cat bowl and I get to sign a lot of contracts. And they send me mad as so many of them have such outrageously immoral, unfair and, I strongly suspect, illegal clauses in them. But if you don’t sign, you don’t get a job.

If the contract says they can get rid of me on a week’s notice, but I have to give them a month, I insist they pick one or the other and it applies to both parties. If there is a clause saying everything I think of belongs to them then I say no – if it is based on their intelectual property or code specific to their application, then it is theirs and I will comply utterly, but if it is the sort of generic data dictionary query that all these client rely on me to use to do my job, it is mine and I want the right to use it {and give it to other people, like I gave it to you, Mr Client}. Another clause that seems to be becoming rampant in the UK contracting arena is the 40-hour working week and signing away any right to complain. I absolutley object to that as it has been proven scientifically that continuous long hours are detrimental to health. If I choose to do 40, 50 hours in a week (and I often do) it is my choice but they damned well are not going to insist on it. I also know if I do the 50 hours for too long, my productivity and quality drops – and I think we all know this is the real case.

There is often a discussion with the actual people you work with, how the contract is just “admin” and they would never treat you in the way it says they can and “just sign it and forget it as we know you will do the job and we will never use clause 17.3.2 on you”. And they probably won’t, but it makes the contract a big, fat lie at best and a potential stick to beat you with at worst.

A few years ago I decided that I had had enough of this and I now challenge the worst of these clauses and I have had some succes. I also challenge them because, just once or twice, I have had someone try and take advantage of me due to these clauses. Usually recruitment agencies, I have to say.

With small organisations I usually can agree fair and equitable terms. With larger organisations it is a fight but I can usually get some sense into the agreement. But with international corporations, it is a blank refusal. They do not need me, they can buy in someone else and they damn well ain’t going to negotiate or treat you as an equal.

I’m facing this one right now. I’m looking at the contract and the blank refusal by the faceless (and probably deeply annoyed {and overworked}) minion in Admin to even consider a single letter change to a contract. And I am thinking “well sod you and your job and your immoral and bullying contract then”. This morning I really considered walking off site and sacrificing any chance of payment to “punish” such unbending unfairness.

But I probably won’t, I’ll probably roll over and sign the abusive, vile document because I have already been on-site for a week and I like the people I work with, I like the job and I want their project to succeed. And the potential unfair aspects of the contract will probably never be a real issue. So why can’t they just be fairer and why does it make me so absolutely incandescant with rage?

I’ve got this USB memory stick which I use to carry around my scripts, documents, presentations, Oracle manuals and enough music to keep me going for a few days. It is on an 8GB Gizzmo Junior and it is tiny. By tiny I mean as wide as my little finger, the length of a matchstick and about the same thickness of said matchstick. So small that I did indeed lose the damn thing for 6 months before I realised it had got trapped behind a credit card in my wallet.

It cost me ten British pounds about 15 months ago (less than most 4GB USB sticks seem to cost now, but then it is nothing more than the memory chip and connectors wrapped in plastic) and it highlights how cheap solid-state “storage” is becoming.

Connected to this, I was looking at buying a new PC this week and this machine comes with 10 USB slots, if you include the ones on the supplied monitor and stubs on the motherboard.
10 USB slots, 8GB gizzmo memory sticks… That would be 80GB of cheap and fast storage. Now get a few USB hubs and bulk-buy a few dozen cheap USB2 sticks and you could soon have a solid-state database of a few hundred GB for a thousand pounds. Then of course you can have fun seeing where the pinch-points in the system are (USB2 has a maximum speed per port and going USB3 right now is going to break that 1 grand barrier. But give it a year…).

This really started me thinking about when memory-based storage would take over from spinning disk as the best option for enterprise-level storage and my gut feeling is in about 5 years. I think it will be both technically possible and financially viable in much less than that, say as little as 2 years, but the cost of solid-state storage per MB will still be higher than disk by then but potentially much faster. A few considerations going through my mind were:-

Disk is getting a lot slower in relation to acreage. By this I mean that, for a single disc drive, capacity is doubling about every 18 months but seek time has hardly reduced in a decade and transfer rate (reading from the physical platters to the units buffer) is again almost stationary, at about 120MB/s for 10,000rpm disk and up towards 180 for those very expensive and noisy 15,000 rpm disks. Being a tad ridiculous to make the point, with modern 3TB disks you could build most Oracle database on one disc. Let’s make it two in a raid 10 configuration for redundancy. My point is, your 3TB database could well be being run right now, for real, across say 5 physical disks with a total sustainable physical throughput of around 500MB a second.

Solid state storage seems to be halving in price in more like 8-10 months.

IO subsystems are made faster by using RAID so that several physical discs can contribute to get towards the 300MB or so speed of the interface – but solid state is already that fast.

IO subsystems are made faster by building big caches into them and pre-fetching data that “might” be requested next. Oh, that is kind of solid state storage already.

Solid state storage, at least the cheap stuff in your USB stick, has the problem that you can only write to each bit a thousand or so times before it starts to get unreliable. But physical disk has exactly the same issue.

There are new methods of solid-state memory storage coming along – “New Scientist” had a nice article on it a few months ago, and these versions will be even higher density and more long-term reliable.

Seek time on solid-state memory is virtually zero, so random IO is going to be particularly fast compared to spinning disk.

Solid state memory needs less power, and thus less cooling, is silent, is potentially denser and is less vulnerable to temperature and humidity fluctuations. I can see it not needing to be kept in a specialist server room with the need for all that air con and ear defenders when you go in the room.
Just somewhere with normal air con and a lock on the door should suffice.
We do not need Solid State storage to match the size of current disks or even be as cheap to take over. As I have already pointed out, it is not acreage you need with physical disks but enough spindles and caches to make it fast enough in relation to the space. Further, we can afford to pay more for solid state if we do not need to keep it in such expensive clean-room like environments.

I can see that in a couple of years for a given computer system, say a mixed-workload order processing system, to support the storage needs we will have maybe a dozen solid-state chunks of storage, perhaps themselves consisting of several small units of memory in some sort of raid for resilience, all able to flood the IO channels into our processing server and the issue will be getting the network and io channels into the server to go fast enough. So don’t, stick all the storage directly into the server. You just got rid of half your SAN considerations.

I’m going to stop there. Partly because I have run out of time and partly because, in checking out what I am writing, I’ve just spotted someone did a better job of this before me. Over to James Morle who did a fantastic post on this very topic back in May. Stupid me for not checking out his blog more often. Jame also mentions that often it is not total throughput you are interested in at all but IOPS. That zero latency of solid-state memory is going to be great for supporting very high IOPS.

Yesterday I posted about the potential for a Oracle in Science community within the UK Oracle user group {and wider for that matter, there is after all a world Oracle Life Science community but it is currently less vibrant than it was, sadly}.

My friend and occasional drinking partner Peter Scott replied to say he felt there was “a place for a SIG for stonking great databases” {now wouldn’t SGDB be a better TLA than VLDB? :-) }.

Well, I would agree but for one small detail. An apparent lack of anyone willing to be part of the community.

When I was building a very considerable VLDB {and I’m sorry I keep going on about it, I’ll try and stop soon} back in the early to mid 2000’s I seemed to be working in a vacuum of information, let alone prior experience. Yes, there was stuff in the Oracle manuals about how big things could theoretically be made and some vague advice on some aspects of it, but an absolute lack of any visible Oracle customers with anything even approaching the sizes I was contemplating. 2TB was about the limit and I was already way beyond that. Was this because I really was pushing the boundaries of database size? Well, I have since found out that whilst I was up there just behind the leading edge, there were several databases much, much bigger than mine and others already envisioned that might hit the Petabyte level, let alone Terabyte.

The thing is, no one would speak about them. At all.

We were left to do it all pretty much from scratch and it would not have been possible if I had not spent years building up with VLDBS as the definition of a VLDB size increased, plus of course cracking support by the other DBAs and Systems Admins around me. And to be fair, Oracle Corp helped us a lot with our efforts to build these massive databases. Interestingly, one Oracle Consultant would regularly tell me that our systems really were not so unusually big and there were plenty larger. He usually said this when I asked, exasperatedly as something else failed to scale, if Oracle had every tested things at this level :-). But despite constantly asking to meet with these people with massive systems, so we could exchange war stories and share advice, and being promised such contacts by Oracle, they never materialized except for CERN – who we already talked to as a fellow scientific organisation – and Amazon, who it turns out did things in a very different way to us {but it was really good to talk to them and find out how they did do their big databases, thanks guys}. Both were at the same scale or just behind where we were.

This is because most of the people with massive oracle databases will not talk about them as they are either run by the largest financial organisations, are to do with defense or in some other way just not talked about. In his comment Peter refers to a prior client with an OLTP-type system that is now around the PB scale. I would be pretty sure Peter can’t say who the client is or any details about how the system was designed.

So although I think there is a real need for a “stonking great databases” forum, I think there is a real problem in getting a user community of such people/organisations together. And if you did, none of the members would be allowed to say much about how they achieved it, so all you could do would be sit around and brag about who has the biggest. There is an Oracle community about such things, called the Terabyte Club, but last I knew it was invite-only and when I managed to get invited, it turned out that mine was biggest by a considerable margin, so I was still not meeting these elusive groups with 500TB databases. Maybe there is an Oracle-supported über database society but as I never signed the official secrets act might not have been eligible to play.

If I am wrong and anyone does form such a user group (or is in one!) I would love to be a member and I would strive to present and help.

I’ll finish with what appears to be a contradiction to what I have just written. There already is a UKOUG User Group that deals with large systems and I chair it – the Management and Infrastructure SIG. {sorry, the info on the web page could do with some updating}. Part of what we cover is VLDBs. But we also cover Very Many DataBases (companies with thousands of instances) and Very Complex DataBases plus how you go about the technical and management aspects of working in a massive IT Infrastructure. It might be that we could dedicate a meeting to VLDBs and see how it goes, but I know that whilst many who come along are dealing with database of a few TB, no one is dealing with hundreds of TB or PB database. Either that or they are keeping quiet about it, which takes us back to my main point. The MI SIG is probably the closest to a VLDB SIG we have in Europe though, and is a great bunch of people, so if you have a VLDB and want to meet some fellow sufferers, we have our next meeting on 23rd September in the Oracle City office.

Many years ago I had a good friend who was a psychiatric nurse. We were talking about his job once and he was saying how some patients just took up much more time than others. These were generally the ones who would be deemed “the most mad” in a non-clinical manner {and is pretty much how my friend the psychiatric nurse put it}. These patient’s actions or need for intervention would put demands on the staff far more than other patients. As a result, all the nurses tended to get to know (or know of) such patients better than others.

I thought of this the other day when a few of us were talking about some awful bit of application we were concerned about. This thing inserts rows from a MSSQL database into a table in an Oracle database. Triggers on the initial table fire and populate another table, in a 1-to-many relationship. This second table also has a trigger on it that further inserts into another set of tables. A regular process then aggregates this data – and sticks it back into the MSSQL database it came from.

Said process is done as a single transaction for all rows inserted for the day. Irrespective of the growth in rows. Or the fact that one source “application” has grown to 10 and soon will grow to 50. All rows in one transaction. The intermediate tables are never cleared out and get bigger and bigger. No one else needs any of this data in the Oracle side of the system.

There are several “madnesses” to this process – why put it into Oracle only to put it back in the source system, why use a busy production system to hold and process transient data, why no clean up, why are the records not processed in batches, the cascading triggers magnifies transaction data volumes…

This process is well-known in our group. I’ve been involved. Both the guys I was talking to have been involved. I can see from my desk 4 or 5 other people who have been roped into bullying this process thought before now. In fact, I reckon half the department have had to work on this damned thing at some point.

Can you see why I was reminded of my conversation with my old nurse friend?

The application is simply mad. And as a result it is demanding not only on our database but on all of the team, as so many of us have had to get involved working on it. We have all got to know it so well.

I’m glad to say that treatment for the application is planned and, hopefully, it will soon be a lot happier.