I've found that the causes of writing high-resource-usage code can be boiled down to 3 things.

1. Demanding schedule which leaves no time to check for alternative methods that may lead to less resource usage.2. The "just get if off my plate as quickly as possible" syndrome exhibited by many developers even when not under schedule pressure.3. Lack of knowledge and data to test with.

The latter reason (#3) is inexcusable considering how easily correctable it is. You're being paid to do a job and to do it well. A lack of knowledge in the area(s) that are required to actually do your job is your fault. Please, no whining about the company not paying for your education. Most people just starting out in the job market didn't have a company pay for their education. If the company you work for won't pay for education, suck it up and get your own education. A great education in SQL Server is relatively inexpensive to obtain. A cheap desktop computer, a $60 USD (including shipping) copy of the Developer's Edition of SQL Server, and a $60 USD book or two are most of what you need. If you want to officially document your education, taking an MS Certification test or two will only set you back a couple of hundred bucks. The rest is simply taking the time to study and practice your trade on your own. Additionally, reading about and solving problems on forums will give you a world of practical problems not available in any book. Studying other people's solutions to these real world problems will give you an incredible education not easily duplicated and usually not available in any book.

The justification of "having a life", especially if you have kids, is a very strong one. Just remember, when your kids get hungry, they can't eat you. You'll need to have a good job to pay for their food and maybe even their education so they won't have to work as hard as you do.

As to the first reason (#1), you managers need to learn to make better schedules. You need to stop over promising in a vain attempt to look good, learn to properly estimate jobs, and learn the lesson that every manager should learn. That is "If you want it real bad, that's the way you'll normally get it". If a bad schedule comes to you "from the top", you need to grow some hair and push back a little because delivering a bad product to the customer is much worse than missing a deadline. And to be sure, delivering a bad product to the customer is the fault of only one person… yours.

Last but certainly not least, you need to learn how to hire good people and pay them well enough to keep them. Treat them with the respect they've earned. Enable your team. Don't try to lead them. Rather, set an example. Follow those suggestions and they'll make you look good without you even trying.

As to the second reason (#2), you people need have a serious change of attitude or find a different career. The only good thing about you is that your continued actions guarantee that people who are in the business of cleaning up performance challenged code will have a lifetime of high paying work.

--Jeff Moden"RBAR is pronounced "ree-bar" and is a "Modenism" for "Row-By-Agonizing-Row".

First step towards the paradigm shift of writing Set Based code: Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column."

(play on words) "Just because you CAN do something in T-SQL, doesn't mean you SHOULDN'T."--22 Aug 2013

Being both a VB.NET developer (yes, VB.NET; coming from a C and C++ background I'd prefer C# but other team members don't) and a SQL Server DBA, spending also one or two days a week on BI solutions with the Microsoft stack, I'm kind of stuck in the middle.

The days you knew in advance what you were building are long gone now. A very efficient solution in terms of processor usage, memory usage, network bandwidth and disk access degrades slowly as more and more changes are piled upon it. It's easy to blame the developer but he didn't plan it that way, neither did the manager or anyone else. With so many software providers to choose from a company must do whatever the customer asks to avoid loosing another valueable account. Only after seeing the nearly finished application that customer will tell you what other features he can't live without. A relational database is not the most optimal tool to support these constant changes in requirements but it is a very stable environment to store data.

Developers think row-based. They cannot work with set-based algorithms. Everyone who tells you otherwise hasn't worked together with developers. LINQ is a beautiful integrated query tool on sets, but most developers still prefer those old-school For Each loops. If your lucky they understand the benefits of joins and use them appropriately. They're good at other things but they're no good at databases. Efficient algorithms are at the base of efficient code, but for most developers getting the job done without subtle flaws or hidden 'features' is already hard enough. I know a lot of performance tuning on many aspects of applications, but I'm generally called AFTER an application becomes too slow to be useful.

Most programmers like to call themselves developers. Most companies want to hire developers while paying only a programmers salary. Most errors will only surface after the solution has been deployed. The use of inefficient techniques are no exception to that rule. Good programmers are hard to find, good developers even harder. Would you be able to learn those so-called developers how to implement their solutions with those efficient techniques? If not, please stop complaining ...

vliet (5/13/2013)Developers think row-based. They cannot work with set-based algorithms. Everyone who tells you otherwise hasn't worked together with developers.

Really? That's a pretty broad brush you're using there. I don't claim to be an expert by any means, but as a SQL Server developer I think I know "set-based algorithms". But then, I guess I'm just fooling everyone...I am a developer after all.

vliet (5/13/2013)Developers think row-based. They cannot work with set-based algorithms. Everyone who tells you otherwise hasn't worked together with developers.

Really? That's a pretty broad brush you're using there. I don't claim to be an expert by any means, but as a SQL Server developer I think I know "set-based algorithms". But then, I guess I'm just fooling everyone...I am a developer after all.

I completely agree. Broad brush indeed! Vliet is hanging around the wrong devs.

When I was doing development work on a regular basis, I always kept my old computer as long as I could, as they were cycling out for the latest and greatest new PCs for the end-users.

Why? Because if I could get acceptable performance from the app/DB on my PC with half the memory and CPU of everybody else, then the performance would be fine in production.

Jeff Moden (5/12/2013)3. Lack of knowledge and data to test with.

These are really two separate things in my view. Education, training, seminars, etc. have no real correlation to test data.

And the problem with the way your scripts generate data is that while random data is great, sometimes it has to be hand entered so the data is consistent, hence small test databases. For example the latest software I've worked with is HIPPA patient data. So it is entered row, by row in a specific way by the end users. (Patient demographics, admission data, insurance info, then allergies and diagnoses, etc.) It all starts off the with the patient and has to be consistent to itself to work, such as the 50 states and valid zip codes, a limited set of diagnosis codes, a limited set of insurers, etc. So using a customer's database and using it is problematic because the data is hard to sanitize, and we can only hold unsanitized data for 180 days.

So our developers were always working with these small fake DBs and then weren't understanding why the SW would break and time out in the real world.

What I have observed living as a DBA in the worlds of Sybase ASE, SQL Server, and Oracle:

Developers, for the most part, are focused on meeting functional specifications and deadlines as a result of managers who impose rigid timelines on them and then less than optimally peforming code falls in the DBA lap to solve. The managers impose the timelines because they are scrambling to meet less than flexible demands from the business-users and are unwilling to push back, most likely due to concerns about their career path (or even their job path).

--

I see the world of IT as having become more and more complex, with expectations of faster and faster response times andor throughputs, but an unwillingness to allow those hired as subject matter expects to perform the jobs they have been asked to do. I submit That extra up-front time to allow the IT folks to think it through is more likely to result in a better end-result than not.

There seems to be a continual need for the Jerry Maguire "Help me help you" up and down the org chart as a starting point to get everyone focused on the ideas that "we are all in this togther" and therefore can accomplish more by working together than apart.

vliet (5/13/2013)Developers think row-based. They cannot work with set-based algorithms. Everyone who tells you otherwise hasn't worked together with developers.

Really? That's a pretty broad brush you're using there. I don't claim to be an expert by any means, but as a SQL Server developer I think I know "set-based algorithms". But then, I guess I'm just fooling everyone...I am a developer after all.

Would SQL query and update statements (without looping and other RBAR constructs) really even be qualified to be called "algorithms" of any sort?

vliet (5/13/2013)Developers think row-based. They cannot work with set-based algorithms. Everyone who tells you otherwise hasn't worked together with developers.

Really? That's a pretty broad brush you're using there. I don't claim to be an expert by any means, but as a SQL Server developer I think I know "set-based algorithms". But then, I guess I'm just fooling everyone...I am a developer after all.

Would SQL query and update statements (without looping and other RBAR constructs) really even be qualified to be called "algorithms" of any sort?

I was wondering the same thing, that's why I used "quotey fingers".

Google definition:

al·go·rithm /ˈalgəˌriT͟Həm/NounA process or set of rules to be followed in calculations or other problem-solving operations, esp. by a computer.

If you think of a SQL query as a "set of rules" then I think "algorithm" absolutely applies. The "process" part may be abstracted away (being performed all or in part by the engine), but you still set up the rules in your query.

I like this editorial. What I like even better is how I saw it differently to a degree than anyone else. The first word that jumped to my mind when Steve said using resources most efficiently was "spaghetti". Often in the old days the most efficient code was rarely readable and understandable. People avoided calling subs/procedures because they took up extra memory when a simple yet devastating GOTO would do the trick. I know that once I had to do a material safety process and I only had 32k of memory to work with. Wow was it ugly code but it worked and fit into the limited memory (that included storing the data sheet). I remember when we had to upgrade it when memory limits were removed, I was definitely badmouthed!

I'm thinking the most efficient code may not always be the most elegant code.