I think all of the answers have merit. I have 30+ years software engineering experience and my current position is Senior Systems Engineer for a tier one mobile phone company. Debugging is hard - that is what my team does more than anything else, and it is mostly a matter of mind-set and pattern recognition capability. You need to be able to look at a mass of code, and recognize where stuff "looks" wrong. Also, a foundation in formal Boolean logic helps a whole lot! :-) In many of my former positions, I would be handed the task of debugging problems that other engineers had worked for weeks on trying to detect. Sometimes I would spend an entire weekend in the debugger and poring over code before the non-obvious became clear.

While the answers above have merit, I think they all look at "where am I know?" instead of "how can I be the best I can be?"

I would suggest that you start learning how to apply tools such as TDD and BDD to your development process. In TDD, for example, you write unit tests for your code functionality BEFORE you write the code itself: your code fails unit tests at first and then passes when you're done.

This isn't easy, especially if you're working in C/C++ on existing codebase which doesn't have comprehensive unit tests already. If you can do it, however, you can become a much better developer, with fewer bugs and more reliable and robust code.

That being said, everyone has to look at where they are in their career and think about how they can improve themselves and, sometimes, identify what changes in direction are required. I'm a KICK ASS DevOps engineer, able to script, deploy, and resolve issues quickly and with great success, across multiple operating systems and programming languages. But if you asked me to write a new Java framework to handle a new NOSQL backend, well, I'm simply not the right guy for that job (I could do it, sure, but that's not my strength).

If you follow the link to SE, the OP was eventually dismissed from the company after being placed in a "employee development program"...I guess my suggestion would have been to use those three months to actively seek out a new position, rather than end up unemployed at the end. That's one of those things that's definitely easier said than done, though...

1- ask your boss for specific metrics2- discuss with colleagues3- is your company treating its workers well in general ?4- have you had training/mentoring in bug finding ?

You seem to work hard, and take criticism harder. Your boss might be playing on that to make her budget.

I cannot rate this comment highly enough.

I'd treat all these points as Boolean Logic. Can you get specifics from the boss? Can you discuss it comfortably with you colleagues? and so on. If the answers are all TRUE, then it's you. However if FALSE turns up one or more times you have all the ammunition you need to challenge your boss.

However, I've also seen that some people seem to have a knack for multithreaded code than others. You shouldn't be discouraged, and this does not mean you can't have a happy successful life. For example, I can't draw. My sketches look like a 10 year old's. Yet I can play music and code and sell. Good enough for me.

The most important factor in debugging is to know the code and side effects of the system in a detailed, minute fashion. All the function calls, all the memory allocation calls, every lock. For primitives (like insert into data structure), what are the concurrency rules of the libraries you're working in?

I find that reading the code base, with notes, over and over is the first step. Writing down flow and code trees. For sections that aren't clear, put prints in and see what happens. This might easily take a week or two for a large project. You might have to bug original authors, or at least someone who seems to be working quickly and well in the code base, for a tutorial. They can often clear up interesting misconceptions - poorly named functions they they had always considered refactoring. This might take very long hours, too. Longer time blocks learning - and less time between - often helps me retain.

I find most people who do well at this can retain about 50,000 lines of code, with a big IF based on complexity and language. After this point, you should be able to lead a code review on the code in question. You'll probably start seeing bugs and timing holes.

If the code is too big to keep in your head, I've found it's important to keep detailed notes on the functions involved. All of them.

If, at this point, there's still struggling, then I would suggest rounds of pair debugging. You'll pick up tricks from your pair. I've often done this with others as the stronger member - I'm a better debugger than "green field" developer - and I've found in a month that with someone willing to learn, they will improve dramatically. I've also seen people who want some kind of magic bullet. Debugging is hard, and you have to sweat it through.

I notice when smart people are slow at finding bugs at work that it's usually their tools.

Things I've seen these people do: - Not use gdb when a simple breakpoint would tell you the cause - instead, they add new debug output, recompile, recreate the problem... repeat. - Not use logging to a text/csv file where you need the output of repeated runs (doing it in gdb will kill you here). - Not use cscope or other code analysis tool, and don't have it integrated in the editor. It's excruciatingly painful to watch vi people (it's almost always vi users for this one) hunt for something in code... :q out... grep for something else... :q... grep... repeat. What you could have done in 10 seconds in cscope takes 5 minutes. - Not know the system tools like lsof or netstat (or the whole /proc tree) that could tell them exactly what they need to know.

I show them how to use the the tools and productivity remarkably improves. Perhaps it's partially your company's education process. Ours certainly doesn't do a good job of it - showing them how to do this is not my job, though of course I do.

On the other hand, some people are just super-methodical and anal - these are great when you need someone to track down a very obscure bug, but not so great when you need to be knocking them down fast. Use them appropriately and give them the time to do it.

And sadly some people just don't have the debugging mindset and never will.

Take a lesson from my days in white-water rafting: When you get sucked underwater by a circulating current on the back-side of a class-4 rapid, your life-jacket will not buoy you back to the surface. The best strategy (though non-intuitive) is to find the bottom of the river, and walk out of the riptide.

Swimming down can sometimes get you out of a hole, but the general best way to escape a hole is to change your shape. Stick your arms and legs out in different directions trying to find current that is going downstream.

I am a retired Software Engineer. I will tell you from experience that this type of situation only gets worse--more oppressive and more demoralizing. It never has a happy ending.

Whether they are trying to make you quit because of some hidden corporate agenda or it's just that your boss is a jerk or you really are out of your element is not immediately relevant. You've got to leave and it's always easier to find a job when you have a job.

Wait, there are people for whom multithreading bugs are fast and easy to fix?!

Warning - Anecdotal: Most hard threading bugs are caused by bad design, which cannot be fixed without starting over. Multi-threading needs a good design phase.

If only one part of code is responsible for something, then when that something doesn't work, you only need to look in one area. If you design it in a modular enough way, then there are only so many cases.

Most of my bugs are fixed on a whiteboard or by just thinking about the characteristics of the bug.

The problem with boss' is the idiots usually compare everyone else to their best employee.

I'm reminded of a quote from an Edwards Demming book. He was talking with a manager at a factory, and the manager said "Half of our agents are below average!" Demming, a statistics guy, got a chuckle out of that, but the manager didn't seem to understand what was so stupid about his statment. Of COURSE half your agents will be below average, that's what "average" is.

Programming shops operate like any other shop. There will usually be a star performer, a few middle folks, and maybe a few lower performers. If the boss is an idiot they will compare everyone to the star. If they do that, however, everyone else notices very fast and you end up with one of two BAD situations..

1) the star makes everyone look bad, which demoralizes the team in general, possible impacting performance ever more. (This is even worse if the star has a fat head and rubs their performance in everyone's nose and runs them down with harassing comments).

2) everyone gangs up on the star to verbally "soap sock" them (ala military style), telling them to slow down or under perform in order to lower the bar that everyone has to hit. Result ... you're not getting the most out of your star.

Good bosses realize they lucked out with the star performer. Maybe they'd like to let the lower performer(s) go and see if they can roll the dice again to get another star performer.

BUT ... you've spent TWO YEARS on a job that involves a) Linux, b) Embedded, c) multi-threading. Throwing you away for another employee would be throwing away 2 years of trench work you've done getting to know the project, the code, etc, etc. I think they would be better off realizing what your really good at and specializing you in it.

There are several ways programmer shops are setup ... which imitates how companies are setup.

The two main ones are...

1) each department is specialized ... you have the coders, you have the bug hunters / QA, etc, etc.2) each department is required to do its own extra-managerial bullshit ...IE: the coder is expected to also QA and bug fix their own code.

Sounds like your shop is setup as #2, where each programmer is expected to own their own code and validate, check, bug-fix, etc, etc, it.

But, just b/c you're having a hard time bug fixing doesn't mean you suck in general. Maybe they should specialize you in cranking out some code. Then have others follow-up, suggest how to optimize/tweak it/refactor it, then others bug check it and fix the bugs.

It just seems like such a waste to toss someone away that has 2 years experience on the job. The time it would take to ramp a new employee up on the code would be long I would think. If they're having such a hard time with you, they could at least let you finish a project, then say "hey, that project is wrapped up, we're going to let you go", and then get someone in before a new project starts up.

These Stack Exchange cross posts need integrated posting imo. But whatever, the guy got fired already, so now we give him belated advice on a forum he probably never posted on or sees.

For me its about the magic 10,000 hours or so it takes to become an expert at something. 2 years fully covers that, so if you are wondering why you still suck at the 2 year mark you need to look in the mirror, not at the people around you.

One other angle is the research on top programmers vs dum dums. There are many factors, but for instance where a top programmer will take a break and accomplish some easy tasks then get back to a hard problem, dum dums will bash their heads into the hard problem and end up accomplishing little, even though they spend roughly similar time on the hard task. Top programmers can be as much as 100 times more productive than bottom programmers. Simple research and reading can get you out of dum dum league and into the middle batch.

Does anyone know when this StackExchange cross-promotion ends? I like both ARS and SE, but I read them for different reasons and this stuff always feels kind of crass. Engagement and brand building are well and good, but this isn't a great fit for ARS' evolution towards bigger picture content. It's like running across "how to repair a carburetor, from howstuffworks.com" on Autoblog.

These Stack Exchange cross posts need integrated posting imo. But whatever, the guy got fired already, so now we give him belated advice on a forum he probably never posted on or sees.

Stack Exchange is not here to help one person who has a problem. It's about helping all the other people in future who run into the same issue.

If a question is only relevant to the person asking it, then it's usually deleted before anyone can answer.

Ars is doing a great job of highlighting interesting questions that many people will run into, and I'm sure there are plenty of people reading this thread who are in a similar situation to the guy who originally asked the question.

Wait, there are people for whom multithreading bugs are fast and easy to fix?!

Warning - Anecdotal: Most hard threading bugs are caused by bad design, which cannot be fixed without starting over. Multi-threading needs a good design phase.

I can agree with this... One time I was tasked with designing some new prototypes, and I had to use the base infrastructure SDK from another team to do it. I talked to other people that used it, and I determined it was a crappy design, because it was built around an idea of being a service-bus, which made it damned near impossible to build anything spec-compliant with their SDK, never mind the endless random crashes and hiccups during certification runs. Trying to fix their implementation was going to be a huuuuuge PITA.

I tossed the entire SDK code-base, and designed my own from scratch, taking feedback from our clients with regards to their requirements... Many (mostly the owners of said code I threw away) thought I wasn't being a team player because I threw away all their code/effort, but at the end of the day, it was the right decision, as we got much better market penetration with the new SDK vs the old, by orders of magnitude. Plus the new code was certified compliant, whereas the old code kept hopelessly failing certification tests.

But I don't know, maybe that's a quality some company's are looking for? Someone that doesn't just blindly follow directions, and chase their tail down a rathole, but stands up to management to convince them to cut their losses and re-architect a better solution that will save more time/money in the long run...

However, I've also seen that some people seem to have a knack for multithreaded code than others.

When you finally get a "knack" for multithreaded code, you'll wind up realizing that it's just horribly broken.

Before threading, we had time sharing, and processes had a few well defined ways of interacting with each other. (Streams, network sockets, shared memory, etc.)

Threading evolved from time sharing and multiprocessing systems, but without keeping the well defined interfaces. It's not a feature that threads can stomp on each other at any point, and you have to worry about things like cache coherency. It's not a feature that threading relies on locks that don't compose, deadlock under trivial circumstances, and typically destroy the performance that you were trying to achieve with threading.

All of the tricks to make multithreaded code work basically boil down to "don't use threads or locks." For example, say I've got a subroutine that's parsing some files and extracting records. And I have another subroutine that wants to accept those records and distribute them to a database of some sort. Perfect job for threads, right, just set up a queue and let them go?

No. For starters, you have to figure out how to get both threads to stop. Okay, we'll bundle a control signal; hopefully "null" isn't taken for something else. Now we need to figure out what to do if one thread stops while the other doesn't. Timeouts cause it to break if a job is large. Maybe your calling thread can listen to another queue for exceptions...

But a coroutine dispenses with all of that complexity, and your language's normal mechanisms take care of error conditions.

Quote:

The most important factor in debugging is to know the code and side effects of the system in a detailed, minute fashion. All the function calls, all the memory allocation calls, every lock. For primitives (like insert into data structure), what are the concurrency rules of the libraries you're working in?

That's a lot of work to make fixing bugs "fast". And I don't think you're wrong; even trivial multithreaded operations require that level of detailed knowledge. For instance, if I want to stop a thread, how do I do that? In Java, you call Thread.interrupt(). (You're welcome to read why they ditched Thread.stop().)

If a thread is waiting on a monitor, it will throw an InterruptedException. If not, it sets the interrupted flag, you'll eventually check it and exit. Unless someone swallowed the interrupt flag, so you have to hope that every method called does the Right Thing.

If the thread is waiting on, say, a network socket, you'll need to close the socket. Assuming you can find the socket, please be nice and warn anyone reading the log that, "hey, this network disconnection was intentional."

Quote:

I find most people who do well at this can retain about 50,000 lines of code, with a big IF based on complexity and language. After this point, you should be able to lead a code review on the code in question. You'll probably start seeing bugs and timing holes.

Yeah, I hardly even see the code any more, that's a blond, there's a redhead...

I notice when smart people are slow at finding bugs at work that it's usually their tools.

Absolutely. Use lots of tools. If you find a particular tool hard to use, odds are that there's a tool for that tool. For example, there are lots of frontends for debuggers. There are so many free tools out there it's crazy. It's not 1995 anymore. You don't have to "just work with what you got" because you can basically keep on getting more stuff!

People think that "there's an app for that" is limited to just phones and tablets... it's like that with developer tools too. There is something for what you're doing and a lot of the time it won't even cost you anything but your time finding it.

Lol yeah sure, in which management book did you find that? I have worked in a couple companies and that never works out. Partly because it's just very uncomfortable to give and receive critical feedback. So if it happens 2 times a year you are golden. Good feedback is a different story but a real assessment with good and part parts? Better left to an official setting.

And Co workers giving assessments sounds like hell and a recipe for a total bitchfest. I wouldn't like feedback from my butcher so it's not up to my Colleagues to give it. Everybody assessing other people sounds like a hippy Waldorf school setup. Unless you specifically ask for it because you sense something is off of course. Which is perfectly valid

For me its about the magic 10,000 hours or so it takes to become an expert at something. 2 years fully covers that, so if you are wondering why you still suck at the 2 year mark you need to look in the mirror, not at the people around you.

Example: 2 years, working 40 hours per week 47 of those weeks, taking courses 2 weeks, and 3 weeks vacation/holidays. Assuming no sick leave, assuming the person is working full time doing only this thing would net us 2*(47*40+2*40) = 3920 hours. but in reality it would be significantly less than that, I'd wager the more realistic number would be a bit over 3300 hours per 2-year period. So, we should expect about five years as more or less required to make the 10k hours threshold.

Incidentally, Malcolm Gladwell claims that it took him ten years to reach the 10k threshold on writing.

If you have a long backlog of bugs to fix, then there will invariably be some bugs that are easier to fix and some that will take quite an amount of effort to pin down. Once it has been found, the amount of code changes needed to make the fix will be mostly constant, though.Of course those developers who first snatch the easy bugs from the backlog will become the "high performers" in bug fixing, and those who assign themselves according to some order (such as priority..) will not.There is no way to numerically measure which bugs are hard, and which are not - such an assessment requires conscious effort by an experienced programmer, and even those will get it wrong half of the time.

I think that it is important to be able to discuss hard-to-find problems with other team members. Sometimes you would need help from co-worker who is more experienced in that module, and sometimes it will help just to have another set of eyes. You, and maybe your coworker, could actually learn something from that. If tracking down a bug takes time, the project manager should also be in on it within reasonable time. Sometimes a bug could be so hard to find, that you would have to create a workaround just to meet the release date for that fix.Unfortunately, there exist deadbeat managers out there, who are difficult to communicate with, who don't understand realities and/or who don't allow co-workers to cooperate properly. A developer should never have to be afraid of having to discuss a bug and ask for help.

Do you create more bugs than others? One great reason for being a slow programmer is because you're a careful one that possibly over-tests things, but hopefully produces less errors as a result. If your bug fix rate is slow then it shouldn't matter so much if you have less bugs to fix overall (or less that are your own fault).

How serious are these bugs? Sometimes the most major bugs are the result of a simple issue, and can be quickly found and eradicated. Other times you get a minor bug that seems easy to fix but that you just can't track down no matter what you try. If a lot of your bugs are in the latter category then a slow fix rate is to expected from even the best programmer in the world.

How busy are you? Or put a different way; is your programming/bug fixing that's slow, or are you struggling to find time to do these things? Do you have a higher workload than others? There's nothing worse than a team project where at a high level three features seem like equal workloads for three people, only to find you've been lumbered with the one that's much more complex than it seemed at first glance.

How slow is slow anyway? In any group there is always going to be someone slower than others, the real question is whether slow in this case is too slow. What about the fastest person? Are they fast because they're cutting corners, or taking on more work than they should and in doing so making you look bad for doing a sane amount of work? Finger pointing isn't usually the healthiest thing to do, but it's definitely no worse than telling your boss you'll do better if in reality you know that forcing yourself to work faster is going to be a bad thing to do.

Just some other food for thought; but it's too easy for a manager to look at something like a JIRA system and say "oh well Bob close 50 bugs this week at an average of one hour each, why don't you do that?" without realising that a heap of those bugs were duplicates and/or extremely simple/obvious things to fix, leaving you all the worst ones.

Threatening people to do better or else on intellectual tasks only makes them slower and more error prone. Here's a video (19min) by Dan Pink on why it doesn't work: http://www.youtube.com/watch?v=rrkrvAUbU9Y BTW, this has been know by physiologists for almost 100 years: Carrot-and-stick management techniques causes slowness and more errors.

How to program better:

1. Do not do deadlines, ever.

2. Write the unit tests as you code (and run them). That way, when you discover a bug, you're not face with a large glob of code to wade thru.

3. Insist that the requirements make sense before you start. It's not your job to do the work of the analysts. (If your company doesn't use analysts, tell your boss to pull up his socks and do the job right. Garbage in = garbage out)

4. Follow good coding practices. Never write a "quick and dirty" piece of code. It will always come back to haunt you.