Comment

I was pondering my previous reply (after saving it, of course) and thinking that in my haste to respond to the fork suggestion, I might have started this thread down a path I didn't intend.

First, I'm not looking for you or anyone else to solve this particular issue for us. We've spent a lot of time looking at a lot of possible solutions, and unfortunately, this particular problem is one that does not lend itself to refactoring into separate process. (The cross-platform issue I mentioned is one reason, but there are also performance issues with fork()ing large memory images that chew up the anticipated benefit from calculating dependencies in parallel.)

Second, I'm much more in the middle of the larger GIL issue than my previous two replies might imply. I'm perfectly willing to believe that the GIL brings with it implementation efficiencies, and that if Python threads were to be re-architected without a GIL, we might pay for it in slower performance in other areas. So overall, the GIL is what it is, and simply has to be accounted for or programmed around. To that extent, I agree with your sentiment that people who cast it as the root of all evil are probably inflating the problem.

That having been said, it would be really handy in our case to not have to program around the GIL, because our architecture already provides a huge, proven performance boost except when we want to do multiple things in Python. Can we program around that? Sure, probably. And we'll have to, if we want to speed up this part of SCons. But that doesn't change the fact that if I could wave a magic wand and create a Python without a GIL (and assume no side effects from that change), we'd get a huge performance win, with virtually no programming effort at all.

So the larger point I want to make here is simply that the fact that you personally haven't run into the situations described in your criteria doesn't mean that there aren't compelling, real-world counter-examples to your assertion that the GIL is only a problem in "one little corner case, boo hoo." Of course, you're free to decide that this is just an example of your "corner case," but I can tell you that we have a significant and growing user community that would benefit hugely if we can speed up our dependency analysis, and that at a minimum the presence of the GIL means that it'll be more work than I'd like it to be...