I find this comment thread on how programmers will "automate" our way out of jobs to be especially pertinent as I work on my proposal. A contention I have, as do many others, is the primary practical challenge standing between us and robust systems is complexity. There are several on this thread who believe we have essentially mastered complexity for a broad number important application areas.

I think I could be convinced of this argument as long as those application areas are very narrow. As long as the creators of those applications are content with relatively standardized solutions. The moment there are A) functional requirements not covered by the standard case or B) differentiating requirements (such as aesthetics etc...) the general tool stops working. This is what makes complexity such a beast. If you make a fully generic tool, you have created a programming language. Perhaps a very high level (even graphical language like LabView) but still a language. Such things require trained people to operate.

"From the point of view of software engineering, the rapid spread of C represented a great leap backward. It revealed that the community at large had hardly grasped the true meaning of the term “high-level language” which became an ill-understood buzzword. What, if anything, was to be “high-level”? As this issue lies at the core of software engineering, we need to elaborate."

from : Wirth, N. (2008). A Brief History of Software Engineering. IEEE Annals of the History of Computing, 30(3), 32–39. doi:10.1109/MAHC.2008.33

"its rules could easily be broken, exactly what many programmers cherished. ... C provided freedom, where high-level languages were considered as straight-jackets enforcing unwanted discipline. It was an invitation to use tricks which had been necessary to achieve efficiency in the early days of computers, but now were pitfalls that made large systems error-prone and costly to debug and “maintain”." -- C is just a set of abstractions which need to be violated in order to get work done.

"at least pseudo-tools like software metrics had revealed themselves as being of no help, and software engineers were no longer judged by the number of lines of code produced per hour." -- Metrics, and measuring in general, have thus far proved quite useless in helping solve process problems in software engineering. You can't view programmers as an optimization problem.

"Just as structured programming had been the guiding spirit behind Pascal, modularization was the principal idea behind the language Modula-2, the successor of Pascal, published in 1979 [15]." -- Modularization is certainly a key idea.

"In the 1990s, a phenomenon started to spread under the name of Open Source. The distrust against huge systems designed in industrial secrecy became manifest." -- Distrust which waxed wained and I think will wax again.

Proposed Solution Space:

"A primary effort must be education toward a sense of quality. Programmers must become engaged crusaders against home-made complexity." -- Complexity is the mind killer. We must always battle complexity, simplify and abstract. When something becomes hard to reason about it is time to make an abstraction such that at the abstract level reasoning is clear. The proof falls out of the setup if you will. However, it must also be clear that meaningless abstraction does no one any good no AbstractFactoryBeanImplementors here.

"It is therefore a sad fact that it[Academia] has remained inactive and complacent. Not only has research in languages and design methodology lost its glamour and attractivity, but worse, the tools common in industry have quietly been adopted without debate and criticism. Current languages may be inevitable in industry, but for teaching, for an orderly, structured, systematic, well-founded introdution they are entirely mistaken and obsolete." -- The Academy has accepted a secondary place in innovation. With pressure from all sides to produce "useful" graduates and to do so at low cost it has settled on providing what industry wants (which is different from what it needs).

"Among its tools languages figure in the forefront. A language with appropriate constructs and structure, resting on clean abstractions, is instrumental in building artefacts, and mandatory in education. Home- made, artificial complexity has no place in them. And finally: It must be a pleasure to work with them, because they enable us to create artefacts that we can show and be proud of." -- A classic rogramming language appology, if only our tools were better all our problems would be solved! I actually disagree with this conclusion I don't think there exists a language which makes writing software simple, clear, and perfect. When the problem is hard so is the solution. Better tooling is important but it can only take us so far.

From http://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/EWD340.html"""Argument three is based on the constructive approach to the problem of program correctness. Today a usual technique is to make a program and then to test it. But: program testing can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence. The only effective way to raise the confidence level of a program significantly is to give a convincing proof of its correctness. But one should not first make the program and then prove its correctness, because then the requirement of providing the proof would only increase the poor programmer’s burden. On the contrary: the programmer should let correctness proof and program grow hand in hand. Argument three is essentially based on the following observation. If one first asks oneself what the structure of a convincing proof would be and, having found this, then constructs a program satisfying this proof’s requirements, then these correctness concerns turn out to be a very effective heuristic guidance. By definition this approach is only applicable when we restrict ourselves to intellectually manageable programs, but it provides us with effective means for finding a satisfactory one among these."""

Essentially what Dijkstra is advocating here is what the approach the most advanced type theories are striving for. They aim to allow the programmer to specify in the type system (a proof system) important semantic invariants of the application. The type system is constructed in such a way the program is only well typed if the specified invariants hold. The hope of the type theorists is that with a sufficiently powerful type (proof) system most if the not all properties one cares to prove are in fact provable (for certain programs).

The difficulty for the practicing is most type systems are not nearly powerful enough to specify properties which are actually interesting. This leaves the programmer doing essentially the proof equivalent of book keeping with no benefit. I should distinguish here between dynamically checked types (properties) and statically checked types. Dynamically checked, that is at run time, are always enormously helpful to the programmer because they provide runtime safety. However, statically checked types can be overly burdensome if they require lots of book keeping without sufficiently powerful proofs.

Unfortunately, for a wide variety of statically checked programming languages the available proofs are uninteresting and the burden is high. This is the challenge to the Dijkstra-ist.