If you've discovered something amazing about Perl that you just need to share with everyone,
this is the right place.

This section is also used for non-question discussions about Perl, and for any discussions that are not specifically programming related. For example, if you want to share or discuss opinions on hacker culture, the job market, or Perl 6 development, this is the place. (Note, however, that discussions about the PerlMonks web site belong in PerlMonks Discussion.)

Meditations is sometimes used as a sounding-board — a place to post initial drafts of perl tutorials, code modules, book reviews, articles, quizzes, etc. — so that the author can benefit from the collective insight of the monks before publishing the finished item to its proper place (be it Tutorials, Cool Uses for Perl, Reviews, or whatever). If you do this, it is generally considered appropriate to prefix your node title with "RFC:" (for "request for comments").

I have been tinkering with a few tools lately, and now want to put up a portfolio of some web applications that I am working on. I have an account on pair Networks (they also host this site), so I set up local::lib and went ahead and tried to install Mojolicious::Lite, since that's the platform I'm working on these days.

No dice -- Mojo::Lite requires 5.10, and pair only has 5.8.9. I checked with the other provider I use, and they have 5.8.8.

So the two options I can see are a) install an up-to-date Perl on one of those accounts, or b) have these web applications run on my home machine (perhaps using http://www.easydns.com to provide consistent name resolution -- not sure is this is still available).

I could go find another web provider, but that's additional expense, and not really my best option right now. Feedback welcome!

This is my very first post on this site!
I wrote my first program, newscript.pl , a few times ago, in order to save some repetitive typing and I really want your opinions, critics, suggestions etc..

It's job is very simple, create an empty script. At first written in bash, with only bash as supported language, now written in Perl it includes Perl, Bash and I started slowly working to include C as well.
It's not very portable and surely not very efficient, but I'm using it everyday, when reading thru intermediate Perl, advanced Bash scripting and learning C the hard way.

The script first ask user for the name of the new script, then it asks for the language you're going to use. It will then print the shebang line as well as wanted modules (if you asked for a Perl script) and save it to a file withyourname.test. It then fires up emacs -nw with your newscript so you are ready to input code right away.

I just read about an IBM programming challenge to try and entice developers to IBMs Bluemix Cloud development environment.

(Don't bother if you're outside the UK; or if you want to use Perl (it ain't supported :(); or if stupid sign-up processes that don't work annoy you; or ... )

What struck me was that the three programming tasks are, at least notionally, so trivial. It took me less than 5 minutes to write (working, but probably not best) solutions to all three.

(Whether they would pass their test criteria I guess we'll probably never know)

I was also struck by this part of the description:

that you can put together a programme that can run within a time limit or on limited resources rather than just lashing together a hideous brute-force monstrosity. And that you can actually read the questions properly in the first place (a useful start, but one that's often forgotten).

I think it would be interesting to see how the best Perlish solutions we can come up with compare with those other languages that get entered to the competition; when and if they are actually made public.

So have at them. (Don't forget to add <spoiler></spoiler> tags around your attempts.)

I'd post the questions here but I'm not sure it wouldn't be a problem copyright wise?

I'll post my solutions here in a few days.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

This is a collection of thoughts I have been slowly putting together based on experience, watching the development (and often bad implementation of) agile coding practices. I am sure it will be a little controversial in the sense that some people may not see agile as something to move beyond and some may see my proposals as being agile.

What's wrong with the Waterfall?

I think any discussion of agile programming methodologies has to start with an understanding of what problems agile was intended to solve and this has to start with the waterfall model of development, where software has a slow, deliberate life cycle, where all design decisions are supposed to be nailed down before the code is started. Basically the waterfall approach is intended to apply civil engineering practices to software and while it can work with very experienced teams in some limited areas it runs into a few specific problems.

The first is that while civil engineering projects tend to have well understood and articulated technical requirements, software projects often don't. And while cost of failure in dollars and lives for a civil engineering disaster can be high, with software it is usually only money (this does however imply that for some things, a waterfall approach is the correct one, a principle I have rarely seen argued against by experienced agile developers).

The second is that business software requirements often shift over time in ways that bridges, skyscrapers, etc don't. You can't start building a 30 floor skyscraper and then have the requirements change so that it must be at least 100 floors high. Yet we routinely see this sort of thing done in the software world.

Agile programming methodologies arose to address these problems. They are bounded concerns, not applicable to many kinds of software (for example software regulating dosage of radiotherapy would be more like a civil engineering project than like a business process tool), but the concerns do apply to a large portion of the software industry.

How Agile is Misapplied

Many times when companies try to implement agile programming, they run into a specific set of problems. These include unstructured code and unstructured teams. This is because too many people see agile methodologies as devaluing design and responsibility. Tests are expected to be documentation, documentation is often devalued or unmaintained, and so forth.

Many experienced agile developers I have met in fact suggest that design is king, that it needs to be done right, in place, and so forth, but agile methodologies can be taken by management as devaluing documentation in favor of tests, and devaluing design in favor of functionality.

There are many areas of any piece of software where stability is good, where pace of development should be slow, and where requirements are well understood and really shouldn't be subject to change. These areas cut against the traditional agile concerns and I think require a way of thinking about the problems from outside either the waterfall or agile methodologies.

Subsidiarity as a Team Design Principle

Subsidiarity is a political principle articulated a bit over a hundred years in a Papal encyclical. The idea is fairly well steeped in history and well applicable beyond the confines of Catholicism. The basic idea is that people have a right to accomplish things and that for a larger group to do what a smaller group can therefore constitutes a sort of moral theft. Micromanagement is therefore evil if subsidiarity is good but also it means that teams should be as small as possible, but no smaller.

In terms of team design, small teams are preferable to big teams and the key question is what a given small team can reasonably accomplish. Larger groupings of small teams can then coordinate on interfaces, etc, and sound, stable technological design can come out of this interoperation. The small team is otherwise tasked with doing everything -- design, testing, documentation. Design and testing are as integrated with software development as they are in agile, but as important as they are in the waterfall approach.

This sort of organization is as far from the old waterfall as agile is but it shares a number of characteristics with both. Design is emphasized, as is documentation (because documentation is what coordinates the teams). Stability in areas that need it is valued, but in other areas it is not. Stable contracts develop where these are important and both together and individually, accomplishments are attained.

Subsidiarity as a Technical Design Principle

Subsidiarity in team design has to follow what folks can accomplish but this also is going to mean that these follow technological lines as well. Team responsibility has to be well aligned with technological responsibility. I.e. a team is responsible for components and components are responsible for functionality.

The teams can be thought of as providing distinct pieces of software for internal clients, taking on responsibility to do it right, and to provide something maintainable down the road. Once a piece of software is good and stable, they can continue to maintain it while moving on to another piece.

Teams that manage well defined technical and stable components can then largely end up maintaining a much larger number of such components than teams which manage pieces that must evolve with the business. Those latter teams can move faster because they have stability in important areas of their prerequisites.

But the goal is largely autonomous small teams with encapsulated responsibilities, producing software that follows that process.

Lately I've been experimenting again with using Perl regexes more like grammars, i.e. parsing inputs via a single big regex that involves lots of branching, instead of the traditional approach of parsing inputs via imperative "spaghetti code" that sequentially matches lots of small regexes.

However, I quickly ran into two limitations relating to regex quantifiers (*+{}). Here's a write-up of the solutions/workarounds I found, both for my own benefit (so I can refer back to them), and in case others might find it interesting.

Also, I'd love to hear the opinions of other monks on which of these techniques should be used in real code, and if it would be worth adding new Perl 5 core featues to make them obsolete.

What makes this sequence interesting is N. J. A. Sloane’s conjecture
that every number eventually appears.

Coding the sequence is simplicity itself; the challenge is to test Sloane’s
conjecture by keeping track of the numbers that have not yet appeared in the
series. My initial, naïve approach was to use a sieve, à la
Eratosthenes:

But this turned out to be far too memory-hungry: for values of MAX of
the order of twenty million, RAM usage on my 3GB system approaches 100%,
thrashing sets in, and the script (along with the rest of Windows) grinds to a
shuddering halt.

Surely, I thought, there must be a memory-efficient way to represent a sieve?
And of course there is, and of course it was already implemented on CPAN.
A little searching led to the Set::IntSpan module which stores runs
of consecutive integers as spans, allowing large (even infinite)
collections of integers to be represented very economically.

Calculation of successive terms in the Recamán sequence is noticeably slower
using Set::IntSpan for lookup than it is using a hash. But, as the
adage says, it’s
better to be late than be dead on time. (This was the slogan of an
Australian safe driving ad campaign some years ago.) For the record: I also
looked at Set::IntSpan::Fast and Set::IntSpan::Fast::XS.
The latter failed to install on my system, and the former actually ran slower
than Set::IntSpan for this use-case.

Turns out that Set::IntSpan not only solves the memory problem, it also makes
it possible to dispense with an upper bound for the sieve. How, then, to display
progressive results? Well, the OEIS has a couple of additional series related to
Recamán’s:

A064228: values of R(n) that take a record
number of steps to appear: 1, 2, 4, 19, ...

A064227: the values of n corresponding to the
values in A064228: 1, 4, 131, 99734, ...

So I recast the script to output successive values of these two series:

First, we all know that micro-optimisation is pointless
until you have first selected the best algorithm(s). But optimising an algorithm may actually consist in optimising its underlying
data structures. Obvious? Yes, but still worth a reminder now and then.

This meditation is about a tied hash package, and how it came into existence. I am still meditating whether this is too obscure, or whether its goal is better achieved using some other technique; is it worth being uploaded to CPAN as yet another strange perl delirium? is the name ok? Any suggestions, review, critics are welcome. Thanks for your time.

Itch

Over 100 poorly performing scripts written in some BASIC dialect for exactly the same purpose (read source records, transform them, write target records), sporting hardcoded parameters and different output assembling code, proliferating with each new customer (copy over, twiddle, tweak).

Scratch

Perl to the rescue to do the data gathering and munging, and write a unified import CSV to be fed into that dratted basic script - one for all. Parameters and data transforming procedures should be kept separate, in a format editable by non-perlers. I choose INI file style, which fitted both that BASIC dialect and perl:

Straight forward. What about the data transforming rules? Since these are concatenations of values from the input record - conveniently present as a hash - and the output of some functions munging those values, these are something that could easily be transformed into subroutines:

and have a magic hash %h which encapsulates all that logic and "knows" what to deliver.

Relief

A tied hash. update: - which is a subroutine factory.

<update>
...due to BrowserUk's immediate reaction below: this is overkill to just execute CODE in the value slots of a hash, of course. Well, the itch was the starting point. The code generation and closure bits (see EXAMPLE in the pod) tell what it might be useful for: currying, building dispatch tables, ... - I have to play more with this, yet.</update>

NOTE: I've been considering translating this five part series I wrote a few years ago to PerlMonks, and I thought I'd go for it to see what others thought. Parts 2-5 are still linked to the blog to give context to the entire series. If people think this should be made into a Tutorial, I'll clean up and translate the rest of the docs and link them properly.

Thoughts?

BEGIN TUTORIAL...

Understanding references and their subtleties in Perl is one of the more difficult concepts to fully wrap one's head around. However, once they are fully understood by the blossoming developer, they find a whole new level of capability and power to exploit and explore.

I often see newer programmers struggle with the concept of references on the Perl help sites I frequent. Some still have a ways to go, but many are at the stage where perhaps one more tutorial may push them over the edge and give them that 'Ahhhh' moment of clarity. My moment of clarity came when I read Randal Schwartz's "Learning Perl Objects, References & Modules" book for the something like the 8th time. Although once the concept of references is understood, the syntax and use cases can still be confusing for quite some time, especially in Perl, because There Is More Than One Way To Do It.

This tutorial is the first in a five part series. This part will focus on the basics, preparing you for more complex uses in the following four parts. I've created a cheat sheet that summarizes what you'll learn in this document.

I will stick with a single consistent syntax throughout the series and will refrain from using one-line shortcuts and other simplification techniques in loops and other structures in hopes to keep any confusion to a minimum. Part one assumes that you have a very good understanding of the Perl variable types, when they are needed, and how they are used. Some exposure to references may also prove helpful, but shouldn't be required.

THE BASICS

References in Perl are nothing more than a scalar variable that instead of containing a usable value, they 'point' to a different variable. When you perform an action on a reference, you are actually performing the action on the variable that the reference points to. A Perl reference is similar to a shortcut to a file or program on your computer. When you double click the shortcut, the shortcut doesn't open, it's the file that the shortcut points to that does.

At this point, if you tried to print out the contents of $aref, you would get the location of the array being pointed to. You know you have a reference if you ever try to print a scalar and you get output like the following:

Before we can use the array the reference points to, we must dereference the reference. To gain access to the array and use it as normal, we use the array dereference operator @{}. Put the array reference inside of the dereference braces and we can use the reference just as if it was the array itself:

You just learnt how take a reference to an array (by prepending the array with a backslash), how to dereference the entire array reference by inserting the reference within the dereference block @{}, and how to dereference individual elements of the array through the reference with the -> dereference operator. That is all there is to it. Hashes are extremely similar. Let's look at them now.

Now we'll iterate over the hash through the reference. To access the hash, we must dereference it just like we did the array reference above. The dereference operator for a hash reference is %{}. Again, just wrap the reference within its dereferencing block:

That's essentially the basics of taking a reference to something, and then dereferencing the reference to access the data it points to.

When we operate on a reference, we are essentially operating on the item being pointed to directly. Here is an example that shows, in action, how operating directly on the item has the same effect as operating on the item through the reference.

This concludes Part 1 of our Guide to Perl references. My goal was not to compete with all the other reference guides available, but instead to complement them, with the hope that perhaps I may have said something in such a way that it helps further even one person's understanding. Next episode, we'll learn about using references as subroutine parameters.

This is meditation; but I also hope that it might start a discussion that will come up with (an) answers to what I see as an ongoing and prevalent problem.

This has been triggered at this time by my experience of trying to wrap my brain around a particular complex module; but I don't want to get into discussion particular to that module, so I won't be naming it.

Suffice to say that CPAN is replete with modules that are technically brilliant and very powerful solutions to the problems they address; and that deserve far wider usage than they get.

In many cases the problem is not that they lack documentation -- often quite the opposite -- but more that they don't have a simple in; a clearly defined and obvious starting point that gives a universal starting point on which the new user can build.

And example of (IMO) good documentation is Parallel::ForkManager. It's synopsis (I've tweaked it slightly to remove a piece of unnecessary fluff):

is sufficient to allow almost anyone needing to use it, for almost any purpose, to put together a reasonable working prototype in a dozen lines of code without reading further into the documentation.

It allows the programmer to get started and move forward almost immediately on solving his problem -- which isn't "How to use P::FM" -- and only refer back to and utilise the more sophisticated elements of P::FM, as and when he encounters the limitations of that simple starting point.

As such, the module is successful in hiding the nitty-gritty details of using fork correctly; whilst imposing the minimum of either up-front learning curve or infrastructural boiler-plate upon the programmer; who has other more important (to him) things on his mind.

Contrast that with something like POE which requires a month of reading through the synopsis of the 800+ modules in that namespace POE::*, and then another month of planning, before the new user could put together his first line of code. As powerful as that module, suite of modules; dynasty of modules is, unless you have the author's help, and lots of time, getting started is an extremely daunting process. In that respect (alone perhaps), POE fails to enable a 'simple in'.

And before anyone says that it is unfair to compare those two modules -- which maybe true -- the purpose was to pick extremes to make a point; not to promote or denigrate either.

Another module that I know I should have made much more use of in the type of code I frequently find myself writing, is PDL. I've tried at least a dozen times to use PDL as a part of one of my programs; and (almost) every time I've abandoned the attempt before ever writing a single line of PDL, because I get frustrated by the total lack of a clear entry point in to the surfeit of documentation.

There's the FAQ, and the Core; and the Index; and the QuickStart; and the Doc; and the Basic; and the Lite; and the Course; and the Philosophy; and the pdldoc; and the Tips; and ... I'm outta here. I'm trying to write my program, which does a little math on some biggish datasets that would benefit from being vectored, but life's too short...

Again; the underlying code is brilliant (I am assured), and it isn't a case of a lack of documentation; just a mindset that says: "this is PDL in all its glory, power and nuance. bathe yourself in its wonderfulness and wallow in its depth". Oh, and then when you've immersed yourself in its glory, understood its philosophy, and acclimated its nuance, then you can get back to working out how to use it to solve your problem.

And that's a real shame; and a waste.

I'm not sure what the solution is. I do know that the modules I use most List::Util, Data::Dump, threads etc. I have rarely ever had to look at the documentation; their functionality has (for me) become an almost invisible extension of Perl itself, and only the occasional (perhaps you forgot to load "sum"?) reminds me that they aren't.

Of course, what they do is essentially pretty simple; but that in itself is a perhaps a clue.

I do know that (for me) the single most important thing in encouraging my use of a module is being able to C&P the synopsis into my existing program, tweak the variable names, and have it do something useful immediately.

In part, that comes down to a well designed API; in part, to well-chosen defaults; and part having a well-chosen, well-written synopsis that addresses the common case; with variable names and structure that make it obvious how to adapt that synopsis for the common case. Once I have something that compiles and runs -- even if it doesn't do exactly what I need it to do; or even what I thought it would do from first reading -- it gives me a starting point and something to build on. And that encourages me to persist. To read the documentation on a as-I-need-to basis to solve particular problems as I encounter them.

And that I think has to be the correct approach to documenting complex modules. They need to:

Offer a single, obvious, starting point. The in.

That needs to be very light on history, philosophy, jargon, technical and social commentary and background. And choice.

It needs to offer a single, simple, well-chosen, starting point, that requires minimal reading to adapt to the users code, for the common case.

It then needs to offer them a quick, clear, simple path to solving their problem.

What it must not do:

It mustn't present them with 'a bloody great big list of entrypoints/methods'.

It mustn't offer them a myriad of choices and configuration options.

It mustn't take them on a deep immersion in the details of either algorithms or implementation.

It mustn't present them with either "Ain't this amazing" nor "Ain't I clever" advert.

It mustn't waste their time with details of your personal preferences, prejudices, philosophies and theologies.

If you want programmers to use your modules, you need to tell them what (the minimum) they *NEED TO KNOW* to get started. And then give a clear index to the variations, configurations and extensions to that basic starting point.

Achieve that, give them their 'in', with the minimum of words, fuss or choice, and they'll come back for all the rest as they need it.

This is ill-thought through and incomplete, so what (beyond risking offending half the authors on CPAN) am I trying to achieve with this meditation?

I'd like to hear if you agree with me? Or how you differ. What you look for in module documentation. Examples that you find particularly good; or bad.

It'd be nice to be able to derive from the thread, a set of consensus guidelines to documenting moderate to complex modules -- that almost certainly won't happen -- but if we managed to get a good cross section of opinions on what makes for good and bad documentation; and a variety of opinions of the right way to go about it; it might provide a starting point for people needing to do this in the future.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

My 1st and 2nd attempts got me warmed up and thought faster is possible. The following demo is my 3rd attempt and writes 1 million cells combined in less than 6 seconds from start to finish and 57 seconds for 10 million cells. Running serially takes 15 and 141 seconds for 1 and 10 million cells respectively. Processors have turbo boost for some time. Thus, serial code is likely to run at a faster GHz.

Writing text data will slow this down a little due to obtaining the next unique id from the shared strTable object. The internal str_table is shared between worksheets in Excel::Writer::XLSX. Thus, synchronization is necessary as well.

Note: This requires MCE from trunk r957 or later which includes MCE::Shared as MCE 1.700 is not yet released. The logic consumes only the memory necessary. There is never duplicate data from running multiple workers.

About a year ago I started with the idea of creating a Perl module based on Net::NTP. The module that I am thinking to create would be named (Net::SNTP::Client). The difference between those two is the precision, from my point of view the Net::NTP module does not get correct millisecond/nanosecond precision. The module is based on RFC4330, where according to the RFC different precision will achieved on LinuxOS and WindowsOS.

In theory the module should be compatible with all OS (WindowsOS, LinuxOS and MacOS) please verify that with me since I only have LinuxOS.

I am planning to create also another module Net::SNTP::Server which is approximately an SNTP server and when I say approximately is because I can not figure it out how to replicate the server side. But any way first thing first.

Is it possible to take a look and assist me in possible improvements and comments. Since this is my first module I have no experience so maybe the module is not well written.

The execution of the script is very simple, create a script e.g. client.pl and put the code bellow.

The first option is to get an RFC4330 printout way, and the second option is to clear the screen before the printout. I think both options will be useful on the printout of the script.

I have chosen to paste the module in the folder path "/home/username/Desktop/SNTP_Module/Net/SNTP/Client.pl". Remember for testing purposes to change the path on client.pl accordingly on the location that you will place the module.

But in many ways, it also sums up my career. (Polite pause as the twitter of laughter dies down.) For most of the past 25 years or so, I’ve been involved in projects. Generally, not ones that I had started. Generally, not healthy ones. Dead ones, or very nearly so. My task was to try to “turn them around,” and I generally did. Whether or not my attempts at resuscitation were actually long-term successful, this experience did teach me a lot of the reasons ... and they are human reasons ... why software projects so often go so badly wrong. I’m not going to do any preaching here, although it may seem so. I’m just relating some of my personal experiences in a mortuary project triage. (FYI: Teams-in-place were anywhere from one to fifteen people, most of whom had “split the coop.”)

First of all, these projects typically started out with “a great deal of enthusiasm, but no real plan.” The usual justification was that the project needed to “hurry up to market,” or that the stakeholders in the project “would know it when they saw it” and the managers of the project (if there were any ...) simply gave-up trying to ask them to make up their minds.

And, of course, in several cases, those stakeholders were assured that they didn’t have to make up their minds. “Self-directed teams,” the programmers purred self-confidently, “would produce a ‘potentially viable product(!)’ every two weeks!

“SOP = SOTP.™” Standard Operating Procedure = Seat Of The Pants.

And yet, what happened ... what inevitably happened ... is that everything in the “software mechamism” turned out to be inextricably coupled to everything else. As layers of code were piled on, and as changes pinged-and-ponged throughout all those layers, the whole thing fell down in a heap as the programmers sailed on to the next green pasture.

Many software projects are actually the work of one Guy. (Sorry, ladies ...) That “one guy” might be surrounded by several other people, but this is simply an attempt to scale-up the onlymodus operandi that this One Guy actually knows: himself. The project “feels its way along” because that’s how he’s used to doing it. (And, because he is a crackerjack programmer, is used to eventually succeeding producing something.) There simply isn’t any experience in being part of a successfully managed project: most programmers, I candidly suspect, haven’t actually seen one. (And there were no Angelic Choirs that started singing when I showed up either, I’m afraid ... no self-sunshine here.)

The underlying reason for these problems, I think, is: a very natural human reaction to what is a virtually-unmanageable technical situation. The objective of the project is to build a self-directing machine ... and to do it perfectly, because nothing less than perfection will do. Viewed as a mechanism, software would be said to have “unlimited degrees-of-freedom.” i.e. “Anything is connected to everything else.” Although the instant-to-instant flow of control within the software is of course described by if/then/else and looping constructs, the actual mechanism is also determined by its internal and external state. This concern for “state” is what causes the coupling. (And it’s also one of the reasons why “Functional Programming” is such a hot research topic.)

My biggest criticism of Scrum, and Agile, and XP, and, well, most “methodologies,” is that they ignore this aspect. They focus, instead, upon the organization and the daily work-activities of the team. They discuss things like “user stories,” which are simply one possible way of trying to express one’s ideas and plans to a customer, but then omit from consideration exactly how that “story” is to become if/then/else, and how that new web of decision-logic is to be tested, and how it both affects and is affected by (“is infinitely coupled to ...”) everything else. As a paradigm, useful in one sense though it may be, it does not and probably cannot (IMHO) go far enough.

“We are building a self-directing machine.” That, quite frankly, is the light-bulb moment that I got from the Managing the Mechanism e-book. It’s something that we can say to business stakeholders, except that it is extremely likely to scare them off. It certainly does, I think, cast some useful insights on what we might be missing in our present-day methodologies. We certainly do need better processes for our work, better ways to describe them, and better ways to inform stakeholders of exactly what we need from them and why.

In closing, one of the most prevalent things that I have seen, in every project that I have tried to turn-around, is disillusionment. On both sides of the aisle. Long before the software had broken down, communication had also broken down, and so had business process (if it ever truly existed). No one builds houses and bridges that way. (For very obvious, flammable and heavy reasons, no one is allowed to ...) I suspect that the seeds of project failure are sown almost as soon as the first plow-blade cuts the soil. This is our problem, as a profession, and we need a better solution to it. Perhaps a different viewpoint is a start.

That’s my Meditation. Borne, as I said, from a most-interesting career path that has not always been a happy one. What do you think? What have your experiences been? For instance, have you worked-through a spectacular success story from one of these other strategies? I’d love to hear it . . . The water in the cooler is ice-cold and there’s beer in the fridge that’s even colder. May the discussions begin?

In 2008, we were pretty much a Scrum company ...
however, a few years later, we had grown into a bunch of teams and
we found that some of the standard Scrum practices were actually getting in the way.
Rules are a good start, then break them when needed.
We decided that Agile matters more than Scrum.

Autonomy means the squad decides what to build, how to build it, and how to work together while doing it.
One consequence of autonomy is that we have very little standardization.
When people ask things like which code editor do you use or how do you plan,
the answer is mostly depends on which squad.
Some do Scrum sprints, others do Kanban ... it's really up to each squad.
Instead of formal standards we have a strong culture of cross-pollination;
when enough squads use a specific practice or tool, such as git,
that becomes the path of least resistance, and other squads tend
to pick the same tool.

Why is autonomy so important?
Well, because it's motivating and motivated people build better stuff.

Instead of blindly following Scrum dogma,
I advise you to
analyse the problems you and your company face daily.
Reason about them.
Consider applying Agile and Lean principles to them.
Experiment to see what works for you and what doesn't.

After feeling isolated and alone in resisting the Scrum imposition,
reading Spotify's story has cheered me up.
I've become more hopeful that things will change, that over time more
folks will come to see the benefits of greater team autonomy.

Autonomy and Alignment

It's kind of like a jazz band, although each musician is autonomous and plays his own instrument,
they listen to each other and focus on the whole song together. That's how great music is created.
So our goal is loosely coupled but tightly aligned squads.

Down here is low alignment and low autonomy, a micromanagement culture, no high level purpose, just shut up and follow orders.
High alignment and high autonomy means leaders focus on what problem to solve but let the teams figure out how to solve it.
Alignment enables autonomy.

For high autonomy to work, you need high alignment.
With low alignment, teams simply do whatever they want,
with each team going off in a different direction.

Specialists vs Generalists

Each system is owned by one squad.
But we have an internal open source model and our culture is more about sharing than owning.
Suppose squad one here needs something done in system B and squad two knows that code best,
they'll typically ask squad two to do it.
However, if squad two doesn't have time, then squad one doesn't necessarily need to wait.
Instead they're welcome to go ahead and edit the code themselves and then ask squad two to review the changes.
So anyone can edit code but we have a culture of peer code review.
This improves quality and spreads knowledge.
Over time we've evolved design guidelines, code standards,
and other things to reduce engineering friction, but only when badly needed.

The basic unit of development is the squad.
Because each squad sticks with one mission and one part of the product for a long time,
they can really become experts in that area.
A tribe is a collection of squads that work in related areas.
The chapter is your small family of people having similar skills and working within the same general
competency area (e.g. QA or Web Development),
within the same tribe.
As a squad member, my chapter lead is my formal line manager, a servant leader,
focusing on coaching and mentoring me as an engineer,
so I can switch squads without getting a new manager.
A guild is a more organic and wide-reaching "community of interest", a group of people that want to share
knowledge, tools, code, and practices. Chapters are always local to a tribe, while a guild usually cuts
across the whole organization. Some examples are: the web technology guild, the tester guild, the agile
coach guild.
Anyone can join or leave a guild at any time.
Most organizational charts are an illusion, so our main focus is community rather than hierarchical structure.

Alistair Cockburn (one of the founding fathers of agile software development) visited Spotify and said
"Nice - I've been looking for someone to implement this matrix format since 1992 :) so it is really welcome to see"

In learning more about Spotify, I was also glad to learn how they
deal with specialization.
You see, this tricky topic has been a chronic nuisance for us,
for a number of reasons.

First, code quality.
New code written by a generalist, and reviewed by a generalist,
is a long term code quality nightmare.
Can you imagine what Perl code written by a ten-year Java veteran with
a few days of Perl experience looks like?
I can because I've seen it.
There have been times when I've opened up a Perl file and rolled on the floor laughing
because it was clear that whoever wrote it had no understanding of Perl at all.

Second, employee engagement.
Though many programmers are happy to become generalists,
a significant minority (including me) are deeply dissatisfied with that role.
I derive much more job satisfaction from doing an expert job
in something I deeply understand than from "cargo-culting" some code
that seems to kinda work even though I lack a deep understanding of why.
I find that dissatisfying.
I do not want to write Java code that causes my Java expert colleague
to roll on the floor laughing.

Another problem is that system architecture
can be compromised if nobody focuses on the integrity of the system as a whole.
To mitigate this risk, Spotify have a "System Owner" role.
All systems have a system owner, or a pair of system owners
(one with a developer perspective and one with an operations perspective is common).
They further have a chief architect role,
someone who coordinates work on high-level architectural issues
that cut across multiple systems.

Healthy Culture Heals Broken Process

Trust is more important than control.
Agile at scale requires trust at scale.
And that means no politics. It also means no fear.
Fear doesn't just kill trust, it kills innovation because if failure
gets punished people won't dare try new things.

I've seen first hand how a healthy organisational culture of good
communication and continuous improvement
can effectively solve process problems.

I've also seen first hand how a culture of fear and an over-emphasis on control and structure
can harm innovation.

Lean Startup

We aim to make mistakes faster than anyone else.
Continuous improvement, driven from below and supported from above.
Failure must be non-lethal and with a "limited blast radius".
Lean startup principles: think it, build it, ship it, tweak it.
The biggest risk is building the wrong thing.
Release first to a small percentage of users, then use A/B testing, then gradually roll out to the rest of the world.
Impact is more important than velocity.
Innovation more important than predictability.

These are some of the ideas used by Spotify to build and release product.
Since there is enough in this installment already, I'll postpone
a discussion of Lean startup
and related ideas to the next episode.

The Agile Alliance formed, conferences were held,
companies went mad wanting a bit of this, everyone was having standup meetings,
burndown charts, product backlogs, Agile coaches all over the place,
more Agile coaches than developers ...
and Post-its, Post-its, Post-its everywhere.
The more Post-its you have the more agile you are.

We spent ten years talking about people, interactions, team building, eliminating waste ...
and at some point agile took a detour ... process became more important than technical practices ...
everyone went crazy at the Post-it party, three years later they woke up
and realised that every two weeks we see the pile of s#?! getting bigger.

Around 2008, some of the original Agile Manifesto-istas, led by Robert C Martin (aka "Uncle Bob"),
felt that Agile had gone off the rails, with too much emphasis on process
rather than technical practices and code quality.
This group felt that Agile projects, every two weeks, relentlessly, steadily, iteratively,
were producing more and more crap code.
Adding to technical debt. Slowing us down.
Though Agile gave good feedback on many things, code quality was not one of them;
it is not visible on burndown charts or the Scrum board.

How does this happen?
Well, there seems to be an unwritten law of the Daily Standup
that every morning you have to move at least one Post-it note.
To comply with this "law", I sometimes pick up the Post-it note I was working
on yesterday and move it a few millimeters to the right on the Scrum board
while explaining why it is
not really done yet.
I always feel bad when I do this though; I would much rather
proudly proclaim that it is done, so as to publicly show-off how productive I am.

Sadly, I've occasionally noticed folks succumb to this psychological pressure
to proclaim something done
when it is not ... resulting in an every increasing pile of poo,
as illustrated by Sandro's next (pair programming) anecdote:

"but what about these names, they don't make sense ... and there's lots of duplication ... and ..."
yeah, well as it's almost done now let's just check it in and add that to the technical debt backlog.
This was a brand new feature in an agile team!
The guy (Sandro was pairing with) was writing brand new code already thinking
to add stuff to the technical debt backlog!

Not only working software but also well-crafted software. Code we can refactor confidently and without fear.

Not only responding to change but also steadily adding value. Not just bug-fixing, but improving code structure, relentlessly keeping it clean.

Not only individuals and interactions, but also a community of professionals. Mentoring, sharing, user groups, passion, professionalism.

Not only customer collaboration, but also productive partnerships. Not a "manager giving orders to a developer" relationship, rather a partnership of equals built on trust.

Well-crafted Software

God forbid that we have one day someone who is a Software Craftsmanship coach
... it is about lead by example, being a mentor
... not about beautiful code, just trying to provide value, not writing crap code for your customers.
The lack of craftsmanship can be one of the main causes of failing projects.

Like Sandro, I disagree the view that "Software craftsmen just care about beautiful code".
Today, for example, it took me quite a while to figure out
why some code that "should" fail was in fact reporting "success".
I almost fell off my chair when I finally realized that it
was silently catching and throwing away all exceptions!
No comment explaining why it would do such a bizarre thing.
To me, this sort of sloppiness shows a basic lack of respect for your colleagues;
lack of care and comments wastes their time.
It is unprofessional. Nothing to do with beautiful code.

Steadily Adding Value

After five years, the software is so s#?! that in the beginning
it took two days to add a feature
a feature of the same size now takes two months ...
so they write a brand new one that is as s#?! as the previous one
that will also be decommissioned in five years time.

Now the two teams are in a race.
The tiger team must build a new system that does everything that the old system does.
Not only that, they have to keep up with the changes that are continuously being
made to the old system.
Management will not replace the old system until the new system can do everything
that the old system does.
This race can go on for a very long time. I've seen it take 10 years.
And by the time it's done, the original members of the tiger team
are long gone, and the current members are demanding that the
new system be redesigned because it's such a mess.

The only way I can see to avoid this sort of fiasco is to
to have the discipline to always keep the code clean in the first place,
relentlessly refactoring as required every time you add a new feature.

Productive Partnerships

As soon as the button was pressed to mute NASA from our meeting,
the managers said "we have to make a management decision", said Boisjoly.

The general manager of Thiokol turned to his three senior managers and asked what they wanted to do.
Two agreed to go to a launch decision, one refused.
So he (the general manager) turns to him and said "take off your engineering hat and put on your management hat" --
and that’s exactly what happened, said Boisjoly.
He changed his hat and changed his vote, just thirty minutes after he was the one to give the recommendation not to launch.
I didn’t agree with one single statement made on the recommendations given by the managers.

The teleconference resumed and NASA heard that Thiokol had changed their mind and gave a recommendation to launch.
NASA did not ask why.

I went home, opened the door and didn’t say a word to my wife, added Boisjoly.
She asked me what was wrong and I told her
"oh nothing hunny, it was a great day, we just had a meeting to go launch tomorrow
and kill the astronauts, but outside of that it was a great day".

As indicated by the appalling "management decision" to launch the Challenger Space Shuttle,
the lack of a true partnership, built on trust, between technical folks and management can
have truly tragic consequences.

Under pressure we cut corners ...
No one wakes up in the morning and says "today I'm going to screw up,
today I am going to write the worst code I can possibly write,
I'm going to f#?! up with this company" ... I met some people who did that,
but normal people don't do that ... everyone is trying to do a good job.

The business asks how long is it going to take? ...
The pressure we keep talking about, the pressure we put on ourselves
... we think we don't have time, even when we have the power to go to the business
and tell them how long something is going to take!

Researchers (Bloom (1985), Bryan & Harter (1899), Hayes (1989), Simmon & Chase (1973)) have
shown it takes about ten years to develop expertise in any of a wide variety of areas,
including chess playing, music composition, telegraph operation, painting, piano playing,
swimming, tennis, and research in neuropsychology and topology. The key is deliberative
practice: not just doing it again and again, but challenging yourself with a task that
is just beyond your current ability, trying it, analyzing your performance while and
after doing it, and correcting any mistakes. Then repeat. And repeat again.
There appear to be no real shortcuts: even Mozart, who was a musical prodigy at age 4,
took 13 more years before he began to produce world-class music.

As indicated by Norvig above, it takes time and effort to become an expert, a true software craftsman.
There appear to be no real shortcuts.
With the current trend towards generalists in agile teams,
finding the time to become a craftsman can be problematic as you
find yourself constantly switching from one domain to another,
from one language to another.
This also affects code quality in that most code is being written by generalists, i.e. non-experts.
This will be the topic of the next installment in this series.