Tuesday, October 09, 2012

I decided my life needed a little bit more math, and so to decided to add a few differential equations in some F# code. Math in code means applied math and lots of arrays, but arrays are a pain functional style. So heres my appoach.

I have written many numerial applications, mostly in C++, yet also some in Fortran. I mention Fortran because it has the notion of equivalence, where you can map arrays on to each other. Equivalence is like C unions for array and there to reminds us is that arrays do not need to be managed as dynamic memory.

A functional style of programming is a lot about preserving your invariants with immutability, yet immutable vectors are not a solution (yet). Still, numerical code has its own invariants, and therefore some immutability is possible, and should be exploited to write robust code. In my case, I have vectors and matrices; Their sizes are invariant, and so is the structure of the matrices, they have lots of zeros, they are sparse. The approach is then to specialize a state monad to maintain size, and structure in an immutable manner, while allocating mutable vectors only within the core of the numerical engine.

Long ago I wrote an matrix assembly algorithm. You can see it as a sparse matrix structure inference code. It was really pretty simple: the final matrix is an assembly of many small sparse matrices, finding the final structure can be seen as solving a set of matrix structure equations, the algorithm defines an order of traversal for all this matrix points and then proceeds to "crawl" across all matrices at once, working out all the local structures. In a functional style of programming, I can use the same newVar like concept you would find in a constraint or inference solver, extend it with an equivalent newEq, to build a monadic environment to support my numerical code. The result is that "application code" just uses variable and defines equations, these are then assembled and only later does the code allocate just a few very large arrays to store the real data associated to the problem

This approach is "easy" because all invariants are built first and then used. Much modern applied math depends on hieararchical and adaptive grids. Hierarchy is probably easy to bring in to a functional style, adaptive grids maybe less obvious.

Wednesday, October 03, 2012

There is nothing special to greenfield projects yet teams that can
tackle greenfield project have something special! It is not the projects that
matter as much as the approach to resolve unknowns when they appear in the projects that matters. My goal here is to give you a few insights on how do this.

Greenfield projects, or greenfield areas of non-greenfield project, is
really about hitting the limits of what you know, it is about meeting unknowns. When
that happens, and it does happen decently often, you should not apply a different
process, but continue with the one you have, one that already has those “off road
tires” that allow you to move in greenfield pastures.This concept of “one process” fits them all
is really the hardest thing to get across to teams that have not really managed
to deepen their agility. Yet processes like Scrum works equally well on
greenfield projects than on non-greenfield projects.

Best explain thing in action, here is how it works. We know that
software development is a combination of different tasks. I’ll simplify
it to the following:

Requirements

Design

Code

Test

Deliverable

The most efficient way for a team to “develop” is follow the flow of dependencies
between these tasks.With this approach,
you start by writing requirements, then you think about the design, then you
code, you test and finally you deliver a software package. You can do this the agile
way, or the waterfall way. You can start
one phase before ending the previous, but you still follow the dependency order.
This works well because although there are cyclic constraints between the
different development phases, they are manageable. For example, requirements
(stories) are analyzed by the team before the sprint, then during the sprint
team members have their independence to pick up work as suites them best.
Another example, the team gets together and debates design, when all are
aligned each can continue with their code development.

A “non-greenfield” project has "manageable" interdependencies
between the different development tasks. By manageable I mean
predictable, and by predictable I mean that the team has enough common
knowledge that they can organize a process around their unknowns and resolve
them together (e.g. as in preparing your next scrum). While, to the contrary, greenfield projects and greenfield areas of a project are,
before all else, situations where it is simply harder for the team to efficiently pick up the development tasks. And the number one reason for this breakdown in
team productivity is that the tasks no longer are supported by sufficiently clear
dependencies, and this results in the team no longer have clear enough objectives when working on the development tasks, and finally too many tasks "just do not go somewhere fast enough"!

You may say: “Ok, so yes
sometime we do not know, that is when we can take the time to learn!” But if you do think that, do read on because you are wrong, projects are not the right place to "take time"! Here is the reasoning:

It is hard to notice what you do not know.

It is hard to learn what you do not know.

Having extra time to “observe” and learn will only help you so much

Let’s start with the first one, which leads me to greenfield project rule
#1:

Do not expect to recognize what you do not understand

What you do not understand is what is missing, what is unknown, and unknowns are mostly invisible and therefore hard to notice. That is because
they are unknown! (Duh… )

Many of us feel anxious when confronted with
unknowns and things we do not understand, yet in a team these feelings are hard to express and hard to use as
drivers that can be shared. In fact, in a team negative feelings may distance
people, which is the last thing you want. As anxious feelings do not support a
team well, we need a surer and easier way to detect the presence of “unknowns” that
are affecting development , we want an “unknown detector”. And we want some automatic way of using this detection
tool, so as not “miss” reacting to the team's lack of understanding. And we are in luck, becausenothing is better than an agile iterative
process to serve as your unknown detector!When you fail your development iteration, when your sprint does not
deliver, that is when you did not know what you thought you did!It could be that you quickly understood what you did not know before, for example that a sick team member is the cause of your troubles, but more often, failed deliverable
are an indication that you have unknowns and that your team as entered “the greenfield zone”…
(add here appropriate music).

Rule #2, for any development, is therefore very important (in scrum
language):

Track velocity, review your sprints truthfully

Velocity tracking and sprint reviews are your team’s unknown detector. The
more newness and unknowns you have the more precious they are in alerting the
whole team of unknown dangers. The next question is “what should you do when
the knowledge alarm bell rings”?As I
forewarned before, the last thing to do is to “go back to school”. It is not
the learning that is wrong, it is what you learn, how quickly you learn, how
productively you learn that is most often the problem. Or to put it another
way, with every development task you can be expected to learn something, the
more you learn the better, as long as you get your task done! In a
non-greenfield project, there are still lots of unknowns, but these get
resolved because your team has a process to conquer these unknowns. In a
greenfield project, unknowns accumulate, and when the team fails a sprint,
it is in fact the team’s unknown resolution process that is failing. Trying to fix
a process by reading up on things rarely does the job. First because, reading
is a personal thing, not a group process, then because learning from reading is
simply not going to happen fast enough.To keep your team going you need your learning to be part of the team process,
not external to it. This bring us to rule #3:

Drive learning within your process

Here is the idea, you are ending your sprint, things have not happened the
way you wanted them to happen, you have delivered half what you hoped you
could. Obviously something is wrong! The easy action is to do nothing about it,
just push the failed tasks into the next sprint and move on. Yet that is
absolutely the wrong thing to do! Each failure needs to bring a learning.The next level of easy action is to talk
about the failure in a sprint retrospective and add future action of
improvement list. Beep… Failed again. That list will not be consumed fast
enough, and no learning will happen.Now
we are ready for rule #4:

Realign your backlog to avoid your unknowns but still touch them

I’ll start by saying: “yes, I realize this sound a little bit crazy”.
Here is the reasoning:
One or more unknowns are keeping your team from achieving its highest
productivity, specific aspects of the sprint’s tasks have failed.Your mission, as a team, is to understand
what changes can be made to the backlog that will bring your productivity back
up AND move your project forwards as much as possible in the right direction.
The whole team’s goal is to move around these unknowns that have been blocking
you and yet to get as close as possible to them because in many cases that is
where the projects added value will be. The added value will be learning but also new deliverables that will make you think differently.

How then to extend your process to tackle these unknowns? The situation is that you have
a scrum process, you are normally able to tackle the deeper unknowns in the
pre-sprint story grooming and design discussion phases, but now you find yourself
starting your sprints neither with good stories, nor good design, nor a clear
idea of how you want to develop. Again you are in the greenfield territory.
This is when you have team members that make suggestions like: "we should use
domain specific languages", or "can we use the GPU for hardware acceleration"? This type of suggestions are
signs that your team is “uneasy”, that they are out of their comfort zone and are looking
for ways to feel better. As I mentioned above, detecting that subtlety of emotions in a team is not a process, but noting your sprint failure and drop in velocity is. The team needs then to take action, but given their current state of "failure" something different needs to be done. Yet I have already told you what you should do, you should revisit your backlog and rework it. The trick is how to make it systematic! This leads to rule #6:

In case of doubt, the product owner bridges the gaps, not the team

Again, you do not change your scrum process, neither do you change your product owner and his role. Yet the product owner's tactics are now somewhat reversed. Normally, it is the team that tries to meet the expectations of the product owner. In greenfield territory, the tables are turned, it is the product owner's task to meet the team's capabilities. The process is actually pretty simple: the product owner asks the team "can you deliver this" over and over, with different variants, until he has rebuilt the best of possible backlogs. Again, it is not the team that decides that it needs to learn, yet in their answer to the product owner they may say: "yes, we can deliver that, we will need to do some learning, yet we are pretty sure we can deliver what you are asking". This is scrum, so the double checking process is known to all: if the team's independent estimations (sizing) of a task are similar enough (and small enough!), then all have a good confidence that they can deliver that task.

In non-greenfield, the team is charge of delivering what it masters. In greenfield projects, it is the product owner responsibility to maintain a backlog that the team can master. Often this means that the product owner needs to be creative in formulating his desires to meet up with the teams capabilities. Here a few examples how such a process works, to make it obvious I'll imagine the product owner wants to build a car, but for software development, the process is no different:

Basic features

Example: Deliver a motor connected to wheels

Limits, boundaries

Example: Work out what is the minimum, maximum, and average weight, power, length, height, etc. of the 50 most popular cars on the market

Out of order

Example: Design a test track

Disconnected, but related

Example: Deliver the driver's seat design, the interior sizing, and the design of the windows

In each of these example, the product owner needed to pull back from his initial goal of getting a full car, each time he/she chooses a new team goal that moves the team in the right direction, delivering something of real value, but while staying in the comfort zone of the team. You may note that the out of order example takes the Test Driven Development approach, but out of order with a good product owner can be much more flexible than TDD, especially when the team has no idea of how to hook in the tests into an nonexistent design. For example, let's say the the team needs to develop a cloud app. They will need to use some new APIs, project's often have given technical constraints, in such a case, the product owner owner can focus on asking for the shell of an app that uses those APIs in order to at least tick off the boxes "code calls API", and "deliver code that works in the cloud". Nevertheless, the product owner should always ask that that the app does something which is line with his business. This is an implicit rule: always deliver some form of business value! Also note that the product owner, and the scrum process, continue to strive to deliver deliverables with enough size so that much of the whole team contributes. Fragmentation of knowledge, and new learning is a high risk in new domains and all should work together to keep the knowledge flowing.