Software Engineering, Improv, Craft of Software Development

“Requirements” is a pre-agile construct based on our false belief that building software was somehow like the construction of buildings.

“Requirements” limits the development team into thinking the end product is fixed and that the team’s goal is to implement a specification.

Agile development teams expect features to change over time. Even with thorough user research and validation, the product will change as soon as we receive user feedback. Even during the implementation of a user story, an emergent conversation on the team might change its acceptance criteria.

In contrast, features are a construct of preferences. Many preferences may stay the same, some will change over time, and most are made up on the spot Lichtenstein and Slovic 2006.

A simple example

Let’s consider a simple and common feature request: “the user logs into the system with a username and password.” If we consider this request to be fixed, we might not consider alternative solutions.

Some websites support magic links. You don’t type in a password, the system emails you an authentication link.

Some websites require two-factor authentication.

Some websites rely on a third-party single sign-on such as Google, Facebook, or Okta.

Several of these solutions also attempt to solve the problem of users creating easy to guess or easy to break passwords.

Treating authentication as a “requirement” misses these opportunities.

System characteristics

How then do we deal with performance, scalability, availability, recoverability?

Pre-agile has the notion of “non-functional requirements” which describe system characteristics or product quality characteristics.

Performance

Before agile, I wrote my share of statements like “the web page should load in under 0.5 seconds.” In reflection, my non-functional “requirements” were arbitrary limitations on the system.

Here’s the typical workflow for agile software development.
* Conduct user research and validation to determine appropriate feature for the opportunity
* Build a minimal implementation of the feature
* Verify that the feature matches the opportunity, if not iterate
* Improve performance where needed
This is a variant of “Make it work, make it right, make it fast.”(http://wiki.c2.com/?PrematureOptimization)

Availability and Recoverability

Many of the system characteristics such as availability can influence Service Level Agreements. Pre-agile treats these as guarantees. If we build the system just right, we can guarantee that we will achieve a particular outcome. Agile treats these as goals. If we want to prove ourselves that the system handles disruptions, then we introduce disruptions much like Netflix Chaos Monkey. We keep iterating on the product and deployment process until we achieve the goal. Adding Chaos Monkey to a system could be considered a “non-functional story” or a chore, since it provides no user facing functionality yet improves the product’s quality.

Avoid pre-agile constructs

It can be hard to remove bias from pre-agile constructs. If you see yourself or your team using the following words, pause and reconsider your options:
* specifications,
* requirements,
* the system shall…

In Summary

Avoid limiting yourself with pre-agile concepts. Recognize the flexibility of features in building software.

Here’s an academic paper demonstrating the negative impacts of framing potential system functionality as “requirements.”

Pair programming is a complex practice requiring skill to master. Although Pair Programming has been around for 20 years and is well researched, some teams simply drop engineers onto a team expecting them to figure Pair Programming out.

As a practitioner, I am still discovering ways that I can improve the pair programming dynamic.

This article describes 1) several techniques that can improve your pair programming experience, 2) strategies on what to do when things get frustrating or go wrong, and 3) concludes with success stories of converting difficult pair-programming relationships into successful pair-programming relationships.

Like any relationship, such as friendships, family dynamics, dating, or marriage, when pairing works well, it is an amazing experience; when pairing falls apart, it can be frustrating and occasionally super hurtful. While unfortunate dynamics can happen while pairing, I have experienced the joy of finding each other in the rift.

Pairing exposes both our strengths and our weakness. How we handle our weaknesses may have a direct impact on the pairing relationship. One of us might not be comfortable exposing our weaknesses. If we are experiencing imposter syndrome, feeling that we are not qualified for the job we have, we might fear that our partner is going to find out that we are faking it.

I have been on a journey to be a better pair. While at Pivotal over 3.5 years, I have paired with ~39 pivots and ~ 20 client developers. I am grateful to each person who has helped me better understand myself and have helped me assemble and create these techniques. If you paired with me and I did not apply the technique, it may be because I learned it later.

Strategies for Success in Pair Programming

In my software development practice, I rely on several techniques for increasing the probability of success with a pairing relationship.

This article contains a toolbox of techniques. Feel free to pick a few to try out and then revisit the list in a few weeks.

Become invested in your pair’s success is the major take-away. When I am taking care of myself and looking out for my pair, an incredible friendship can form. Consider “what are my pairs strengths? And how do they map to my weaknesses?” and vice-versa. Have I always been invested in my pair’s success? No. My pairs probably have stories on how I could have improved in my pairing. Shifting from “what is in it for me” to “what is in it for us” enables new pairing dynamics.

When Pair Programming Goes Wrong / Weird

Since pairing is a relational practice relying on imperfect communication, misunderstandings and slights can emerge.

What could go wrong with pairing? Here are some situations that have bothered me.

You come across some code that is difficult to read. Your partner tells you not to refactor the code.

Your partner dominates the keyboard for most of the day.

Your partner does all the work without involving you.

Your partner tells you how horrible is the code that you helped work on the previous day.

You discuss a situation and agree to a plan, but then your partner starts doing his / her own thing anyway.

In this section, I list out several strategies for coping with a situation that is getting tense, de-escalating a situation that is tense, and how to re-engage with your partner after a tense situation.

There are several ways in which Sustainable Software Development has evolved from Extreme Programming (XP)

Sustainable Software Development

combines user centered design with agile software development. Product designers apply design thinking and user research to understand what product and features to build. Extreme Programming relies on an onsite customer to tell the developers what to build. This is problematic in two ways. Often an onsite customer is unavailable, impractical, or too expensive to have on a team. Onsite customers are not trained in design thinking and information architecture.

reduces the commitment level of the team to the work in progress, the stories that each developer is currently working on. Unlike Extreme Programming, the team does not commit to an amount of work to be done during a weekly cycle or iteration. There are no iterations in Sustainable Software Development.

focuses the technical practices on two main goals

caretaking the code

removing knowledge silos

Extreme Programming has many practices around improving code quality, but removing knowledge silos is not an explicit goal.

replaces predictions with focusing on “what is the minimal that we need to ship?” Planning Extreme Programming utilizes the team’s velocity and backlog to create burn down charts for forecasting the team’s progress. Like Extreme Programming, the team maintains a shippable product.

identifies and removes software development waste. Weekly retros identify wastes and the team removes wastes with action items and chores.

Both Sustainable Software Development and Extreme Programming deal with the tension of dates and features by asking “how do we reduce scope?” and “what should be done first?” In other words, developer happiness and sustainability is achieved by prioritizing the work.

This table compares Sustainable Software Development to Extreme Programming (XP).

Sustainable Software Development

Extreme Programming 2.0 (XP 1.0 names)

Balanced Team
pulls together various disciplines into one collaborating team. The team works together to accomplish a common goal.

Whole Team (Balanced Team)
Whole team is also known as cross-functional teams. Include people on the team with the skills needed for success. Beck points out that a sense of team fulfills an individual’s psychological needs. “We belong. We are in this together. We support each others’ work, growth, and learning.” [1]

Track One Practices

User research
includes writing interview guides (questions), interviewing users and synthesizing the results.

Real Customer InvolvementC (On-site customer)
Make the customer a part of the team. “The point of customer involvement is to reduce wasted effort by putting the people with the needs in direct contact with the people who can fill those needs” [1]. Even if a customer is busy, involving them whenever possible is preferred to not involving them at all. In this situation, the team needs to find alternative ways of validating features with users.
It is impractical to have an onsite customer.

Road Mapping
the product manager diagrams his or her long term vision for the product.

Quarterly Cycle (Planning Game, Small Releases)
Each quarter, the team identifies the stories for the next quarter. During a meeting, the team identifies bottlenecks, initiates repairs, plans the feature sets for the quarter, and selects a quarter’s worth of stories.

Code and TestsC
The tests and code are the only permanent artifacts. Everything else (e.g. documentation) is generated from the code. Anything that contributes to what the system does today and what it will do tomorrow is valuable, “everything else is waste” [1].

Fast test suite and builds
means that the tests execute as quickly as possible. A fast test suite provides speedy feedback to the team when something breaks.

Ten-minute Build
Building the code and running the test suite should be automated and short. Longer builds increase the feedback loop.

Test Driven Development / BDD
has the engineers incrementally build the software by always writing a failing test, then writing just enough code to get that test to pass.

Test-First Programming
The developers write tests before writing code. Every production line of code is preceded by a test. Test-first programming addresses several concerns: scope-creep, coupling and cohesion, trust, and rhythm. The practice mitigates against scope-creep; without a clear goal, a programmer can easily write more code than is necessary. The practice improves coupling and cohesion as easy-to-test code often has beneficial design qualities. The practice builds trust in the code as engineers trust tested code more than un-tested code. The practice produces rhythm as it provides a cadence of accomplishing goals.

Team code ownership is the ability for any developer on a team to change any of the team’s code.

Shared CodeC (Collective ownership)
Everyone on the team can modify any of the system. When engineers sees something broken, they fix it.

Continuous refactoring

Incremental Design (Simple Design Refactoring)
Build the system incrementally and evolve the design every day. In the past, software teams might often had long design phases that assumed features would not change. When the team’s understanding of the system changes, make incremental changes to the system to align the code’s design with the team’s desired goal. Teams routinely remove duplication.

Integrate frequently

Continuous Integration
When working on new features, routinely integrate new code into the master code repository. Avoid long-lived branches. Beck suggests un-integrated code live on its own “no more than a couple of hours” [1]. The longer the team waits to integrate, the more unpredictable and expensive integration becomes.Single Code BaseC
Ideally there is only one branch of the code. Multiple branches, (e.g. for releases, or customers) creates extra work. Branches live a few hours at most.
branches live a few hours at most

Frequent releases
means shipping or deploying the product frequently. For a website, “frequent” might mean several times a day; for enterprise software, “frequent” might mean once a month.
Each team releases when features are done or at a given cadence

Daily DeploymentC
Every day put a new version of the code into production.

Acceptance / staging environment
means that the product manager accepts a story in another environment that closely resembles the user’s environment. For installed software, this is another machine or cluster. For web development, this is a staging server.

(implied)

Build system
is for continuous integration or continuous delivery. The build monitor indicates the status of running all the tests with the latest delivered code.

(implied)

Stop on red builds
means that when a build fails (the build monitor turns red), one pair immediately stops what they are doing and fixes the issue. This is not necessarily the pair that caused the build to fail. Shaming whoever introduced the change to cause the build to break is antithetical to the goals of sustainable development. The occasional failing test is not a big deal, the team handles it and moves on.

(implied)

Remove Knowledge silos

Continuous pair programming
refers to two engineers co-creating source code and tests on pairing workstation with two displays, two keyboards, two mice, and identical development environments. Other models of pair programming exist, but this setup reinforces the equality of the relationship.

Pair Programming
Every part of a production system is written by two programmers sitting side by side. The pair works together to solve problems, keep each other on task, learn from each other, and abide by the team’s practices.

Overlapping pair rotations
means that, each day, engineers change pairs such that over time, everyone works with everyone else. One member of each pair always retains work in progress from the previous day.

N/A

Boundary Spanning Practices

Backlog grooming
the product manager routinely re-sequences the stories in the backlog and verifies that stories at the top of the backlog are ready for engineers to start.

N/A

Accepting stories
after the team delivers a story, the product manager verifies that the delivered work achieves the acceptance criteria. The product manager tries to break the feature with common unexpected input.

(implied)

Story showcase meeting
a 60 minute weekly meeting to discuss stories that the team will be working on in the immediate future. The product manager describes upcoming stories and the engineers provides feedback on the expected difficulty of each story. The story showcase helps improve the team’s shared understanding of the product.

Weekly Cycle (Iteration Planning Meetings)
Each week, the team accepts a certain amount of work to accomplish. During a meeting at the beginning of the week, the team reviews progress to date, the customer chooses a week’s worth of stories, and the engineers decompose the stories into tasks and individually estimate each task. “The goal is to have deployable software at the end of the week which everyone can celebrate as progress” [1].

Project Management Practices

Core work hours
means the team prioritizes working together in the same office during the same hours. Flexibility with the schedule helps with outside of work constraints provided that the team has enough time in common.

Energized Work (40-hr week)

Stand-up is a 5-10 minute, daily meeting where team members discuss changes that affect the team from the previous day’s work and impediments to the current day’s work. If physically possible, people stand to encourage a short check-in. Usually this meeting is at the start of the day, otherwise it is a daily spin-down meeting..

Present in http://www.extremeprogramming.org/, but not in the 1999 or 2004 books

Retro
a 60 minute weekly meeting in which the team re- flects on the week, celebrates successes, and identifies ways to improve. The ideal retro is a safe environment where a team can discuss any topic constructively.

N/A

Flexible scope contracts
establish an ongoing relationship between a development organization and a client organization without unnecessary details. It might state that the client owns the code, that new releases are expected at most every two weeks, or the daily rate at which engineers are billed. It typically would not state a fixed scope, schedule or budget. If a schedule or budget is fixed, scope is not and vice versa.

Negotiated Scope ContractC<
“Write contracts for software development that fix time, costs, and quality but call for an ongoing negotiation of the precise scope of the system.” [1]

No overtime
no one is allowed to work overtime except in genuinely exceptional circumstances, such as solving a customer escalation.

Energized Work (40-hr week)
“Work only as many hours as you can be productive and only as many hours as you can sustain” [1]. Working longer hours may make the team less productive. Team members need to take care of themselves; this includes staying home to rest when sick. The work patterns of the team should be sustainable in the long run.

Core Work Hours, Be Present

Sit Together
Software development is a collaborative endeavor. Co-locating teams increases collaboration. Distributed teams are still possible, yet “Sit Together predicts that the more face time you have, the more humane and productive the project” [1]

Informative Workspace
“An interested observer should be able to walk into the team space and get a general idea of how the project is going in fifteen seconds” [1]. The team can modify the workspace to make it more conducive for their work.

Not needed
The team does not commit to accompilshing certain stories each week.

Slack
During the weekly cycle meeting, include tasks in the plan that the team can drop if the team gets behind. Adding slack to the schedule enables a sustainable work environment.

Not needed for all projects

Incremental DeploymentC
Switch between a legacy system and a replacement system in incremental steps. Avoid a hard switch over from a legacy system to a new system.
Slowly replace a legacy systems.

Allocations
Move engineers where needed

Shrinking TeamsC
Try to improve team efficiency so that the team can shrink in size. When a team becomes more productive, re-allocated extraneous team members to other teams.Team ContinuityC
“Keep effective teams together” [1]. People are not fungible, and building relationships is part of creating a high performing team. Rotating people through teams helps spread knowledge.

Optional, depends on situation

Root-cause analysisC
When a defect occurs, determine why it happened. Modify testing strategy to prevent that kind of mistake from happening again. Write an automated system test and an automated unit test then fix the code.

Pay-per-useC
Pay-per-use provides alignment between revenue and the features that generate the revenue. “Connecting money flow directly to software development provides accurate, timely information with which to drive improvement” [1].

C are optional XP 2.0 Corollary Practices

For a team transitioning to Extreme Programming, Beck recommends starting with the primary practices. Beck claims that “the primary practices are useful independent of what else you are doing” resulting in an immediate improvement, whereas the corollary practices work well once a team has implemented the primary practices. Extreme programming is not dogmatic about the list of practices. The team chooses which practices to apply.

Once a team has implemented the primary practices, then the team can start implementing the corollary practices. Beck says that it seems “difficult or dangerous to implement [the corollary practices] before completing the preliminary work of the primary practices” [1].

This month I finished my Ph.D. I have used grounded theory for the last three years. I have found the method extremely rewarding. I loved interviewing, participant observation, and analyzing the data. There will be moments of confusion with the data. Embrace the confusion, on the other side tends to rewarding research results.

While at Pivotal, I have iterated on remote pair programming while working on two distributed teams. This shares what I have learned about remote pairing.

Best Practices

Periodically check in with your pair about the audio, video, and screen sharing quality. Fix issues promptly. If you are experiencing issues, bring them up. Quickly iterating is the key to improving the remote pairing experience.

Setup a video feed (zoom, appear.in, hangout, slack) that is always on and instant messaging (slack) in the secondary window. Much of communication is nonverbal. Seeing my partner excited or frustrated is super helpful.

Leave the video feed running during breaks. This allows each person to see what is happening at the other end. If my pair is in an office, I might see that my pair is talking to another member of the team.

In the office, leave the microphone open. This allows a remote person to feel connected to the office vibe when their pair is away from the desk. Osmatic communication can still happen.

When a team is distributed between two offices but pairing locally, setup a remote video feed. This allows either office to get the attention of the other office by waving frantically. (Often pairs do not check slack frequently while pairing.)

Use screenhero for screen sharing and zoom for video and audio. The audio in screenhero does not work as well as zoom or appear.in

Have all pairs join the same appear.in channel. Mute everyone but your pair. Whenever you need a team huddle, just enable everyone’s channel. This reduces the friction for team huddles.

Here the team is in one channel with multiple conversations. Note that the team has an extra camera setup to see what is happening in the aisle:

Ask for feedback about the development environment. While Pivotal uses workstation-setup for consistent development environments, the screen resolution difference between a laptop and an iMac makes that combination super tricky. I prefer iMac to iMac.

If you are working across timezones, have empathy and be mindful of break times and lunch time.

Equipment

Use a two display system. One screen is shared. The other is for supporting remote pairing.

My observations clearly indicate that team code ownership is a feeling that can be engendered, not a policy that the team adopts

Pivotal developers more acutely feel team code ownership when i) they understand the system context; ii) they have contributed to the code in question; iii) they perceive code quality as high; iv) they believe the product will satisfy user needs; and v) they perceive team cohesion as high.

The paper shows that diverse events and trends that can undermine sense of ownership

The paper replaces Beck’s definition of Collective Code Ownership (“anyone can change any piece of code in the system at any time”) by introducing the new term Team Code Ownership (“the ability for any developer on a team to change any of the team’s code”)