Why Short Feedback Cycles Lead to Great Software

Sunday February 24, 2019
Software Development

I think most software developers love short feedback cycles, whether they
realize it or not. And it makes sense! Really short feedback cycles are one of
the first things most developers experience when they write their first “Hello,
world!” program. A lot of developers get hooked when they see that any change
they make to the code is reflected immediately in the output. This feedback
cycle is nearly instantaneous, and many developers love that about programming.

Unfortunately, most professional software developers don’t experience rapid
feedback like this in their day-to-day work. Actually, it seems like the length
of feedback cycles tends to increase as the size of an organization increases.
In a large enterprise with hundreds or thousands of developers, it’s possible
that you won’t see your change in production until months after you write it. In
contrast, if you’re working on a personal project or a very small team, it’s
likely that you can deploy a change to production in minutes.

These feedback cycles apply to many components of software development besides
time between making a code change and seeing that change in production. For
example, the amount of time it takes to run the test suite is likely to follow
a similar pattern. Smaller teams or personal projects are likely to have small
unit test suites that can run in a matter of seconds while large enterprises are
likely to have large unit test suites along with complex integration test
suites, UI tests, or end-to-end tests. These suites are likely to take much
longer to run, and in some cases take hours or days to complete.

What does the data say?

I spent a small part of my career teaching high school math, and I actually have
a master’s degree in education from the alternative licensure program I
completed. One of the things I learned while studying education is that short,
frequent feedback cycles are incredibly important for both learning and
motivation. For example, if the goal is to correct misconceptions in a student’s
understanding, it’s much more effective to have them do a quick problem and then
trade papers to check a classmate’s work than it is to have them turn in a quiz
that might not be graded and returned for a week or more. Getting feedback while
the problem is still fresh in their minds is critical, and
severalstudies
have shown that short, frequent feedback cycles keep people engaged and promote
learning and achievement.

Rapid Feedback in Practice

If the tendency is for organizations to slow down with longer and less frequent
feedback cycles as they grow, how do we fight that tendency? I have some ideas
about software engineering practices that can promote shorter and more frequent
feedback cycles to help engineers deliver higher-quality code more quickly.

Make small commits.

Commits are the primary way that we make changes to and get feedback from the
system we’re developing. A smaller commit makes every step in the change cycle
faster, while a larger commit usually makes every step slower. Smaller commits
will lead to more isolated unit test changes, faster code reviews, more targeted
manual testing or UI verification, and easier monitoring after deployment. All
of these steps can have an impact on feedback cycle time, so commits play a big
role in slowing or speeding up the cycle.

Respond to code review requests quickly.

If you are slow to respond to pull requests, you make the feedback cycle longer
for the person who wrote the code. It’s most effective for the code author to
receive and act on feedback while the code he wrote is still fresh in his mind.
Ideally, I think it’s reasonable to expect most code reviews in less than a day
(in a professional work environment). If your code takes longer to review than
that, you should probably make smaller commits (see the section above).

Ensure unit tests run quickly.

When I’m working on implementing a new feature or fixing a bug, I prefer to use
test driven development (TDD). If my unit test runs quickly (in seconds), it’s
very easy to make a minor change to the code and see how it affected the tests.
But if my tests are slow (even tens of seconds is too long), my work cycle slows
down significantly. In reaction to my slow tests, I start running them less
frequently and making more changes between test cycles, which ultimately adds
complexity to my development workflow. The same principle extends to entire unit
test suites and the feedback cycles they create on pull requests.

If your unit test suite isn’t this fast, what’s slowing it down? Oftentimes, the
slow components of a “unit” test are dependencies on external services like a
database, filesystem, or HTTP API. Generally speaking, it should be relatively
easy to mock these services and inject the mock into your unit test suite, which
would allow you to get the same assertions on your unit under test (the business
logic in the class you’re writing) without waiting for a real database, API, or
filesystem. If your unit tests are too slow, profile them, and see if it would
be possible to replace any of the slow components with a mock in the slowest tests.

Ensure integration and end-to-end tests run quickly.

Integration and end-to-end tests have a similar feedback cycle problem to unit
tests, but on a larger scale. With integration and end-to-end test suites, a
realistic goal is to have them run in minutes rather than hours or days. A slow
integration or end-to-end test suite ultimately leads to the same problems as a
slow unit test suite - slower suites are run less often, and as a result they
also run against bigger change sets.

If your integration or end-to-end test suite is slow, a common cause is trying
to test too much in integration layer. It should be possible to validate the
majority of your business logic in the unit test layer. And you should prefer to
do so because of the faster feedback cycle there. In the integration layer,
then, we’re left only with things that couldn’t be validated in the unit test
layer. A relatively small test suite should be able to validate that the
services are integrated properly. If your integration suite is too slow because
it covers a lot of business logic, consider if some of those tests could be
moved to the unit layer to make the integration suite faster, and to get faster
feedback on the assertions in those tests than the integration suite can
provide.

Implement a fast development and release cycle.

Feedback cycles don’t end when the code is merged and released. There’s a
also a feedback cycle from the production code, which provides information to
the development team through logs, monitoring tools, A/B testing, etc. It’s
important to make sure developers have easy access to these tools so they can
receive feedback from the code running in production. I’ve seen organizations
where it was exceptionally tedious to pull production logs, and business
processes like that make the production feedback loop nearly non-existent.

If developers are already getting good feedback from production, the goal
(again) is to shorten the feedback loop. Developers who get feedback from
production quickly will be able to detect and fix problems while the code is
still fresh in their minds, and while the change set is relatively small.
They’ll also be able to iterate more quickly - particularly if they’re using
something like feature flags to perform dark launches. A slow feedback cycle
with production has the same symptoms as a slow feedback loop on unit or
integration tests - changes become bigger, and it takes a longer time for
developers to know how their solution is performing. The best way to shorten the
production feedback cycle is to release more frequently. I think a goal of one
release per day is a good target for most organizations, but it’s not unheard of
to go much faster than that.

Shorten Your Cycles

Over the course of my career, I’ve seen time and again how slow feedback cycles
can be a burden to a development team, and I’ve seen how making improvements to
those feedback cycles can not only motivate the team, but also lead to more
stable releases and faster feature delivery. In Chuck Rossi’s
@Scale 2017 talk,
Rapid Release at Massive Scale (14:00),
he notes that faster iteration is worthwhile because it leads to tighter
coupling between developers and users. It’s worthwhile to examen your own
processes because there’s always some room for improvement.