Software Development: It’s all about shortening the feedback cycle

• Geoff Ellis
• Jan 11, 2021
• 9 min read

The past two decades have seen a dramatic transformation in software development practices. What began with Agile and Xtreme Programming led us to Lean and later DevOps, Site Reliability Engineering, and more. At Ally, our development teams embrace many of these concepts, and they guide many of our practices. While each of these concepts are independent and with their own unique focus, in many ways, there are common threads that run through them. One of these threads is reducing the time between having an idea or starting a task and validating the outcome of that idea or task — in other words, shortening the feedback cycle.

The purpose of this article is to connect the dots between the contemporary software development practices and to highlight this common thread. The teams at Ally leverage these practices to shorten the feedback cycle at every step of the software design, development, and engineering process. As more of these practices are adopted and combined, there is a multiplying factor that drives significant improvements in team effectiveness. By understanding how each of these practices plays a part in achieving the end goal of rapid and effective software development, it can become a roadmap for creating high-functioning teams.

Feedback Loops Within Teams

Broadly speaking, Agile methodologies are centered on fast feedback through timeboxed sprints; work is divided into short intervals with a review of the output at the end of each iteration. This ensures the team gets feedback quickly. Sprint demos (also referred to as show-and-tell sessions) can be used to gain fast feedback, but they are truly effective only if the team is encouraged to have a two-way conversation. More than just showcasing recent work, it allows the other team members to understand what changes were made, catch technical errors or potential conflicts, and provides an opportunity for business partners to review the output and confirm “yes, this is what I was expecting.” Since the intervals are kept small, the team gets feedback on their changes in small increments before too much effort has been invested. Any technical tweaks or feature changes can be quickly addressed, which limits the time taken in the wrong direction — reducing the feedback cycle as work is progressed.

Feedback Loops Between Developers

On a more technical level, practices introduced with Xtreme programming (XP) dramatically shorten the feedback cycle for coders. Test-driven development is a prime example: developers literally test that their code is achieving the required output as they are writing it, so they constantly get feedback on whether they are writing the right code. Red-Green-Refactor describes the behavior where the unit test is written first, the code is written and validated if the tests pass, and later the code can be perfected (through a revision process known as refactoring) knowing that the tests still validate working functionality.

Another fundamental component of XP is paired programming, a practice where two developers work side by side developing software as a pair: and in doing so solving problems together and giving each other feedback on their code.

My experience with paired programming and coaching teams through the transition shows there is a steep learning curve, but the end results are phenomenal. We may attribute this to ‘two heads being better than one,’ but the success is derived from the availability of immediate and relevant feedback. Who among us appreciates negative feedback after we have completed a task? Whether it be at work or at home — it can be quite deflating. Yet that is exactly how we approach the traditional code review: the Senior Developer or Tech Lead examines code that the developer has so proudly completed and poured so much energy into, and they criticize it. They pull it apart, demand changes, or re-work. Not only is the developer feeling frustrated, but they also must go back in time and remember exactly what the problem statement and solution was; this may occur many days or weeks after they wrote the code.

Paired programming reduces this friction; the beauty of the pair is that it effectively becomes a code review in real-time. The conversation and constant interplay between the developers shortens the feedback loop to zero, and in doing so improves the code and ensures basic mistakes are not made. Code reviews still may exist with the process, but their purpose shifts from catching basic issues. Of course, this is not the only benefit of XP, but continuous feedback is a strong factor.

Some organizations mandate paired programming 100% of the time. At Ally, our teams take a more nuanced approach and apply this technique where they feel it is valuable or necessary. Whatever your approach, avoid pushing out the code reviews to the end of the sprint — instead, use all available techniques to shorten the feedback loop as much as possible and provide the developer with quick feedback on their code.

Feedback Loops for the Codebase

The next important practice is Continuous Integration, with the purpose of reducing the cycle time between a developer completing their changes and finding out whether they work in the codebase at large.

The process of merging code and eliminating all the conflicts is usually considered a significant event. In a traditional team, a developer may have completed many days of work on their local machine before they ‘commit’ their changes, and only then do they find out if these changes overlap, conflict, or otherwise impact changes being made concurrently by their peers. Feature branches are a convenience that seemed the perfect solution at the time but is not conducive to the quick and constant flow of changes through the development pipeline.

Moving to trunk-based development eliminates the delay: developers make small code changes and quickly check them back into the main code base (the trunk) instead of keeping them separate (in a branch). Meanwhile, tools are in place to ensure concurrent changes are reconciled every time code is checked in. Not only does this give the developer much faster feedback on their changes, but it keeps all changes in sync across the whole team.

It should be noted that successful trunk-based development is made possible through many other corresponding changes, such as test automation and the availability of a rapid frequent and automated build process; otherwise, developers would forever be held up waiting for the new build to be generated or would be stopped in their tracks by someone breaking the build with untested code.

Feedback Loops for Managing Quality

In fact, test automation is another key component in reducing feedback loops. Manual testing takes time and can only be repeated so often. Even if test scripts are well documented, it’s simply not feasible to have manual QA testers perform the same tests again and again over a given time interval.

For many early adopters of Agile, teams struggled to complete both development and testing within the sprint timeline. Inevitably, in most organizations new to Agile, QA validation gets pushed to end of the sprint with manual tests of the new changes followed by a regression test of the existing code. Teams are directed not to let testing fall behind or carry over into the next sprint — the resulting pressure is aptly described by the metaphor of standing at the conveyor belt trying to keep up with the flow of work.

Those teams were still effectively working a mini-waterfall process of 2-week intervals, and the net result was that defects were only found at the end of the 2-week cycle, long after the developer had completed the code and moved on to other things. The feedback cycle was simply too long — by the time the feedback was received, it forced them to go back, revisit the code, and try to fix it after the fact.

Test automation allows the team to run tests multiple times a day, either on-demand or triggered whenever new code is checked in. The test automation platform never gets tired, it simply executes the tests and provides instant feedback to the developer. It’s not magic; it needs a huge amount of effort to create and maintain an effective test automation suite — but the impact is worth the effort. With the right structure and balance of resources, the test automation engineers can be updating and verifying the test automation suite at the same time that the developer is writing the code, and they work together to ensure that the code is functioning correctly (and the test scripts are working correctly) before they move on to the next task — team members collaborating and getting continuous feedback on their work.

Feedback Loops for Production Code

The practices discussed so far help the teams plan work in small chunks, develop high quality code through constant peer review, and then get fast feedback and verification that the newly developed code changes are working as intended: both in isolation (unit tests) and within the application (or system) as a whole (end-to-end tests).

The next hurdle is getting that code into production so that we can verify that it really is working with real users. Remember, features only add value once they are in production, and the faster we can get new features in the hands of real users the faster we can gather real feedback.

Legacy thinking told us that moving code to production is risky, so processes evolved with the express intent of protecting production. If most incidents or outages were tied to production changes, engineering practices were built to limit the frequency of any production changes — in other words hold back multiple weeks of development into large and infrequent deployments, and make sure that those large changes are thoroughly tested.

But, if you step back for a minute, that whole premise is contradictory, and serves only to create an arms race of larger, more complex changes requiring more thorough testing and validation. The longer you wait to deploy, the more work gets included; the changes become more significant, and more time is required to fully test.

The same thinking also led to the practice of scheduling deployments at night during system downtime; changes need to be assembled, executed, and validated before the system can be deemed safe to bring back online. This involves multiple people spending nighttime hours deploying and possibly fixing code — which limits how frequently those same team members can be asked to repeat the same exercise, and in effect influences the team towards less frequent deploys.

Organizations are now moving in the opposite direction — make the changes to production so small and inconsequential that the risk and impact of the change is reduced. This has the added benefit that testing can be easier and simpler, which makes it more suitable for automated test validation. Plus if those changes can be made seamlessly without needing to take the application offline, now the change can be made during regular hours, without stress, and with the full team available to review and validate the change. In this scenario, Agile teams now have much more freedom to deploy code more frequently and truly accelerate the feedback cycle. This is a significant and far-reaching transition in itself and requires organizations to reimagine their traditional change review processes so that it is less focused on formal approval gates and more focused on creating guardrails.

This is where modern deployment management techniques can help. Leveraging techniques such as canary deploys, traffic splitting, A/B testing, and feature flags, teams can push features live and validate the new code on a select, limited number of end-users (minimizing the risk), and quickly turn off the feature or reverse the change if any issues occur (minimizing the impact).

DevSecOps Feedback Loops

Detecting and reacting to production issues is the next area that benefits from reducing the feedback cycle and is one of the enablers for a true DevOps approach. The starting point is robust application monitoring that checks and alerts for system errors, but the true goal is not just to verify the system is running, but rather to validate that the application is functioning correctly for end users.

Monitoring provides rapid feedback to the development team who are encouraged to react to that feedback with improvements or changes to the application. In the past, a system alert would be handled by the system team — reboot the server or increase memory allocations. But feedback would not be passed to the development team, and rarely would it lead to follow-on work such as refining and refactoring the code to avoid similar future outcomes.

We still need to protect production and provide a reliable experience for our users. Speed cannot be at the expense of quality or stability. But successfully reducing the feedback loop at every point actually enables our teams to work faster and appropriately balance risk.

Putting it All Together

In summary, all of these points are interconnected. Success means addressing pieces of this puzzle from multiple angles, at the same time. Agile teams at Ally are measured on each of these areas and they constantly evaluate and improve these practices. The end goal is very much dependent on getting all of the smaller levers in place. Letting things drift in any one of these areas can negatively impact the end result.

The results are impressive when it all falls into place — teams at Ally thrive in an environment where they receive feedback in a timely manner at every step of the development cycle.