Release Modernization: How We Got Our Weekends Back, One Feature at a Time

• Jeremy Cox & Nate Gray
• Dec 12, 2024
• 4 min read

In a world of constant change, technology is ever evolving. Several years ago, when Ally Financial recognized the need to innovate to keep up with the rapid pace of digital transformation, our Digital engineering and product teams in Ally Tech were empowered to drive much of that innovation. Leadership's belief was simple: prioritizing employees—by ensuring engineers have the right tools to do the job while reducing burnout—leads to better outcomes for our customers. With this vision, Ally embarked on a journey to modernize its software delivery process, focusing on Agile delivery, DevOps best practices, and feature management. 

Laying the Groundwork: Agile and DevOps  

Our initial approach required evolving from Waterfall to Agile on the delivery side to focus on incrementally delivering value for our customers and stakeholders, while enabling us to change course along the way with each lesson we learned. And on the tech side, our focus was on breaking up large monolithic applications into micro-services, modernizing our data centers, and embracing cloud computing. These changes laid the groundwork for increasing developer productivity and maintaining high system performance, but also unveiled brand new challenges we had to overcome.  

Addressing the Next Challenge: Independent and Incremental Value Delivery  

As Ally scaled its teams for Agile delivery and built more features as the company itself grew, the complexity and coordination required for releases grew along with it. Scheduled overnight and weekend releases were becoming unsustainable as we started improving productivity and increasing velocity—development and operations teams can’t work 40+ hours during the week and then also be expected to give up nights and weekends on top of it—that’s a great way to burnout your best engineers and lose your top talent.  

Finding Our Way: Collaborating Externally  

To deliver changes faster and more safely, Ally’s Digital Engineering team began experimenting with feature flags for its Web and Mobile applications using LaunchDarkly in 2020. The use of feature flags as a core tech capability opened the door to separating releases from deployments—allowing us to selectively release code, experiment with A/B testing, and toggle features on and off without the risks and ceremonies that come with deploying new code into production with every change.  

This became a key piece of the overall puzzle we were trying to solve, and one that immediately helped us serve our customers better. The very first example of this was in March of 2020, when COVID was in full effect, we had a feature that was about to go out to production for customers to verify their employment status. However, we were sensitive about the fact that we were asking customers to verify their employer when many of our customers were experiencing employment instability as so many companies were struggling to stay afloat. Using our very first feature flag, we were able to simply turn off the feature without extensive rebuilds or testing, demonstrating the benefits of separating deployment from releases. Four years later, nearly all our new features are being built behind a feature flag. 

Building Momentum: Modernizing the Release Process  

In addition to making feature flag management a key component of our software delivery strategy, Ally has also taken steps to decouple our Web and Mobile UIs from our APIs by moving towards microservices and even a micro-frontend architecture. By decoupling UIs from APIs, we could adopt an API first approach that allows API teams to deliver new and updated services to production without a dependency on the UI development, while the UI teams can use feature flags to point to the new API version on their own schedule. This shift has facilitated independent releases and feature-level rollbacks, significantly reducing risks to our customers and release-related downtime. And by eliminating downtime during releases, we no longer had to wait until the middle of the night or work all weekend to release new features for our customers.  

Breakthroughs: Seamless Feature Rollouts 

Implementing feature flags across web and mobile platforms has allowed Ally to deploy features to each channel on its unique schedule, while giving our feature teams full control over employee pilots and canary rollouts, until they are ready to enable their features for all customers simultaneously once all the components have been independently deployed and validated in Production.  

Results: Transforming the Way We Deliver Features  

Our journey was driven by several guiding principles for release management: providing feature teams a well-defined, easy-to-follow path for production deployments; granting application teams autonomy to move at their own pace; and ensuring each component of a feature is independently releasable. The results have been impressive, and something this team is incredibly proud of—over the past four years there has been more than a 75% reduction in overnight and weekend releases, all while enabling a 275% increase in production deployments.  

Continuous Learning and Future Steps 

We still have plenty to learn—Ally Tech is constantly looking for new and innovative ways to keep improving how we deliver value to our customers, how we manage our change-related risks, and how we give our engineering teams a better work-life balance. Looking ahead, we have plans to continue to increase federation and team independence, implement release and rollback automation, and expand our experimentation and personalization capabilities that can be built off the strong technical foundation and culture we’ve established.