Agile Transformation initiatives need not originate at the senior management level. By examining the successes and failures along the journey of a mission-critical system for the United States Federal Government, this experience report discusses how the business and IT established a true partnership to effect organizational change from the grassroots level. We demonstrate the importance of creating a culture of transparency, experimentation and trust towards achieving organizational agility, especially in highly structured and bureaucratic organizations, and provide insights that other organizations may use to jumpstart their agile transformation initiatives.
1. INTRODUCTION
“I noticed G-REX is important but it sure isn’t winning popularity awards with the users,” I, Ankur, joked with Colleen having just been appointed to manage software development for G-REX. “They hate it! All of it!” I, Colleen, replied back. It was late 2016 and we were talking about the GSA Real Estate Exchange (G-REX) that had quickly become a perennial source of woes for me, Colleen, since I had inherited the responsibilities for G-REX in 2015 for National Office of Leasing. The modernization effort intended to bring the legacy leasing system current with times had resulted in a system that was out of sync with how users went about their work. Errors and performance degradation were rampant, basic necessities of work, e.g. revising information, needed help desk intervention, and response times from the help desk were less than stellar. While everyone agreed that things needed to get better there was little consensus on how to get to a better place.
I, Colleen, served as a program manager with the General Services Administration (GSA), Office of Leasing responsible for leasing real estate for the United States Federal Government. I started as a leasing specialist, and a G-REX user, in Philadelphia before transitioning, 5 years later, into a business line project management role for G-REX in 2015. During my time working on the G-REX system, I was exposed to the concept of Agile software development and educated myself on the methodologies employed in the industry. Having experienced years of unsuccessful waterfall development I was eager to implement Agile on G-REX.
I, Ankur, on the other hand, joined GSA from the private sector in 2016. With my eclectic background of working in roles that supported application development, infrastructure operations, IT architecture and pre-sales, primarily delivering projects to the federal government as a contractor, I was well-positioned to serve as a GSA insider with deep insights into the vendor community supporting GSA. Having an outsider’s perspective to some of the long-standing problems that plagued G-REX worked to my advantage.
Despite our very different backgrounds, we shared the sentiment that status quo was not an option. G-REX was failing its primary users and something needed to be done to alleviate some of the pain for the users while putting in a long-term plan for success in place. This shared understanding laid the foundation for a relationship where I, Ankur, relied on Colleen to define success for G-REX and Colleen could depend on me to deliver a technology solution that materialized that success.
2. Background
G-REX, developed in 2014 as a replacement for a legacy system eLease, is a mission critical system, built on Appian Low-Code Platform, used to manage GSA’s lease portfolio. It serves as the system of record for the procurement of leases in the federal government, supporting a real estate portfolio valued at approximately $2.7 billion in annual rent billing through 77 million rentable square feet across 8,000 leased assets. Approximately 2,500 leasing practitioners, ranging from novice to expert, who are geographically dispersed across the continental United States rely on the application to accomplish their duties on a daily basis in support of about 3,300 active projects at any given time in the G-REX application. The mission-critical nature, as well as the revenue impact of the application makes the development projects and operations associated with G-REX highly visible.
In economic terms, G-REX presents a perfect monopolistic market structure – a consumer base that is unique and a product that is specific to them and mandated for use by policy. However, after a year or so of being subjected to bad experiences, and by the time I, Colleen, formally assumed responsibilities for the system in 2015, the users had started to revolt. In the summer of 2015, we solicited system feedback via group meetings with regional users and an assessment was conducted by GSA-IT to determine the viability of the current G-REX application. That analysis uncovered a number of design and technical issues that, if addressed, would decrease technical breaks and increase flexibility, usability and user adoption.
G-REX implemented the leasing process as a set of rigid and linear business processes that required the user to conduct his business in a predefined sequence of actions. However, this was very disconnected with how the leasing practitioners operated. This was the biggest contributor to the bad experience users were reporting. There was news of spreadsheets and local drives being used to circumvent the system, complaints being made as high up in the reporting chain as possible, and a general sense of discontent all around. Status quo was not an option but change in a highly structured environment, like GSA, is especially difficult.
Software development at GSA is a careful dance between the lines of business and IT, which adds another layer of complexity. The lines of business are the primary consumers of the application and source of funding for all IT initiatives. IT, on the other hand, procures all technology products and services and is responsible for delivering solutions that are fit-for-purpose and fit-for-use. While the lines of business are intimately aware of the user needs, the technology needs that make the wheels go around are not as transparent to them. Hence, every change to a system is a negotiation or at least coordination between the customer paying for the system and associated support services and the organization providing the development services.
With that said, there were significant developments in favor of software development. First, desperate times had caused Colleen to work pretty closely together with the user community and the IT personnel, which ensured that the voice of the customer was woven into any future initiatives. Secondly, there was an organizational push towards agile. Agile was enough of a buzzword in the industry to have a cool tag associated with it. So by the time I, Ankur, walked in to GSA in late 2016, every organization, IT or otherwise, were sending their staff to internal 2-day agile training classes. In addition, you could avail agile coaching services from the Office of the Chief Technology Officer (CTO) and the leader for every IT organization had a line item on their performance plan that was related to implementing agile practices. As it happens with most agile transformations, most teams were now working in sprints and JIRA or Trello instances were popping up. Call it serendipity or whatever but it was nearly certain that no one was going to question or frown upon us for suggesting we work in an agile manner.
3. Our Story
The source of woes for G-REX was not very different from the systems that become the subject of horror stories. G-REX did not provide an efficient means for the users to perform their tasks. It was a classic case of picking a platform and maximizing utilization of platform capabilities with the assumption that the user base will fall in love with it – because it is new. Well, that was not to be.
3.1 The G-REX Front-line
Although well intentioned, the standardization of the leasing process rubbed a lot of people the wrong way. Users in different regions performed the work with nuances unique to each region. Codified constraints did not account for those nuances and users felt that the headquarters-developed processes boxed them in. One could feel the headquarters vs. the regions sentiment brewing organizationally. In addition, developed features fell well short of user expectations and improvements were deemed costly and difficult to make. “This is the worst application that I am forced to work with. Terribly designed, not functional, and the help desk is not helpful at all for it. This should be junked as soon as possible. Even no leasing application is better than G-REX,” noted a user when asked to provide feedback in 2016 to summarize the general views around G-REX.
The technical solution was on shaky grounds itself. The process to procure and award a federal lease may span 3-4 years on average, and involves hundreds of steps. While implementing a process automation solution made sense, the architecture and interaction of the various processes were critical to the success of the system. As designed, G-REX featured linear and long-running processes that did not mimic the actual workflow from a user’s standpoint. On the contrary, the processes added constraints that limited users’ ability to go back and forth between steps – something which was a necessity of the job. Under the hood, the Appian platform, by design, kept all running processes in memory, causing capacity constraints and bottlenecks as utilization increased. If that was not enough, in late 2016, GSA was made aware that the Appian Platform had undergone a major transformation and the platform version leveraged by G-REX was scheduled to go end-of-life in 2 years.
Users weren’t the only ones frustrated. By late 2016, I, Colleen, was at wits end. Over the course of 4 years the inability to collaborate closely with the development team led to misinterpreted or missed requirements in production and numerous costly change orders that would alleviate but not entirely address system issues. The constant complaints of 2,500 users were too much to bear for me and constantly requesting funding was becoming increasingly embarrassing for me. On the other hand, I, Ankur, had only been at for a few weeks GSA but they were already enough to see my fair share of dysfunction. There was no visibility into work. The development team claimed to work in an agile manner but only to deliver code with varying standards for End User Testing in one large batch in accordance with a Functional Requirements Document. I realized that a beautiful waterfall had been named Agile. There was a massive communication overhead between the line of business, development team, and operations teams. Delays were rampant, and ‘It works on my machine’ card was played more during every deployment than a Vegas poker table. In addition, the existing development contract for G-REX was coming to an end in early 2017 and there was uncertainty in terms of how the new support contract would be structured, whether there would be a new vendor, and whether the support cost would increase.
By the time 2017 rolled around, we were looking at: (a) a discontented user population that felt unsupported; (b) massive technical debt in the existing system; (c) a platform that was soon to go end-of-life; and (d) recompetition of the support contract. This called for a strategy to balance the immediate needs of the users with the long-term needs of sustainment, and more importantly a need for more funding. With that said, with this great challenge and uncertainty came a great opportunity.
3.2 A Three-Pronged Approach to Agility
The multidimensional problems needed a problem-solving approach that addressed all aspects of the problems while being considerate of the time sensitivity that accompanied each issue. October 2016 through September 2017 represented 12 months of great change for G-REX.
Figure 1. Summary of Improvements
We, with the support of other business line and IT stakeholders, established a way of doing things for G-REX, illustrated in Figure 1, that not only yielded significant results but later became a guiding light for other systems at PB-ITS to improve software development practices. This was our foray into improving customer experience using practices rooted in the human-centered design (HCD) framework without labeling it as such. We conjured up this methodology to optimize a series of interrelated feedback loops, from product conceptualization through production deployment, by focusing on interactions between the various stakeholders to deliver a customer-centric version of G-REX. As described in the following sections, the three-pronged approach included: (1) getting the right people involved at the right time through the Product Organization and Change Control Board; (2) ensuring acquisition strategy aligned with the expected outcome; and (3) modifying the software development processes to deliver results.
3.2.1 The Prototype, Product Organization and Change Control Board
I, Colleen, had learned from the past that Office of Leasing leadership, G-REX users and myself would need reasonable assurance of successful adoption prior to committing significant funding towards redesigning G-REX and undoing the mistakes of the past. “What if we can build something quickly? Something simple but representative of what the new system would look like,” I had asked Paul Done, Ankur’s predecessor in PB-ITS. And just like that, out of a simple conversation the idea of developing a prototype for modernized G-REX was born. But getting funding approved for an effort to develop non-production capabilities is not an easy feat to achieve, especially in the Government where funds represent taxpayer dollars. I spent a good part of the second quarter of 2016 preparing a strong business case for the National Office of Leasing leadership. I needed the leadership to fund a 6-month effort to test not only the Appian product but also the organizational ability to participate in and dedicate the appropriate resources for iterative development of a new system. The cost-benefit analysis of continuing on the existing path spoke for itself. The amount of money expended to continuously correct the existing application far exceeded the expense of this short term trial. I also recognized that developing a system aligned with user needs required leasing professionals, with divergent viewpoints, to actively engage with the developers and provide feedback as things are being built. “We have to work in partnership with IT and it has to be a full-time job” I explained to the leadership. As a result of these conversations, National Office of Leasing established the Center for Lease Technology and Innovation that included experienced leasing practitioners, including myself, who were dedicated to serve as the voice of the customer for software development initiatives. I was to mentor and a guide the new Product Owners and mature this product organization over time.
For the prototype itself, leveraging a scrum-based development approach, I, Colleen, would act as the Product Owner supported by a Scrum master, Business Analyst and an Agile Coach. Paul Done would serve as the IT Project Manager, and help contract for a lean team of developers dedicated to developing the prototype.
However, the biggest challenge I faced was to unite the customer needs from the 11 GSA regions and ensure everyone felt heard and represented. To overcome this challenge, I established a focus group representative of all regions that would meet on a bi-weekly basis to define and refine the user needs, attend product demonstrations from development team, and provide feedback throughout the development of the prototype. This group was referred to as the Change Control Board (CCB) and would later prove to be a critical element in managing the change associated with modernizing G-REX. Through the course of prototype development, not only were we able to prove that we could build a successful system on the Appian platform but we now had a practical experiment to prove that we were capable of implementing and resourcing an agile project, if we tweaked some of our internal processes and wrote contracts to support iterative development.
3.2.2 Restructured Contract and a Dedicated Helpdesk
By the time I, Ankur, joined GSA in October 2016 and took over the reins of G-REX from Paul Done, preparations were already underway to award a new contract in March 2017. I was quickly frustrated by being handcuffed by the existing Operations and Maintenance (O&M) contract that basically allowed no changes to the system other than bug fixes. Colleen was equally frustrated by me repeatedly asking her to secure additional funding for providing a user experience that the user base was craving for since the initial launch of G-REX in 2014. I and Colleen both worked with Paul Done to craft a Statement of Work (SOW) that would alleviate the pain of the existing contract and be more conducive to Agile.
A shift in mindset was needed. Rather than considering software to be in a ‘Done’ state, the new approach considered software to be a solution to a business problem with the constraints as they existed at a certain point of time. If the conditions changed then the solution would have to adapt. Therefore, under the new approach, all maintenance tasks – preventive, corrective, adaptive or perfective would be covered under the Operations and Maintenance (O&M) contract. There would be a dedicated team for O&M, i.e. fix issues and bugs, upgrade software, make changes to support changes in regulations and business rules and/or fine tune functionality. There would be no additional charge for these. Additional team(s) would be onboarded for any major enhancements that need to be made to the system or for modernizing the existing application to better align with the user needs and getting off the soon-to-be end-of-life platform. Both, I and Colleen, agreed that this was a good approach for alleviating immediate pain for the users while positioning for a better modernized version of the application.
Learning from the challenges encountered with the previous contract, I and Colleen were diligent about ensuring transparency and a collaborative approach. We ensured that the SOW explicitly called for using an agile software development approach, where the contractor worked in small batches and provided multiple opportunities for the product owners to provide feedback prior to capabilities being released for general use. Also, to ensure transparency that was lacking earlier, we mandated use of a Government-owned central repository for tracking in-process work, including user stories, which would be accessible to all team members, Government or contractors, alike.
Another feature of the new contract was the establishment of a dedicated help desk for G-REX users to provide a single point of contact for all their issues related to G-REX. Initially, this was a contentious move. While establishing a dedicated support line made sense from the standpoint of enhancing customer experience, I, Ankur, believed that the cost was somewhat high and the helpdesk was always likely to have excess capacity given the historical volume of contacts and the personnel required to maintain coverage throughout the day. I and the IT organization, felt that there was an opportunity to improve the support processes of the existing shared service prior to making this investment. I, Colleen, and the National Office of Leasing stakeholders, on the other hand, felt strongly about having a dedicated helpdesk and had funding approved for this initiative. Everyone agreed that having a dedicated helpdesk would have an immediate positive impact on customer experience when requesting support though. Given the availability of funds, a dedicated helpdesk for G-REX was established in March 2017.
3.2.3 Software Development Mechanics
Figure 2. Delivery Streams at PB-ITS
The recompetition for the G-REX contract resulted in the development contract being awarded to a new vendor, which was a blessing in disguise. The whole contract could start afresh as there were no lingering references to the past, and no people on the team carrying the baggage of the past. The resistance to change had been lowered by a great degree.
One of the very first things I, Ankur, did with the incoming development team was to map out the various delivery streams that put capabilities into production. There were three distinct streams: Planned maintenance releases, Unplanned eFixes or emergency releases, and Planned major releases for enhancements. The new O&M team was primarily focused on the first two but could deliver planned major releases when it came to software upgrades or integration with other systems/components. Each of these delivery streams required careful coordination with the other IT support groups, namely TechOps for deployment activities, Independent Verification and Validation (IV&V) for independent testing of the application, and Security Operations for scanning the application for vulnerabilities and security requirements. And since this time around everyone wanted to do things right, Colleen and her Product Organization became a critical stakeholder. The goal was to agree upon a schedule of activities that could be executed repetitively over a period of time. This period would constitute a sprint, and would help establish a predictable cadence for the benefit of everyone involved.
A 2-week sprint schedule was determined to be the best fit for O&M releases with a Production deployment every 2 sprints. This schedule was made available to all stakeholders using a shared calendar so everyone could plan for what is coming down the pike without waiting for emails and phone calls.
Then came the hard part: developing trust. Everyone in the Government or in the Government contracting business can recite a story or few about how either they got burned by the other party on a contract or how they were taken for a ride. Having worked for over 15 years as a Government contractor, I, Ankur, knew that it was important to address the trust issues up front. Therefore, as the new vendor was transitioning in, I emphasized creating a few norms of working. First, and most important, was to clearly establish that the contract was a partnership and success or failure was going to be a shared outcome. The development team would not be winning awards if I and Colleen could not deliver a worthy product to the user, and it was highly unlikely that Colleen and I would be lauded for our work if the development team was struggling. Second, and not too far behind, was establishing clear roles and responsibilities. Colleen’s role on the project was to represent the user base and bring leasing expertise to the table. The development team was there for their software development expertise. And I, Ankur, was to ensure that the team was not heading down an undesirable path and had everything they needed for the work to keep flowing. Anyone could provide input outside their realm but the authority of making the decision was with the person closest to doing the work. Finally, it was essential to make delivery of bad news easy. “If you run into an issue and engage me early on, I am sure I, being a Government employee, have access to a lot more resources that can help you solve the issue a lot faster,” I, Ankur, said to the development team to encourage open communication between the team. This last bit would be put to test not too long after. Within the first couple of months of the contract, a developer accidentally deleted a configuration file in Production environment while deploying a release, which obviously caused issues. However, the developer engaged me within 2 minutes of realizing the mistake, and I was able to coordinate efforts between all other parties relevant to the matter while the developer focused on fixing the issue. The whole matter was resolved within 10 minutes, and the only consequence to this episode was a tweak to the process that decoupled application configuration from code deployment.
3.3 Results
The efforts to jumpstart agile yielded a lot of positive results, some immediate but a lot more long-term. The G-REX help desk had an immediate impact on customer experience, as expected.
Figure 3. Improved Resolution Time for G-REX Support Tickets as a Result of Dedicated G-REX Helpdesk
As shown in Figure 3, over 95 percent of the issues reported for G-REX were now being resolved within 5 days, with the average resolution time of 1.2 days. This was not the case prior to March 2017 when it was not unusual to have tickets open that were over 30, 60 or even 120 days. While the helpdesk served the immediate needs of the customers, the CCB provided them with a platform to contribute towards the long-term modernization of the application. The ideas and suggestions were now being funneled from all GSA regions in the CCB meetings, debated in presence of the leasing community, and then prioritized in terms of whether they could be implemented immediately with the help of the O&M team or could be ingested into the modernization backlog. Such was the beauty of this new contract structure that items which had been considered separately-priced enhancements previously could now be delivered under preventive, adaptive or perfective maintenance with no additional cost or contracting overhead. Colleen just needed to adjust the prioritization of the backlog and decide, with the help of the CCB and her Product Organization. In August 2017, when an unanticipated regulation change required a new Disaster Leasing module to be implemented to support the new regulatory requirement, the G-REX team was able to deliver the solution within 60 days of the regulation change. Over time, G-REX would boast having one of the fastest times-to-market in the GSA application portfolio while being one of the most reliable applications.
Every year GSA IT conducts a Business Value Survey for the application users in the lines of business to rate the applications in six areas, as shown in Figure 4, that contribute to an Overall Business Value Score (BVS) – on a scale of 0 to 5, 0 being least effective and 5 being highly effective. It was no surprise that the G-REX scores showed a significant increase, the highest increase among PB-ITS applications, in all categories in Fiscal Year 2018. The quick wins gained resulted in establishment of credibility and trust with the stakeholders, and allowed me and Colleen to further experiment with the contracting structure andthe software development process. We would go on to deliver the modernized version of G-REX in March 2019, which would completely revamp the user experience for leasing professionals, and in August 2019 deliver Lease Offer Platform enhancement at 45% of the projected cost using a modular contracting approach. As of 2021, G-REX is the highest rated application in the PB-ITS Portfolio.
Figure 4. Increase in BVS Scores for G-REX from FY2017 to FY2018
G-REX became a model for implementing Agile practices on software development projects at PB-ITS. It influenced a change in the acquisitions process for procuring technology, and demonstrated the value of delivering value quickly and continuously to a product’s customer base. Most of all, it highlighted the importance of establishing a strong partnership with the user base to deliver a product successfully. On a personal note, I, Ankur was nominated by the National Office of Leasing for Customer Service Recognition award three years in a row, 2017-2019, and for the GSA CIO Award, the highest IT award at GSA, for my work on G-REX.
4. What We Learned
Most improvement efforts, especially in a big machine, like GSA remain stalled because most guidance assumes there is a green field that allows you to make whole-sale changes. However, that couldn’t be further from reality. The starting point for each change initiative is different based on the relevant circumstances that exist. Therefore, it was critical that the G-REX team started where they were and not where they wish they could be. Once the underlying issues in terms of technology, process and training, were acknowledged, the team needed to focus on making small improvements from there on. While prototyping helped me, Colleen, to quickly assess user preferences and the issues that plagued the existing product, retrospectives from each sprint helped me, Ankur, fine tune the development process and coordination with the other support groups. There was existing infrastructure and cadre of tools that could be used to make incremental changes, and people who, when provided the right support and guidance, were willing and able to contribute to the change initiative.
You have to be agile to do agile. This is a much used cliché in the agile community. However, it stresses on the reality that the agile transformation would not be as successful had we not focused on improving the budgeting, governance and contracting processes that added layers of constraints to the development process. Simply introducing Scrum would’ve meant that the G-REX team would be building pre-fixed requirements in sprints at a predetermined cost, a sad story that is unfortunately not unheard of. However, managing those constraints within GSA’s framework allowed a true public-private partnership where a shared understanding exists between the business line and technology. In 2020, I , Ankur, applied these learnings from G-REX in an interagency project to develop an Organizational Agility Playbook that describes 10 plays focused on setting an organization up for success to deliver timely best-value solutions.
It was also important to fight the right battles at the right time. Addressing the immediate pain points, then resolving the root cause of that pain and ultimately getting rid of the problem are three related concepts that when approached in an untimely fashion can lead to more problems. “It’s like playing doctor,” as I, Ankur, explained at my Agile2019 session, “You don’t want to be talking about finding a cure for a disease two years down the line when the patient is in pain. They probably appreciate an ibuprofen more at that point.” Addressing the immediate needs of the G-REX users built credibility and allowed Colleen the time to secure funding and me to devise a solution that was more conducive to users and more cost-effective in the long-run.
This story would not be complete without the serendipitous impact of the dedicated helpdesk. While, the helpdesk was established to expeditiously address the support needs of the G-REX users, it quickly became a source of user feedback from across regions. The G-REX team was able to collectively analyze trends in user reported issues, and talk to individuals to discover preferences and issues that did not surface during the CCB discussions. This gave a voice to the user whose feedback did not get funneled through the CCB.
5. Acknowledgements
The successes and wins mentioned in this account have certainly not come overnight and definitely without the help and support of individuals that have believed in us and supported us along the way. In particular we would like to express our sincere gratitude to: Andi Schernecke, who unequivocally supported me, Colleen, in committing time and resources from the business side; Phil Klokis and Paul Butler for providing the much needed management support for this change in the technology organization; Paul Done for sowing the first seeds of agile with me, Colleen, at PB-ITS; and Justin Pinkney for keeping us out of contracting troubles and being the voice of reason when creativity wandered into grey areas. In addition, we would like to acknowledge the Operations (AppOps) and IV&V team for collaborating with us on the many experiments with the deployment and release management process, and supporting us through tough times. Finally, we would like to thank our shepherd, Rebecca Wirfs-Brock, for her time, ample patience and feedback for making this idea in our heads into an experience report. This would still be an unorganized list of bullet points without your shepherding.
