Mobile Technology

This blog is generally about mobile technology, concentrated on the Android/Chrome OS arena. However, there will be other topics delved into from time to time.

Tuesday, August 4, 2015

When Software Development Goes Bad Part .9

(Title changed to fit in with the current series of posts on software development gone bad. Formerly titled: The Problem With Scrum: The Business Side)

Having spent the last 4 years of my career in an organization that wanted to practice Scrum, but behaved in a completely Waterfall way, I have some serious insight into why software development organizations fail at Agile development.

And it isn't the methodology's fault. It's the business' fault. (Note, this article uses business and product interchangeably.)

The business side of the company I just left so wanted to be agile. They spent time talking about Scrum. They wanted more releases, faster. But they couldn't behave in agile ways.

Here were the core issues:

  • No, "well-groomed" backlog
  • No prioritization ("it all has to get done")
  • No business involvement beyond a cursory "this is what we want" at the beginning
  • No test environment that mimicked production
  • No operations and infrastructure staff
  • Not enough resources to begin with
Let's dissect each of these and discuss why this was such a problem.

First, the backlog. Everyone agreed we needed a backlog. We, in fact, had a backlog. But it was holding tank for anything that didn't fit into the current "sprint". I quote sprint because even the idea of a sprint was destroyed by the business. 

Sprint is defined as: "a sprint is a set period of time during which specific work has to be completed and made ready for review."

However, the business didn't want a set period of time. One sprint we had ended up taking months because the Product Managers refused to close it until all of the features they wanted were delivered. (See, NOT Scrum.)

Without a "well-groomed" backlog there was no sprint planning meeting. The way it worked went something like this: PDMs and BAs picked a bunch of "items" (some defined, many not so much) and loaded them into the sprint. Negotiations began where the dev leads would review the items, and kick back any that were poorly defined (which was most). Then the PDMs badgered the BAs to provide better requirements, insisting that their loading of the sprint was doable in the sprint time-frame. This resulted in the first half to the sprint being consumed by providing proper requirements before dev could start. The PDMs then went away for the rest of the sprint and came back at the end to yell about what wasn't complete.

Seriously. The business would provide as little guidance as possible, and expected the end result they were anticipating. And anything not done, after verbally being dressed down, went to the top of the list for the next sprint. And so it went leading to.....

Lack of prioritization. When the dev group pushed back saying, "we can't deliver all of this, which of these are the highest priority?", the business came back with "all of it!" Not helpful. This usually meant that none of it got completed, at least to any real level. The biggest features languished for many sprints if the dev group insisted on sprint cadence. Or just long periods of time otherwise. 

One dev project, which was promised to be delivered in 2 months because the PDM, BAs, and devs were going to work closely together, was still not done 2 years later. Mainly, because the PDM and BAs got pulled away into other things. With no prioritization, dev was left to try and guess what was most important to get done. The success rate of that hovered around 10%. All of this was a direct result of......

The business not being fully engaged throughout development. Scrum CANNOT be successful, especially without a well-groomed backlog and prioritization, without PDM and BA engagement throughout the dev cycle. It's like throwing water against a wall to see what sticks. 

Much of this falls into a bigger category (and discussion) called Unrealistic Expectations. The business side loved the idea of Scrum regarding lighter requirement documentation, but they weren't willing to invest themselves into the rest of Scrum. Especially the part where they remained engaged throughout the dev cycle. One BA even was famous for insisting that dev only come to him once they had fully tested and certified that the solution was 100% bug free. -insert hysterical laughing here-

This thinking from the business side was pervasive, with the director of product once being quoted as saying: "when we make a software change we have to be 100% sure we don't break anything." You can't make this stuff up. And leads us into......

Test (dev and QA) environments not mimicking production. Now obviously this is impossible for every environment. But the point stands that there should be an environment for testing that is a replica of the production environment. Especially with decrees like "changes can't break anything." How does one certify that in environments that aren't like the environment that the user is going to be using? (It should be pointed out that this director was also the head of infrastructure, so you can see the hypocrisy in his decree.)

So as a dev group we not only had to guess what the business really wanted, but we had to deliver to their expectations, and be completely bug free, but had no test environment in which that could all be certified. I often said that when I perfected all of that I would make billions sharing the method with other software organizations.

Oh, and by the way, not only was this the expectation, but the devs weren't allowed to just be devs. Which takes us to....

The lack of operations and infrastructure staff. Of course, many people would say "in Scrum, you have DevOps, not mere devs!" However, anyone that says that doesn't understand DevOps. DevOps simply means that devs are capable of doing deployment and operations in the QA and dev environments. IT NEVER MEANS THAT DEVS SUPPORT THE PRODUCTION ENVIRONMENTS.

However, without operations and infrastructure resources, and without a true Operations Manager, then suddenly the devs were thrust into the roles of supporting the production operations and infrastructure. I frequently asked "why, as a dev, do I have access to the production environment?" The answer: "Because we need this done." 

Never mind the obvious security issues this raises, but remember the problem above? Too much to do and too little time (and resources). That was exasperated by the draw on the dev resources that this added responsibility caused. I was quoted multiple times, when myself and other devs were pulled into this support work, "to remember this at the end of the sprint when PDMS and Project Managers ask why more wasn't done." They didn't like to hear that.

I could go on but I think you are getting the point. It was no wonder that a project meant to be in market in 18 months was still not in market after 4 years. (See: Unrealistic Expectations) The compounding issues included:
  • Not enough resources across the organization (BAs, Devs, Ops and Infrastructure, Testers)
  • Poor processes that were exasperated by the lack of resources
  • Way too many hours logged by senior staff (I routinely worked 50-60 hours, over 80 during certain times. And that was short compared to our architect)
  • Lack of good leadership
The last item was really the main issue. All of this could have been fixed with strong leadership, reacting quickly to the issues. Not enough resources? Get more. Not the right resources? Get the right ones. Wrong processes in place? Mandate from above what the process was going to be. Rogue agents in the staff? Reign them in. Wrong attitude in the organization (like a director commenting that the prod environment, running a live customer's instance, was really an "experiment")? Reset the of organization to reality.

Unfortunately, no amount of hard work by very good staff could overcome these issues. The team assembled was topnotch. They were all hardworking individuals willing to do whatever it took to be successful. But their efforts were ultimately doomed because the business' behavior was not conducive to successful delivery. In the end the project was doomed to failure.

The last point is:the business, and product team, reacted to all of this by insisting everything go faster. Their failure to recognize their own poor behavior meant that the root cause of the issue was never going to be corrected. They didn't realize that going faster meant the path to final solution was actually slower. Sometimes the prudent approach is to slow down so that you can do things the right way.

All of this resulted in no chance of successful Agile software development.

(The author is currently in the process of writing a book on his experiences that should be out in the next few months. More details to come.)