3/6/2018
"Requirements engineering" was supposed to ensure that development teams delivered applications that users wanted. That didn't happen (as the process of requirements engineering itself demonstrated). The answer to giving users what they want turns out to be getting rid of requirements altogether.
Requirements exist because everyone recognizes that users aren't developers. On that understanding, requirements formed the channel through which user needs were communicated to developers.
Except, as communication channels go, requirements were pretty awful at communicating user needs. Back when I started building applications, despite the time we spent gathering requirements, we frequently delivered applications the customer didn't want. I still remember a project where we had to back the whole application out shortly after installation and re-design it before the customer would let us try again. That project sticks in my mind not because it was unusual but because I ended up taking over as the lead (lucky me!).
Amazingly, we weren't stupid -- we were just typical. We even, like every other development team, recognized that we had a problem and needed to do something about it.
The Birth and Death of Requirements Engineering
The solution, at the time, was called "requirements engineering" which boiled down to spending more time on gathering, defining, and documenting requirements. It also included managing those requirements through the life of the project (by and large, by "managing requirements" we meant "rejecting changes and additions"). An important part of the requirements engineering process was "traceability." Traceability allowed you, for any requirement, to see where the requirement was reflected in the application and, for any part of the application, see what requirement drove it.
All by itself, the existence of "traceability" gives away the awful truth that requirements engineering didn't solve the problem. Traceability, like much of the work added by requirements engineering, was a symptom of the problem and not a solution to it. After all, you only need traceability if you're frequently delivering applications that have nothing to do with the customer's requirements. To put it another way: If requirements engineering solved the channel problem then traceability wouldn't have been required.
In fact, the real purpose of traceability and requirements engineering was to shift the blame from developers to users. If the delivered application wasn't what the user wanted, it didn't matter to the developers: Thanks to traceability, the documentation generated by requirements engineering would demonstrate that the functionality the developers delivered was exactly what the user had agreed to.
Sadly, end users kept stubbornly pointing out that the application wasn't right and it didn't matter what the documentation "proved."
The Solution
What's interesting is that the solution to the channel problem was in place all the time. The requirements problem really only existed in big projects for building whole new applications. As I knew from my own experience, the requirements problem didn't exist for software maintenance (changes to existing applications to extend, enhance, adapt, or fix them) .
In retrospect, it's easy to see why. To begin with, for any application there was often a dedicated maintenance team. Over time, the maintenance team often got to know the application's end users well. As a result, the "requirements process" often consisted of "Go and talk to Judy in shipping, she knows what needs to be done." It was not only easy to consult with the affected users but, because those users often had input to the team members' appraisals, developers were incentivized to consult with end users. The maintenance developer's mantra was "Happy users, happy boss."
Replacing Requirements
Agile moved that process into building new applications. The closest approximation to requirements gathering in the Agile process are "user stories" (we discuss how to create stories in Learning Tree's Effective User Stories course). Like requirements, stories are gathered from users...but that's where the similarities end.
For example, a story focuses what a specific user wants to do and why. Here's a typical example of a user story from a time entry system for business consultants:
As a consultant for this company I want to be able to enter the time spent on project tasks so that I can get back to work on my projects as quickly as possible.
This story bears very little relationship to the output of the requirements engineering process, which focused on how the application was to behave. Because of that focus, a typical requirement for that same time entry system would look very different -- something like this:
- Consultants will be able to select from a list of valid projects they are assigned to
- Consultants will be able to enter their time to two decimal points
- The application will edit time entered to ensure that it is a positive number and less than 24 hours
- The application will accept, store, and provide a response within 2 seconds when time is saved
- ...more...
Several differences stand out...but the the most important one is that a story looks more like a request than a solution; A requirement, on the other hand, looks more like a solution. A story is a beginning not an end (Jake Bartlett at Testlodge.com sums up all the differences between stories and requirements nicely, I think).
Extending Stories
Of course, because a user story is just a beginning, to deliver a solution we need additional details. However, with user stories, those additional details will be worked out with the client at the last possible moment (probably when the story moves to the top of the backlog and the odds of it being included in an upcoming sprint become highly likely).
That "working out" process is handled through a conversation between all the interested parties (the client, the coder, the tester, and so on). It's typically at this stage that non-functional aspects of the story are determined. Non-functional aspects include how to reduce technical debt in the implementation, what response times are required, and how this change will work with the larger application to provide a consistent user experience (as we discuss in Learning Tree's Splitting Stories: Getting to Ready).
However, unlike requirements documentation, these decisions may never appear in a formal document, let alone support "traceability." The goal of this process is to implement requirements, not document them (to quote the Agile Modeling site). For example, the conversation may involve writing some code or building a simple wireframe to give stakeholders something concrete to make decisions about. That initial sample screen or wireframe may be the single most important "document" created to help stakeholders define what's to be built...and it may have been thrown away by the time the functionality is delivered.
But there's something more important going on here. Defining functionality with a focus on the end user and involving all the stakeholders in the conversation addresses the essential problem in requirements: The separation between developers and users. Over time, this process gradually erases many of the differences between end users, business analysts, and developers. Those distinctions won't go away, of course (users will still spend much of their time in the business; developers will spend much of their time building applications). But, more and more, developers come to understand user needs while users come to better understand developers' challenges. The channel may not disappear but it gets shorter and clearer with every delivery.
Related Training:
Agile and Scrum Training
Project Management Training
Business Analysis Training