Our experience with this agile methodology for building better products
I’m a computer engineer and at Octobot I’m part of the Product Quality team of Turntide, a US company that develops electric and sustainable motors. In this article I want to share with you our experience working under the Specification By Example scope, a methodology that fosters collaboration among software development team members and makes work more efficient and accurate.
Turntide and Octobot
Turntide’s purpose is intrinsically connected with sustainability, since its motors use electrical energy in a more intelligent, efficient way by minimizing the amount of energy that is dissipated. Therefore, they contribute to reducing the carbon footprint and relying on more sustainable energy sources without resorting to fossil fuels.
As for the project we are developing in the Product Quality team, it is a framework that is responsible for automated testing of engine performance. This solution makes testing processes more efficient, saving time for every other development team in Turntide.
There are many variables and parameters that are part of the operation of an engine, which are studied and tested to certify it is working properly. For example, the speed of the motor; the direction in which it is moving, whether it is forward or backward; the voltage, current and resistance value, and more. All these parameters are under the magnifying glass and are constantly evaluated during the testing stage.
Turntide motors are controlled by software, or a motor controller, built by their developers. Octobot joined their in-house team to complement them with specific skills and support their business expansion.
In the Product Quality team, we have frequent meetings with different Turntide stakeholders to understand how the engines work, define the requirements to be tested, and write the detailed requirements specification, which at the same time are the automated test cases of the system. The framework is written in Python, and the library we use to interpret the Gherkin code is called Behave.
All of these practices are part of the Specification by Example recommendations, which we’ll dive deep into now.
About SBE
Specification By Example, or SBE, is a software development methodology that is encompassed within agile methodologies. It has to be seen as a complement to Scrum, not as something separate or different, and can be applied within the framework of a Scrum project. SBE can also be known under the acronym BDD – or Behavior Driven Development.
It is a methodology that focuses on collaboration between all the disciplines and roles that interact during a project, in order to reach a shared understanding of what is being built. To achieve this goal, it’s necessary to specify the project requirements based on concrete examples that are free of any ambiguity.
This has a positive impact on the final result of the software product, because it increases its quality and the delivery time. The number of defects that developers introduce into the system is greatly reduced, which saves time and money in the long run for system support and maintenance.
Purpose and added value
In the world of software development, there are 3 problems that are very common to most projects according to Leslie Brooks:
Little “b” bugs
Untested developments will result in the detection of bugs when the customer starts using the product. In other words, what was specified in the requirements gathering stage was developed, but afterwards a sufficient number of test cases were not tested. As a result, the product is delivered to the customer with many defects or bugs.
Big “B” bugs
Certain product enhancements requested by the customer were overlooked or misunderstood at the development stage. This is the typical case when the developer did exactly what was specified in the requirements, but there were certain requirements that were either not included in the development, either by mistake, by forgetfulness, or simply by an error in interpreting what the customer wanted.
Postponing testing to after development
It’s common to work on testing after the development is finished, which extends the defect creation cycle. Not testing well enough at the same time it is being developed has a negative impact that will result in future issues. When testing is postponed to after the development stage, it generates a kind of ping-pong effect where a lot of time is spent going back and forth between the developer and the tester.
The Specification By Example methodology seeks to address and solve these three common problems, to ensure that all requirements are well defined, and that exactly what has been defined in the requirements stage is developed. This avoids any misunderstanding or loss in the process and, at the same, increases the chances of having everything developed and correctly tested in time.
To summarize in a few words, Specification By Example seeks to ensure that what has been defined is built, tested and also automated.
Specification by Example in practice
To work with SBE it is necessary to follow three main principles:
1. The requirements must be written using concrete examples that are easy to understand.
Let’s suppose we are building a system in which users have to fill out a form and, among other data, they must enter their date of birth. In this system, only birth dates of adult users are accepted, so the system must give an error for ages under 18, and must allow users to upload the form if the date of birth implies an age equal or superior to 18 years old.
Following Specification By Example, we’d explain this requirement by detailing 3 possible birth dates as input and what the expected output the system should give us for each case:
Example if we’re implementing the system today:
- for 01/20/2000, the result of the test is OK
- for 02/28/2002 the test result is OK
- and for 04/25/2015 the test result is ERROR.
As you can see, we are specifying the desired behavior of our system in a concise way. This allows us to follow clear examples to develop the software and, at the same time, build the automated test cases using the same reference.
2. The requirements are written collaboratively, which makes it difficult to have different views on the same requirement.
They must be defined through different meetings, usually attended by the product owner, the developer, and the QA manager. Between the 3 parties, the system requirements and their examples of expected input and output are assembled in a more precise way.
3. The requirements are exactly the test cases.
No additional interpretations are needed: once the examples for that functionality are defined, both the developer and the QA leader are going to build a product starting from the same example, minimizing the possibility of misunderstandings. This brings us back to the concept I mentioned earlier: what is defined is exactly what is developed, tested, and automated.
Benefits of SBE
Shared understanding
Both the client and the team agree on how the system requirements should be written.
Clear acceptance criteria
The software is accepted if all test cases pass successfully. In other methodologies it usually happens that the developer finishes the task and writes unit test cases not only to check accuracy, but also to ensure a certain degree of coverage in the code.
However, this does not mean that if all the unit tests pass, the software will be accepted by the client. In SBE the acceptance criteria are the same test cases that were defined earlier, which means that you are testing directly against the acceptance criteria, not against a set of tests defined with arbitrary criteria to cover code and test an isolated portion of the software.
Living, versioned documentation
Documentation equals requirements, which in turn equals the test cases themselves. They can evolve and change over time, and every time a requirement changes, it can be easily retested, since the requirement is nothing more than the same test case that will be executed automatically. At the same time the written requirement itself works as the system functional documentation: by reading the requirements I can easily understand how the system works.
Regression testing is implicit
Doing regression testing is the same as running all the automated test cases that were defined, and checking that everything works and nothing was altered by a last minute code change.
Efficiency and accuracy
The requirements are rigorous and complete, which fosters the ground for making automation more efficient over time. Once everything is up and running, adding a new requirement to the system can be as simple as adding a new text file with the requirement description. The rest is running the software and seeing what results the test gives, and that has zero or minimal development effort.
Mitigates (or virtually eliminates) defects in the system
By eliminating most of the problems that arise from misunderstandings and ambiguities, we avoid mistakes or defects.
Allows development and QA teams to work in parallel
In traditional methodologies, the testing team usually has to wait for the developers to release the features to define and develop the test cases. In SBE, since both teams start from a single requirements definition and a set of expected results examples, there is no need to wait for everything to be developed in order to implement automated test cases. The two teams can work in parallel, which saves time.
In fact, in some situations, the QA team may finish developing the automated test cases even before the developers finish implementing the product.
SBE challenges
Requires a lot of training and coordination
On one hand, implementing SBE requires training the team so everyone can learn how to write the system requirements. It also uses a language that is simple in the sense that it is readable, but you need to be able to write the key system operation scenarios in a concise and understandable way.
SBE’s ultimate goal is that all parties involved understand what is expected from the system, and this shared understanding is unique. On the other hand, it requires coordination, because as we said before, it’s based on constant collaborative work between several stakeholders.
From the technical point of view, the focus is on the product’s quality
In SBE, prioritizing clean, maintainable code is vital. From the technical design of the solution, it has to be very well designed and consider the reuse of components, since the idea is to save support and maintenance time in the long term. This can only be achieved if the automated testing software allows adding new requirements or test cases with the minimum possible effort.
This, as mentioned above, implies an initial investment of time and effort in the technical design and development of the solution, but it has many benefits in the long run. Later, adding new test cases becomes a trivial task or with minimal maintenance effort. Long-term time savings is where SBE wins the battle against traditional development methodologies.
Myths & truths about SBE
Scope
“SBE is a QA methodology.” – No, SBE is a product development methodology. It is not something that is limited only to the testing stage.
Time
“SBE takes much longer to implement.” – No, SBE is a methodology focused on improving product quality, but contrary to popular belief, it takes much less implementation time than the sum of all stages in traditional development.
Efficiency
“SBE takes a lot of valuable developer time, and requires more QA time.” – It is proven that using SBE significantly reduces development and QA time. An example mentioned in Cucumber Podcast shows that “in just 9 months the defect rate decreased from 34% to 4% and they avoided a very costly rewrite.”
Creativity
“SBE removes the opportunity for developers to be creative with their work.” – That is not the goal of SBE. The methodology attempts to eliminate ambiguities and reduce individual interpretation, but creativity remains latent.
Agile
“SBE is a different type of agile methodology.” – No, SBE can be applied in conjunction with Scrum, and should not be seen as different or separate from the second.
“It is the same as TDD (Test Driven Development).” – Wrong again 🙂 TDD is focused on unit testing, while SBE is focused on functional and system acceptance testing.
Specific language
A language called Gherkin is used to write SBE requirements. It is a descriptive language (one of the Domain Specific Languages, for those who are curious and want to go deeper into this concept) that allows using some keywords in pure English. With Gherkin we can detail the requirements through the description of all the expected input and output examples that a system must fulfill to meet acceptance criteria.
The principle concepts used behind code written in Gherkin language are: “Given”, “When” and “Then”.
“Given” describes the preconditions, i.e., the state in which the system must be prior to the test case execution. For example, to test that a user under a certain profile is able to view certain options on the screen, I should first make sure that the user is logged into the system, otherwise I would not be able to perform the test.
“When” is used to describe the action(s) that initiate the test. Continuing with the previous example, a possible action could be pressing a search button, or accessing a certain menu in order to visualize the desired information.
“Then” is intended to describe what the expected test result will be after the action indicated by the “When” command is performed. If, for example, the action indicated in the “When” statement is to press the search button, the “Then” command should indicate the expected search results that the system should show the user to validate that the test works correctly.
Condensing everything into a concrete example of Gherkin code, a possible test scenario could very well be written as follows:
Scenario: A regular logged-in user should be able to perform a search in the catalog based on one or more keywords.
Given these books | Book Title | Author | | Where the Crawdads Sing | Delia Owens | | Book Lovers | Emily Henry | | Ugly Love: A Novel | Colleen Hoover | And a regular user When we search for books with these attributes | Book Title | | crawdads | Then the following books will be returned | Book Title | Author | | Where the Crawdads Sing | Delia Owens |
A human-readable language
The goal is to have a language that is understandable by everyone involved in the process, from the product owner to the developer to the QA team. Gherkin combines keywords in the English language and is free of the typical technical symbology that exists in a traditional programming language.
On a technical level, there are different software packages that interpret the text files written in Gherkin language. That allows the developer to write the different test cases that will be automated to be able to run those test cases against the system implementation.
The important thing about this is that the same set of requirements that are written in Gherkin are executed against all these test targets, as it’s known. The advantage of Specification By Example as a methodology is that it abstracts from the underlying characteristics of the implementation, i.e., the same requirement can be tested against more than one API or more than one type of engine at the same time, which allows reusing the same test case for automation purposes.
It is very important to design the automation framework well from the technical point of view, so that the abstraction is as flexible and maintainable as possible. If tomorrow a new engine API for testing emerges, our entire framework should be reusable and testing the new API would mean a minimum effort for us in terms of additional work time.
Recommended resources
There are 2 books that I recommend to keep learning about SBE:
Specification By Example, by Gojko Adzic.
It explains the methodology, and its advantages from a theoretical point of view. It does not show code examples, but rather focuses on the motivation for the methodology and the value it adds to software development.
Behavior Driven Development with Cucumber, by Richard Lawrence and Paul Rayner.
This book contains code examples using a software package called Cucumber for the Java language. At the beginning, the authors introduce the methodology using fictional dialogues between a product owner and a developer, telling you what SBE is all about through a story and from a practical point of view.
To enter the world of the Gherkin language, and the Python software package called behave, we also recommend this blog post I wrote recently and the following link:
https://behave.readthedocs.io/en/stable/
You can also listen to my experience with SBE in our podcast, Octobot Tech Talks, for Spanish-speakers only:
See related posts
Unlock the Power of Radical Candor
Our CEO Guillermo Perez will lead you through the Four Quadrants that are the fundamental pillars of author Kim Scott’s work.
The Delivery Lead Role
At Octobot, Delivery Leads help us keep our software development projects on track. Let’s take a look a closer look at the role.
