Recently I had similar discussions with different customers. The topic was the same - Customer testing. It looks like this area is very often misunderstood. And I am asking why.
I suppose we all know that Customer testing is a critical aspect of Agile development. It is obvious, isn’t it?
We use Scrum in this company. On the other hand a lot of our customers are used to Waterfall and Agile is something new for them. This usually ends with some Hybrid, which is not necessarily bad. Any step from Waterfall to earlier customer involvement during incremental development is good.
It looks like all these hybrids are very similar
- Agile with “fixed scope” - that is risky
- Requirements are defined before development starts
- At least on high level
- Of course they are clarified and changed later
- Agile with “fixed date” - that is fine
- All critical milestones are defined in advance
- Internal Development is done in Sprints (iterations)
- All requirements are in backlog
- Sprint is planned based on priorities and risk mitigations
- Internal QA is a part of each Sprint
- There is a demo after each Sprint and build is delivered to customer
- Customer Involvement in Sprints
- Usually works
- During planning
- During Sprint for clarification
- On demo
- Often problematic
- Customer testing – after demo
- Usually works
If we are lucky, the customer takes part in Sprint planning – his input is mainly by means of business prioritization. This gives them also an idea what they can expect to be delivered by the end of the Sprint. The customer also needs to be prepared to answer clarifications.
The Demo is a time to show our results to the customer and to get their feedback. We cannot expect that customer will accept features based on the demo and that is the reason why the ensuing Customer testing is critical. But this is where the problems start.
We said that they are used to waterfall and Customer testing for them equals UAT (User Acceptance Testing) at the end of the whole project. Sometimes they also count with limited Pre-UAT. Now we ask our customers to test regularly after each Sprint and this is a big change or even a surprise for them. The Customers will definitely ask why they should do it and we need to be prepared to answer and fight for it.
This was quite a long introduction, but I hope you have the context and now I can share my typical “Defense of Customer Testing”.
I always start with a definition of testing as an activity and required artifacts. I would like to come to a point when the customer understands that QA is not only about Functional Testing.
- Assurance that the best value is delivered to all stakeholders
- Testing is not about artifacts, it is about activity
The next step is a definition of Functional testing and Test Case, because this is the most important activity and artifacts we expect to be done by customer.
They should prepare their own test cases to describe their Acceptance Criteria. (See this article about Test Case Abstraction)
Very important is to explain that Test Cases are role based. It means you always need to start with roles identification – more like personas. Also make sure their TCs will have a correct level of abstraction.
UAT success is the goal of each project but it is very often understood as the time to find all bugs. This is one of the biggest misunderstandings. UAT is a verification that Business needs are fulfilled and system delivers expected value. It is not about finding bugs – they should be found during regular testing.
Second part of my presentation is a comparison of two hypothetical projects
- Sprint based without regular customer testing
- Sprint based with regular customer testing
My goal is to explain that total effort in both cases is roughly the same while the second approach mitigates a lot of risks and surprises.
The following slides assume that customer knows he needs to test before UAT and his testing should be based on Test Cases.
Note: FRD = Functional Requirement Design (the main document to capture requirements).
None or minimal regular customer testing
Continuous customer testing
As I said, hybrids are not ideal, but they can be better than pure Waterfall. Try to involve your customers as much as possible and help them with it. I think they are sometimes only scared because this is something new for them.