At the top of my personal list of wasted activities, starting with the most dangerous, I have: building a feature that no-one needs.
This is the stuff that kills startups. It is also the stuff that kills large companies as well, slowly over time.
Building Waste
In my career working in startups, sometimes I have fallen into the spell of a product manager, or otherwise any other product authority, that strongly believed in a specific feature or product and were pretty good at convincing people to build it.
Often, it looks amazing on paper, sometimes accompanied with incredible slide decks and other documents.
An entire engineering team is mobilized to build it. Many times, top engineers that are very good at getting stuff done. A manager is hired for the team. Thousands of lines of code get written.
The team focuses their time 6 months to put the first version out, then 6 more months to finalize the long tail of details needed to put anything into production, from minor sub-features to performance improvements.
Closer to the release time, the product manager involves marketing and sales, and the engineering team works with the support team to provide training.
Then – almost no one buys it. And months after the release, we stare at the product dashboards to find more support tickets than API usage hits.
Supporting Waste
Note that I said “almost no one buys it”. The handful of customers that bought it, turns out, are either heavily using some other feature, or are in the pipeline to expand their product usage in other areas. So we don’t want to disrupt them.
We need to continue supporting it.
Every single change done in or around the feature now requires the developer to spend time understanding it, then carefully modifying it, then testing it.
Infrastructure teams are constantly asking who the hell owns this thing, just to find that some guy kind of owns it after some other girl left the company 2 years ago.
Then support once in a while gets some weird tickets from the only company using the feature. The tickets always come 2 weeks before renewal time. They are threatening to cancel the contract. Management then pulls in ninja engineers from other mission critical projects to patch it.
At this point, in order to calculate the waste, you need to sum all of the collective resources across engineering, support and product, including the time to build it, the time to fix issues and make it production ready, the overhead of slowing down other missing critical projects, and the opportunity cost of not doing something else.
This can be millions and millions of dollars.
The Right Mindset
Let’s review the definition of waste:
waste here is anything that does not bring direct customer value. It is something that the customer is not willing to pay for.
Following the need to reduce waste, the ultimate way to know if something brings value to a customer is to actually release it. If enough people pay for it, then it is valuable.
I don’t want to over simplify the problem. I understand that there are numerous techniques to shift left, from customer interviews to early prototyping.
In my experience, however, these techniques are often flawed. Many times they are based either on only a few data points, or second hand information from sales. Sometimes it might be based on a single point of view from an investor or board member. Or on a whim of someone undisputed in the command chain.
Engineers might have little control and visibility over this process. The feature request will likely come top down. Good engineers will ask questions, but without access to first hand information it is not possible to completely be sure the due diligence was done.
Releasing the code is the ultimate de-risker. If the code is out, you will know the truth quickly, one way or another.
This is the right mindset: think of unreleased code as a hot potato. It is a ticking bomb. 100% a liability. You need to get it out of your hands as soon as you can.
Just Put It Out
In order to really be successful in reducing the amount of unreleased code you have, you need to be able to release quickly and release often.
However, I learned the hard way that this is a problem that goes well beyond engineering. It depends on how the entire organization is stitched together.
On the engineering side, there are an obvious number of challenges:
- The CI/CD pipelines;
- Keeping the main branch green;
- Having a proper test pyramid;
- Having a proper staged deployment strategy;
- A/B tests;
- Shadow deployments; amongst others.
These are things you can and should build. They won’t solve the problem by themselves, however.
In the end, it boils down to (1) being able to release the minimal amount of surface area without making the feature meaningless and (2) being able to release potentially unstable or non-performant code in safe environments that won’t burn your company down.
Both of these require you to not only have the right engineering release infrastructure and processes, but also an alignment amongst product, sales, marketing and support.
However, the incentives might be in the wrong direction. Product managers might want to hold the product until it is “just right”. Marketing will want to go out with a bang, not half-features. Operations might want to align the roadmap planning with the headcount planning process. Engineers like to put in all of the final non-functional requirements before release.
Left unchecked, these forces will increase the release cycle and the planning cycle of the company.
Meanwhile, no one knows yet what customers really want. No one knows which assumptions are based on shaky grounds.
The Code Will Bring The Truth
At some point along this process, people forget that any roadmap item carries risk. Even an idea that looks pretty risk-free on paper, such as copying another successful competing product, carries risk.
There is always a divergence between how any work is imagined and how work is actually performed. There will be limits and roadblocks in the most unknown of unknowns.
When in doubt, do the work. Code it out, release it. Then you will know.
Keys
Here some keys I carry with me.
If you want to release potentially unstable or non-performant code in safe environments that won’t burn your company down, try:
- Design the product in a way that properly communicates the level of risk a feature carries. Features that are in an “experimental” area for example is a great way to communicate high risk features;
- Carve out other contained customer visible deployment areas, such as a “nice customers that play along” deployment cohort, or some other “low stakes” deployment area;
- Use probability smartly: the perceived quality of your product is the number of issues it has times the probability that someone will see them; so be sure to keep the probability low, but never too low;
If you want to release the minimal amount of surface area without making the feature meaningless, try:
- Develop a culture where people are comfortable releasing incomplete products, because people will not be punished if a product does not work out;
- The first release should feel incomplete; if not, you are releasing too late;
- Target 90% of issues to be found before the rollout; targeting 100% is too costly (and impossible);
- For the remaining 10%, invest in rapid resolution of customer issues, preferably through rollbacks or feature flags;
- Don’t align experimental releases with marketing releases; these are incompatible cycles that are detrimental to the company in the long term; aligning them makes things slower and riskier;
Finally, assign risk and build in redundancy in the roadmap, so that the overall strategy and overall vision still can be achieved and marketed even if one of the pieces falls apart during experimentation.
In the next post, I will write more about another form of dangerous waste: product defects.