Why would a company or person be interested in creating a prototype? The answer is simple - to test a new idea, improve a product, or find a better or more optimal way to solve a business problem. According to Statista, 56% of organizations worldwide expect generative AI to enhance these aspects, though many likely don’t have an exact plan for implementation. In this blog post I want to share my experience on building prototypes, quickly testing ideas and most importantly, innovating.
You would be right to question my authority about prototypes, as I don’t have a lifetime of experience. However, I spent over two years working for the AWS prototyping team, where I engaged with dozens of companies from various verticals and of different sizes - from startups to regional banks and beyond. So, I have a couple of anecdotes from that experience.
The good one
One example in particular highlights why it’s important to test and validate ideas. We joined an ongoing customer project with the main goal of onboarding them on AWS ML offerings, especially deep learning. Prior to our engagement, the sales pitched that an ML model could be built and the process automated by eliminating a human in the loop, which was a low-level role for students. The sales play was that $70,000 expenses could be replaced by AI, though AI might have lower accuracy.
We spent some time enabling the team to build the models and use the tools on AWS platform, but you could feel that the customer’s engineering team was getting frustrated - the model was giving us 80% accuracy, meanwhile a student could spot with almost 100% accuracy. I relayed the message to the sales team and management that the customer’s tech team wasn’t really happy just getting educated about cloud and AI capabilities. However, the leadership teams on both sides were very happy about this progress.
Nonetheless, I had a hunch that there might be a simple solution avoiding deep learning altogether, so I scoped some time to sift through research papers for a potential solution. A few days later, my colleague happily pinged me back saying that he found a solution - 3 lines of code based on opencv library. The most important thing was that we were able to deploy our code into a serverless function (AWS Lambda) and avoid expensive compute instances used for deep learning, spending only $3 vs. the original $70,000.
Was the customer happy - very! Was the sales team happy? Hell, no! What’s great about Amazon is that everything revolves around customer obsession, and the long-term game is that by saving money there, we returned this budget back to the customer for other experiments and expansion. A takeaway from this experience is that without extra effort and experimentation, the customer would keep spending $70K on what could be automated with a library from 1999.
Prototyping today
I agree with Andrew Ng, who was a keynote speaker at an internal ML/AI conference: nowadays, it’s very cheap and quick to validate an idea. What he meant is that with advancements in AI and ML, we can build a prototype in a week or even a day, which is an amazing thing. Now the catch is that other parts of the project might still be lagging.
I wrote a blog post where I shared my mental model of running data science projects and implicitly mentioned that preparation and scoping will take most of your time in a project. GenAI might help you with writing a scoping doc or a proposal, but the savings won’t be significant. Meanwhile, the Prototyping stage not only can be reduced significantly, but you can try 10x-100x more ideas and choose the best one from an enormous pool of good ones.
To contrast possibilities today with those from a few years back, here is a relevant example. I had a prototyping engagement with a customer who needed to extract and structure data from reports of different companies. After four weeks of heavy development with cloud tools such as AWS Extract, which supposedly helps to extract entities and specific keywords like profit or expenses, we concluded that the approach was still too fragile to be automated and the added value was not significant.
Nowadays, with the help of LLMs, unstructured data has a higher chance of being structured, and additional insights can be extracted more easily. The lesson to take away is that not every prototype will end with business growth, and stakeholders need to be prepared for that. It’s a risky investment decision - you pull a team or more for a week, burn some IT resources, which can sum up to $10K-$50K a week or more. And your probability of success should be below 100%; otherwise, why prototype?
The cost of not doing it might be much greater, but this might not be in sight of the management or decision-makers. For example, in this case, the company needed to test new grounds because the speed at which they were parsing documents was core to their business.
Constant push for innovation
Let’s zoom out from prototyping - it is just a small piece in the grand scheme of innovations. How do you make sure that you, your team or the company with which you are engaging is constantly innovating?
For innovations at Amazon, I would emphasize two high-impact factors: an influx of new people and the Working Backwards innovation framework. For the former, a person without historical baggage would be keen to try the same idea which may have failed 10 times already, simply because they don’t know its history. Meanwhile, due to the rapid pace of technological advancement, the timing now might be right for success.
I recall a colleague suggesting setting up a phone line with a voice message box for internal usage, where an issue would be dictated to a machine, analyzed and dispatched to the right person for resolution. Back in 2018, every step of this idea, such as setting up an international line, converting voice to text, or classifying the message, was either a complete blocker (e.g., a person’s accent in a voice message) or months of work (e.g., text classification with NLP tools). If someone is assigned a similar task today, it would be a walk in the park - LLMs to the rescue!
The lesson here is to be conscious of your biases, try to break out of your bubble, and review previous failures; maybe now is the right opportunity.
Now, there are many frameworks to help with innovations and creation of new products. What’s special about the Working Backwards and PR/FAQ framework is that it’s customer-centric, pushing you to think about the customer rather than the product. Basically, you start with:
- Listen and understand the customer’s perspective. We are interested here in pain points, challenges, desires, etc.
- Define stage. The inputs from the previous stage translate into a clear problem or an opportunity - our North Star that will guide us through the project.
- Invent stage. Here we look for the best solutions given a problem, dream big, and involve others in the development process.
- Refine. At this stage, we want to be crystal clear about how the idea works. Here’s an interesting part: we write a mock press release, not for customers, but to peek into the future when the product is developed. This helps clarify how customers will experience the product and allows for internal communication and discussion. For this purpose, the main document should not be longer than one page, but the appendix will have an FAQ section, which is why it’s called PR/FAQ.
- And finally, we want to test with some experiments, identify the issues, and continuously iterate.
Connecting the dots
Now that we’ve discussed prototyping and innovation, how are these applied in practice? Let me share an example. A finance industry customer was interested in pushing the boundaries with innovations, so we allocated a couple of days for business and tech teams to learn about the innovation framework and think big. As a result, we scoped one big, challenging idea for a prototype. A prototype was needed because the idea was ambiguous, with some known unknowns and many unknown unknowns. For example, during the prototype development, we learned that the data possessed by the company was insufficient as counterfactuals were missing, i.e., what would be the outcome if the world were slightly different.
As planned, we built data pipelines, created a user interface, identified challenging or missing parts, and presented a minimally viable product. Through this exercise, we found the project was far from a positive outcome, which was totally fine as we learned about the gaps and a potential path forward. But a big surprise to me was when the business side took that outcome and broadcasted it as a big win with an international press release about cooperation and pushed boundaries. The lesson here is that there are always different points of view, and one should be ready to accept that.
The difference
For data science projects, you might wonder which approach is better - prototyping or the scientific approach? The former came from product design and engineering fields and is driven by agile methodology and rapid development. The latter requires rigorous methodology based on statistical principles, peer review, hypothesis testing, and reproducible results. In my experience, the choice depends on the team and organization you’re working for. The scientific approach is less common in business environments when it’s not a requirement (unlike in the pharmaceutical industry, for example).
Final word
Probably everyone agrees that innovations are a must for a company’s or country’s survival. However, there is an edge case where this is not true - there are companies, and I wouldn’t say more than a dozen, where their strategy is to “keep the lights on” or basically extract profits. We know that there are different stages in a company’s lifecycle, and if a company has entered this particular stage, you’re unlikely to succeed with innovations there. The reasons for a company to be in this stage might be many - investors interested in value extraction rather than growth, lack of ownership from the owners, lack of investment, etc. Regardless of the reasons, if the leadership is not interested in innovations, pursuing them has a lower, if any, probability of success.
Good luck with prototyping and innovations! And if you’re interested in discussing this in more detail or running an innovation project, let’s connect.