ROI Machine Learning: How to model the investment value of Machine Learning?

By Kimberly Cook |Email | Sep 26, 2018 | 13956 Views

A dialogue and analysis tool for distinct teams

Bizman tells something to which Techie seems useless. Techie responds with answers to which Bizman interprets as objections which would never make their business, their salaries, profit.

This mini parody and its variants repeat themselves continually when business orientated teams interact with technical teams. There are those who have the innate ability to build bridges and are central for everybody, even though they are not usually recognized for their value.

Nowadays, with the fashion and hype behind Artificial Intelligence and Machine Learning the point doesn't lose validity, on the contrary: the SciFied promises of many marketing teams irresponsible?- in every conference generate magical expectations that rarely end in something other than disenchantment... and this is when everyone loses.

At Machinalis we have worked for around a decade implementing Machine Learning solutions. And it has to be said: we have often been protagonists of this hype. Today, with a bit of a trail and scars, we are extra specially careful with this interaction the moment when everyone can contribute the best of their talents and achieve effective science with real business value...and rules. For this reason, one of the tools we have developed and now use is the Machine Learning Value Proposition Model. We have found other tools (some recently) but other then encourage us to invest in the topic, we have not found them truly useful: fundamentally because of the impracticality in the early stages of the decision making process to invest in Machine Learning. These early stages are more sensitive and generate potential conflict between the teams mentioned at the beginning of the article, also in these stages we have found more value dedicating effort in generating shared communication tools.

The Canvas we present today is essentially a communication tool, interactive, which motivates the organization of questions and benefits for all sides. In the best case scenario it becomes a storytelling tool. Whether you are a business person who wants to seed a business idea and intuit that Machine Learning can add value, or your are on the technical side and want to establish common definitions useful for business people and the rest of the team, we hope that this (long) post helps: we share a short description of each field in the outline.

Machine Learning Value Proposition

The What and How. The task at hand. What problems do we resolve with Machine Learning? Typically with Machine Learning, predictions.

It is important to identify what specific contribution we are making before developing a story. It is not only identifying "the problem," rather the factor or trigger that makes this proposal inevitable. A problem to predict the delivery date of a shipment can be explained from the point of view of precision and buffers, yet a trigger goes beyond this, towards generating positive expectations among users.

Organizational Objectives

The organization or company search behind this investment. The objective abstract depends on many factors: typically, the more concrete and coherent they are with the rest of the model elements, the better.

From a technical point of view, the organizational objectives pose a work horizon (scope), which permits us to trace from the proposition to what mobilizes the campaign, whether it is cost effectiveness, profit growth, or market share. From the story point of view, the objective poses the first point of empathy. We don't begin a story for Doctors without Borders with "if you really want to make money...". And it is important because of its functionality, a Machine Learning project can nourish different objectives without entering in conflict. The objectives also become contextualized if Machine Learning tackles a problem that is core for the company or not. Or in the worst case scenario, if it attacks a problem which comes into conflict with the objective. For example: If Machine Learning is used to detect fraud in account generation for a platform, and at the same time the platform's objective is to grow its active user base, our story should not focus on the speed with which the objective is met, but rather on how the Machine Learning solution eliminates the risks of not achieving it in the proposed manner.

However, not being associated with the company objectives at some level does not make the project automatically non-viable.

Product Objectives
This is the space where the multiple leaderships and interests of the project battle it out. In terms of Machine Learning, this is the space where Metrics are defined, one of the principal resources of the technical teams. It is the point where the product make or break definitions are explicit. In our experience, a central point is that the objectives are spoken of technically (for example, precision/recall) but rather in a way all stakeholders understand without the need of a dictionary. Otherwise, the risk is run of having to waste a half hour of every meeting in reviewing the meanings of precision and recall.

The product objectives delineate the relative value of the Machine Learning contribution for the organization and the Machine Learning contribution to the product in question. For example: If a project plans to replace human classification of NSFW videos for one based on Machine Learning, we could generate an attractive reduction of costs for some stakeholders. At the same time, the classifier could have a level of precision that is not in line with the product objectives or introduce unacceptable risks, generating a conflict. A cohesive story sets up a thread between the organizational and product objectives and the Machine Learning value proposition.


The actors directly involved in the use and consumption of the Machine Learning product (normally internal areas of your business). Concrete steps, behavior, stakeholder KPI's that will be affected by the product. In what manner these steps are represented in the data which stakeholders use and generate daily. Specific interactions and types ofstakeholders related to the system.

We should understand the impact of the Machine Learning solution on the journey of our stakeholder. It is not the same to implement a moderation solution of product commentaries in an online catalog for the community manager than the legal team, because what they have to win and lose varies substantially and with it the focus of the solution. In the first case, an omitted moderation could strengthen engagement (even with the cost of obliging the payment of a small fine over copyright for example). In the second, a series impact on the business objectives.

End User Impact

Some Machine Learning solutions pass inadvertent to the final user. Others are the essence of interaction. Knowing how we impact the user is a point of departure of the whole proposal, and thus its location on the canvas. How a malfunction impacts a user is another key point. Even something apparently inoffensive, like a recommendation, can have a negative impact on the final interaction, and so, impact a business decision.

Distinct angles can signify more or less interaction by users and so corresponding viability. (What happens if we need each user to give feedback for 100 days straight for the predictive model to function? Is it a reasonable time from the UX perspective?)

Data Input
The resources you plan to use and process with Machine Learning. Possibly the list is not restricted to data, and could include heuristic previews, model previews, rules considered as data, etc. In a nutshell, if you don't have a technical background it is one of the aspects, along with Product Objective and System Output, in which more effort should be put to generate precise definitions so as to make pitching the idea to the technical team viable (and so they give faster and more usable answers).

The success rate of a Machine Learning project is associated with the quality and availability of relevant data. This factor should be explicit when telling the story, or we fall into the magical belief of an algorithm that, with a bit of sketchy data can provide an extraordinary result. Has it happened to you to be in a conference where a sales team promises solutions too fantastic to be of this world? Has it happened that someone returned from a conference inspired by a sales team to ask for the development of a product as if you were Amazon...without the resources of Amazon? Here we should include suppositions of style "how do we know the user purchase history for the last year" or "how can we detect emotions based on security camera images". Many stories break when they get to this point and turns out that the intrinsic data needed is not available or can not be generated. It is a good moment to make a cost evaluation of the data, who is their owner and what policies of governance pertain.

System Output

It could be information, a concrete number, even the taking of a decision: the closer to the last the more complex is the product to build, but maybe there is more value to be gained. In either case, it is extremely useful to define the format in which the output will be generated and the mechanism and format to interact with other systems, so as to facilitate analysis and shrink reworking.

The data resulting from a Machine Learning solution is, in story terms, where we explain the gains leaving the evaluation up to the listener. Through the output data we finish validating our understanding of the problem, with the possibility that it may open in each case. In explaining that our solution can determine if an anonymous user is or not a millennial with tech brand loyalty with a couple of inputs, we let the receiver fill in blank spaces, not only in terms of the value proposition, but also in the impact potential in other business problems.

Competitive Landscape
A reference of the characteristics that the competition has in relation to the task at hand is one of the most relevant elements we have found for the moment of defining the value proposition. Whether it be in the search for parity in your product capacities or the contrary, for differentiation, the main benefit is in the power to exemplify the desired direction: enables more precise discussions and definitions of technical viability, aligns with the business strategy, anticipates friction with the users, etc.

The analysis of the competitive environment does not only consist in identifying other options to our solution, but also to know what those competitors have already installed. Some competitors may not even belong in the domain. Since Amazon and the later adoption by many ecommerce, the idea of free-returns is now the norm expected by buyers. If we consider penalizing, charging or not accepting returns we put ourselves in a situation of conflict of expectations born from the competition.

Structural Restrictions. Costs and Debts.
The costs taken on by building this product: the current and future ones. A mature vision of the investment in systems or ML intensive products implies conscientious decisions about the debt we take on to validate a value proposition, not always "visible" at the beginning of the product unless energy is dedicated to identify them (e.g. "they lack a data culture").

Estimating the cost of development of a solution is not only associated with coding time. Some solutions can be simple to develop and costly to test. The story to tell should reflect what technical concessions a MVP of the solution entails so as to gel correct expectations. If we believe that the point of inflection in the story is 10K transactions, we can base our model for doubts sake to support 20K, later needing to be recalculated (but with the principal hypothesis verified).

Business Impact

The impact on business is direct or indirect? The organizational and product objectives are not always transparent in relation to the benefit which the business generates. Also the same combinations of Organizational Objectives + Product Objectives + Value Proposition can pursue an impact or other in the business: for example, increase the value of the company vs increase the cashflow.

What will be the business impact of our solution? Is it saving or earning? Capitalize-able? Is it a vitamin, something to gain an opportunity or a painkiller, something to resolve a need? If we have a tangible impact ( a conversion increment aligned to the revenue objective, for example) we can support our story on the company objective. If the impact is more diffuse and looks to generate engagement, we can associate the story with the positive impact on interactions in the user journey.

Moving away from stakeholders considerations and from the impact on specified user journeys, it is not strange that a Machine Learning intensive product has the potential to expand and grow toward other stakeholders and other corners of users journeys. To articulate in a unique tale the connection between the current value proposition, the expected impact on the business and the growth possibilities, helps that all actors involved can share the direction laid out.

Evaluation and Real Time Monitoring
Legacy of previous models, the idea of this component in the Canvas is to make explicit what factors we observe to guarantee the health of the system. It is not uncommon to observe technical factors such as stuck models, data source corruption and unstable relationships between training vs production (in fact we have internal tools which I hope we can eventually share with the community), nevertheless in this point we intend a slightly different focus: share within the team, technical or not, which of the components mentioned above will be the factors to observe for the value proposition to keep being the coherent axis within the Canvas.

Implies a decision about the risk you wish to manage being online.

If you got here, Congratulations! It is the moment to try it out. Fill in at least 5 fields (with the Value Proposition as the only non-negotiable) and tell someone "from the other side" a story with those five fields. It wouldn't surprise me if the others feel that you speak their language and the value analysis flows.

The article was originally published here

Source: HOB