How to track your AI ROI ?
One of the most pressing issues data and analytics leaders are facing is how to optimize business value from their AI investments. If most companies do not achieve maximum leverage from their AI investments, despite increased spending, it is largely because they struggle with the ability to assess the performance of your AI investment.
Prior to digital industrialization, almost all companies tracked their return on investment using metrics that were optimized for real assets, as exemplified by the Return on Investment (ROI).
Generally used by financial investors to evaluate the profitability of an investment, what is the particularity of the ROI when it applies to the field of AI? And more importantly, what kind of tools should be used to track your AI ROI?
This article summarizes everything you need to know before assessing the performance of your AI investment.
Key learnings
- The ROI (Return on Investment) is a common measure used to calculate the effective return on profit compared to the money invested.
- Applied to Artificial Intelligence projects, ROI is a relevant key performance indicator that allows companies to judge the opportunity to launch a new project and evaluate the potential gains and eventual risks related to this project.
When calculating AI ROI, companies must take into account both “hard” and “soft” values created.
What does ROI mean in AI ?
The Return on Investment or ROI is the financial ratio that is calculated to know the relative return on profit compared to the capital invested. In any type of investment project, you are considering risks and returns. Making a smart investment is estimating stronger benefits than the costs.
For AI this is specific to tech-related questions and subjects, related to helping maximize the ROI by reducing the uncertainty. For example, there are subjects of IT resources, data availability and quality, what stakeholders may be involved and how the business goals are in line with the tech-feasibility.
In companies, you will generate a positive ROI from your AI initiatives, by implementing key practices in tracking your results, security, privacy and management. In the industry, the most performant areas for positive ROI from AI are in customer service and experience (74 %), followed by IT operations and infrastructure (69 %), as well as planning or decision-making (66 %). On a world-basis, there is an average 4.3% ROI according to Deloitte in leader-companies.
Unlike traditional financial projects, the benefits of an AI investment are not limited to their financial dimension. Indeed, two kinds of ROI must be defined when it comes to measuring AI : “hard ROI” and “soft ROI”.
On the one hand, hard ROI refers to the standard benefits studied when calculating the profitability of an investment. For example, revenue increase can come from the implementation of new services developed thanks to augmented intelligence, as well as productivity improvement can arise from gain in effectiveness and efficiency allowed by assisted intelligence.
Automated intelligence can also have an impact on time savings (automation of repetitive tasks). All of these improvements can also generate cost savings (lower staff needed).
On the other hand, soft ROI allows to emancipate the ROI from its purely financial perspective. It integrates broader benefits such as social, health or environmental-related values. These soft returns can include skills retention for example, a major challenge for any company. It can also include the experience of the customer, to continuously provide the best service or product, and help them stay in business.
Why should you track your AI ROI ?
Tracking the AI ROI allows a company to estimate the potential gains and the eventual risks of an AI investment. Such estimations are essential in order to size the initial budget and to judge the opportunity to launch the project.
Calculating the AI ROI will be necessary in one way or another to provide an answer to the following questions : What is the potential value of the investment? What do we risk if we do not implement the project ? What are the potential quantitative and qualitative gains of the investment ? Calculating the ROI of an AI project also enables a company to know if it is mature enough to enter a new market or to launch new offers and encourages to widen the scope of possibilities and think about the evolution of its R&D model.
How to measure ROI of AI ?
AI is shaking up technological, economic and societal environments and will continue to change organizations with an expected growth rate of 18.8% for 2022.
A large variety of AI tools are constantly developed and are difficult to apprehend by the companies themselves. Faced with this fact, companies need to manage ROI and risk tracking before triggering any AI investment.There are six key stages to follow to track and optimize ROI for AI investments.
Step 1 : Estimate revenue gains
AI projects such as production line optimization, packaging optimization, quality control automation, or image recognition will lead to revenue gains that can be easily estimated with assumptions on KPIs such as new production capacities, amount of new products sold, number of newly acquired customers, etc.
A very important element to keep track of is the gain in productivity through the provision of automation. This will as well impact on cost reductions in the duration of processes.
When it comes to obtaining hard numbers to calculate the ROI from AI investment, many executives have lost sight of the idea of the variability of performance/revenue and of the KPIs that express the real goals of the investment.
The common mistake is to focus on estimating the best case scenario and only consider the attractive outcome of an investment based on a set of assumptions that are highly favorable to the project. That being said, estimated revenue gains must take into account both the risk of false negatives and the cost of false positives for classifiers (or recommendation tools).
Step 2 : Identify costs associated with the AI project
Correcting budgeting and cost control are critical to your AI ROI calculation. A rough calculation of …
- Tasks (what will be done),
- Resources (technology stack for example),
- Rates (at what billing rate, in what currency, with what tax amount),
- Duration (for how many hours or days),
- Team composition,
- Number of iterations (training),
- Implementation and maintenance cost,
- Potential amount of data to have to collect
Can be used to generate an estimate of total costs associated with the AI project.
AI costs are specific because they include the production costs (building and feature engineering), the running costs and the costs of error. The variability of the building costs must be considered since costs for adjusting the models can vary from very high to “not affordable” and storage costs can rapidly become exorbitant.
As for the running costs, it can be difficult to estimate the platform size that will be necessary to meet the service-level agreement (SLA).
The final bill depends on the project’s complexity, but before estimating it, companies must be aware of the risk of “bill shock” i.e. unexpected charges and high bills. A risk that is minimized with Aleia’s solution.
Step 3 : Determine intangible gains
Intangible gains are part of the “soft” ROI calculation and are not easy to quantify. However, increasingly more project sponsors are pushing to translate them into something measurable regardless of the fact that it brings a business edge.
In fact, “soft” values are much more specific to each AI project than “hard” values. These metrics should be determined according to each particular project and the related objectives.
For example, brand recognition or employee loyalty have a real impact on a company. It is important to value intangible benefits as the company goes through technology investments as it has been observed it truly impacts positively on the long run.
To evaluate and leverage on these intangible benefits, companies should rather have a qualitative approach rather than a quantitative one.
As an example of intangible gains, by automating certain tasks, AI assists and augments the human and therefore reduces many physical and mental disorders such as burn-out. In the same way, the company will be better off because human errors will decrease, together with turnover.
Step 4 : Fix a realistic and sustainable time frame for measuring results
Setting a time frame for measuring results is a key element in AI because AI projections never stop ! The model developed will not be valid forever so it will need a maintenance period which represents additional costs that must be taken into account.
Step 5 : Develop a status quo scenario for the same time period
When calculating a return on AI investment, it is necessary to measure what it may cost not to carry it out. We are no longer talking about ROI, but about RONI (Risk Of Non Investment)! What will be the financial impact of not having invested more in an AI solution? These are the kind of questions that the concept of risk of non-investment is able to solve by providing quantified estimates.
Step 6 : Calculate the cost/benefit ratio for the AI scenario
Measuring the difference of the cost/benefit ratio between the AI scenario and the status quo is the best way to determine if your AI project will generate significant improvements as compared to the current situation.
3 big mistakes to avoid when computing ROI
Before starting to track your ROI on AI projects, you should be aware of four common mistakes that are constantly made by companies.
Consider that the investment will necessarily result in gains
Some companies do include hard and soft returns in their AI ROI calculation but forget to take into account the error bias that is inherent to any artificial intelligence development project.
Many company leaders are far too optimistic thinking that if they build AI projects, profits will necessarily come. Indeed, it is well known that AI models are prone to error, and their accuracy is less than 100% guaranteed.
When calculating an AI ROI, both the error rate and the estimated cost of these errors must be integrated to reach a level of ROI that is the closest to reality. The task of estimating the margin of error and its cost is far from being an easy one!
The margin of error generally results from a comparison between a baseline of human performance and the performance of the AI model, while the related costs are usually estimated on a sliding scale.
Use a “point-in-time” method
As mentioned earlier, AI projects are part of long-term business investments. Thus, a big mistake to commit would be to calculate an AI ROI at a given moment that is way too early, for example, only a few months after the deployment of the AI system.
The average ROI for most experienced companies is 4.3%, whilst companies first getting started with AI projects is 0.2%. On average, the payback period will vary between 1,2 and 1,6 years. This is the point at which the net benefits will equal the initial investment.
Calculating your ROI too early, would generate an inaccurate ROI, as it would not consider the potential deterioration of AI models performance over time. Every company investing in AI projects must measure AI ROI on an ongoing basis so that the value of the AI model doesn’t degrade and erode the gains already made.
To conclude, AI investment models will differ across industries and organizations. There is no generic best practice applicable to all when it comes to optimizing AI ROI, but best practices in AI investment will develop as organizations become practiced at strategy-driven allocations.
Aleia is an open, industrialized, digitally sovereign and secured platform to create, deploy and operate your Artificial Intelligence applications. Don’t hesitate to contact us, to learn more about our methodology.