The public release of ChatGPT threw the world into a frenzy, and users started assigning all sorts of personal tasks to the new expert in town. Within 5 days from launch, ChatGPT had a million users, which is beyond impressive. For reference, check out the other 4 fastest growing apps in the top 5. All are in the entertainment space, while a productivity app stands alone, showcasing just how much its value addition was in people’s lives.
ChatGPT is the only app on the list not in the Entertainment Space
This launch sparked a greater debate around the advent of AGI, or Artificial General Intelligence. AGI is a model that can rival human intelligence in a wide range of tasks. If I were to put it in more quantifiable terms, an AGI will rival the human most skilled at a particular task in all possible tasks to be done. Creating an AGI is often said to be the last invention of mankind, since the AGI will be better at inventing new technology than the most skilled human out there.
Even though ChatGPT was able to complete a very wide range of tasks better than an average human, experts doubted that LLMs were the final step before achieving AGI. There have been several numbers put out in the media regarding when we will be achieving AGI, including Elon Musk’s most aggressive estimate of 2025.
However, the actual number might be far, far away. Chief AI scientist at Meta, Yann Lacun, argues that LLMs are not the way to achieve AGI. LLMs merely learn from human language, and the skills required to outsmart humans include a more multi-modal sourcing of information. The quartet of human intelligence includes ability to plan, reason, memory persistence, and understanding of the world. LLMs lack 3 of these, exception being memory persistence.
So what is the current scope of LLMs in the industry?
The impact that LLMs have had on the world since the launch of ChatGPT cannot be overstated. But how much difference are they really making in the industry?
Only 14% Enterprises have at least 1 LLM in production, rest are still experimenting
This data shows a promising future, but take a look at how many enterprises have actually LLMs in production. Less than 15%! So even the latest models like GPT 4 or Gemini Ultra are not enough to give confidence to organizations that they can be productionized. This is because organizations are structured to operate at a six sigma (99.999%) accuracies when dealing with autonomous tech, whereas LLMs operate at an accuracy of 95% at best for even easy tasks. Currently organizations cannot hand over any task to an LLM, and must adopt a Co-Pilot approach instead, that is, have a human in the loop to overlook the work done by an LLM before taking any action.
Then what is the way forward for increasing AI adoption in Enterprises?
Rather than looking at a single General Purpose AI to complete all the tasks in the industry, a mixture of Objective Oriented AI models will lead to much greater efficiency today. Objective Oriented AI models are trained for specific tasks that they can do with much higher efficiency and accuracy than a General Purpose model trying to do that task. The downside being Objective Oriented models are completely unable to handle queries outside of their scope of data.
However, if we look at how enterprises function, 99%+ tasks that an execution level worker sees are not unprecedented. So, is it really prudent to employ a General purpose model just so that the long tail can be addressed? The data above shows not.
Hence, with the current capabilities of General Purpose Models, the architecture that enterprises would look at to generate high accuracy of tasks at a low costs would be a mixture task oriented models. The purpose of a language model can be limited to a small percentage of tasks which require logical reasoning, while the rest can be be addressed with task oriented models.
Now, lets say we used a mixture of task oriented models to achieve 99%+ accuracy for a range of tasks. The next step is to bring all stakeholders on the same page regarding the limitations and the benefits of deploying this AI model. 99% may sound sufficient to convince anyone on adoption, but in the organization that I worked with at Amazon, each bps (0.01%) decrease in model accuracy resulted in a loss of~$15M for the organization. So leaders are not ready to adopt even such high accuracy solutions unless you add an extra layer of convincing.
One way to do this is to benchmark human accuracy against the model. If the model is performing at par with the most skilled humans on the job while incurring significantly lower costs, it should be accepted by default. However, if the model is not performing on par with expert humans, then the effectivity of the model must be proven through the overall ROI calculations.
Costs and Accuracy of Humans and Machines modeled
In this case if Y + (1-a) * Z < X + (1-b)*Z, or if X - Y < (b-a) * Z, then the solution must be adopted.
Doing such a cost benefit analysis turns exponentially more complex as the workflows become complex.
Not even all Fortune 500 have an internal team that is sufficiently equipped to study the market and do such cost-benefit analyses with accuracy for their own company. The only option is to search for 3P partners who have expertise and experience having implemented such projects for other customers. The only problem with this approach is lack of alignment of incentives between the partner and the company, each wanting to maximize their own profits.
Aligning the Incentives between Partners and Enterprises
Incentive alignment is a business problem more than a technical problem. Even after finding partners who you can trust to do right by you, it is important to align monetary incentives. Essentially, the monetary pie that the 3P partner is helping you create should be fairly split. Lets explore multiple pricing strategies employed by such partners, and which one would likely stick around in the long run.
Hourly Billing
In AI Projects, the outcomes are often uncertain. The risk of uncertainty is borne by the customer entirely in this case. However, if the value created is far too great, then the entire reward also goes to the customer.
In this case, aligning incentives is difficult. For the partner to maximize profitability, they will propose multiple approaches and conduct multiple cost benefit analyses so that the project draws long. As long as the partner knows that the pie they are going to create is higher in value than the overall price they are charging the customer, they are guaranteed to satisfy the customer.
The entire power lies in the hands of the partner once such a pricing option is agreed upon. Hence, this is unlikely to work in the industry.
Upfront Pricing
Getting an upfront estimate of the project value from the partners helps the customers plan better. The partner has an incentive to inflate the cost of the project, but also faces downward pressure from competitors who might outbid them in case the quote goes too high.
However, now the partner has an incentive to finish the project as quickly as possible once the project cost is fixed. This can result in sub-par performance, and the company can miss out on better alternatives that may have been found if the partner was allowed more time. Hence, this pricing option is also unlikely to stick around.
Success Based Pricing
Success based pricing attempts to link the success of the customer with the success of the partner. There can’t be one without the other. This is an attempt to fairly split the value created by the partner. The ambiguity arises in defining success.
If the partner is merely an implementation partner, then the success should be limited to metrics like Accuracy of the Model, Uptime, Latency, etc. However, if your partner is an innovation partner who is partnering with you to deliver business success, then the only success metric worth looking at is improvement in the bottom-line. However, this is difficult to calculate accurately since the bottom line is influenced by multiple external factors. So, a better metric for success usually becomes reduction in Cost Per Task (CPT) that the AI model is supposed to automate. Although this can also be influenced by other factors like better training pipelines for humans, those are negligible in comparison with the cost savings that are brought about by automation. Now, the amount paid to the partner increases with the % reduction in CPT. Now, the partner has an incentive to add maximum value in the minimum amount of time, and only those partners who are confident about delivering reduction in CPT will be willing to take on such projects. Everyone wins.
Hence, I believe strongly that Success Based Pricing is the way forward in the AI consulting industry, not dissimilar to the transition that happened in the management consulting industry a few years back. From hourly billing, the Big 4 and Bain, BCG, McKinsey now get paid for the value added for the customer. In AI Consulting, there are very few players confident enough to take this bet, since the field is new, and there are few players who understand both the business as well as technical aspect to take this bet. Tailored AI is one of the only players from India to take this bet of guaranteeing business outcomes to customers, or not getting paid at all.
Concluding Remarks
I strongly believe that AGI is at least a few years away, and a single AI model is not going to replace humans in enterprises any time soon. What will replace them, is a combination of task oriented models, and who will drive the adoption of these models are AI implementation partners who are willing to share risk of implementing projects with their customers.
Transform your operations, insights, and customer experiences with AI.
