Using classic ML to empower AI agents

These days at work I feel like sometimes they are surprised when I say “build CatBoost classifiers” or “customize isolation forests”. But that’s exactly what I spend a lot of my time doing. My company’s product is a sophisticated agent AI platform, but I don’t do much prompt engineering in my everyday life.

If you had asked me a year or two ago, I would have been seriously worried about the future of classic ML work in machine learning because we were so deep into a paradigm of using LLMs to get things done regardless of whether they did good work or not. However, it turns out that agent AI needs classic ML a lot more than we probably thought.

For a quick summary in case readers are unfamiliar:

An AI agent means combining LLMs and other software tools together to create workflows with minimal or no human intervention and orchestrating any number of models or tools.
The LLM is typically the interface between human users and all other software tools, including tasks such as translating human prompts into computer language. The LLM also interprets tool outputs and selects which tools to invoke at appropriate times.
This allows for much more functionality than an LLM alone, because as I’ve mentioned here many times, an LLM is simply a token-generating model that predicts the next word or phrase in a passage based on its context.
Aside from autonomous agents, combining an LLM interface with other tools is required to do a variety of the things we colloquially think of as LLM chatbots. ChatGPT, Gemini and Claude do something like this by chaining the LLM interface to things like data retrieval, web search, math calculators, etc.

As you can see, tooling is an important aspect of the entire agent AI ecosystem – your agent needs to have access to tools to effectively complete tasks outside of the core LLM functions.

These tools can take all sorts of forms – many of them in business today are data retrieval and organization tools, graph databases, RAG knowledge bases, query construction and validation, etc.

Table of Contents

Why classic ML

However, I would like to remind you that classic ML models can also be very valuable tools for your agent. Go one step further than just calling rudimentary tools and provide models to your AI agents! For example, consider a broker who is designed for real estate analysis. If you want to know the fair market price for a property, give your agent the address. An API retrieval tool can be used to retrieve details about the property and then pass these formatted details to a regression model that generates a price estimate.

Of course, you could theoretically ask the LLM to simply estimate the values themselves. However, this is questionable and even risky for several reasons.

accuracy: An LLM is particularly bad for any task that requires you to calculate a meaningful number – it involves guesswork rather than performing an empirical, evidence-based calculation. A well-trained classic ML model will be much more accurate and trustworthy.
Interpretability: They have minimal interpretability and explainability by the LLM conjectures. We know that LLMs tend to be a black box and this significantly limits your ability to assess the journey you have taken to arrive at the quote you have received. A classic ML model allows you to identify the decisions you made to arrive at your conclusions and validate them against your expertise.
Cost: Running an LLM becomes expensive very quickly (see my articles from the last few months on token costs). When you have a lot of cases to handle, token prices quickly become significant. Running a classifier or regression model is incredibly easy and inexpensive, even at large volumes. Additionally, in an LLM you don’t have control over the cost of each call, and token usage and spending can increase – we’re already seeing this across the tech industry.
precision: You have no control over the training or optimization of the LLM (unless you optimize a base model). You can trust the generic LLM to do the work, but you are taking a big risk and as mentioned above, validating the work is extremely difficult. On the other hand, fine-tuning a base model could be effective, but requires much more data and more specialized skills than just training a regression or classifier and still introduces interpretability issues.
Control over your data: Your data may leave your controlled environment and be accessed by a third-party LLM model provider, which may pose a risk.
Control of infrastructure: With an LLM, you have no authority over infrastructure management, so third-party downtime poses a risk to your business.

Of course, creating a classical model requires different skills than simply setting an LLM task. You must understand your data well, be prepared to perform feature engineering with expertise, and have sufficient computing power and data to train the model. If you don’t have labeled data, you’re limited to unsupervised learning or perhaps bootstrapping your own labels. Fortunately, there is a wealth of content on how to build these models and how to thoroughly evaluate and monitor them after deployment.

Connect your model with your agent

You may be convinced to give it a try, but before you get started, you also need to consider some architectural decisions. How will your model and agent interact?

Direct calls

Perhaps the quickest way to get started is to simply provide the model as a tool to the agent to call directly. This is the form of my real estate research tool example: The agent can invoke a model for just-in-time inferences based on a prompt. To set this up, your AI agent must be able to correctly adapt its requests to the classic model. Your agent needs to understand what this model is for, when to invoke it, and when to use something else. This means clearly documenting the purpose and capabilities of the model. However, if you are already building an agent AI, this is a familiar task.

When output, your model response must be structured in a way that the AI agent can process it effectively. Simply returning a numerical result may not be enough as the agent needs contextual information to interpret and make best use of it. For my models, I often use F-strings to create text descriptions as part of the inference, indicating for example what the key features of the model were, what the probability of the outcome is, and so on. Simply returning a probability limits your agent’s ability to interpret the output and create a useful answer for the end user.

Database access

Another possibility is to make the model not a direct tool of the agent, but a provider of contextual data. You can precompute the inferences by running your classic ML model as a scheduled job and storing those inferences in the data storage solution your agent has access to. Instead of the agent making an initial inference call directly to a model API, it writes a query and passes it to your database.

If you have a finite set of cases that you could potentially need conclusion completion for, this can be a good solution. For example, if you have 500 people in your database and you want your agent to retrieve information about their financial health, you could use a credit scoring model and pre-calculate their creditworthiness so that the agent retrieves it at runtime along with all the other data it collects. Depending on your model infrastructure, this can reduce latency and retries by effectively caching the results.

This approach has different invocation and retrieval requirements than direct tool invocation. If you pre-calculate the conclusions and simply provide them to your agent through a database, your agent needs to know that these results exist. If the table or available content is not known, it will not be used when necessary. You may already have the infrastructure in your prompt engineering to tell the agent what the database contains when your agent calls the database for other information. So by reusing you can save yourself from duplicating work. If not, your agent may need special tools to check the database metadata.

As for the result format, the requirements are similar to the case of direct tool access. A text-based description of the results is a good choice because the agent needs to be able to interpret what it is retrieving, regardless of the source.

Diploma

Before LLMs came along, classic ML models were the cutting edge capabilities across many different industries for over a decade, giving people insights into data they otherwise wouldn’t have been able to gain. This performance should not be neglected but can be combined with the capabilities of LLMs. We can leverage the strengths of the LLM by converting human language into computer language, stringing together various tool calls, and getting results from those tools, and within that framework still use classical models to do the work that an LLM is not suited to.

The barrier to entry is the ability to build high-quality classic ML models, which unfortunately isn’t as glamorous as some of the AI-related work these days. However, the effort is worth it because of the benefits: accuracy, precision, interpretability, cost and control. I recommend practitioners brush up their skills with tools like XGBoost, LightGBM and scikit-learn to see the results for themselves.

Useful Links

Edtior's Picks

Latest Articles