As a child, I, like many engineers, could quickly solve basic math problems in school simply by providing the answers. However, when I failed to "show my work," my teachers deducted points; the right answer didn’t hold much value without an explanation. Strangely, this level of accountability doesn't seem to extend to many AI systems today, even those making critical, life-altering decisions.
The leading AI companies featured in recent headlines — including OpenAI, Google, and Microsoft — primarily utilize black-box models. This means that when a question is posed, an answer emerges without any transparency regarding the data or reasoning behind it. These black-box AI platforms are built on a decades-old technology called “neural networks.” Essentially, these AI models abstract vast amounts of training data but do not directly reference it. Consequently, black-box AIs make inferences based on what they predict to be the most likely answer, rather than relying on concrete data.
Occasionally, this intricate predictive process can lead to issues where the AI “hallucinates.” By its very nature, black-box AI is frequently untrustworthy because it escapes accountability for its outcomes. If you can't understand how an AI arrived at a prediction, there's no way to assess whether it utilized flawed, biased, or compromised information during its reasoning.
While neural networks offer substantial capabilities and are here to stay, another AI framework — instance-based learning (IBL) — is quietly surging in importance. IBL stands in stark contrast to neural networks: it is a form of AI that is transparent, audit-friendly, and easily understood. This approach allows every decision made by the AI to be traced back to the training data that informed it.
IBL offers clarity for every decision because it operates directly from the data itself rather than generating an abstract model. Users can easily audit IBL-based AI, probing into the reasons behind decisions and making interventions when necessary to correct errors or counteract biases.
The effectiveness of IBL stems from its method of storing training data (“instances”) in memory and employing principles of “nearest neighbors” to make predictions based on proximity to existing instances. Being data-centric, IBL facilitates direct comparisons of individual data points, granting insights into both the dataset and the resulting predictions. Simply put, IBL “shows its work.”
The implications for this understandable AI are substantial. Businesses and regulatory entities eager to implement trustworthy, explainable AI can turn to IBL to ensure compliance with industry standards. This approach is particularly indispensable in scenarios where allegations of bias are prevalent, such as hiring, college admissions, and legal contexts.
Numerous companies are already leveraging IBL. For instance, my organization has developed a commercial IBL framework utilized by major financial institutions to detect anomalies in customer data and produce auditable synthetic data compliant with the EU’s General Data Protection Regulation (GDPR).
However, IBL does face challenges. Its primary limitation is scalability, which was also a hurdle for neural networks for decades until advances in computing made them viable. With IBL, every data point must be queried, cataloged, and stored in memory, making scalability increasingly difficult as datasets expand.
Nevertheless, researchers are developing rapid-query systems based on new breakthroughs in information theory, significantly enhancing this process. This cutting-edge technology is enabling IBL to compete effectively with the computational capabilities of neural networks.
Despite these challenges, the potential for IBL is undeniable. As companies increasingly demand safe, explainable, and auditable AI solutions, the traditional black-box neural networks may soon become inadequate. If you're a business owner — be it a small startup or a larger enterprise — consider these actionable strategies to start implementing IBL today:
1. Adopt an Agile and Open Mindset
When working with IBL, it's best to explore your data for the insights it may provide rather than restricting it to a specific task, such as “predicting the optimal price” of a product. Keep an open mind and allow IBL to inform your understanding. It might reveal that while it struggles to predict optimal pricing well, it can effectively identify peak purchase times or customer contact methods.
2. Embrace “Less is More” for AI Models
In traditional black-box AI, a single model is trained for specific tasks, which can lead to a plethora of AI models to manage in large enterprises — this can be costly and complex. In contrast, IBL allows for versatile, multitask analysis. A single IBL model can handle supervised learning, anomaly detection, and synthetic data generation, all while ensuring full explainability.
3. Diversify Your AI Toolkit
Neural networks excel in areas where decisions don’t require detailed explanations or auditing. However, when AI influences significant decisions—such as substantial investment strategies or hiring choices—explainability is crucial. The lack of transparency in black-box decisions can lead to discontent and potentially costly litigation. Therefore, select your AI frameworks based on their application: use neural networks for rapid data processing and IBL for trustworthy, explainable outcomes.
Instance-based learning has been in development alongside neural networks over the past two decades but has received less attention. It's now garnering renewed interest amid the current AI revolution. IBL has demonstrated a capacity to scale while maintaining explainability—a much-needed alternative to the unreliable outputs of neural networks that can generate erroneous information.
As more companies indiscriminately adopt neural network-driven AI, we can anticipate a rise in data leaks and lawsuits stemming from bias and misinformation allegations. When the repercussions of black-box AI start affecting companies’ reputations and finances, IBL is likely to come into the spotlight. We’ve all learned the importance of “showing our work” in elementary school, and it’s time we demand that same level of scrutiny from the AI systems that shape our decisions.
Ranking
