homehome Home chatchat Notifications


Opening the AI Black Box: Scientists use math to peek inside how artificial intelligence makes decisions

Researchers find a mathematical key to understanding how AI makes decisions.

Tibi Puiu
January 16, 2025 @ 7:13 pm

share Share

Illustration by Midjourney.

Artificial intelligence is everywhere, from the personal assistant in your pocket to self-driving cars navigating busy streets. Yet, for all their apparent intelligence, the neural networks powering these technologies have long been a mystery. AI developers don’t write explicit rules for these systems. Instead, they feed vast quantities of data into these systems and they learn on their own to spot patterns. Since they don’t provide insights that point to their decision-making process, AIs have often been referred to as “black boxes”.

A team of researchers at Western University has now taken a major step toward opening these AI black boxes. By applying advanced mathematics, they’ve developed a method to peer inside the step-by-step output of these algorithms and understand their decision-making processes.

Their work marks a milestone in the quest for so-called explainable AI.

“We create neural networks that can perform specific tasks, while also allowing us to solve the equations that govern the networks’ activity,” said Lyle Muller, director of Western’s Fields Lab for Network Science. “This mathematical solution lets us ‘open the black box’ to understand precisely how the network does what it does.”

A New Way of Seeing

The researchers demonstrated their method by tackling image segmentation, a key process in computer vision. Image segmentation enables systems to divide an image into distinct parts, such as separating a polar bear from the snowy background. But how does a computer decide where one object ends and another begins?

Starting with basic shapes like squares and triangles, the team created a recurrent neural network (RNN) capable of identifying these components. Using a mathematical approach developed for studying other types of networks, they mapped how the neural network processed each image step by step.

The innovation introduced by the researchers hinges on the neural network’s ability to operate with complex numbers. This design, inspired by natural processes like brain waves, creates dynamic, flowing patterns across an image. These patterns, or “traveling waves,” separate different regions, such as a triangle from its background.

Surprisingly, their network was able to extend this understanding to more complex scenarios, such as photographs of wildlife in natural settings.

“By simplifying the process to gain mathematical insight, we were able to construct a network that was more flexible than previous approaches and also performed well on new inputs it had never seen,” said Muller.

The flexibility of this new network comes with a significant bonus: it can be explained. Unlike traditional neural networks, whose operations are often opaque even to their creators, this system reveals exactly how its computations unfold.

From the Cortex to the Computer

The team drew inspiration from neuroscience. The visual cortex, a brain region processing visual information, exhibits traveling waves during sensory tasks. These waves aren’t just noise — they carry computational meaning.

By modeling these waves mathematically, the team created a network that doesn’t require repeated training to handle different images. Instead, a single set of parameters guides the RNN to identify objects in simple geometric patterns, natural scenes, and everything in between.

The network, called a complex-valued RNN (cv-RNN), uses two key features:

  1. Amplitude and Phase: Each pixel is represented as a complex number, with its brightness encoded in amplitude and its group (object) determined by phase.
  2. Recurrent Connectivity: Pixels influence their neighbors, much like neurons in the brain. This interaction generates waves that flow through regions belonging to the same object.

These dynamics unfold over two layers. The first separates objects from the background, while the second distinguishes individual objects. The results are astonishingly clear, even for overlapping shapes or detailed natural scenes.

The team even connected their network to a living brain cell in collaboration with physiology professor Wataru Inoue. The result was a hybrid system that merged artificial and biological neural networks, opening the door to futuristic possibilities in neuroscience and AI.

“This kind of fundamental understanding is crucial as we continue to develop more sophisticated AI systems that we can trust and rely on,” said Roberto Budzinski, a post-doctoral scholar on the team.

Why It Matters

Neural networks have revolutionized industries but remain a double-edged sword. Their power comes with uncertainty, as their lack of transparency has raised concerns about trust and safety. By creating systems that are both effective and explainable, researchers are paving the way for more responsible, better AI.

“This is just the beginning,” Muller said. “We believe this mathematical understanding can be useful far beyond this first example.”

As AI continues to shape the world, understanding how it thinks is more than an academic exercise — it’s a necessity. This work is a step toward ensuring that the technologies of tomorrow are not only smarter but also safer and more accountable.

The findings were reported in the journal PNAS.

share Share

Largest Study of Its Kind Finds How Long-Term Cannabis Use Affects Memory

The study looked at the effect of cannabis use on young adults who are recent or heavy users.

This Moth’s Wings Create a Mind-Bending 3D Optical Illusion to Avoid Being Eaten

A moth's flat wings fool predators into seeing an inedible 3D leaf.

Scientists Just Linked Two Quantum Computers With "Quantum Teleportation" for the First Time and It Changes Everything

The future of computing might not be one giant quantum machine but many linked together.

Human-like Teeth Grown in Pigs Could Make Dental Implants a Thing of the Past

It's a glimpse into the future of tooth replacement.

Paleolithic culture cannibalized their enemies — and maybe their friends as well

In the 19th century, archaeologists in Poland unearthed a stunning cave filled with prehistoric secrets. The Maszycka Cave, as it’s called, once sheltered Magdalenian people 18,000 years ago. Now, a new study offers compelling evidence that the cave was the site of a grisly ritual — or perhaps something even darker. Did these ancient people […]

AI Is Supposed to Make You More Productive — It's Making You Dumber and Overconfident

Generative AI is supposed to make life easier. It drafts emails, summarizes documents, and even generates creative content, helping you offload some of that dreaded cognitive effort. But according to a new study from Carnegie Mellon University and Microsoft Research, it may also be making you dumber in the process. The study, based on a […]

Beavers Built a $1.2M Dam for Free — And Saved a Czech River

A Czech project that was stalled for years is now completed — by beavers.

A Single High Dose of Creatine Might Help the Brain to Power Through Sleep Deprivation

From the gym to your brain: the surprising new use of creatine.

A Spoonful of Peanut Butter Might Be the Key to Overcoming Peanut Allergies

A new study suggests that children with peanut allergies may be able to safely build tolerance through a simple, cost-effective treatment.

Inside China's 600 MPH Floating Train Faster Than a Boeing 737

It's basically a Hyperloop design on steroids.