Where AI Breaks Down—and How to Fix It
By Michelle Fiscus, Senior VP & Chief Communications Officer
AI systems are often only as good as the data they’re trained on, but the gap between training data and real-world conditions isn’t always due to oversight. In many cases, the most challenging or edge-case environments are difficult to capture because collecting such data can be risky, expensive, or logistically demanding. As a result, models are frequently trained on cleaner, more controlled datasets and then deployed in more complex settings where this unavoidable mismatch can lead to performance breakdowns.
Dr. Sambit Bhattacharya, a computer science professor at Fayetteville State University, studies what happens when AI leaves the lab and meets reality, especially in cases where the hardest conditions are too risky or expensive to capture in training data.
"AI can look impressive in controlled settings,” he said, “but the real test is the messy world where the data is hardest to get, and that’s exactly where systems tend to break.
The issue isn’t theoretical. It shows up in the kinds of situations these systems are increasingly used to support.
In maritime monitoring, an AI system may be tasked with detecting unusual vessel behavior under difficult conditions — low visibility, cluttered environments, or patterns it hasn’t encountered before. If those scenarios aren’t represented in its training data, it may miss the signals entirely.
In healthcare, the same limitation appears differently. AI models used in treatment planning rely on understanding how radiation interacts with different tissues. Without enough representative data, those predictions become less precise.
Across both domains, the pattern is consistent: the most critical situations are often the least represented in the data used to train these systems.
His work is focused on addressing that gap by changing what AI systems are trained on in the first place.
His team is developing a way to generate highly realistic synthetic data — data designed to replicate conditions that are too rare, too complex, or too risky to capture in real-world datasets.
“We’re creating a way to generate highly realistic ‘synthetic’ data, so AI systems can learn from situations that are too dangerous, too rare, or impossible to capture in real life,” he said.
The goal is not just to simulate environments, but to improve how AI systems perform when those environments are encountered in practice.
Better training data leads to more reliable systems — systems that can detect anomalies earlier, support faster decision-making, and operate with greater accuracy in complex conditions.
With support from NCInnovation, the work is moving beyond early-stage development.
“This funding allows us to move beyond early prototypes and build a complete, integrated system,” Bhattacharya said.
That includes testing the technology with defense and healthcare partners, validating its performance in operational settings, and preparing it for use outside the lab.
Without that support, progress would likely remain incremental.
“We could advance individual components, but not bring them together into a validated system,” he said.
In North Carolina, the implications are tied directly to the sectors where these systems are used.
The state has a strong presence in both defense and healthcare, including military installations and major research institutions. Improvements in how AI systems are trained affect the people working in those environments — analysts, clinicians, and researchers — who rely on these tools to make decisions.
“If AI systems are trained on poor-quality data, they can produce unreliable results,” Bhattacharya said. “In defense, that could mean missed threats. In healthcare, it could affect treatment planning and patient safety.”
The consequences are not abstract — they are tied directly to outcomes.
If the underlying problem isn’t addressed, those limitations persist. AI systems continue to perform well in controlled settings but fall short in the conditions where accuracy matters most.
If it is addressed, the change is straightforward: more reliable systems, better-informed decisions, and greater confidence in how these tools are used.