A Look at the ‘World’s First’ Full AI-Based Image Signal Processor

Two companies are collaborating to create the “world’s first full AI-based image signal processor” to replace the hardware-based ISPs that have been core to digital imaging for decades. Chips&Media, a Korean IP provider for image processing, is working with Visionary.ai, an Israeli startup focused on low-light image processing, to develop this new ISP.

The collaboration aims to use AI to move the entire image formation process into software running on neural processing units. Both companies see this as a way to tune, retrain, and update video processing in real time. While there are implications for still photography, both companies have identified low-light video needing more of this kind of structural shift.

From Fixed Hardware to Software-Defined Imaging

ISPs are common in many cameras in the digital age, but their overall hardware architecture hasn’t changed much over time. Chipmakers largely build these to complete mathematical stages that leave little room for tinkering outside the factory, aside from manual per-sensor tuning. The two companies see this as a limiting factor because it no longer scales with the imaging demands consistent with expansion from smartphones into autonomous driving, XR devices, and even mirrorless cameras.

“This is the first full end-to-end ISP pipeline that runs entirely on an NPU, without relying on a hardware ISP at all,” says Oren Debbi, Visionary.ai’s co-founder and CEO, in an interview with PetaPixel. “Existing pipelines bolt neural blocks onto a fixed-function ISP. We replace the conventional ISP entirely with an end-to-end neural imaging pipeline.”

That means it processes RAW sensor data directly on an NPU or GPU. Since it’s all software-based, there’s wiggle room to adjust tuning and optimization through over-the-air updates that bear no effect on the actual silicon.

Central to this approach is sensor-specific training. Visionary.ai trains a custom neural network for each image sensor, but has developed an automated training platform that can produce a new model within a few hours using only a small number of short video clips. Debbi says this significantly reduces integration overhead and allows the company to scale across sensors and platforms without the lengthy tuning cycles associated with classical ISPs.

AI-enhanced ISPs already play a role in smartphones and cameras, though both companies argue those systems are still overtly hardware-centric. Manufacturers usually add neural networks as isolated blocks, only they don’t process core RAW data because fixed-function hardware and mathematical pipelines handle that load instead.

“The image formation pipeline is neural-first, not a classic ISP with a few AI add-ons,” says Debbi. “Some camera control functions can remain conventional today, but the core image pipeline no longer depends on fixed-function hardware.”

That does suggest a hybrid approach where neural networks handle image formation while
camera control functions, like exposure and white balance, still get the conventional treatment. Debbi notes that AI-based solutions for those components already exist, yet expects them to mature rapidly.

The advantage, he adds, is that a neural-first pipeline means fixed hardware blocks or manual parameter tuning should no longer constrain image quality improvements. If manufacturers have the flexibility to update, specialize, and retrain solely through software, they can optimize imaging output per sensor and use case while also addressing power and latency concerns.

Image Quality in Challenging Conditions

Low-light conditions offer the most visible improvement, he adds. Standard ISP pipelines often have to suppress noise and lose fine detail, thereby forcing sharpening algorithms across the entire frame that may make an image look artificial or introduce artifacts like halos and pixel bleed.

“You see the biggest difference in the hard cases where classic ISPs have to trade off detail, noise, and artifacts — very low light, high dynamic range, and mixed lighting,” says Debbi. “Practically, that means cleaner shadows without waxy textures, fewer halos and oversharpening artifacts, more stable color, and fewer temporal artifacts in video. Because the pipeline is learned end-to-end, we can optimize for perceptual quality and stability across scenes, not just isolated blocks like denoise or HDR.”

In addition, the neural pipeline is designed to adapt to scene dynamics to reduce ghosting and shimmer without sacrificing natural detail when subjects move, a long-standing challenge for multi-frame classical pipelines.

While the current product focus is firmly on video, Debbi acknowledges that still photography can also benefit from a full AI-based ISP. He says market demand and deployment opportunities have so far centered on video-centric use cases, but the underlying architecture is built around processing sequences of images to achieve the best results.

Since phone cameras often bracket and stack images to produce everything from HDR and low-light images, decoupling some of that process from the hardware could theoretically deliver improved results.

Visionary.ai acknowledges this by noting that most on-device neural imaging today happens after the ISP, operating on YUV or RGB data where “significant sensor information” has already been discarded. Debbi believes his company’s expertise lies in efficient RAW-domain processing, either by replacing the ISP entirely or by integrating into an existing pipeline “Bayer to Bayer” to perform specific functions such as AI denoising.

Beyond that, the software-defined AI ISP can effectively fill that gap for platforms with limited or no ISP hardware, enabling those chips to support camera capabilities they wouldn’t otherwise be able to reach.

Balancing Performance and Power

The thing about AI-based imaging is that it can draw power when consistently running in the background. The system also supports different operating modes, allowing manufacturers to trade power for quality depending on the application.

“We’re able to run on a very small NPU and consume only slightly more than imaging with a traditional ISP, and that gap continues to shrink,” says Debbi. “As NPUs get stronger and our models continue to optimize, we expect this to consume even less power than hardware ISPs.”

Chips&Media’s WAVE-N NPU is designed for high-throughput vision workloads, serving as a full reference implementation for the AI ISP, demonstrating an end-to-end neural imaging pipeline running in real time on video-focused AI hardware.

At the same time, the AI ISP itself is hardware-agnostic, so manufacturers can map the software pipeline to a wide range of NPUs or GPUs depending on their SoC architecture, power envelope, and cost targets. They can also deliver substantial imaging improvements over time, including better HDR, improved exposure fusion, enhanced segmentation, and use-case-specific modes for applications, ranging as far wide as automotive night driving or video conferencing.

Fitting In With Existing Hardware

Despite attempting to disrupt the way ISPs work in cameras and imaging devices, both companies recognize fixed-function ISPs won’t disappear overnight, “but the center of gravity is clearly moving toward programmable AI compute.”

Being software-based, integrating this AI ISP largely depends on where an OEM is in chip development. For existing silicon, Visionary.ai can deploy it “within months” through software integration alone. For chips that are still pre–tape-out, moving more imaging functionality into AI can reduce dedicated ISP silicon area within the same generation.

“Software updates faster than silicon, adapts better to new sensors and use cases, and ultimately reduces cost and complexity,” he says. “The winners will be the solutions that hit real-time latency, power, and consistent visual quality at scale.”

The companies do not expect fixed-function ISPs to disappear immediately, but they believe the long-term trajectory is clear. As AI compute becomes more capable and deployment tooling matures, software-defined imaging pipelines are expected to overtake classical ISPs across many categories.

By debuting a full AI-based ISP at CES 2026, Chips&Media and Visionary.ai are positioning their collaboration as an early indicator of that shift, one that could reshape how image quality is delivered, updated, and scaled across the imaging industry.

View Original Article