The image sensor is an important component in defining the camera’s image quality. Many real-world applications pushed for smaller pixel sizes to increase resolution in compact form factors.  To address this demand, Sony has been improving its image sensor technology across generations. Over the years, this evolution has been focused on key aspects such as pixel size optimization, saturation capacity, pixel-level noise reduction, and light arrangement.

The advancements in Sony’s sensors have spanned four generations. Of these, Pregius S is the latest technology. It provides a stacked sensor architecture, optimal front illumination, and increased speed, sensitivity, and improved exposure control functionality relative to earlier generations.

Key Takeaways:

• What are the IMX264 and IMX568 sensors?
• The architectural differences between the second-generation Pregius and the fourth-generation Pregius S sensors
• Key technologies of IMX568 over IMX264 in embedded cameras

What Are the IMX264 and IMX568 Sensors?

The IMX264 sensor was the first small-pixel sensor in the industry, with a pixel size of 3.45 µm x 3.45 µm when it was introduced. Based on Sony’s “Pregius” Generation two, this sensor takes advantage of Sony’s Exmor technology.

The IMX568 sensor is a Sony Pregius S Generation Four sensor. The ‘S’ in Pregius S refers to stacked, indicating that the sensor has a stacked design, with the photodiode on top and the circuits on the bottom. This sensor is designed with an even smaller pixel size of 2.74 µm x 2.74 µm.

Comparison of key specifications:

Architectural Description: Second vs. Fourth Generation Sensors

Second-generation front-illuminated design (IMX264)
The second-generation Sony sensor uses front-illuminated technology. In front-illumination technology, the conductive elements intercept light before it reaches the light-sensitive element. As a result, some of the light might not reach the light-sensitive element. This affects the performance of the camera with small pixels.

Fourth-generation back-illuminated design (IMX568)
The Pregius S architecture revolutionizes this design by flipping the structure. The photodiode layer is positioned on top with the conductive elements beneath it. This inverted configuration allows light to reach the photodiode directly, without obstruction. It dramatically improves light-collection efficiency and enables smaller pixel sizes without sacrificing sensitivity.

The image below provides a clearer view of the difference between front- and back-illuminated technologies.

IMX264 vs. IMX568: A Detailed Comparison

Global shutter performance
IMX264 already delivers true global shutter operation, eliminating motion distortion. However, IMX568 introduces a redesigned charge storage structure that dramatically reduces parasitic light sensitivity (PLS). This ensures that stored pixel charges are not contaminated by incoming light during readout.

It results in a clear image, especially under high‑contrast or high-illumination conditions in the high-inspection system.

Frame rate and throughput
The IMX568 has a frame rate that is nearly double that of the IMX264 at full resolution. The reasons for this are faster readout circuitry and SLVS‑EC high‑speed interface. For applications such as robotic guidance, motion tracking, and high‑speed inspection, this increased throughput directly translates into higher system accuracy and productivity.

Noise performance and image quality
Pregius S sensors offer lower read noise, reduced fixed pattern noise, and better dynamic range. IMX568 produces clear images in low‑light environments and maintains higher signal fidelity across varying exposure conditions.

Such an improvement reduces reliance on aggressive ISP noise reduction, preserving fine image details critical for machine vision algorithms.

Power consumption and thermal behavior
Despite higher operating speeds, IMX568 is more power‑efficient on a per‑frame basis. Improved charge transfer efficiency and readout design result in lower heat generation, making it ideal for compact, fanless, and always‑on camera systems.

System integration considerations
IMX264 uses traditional SLVS/LVDS interfaces and integrates well with legacy ISPs and FPGA platforms. IMX568 requires support for SLVS‑EC and higher data bandwidth. While this demands a modern processing platform, it also future‑proofs the system for higher-performance vision pipelines.

What Are the Advanced Imaging Features of the IMX568 Sensor?

Short interval shutter
IMX568 can perform short-interval shutters starting at 2 μs, which helps reduce the time between frames by controlling registers. This allows the cameras to capture images of fast-moving objects for industrial automation.

Multi-exposure trigger mode
The IMX568 allows multiple exposures within a single trigger sequence. This feature allows obtaining several images of the same scene at differing exposure times, both in illuminated and dark areas of the object. This reduces dependency on complex lighting and strobe tuning.

It enables IMX568-based cameras to handle challenging lighting conditions more effectively than single-exposure sensors in vision applications such as sports analytics.

Multi-frame ROI mode
This multi-ROI sensor enables simultaneous readout of up to 64 user-defined regions from arbitrary positions on the sensor.

In the image below, you can see how data from two ROIs have been read from within a single frame. The marked areas represent the ROIs.

Full Frame

Selected Two ROIs

Cropped ROIs

e-con Systems’ recently-launched e-CAM56_CUOAGX is an IMX568-based global shutter camera capable of multi-frame Region of Interest (ROI) functionality. It supports a rate of up to 1164 fps with the multi-ROI feature.

This can be very useful in real-time embedded vision use cases, where it is necessary to focus only on a specific region of the image. e-CAM56_CUOAGX can be deployed in traffic surveillance applications where the focus should only be on car motion, facial recognition applications. That way, only the facial region of the subject can be zoomed to achieve superior security surveillance.

Short exposure mode
The IMX568 supports exposure times that can be very short while ensuring image stability and sensitivity at the same time. Exposure times for this mode may vary by up to ±500 ns depending on the sample and environmental conditions, as well as other factors such as temperature and voltage levels.

Dual trigger
The IMX568 enables dual trigger operation, allowing independent control of image capture timing and readout by dividing the screen into upper and lower areas.  This enables precise synchronization with external events, lighting, and strobes, and allows flexible capture workflows in complex inspection setups.
Read the article: Trigger Modes available in See3CAMs (USB 3.0 Cameras) – e-con Systems, to know about the trigger function in USB cameras

Gradation compression
IMX568 features gradation compression to optimize the representation of brightness levels within the output image. This preserves important image details in both bright and dark regions. With this feature, the camera can deliver more usable image data without increasing bit depth or lighting complexity.

Dual ADC
The dual-ADC architecture provides faster, more flexible signal conversion. This supports high frame rates without compromising image quality and optimizes performance across the different bit depths: 8-bit / 10-bit / 12-bit. The dual ADC operation also helps IMX568-based cameras maintain high throughput and low latency in demanding vision systems.

IMX568 Sensor-Based Cameras by e-con Systems

Since 2003, e-con Systems has been designing, developing, and manufacturing cameras. e-con Systems’ embedded cameras continue to evolve with advances in sensors to meet the growing demand for embedded vision applications.

Explore our Sony Pregius Sensor-Based Cameras.

Use our Camera Selector to check out our full portfolio.

Need help selecting the right embedded camera for your application? Talk to our experts at camerasolutions@e-consystems.com.

FAQS

1. What is Multi-ROI in image sensors?
   Multi-ROI (Multiple Regions of Interest) allows an image sensor to crop and read out multiple, user-defined areas from different locations on the sensor within a single frame, instead of reading the full frame.

1. Can multiple ROIs be read simultaneously in the same frame?
   Yes. Multiple ROIs can be read out simultaneously within the same frame, allowing spatially separated regions to be captured without increasing frame latency.

1. How many ROI regions can be configured on this sensor?
   The multi ROI image sensor supports up to 64 independent ROI areas, enabling flexible selection of multiple spatial regions based on application requirements.

1. What are the benefits of using Multi-ROI instead of full-frame readout?
   Multi-ROI reduces data bandwidth and processing load, increases effective frame rates, and enables efficient monitoring of multiple areas of interest.

1. Are all ROIs captured at the same time?
   Yes. All selected ROIs are captured within the same frame, ensuring consistent timing.

Prabu Kumar
Chief Technology Officer and Head of Camera Products, e-con Systems

The post Sony Pregius IMX264 vs. IMX568: A Detailed Sensor Comparison Guide appeared first on Edge AI and Vision Alliance.

Studies show that about 34% of manufacturing defects are missed because inspection systems make mistakes.[1] These numbers show a big problem—when the inspection AI misses something, even a tiny defect can spread across hundreds or thousands of products.

One small scratch, crack, or colour mismatch can lead to rework, slowdowns, customer complaints, or even product returns. And because the production line moves quickly, these mistakes can multiply before anyone notices. That’s why an inspection AI failure affects not just one product, but the entire production line.

But here’s the good part: the problem usually comes from fixable issues like poor training data, bad lighting, or camera setup problems. When manufacturers study these mistakes closely, they can upgrade the AI, improve the dataset, and build a stronger, more reliable inspection system.

This blog explains what happens when inspection AI fails, and how these failures can actually help companies build a smarter, more accurate quality control process.

What is Inspection AI Failure?

Inspection AI failure happens when an AI system designed to spot defects in products misses, mislabels, or incorrectly flags issues. This can occur due to poor training data, changes in product appearance, lighting problems, or limitations in the AI model itself.

Such failures lead to missed defects, false alarms, and reduced confidence in automated quality checks, affecting production efficiency and product quality. DeepVision (a company working on AI vision) claims that with AI visual inspection, defect “escape rates” in some manufacturing lines dropped by as much as 83%.[2]

Why Do Visual Inspection Systems Miss Defects?

Visual inspection systems miss defects for several reasons. Sometimes, the AI isn’t trained on enough examples of real-world defects, so it doesn’t recognize unusual scratches, cracks, or color changes.

Other times, the lighting, camera angles, or image quality make it hard for the system to see small imperfections clearly. Even minor changes in product shape or texture can confuse the AI, leading to missed defects.

Another common reason is a lack of proper visual inspection error analysis. Without reviewing mistakes and understanding why the AI failed, the same errors can keep happening.

By analyzing these errors carefully, manufacturers can improve training data, adjust cameras and lighting, and fine-tune the AI model to catch more defects and reduce costly mistakes on the production line.

Real-World Impact of AI Defect Detection Failures

AI defect detection failures don’t just affect machines; they impact the entire production chain, from efficiency to customer trust.

1. Production Delays and Increased Costs

When AI defect detection misses problems, products often need rework or replacement, slowing down the production line. For example, Foxconn, a major electronics manufacturer, faced delays when their AI inspection system missed minor defects in smartphone assembly, causing additional labor and wasted components.

Similarly, Toyota reported production slowdowns in certain plants when AI visual inspection failed to catch paint imperfections, leading to costly rework and delayed deliveries.

2. Customer Dissatisfaction and Brand Damage

Defective products reaching customers can hurt a company’s reputation. Samsung once had to recall devices due to overlooked micro-defects in components, showing how AI inspection failure can impact customer trust.

Nike also faced quality complaints when automated inspection missed stitching errors in footwear. These cases highlight why reliable AI defect detection and thorough visual inspection error analysis are critical to prevent defects from reaching customers and protect brand reputation.

Ultimately, addressing AI defect detection failures through careful error analysis and improved models helps manufacturers save costs, maintain efficiency, and keep customers satisfied.

Common Causes Behind Production Line Mistakes

Understanding inspection AI failure starts with knowing why mistakes happen on the production line.

1. Poor Training Data – AI models may miss defects if they haven’t seen enough examples during training.

1. Changes in Product Appearance – Variations in color, shape, or texture can confuse the AI.

1. Lighting or Camera Issues – Poor lighting, glare, or misaligned cameras can hide defects from the system.

1. Outdated AI Models – Models not retrained for new products or updated production conditions can fail.

1. Lack of Error Analysis – Without reviewing AI mistakes through visual inspection error analysis, recurring defects go unnoticed.

By solving these causes, manufacturers can reduce errors and improve overall production quality.

5 Easy Steps to Conduct Effective Visual Inspection Error Analysis

Performing visual inspection error analysis helps identify why AI missed defects and improves overall accuracy. Here are five simple steps:

Step 1: Collect Failed Samples – Gather images or products where the AI missed defects or gave false positives. This creates a clear starting point for analysis.

Step 2: Compare with Training Data – Check if the AI has seen similar defects before. Missing examples in the training set often cause errors.

Step 3: Check Image Quality – Review lighting, camera angles, resolution, and focus. Poor image conditions can hide defects from the system.

Step 4: Analyze Model Confidence – Look at confidence scores or outputs from the AI. Low confidence often points to areas where the model struggles.

Step 5: Document and Retrain – Record all errors and their causes, then retrain the AI with new examples to reduce future inspection AI failures.

This step-by-step process ensures errors are understood, fixed, and less likely to repeat, making your AI defect detection more reliable.

Learning From Failures: Fixing the Root Cause of AI Mistakes

Learning from inspection AI failure is not about blaming the system; it’s about understanding why mistakes happen and preventing them in the future. Here’s how manufacturers can approach it effectively:

1. Identify the Exact Error

Start by pinpointing what went wrong. Was it a missed defect, a false positive, or a misclassification? Breaking down errors into clear categories makes it easier to address the root cause.

2. Investigate the Cause

Look into the source of the error:

• Was the AI model trained on enough defect examples?

• Did changes in product design or material confuse the system?

• Were environmental factors like lighting, vibration, or camera setup involved?

3. Improve Data Quality

Many failures occur because the AI hasn’t seen enough diverse defect examples. Collect new images or product samples representing edge cases, rare defects, or variations, and add them to the training dataset.

4. Update and Retrain the AI Model

After enhancing the data, retrain the AI. Fine-tune parameters and test against real production scenarios. Continuous retraining ensures the AI adapts to evolving products and production conditions.

5. Monitor and Review Continuously

Even after fixes, monitor the AI’s performance regularly. Conduct periodic visual inspection error analysis to catch new failure patterns early and maintain high-quality standards.

By following these steps, companies turn AI mistakes into actionable insights, reducing inspection AI failure and improving overall production efficiency.

Preventing Future Failures: Building a More Accurate, Reliable Inspection AI

Preventing inspection AI failure starts with creating a system that learns and adapts continuously. By using diverse and high-quality training data, improving camera setups and lighting, and retraining models regularly, manufacturers can catch even rare or subtle defects.

Adding human checks for unusual cases and monitoring AI performance in real-time further reduces errors. The goal is to build an AI-based quality inspection system that is not only fast but also consistent and dependable, keeping production smooth and products defect-free.

Why Choosing the Right AI-Based Quality Control Partner Matters

Selecting the right partner can make a huge difference in reducing inspection AI failure. Here are three key reasons:

1. Expertise in AI and Machine Vision

A skilled partner knows how to train, fine-tune, and deploy AI defect detection systems that work reliably in real production conditions.

AI-powered defect detection systems typically achieve 95‑99% accuracy, compared to just 60–90% in manual inspections.[3]

2. Customized Solutions for Your Production

Every production line is different. The right partner designs AI inspection workflows tailored to your products, lighting, cameras, and quality standards.

AI-driven QC can reduce defect rates by 20–50%, depending on the implementation.[4]

3. Continuous Support and Improvement

Reliable partners offer ongoing monitoring, retraining, and error analysis, ensuring the AI keeps improving and defects are caught before they reach customers.

In real-world deployments, AI inspection systems have reduced production‑line defects by up to 30% through continuous learning and anomaly detection.[5]

Choosing the right partner not only improves accuracy but also helps prevent costly inspection AI failure, keeping your production line efficient and your products defect-free.

Why Lincode Stands Out as Visual Inspection AI

When it comes to reliable AI defect detection, Lincode sets itself apart with a combination of advanced technology and practical design. Here’s why it’s trusted by manufacturers worldwide:

Key Reasons Lincode Excels

• High Accuracy Detection – Lincode’s AI models detect defects with over 98% accuracy, catching even the smallest scratches, cracks, or misalignments.

• Easy Integration – It can be integrated into existing production lines in less than 48 hours, reducing downtime and implementation costs.

• Real-Time Monitoring – The system provides instant alerts and detailed reports, enabling teams to resolve issues up to 3x faster than traditional inspection methods.

• Continuous Learning – Lincode adapts to new products and defect types through ongoing retraining, improving defect detection rates by 15–20% within the first few months.

In short, Lincode doesn’t just detect defects; it helps companies prevent costly mistakes, improve production efficiency, and reduce inspection AI failure, keeping product quality consistently high.

FAQ

1. What is the main reason for inspection AI failure?
The main reason is usually a lack of diverse training data or changes in product design that the AI wasn’t trained to recognize. Environmental factors like poor lighting or misaligned cameras can also cause failures.

2. How often should visual inspection error analysis be conducted?
It’s best to review errors regularly, ideally once a month or after introducing a new product, to catch recurring mistakes and improve AI accuracy.

3. Can AI defect detection replace human inspection completely?
While AI can catch most defects, combining it with human checks ensures rare or unusual defects are not missed. A human-in-the-loop approach reduces inspection AI failure significantly.

4. How does retraining the AI improve defect detection?
Retraining with new defect examples and updated production data helps the AI learn from past mistakes, improving detection accuracy and reducing future failures.

5. What industries benefit most from inspection AI?
Industries like electronics, automotive, pharmaceuticals, food packaging, and consumer goods see the biggest gains because even small defects can cause costly rework or quality issues.

Bibliography:

[1] Micromachines, Journal article, 27 February 2023.
[2] AI.Business, Case‑study article, 01 May 2024.
[3] Dhīmahi Technolabs, Blog post / Insight,2025
[4] International Journal of Intelligent Systems and Applications in Engineering Journal article, 2024.
[5] International Journal of Scientific Research and Management,  Journal article, October 2024.

The post What Happens When the Inspection AI Fails: Learning from Production Line Mistakes appeared first on Edge AI and Vision Alliance.

As the need for security becomes increasingly more critical, MIPI Alliance has continued to broaden its portfolio of standardized solutions, adding two more specifications in late 2025, and continuing work on significant updates to the MIPI Camera Security Framework specifications slated for completion in mid-2026.

Read on to learn more about the newly released specifications and what lies ahead for the MIPI Camera Security Framework.

MIPI CCISE: Protecting Camera Command and Control Interfaces

The new MIPI Command and Control Interface Service Extensions (MIPI CCISE) v1.0, released in December 2025, defines a set of security service extensions that can apply data integrity protection and optional encryption to the MIPI CSI-2® camera control interface based on the I2C transport interface. The protection is provided end-to-end between the image sensor and its associated SoC or electronic control unit (ECU).

MIPI CCISE rounds out the existing MIPI Camera Security Framework, which includes MIPI Camera Security v1.0, MIPI Camera Security Profiles v1.0 and MIPI Camera Service Extensions (MIPI CSE) v2.0. Together, the specifications define a flexible approach to add end-to-end security to image sensor applications that leverage MIPI CSI-2, enabling authentication of image system components, data integrity protection, optional data encryption, and protection of image sensor command and control channels. The specifications provide implementers with a choice of protocols, cryptographic algorithms, integrity tag modes and security protection levels to offer a solution that is uniquely effective in both its security extent and implementation flexibility.

Use of MIPI camera security specifications enables an automotive system to fulfill advanced driver-assistance systems (ADAS) safety goals up to ASIL D level (per ISO 26262:2018) and supports functional safety and security mechanisms, including end-to-end protection as recommended for high diagnostic coverage of the data communication bus.

While the initial focus of the camera security framework was on securing long-reach, wired in-vehicle network connections between CSI-2 based image sensors and their related processing ECUs, the specifications are also highly relevant to non-automotive machine vision applications that leverage CSI-2-based image sensors.

A downloadable white paper, A Guide to the MIPI Camera Security Framework for Automotive Applications, provides a detailed explanation of how these specifications work together to provide application layer end-to-end data protection.

MIPI Security Specification for Debug: Enabling Remote Debug of Systems in the Field

The recently adopted MIPI Security Specification for Debug defines a standardized method for establishing secure, authenticated debug sessions between a debug and test system and a target system.

Designed to enable remote debugging in potentially hostile real-world locations outside of a test lab, the specification allows secure remote debugging of production devices without relying solely on traditional physical protections such as buried traces or restricted access to debug ports. Instead, it introduces a trusted, cryptographically protected communication path that spans end-to-end, from the physical debug tool to the target device’s package pins, through all connectors, cabling, routing and bridges.

The new speciation adds a secure messaging layer to the existing MIPI debug architecture, wrapping debug traffic in encrypted, authenticated messages while remaining interface-agnostic. Core components include a secure communications manager that is responsible for security protocol, data model processing and key generation; cryptographic message-protection functions; and secure communication management paths. To accomplish this, the specification leverages the DMTF Security Protocol and Data Model (SPDM) industry standard for platform security.

This approach ensures authenticity, confidentiality and integrity for all debug communications, regardless of the underlying transport interface, whether MIPI I3C®, USB, PCIe or others. Debugger behavior remains consistent across interfaces, simplifying implementation and validation.

The specification complements the broader MIPI debug ecosystem.

Coming in 2026: New “Fast Boot” Options for MIPI Camera Security

Enhancements to the suite of MIPI camera security specifications are being developed to enable faster boot times for imaging systems, minimizing the time taken from power-on to streaming of secure video data.

These enhancements will continue to leverage the DMTF SPDM framework and message formats, but will introduce an optional new security mode that will half the number of security handshake operations required to complete the establishment of a secure video streaming channel compared with currently defined security modes. Image sensors will be able to implement both current and new modes of operation to provide backward compatibility, and SoCs may only require software updates to implement the new mode of operation.

Both the MIPI Camera Security and the MIPI Camera Security Profiles specifications are scheduled to be updated to v1.1 in mid-2026. However, the companion specifications that will fully enable the enhancements, MIPI CSE v2.1 and the new CSE Exchange Format (EF) v1.0, will follow later this year.

All security specifications are currently available only to MIPI Alliance members.

Ian Smith
MIPI Alliance Technical Content Consultant

The post What’s New in MIPI Security: MIPI CCISE and Security for Debug appeared first on Edge AI and Vision Alliance.

Introduction

Continuous innovation in high-performance, power-efficient systems-on-a-chip (SoCs) is enabling safer, smarter and more autonomous driving experiences in even more vehicles.

As another big step forward, Texas Instruments and Synopsys developed a Virtualizer Development Kit (VDK) for the TDA5 high-performance compute SoC family, which includes the TDA54-Q1. The TDA5 VDK enables developers to evaluate, develop and test devices in the TDA5 family ahead of initial silicon samples, providing a seamless development cycle with one software development kit (SDK) for both physical and virtual SoCs. Each device in the TDA5 family have a corresponding VDK to enable a common virtualization design and consistent user experience.

Along with the VDK, TI and Synopsys are providing additional components to create the full virtual development environment. Figure 1 provides an overview of available resources, which include:

• The virtual prototype, which is the simulated model of a TDA5 SoC.
• Deployment services from Synopsys, which are add-ons and interfaces that enable developers to integrate the VDK with other virtual components or tools.
• Documentation for the TDA5 and the TDA54-Q1 software development kit.
• Reference software examples for each TDA5 VDK and SDK to help developers get started.

Figure 1 Block diagram showing components provided by TI and Synopsys to get started with development on the VDK.

Why virtualization matters

Virtualization designs greatly reduce automotive development cycles by enabling software development without physical hardware. This allows developers to accelerate or “shift-left” development by starting software earlier and then migrating to physical hardware once available (as shown in Figure 2). Additionally, earlier software development extends to ecosystem partners, enabling key third-party software components to be available earlier.

Figure 2 Visualization of how software can be migrated from VDK to SoC.

Accelerating development with virtualization

The TDA5 VDK helps software developers work more effectively and efficiently, allowing them to use software-in-the-loop testing, so they can test and validate virtually without needing costly on-the-road testing.

Developers can use the TDA5 VDK to enhance debugging capabilities with deeper insights into internal device operations than what is typically exposed through the physical SoC pins. The TDA5 VDK also provides fault injection capabilities, enabling developers to simulate failures inside the device to get better information on how the software behaves when something goes wrong.

Scalability of virtualization

Scalability is another key benefit of the TDA5 VDK because virtualization platforms don’t require shipping, allowing development teams to ramp faster and be more responsive with resource allocation for ongoing projects. The TDA5 VDK also enables automated test environments, since development teams can replace traditional “board farms” with virtual environments running on remote computers. This helps automakers streamline continuous integration, continuous deployment (CICD) workflows to more efficiently and effectively accomplishing testing.

Since the TDA5 VDK is also available for future TDA5 SoCs, developers can scale work across multiple projects. If a developer is using the VDK for a specific TDA5 device (for example, TDA54), they can explore other products in the TDA5 family in a virtual environment without needing to change hardware configurations.

System integration

Virtualization designs such as the TDA5 VDK serve as the foundation for developers to build complete digital twins for their designs. By virtualizing the SoC, it can be integrated with other virtual components and tools to create larger simulated systems such as full ECU networks. Figure 3 shows how developers can leverage the capabilities of the Synopsys platform to integrate the VDK with other virtual components and simulate complete designs.

Figure 3 Diagram showing how the VDK can integrate with other virtual components and simulate complete designs.

Digital environment simulation tools can also be integrated with the TDA5 VDK to enable virtual testing in simulated driving scenarios, allowing developers to quickly perform reproducible testing. The TDA5 VDK also allows developers to leverage the broad ecosystem of tools and partners from Synopsys to get the most of their virtual development experience.

Getting started with the TDA54 VDK

The TDA54 SDK is now available on TI.com to help engineers get started with the TDA54 virtual development kit. Samples of the TDA54-Q1 SoC, the first device in the TDA5 family, will be sampling to select automotive customers by the end of 2026. Contact TI for more information about the TDA5 VDK and how to get started.

The post Accelerating next-generation automotive designs with the TDA5 Virtualizer™ Development Kit appeared first on Edge AI and Vision Alliance.

NVIDIA Editor’s note: This post is part of Into the Omniverse, a series focused on how developers, 3D practitioners and enterprises can transform their workflows using the latest advancements in OpenUSD and NVIDIA Omniverse.

New NVIDIA safety frameworks and technologies are advancing how developers build safe physical AI.

Physical AI is moving from research labs into the real world, powering intelligent robots and autonomous vehicles (AVs) — such as robotaxis — that must reliably sense, reason and act amid unpredictable conditions.

To safely scale these systems, developers need workflows that connect real-world data, high-fidelity simulation and robust AI models atop the common foundation provided by the OpenUSD framework.

The recently published OpenUSD Core Specification 1.0, OpenUSD — aka Universal Scene Description — now defines standard data types, file formats and composition behaviors, giving developers predictable, interoperable USD pipelines as they scale autonomous systems.

Powered by OpenUSD, NVIDIA Omniverse libraries combine NVIDIA RTX rendering, physics simulation and efficient runtimes to create digital twins and simulation-ready (SimReady) assets that accurately reflect real-world environments for synthetic data generation and testing.

NVIDIA Cosmos world foundation models can run on top of these simulations to amplify data variation, generating new weather, lighting and terrain conditions from the same scenes so teams can safely cover rare and challenging edge cases.

In addition, advancements in synthetic data generation, multimodal datasets and SimReady workflows are now converging with the NVIDIA Halos framework for AV safety, creating a standards-based path to safer, faster, more cost-effective deployment of next-generation autonomous machines.

Building the Foundation for Safe Physical AI

Open Standards and SimReady Assets

The OpenUSD Core Specification 1.0 establishes the standard data models and behaviors that underpin SimReady assets, enabling developers to build interoperable simulation pipelines for AI factories and robotics on OpenUSD.

Built on this foundation, SimReady 3D assets can be reused across tools and teams and loaded directly into NVIDIA Isaac Sim, where USDPhysics colliders, rigid body dynamics and composition-arc–based variants let teams test robots in virtual facilities that closely mirror real operations.

Open-Source Learning 

The Learn OpenUSD curriculum is now open source and available on GitHub, enabling contributors to localize and adapt templates, exercises and content for different audiences, languages and use cases. This gives educators a ready-made foundation to onboard new teams into OpenUSD-centric simulation workflows.​

Generative Worlds as Safety Multiplier

Gaussian splatting — a technique that uses editable 3D elements to render environments quickly and with high fidelity — and world models are accelerating simulation pipelines for safe robotics testing and validation.

At SIGGRAPH Asia, the NVIDIA Research team introduced Play4D, a streaming pipeline that enables 4D Gaussian splatting to accurately render dynamic scenes and improve realism.

Spatial intelligence company World Labs is using its Marble generative world model with NVIDIA Isaac Sim and Omniverse NuRec so researchers can turn text prompts and sample images into photorealistic, Gaussian-based physics-ready 3D environments in hours instead of weeks.

Those worlds can then be used for physical AI training, testing and sim-to-real transfer. This high-fidelity simulation workflow expands the range of scenarios robots can practice in while keeping experimentation safely in simulation.

Lightwheel Helps Teams Scale Robot Training With SimReady Assets

Powered by OpenUSD, Lightwheel’s SimReady asset library includes a common scene description layer, making it easy to assemble high-fidelity digital twins for robots. The SimReady assets are embedded with precise geometry, materials and validated physical properties, which can be loaded directly into NVIDIA Isaac Sim and Isaac Lab for robot training. This allows robots to experience realistic contacts, dynamics and sensor feedback as they learn.

End-to-End Autonomous Vehicle Safety

End-to-end autonomous vehicle safety advancements are accelerating with new research, open frameworks and inspection services that make validation more rigorous and scalable.

NVIDIA researchers, with collaborators at Harvard University and Stanford University, recently introduced the Sim2Val framework to statistically combine real-world and simulated test results, reducing AV developers’ need for costly physical mileage while demonstrating how robotaxis and AVs can behave safely across rare and safety-critical scenarios.

Learn more by watching NVIDIA’s “Safety in the Loop” livestream:

These innovations are complemented by a new, open-source NVIDIA Omniverse NuRec Fixer, a Cosmos-based model trained on AV data that removes artifacts in neural reconstructions to produce higher-quality SimReady assets.

To align these advances with rigorous global standards, the NVIDIA Halos AI Systems Inspection Lab — accredited by ANAB — provides impartial inspection and certification of Halos elements across robotaxi fleets, AV stacks, sensors and manufacturer platforms through the Halos Certification Program.

AV Ecosystem Leaders Putting Physical AI Safety to Work

Bosch, Nuro and Wayve are among the first participants in the NVIDIA Halos AI Systems Inspection Lab, which aims to accelerate the safe, large-scale deployment of robotaxi fleets. Onsemi, which makes sensor systems for AVs, industrial automation and medical applications, has recently become the first company to pass inspection for the NVIDIA Halos AI Systems Inspection Lab.

The open-source CARLA simulator integrates NVIDIA NuRec and Cosmos Transfer to generate reconstructed drives and diverse scenario variations, while Voxel51’s FiftyOne engine, linked to Cosmos Dataset Search, NuRec and Cosmos Transfer, helps teams curate, annotate and evaluate multimodal datasets across the AV pipeline.​

Mcity at the University of Michigan is enhancing the digital twin of its 32-acre AV test facility using Omniverse libraries and technologies. The team is integrating the NVIDIA Blueprint for AV simulation and Omniverse Sensor RTX application programming interfaces to create physics-based models of camera, lidar, radar and ultrasonic sensors.

By aligning real sensor recordings with high-fidelity simulated data and sharing assets openly, Mcity enables safe, repeatable testing of rare and hazardous driving scenarios before vehicles operate on public roads.

Get Plugged Into the World of OpenUSD and Physical AI Safety

Learn more about OpenUSD, NVIDIA Halos and physical AI safety by exploring these resources:

• Watch the on-demand NVIDIA GTC session, “Reconstructing Reality: Simulating Indoor and Outdoor Environments for Physical AI.”
• Visit the NVIDIA Halos AI Systems Inspection Lab webpage.
• Follow the NVIDIA DRIVE LinkedIn newsletter: “NVIDIA Safety in the Loop.”
• Read the corporate blog explainer: How AI Is Unlocking Level 4 Autonomy.
• Get started with the Learn OpenUSD curriculum, now open source.

Katie Washabaugh, Product Marketing Manager for Autonomous Vehicle Simulation, NVIDIA

The post Into the Omniverse: OpenUSD and NVIDIA Halos Accelerate Safety for Robotaxis, Physical AI Systems appeared first on Edge AI and Vision Alliance.

Key Takeaways

• Why multi-sensor timing drift weakens edge AI perception
• How GNSS-disciplined clocks align cameras, LiDAR, radar, and IMUs
• Role of Orin NX as a central timing authority for sensor fusion
• Operational gains from unified time-stamping in autonomous vision systems

Autonomous vision systems deployed at the edge depend on seamless fusion of multiple sensor streams (cameras, LiDAR, Radar, IMU, and GNSS) to interpret dynamic environments in real time. For NVIDIA Orin NX-based platforms, the challenge lies in merging all the data types within microseconds to maintain spatial awareness and decision accuracy.

Latency from unsynchronized sensors can break perception continuity in edge AI vision deployments. For instance, a camera might capture a frame before LiDAR delivers its scan, or the IMU might record motion slightly out of phase. Such mismatches produce misaligned depth maps, unreliable object tracking, and degraded AI inference performance. A sensor fusion system anchored on the Orin NX mitigates this issue through GNSS-disciplined synchronization.

In this blog, you’ll learn everything you need to know about the sensor fusion architecture, why the unified time base matters, and how it boosts edge AI vision deployments.

What are the Different Types of Sensors and Interfaces?

Coordinating Multi-Sensor Timing with Orin NX

Edge AI systems rely on timing discipline as much as compute power. The NVIDIA Orin NX acts as the central clock, aligning every connected sensor to a single reference point through GNSS time discipline.

The GNSS receiver sends a Pulse Per Second (PPS) signal and UTC data via NMEA to the Orin NX, which aligns its internal clock with global GPS time. This disciplined clock becomes the authority across all interfaces. From there, synchronization extends through three precise routes:

1. PTP over Ethernet: The Orin NX functions as a PTP Grandmaster through its USB NIC. LiDAR and radar units operate as PTP slaves, delivering time-stamped point clouds and detections that stay aligned to the GNSS time domain.
2. PPS-derived camera triggers: Cameras linked via GMSL or MIPI CSI receive frame triggers generated from the PPS signal. This ensures frame start alignment to GNSS time with zero drift between captures.
3. Timed IMU polling: The IMU connects over I²C and is polled at consistent intervals, typically between 500 Hz and 1 kHz. Software time stamps are derived from the same GNSS-disciplined clock, keeping IMU data in sync with all other sensors.

Importance of a Unified Time Base

All sensors share the same GNSS-aligned time domain, enabling precise fusion of LiDAR, radar, camera, and IMU data.

Implementation Guidelines for Stable Sensor Fusion

• USB NIC and PTP configuration: Enable hardware time-stamping (ethtool -T ethX) so Ethernet sensors maintain nanosecond alignment.
• Camera trigger setup: Use a hardware timer or GPIO to generate PPS-derived triggers for consistent frame alignment.
• IMU polling: Maintain fixed-rate polling within Orin NX to align IMU data with the GNSS-disciplined clock.
• Clock discipline: Use both PPS and NMEA inputs to keep the Orin NX clock aligned to UTC for accurate fusion timing.

Strengths of Leveraging Sensor Fusion-Based Autonomous Vision

Direct synchronization control

Removing the intermediate MCU lets Orin NX handle timing internally, cutting latency and eliminating cross-processor jitter.

Unified global time-stamping

All sensors operate on GNSS time, ensuring every frame, scan, and motion reading aligns to a single reference.

Sub-microsecond Ethernet alignment

PTP synchronization keeps LiDAR and radar feeds locked to the same temporal window, maintaining accuracy across fast-moving scenes.

Deterministic frame capture

PPS-triggered cameras guarantee frame starts occur exactly on the GNSS second, preventing drift between visual and depth data.

Consistent IMU data

High-frequency IMU polling stays aligned with the master clock, preserving accurate motion tracking for fusion and localization.

e-con Systems Offers Custom Edge AI Vision Boxes

e-con Systems has been designing, developing, and manufacturing OEM camera solutions since 2003. We offer customizable Edge AI Vision Boxes powered by NVIDIA Orin NX and Orin Nano. It brings together multi-camera interfaces, hardware-level synchronization, and AI-ready processing into one cohesive unit for real-time vision tasks.

Our Edge AI Vision Box – Darsi simplifies the adoption of GNSS-disciplined fusion in robotics, autonomous mobility, and industrial vision. It comes with support for PPS-triggered cameras, PTP-synced Ethernet sensors, and flexible connectivity options. It also provides an end-to-end framework where developers can plug in sensors, train models, and run inference directly at the edge (without external synchronization hardware).

Know more -> e-con Systems’ Orin NX/Nano-based Edge AI Vision Box

Use our Camera Selector to find other best-fit cameras for your edge AI vision applications.

If you need expert guidance for selecting the right imaging setup, please reach out to camerasolutions@e-consystems.com.

FAQs

1. What role does sensor fusion play in edge AI vision systems?
   Sensor fusion aligns data from cameras, LiDAR, radar, and IMU sensors to a common GNSS-disciplined time base. It ensures every frame and data point corresponds to the same moment, thereby improving object detection, 3D reconstruction, and navigation accuracy in edge AI systems.

1. How does NVIDIA Orin NX handle synchronization across sensors?
   The Orin NX functions as both the compute core and timing master. It receives a PPS signal and UTC data from the GNSS receiver, disciplines its internal clock, and distributes synchronization through PTP for Ethernet sensors, PPS triggers for cameras, and fixed-rate polling for IMUs.

1. Why is a unified time base critical for reliable fusion?
   When all sensors share a single GNSS-aligned clock, the system eliminates time-stamp drift and timing mismatches. So, fusion algorithms can process coherent multi-sensor data streams, which enable the AI stack to operate with consistent depth, motion, and spatial context.

1. What are the implementation steps for achieving stable sensor fusion?
   Developers should enable hardware time-stamping for PTP sensors, use PPS-based hardware triggers for cameras, poll IMUs at fixed intervals, and feed both PPS and NMEA inputs into the Orin NX clock. These steps maintain accurate UTC alignment through long runtime cycles.

1. How does e-con Systems support developers building with Orin NX?
   e-con Systems provides customizable Edge AI Vision Boxes powered by NVIDIA Orin NX and Orin Nano. They are equipped with synchronized camera interfaces, AI-ready processing, and GNSS-disciplined timing. Hence, product developers can deploy real-time vision solutions quickly and with full temporal accuracy.

Prabu Kumar
Chief Technology Officer and Head of Camera Products, e-con Systems

The post What Sensor Fusion Architecture Offers for NVIDIA Orin NX-Based Autonomous Vision Systems appeared first on Edge AI and Vision Alliance.

Imagine capturing the perfect landscape photo on a sunny day, only to find harsh shadows obscuring key details and distorting colors. Similarly, in computer vision projects, shadows can interfere with object detection algorithms, leading to inaccurate results. Shadows are a common nuisance in image processing, introducing uneven illumination that compromises both aesthetic quality and functional analysis.

In this blog post, we’ll tackle this challenge head-on with a practical approach to shadow correction using OpenCV. Our method leverages Multi-Scale Retinex (MSR) for illumination normalization, combined with adaptive shadow masking in LAB and HSV color spaces. This technique not only removes shadows effectively but also preserves natural colors and textures.

We’ll provide a complete Python script that includes interactive trackbars for real-time parameter tuning, making it easy to adapt to different images. Whether you’re a photographer, a developer working on augmented reality, or just curious about image enhancement, this guide will equip you with the tools to banish shadows from your images.

How Shadows Affect Image Appearance

Before diving into solutions, let’s understand shadows and their challenges in image processing. A shadow forms when an object blocks light, reducing illumination on a surface. This dims the area but doesn’t alter the object’s inherent properties.

Key points to consider,

• Shadows impact illumination, not reflectance (the object’s true color and material).
• The same object may look dark in shadow and bright in light, confusing viewers and algorithms.
• Shadows vary: soft (smooth transitions) or hard (sharp edges), needing precise detection to prevent artifacts.

Simply brightening an image won’t fix shadows; it can overexpose highlights or skew colors. Instead, effective correction separates illumination from reflectance. The image model is I = R × L, where I denotes the observed image, R denotes reflectance, and L denotes illumination. To recover R, estimate and normalize L, often using logs for stability.

Real-world examples show how shadows cause uneven lighting, which our method corrects by isolating and adjusting these components.

These visuals illustrate uneven lighting from shadows, guiding our approach to preserve true colors.

Understanding the Fundamentals

Before diving into the code, let’s build a solid foundation on the key concepts.

Color Spaces Explained

Images are typically represented in RGB (Red, Green, Blue), but for shadow removal, other color spaces are more suitable because they separate luminance (brightness) from chrominance (color).

• LAB Color Space: This is a perceptually uniform color space where L represents lightness (0-100), A the green-red axis, and B the blue-yellow axis. It’s ideal for shadow correction because we can manipulate the L channel independently without affecting colors. In OpenCV, we convert using cv.cvtColor(img, cv.COLOR_BGR2LAB).

• HSV Color Space: Hue (H), Saturation (S), and Value (V). Shadows often appear as areas with low saturation and value. We use the S channel to help identify shadows, as they tend to desaturate colors.

Switching to these spaces allows us to target shadows more precisely.

Retinex Theory Basics

Retinex theory, proposed by Edwin Land in the 1970s, models how the human visual system achieves color constancy, perceiving colors consistently under varying illumination, much like how our eyes adapt to different lighting without changing perceived object colors. The core idea is that an image can be decomposed into reflectance (intrinsic object properties, like surface material) and illumination (lighting variations, such as shadows or highlights).

Multi-Scale Retinex (MSR) extends this by applying Gaussian blurs at multiple scales to estimate illumination, inspired by the multi-resolution processing in human vision. For each scale:

1. Blur the image to approximate the illumination component and smooth out local variations.
2. Subtract the log of the blurred image from the log of the original (to handle the multiplicative nature of illumination effects, as log transforms multiplication to addition for easier separation).
3. Average across scales for a robust estimate, balancing local and global corrections.

This results in an enhanced image with reduced shadows, improved dynamic range, and better contrast in low-light areas. In our code, we apply MSR only to the L channel for efficiency, focusing on luminance where shadows primarily affect brightness.

Shadow Detection Challenges

Simple thresholding on brightness fails because shadows vary in intensity (from subtle gradients to deep darkness) and can blend seamlessly with inherently dark objects, leading to false positives or missed areas. We need an adaptive approach that considers context:

• Combine low luminance (L < threshold) with low saturation (S < threshold), as shadows not only darken but also desaturate colors by reducing light intensity without adding new hues.
• Use morphological operations, such as closing to fill small gaps in the mask and opening to remove isolated noise specks, to refine the mask for better accuracy and continuity.
• Smooth the mask with a Gaussian blur to achieve seamless blending and prevent visible edges or halos in the corrected image.

This ensures we correct only shadowed areas without over-processing the rest of the image, maintaining natural transitions and avoiding artifacts.

Overview of the Shadow Removal Pipeline

Our pipeline processes the image step-by-step for effective shadow correction:

1. Load and Preprocess: Read the image and resize for faster preview (e.g., 50% scale).
2. Color Space Conversion: Convert to LAB (for luminance/chrominance) and HSV (for saturation).
3. Compute Retinex: Apply Multi-Scale Retinex on the L channel to create an illumination-normalized version.
4. Generate Shadow Mask: Use adaptive conditions on normalized L and S, then blur for softness.
5. Remove Shadows: Blend the original L with Retinex L in shadowed areas. For A/B channels, blend with estimated background colors to avoid color shifts.
6. Interactive Tuning: Use OpenCV trackbars to adjust strength, sensitivity, and blur in real-time.
7. Display Results: Show original, mask, and corrected image side-by-side.

This approach is adaptive, meaning it responds to image content, and the parameters allow customization for various lighting conditions.

Diving into the Code: Step-by-Step Breakdown

Let’s dissect the Python script. We’ll assume you have OpenCV and NumPy installed (pip install opencv-python numpy).

Prerequisites

• Python 3.x
• OpenCV (cv2)
• NumPy (np)

Core Functions

Multi-Scale Illumination Normalization (Retinex Processing)

This function computes the Multi-Scale Retinex on the lightness channel.

def multiscale_retinex(L):
    scales = [31, 101, 301]  # Small, medium, large scales for different illumination sizes
    retinex = np.zeros_like(L, dtype=np.float32)
    for k in scales:
        blur = cv.GaussianBlur(L, (k, k), 0)  # Blur to estimate illumination
        retinex += np.log(L + 1) - np.log(blur + 1)  # Log subtraction for reflectance
    retinex /= len(scales)  # Average across scales
    retinex = cv.normalize(retinex, None, 0, 255, cv.NORM_MINMAX)  # Scale to 0-255
    return retinex

Why these scales? Smaller kernels capture fine details, larger ones handle broad shadows. The +1 avoids log(0) issues. Normalization ensures the output matches the input range.

Adaptive Shadow Detection and Mask Generation

Creates a binary shadow mask and softens it.

def compute_shadow_mask_adaptive(L, S, sensitivity=1.0, mask_blur=21):
    shadow_cond = (L < 0.5 * sensitivity) & (S < 0.5)  # Low brightness and saturation
    mask = shadow_cond.astype(np.float32)  # 0 or 1 float
    mask_blur = mask_blur if mask_blur % 2 == 1 else mask_blur + 1  # Ensure odd for Gaussian
    mask = cv.GaussianBlur(mask, (mask_blur, mask_blur), 0)  # Soften edges
    return mask

Sensitivity scales the luminance threshold, allowing tuning for faint or dark shadows. The blur prevents harsh transitions.

Mask-Guided Shadow Removal and Color Preservation

The heart of the correction: refines the mask and blends channels.

def remove_shadows_adaptive_v3(L, A, B, L_retinex, strength=0.9, mask=None, mask_blur=31):
    kernel = cv.getStructuringElement(cv.MORPH_ELLIPSE, (7, 7))  # Elliptical kernel for morphology
    shadow_mask = cv.morphologyEx(mask, cv.MORPH_CLOSE, kernel)  # Close gaps
    shadow_mask = cv.morphologyEx(shadow_mask, cv.MORPH_OPEN, kernel)  # Remove noise
    shadow_mask = cv.dilate(shadow_mask, kernel, iterations=1)  # Expand slightly
    shadow_mask = cv.GaussianBlur(shadow_mask, (mask_blur, mask_blur), 0)  # Smooth
    mask_smooth = np.power(shadow_mask, 1.5)  # Non-linear for stronger effect in core shadows

    L_final = (1 - strength * mask_smooth) * L + (strength * mask_smooth) * L_retinex  # Blend L
    L_final = np.clip(L_final, 0, 255)  # Prevent overflow

    mask_inv = 1 - mask_smooth  # Non-shadow areas
    A_bg = np.sum(A * mask_inv) / (np.sum(mask_inv) + 1e-6)  # Average A in non-shadows
    B_bg = np.sum(B * mask_inv) / (np.sum(mask_inv) + 1e-6)  # Average B

    A_final = (1 - strength * mask_smooth) * A + (strength * mask_smooth) * A_bg  # Blend A/B
    B_final = (1 - strength * mask_smooth) * B + (strength * mask_smooth) * B_bg

    return L_final, A_final, B_final

Morphological ops refine the mask: closing fills holes, opening removes specks, dilation ensures coverage. The power function makes blending more aggressive in deep shadows. Background color estimation for A/B preserves chromaticity.

Trackbar Callback Utility

A placeholder for trackbar callbacks, as required by OpenCV.

def nothing(x):
    pass

Full Code:
The entry point handles image loading, setup, and the interactive loop.

import cv2 as cv
import numpy as np

# Retinex (compute once)
def multiscale_retinex(L):
    scales = [31, 101, 301]
    retinex = np.zeros_like(L, dtype=np.float32)
    for k in scales:
        blur = cv.GaussianBlur(L, (k, k), 0)
        retinex += np.log(L + 1) - np.log(blur + 1)
    retinex /= len(scales)
    retinex = cv.normalize(retinex, None, 0, 255, cv.NORM_MINMAX)
    return retinex

# Adaptive Shadow Mask
def compute_shadow_mask_adaptive(L, S, sensitivity=1.0, mask_blur=21):
    shadow_cond = (L < 0.5 * sensitivity) & (S < 0.5)
    mask = shadow_cond.astype(np.float32)
    mask_blur = mask_blur if mask_blur % 2 == 1 else mask_blur + 1
    mask = cv.GaussianBlur(mask, (mask_blur, mask_blur), 0)
    return mask

# Shadow Removal
def remove_shadows_adaptive_v3(L, A, B, L_retinex, strength=0.9, mask=None, mask_blur=31):
    kernel = cv.getStructuringElement(cv.MORPH_ELLIPSE, (7, 7))
    shadow_mask = cv.morphologyEx(mask, cv.MORPH_CLOSE, kernel)
    shadow_mask = cv.morphologyEx(shadow_mask, cv.MORPH_OPEN, kernel)
    shadow_mask = cv.dilate(shadow_mask, kernel, iterations=1)
    shadow_mask = cv.GaussianBlur(shadow_mask, (mask_blur, mask_blur), 0)
    mask_smooth = np.power(shadow_mask, 1.5)

    L_final = (1 - strength * mask_smooth) * L + (strength * mask_smooth) * L_retinex
    L_final = np.clip(L_final, 0, 255)

    mask_inv = 1 - mask_smooth
    A_bg = np.sum(A * mask_inv) / (np.sum(mask_inv) + 1e-6)
    B_bg = np.sum(B * mask_inv) / (np.sum(mask_inv) + 1e-6)

    A_final = (1 - strength * mask_smooth) * A + (strength * mask_smooth) * A_bg
    B_final = (1 - strength * mask_smooth) * B + (strength * mask_smooth) * B_bg

    return L_final, A_final, B_final

def nothing(x):
    pass

# Main
if __name__ == "__main__":
    img = cv.imread("image.jpg")
    if img is None:
        raise IOError("Image not found")

    scale = 0.5
    img_preview = cv.resize(img, None, fx=scale, fy=scale, interpolation=cv.INTER_AREA)

    lab = cv.cvtColor(img_preview, cv.COLOR_BGR2LAB).astype(np.float32)
    L, A, B = cv.split(lab)
    L_retinex = multiscale_retinex(L)

    hsv = cv.cvtColor(img_preview, cv.COLOR_BGR2HSV).astype(np.float32)
    S = hsv[:, :, 1] / 255.0

    cv.namedWindow("Shadow Removal", cv.WINDOW_NORMAL)
    cv.createTrackbar("Strength", "Shadow Removal", 90, 200, nothing)
    cv.createTrackbar("Sensitivity", "Shadow Removal", 90, 200, nothing)
    cv.createTrackbar("MaskBlur", "Shadow Removal", 31, 101, nothing)

    while True:
        strength = cv.getTrackbarPos("Strength", "Shadow Removal") / 100.0
        sensitivity = cv.getTrackbarPos("Sensitivity", "Shadow Removal") / 100.0
        mask_blur = cv.getTrackbarPos("MaskBlur", "Shadow Removal")
        mask_blur = max(3, mask_blur)
        mask_blur = mask_blur if mask_blur % 2 == 1 else mask_blur + 1

        mask = compute_shadow_mask_adaptive(L / 255.0, S, sensitivity, mask_blur)

        L_final, A_final, B_final = remove_shadows_adaptive_v3(
            L, A, B, L_retinex, strength, mask, mask_blur
        )

        lab_out = cv.merge([L_final, A_final, B_final]).astype(np.uint8)
        result = cv.cvtColor(lab_out, cv.COLOR_LAB2BGR)

        # BUILD RGB VIEW
        orig_rgb = cv.cvtColor(img_preview, cv.COLOR_BGR2RGB)
        mask_rgb = cv.cvtColor((mask * 255).astype(np.uint8), cv.COLOR_GRAY2RGB)
        result_rgb = cv.cvtColor(result, cv.COLOR_BGR2RGB)

        combined_rgb = np.hstack([orig_rgb, mask_rgb, result_rgb])

        # Convert back so OpenCV shows correct colors
        combined_bgr = cv.cvtColor(combined_rgb, cv.COLOR_RGB2BGR)

        cv.imshow("Shadow Removal", combined_bgr)

        key = cv.waitKey(30) & 0xFF
        if key == 27 or cv.getWindowProperty("Shadow Removal", cv.WND_PROP_VISIBLE) < 1:
            break

    cv.destroyAllWindows()

Key points:

• Resizing speeds up processing for previews.
• Retinex is computed once outside the loop for efficiency.
• The loop updates on trackbar changes, recomputing the mask and correction.
• Display stacks original, mask (grayscale as RGB), and result for comparison.

Running the Code and Tuning Parameters

Setup Instructions

1. Save the code as a .py format.
2. Replace “image.jpg” with your image path (JPEG, PNG, etc.).
3. Run: python shadow_removal.py.

A window will appear with trackbars and a side-by-side view.

Output:

Interactive Demo

• Strength (0-2.0): Controls blending intensity. Higher values apply more correction but increase the risk of artifacts.
• Sensitivity (0-2.0): Adjusts shadow detection threshold. Lower for detecting subtle shadows, higher for aggressive ones.
• MaskBlur (3-101, odd): Softens mask edges. Larger values for smoother transitions in large shadows.

For outdoor scenes with cast shadows, increase sensitivity. For indoor low-light, reduce the strength to avoid over-brightening.

Potential Improvements and Limitations

Enhancements

• Batch Processing: Extend the pipeline to process multiple images or video frames, enabling use in real-time or large-scale applications.
• ML Integration: Incorporate deep learning models (such as U-Net) to generate more accurate, semantic shadow masks using datasets like ISTD.
• Colored Shadow Handling: Improve robustness by detecting and correcting color shifts caused by colored or indirect lighting.
• Performance Optimization: Speed up processing for large images by parallelizing Retinex scales or working on downsampled inputs.

Limitations

• Visual Artifacts: In textured regions or near shadow boundaries, blending can introduce halos or inconsistencies, requiring more refined masks.
• Computational Cost: Multi-Scale Retinex with large kernels can be slow on high-resolution images; preprocessing steps like downsampling are often necessary.
• Lighting Assumptions: The method works best for neutral (achromatic) shadows and may struggle under colored or complex illumination conditions.
• Low-Light Noise Amplification: Shadow enhancement can amplify image noise in dark areas; denoising may be needed beforehand.
• Compared to Deep Learning: OpenCV methods don’t match deep learning for complex shadow removal, and images with heavy shadowing can be tough to fully correct.

Overall, this is a solid baseline for many scenarios, and performance can be improved by tuning parameters to the specific image and lighting conditions.

Conclusion

Shadows pose a challenge in image enhancement because they affect illumination without changing object properties. This blog presented an adaptive shadow-correction pipeline using OpenCV that combines Multi-Scale Retinex with color-space–based shadow detection to reduce shadows while preserving natural colors. Interactive parameter tuning makes the method flexible across different images. Although it cannot fully match deep learning approaches for complex scenes, it provides a lightweight and effective baseline that can be further improved or extended.

Reference

Image Shadow Removal Method Based on LAB Space

Shadow Detection and Removal

Image Shadow Remover

Frequently Asked Questions

Why not simply increase the brightness to remove shadows?

Increasing brightness affects the entire image and can wash out highlights or distort colors. Shadow removal requires separating illumination from reflectance to selectively correct shadowed regions.

Why are LAB and HSV color spaces used instead of RGB?

LAB and HSV separate brightness from color information, making it easier to detect and correct shadows without introducing color shifts.

Sanjana Bhat
OpenCV

The post Enhancing Images: Adaptive Shadow Correction Using OpenCV appeared first on Edge AI and Vision Alliance.

In today’s automotive industry, onboard AI inference engines drive numerous safety-critical Advanced Driver Assistance Systems (ADAS) features, all of which require consistent, high-performance processing. Given that AI model engineering is inherently iterative (numerous cycles of ‘train, validate, and deploy’), it is crucial to assess model performance on actual silicon at every step of product development. This hardware-based validation not only strengthens confidence in model engineering decisions but also ensures that AI solutions are reliable and meet the target KPI for deployment into in-vehicle AI applications through the product lifecycle.

Meet RoX AI Studio, designed specifically for today’s innovative automotive teams. With RoX AI Studio, you can remotely benchmark and evaluate your AI models on Renesas R-Car SoCs within your internet browser (Figure 1), all while leveraging a secure MLOps infrastructure that puts your engineering team in the fast lane toward production-ready solutions.

This platform is a cornerstone of the Renesas Open Access (RoX) Software-Defined Vehicle (SDV) platform, offering an integrated suite of hardware, software, and infrastructure for customers designing state-of-the-art automotive systems powered by AI. We’re dedicated to empowering products with advanced intelligence, high-performance, and an accelerated product lifecycle. RoX AI Studio enables you to unlock the full potential of next-generation vehicles by embracing a shift-left approach.

UX Overview

MLOps with Workflows and HyCo Toolchain

The API endpoints in RoX AI Studio are underpinned by robust MLOps business logic, which ensures reliable execution for every incoming API request. Each experiment initiated through the platform follows a systematic and predefined sequence of steps. These steps are organized as Directed Acyclic Graphs (DAGs) and orchestrated using Apache Airflow, a proven workflow management tool.

MLOps Overview

Workflows

Apache Airflow manages the queuing, scheduling, and concurrency of experiment tasks automatically, allowing the system to efficiently handle multiple simultaneous user requests with finite computational resources on the cloud. The backend architecture leverages a suite of MLOps and third-party microservices, each deployed as Docker containers or coupled through third-party API. This design separates the execution of individual intermediate steps from the overarching control plane, which is governed by the DAG workflows. Such separation provides greater flexibility, enabling the platform to scale dynamically across distributed cloud computing environments and adapt to fluctuating user demands.

Moreover, this approach promotes more granular product development for each microservice. By supporting out-of-the-box (OOB) execution for individual components, RoX AI Studio enables rapid iteration and targeted enhancements, aligning with evolving platform requirements and user needs. Each workflow incorporates model management, data management, and experiment management, powered by Model Registry, Managed DB, and Board Manager.

HyCo Toolchain

Custom layers and operators are increasingly prevalent as AI model architecture continues to evolve. To address this opportunity, a high-performance custom compiler known as HyCo (Hybrid Compiler) is offered specifically for the R-Car Gen4 product line. HyCo has a hybrid compiler architecture, comprising both front-end and back-end compiler components, to ensure scalability and adaptability for custom implementations. At the core of this approach, TVM functions as a unifying backbone, enabling seamless integration of customizations in the front-end compiler with accelerator-specific back-end compilers. This design supports efficient compilation and optimization tailored to heterogeneous hardware accelerators within the SoC.

HyCo is seamlessly integrated into a developer-oriented HyCo toolchain, also referred to as AI Toolchain. Beyond the compiler itself, AI Toolchain provides interfaces for ingesting open-source model zoo assets as well as BYOM assets, encompassing both pre-processing and post-processing software components. This approach demonstrates how an AI toolchain can integrate with customer-specific model zoos, enhancing flexibility in deploying diverse AI workloads. Within the MLOps framework, various configurations of the AI toolchain are containerized into independent microservices. This modular approach emphasizes robust integration within MLOps workflows, allowing for the deployment of standalone AI toolchain components that can dynamically scale in cloud environments.

Infrastructure with MLOps Cloud and Device Farm

The hybrid infrastructure enables comprehensive end-to-end MLOps workflows, seamlessly delegating HIL inference tasks to Renesas Device Farm. Currently, the MLOps cloud platform is hosted on Azure, but its architecture is designed to support flexible deployment across other public or private cloud environments in the future.

Infrastructure Overview

MLOps Cloud

By utilizing a workflow-based MLOps architecture, we can securely enable multiple users within a single tenant to share computational resources, optimizing capital expenditure. This approach empowers customers to develop AI products without the need for significant individual investment for each developer. The architecture is also built to support seamless integration with private customer clouds, accommodating custom hardware configurations (such as CPU and GPU servers and shared bulk storage) alongside robust on-premises security infrastructure.

Renesas Device Farm

A secure on-premises device farm hosts multiple R-Car SoC development boards, providing the foundation for hardware-in-the-loop (HIL) inference experiments essential for AI model benchmarking and evaluation. The cloud-based Board Manager microservice efficiently handles board allocation, setup, and release, streamlining resource management and eliminating the need for direct developer involvement. The MLOps workflow leverages the device farm to execute HIL inference experiments without common delays associated with traditional board provisioning, updating, and maintenance. A robust networking architecture ensures secure HIL inference sessions for users, maintaining the integrity and confidentiality of both data and AI models.

What Advantages does RoX AI Studio bring to the customers?

• Faster Time-to-Market: Shift-left your AI product lifecycle. Start model evaluation and iteration early, long before our silicon gets delivered to your labs!
• Managed, Scalable Infrastructure: Forget about maintaining costly labs. RoX AI Studio delivers scale, security, redundancy, and automation out of the box.
• Effortless Experimentation: Register your own models (BYOM), spin up inference experiments, and compare results easily—all through a simple dashboard.
• Collaborate with Confidence: Centralized, cloud-based access lets distributed global teams work together seamlessly on model benchmarking and evaluations.

Imagine a world where your AI engineers are instantly productive, your teams collaborate without boundaries, and your prototypes move from idea to reality faster than ever before. With RoX AI Studio, that world is already here!

Sign up for a hands-on demo of RoX AI Studio on your journey to intelligent, efficient, and safe software-defined vehicles.

Shashank Bangalore Lakshman
SoC MLOps Engineering Manager

The post Driving the Future of Automotive AI: Meet RoX AI Studio appeared first on Edge AI and Vision Alliance.

We all know the AI revolution train is heading straight for the Embedded Station. Some of us are already in the driver’s seat, while others are waiting for the first movers to pave the way so we can become fast adopters. No matter where you are on this journey, one thing becomes clear: AI must adapt to the embedded application sandbox—not the other way around.

Embedded applications typically operate within a power envelope ranging from milliwatts to around 10 watts. For AI to be effective in many embedded markets, it must respect the power-performance boundaries of the application. Imagine your favorite device that you charge once a day. If adding embedded AI to a product means you now need to charge it every four hours, you are likely to stop using the product altogether.

This is where embedded AI fundamentally differs from cloud AI. In the cloud, adding more computations is often the default solution. But in embedded systems, the level of AI compute must be dictated by what the overall power and performance constraints allow. You can’t just throw more compute silicon at the problem.

There are two key approaches to scaling AI effectively for embedded applications:

1. Process Technology

At the foundational level, advanced process technologies like GlobalFoundries’ 22FDX+ with Adaptive Body Biasing offer a compelling solution. These transistors can deliver high performance during compute-intensive tasks while maintaining low leakage during idle or always-on modes. This dynamic adaptability ensures that the overall power-performance integrity of the application is preserved.

2. Alternative Compute Architectures

Emerging architectures like neuromorphic computing are gaining attention for their ability to run inference at a fraction of the power—and with lower latency—compared to traditional models. These ultra-low-power solutions are particularly promising for applications where energy efficiency is paramount and real-time response is also important.

BrainChip’s AKD1500 Edge AI co-processor, built on GlobalFoundries 22FDX platform, demonstrates how neuromorphic design can make AI practical for the smallest and most power-sensitive devices. Powered by the company’s AkidaTM technology, the chip uses an event-based approach, processing only when there’s information thereby avoiding the constant compute cycles that waste energy by reading and writing to either on-chip SRAM or off-chip DRAM as in traditional AI systems.  The co-processor performs event-based convolutions that leverage sparsity throughout the whole network in activation maps and kernels, significantly reducing computation power and latency by running as many layers on the Akida TM fabric. The diagram below shows all the interfaces, as well as the 8 Node Akida IP as the centerpiece of the AI co-processor.

The design further improves efficiency by handling data locally and using operations that cut power consumption dramatically. The result is a chip that delivers real-time intelligence while operating within just a few hundred milliwatts, making it possible to add AI features to wearable, sensors, and other AIoT devices that previously relied on the cloud for such capability.

The Akida low-cost, low-power AI co-processor solution offers a silicon-proven design that has already demonstrated critical performance metrics, substantially reducing risk for developers. With fully functional interfaces tested at operational speeds and proven interoperability across multiple MCU and MPU boards, the platform ensures seamless integration. The AKD1500 co-processor supports both power-conscious MCUs via SPI4 and high-performance MPUs through M.2 and PCIe interfaces, providing flexibility across many configurations. Enabling software development early with silicon prototypes accelerates time to market. Several customers have already advanced to prototype stages, validating the design’s maturity and readiness for deployment. As an example, Onsor Technologies’ Nexa smart glasses utilize the AKD1500 for low power inference to predict epileptic seizures, providing quality-of-life benefits for those suffering from epilepsy.

The best part of this is that the AKD1500 can be used with any low cost existing MCU with a SPI interface or an Applications processor where there is a PCIe connection available for higher performance. Adding the AKD1500 AI co-processor makes the time to market very short with available MCUs today.

Final Thoughts

As AI starts to sweep across the length and breadth of embedded space , right sizing becomes not just a technical necessity but a strategic imperative. The goal isn’t to fit the biggest model into the smallest device – it’s to fit the right model into the right device, with the right balance of performance, power, and user experience.

Anand Rangarajan
Director, End Markets, GlobalFoundries

Todd Vierra
Vice President, Customer Engagement, BrainChip

The post Right Sizing AI for Embedded Applications appeared first on Edge AI and Vision Alliance.

The gap between robotics prototypes and production deployments has always been an infrastructure problem disguised as a hardware problem. Teams build incredible computer vision models and robotic control systems on NVIDIA Jetson developer kits, only to hit a wall when scaling to production fleets. The bottleneck isn’t the AI or the algorithms—it’s the months spent building custom Linux systems, provisioning infrastructure, and OTA mechanisms that should have been solved problems.

Today, we’re announcing native provisioning support for NVIDIA Jetson Orin Nano, Orin NX and AGX Orin in Avocado OS. This completes our production software stack for the industry’s leading AI edge hardware, delivering deterministic Linux, secure OTA updates, and fleet management from day one.

What We’ve Learned About Production Jetson Deployments

Through partnerships with companies like RoboFlow and SoloTech, and conversations with teams building everything from autonomous mobile robots to industrial smart cameras, a clear pattern emerged. The technical challenges weren’t about AI models or robotic control algorithms—teams had those figured out. The bottleneck was infrastructure.

Teams consistently hit the same obstacles:

• Custom Yocto BSP builds consuming 3-6 months of engineering time
• RTC configuration issues causing timestamp failures in vision pipelines
• Fragile update mechanisms that break when scaling beyond dozens of devices
• Manual provisioning workflows that don’t translate to manufacturing partnerships
• Security compliance requirements eating bandwidth from core product development

These aren’t edge cases. This is the standard experience of taking Jetson from prototype to production. And it’s exactly backward—teams solving hard problems in robotics and computer vision shouldn’t be rebuilding the same embedded Linux infrastructure.

Premium Hardware Deserves Production-Ready Software

NVIDIA Jetson Orin Nano delivers 67 TOPS of AI performance with exceptional power efficiency. It’s the computational foundation for modern edge AI—supporting everything from multi-camera vision systems to real-time SLAM processing to local LLM inference. The hardware is production-ready.

The software needs to match.

What “production-grade” actually means:

Stable Base OS: Deterministic Linux that supports robust solutions. Not Ubuntu images that drift with package updates. Reproducible, image-based systems where every device runs identical, validated software.

Full NVIDIA Tool Suite: CUDA, TensorRT, OpenCV—pre-integrated and production-tested. Not reference implementations that require months of BSP work. The complete NVIDIA stack, ready to support inference solutions from partners like RoboFlow and SoloTech.

Day One Provisioning: Factory-ready deployment without custom scripts and USB ceremonies. Cryptographically verified images, hardware-backed credentials, and deterministic flashing workflows that integrate with manufacturing partners.

Fleet-Scale Operations: Atomic OTA updates with automatic rollback. Phased releases with cohort targeting. Air-gapped update delivery for secure environments. Infrastructure that works reliably across thousands of devices.

This is what we mean by production-ready hardware meeting production-grade software. Jetson provides the computational horsepower. Avocado OS and Peridio Core provide the operational infrastructure to actually ship products.

Complete Stack: From Build to Fleet

With Jetson provisioning now available, teams get the complete deployment pipeline:

Build Phase

• Pre-integrated NVIDIA BSPs with validated hardware support
• Modular system composition using declarative configuration
• Reproducible builds with cryptographic verification
• CUDA, TensorRT, ROS2, OpenCV—all validated and integrated

Provisioning Phase

• Native Jetson flashing via tegraflash profile
• Automated partition layout and bootloader configuration
• Factory credential injection for fleet registration
• Deterministic provisioning from Linux host environments

Deployment Phase

• Atomic, image-based OTA updates with automatic rollback
• Phased releases with cohort targeting
• SBOM generation and CVE tracking
• Air-gapped update delivery for secure environments

Fleet Operations

• Centralized device management via Peridio Console
• Real-time telemetry and health monitoring
• Remote access for debugging and diagnostics
• 10+ year support lifecycle matching industrial hardware

This isn’t a reference design or example code. It’s production infrastructure that scales from 10 devices to 10,000 and beyond.

Why This Matters: Robotics is Moving Faster Than Expected

The robotics industry is accelerating at an unprecedented pace. The foundational layer—perception—is rapidly maturing, unlocking capabilities that seemed years away just months ago. Vision language models (VLMs) and vision-language-action models (VLAs) are fundamentally changing how robots understand and interact with their environments. Engineers who once relied entirely on deterministic control systems are now integrating fine-tuned AI models that can handle ambiguity and adapt to novel situations. The innovation happening right now suggests 2026 will be a breakout year for practical robotics deployment.

Last week at Circuit Launch’s Robotics Week in the Valley, we saw this firsthand. Teams that aren’t roboticists or computer vision experts were training models with RoboFlow, integrating VLA platforms like SoloTech, and building working demonstrations in hours—not weeks.

The AI tooling has advanced exponentially. Inference frameworks are mature. Hardware platforms like Jetson deliver exceptional performance. But embedded Linux infrastructure has been the persistent bottleneck preventing teams from shipping at the pace they’re prototyping.

This matters because:

When prototyping velocity increases 10x, production infrastructure can’t remain a 6-month investment. Teams building breakthrough applications need to move from working demo to deployed fleet at the same pace they move from idea to working demo.

The companies winning in robotics will be the ones focused on their core innovation—better vision algorithms, more sophisticated manipulation, smarter navigation. Not the ones rebuilding Yocto layers and debugging RTC drivers.

Technical Foundation: Why Provisioning is Hard

The challenge with Jetson provisioning isn’t technical complexity—it’s reproducibility at scale. Most teams start by configuring their development board manually: installing packages, setting up environments, tweaking configurations until everything works. Then they try to capture those steps in scripts to replicate the setup on the next device.

This manual-to-scripted approach falls apart quickly. What runs perfectly on your desk becomes unpredictable in production. By the time you’re managing even a handful of devices, you’re troubleshooting subtle environment differences, dealing with drift from package updates, and questioning whether any two devices are truly running the same stack.

Production provisioning solves this fundamentally differently. Instead of scripting manual steps, you’re building reproducible system images where every device boots into an identical, validated environment. The OS becomes a clean foundation—deterministic, verifiable, and ready to run whatever AI toolchain your application requires. No configuration drift. No “it works on my machine” surprises.

This is where Avocado OS and NVIDIA’s tegraflash tooling come together. We’ve integrated deeply with NVIDIA’s BSP to automate the entire provisioning workflow—partition layouts, bootloader configuration, cryptographic verification, hardware initialization sequences. The complexity is still there, but it’s handled systematically rather than cobbled together through scripts.

We document the Linux host requirement explicitly because it matters. Provisioning workflows require reliable hardware enumeration and direct device access. macOS and Windows introduce VM-in-VM architectures that create timing issues and device passthrough complexity. Native Linux (Ubuntu 22.04+, Fedora 39+) ensures consistent, reliable provisioning.

For production deployments, this integrates with manufacturing partners. Advantech, Seeed Studio, and ecosystem partners can run provisioning at end-of-line, delivering pre-configured devices directly to deployment sites. Zero-touch deployment at scale.

Scale Across the Jetson Family

Teams can scale up and down within the Jetson family with unified toolchains and processes across the Jetson family:

• NVIDIA Jetson Orin Nano: 67 TOPS, efficient edge AI for vision and robotics
• NVIDIA Jetson Orin NX: Up to 157 TOPS for balanced performance for production deployments
• NVIDIA Jetson AGX Orin: Up to 275 TOPS for demanding AI workloads
• NVIDIA Jetson Thor (coming soon): Next-generation automotive and robotics platform

One development workflow. Consistent provisioning. Predictable behavior across the product line. This matters when your prototype needs to scale, or when different deployment scenarios require different performance tiers.

Getting Started: Production-Ready in Minutes

For teams ready to move from prototype to production, our provisioning guide walks through the complete workflow—from initializing your project to flashing your first device.

The entire process, from clean hardware to production-ready deployment, takes minutes, not months. The guide covers everything you need: Linux host setup, project initialization, building production images, and first boot configuration.

What’s Next: NVIDIA Momentum

Provisioning is the foundation. What comes next is ecosystem momentum.

We’re working with partners across the robotics and computer vision stack—from inference platforms like RoboFlow and SoloTech to hardware manufacturers like Advantech. The goal is creating a complete solution ecosystem where teams can focus entirely on their application layer while we handle everything below it.

We should talk if you are:

• Building on Jetson and struggling with the path to production.
• Evaluating hardware platforms and need production software from day one.
• Just getting started and want to avoid months of infrastructure work.

Production Software That Matches Production Hardware

Our thesis has always been that embedded engineers should ship applications, not operating systems. The robotics acceleration we’re seeing validates this more than ever. Teams have breakthrough ideas for autonomous systems, vision AI, and robotic manipulation. They shouldn’t spend months on Linux infrastructure.

Jetson provisioning is production-ready today. It’s the result of deep technical work, extensive partner validation, and clear understanding of what teams actually need when taking hardware to production.

Production-ready hardware. Production-grade software. Available now.

Ready to deploy production-ready Jetson? Check out our Jetson solution overview, explore the provisioning guide, or request a demo to discuss your use case.

If you’re working with Jetson and want to connect about production deployment challenges, join our Discord or reach out directly—we’d love to learn about your use case and how we can help.

Bill Brock
CEO, Peridio

The post Production Software Meets Production Hardware: Jetson Provisioning Now Available with Avocado OS appeared first on Edge AI and Vision Alliance.