Future Electronics — How New Image Sensors Meet Needs of Latest AI-Based Vision Systems
onsemi

How New Image Sensors Meet Needs of Latest AI-Based Vision Systems

By Davide Osto, EMEA Business Development Manager (Sensors), Future Electronics and

David Mills, EMEA Strategic Business Development Manager in the Intelligent Sensing Group, onsemi

The technologies underpinning Artificial Intelligence (AI) and machine learning have entered the mainstream of industrial electronics in the past three years. For most of the 21st century, implementing AI has been a high-cost, high-effort endeavor possible only for a handful of the world’s richest and most powerful companies and government agencies.

 

Now, however, low-cost microprocessors and FPGAs have gained sufficient computing power to run AI inference engines, and the massive scale of the internet has enabled the generation of large stores of tagged data for use in training data sets for machine learning.

 

As a result, almost any industrial OEMs, including start-up companies, can consider deploying AI techniques to enhance the application of existing products, or even to create entirely new applications.

 

It has emerged that image processing is one of the most effective applications for AI technology. In Intelligent Transportation Systems (ITS) for instance, imaging systems may be used to assess traffic flows and automatically adjust traffic signals, or to provide real-time information to nearby vehicles about the location of unoccupied parking spaces (see Figure 1). In smart farming, cameras can measure the color of fruit on the tree or bush, and direct automatic harvesting equipment to pick only ripe fruit. In security and surveillance, imaging can support automated access control equipment which can recognize the faces of authorized users.

 

These applications are already in use or under development, and there are thousands more potential applications for intelligent imaging which have yet to be devised or discovered.

Future Electronics — How New Image Sensors Meet Needs of Latest AI-Based Vision Systems

Fig. 1: Real-time information about the location of available parking spaces provided automatically to nearby cars could save drivers time and reduce congestion locally. (Image credit: Rachmaninoff under Creative Commons license.)

 

This means that the pattern of deployment of CMOS image sensors, and the cameras in which they are embedded, are about to be utterly transformed. Since the invention of the semiconductor image sensor, the market has been configured to supply a small number of manufacturers which each used huge volumes of the devices: the handful of manufacturers of still and video cameras and, latterly, smartphones. Manufacturers of cameras for industrial, professional or consumer uses are imaging specialists: they have their own, proprietary development tools and technologies for implementing image sensors in camera designs, and the image sensor manufacturers have provided specialized or even custom products to meet their exact specifications.

 

The new, AI-driven market for image sensors is very different: a manufacturer of farm harvesting machines, or of traffic-control signals, or of smart door locks, will normally have little or no imaging expertise in-house. And since the quantities of image sensors they require are tiny compared to a camera manufacturer, they will want to choose from a range of standard products so that they benefit from lower unit prices.

 

Now image sensor manufacturers are developing new product offerings to meet this emerging demand from embedded systems manufacturers which want to add intelligent imaging to the capabilities of their products. And they are finding that the requirements of these new users are different from those of big camera manufacturers. The traditional focus of image sensor technical evaluation is on optical performance characteristics such as:

  • Resolution
  • Dynamic range
  • Frame rate
  • Sensitivity and low-light performance
  • Noise

 

This article, however, argues that embedded systems developers will be as interested in two important features of the new generation of sensors: platform capability, and the availability of user-friendly development tools and resources.

 

Platform for Developing Multi-Functional Families of End Products

The emerging demand for standard image sensor products for niche applications in ITS, smart farming, security and elsewhere is inducing the handful of dominant image sensor manufacturers to extend their product ranges with new, standard products.

 

One of these manufacturers, onsemi, has recognized that many of these new, niche customers want to develop multiple product variants to meet different performance or application requirements, and to market them at different price points. This is why, in onsemi’s XGS family of image sensors, customers can find parts with resolutions ranging from 2Mpixels up to 45Mpixels.

 

These sensors in the XGS family are based on the same platform, with two footprint options: XGS sensors with resolution of 2Mpixels, 3Mpixels, 5Mpixels, 8Mpixels, 9.4Mpixels, 12Mpixels, or 16Mpixels share a common footprint and are compatible with the industry-standard, compact 29mm x 29mm camera size. XGS sensors with resolution of 20Mpixels, 30Mpixels, 32Mpixels, or 45Mpixels support the Super 35mm optical format.

 

The XGS family is broad enough to support a range of price and performance requirements. At the high-performance end, the XGS 45000 produces an 8K (8192px x 4320px) video output at up to 60 frames/s. The lowest-cost device, the XGS 2000, produces a 12-bit output at full 1920px x 1200px resolution at up to 220 frames/s, in monochrome or Bayer color configuration.

 

All XGS family sensors benefit from an advanced pixel design incorporating a global shutter, which produces no blurring or image artefacts when capturing fast-moving images. The pixel architecture offers high bandwidth while operating at low power with low noise.

 

Giving New Camera Designers the Tools to do Their Job

One response to the needs of the new customers developing AI-driven imaging systems, then, is to provide a flexible platform of image sensor products.

 

But there is another way in which this emerging set of customers is different from established camera manufacturers: they do not have sophisticated, proprietary optical development tool suites and prototype development platforms available in-house. Without such tools, the task of developing a new, application-oriented camera is dauntingly difficult and time-consuming.

 

onsemi has responded to this emerging need by providing its own suite of resources to support every stage of development, from evaluation to software development to camera design.

 

The DevWare PC-based application software supports sophisticated image sensor evaluation. When used in combination with the USB3 imaging baseboard and XGS image sensor headboard, it gives a high capture rate and flexible options for evaluation. The DevWare platform is ideal when the developer wants the freedom to evaluate every aspect of the XGS image sensor performance and pixel operation, with access to all register settings.

 

System software development is enabled by X-Celerator: this consists of an FPGA Mezzanine Card (FMC) format camera head which provides a direct interface to standard FPGA development environments (see Figure 2). X-Celerator is backed by public FPGA code, so no software license agreement is required for its use. An important advantage of this development platform is that the development team can write camera software before their own hardware design is finalized, enabling the hardware and software teams to work in parallel, shortening time-to-market.

Future Electronics — How New Image Sensors Meet Needs of Latest AI-Based Vision Systems

Fig. 2: The X-Celerator camera kit enables software development in an FPGA-based hardware environment. (Image credit: onsemi)

 

Design of the camera hardware may be accelerated by using onsemi’s X-Cube reference design, a complete, end-to-end camera implementation which has a MIPI interface (see Figure 3). The X-Cube design supports all XGS image sensors with resolution up to 16Mpixels in an industry-standard 29mm x 29mm form factor, allowing the use of C-mount or S-mount lenses.

 

An adapter board provides an interface to the USB3 baseboard, which links the reference design to the DevWare software, enabling image capture, processing and analysis on a PC. onsemi supplies all design files, including schematics, layout, and lens mount CAD, to customers to use and modify.

Future Electronics — How New Image Sensors Meet Needs of Latest AI-Based Vision Systems

Fig. 3: The X-Cube reference design provides a complete camera system based on an XGS image sensor with resolution up to 16Mpixels. (Image credit: onsemi)

 

Support to Build Out a Complete AI-Based Imaging System

onsemi’s XGS image sensors are comparable with the best that the market has to offer in conventional optical terms, providing high resolution, fast frame rates, and low power consumption. The big breakthrough for industrial customers is the new way in which onsemi supports OEMs with no or little in-house camera expertise, providing a wide range of development resources and a flexible image sensor platform.

 

The support for embedded system OEMs which are not vision specialists may be supplemented by turning to onsemi’s franchise distributor, Future Electronics. Future Electronics can provide in-depth technical support for new camera developments through its regional imaging experts in the UK and Europe, as well as industry-leading supply-chain services for buyers of image sensors. Future Electronics also provides a full range of supporting components for AI-based imaging applications, including a range of FPGA options from Microchip, Lattice and Quicklogic for the AI back end.

Share This

Share on facebook
Share on twitter
Share on linkedin

Subscribe to our newsletters

Subscribe to Future Electronics

Get access to the latest product information, technical analysis, design notes and more

Choose your region

Resources

Related Articles