Extreme Early Image Recognition Using Event Based Vision

While deep learning algorithms have advanced to a great extent, they are all designed for frame-based imagers that capture images at a high frame rate, which leads to a high storage requirement, heavy computations and very high power consumption. Unlike frame-based imagers, event-based imagers output asynchronous pixel events without the need for a global exposure time, therefore lowering both power consumption and latency. In this work, we propose an innovative image recognition technique that operates on image events rather than frame-based data, paving the way to a new paradigm of recognizing objects prior to image acquisition. To the best of our knowledge, this is the first time such a concept is introduced featuring not only an extreme early image recognition but also reduced computational overhead, storage requirement and power consumption. Our collected event-based dataset using CeleX imager and five public event-based datasets are used to prove this concept, and the testing metrics reflects how early can the neural network (NN) detects an image before the full frame image is captured. It is demonstrated that on average, for all the datasets, the proposed technique recognizes an image 38.7 ms before the first perfect event and 603.4 ms before the last event is received, which is a reduction of 34% and 69% of the time needed respectively. Further, less processing is required as the image is recognized 9,460 events earlier, which is 37% less than waiting for the first perfect recognized image. An enhanced NN that is trained on partial noisy images is also introduced to reduce this time.