3D Mobile Mapping Artificial Intelligence (3DMMAI), is supported by the NSERC (Natural Sciences and Engineering Research Council of Canada) Collaborative Research Development (CRD) program and Teledyne Optech in Toronto. Research grant worth $1,024,000, from NSERC, for this project. Additional cash and in-kind contributions made from Teledyne Optech were also significant: the total cash contribution is approximately about $1.5 million ($1,536,000 total; $512,000 from Teledyne Optech) and $1 million in-kind contribution ($1,048,146). Total funding is $2.5 million over four years. This research project seeks to update Canada’s critical infrastructure – the independent network of utilities, transportation, and facilities. Although Canada is the second-largest country in the world (in terms of area), with the world’s 10th largest economy, one-third of its infrastructure needs a significant update. In collaboration with Teledyne Optech, this project will develop an advanced data processing system using a deep neural network pipeline, which has recently achieved remarkable success in a computer and robotic vision and machine learning. This work will allow for the autonomous recognition of infrastructure assets using high-quality 3D models of critical networks, thus contributing to the field of infrastructure management and improving urban sustainability as a whole.

Overview: Our lab will lead the development of deep learning-based 3D scene classification pipelines including noise filtering, terrain filtering, and semantic segmentation for the point cloud. We will also work on high perception pipelines for LIDAR and visual SLAM and development of the co-alignment network. It will also focus on transmission network modeling.

3D Scene Classification

Noise Filtering:

The 3DMMAI pipeline includes several stages of post data acquisition processing, and the first significant step is noise filtering.LiDAR (Light Detection and Ranging) mounted with static and mobile vehicles has been rapidly adopted as a primary sensor for mapping natural and built environments for a range of civil and military applications. Recently, technology advancement in electro-optical engineering enables acquiring laser returns at high pulse repetition frequency (PRF) from 100Hz to 2MHz for airborne LiDAR, which leads to an increase in the density of 3D point cloud significantly. Traditional systems with lower PRF had a single pulse-in-air zone (PIA) big enough to avoid a mismatch between pulse pair at the receiver. Modern multiple pulses-in-air (MPIA) technology ensures multiple windows of operational ranges for single flight lines and no blind-zones; the downside of the technology is a projection of atmospheric returns closer to the same PIA zone of neighboring ground points and more likely to be overlapping with objects of interest. These characteristics of noise compromise the quality of the scene and encourage usage of noise filtering neural network as existing filters are not effective and scalable for the large-scale point cloud dataset.

Contributors: Maryam Jameela

Semantic Segmentation:

The next natural step of the pipeline for scene understanding and classification is semantic segmentation, which labels every point and pixel in the point cloud and image respectively of their enclosing object or region. There are multiple existing deep neural networks available for semantically segment indoor and outdoor environments, but they lack scalability for large-scale dense point cloud, efficient downsampling strategies, and effective deep feature aggregation modules. These systems also failed to achieve high mean intersection over union (mIoU) and overall accuracy (OA) on all classes, especially for long-tailed data distribution. With these challenges in mind, it is beneficial to develop a deep neural network scene classification solution utilizing optical images and clean point cloud to extract characteristics unique to multimodality.

Contributors: Jacob Yoo, Maryam Jameela