Artificial Intelligence for 3D multimodal point cloud classification

The aim of this project is to develop software tools that can efficiently analyse images in a three-dimensional context – for example medical images or pictures from cameras mounted on self-driving cars. Artificial intelligence (AI) is already capable of detecting anomalies in MRI images, for example, by classifying the image data. However, many of the existing AI algorithms only work with two-dimensional images. While they can analyse neighbouring pixels in the image, they cannot recognise whether they reside on the same plane as each other in reality.

Rainbow-coloured image of a mountain landscape to reveal different layers within the rocky structure

The aim of the project Hyper3D-AI is to develop software that can simultaneously capture both the image data and the spatial relationship between different objects. This will involve the fusion of three-dimensional information with two-dimensional data. “We work with point clouds, where we have three-dimensional coordinates for each point,” says Dr. Sandra Lorenz of the Helmholtz Institute Freiberg for Resource Technology. “That is a completely different architecture from what is used for analysing pixels in photos. However, the current methods can’t really cope properly with these point clouds yet, even though point clouds offer a much better depiction of the real world.” The researchers now want to close this gap. By characterising pixels in 3D space, this will open up new possibilities in fields like exploration and mining, medicine and autonomous systems.

Image of landscape marked with dots for analysation

AI should then be able to achieve multimodal classification, or in other words distinguish objects or domains out of data coming from multiple sensors. For mining, as one possible application, this could mean the software would automatically recognise deposits of mineral raw materials, for example, based on spectral properties or colours.