Hyper 3D-AI

Artificial Intelligence for 3D multimodal point cloud classification

The aim of this project is to develop software tools that can efficiently analyse images in a three-dimensional context – for example medical images or pictures from cameras mounted on self-driving cars. Artificial intelligence (AI) is already capable of detecting anomalies in MRI images, for example, by classifying the image data. However, many of the existing AI algorithms only work with two-dimensional images. While they can analyse neighbouring pixels in the image, they cannot recognise whether they reside on the same plane as each other in reality.

“We work with point clouds, where we have three-dimensional coordinates for each point,” says Dr. Sandra Lorenz of the Helmholtz Institute Freiberg for Resource Technology. “That is a completely different architecture from what is used for analysing pixels in photos. However, the current methods can’t really cope properly with these point clouds yet, even though point clouds offer a much better depiction of the real world.” The researchers now want to close this gap. By characterising pixels in 3D space, this will open up new possibilities in fields like exploration and mining, medicine and autonomous systems.

AI should then be able to achieve multimodal classification, or in other words distinguish objects or domains out of data coming from multiple sensors. For mining, as one possible application, this could mean the software would automatically recognise deposits of mineral raw materials, for example, based on spectral properties or colours.

Publications

Afifi, A. J., Thiele, S. T., Rizaldy, A., Lorenz, S., Ghamisi, P., Tolosana-Delgado, R., Kirsch, M., Gloaguen, R., & Heizmann, M. (2024). Tinto: Multisensor Benchmark for 3-D Hyperspectral Point Cloud Segmentation in the Geosciences. IEEE Transactions on Geoscience and Remote Sensing, 62, 1–15. https://doi.org/10.1109/TGRS.2023.3340293

Bihler, M., Roming, L., Jiang, Y., Afifi, A. J., Aderhold, J., Čibiraitė-Lukenskienė, D., Lorenz, S., Gloaguen, R., Gruna, R., & Heizmann, M. (2023). Multi-sensor data fusion using deep learning for bulky waste image classification. Automated Visual Inspection and Machine Vision V, 12623, 69–82. https://doi.org/10.1117/12.2673838

Thiele, S., Afifi, A. J., Lorenz, S., Tolosana-Delgado, R., Kirsch, M., Ghamisi, P., & Gloaguen, R. (2023). LithoNet: A benchmark dataset for machine learning with digital outcrops (No. EGU23-14007). Copernicus Meetings. https://doi.org/10.5194/egusphere-egu23-14007

HIF-EXPLO. (2022). hifexplo/hylite. https://github.com/hifexplo/hylite (Original work published 2020)

Schambach, M., Shi, J., & Heizmann, M. (2021). Spectral Reconstruction and Disparity from Spatio-Spectrally Coded Light Fields via Multi-Task Deep Learning. 2021 International Conference on 3D Vision (3DV), 186–196. https://doi.org/10.1109/3DV53792.2021.00029

Thiele, S., Lorenz, S., Bnoulkacem, Z., Kirsch, M., & Gloaguen, R. (2021). Hyperspectral mineral mapping of cliffs using a UAV mounted Hyspex Mjolnir VNIR-SWIR sensor. 2021, 1–3. https://doi.org/10.3997/2214-4609.2021629011

Thiele, S. T., Lorenz, S., Kirsch, M., Cecilia Contreras Acosta, I., Tusa, L., Herrmann, E., Möckel, R., & Gloaguen, R. (2021). Multi-scale, multi-sensor data integration for automated 3-D geological mapping. Ore Geology Reviews, 136, 104252. https://doi.org/10.1016/j.oregeorev.2021.104252

Kirsch, M., Lorenz, S., Thiele, S., & Gloaguen, R. (2021). Characterisation of Massive Sulphide Deposits in the Iberian Pyrite Belt Based on the Integration of Digital Outcrops and Multi-Scale, Multi-Source Hyperspectral Data. 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, 126–129. https://doi.org/10.1109/IGARSS47720.2021.9554149

Schambach, Maximilian. (2021). A highly textured multispectral light field dataset. Karlsruhe Institute of Technology (KIT). https://doi.org/10.35097/500

Li, L., & Heizmann, M. (2021). 2.5D-VoteNet: Depth Map based 3D Object Detection for Real-Time Applications. The 32nd British Machine Vision Conference 2021, 1. https://publikationen.bibliothek.kit.edu/1000140306

Schambach, M., & Heizmann, M. (2020). A Multispectral Light Field Dataset and Framework for Light Field Deep Learning. IEEE Access, 8, 193492–193502. https://doi.org/10.1109/ACCESS.2020.3033056

Lorenz, S. (n.d.). Hyper 3D-AI: Artificial Intelligence for 3D multimodal point cloud classification. Retrieved July 7, 2022, from https://www.hzdr.de/publications/Publ-33167

Project partners

HZDR, KIT

Dr. Sandra Lorenz
Prof. Dr. Michael Heizmann
Dr. Pedram Ghamsi
Norbert Mitschke

Duration

36 months

Primary Contact

Dr. Sandra Lorenz

s.lorenz@hzdr.de

Other projects

Image: Paul Kamm, Helmholtz-Zentrum Berlin

Avanti

X-ray tomoscopy of dynamic manufacturing processes
How can the manufacturing processes of materials be mapped at the smallest level? How do you train an artificial intelligence to analyze these processes automatically? That's the focus of the Avanti project, which aims to improve X-ray tomoscopy – the imaging and quantification of three-dimensional images of very fast-moving processes.

SATOMI

Tackling the segmentation and tracking challenges of growing colonies and microbialdiversity
An artificial intelligence will observe the growth of bacteria: from microscope images of bacterial cultures taken at regular intervals, it will precisely track the development and division of individual cells – even when multiple bacterial species are cultivated together.

Image: Aaron Christian Banze

HYPER-AMPLIFAI

Advancing Visual Foundation Models for Multi-/Hyperspectral Image Analysis in Agriculture/Forestry
The project aims to make advanced AI models accessible for Hyperspectral Earth Observation, reducing computational demands, and improving environmental assessments through user-friendly interfaces.