Hyper 3D-AI

Artificial Intelligence for 3D multimodal point cloud classification

The aim of this project is to develop software tools that can efficiently analyse images in a three-dimensional context – for example medical images or pictures from cameras mounted on self-driving cars. Artificial intelligence (AI) is already capable of detecting anomalies in MRI images, for example, by classifying the image data. However, many of the existing AI algorithms only work with two-dimensional images. While they can analyse neighbouring pixels in the image, they cannot recognise whether they reside on the same plane as each other in reality.

“We work with point clouds, where we have three-dimensional coordinates for each point,” says Dr. Sandra Lorenz of the Helmholtz Institute Freiberg for Resource Technology. “That is a completely different architecture from what is used for analysing pixels in photos. However, the current methods can’t really cope properly with these point clouds yet, even though point clouds offer a much better depiction of the real world.” The researchers now want to close this gap. By characterising pixels in 3D space, this will open up new possibilities in fields like exploration and mining, medicine and autonomous systems.

AI should then be able to achieve multimodal classification, or in other words distinguish objects or domains out of data coming from multiple sensors. For mining, as one possible application, this could mean the software would automatically recognise deposits of mineral raw materials, for example, based on spectral properties or colours.


Afifi, A. J., Thiele, S. T., Rizaldy, A., Lorenz, S., Ghamisi, P., Tolosana-Delgado, R., Kirsch, M., Gloaguen, R., & Heizmann, M. (2024). Tinto: Multisensor Benchmark for 3-D Hyperspectral Point Cloud Segmentation in the Geosciences. IEEE Transactions on Geoscience and Remote Sensing, 62, 1–15. https://doi.org/10.1109/TGRS.2023.3340293
Bihler, M., Roming, L., Jiang, Y., Afifi, A. J., Aderhold, J., Čibiraitė-Lukenskienė, D., Lorenz, S., Gloaguen, R., Gruna, R., & Heizmann, M. (2023). Multi-sensor data fusion using deep learning for bulky waste image classification. Automated Visual Inspection and Machine Vision V, 12623, 69–82. https://doi.org/10.1117/12.2673838
Thiele, S., Afifi, A. J., Lorenz, S., Tolosana-Delgado, R., Kirsch, M., Ghamisi, P., & Gloaguen, R. (2023). LithoNet: A benchmark dataset for machine learning with digital outcrops (No. EGU23-14007). Copernicus Meetings. https://doi.org/10.5194/egusphere-egu23-14007
HIF-EXPLO. (2022). hifexplo/hylite. https://github.com/hifexplo/hylite (Original work published 2020)
Schambach, M., Shi, J., & Heizmann, M. (2021). Spectral Reconstruction and Disparity from Spatio-Spectrally Coded Light Fields via Multi-Task Deep Learning. 2021 International Conference on 3D Vision (3DV), 186–196. https://doi.org/10.1109/3DV53792.2021.00029
Thiele, S., Lorenz, S., Bnoulkacem, Z., Kirsch, M., & Gloaguen, R. (2021). Hyperspectral mineral mapping of cliffs using a UAV mounted Hyspex Mjolnir VNIR-SWIR sensor. 2021, 1–3. https://doi.org/10.3997/2214-4609.2021629011
Thiele, S. T., Lorenz, S., Kirsch, M., Cecilia Contreras Acosta, I., Tusa, L., Herrmann, E., Möckel, R., & Gloaguen, R. (2021). Multi-scale, multi-sensor data integration for automated 3-D geological mapping. Ore Geology Reviews, 136, 104252. https://doi.org/10.1016/j.oregeorev.2021.104252
Kirsch, M., Lorenz, S., Thiele, S., & Gloaguen, R. (2021). Characterisation of Massive Sulphide Deposits in the Iberian Pyrite Belt Based on the Integration of Digital Outcrops and Multi-Scale, Multi-Source Hyperspectral Data. 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, 126–129. https://doi.org/10.1109/IGARSS47720.2021.9554149
Schambach, Maximilian. (2021). A highly textured multispectral light field dataset. Karlsruhe Institute of Technology (KIT). https://doi.org/10.35097/500
Li, L., & Heizmann, M. (2021). 2.5D-VoteNet: Depth Map based 3D Object Detection for Real-Time Applications. The 32nd British Machine Vision Conference 2021, 1. https://publikationen.bibliothek.kit.edu/1000140306
Schambach, M., & Heizmann, M. (2020). A Multispectral Light Field Dataset and Framework for Light Field Deep Learning. IEEE Access, 8, 193492–193502. https://doi.org/10.1109/ACCESS.2020.3033056
Lorenz, S. (n.d.). Hyper 3D-AI: Artificial Intelligence for 3D multimodal point cloud classification. Retrieved July 7, 2022, from https://www.hzdr.de/publications/Publ-33167

Other projects


Hyper 3D-AI

Artificial Intelligence for 3D multimodal point cloud classification

The aim is to develop an artificial intelligence that can achieve the fusion of two-dimensional data with three-dimensional information. Based on this, the software would simultaneously be able to recognise image characteristics as well as the spatial relationships between different objects.
Image: DLR


On the trail of the mystery of the laws of calving

Researchers still face a mystery when it comes to the laws by which glaciers calve. This project aims to use satellite imagery, artificial intelligence, mathematical optimisation and a new data processing pipeline to track the movements of glacier fronts in Antarctica to get closer to solving the mystery.


Solar Image-based Modelling

The aim of the project is to develop an algorithm by which computers can automatically predict the space weather. This will make use of datasets of solar images that have been captured from space. The method could replace computationally demanding physics-based models and deliver space weather forecasts long before the effects of solar events are […]