Advances in AI-supported image processing now enable precise and automated recognition, localization, and segmentation of objects, and thus processes that are used in autonomous driving, for example. This potential can also be transferred to other fields of application, such as the detection of vehicle damage, dermatological abnormalities, or structural and construction damage to buildings and monuments.
Modern smartphones offer a possible technical basis for this. In addition to classic motion sensors, they have high-resolution cameras and the first models also have lidar sensors. Combining this sensor data opens up new possibilities for visual and spatial analysis.

This is where the project comes in: it investigates how aspects of human perception, in particular the interaction of different senses, i.e. different modalities, can be transferred to AI-supported analysis methods. The fusion of camera images and lidar point clouds is intended to create new computer vision models that enable more reliable detection of complex conditions and damage. This multimodal approach is based on successful methods from autonomous driving and is now being transferred to smartphone-based applications.
Together with three application partners from different fields, the method is being tested in practical scenarios, including the detection of paint and surface damage on vehicles, the analysis of conspicuous skin features, and the detection of structural damage.