A city stroll through time and space

By Ute Schönfelder

Virtual strolls through Dresden’s Old Town, taking in the city’s history, are now possible thanks to a web application called »4D Browser«. The app has been developed by Prof. Dr Sander Münster and his team in the course of a research project funded by the Federal Ministry of Education and Research (BMBF). This interactive platform allows users to explore a 3D model of the city along with historical images and texts about its buildings and architecture.

»What’s so special is that the historical photos in the model are shown from where they were taken and from the precise camera position,« as Sander Münster explains. »This makes it possible to see, for example, from which direction buildings were often photographed, the view of the city enjoyed by photographers in the past, and how they framed a given part of the city.« A timeline makes it possible to trace changes in buildings and neighbourhoods over time, documented in photographs. The 3D models and photos are also connected with text sources.

AI identifies the precise camera position

Data is constantly being added to the »4D Browser«, with new images automatically located and visual content linked with texts. In the past, photos were primarily selected and allocated to certain buildings using metadata. However, as this metadata is often inaccurate or incomplete, the researchers have now turned to a range of AI tools and are analysing similarities between images with the help of ML models. In order to locate a photograph at the exact location it was taken, the team apply neural networks that use feature identification and matching processes. Computer vision techniques help to fine-tune the positioning, drawing on AI-based edge detection and feature matching.

The researchers also apply AI technologies to connect a variety of image, text and 3D data and also to process text data. This includes automatically annotating the names of people and places in text sources as well as semantic text annotations based on similarities between words. The annotations in image, text and 3D data are analysed and compared in a graph database to establish links between similar content. The tool can also automatically generate suitable captions for these photographs. Although this is a rule-based process at present, it is set to be refined with AI in the future.

Limited data sets restrict the AI's performance

»None of the AI technologies we use are new or developed specifically for this project,« emphasizes Münster. »Instead, we adapt existing approaches and combine them with other techniques to achieve higher accuracy and reliability.«

However, as Münster explains, working with historical sources presents a whole host of challenges. For example, the historical photos are digital copies that do not usually contain any information about the parameters of the camera. This limits the researchers’ ability to draw on technologies that require such information to reconstruct the camera position in photos.

There is also one fundamental problem with regard to the use of AI in the historical sciences that has yet to be solved: the amount of historical data available to train AI models is rather small, which limits the models’ capabilities. This is particularly evident with regard to architecture. While AI models are already proficient in the automatic identification of roofs and windows, they do not yet have the ability to identify other elements of a building’s façade—such as pilaster or architraves.