How AI digital twins are nurturing churches, temples, and mosques

The twin, developed in just a month in partnership with Microsoft and a Paris-based digital preservation company, Iconem, will provide millions worldwide with an immersive virtual experience of the Basilica. 

While Iconem used multiple devices,including drones, 3D cameras, and lasers to capture more than 400,000 high-resolution images of the Basilica, the data was securely backed up and stored on Microsoft’s Azure Cloud. Microsoft’s AI tools refined the digital twin, producing a 3D representation accurate to the millimeter, and also used AI to help detect and map structural vulnerabilities like cracks and missing mosaic tiles.

Digital twins or virtual representations of physical entities like manufacturing plants, buildings, temples, churches, mosques, or heritage sites, have been created over the last two decades using technologies like light detection and ranging (LiDAR) scanning, 3D mapping, and photogrammetry–the process of capturing images and stitching them together to build 3D models. Heritage Building Information Modelling (H-BIM) enhances digital twins by providing accurate 3D models.

For instance, the Aga Khan Trust for Culture (AKTCI) and Tata Trusts partnered with the Archaelogical Survey of India (ASI) for over a decade to restore Delhi’s Humayun Tomb and Nizamuddin Basti using laser scanners and 3D mapping. The ministry of tourism collaborated with Google for ‘Incredible India’ to create virtual walk-throughs of famous historical places. However, most of these models have traditionally been static.

Digital twins are now being empowered with AI, enabling detailed documentation, monitoring, and proactive conservation of heritage sites. They predict vulnerabilities, simulate impacts, and guide restoration using real-time data. The data is processed with computer-aided design (CAD) and geographical information system (GIS), integrated with internet of things (IoT) data from sensors, and simulated for various scenarios.

The ancient Roman city of Pompeii, as an example, has been digitally captured multiple times, but the AI-powered twin developed as part of the EU-funded RePAIR project used machine learning algorithms to help the digital twin reconstruct broken artefacts by simulating missing pieces based on historical patterns and materials analysis. 

Likewise, following extensive war damage, Unesco is leveraging AI-powered digital twins to restore the iconic Mosul Al-Nuri Mosque in Iraq. While AI interprets drone and satellite data to create precise reconstructions and assess structural stability, the process includes simulating the mosque’s original acoustic properties and sunlight patterns to preserve its heritage. 

After the devastating fire in 2019, a traditional digital twin of the Notre-Dame Cathedral in France was created to record its pre-fire condition. An AI-powered twin, however, has since been developed to guide restoration. IBM and Ogilvy for the Voice of Art project partnered with Pinacoteca, Sao Paulo’s oldest museum, to use natural language processing (NLP), cognitive computing and machine learning (ML), aimed at allowing visitors to interact with specific paintings and sculptures in the museum, and enabling a personalised visiting experience.

Also Read: Urban India’s 3D digital twins are on their way to be born

Closer home, the Virtual Experiential Museum Ajanta (VEMA) uses digital technology to present Ajanta’s Unesco-listed cave paintings on large screens and online. Vizara Technologies, funded by the Department of Science and Technology in 2016, built 3D digital models of Indian heritage sites, including the Taj Mahal in Agra, Kashi Vishwanath temple, Konark Sun Temple in Odisha, Rani Ki Vav (stepwell) in Gujarat and the ruins of Hampi in Karnataka. 

While early 3D scans created a static model of the Sun Temple, AI-powered enhancements now simulate the temple’s weathering due to salt-laden winds and heavy rainfall. Using predictive models, conservationists can plan targeted restoration for high-risk areas. AI also aids in reconstructing intricate sculptures by learning from existing patterns and comparing them to historical records.

Likewise, the archaeological site of Hampi in Karnataka, which is known for its complex stone architecture, benefits from AI-powered digital twins that analyze how foot traffic and vibrations from nearby modern activities are affecting the stability of its monuments. Machine learning helps predict structural vulnerabilities and recommends visitor management strategies to reduce impact. 

Similarly, Hyderabad-based Hexagon created a digital twin of the tomb of Mohammed Quli Qutb Shah in Hyderabad in February by capturing images of the tomb from within and outside, and using cutting-edge technologies including reality capture, geospatial mapping, and 3D scanning along with AI tools.

The opportunities are immense, as we see. Advances in 3D AI generative content and modern photogrammetric scanning now enable non-specialists to create digital twins for historical research, according to a February 2023 paper by researchers from Lindenwood University who developed a digital twin of the Church of Santi Apostoli in Florence. 

Also Read: Your body parts will now have their AI-powered digital twins

But challenges remain

The European Union points out that the fear of the infringement of data protection or copyright laws severely limits the uptake of the technology by museums and the cultural heritage sector. Besides, there’s a lack of clarity on the commercial or business opportunities linked to the use of AI in museums and heritage organisations.

The Lindenwood Univeristy researchers point out that cultural heritage institutions face significant challenges in adopting standardized technologies for international collaboration. Diverse software, hardware, and pipelines hinder interoperability and reproducibility. 

For instance, while HoloLens 2 has been popular for AR projects, its discontinuation has created uncertainty. Similarly, VR development faces fragmentation, with Meta Quest 2 dominating but competition looming from Apple and Google, they say in the paper cited above. 

Further, varied software like Blender and Unity complicate unified workflows. The lack of global standards for methodology and archival longevity, combined with reliance on specialized technical expertise further limits cohesive collaboration and sustainable development in the field. Given the opportunities that digital twins offer, these challenges need to be addressed.

Also Read: Digital twins of organs can help us solve big challenges