The Old Narva prototype was born out of search for new engaging solutions for access to, and the application of, digital photo archives. Digitised collections continue to be relatively unused and unexploited, despite formidable investments over several decades. Innovation in design and implementation could open the way to a plethora of new techniques for the display and dissemination of digitised resources. For the prototype we drew from features found in common ‘analogue’ games such as the jigsaw puzzle and the ‘Hot & Cold’ game. In addition to this experimental design for a ‘Photo Positioning Puzzle’, the application includes a dynamic 3D–reconstruction of an historic city square, Raekoja Plats in Old Narva, as well as an information layer for access to background information, including additional historical photos and online resources. Testing showed that ‘Photo Positioning Puzzle’ and its integration with other forms of mobile AR functionality was successful, and should have general value and suitability for additional historical places with corresponding photo collections and archived documentation.
The application has three stages and corresponding modes that the visitor/user must explore and accomplish in sequence:
- The puzzle where finding the vantage point of sixphotos based on image analysis and movement is the aim.
- Reconstruction of the Narva square in the 1930s with animation of everyday actions and events.
- Information mode where the temporal sequence is paused and one can access background information via spatially positioned hypertext links.
This triple structure starts with a problem to be solved. When the puzzle is pieced together, the reconstructed square presents itself as an audio-visual reward, a full blown indirect AR experience where one can walk around and explore the square and the facades of the various buildings. The perspective on the screen is instantly updated based on the movement of the device which accordingly also controls the virtual camera inside the 3D environment. The dynamic simulation with elements of everyday life in the square is not only a reward, but also an introduction that motivates the user to explore the background information made available after pausing the sequence of events.
Creation of the prototype was led by Gunnar Liestøl from University of Oslo. In earlier indirect AR situated simulations (sit-sims) they have explored various types of historical reconstructions of cultural heritage sites, as well as historical buildings and events. In the current project the initial idea was to scrutinise how use of digitised historical photos could be somehow integrated with the digital reconstructions for some new kind of viewing experience on location. With the Narva material the old baroque Town Hall Square stood out as a suitable site for reconstruction. Given the affinity between the process of rephotography and the ‘Hot & Cold’ game, we also had a procedure for how to connect the selected photos with a digital reconstruction: Finding the vantage point of each photo would be rewarded with a positioned 3D reconstruction of the building(s) depicted in the image. However, this solution had its obvious limitations: It would only allow us to reconstruct the buildings represented in one separate photo. We had no strategy for combining the process between several photos, in order to reconstruct all sides of the square into a unified 3D environment. The various photos selected to document the former version of the square thus needed to have some visible relationship to each other.
The most obvious game that concerns analysis of images for the purpose of combining them into a whole is the jigsaw puzzle. To finish a jigsaw puzzle one needs to analyse the visual information in each piece in order to join up those that are directly adjacent to each other. In the jigsaw puzzle the player pays attention to the shape of the individual piece, to see if it physically fits in with the neighbouring piece. Although photos follow certain formats or ratios between vertical and horizontal sides, the shape and size of a photographic picture is irrelevant here, at least compared to a piece in a jigsaw puzzle. Thus the most apparent method to connect the images was to look for horizontal overlap, a feature which was abundantly represented in the Ajapaik database. If each photo shared visual information with two other images, one on each side, and all buildings were depicted in the selected photos, a continuous panorama of images representing all facades of the old square was achieved.
It was also evident that the search for the vantage point needed to start with one of the three photos that depicted the whole or parts of the only shared structure between the old Town Hall square and the present, that is, the Town Hall building itself. In our design we have combined a solution where finding the approximate vantage point of six pictures leads to a 3D digital reconstruction of the lost environment that is the historic subject of the photos.
The application as described was tested with 18 people on location at Raekoja Plats in Narva on May 3, 2018. Six iPads were used for the test. All participants were given a quick oral introduction to how the system worked and were then individually assisted by one Norwegian student from the development team. The feedback was generally very positive.