Unfolding is a new take on our past project with Filip Ruisl, focusing on architectural reconstruction using the technology of augmented reality. We used augmented reality, because it allowed us to reveal the long forgotten values of old buildings with minimal intervention in the physical environment and to show them in their original location in the same time. The project was developed during our residency at the Budapest based innovation laboratory, Kitchen Budapest.

Demonstration of our final prototype.

Upon the arrival to Kitchen Budapest we were facing two major technological challenges: tracking large objects in 1:1 scale in AR and adjusting the app for varying light conditions. Large objects can be challenging to track, because even the smallest inaccuracies can lead to dissonance between the real and the augmented worlds. Varying light conditions often result the loss of tracking, when using camera image based AR tracking.

Presentation of our first proof of concept after a 24 hour hackathon.

Throughout the project we were in parallel focusing on the above mentioned challenges and also on content creation. We were researching interesting architectural sites across Budapest. Our main criteria was to find buildings of which current state largely differs from their past and original states. Online databases of old postcards and illustrations helped us to compare these old locations to their current condition. It was crucial for us to find buildings with reliable data, in order to recreate authentically. The Budapest Archives holds the largest database of architectural drawings in Budapest and it has provided us the most accurate source materials.

The Unfolding app contains three main views of the augmented building. An architectural drawing view revealing the original drawings of the architect(s) derived from the plans found in the archives, a three dimensional reconstruction created from the drawings and postcards and a textual analysis of the points of interest.

Even though the data from the archives told us a lot about the look and functions of the former buildings, it did not provide enough information to create a photorealistic rendering of the original state. After consulting with art historians and archeologists we have decided to create a visual style that leaves space for people to project their own viewpoint onto it. The ambiguity of this style is ought to be more illustrative than accurate, highlighting only the features we had reliable data to reconstruct. We have achieved this by creating a specific shader for this purpose that provides a coherent visual style even when applied to various buildings. 

We built our prototype around one specific location, the former Makkhetes Vendéglő on the corner of Rottenbiller and Dob street. This featureless one story building was hiding exciting historical layers under its rather meaningless façade.

“The street fronts of the house show traces of later remodelling that robbed them of their character―of all architectural details, ornaments, window or door frames that might suggest the original function and the approximate date when this house was erected. It is only the volume, forms and measures of the building that may point to its first form. The present stage is recorded on a 1978 architectural plan. Since then an only side entrance door in Dob St. has been substituted by a window.” — Endre Raffay

At the archives we found three distinct architectural styles of the building. The earliest from 1885, showed signs of classicism. The second phase from 1930 revealed an eclectic styled redesign, clearly showing us the former restaurant function of the building. The latest drawing from 1978 is free of any ornaments and features, the façade was stripped down to bare windows and doors.

We built our app using Unity with the help of the Vuforia framework. After testing dozens of marker solutions, we have decided to tweak the built in frame marker system of Vuforia. By using multiple markers at the same time for a single object we were able to smoothen the inaccuracies. In order to fine tune the tracking further, we limited the update rate of the tracked object's position to every 5th frame, interpolating between the incoming transform values. These solutions helped us to overcome the amplified inaccuracies of the large tracked objects. 

The user interface was designed alongside the Material Design guidelines by Google with AR first interaction on mind. Once the app has found the trackable, the user is able to take action. By horizontal swipe gestures one can easily switch between the three dimensional and architecture drawing views. By selecting a particular phase, the app changes the visualisation seamlessly to the given year. Three dimensional tooltips are constantly facing the user as he/she is moving around the building, showing various points of interests. The user is able to expand a tooltip into a detailed description accompanied by imagery from the archives.

The project was presented first time for the broader audience at the Reality Research Festival, organised by Kitchen Budapest, in Ocotber 2016. During the festival we arranged multiple guided tours for the visitors enhancing the experience provided by the application by on location presentations. The application was also exhibited using a 1:50 scale model indoors.