Telling the stories of 100 Detroiters through image & sound
The ‘Human Atlas’ is a series of projects that map, collate & interview groups of inspirational people all over the world. Developed by acclaimed British artist, Marcus Lyon, the series tells the stories of places and the people who inhabit them, through photography and immersive sound.
The first work in the series, Somos Brasil, saw Marcus and his team meticulously map the ancestral DNA of more than 100 Brazilians, photographing them and recording their stories. Calvium came on board to add an interactive dimension to his portrait exhibition and beautiful accompanying book.
Calvium worked closely with Marcus and his team to develop an immersive digital experience that would enrich a viewer’s experience of the portraits; using image recognition technology that triggers a piece of audio when an image is scanned, creating a layer of interactivity. Somos Brasil was launched in 2017.
Now, we are continuing this long-standing partnership, teaming up again with Marcus to develop the supporting immersive app for his new book, ‘i.Detroit’ – the latest project in the Human Atlas journey. Marcus is telling the stories of Detroiters through image and sound, giving voices to those who are underrepresented or misrepresented in what was once known as America’s Motor City. i.Detroit was commissioned by the Kresge Foundation in partnership with the Charles H. Wright Museum in Detroit.
The Immersive Challenge
Over two years, Marcus and his team analysed 100 Detroiters of all ages, races, and backgrounds who are pushing forward social change. He then traced their generational migration to the city and interviewed each one to record their personal stories. As with Somos Brasil, Marcus took the time to analyse and understand each participant – telling their stories through photography and sound.
To provide an immersive and multisensory engagement with this community of Detroit residents, Calvium were invited to develop the i.Detroit app. The interactive app enables the user to hear the voices and stories of each individual. Critically, the app needed to play a resident’s specific audio when the smartphone was placed in front of the image, and do so gracefully, in a way that wouldn’t obstruct the user’s enjoyment of the work.
The i.Detroit app needed to be a seamless element of the artistic work, not an adjunct; the essence, values and emotional resonance of the project needed to be present in the technology.
How Calvium met the challenge
As with our first project for Marcus, Somos Brasil, Calvium’s main challenge was to create an audio experience that seamlessly integrated with the visual elements of the book. Within this, there were further challenges the team had to overcome.
Critically, the technology we employed for i.Detroit was different to the one that we had used for Somos Brasil. There were two reasons:
- We were now using the React Native framework to develop the app, as opposed to using a pure Native App for iOS. A key benefit of this approach is that there is just one code base that can be used for both Android & iOS apps – instead of developing an Android app and a separate iOS app.
- Somos Brasil used a third party technology for its image recognition capability. This technology was bought by Google, so no longer freely available to us. After exhaustively testing alternatives, we chose Firebase for iDetroit’s image recognition capability.
Firebase is Google’s software development platform and is used for Machine Learning. It’s not the only software available, but it’s perfectly suited to this project.
Machine Learning for i.Detroit
As mentioned, when the user places their smartphone in front of any single portrait, the app needs to play a resident’s specific voice. To do so the app has to recognise the image immediately and then play the associated audio track. We used Machine Learning to train the app to recognise each of the 100 portraits, and this section provides an overview of our approach.
When the team first started training the Machine Learning models, we used JPEGs of the 100 images in the book, provided by the client. With these, we were doing some automatic manipulation of the images e.g. changing the size and orientation, and we were training the model with that set-up. However, we quickly learnt that this wasn’t the best solution because when people use the app, they are taking their phone to the portrait and this experience is a very different scenario than the context we used to train the machine.
Our next way of tackling this challenge was to print the images and spend time photographing the physical copies. Although this worked well, and meant we were able to train the machine learning models to recognise the images a lot better, we still wanted to ensure the quality of the image recognition was as seamless as possible. So, having obtained a copy of the book as soon as it was available, we took photos from the publication itself, using two different mobile phone cameras. Using these photos and concentrating on the face and as much of the body as possible, we trained the model once more – fully optimising its facial recognition capability. This painstaking attention to detail ensured a seamless, immersive experience for audiences.
“We made a huge effort to have the image recognition working really well. We had to iterate a few times to get the final result – it works well and it works fast, in fact, Marcus initially said that it was working too fast! Overall, I’m very happy.” – Fernando Sanchez, Developer, Calvium
React Native is one of Calvium’s preferred frameworks for development and we have used it extensively to build a wealth of digital mobile services. The i.Detroit app is built in React Native as it allows us to create one code base for smartphones using both Android & iOS. This enabled us to develop the app faster, without losing any quality in the final product or end user experience.
Building the i.Detroit app using the React Native framework with Firebase software development platform posed a potential layer of complexity, but as explained above, ultimately it worked well for the project.
“The i.Detroit app not only opens our audiences to another level of experience of our Human Atlas Project on the City of Detroit, but does so so simply and powerfully that the tech is invisible.” – Marcus Lyon
This ‘invisibility of the technology’ is a key feature of the success of i.Detroit. The audience receives a multi-sensory and immersive experience due to the thoughtful and holistic nature of the project. The i.Detroit team has approached each of the perceptual elements of the project as a whole and therefore the audience benefits from all of its elements being united, as opposed to a disparate collection of media. Similarly, the visual design of the app’s interface echoes the visual language of the book, again ensuring a unity.
For something that feels to the reader as if it is very simple, natural and intimate, there has been a great deal of skill and expertise employed by Calvium to create this interactive experience.
“Working with Matt Hill and Calvium on the i.Detroit app to create an interactive sound platform for my latest book and art exhibition was a delight… even with all the challenges of 2020 the team at Calvium drove the project to its conclusion with elegance and professionalism.” – Marcus Lyon
Calvium has worked with Marcus and his team for a number of years. In this time, we’ve forged a strong, collaborative partnership. As with all our client projects, we were able to recognise and apply quick problem solving to deliver the best solutions for i.Detroit. The project saw us further develop and demonstrate our Machine Learning and image recognition capabilities.
i.Detroit was launched in October 2020 at a live, virtual event. Marcus is now continuing on his Human Atlas journey, and Calvium looks forward to collaborating with him every step of the way!
We would be delighted to explore with you how we could deliver digital products and services for your organisation. Get in touch: 0117 226 2000 / email@example.com