Author: Miguel Andres-Clavera, Creative Technology and Innovation Director, BBH Asia Pacific
For some time now, artists, programmers and marketers have been seeking innovative ways to collaborate, to blur the line between art and technology, thereby creating complex systems that merge the real world with the digital world. As these new experiences transcend digital mediums and permeate our physical experience, we begin to witness the emergence of public performance as a spectacle.
We had a great opportunity to explore some of these ideas when we were tasked to launch Nike’s new Hypervenom football boot collection in Southeast Asia. Our imagination ran wild at the thought of of creating an experience that combined real football with virtual challenges and got us really excited. In a way we wanted to allow fans to experience a whole new way of playing football, to make fans feel as if they were personally immersed in an epic video game.
The challenge was to create an authentic experience that preserved the physical skills and attributes required when playing football in real life, then take the player through an intense emotional journey culminating in a ‘deadly’ twist, giving him or her a sense of empowerment from the game.
The ‘house of deadly’ was born, a mixed-reality gameplay experience in a controlled environment where players were monitored and challenged to perform actual football skills but in a virtual context using an adaptive interface.
The next step was to persuade the client and prepare him for several months of lasers, interactive projection mapping, motion detectors, sensors and even virtual beasts. The pitch process was intense and exciting. Nike is a dream client when it comes to embracing big endeavours, and this project was certainly one.
In order to make these kind of experiences possible, it is critical to work closely with all the teams involved in the production process. Given the time and budget constraints in this particular case, I was well aware of the fact that our technology partners had to embrace the project beyond a statement of work, as the uncertainties were manifold. In other words, a personal commitment is key where everyone involved is equally passionate about making it happen, come rain or come shine. We truly believed the world deserved to see this project!
We were lucky enough to know the remarkable engineers at Arstic, a fantastic software development company based in Zaragoza (Spain), specialized in interactive installations and custom software. More than software developers, they’re problem solvers. Their talent and commitment was key. If had to choose a word to define our partnership, that word would be empathy.
We created the first computer vision system for real-time tracking of a football and created the first depth camera tracking system to detect point of impact for a football. Using several Arduino hacks we developed software that control several peripherals, critical for the experience: ball feeders that launched footballs, light heads, infrared cameras and different sensors. In addition, we had to create a 10m x 10m x 5m large virtual football stadium, almost half a million litres large. All this in four months.
Eventually, we came up with a game interface that used the physical football as the object or controller that linked the physical world with the digital realm – the football had to be used to interact with the world of projections around the player. Combining the elements of motion detection and interactive projection mapping, a surround sound system and dramatic 3D effects, we managed to create a game that would bring a football fan’s imagination to life. We worked with the motion graphics studio FutureDeluxe based in the UK to fabricate the 3d landscapes and targes that appeared during gameplay.
It is always a challenge to design an adaptive space where players can freely perform tasks, as there will always be unexpected situations. Thus, calibrating the system and adjusting the behavior of the different components to react to the player’s actions through software programming became crucial. We designed different behaviors for the virtual football defenders based on the real-life actions and reactions of footballers out on the field. We had fun simulating real defenders like Pique, Rio Ferdinand, Ramos, Puyol… to play with the imagination of the football fans.
Players who entered the stadium had to play through five levels, controlling a football thrown at them by two ball feeders on each level, dribbling a real soccer ball and hitting the on-screen targets of virtual beasts. Laser beams chased the ball and served as ‘defenders’ that players had to avoid whilst dribbling.
Simple game mechanics made the experience very exciting for the players. An agility bar determined the time players stayed on the game. Every time a virtual defender touched the football, the agility bar would drop.
Difficulty increased on each level, with the players having to dodge up to three defenders for longer. Once players managed to dribble the defenders, they had to hit moving targets that were projected onto the three screens surrounding the gameplay area. Depending on the accuracy of the strike players were given different scores. The game ended when the player either run out of agility points, time expired, or level five is reached and completed successfully.
At the end of the experience, players received an accuracy and agility score in the form of a digital and physical personalized memento using a photo taken prior to entering the gameplay area.
Interestingly enough, designing a system that accurately tracks the path of the ball and direct the laser beams would prevent any player from winning the game. Thus it was always important to keep in mind the delicate balance between reality and virtual reality by engineering in a human factor to the virtual defenders.
To design an experience that simulated the skills necessary to play football like in the real world we had to overcome several technology challenges: control of ball-feeders and DSLR cameras over an intranet, develop a realistic behaviour of virtual defenders using strategic algorithm and an tracking system to control mobile light heads, warping and synchronising of three HD projections to trigger different animations based on the player’s performance and the synchronization of tracking, visual and gameplay processes.
The event’s reception was overwhelming. Having one of Asia’s highest concentrations of football fans, Bangkok, Thailand was the choice to erect ‘The House of Deadly’ over two nights, employing more than 50 people in the construction. We were thrilled to see the faces of thousands of people who walked by to see the commotion created by the game, only to be blown away by the realistic simulation and stunning graphics. Over 400 people tried it in less than three days, and professional players from FC Barcelona, such as top striker Neymar, absolutely loved the game due to its realistic features and interactive elements.
Authenticity is key for Nike. Not only the experience should feel real for football fans but it should also reflect the key attributes of the product, of deception, agility and accuracy and fully encapsulated Nike’s Hypervenom concept of fierce beasts, represented in the different landscapes and targets designed for the game.
Knowing that our game allowed young football fans to feel like their heroes, performing their skills in front of a crowd of hundreds, defeating virtual fiends and sharing their achievements on Facebook made us feel a great sense of accomplishment.
Given the great feedback given by the participants and professionals, the future looks bright for ‘The House of Deadly’. We are planning to package and bring this experience to several other markets, subsequently adding more exciting and groundbreaking features.
TECHNICAL DESCRIPTION OF THE SYSTEM:
The main hardware components of the system are:
– Truss structure 10x10x5m
– Three 10x5m tarps as walls that close the gameplay area
– 10x10m floor with artificial grass and slopes to retrieve the balls
– Two balloon launchers controlled by an Arduino + Ethernet Shield + relay
– Two DLSR cameras controlled by an Arduino + Ethernet Shield + optocouplers
– A central DMX focus indicates the player’s starting position
– Two DMX spotlights that illuminate the pitcher from which you will receive the next ball
– Three DMX moving heads that function as virtual defenders
– Two DMX fog machines that allow visible light beams
– Three video projectors lumen 20,000
– A quadraphonic sound system with subwoofer reinforcement
– Two cameras and infrared illuminators zenith
– Three Kinect devices
– Balls modified with reflective material for a robust tracking
– Area record with a DSLR camera with an SD-EyeFi and chroma background
– Computer networking, routers, hubs, signal extenders, etc..
On the other hand they have developed the following software applications:
– A microsite that manages the players registration system, the queue to access the game and printing of memento
– A main application manages gameplay, plays sound effects and synchronize the entire system (projections, and spotlights DMX moving heads, ball launchers, DSLR cameras, etc.)
– An application handling and processing of visual content with three instances synchronized from the host application for each of the three walls of the game
– A tracking application using two infrared to determine the position of the ball in the game
– An application for tracking devices using three Kinect that identifies the point of impact when the player shoots a target
– A scheme to control Arduino through OSC messages
– A scheme to control Arduino through OSC messages DSLR cameras that capture snapshots of the game
The microsite is based on PHP + mySQL and manages communication with a Facebook application that allows to automatically share the outcome of the game on the wall of the participants.
The main application of the two visual tracking applications are based on Openframeworks and the following addons: ofxCv, ofxDmx, ofxFTGL, ofxHttpUtils and ofxKinect.