Augmented Reality TV

00. Table of Contents

01. The Call

01.01. The Recommendation

I had just finished a large project and the sun was already coming up. I was about to fall asleep when my phone rang. It was a friend telling me he had just recommended my company to a startup that was looking for a Creative Director for an augmented reality project. It seemed urgent, and he had given them my number. I thanked him and hung up.

01.02. The First Meeting

About an hour later, the startup called and asked if I could come to their office that day to discuss a project. I looked up the distance and told them I could be there in two hours. It was a 45-minute drive or an hour by commuter train, so I took the train so I could sleep on the way.

When I arrived, we discussed the project and the requirements. They needed someone to create concepts, design the experience, and direct the creative for an augmented reality integration with Galileo, ProSieben's flagship science show. They asked how I would approach it, and we talked through ideas on the spot. My thinking aligned with their vision, and that is how we started working together.

02. The Audition

02.01. The Unknowns

The project was a cascade of unknowns. eyecandylab had been founded just weeks before I met them, and we did not know how many episodes there would be, what topics they would cover, or what content we were supposed to create AR for.

02.02. The Room

The next day, I was sent alone to meet with ProSieben directly: high-level executives, producers, and the journalists who would be responsible for filming the episodes. The journalists still needed to go out with their crews to shoot the content, so nothing was ready yet. That meeting was a test to see if I could handle the pace, the volume of information, and whether I could generate ideas under pressure.

The room had a long conference table with executives and journalists sitting on both sides. A television was at one end of the table, and I sat at the other end facing it.

02.03. The Performance

After a quick introduction, they explained they had a rough edit of one episode with no sound and no voiceover, just the raw footage. While it played, the journalist responsible for that episode would read from the script. My job was to tell them, in real time, what would happen in augmented reality: what information would appear, how the animation would look, where it would appear on screen, and how it would interact with the broadcast.

(A recreation of that first meeting at ProSieben headquarters.)

So I did. For 30 minutes straight, the rough edit played silently on the television while the journalist read the script aloud and I described the AR experience as it would unfold, moment by moment. No one interrupted, no one commented, and no one spoke.

02.04. The Verdict

When I finished, the Executive Producer slammed their fist on the table and said: "That is what you call creative!"

They told me they would provide the information and details for the remaining episodes and that we would have regular meetings going forward. There would be 10 episodes. Later, it became 11 because they decided they wanted one more while we were already deep in production. I had to figure out how to add an entire episode to an already impossible timeline.

03. The Problem

03.01. Timeline and Scope

From that meeting, we had 5 months until broadcast. The budget was limited and the scope was massive. We needed to create all the AR assets, program and code everything, get it ready for live television synchronization, and design and build a companion app for Android and iOS, for both phones and tablets. The app also had to work on older, slower devices. There was no precedent because augmented reality for live broadcast television did not exist. Everything was new territory.

03.02. Finding Talent

We could not do this locally because the cost and speed requirements made it impossible. We needed to look internationally. For the next week, my company made calls and sent emails around the world, trying to find anyone with the skills to build what we needed. We had to figure out what skill set was even required for AR television, because no one had ever done it before.

Mobile game developers turned out to be the closest match. The specific type of mobile game studios we needed were concentrated in Asia, so we narrowed our search there, balancing budget and quality.

03.03. Selecting Malaysia

We chose Kuala Lumpur, Malaysia.

04. The Solution

04.01. Setting Up Operations

While I stayed in Munich for meetings and planning with ProSieben, I sent my company's Managing Director to Kuala Lumpur to set up an office and build a team. She had one month to find the right people, conduct interviews, and have everything ready for production. She did. When I arrived in Kuala Lumpur a month later, the office was operational and the team was assembled. We went to work immediately.

04.02. The Design System

I had to design a workflow where no one was ever waiting and every person had something to work on at all times. I started with the foundation by designing the components and creating a style guide for augmented reality television: text boxes, lower thirds, maps, data visualizations, and information overlays. Everything had to align with Galileo's existing brand guidelines while creating new patterns that could be reused for future AR broadcasts.

04.03. Content Creation

Once the team started building the component library, I moved to episode-by-episode concepts. I designed every AR experience and planned every animation.

I also had to create the actual content, the information and data that would appear in augmented reality, because ProSieben did not provide this. They gave me access to their research department, and I coordinated with their team to get accurate information for each episode.

04.04. The Schedule

Malaysia is 7 hours ahead of Germany. During the day, I was in the Kuala Lumpur office designing, creating storyboards, directing the team, answering questions, and explaining how things should look, feel, and animate. At night, I was on calls with ProSieben reviewing progress, getting feedback, and discussing changes. When the calls ended, it was already morning, and I went back to the office. This continued for 3 months. I do not remember sleeping. These were the months in my life where I slept the least.

05. Life in Kuala Lumpur

Malaysia sits on the equator, and especially in the middle of summer, the sun is directly on top of your head. The extreme heat combined with 90% humidity absolutely did not help with the sleep deprivation.

05.01. The Food

What kept me alive was the food. Malaysia has three major ethnicities: Malay, Chinese, and Indian. Kuala Lumpur is divided into districts for each, so when you are in the Chinese district, it is like being in China with authentic Chinese food, and in the Indian district, authentic Indian food. Entire meals cost between $1 and $5. Near the office was a high-end Japanese restaurant where the most expensive course cost about $12. I went there every day for lunch, and the staff thought I was a billionaire.

05.02. The Working Culture

I had to adjust to many small things, but what left the biggest impression was when our lead software engineer got sick with a fever.

She was in the office visibly struggling. When I told her she should go home and rest, she apologized and said she would keep working. I kept insisting, until I understood that she thought I was letting her go. I had to explain that this was the last thing on my mind, that her health was more important than the project, and that she should not worry because she would still get paid. I called her an Uber, paid for it myself so she did not have to take public transportation, and told her to come back only when she was not dying anymore. She reluctantly went home.

I contacted ProSieben and requested to move some episodes back, explaining that one of our lead software engineers was sick with a fever. As expected, they approved immediately.

I thought that was the end of it and I would see her a week later. The next day, there she was again. It was impossible to convince her not to work. The only thing I could do was make her as comfortable as possible and get her supplies from the pharmacy. I told her to at least take an Uber to and from the office instead of public transportation, and that I would pay for everything. She never did.

06. The Technical Challenge

06.01. Optimization

After 3 and a half months, we had everything built and I flew back to Germany for final testing and launch preparation.

Testing the app was painful because the AR synchronization required significant processing power.

We had to figure out how to prevent phones from overheating during extended use while also optimizing performance for older devices without degrading the experience.

06.02. TELEMARK Technology

The technology itself was unprecedented. eyecandylab's TELEMARK system used moving video as the AR trigger, not static markers.

The app detected the television screen through the phone's camera, fingerprinted the broadcast content, and synchronized AR overlays with 20-millisecond accuracy.

The AR content appeared to float around and extend beyond the physical television.

07. The Outcome

07.01. Launch

• The broadcast premiered on November 6, 2017. Galileo AR Week ran Monday through Friday, with 2-3 AR-enhanced segments per episode.
• 4 million viewers.
• The companion app reached number 1 on both the Apple App Store and Google Play Store in Germany.
• This was the world's first augmented reality television broadcast.

07.02. The Experience

07.03. Legacy

eyecandylab used this project to expand internationally and went on to work with VIACOM, NBCUniversal, WWE, Red Bull, LG, and Google. Their work earned an Emmy nomination in 2021, and in 2023, they were acquired by Accedo.