You are the chairman of the JPEG standardization committee. JPEG is an image format that was invented quite some time ago. What kind of work is left to do?
Actually, the acronym stands for “Joint Photographic Experts Group”. The JPEG format you referred to is the result of work this committee and was done almost decades ago. At that time many electronic imaging applications took off and the need to exchange digital pictures arose. JPEG still is the most widely used format for still images. But the committee has continued to work on other relevant standards which are less widely known. The JPEG 2000 format, for instance, is a format produced by the JPEG committee to store and to transmit digital cinema movies, medical as well as aerial images.

The work at this committee gives you a good overview of where technology is heading, I imagine. But you’re also involved in several startups. Why?
I believe good academics don’t just sit and reflect on problems inside their labs, but they should also be driven by real-world problems. The best way to know about real-world problems is to work with companies. This includes both established companies and smaller startups.  Startup companies are usually more agile and more open to transformative technologies for reasons easy to understand. I particularly like to work with spin-off companies that rely on technologies coming from research labs as it allows me to better understand new challenges and focus on them.

As a professor, you head the multimedia signal processing group of EPFL. What is the most interesting project you’re working on at the moment?
It’s not easy to choose which is the most interesting. But instead, I can tell you which is the most forward-looking. We have been working on a way to analyze the brain activities of human subjects when they consume media contents. The objective is to implicitly find out how much they like or dislike a song or a film, without explicitly asking them, which is the only way to do it today. This kind of information would be the holy grail of recommendation systems. It could teach an AI system to deliver more suitable content to a given person or audience. Imagine the impact on personalized advertisements for example. As IoT and wearable technologies become more popular, their sensors can be used for this purpose. In longer-term, it could be applied to modalities beyond audiovisual. For example to find out which coffee recipe or which combination of spices in a meal is preferred by an individual or a specific category of individuals.

What are other projects you’re working on?
There is a particular project that I believe will have a huge impact on society. It’s about future modalities of content. This is where light field technology comes into play.

Can you explain in simple terms what a light field is?
Think about how pictures and movies are captured today: by small boxes with a sensor in it that we call cameras. Our world is three-dimensional. But pictures today are two-dimensional. Cameras remove a dimension of the real world we take a picture or record a video. Now imagine that we could have a box that would capture all dimensions of the real world. The resulting picture is called a light field.

How would that look?
This allows you to create both real and imaginary worlds in which users can move around freely. There isn’t just one perspective imposed by the camera like today, but infinite ways to look at things and interact with them. If you think of the holodeck in Star Trek or the projection of Princess Leia in Star Wars, that’s what a light field technology offers.

“The idea of light field is actually as old as the photography itself”

This sounds like, well, Science Fiction.
The idea of light field is actually as old as the photography itself. Gabriel Lippmann, who received the Nobel prize in physics in 1908 for his invention of color photography, proposed to capture reality as it is in a way similar to light field technology. Lippman said that color photography was just the first baby step towards that goal. At that time, of course, technology wasn’t ready for it yet.

Is it now?
Talking about light field technology we have to distinguish three different challenges. The first one is how to build a box that is able to capture light field content. The second is how to display light field content. And the third challenge is how to process light field content for efficient storage and delivery. The last is motivated by the fact that light field contents require huge amounts of data, in the order of 100 to 1000 times as much in volume as a regular picture or movie today. This is way too much to handle, as it would be both too expensive and impractical. The challenge lies in finding a compression technique that reduces the volume of data without a significant impact on the quality of the content it represents.

This sounds like a task for a committee that has experience with digital data formats.
Exactly. JPEG is at the final stage of compression specifications to define an efficient and universal format for such contents. The work that is now completed at 95% started 5 years ago by contacting and by talking to all stakeholders from industry, academia, and creative communities. This resulted in a new activity within the JPEG committee to create a new standard format for immersive contents including light field. The format is called JPEG Pleno. We expect it to be finalized in the first half of 2020. It will encompass not just light field, but also holographic and the point cloud representations that together with light field define plenoptic contents.

Where does the name Pleno come from?
It comes from the plenoptic function. This is a mathematical 7-dimensional function that describes how light flows through every point in space in every direction. The goal of setting such a standard is that it enables different actors in an emerging ecosystem to communicate with each other. No single company will be dominant with a proprietary solution and the door will be open for all actors in the ecosystem. This is the beauty of an open standard like JPEG Pleno.

Now that the JPEG committee has taken care of one of the three challenges you mentioned, let’s talk about the other two. With what kind of box will light field content be captured?
You need to capture the image from many different perspectives to create a light field content. But you cannot just move one camera around the scene because as time passes its content can change. So you need to multiply the cameras. In practice, the box is made of an array of hundreds of cameras arranged in a matrix and depending on the exact way things are implemented, one talks about either a lenslet based or camera array-based light field capture device. Of course, nothing prevents to also create a completely synthesized light field content using computer graphics techniques like those used in special effects in movies. For the latter, no capture is needed.

These cameras already exist?
Yes, and it shouldn’t come as a surprise that Google recently bought a light field camera startup called Lytro. RED, Facebook, Microsoft, Sony, Samsung and basically all the big names in multimedia are working in light field. The open question so far is how to create compelling content that truly shows the added value of light field representation when compared to conventional content we use today.

“All the big names in multimedia are working in light field”

Where does it add value in your opinion?
It is obvious that watching a news anchor in 3D isn’t going to be the most exciting use case for light field. The question you’re asking goes way beyond technology itself. It touches creativity and storytelling. Light field will enable completely new ways of storytelling because it is truly immersive. But how exactly light field technologies are going to be used as narrative tools still needs to be discovered. In a way, light field is like an exotic fruit that not many have tasted yet. Before you can light field devices you have to have compelling contents tht clearly show their added values.

Let’s talk about display technology. There seems to be quite some activity in this field.
This is true. Especially in the US, there are a lot of companies active in light field displays, often driven by the military and channeled through DARPA, a US government agency. The interest there is to get a clearer picture of what is going on in various situations by advanced visualization techniques. Imagine a hologram of a battlefield which generals can look at from any perspective to get a better sense of what is going on in real-time. The content that is displayed doesn’t necessarily need to be captured by cameras, it can also be created synthetically, like in a game.

How exactly will content be displayed?
There are several different approaches this is achieved. The first is a sort of table which displays things on top of a surface. You can stand around it and look at objects or a battlefield or anything as a matter of fact. This method has gathered a lot of interest in the US and in Japan which has a long tradition in display devices like TVs. A second approach is to have a big monitor on a desk or against a wall. It will be similar to when looking outside through a window. And a third approach is to wear head-mounted displays. This is considered by some, to be the next evolution of mobile phones. Instead of looking at a small screen on your smartphone, information will be displayed on your smart glasses. This will allow doing things that are unimaginable today.

This is the approach the Swiss startup CREAL3D has taken. Does that make sense?
Yes, there is a very good reason they have selected this approach. Today’s virtual reality goggles have a big problem, they create eye strains and fatigue and can even make people sick when they wear them. This is because they use a simple projection using conventional contents and this gives the brain contradictory signals when the head moves and when the eyes focus on different objects. Light field is the technology that will allow creating the immersive experience that virtual reality has promised in a way that it can become indistinguishable from the real world.

What about augmented reality?
The mixture of reality with synthetical objects creates additional challenges. If the renderings of virtual objects aren’t quite realistic, the experience just isn’t good enough. Here again, light field is the way to enable a much richer and more realistic experience, for augmented reality.

I’ve seen CREAL3D’s prototype and was quite amazed by the quality of the images. But it is quite a large box and it is difficult to imagine how it will be possible to shrink it to the size of a pair of glasses.
The goal of a prototype is to show the feasibility of the technology and in this case, to show what a light field content looks like. To minimize the size is a challenge and the same goes for other things like efficient power management. This reminds me of when I visited the Ericsson Research Lab in Stockholm many years ago. They showed us the first prototype of a GSM mobile phone. The thing was so huge, that it was filling an entire room. A researcher told us that one day, we’ll be able to put a GSM phone like what they were showing, in our pocket. Everybody burst into laughter. But as we all know, he was right, it just took a few years time.