The interest in computer vision is growing, especially as it is a key ingredient in autonomous driving. How would you characterize the biggest changes in the field in recent years?
At the beginning, computer vision was quite limited in its applications. Companies specialized in narrow applications where a good control over the environment was possible. Based on recent progress, computer vision is now able to enter the consumer market, with applications in cars and on mobile phones, for example. This also went hand in hand with much larger companies getting active in the field. From a technical point of view, computer vision now is able to deal with much less controlled environments, i.e. with large variations `in the wild’. In particular, the advent of deep learning has changed computer vision enormously.

How does a computer actually see?
Before the advent of neural networks, humans would determine what the features were that needed to be extracted from the images. The approach was based on pattern recognition and engineers would sit down to design features that would allow for good classification. It is a bit frustrating, but deep learning replaces such expert-based feature design by the automatic learning of the features the vision system can best use to solve a problem. In many cases, the system performance goes up drastically.

How does that work in, say, the application of autonomous driving?
The basic goal is to recognize and localize different objects, be it persons or other cars, and tracking them over time. As a matter of fact, driving is a difficult task. First, we have a very unpredictable environment. Second, it’s not just a two-dimensional picture we’re looking at. Distances and speed matter a lot. Third, we need to combine input from different sensors, not just visual ones but also radar, possibly Lidar as well. For now, Lidar is still pricey compared to what it adds. In the end, fortunately, there are only a few decisions to make, like continuing to drive, accelerating, breaking, taking a turn, etc.

Is the advance of computing power the main ingredient that has made computer vision advance?
It is one ingredient, but not the only one. To interpret the content of an image, you need to have examples, which means data. And the data needs to be annotated, which is usually an expensive process. So if you think about who has these amounts of data and the money to afford massive computational resources, it’s the big tech companies. Academia sometimes finds it difficult to compete with them, even if we do a lot of dataset sharing. To get back to your question, it’s not just the raw amount of computing power that counts, but also picking a good architecture for the neural networks. Current work typically picks a network architecture that has been shown to be successful before. In the future, a lot of research needs to be done to answer the question of what the best architecture is to solve a specific problem, a field called neural architecture search. At the moment, our understanding of this question is still very limited.

This sounds a bit like it’s a lot of trial and error at the moment.
It is, and a coherent grand theory of deep learning is still lacking. While we don’t quite know how to determine the best architecture to solve a particular task, the community is already looking at connecting different networks that together solve a multitude of tasks (multi-task learning).

In such cases there are even more architectural design choices to be made. Almost nothing is based on established theoretical insights, and for a scientist, that’s a bit scary and, even more so, unsatisfactory. But given the power of these networks, there is no point in returning to old school methods. There are networks that can add captions to images, that can generate faces that one cannot distinguish from real ones, that can boost the quality of an amateur image to a professional optical level, etc.

When you started in the field of computer vision at ETH at the end of the 1990s, you were a pioneer and put Zurich on the map in this field. Since then, there have been a lot of successful startups coming out of your lab, and big tech companies such as Nvidia, Apple or Facebook have established subsidiaries in Zurich. What is the reason why this cluster has become successful?
To be fair, there was already a solid group working on computer vision. But its head – prof. Olaf Kübler – became President of ETH – so I could enter in his footsteps. As to the success of Zurich as a hub for computer vision, academic and industrial, I would attribute that to a combination of factors. Of course, the city is wonderful, but with ETH there was a strong academic player in place. This in turn generated multiple startups, which in more than one case became the kernel around which bigger companies formed their labs in Zürich, upon acquiring these. That put into motion a dynamic process. The more know-how is assembled in a place, the more likely it is that new people will join. Apart from low taxes, a stable government, and strong academic foundations, Switzerland also offers a great openness towards the world. When I came here I saw that many of the students would go to the US or Japan or other faraway places, not just to Germany or France. The same is true for the people who come to study here. Our students do not just come from the EU, but also from places farther away. And as to the presence of industry, It’s not just the companies you mentioned, but also Google, IBM and Microsoft. They have big research centers here, which turns Zurich into a small version of Silicon Valley, in Europe. But one thing worries me a lot. Europe isn’t home to such internet giants, but instead has an almost 20th-century industry in a way. This will come to haunt us. The technology we develop here often benefits large players in other regions. Of the 12 companies I co-founded, 7 made an exit so far, but all except one were to American or Asian companies. To conserve the knowledge we would need to have more European buyers of startups. Europe should innovate its industrial base also through disruptive additions, and when it does, be more apt to exploit the size of its market. Remember Nokia and Ericsson, which once led the mobile phone industry? It is a missed chance that they’ve been reduced to far smaller players. And I fear that the same fate could befall other industries such as banks or car manufacturers in the future.

“Of the 12 companies I co-founded, 7 made an exit so far”

How come there are so many startups coming out of your lab, is computer vision a topic that lends itself more readily to the creation of new companies than others?
I think now the time of computer vision has really come, even if the first spin-offs happened more than a decade ago. Even if the requirements of data and computing power make it more difficult to enter the field, there are still niches or gaps that can be filled. And if Ph.D. students see the examples of their colleagues who have created startups, this is, of course, a good source of inspiration for them. By now, if you study computer vision, Zurich is a very good place to find an interesting job at one of the big tech companies, or to create your own company. Still, there are big challenges for startups in Europe.

Such as…?
First of all, not enough money gets invested into startups, the US invests much more than Europe. This means that it is not a level playing field. I’ve seen instances where startups with  superior technology attracted just a few hundred thousands of investment in a Seed round in Europe, and a US competitor with an inferior technology gets 3 million of Seed funding and then rapidly much more in later rounds. You can imagine which companies prospered. The second element is the background and network of investors. In the US, investors might have a background in the industry or at least know people in the big tech companies they can introduce the startups to. In Europe, an investor might have business experience and a large network, but in a field that isn’t really valuable for the startup. Combine this with the many boundaries in the European market and a certain complacency that wealth is here to stay, and you see why it sometimes is an uphill battle to innovate in Europe.

What can be done in Europe to support innovation?
I don’t know whether this would be possible or legal. But printing money – quantitative easing – has become quite usual. The problem with much of that money is that it hardly reaches the real economy. One could imagine using part of these resources to create venture funds. If they are successful, money will also flow back to the taxpayers as ultimately one would want to sell those shares to private investors.

We’ve talked about the US and Europe, but we haven’t mentioned China yet. How far is the country in this technology?
Europe has a very strong tradition in computer vision, but the advent of deep learning was like a reset that allowed newcomers like China to leapfrog ahead. The Chinese government fully realized the importance of deep learning very early on. The country established a program with the goal of becoming the dominant force in this field. I wouldn’t be surprised if they achieve it. Today, there is no other country that delivers so many authors of papers in our main conferences and journals as China, and by far. There are a number of initiatives in Europe, but these often were established relatively late.