What challenges does the wider adoption of AI bring to computing infrastructure?

There are a couple of major challenges depending on where AI is deployed. When it comes to the cloud, we need to adapt our cloud infrastructure to the typical workloads of artificial intelligence. There are two big challenges when it comes to this. One is inference or how fast a pre-trained AI model provides responses to queries. This requires the ability to respond rapidly with significant computation done in a short amount of time. The second challenge is AI training. We have a lot of datasets that we use to train our machine-learning models and it’s a slow process. It can take weeks and is very demanding in terms of total computing power. It also requires storage, because we need to store the examples, the datasets, and the intermediate results of the computation that are needed to get to the full training of the neural network.

Secondly, there are the challenges of deploying AI directly where data is being collected, namely at the edge. Edge computing is used when a neural network cannot be used in the cloud for several reasons. For example, when privacy is a concern, and we don’t want critical data to travel across an unchecked network. Another example is latency, when we want a reaction time that is not compatible with sending the data to a remote server and waiting for an answer. A third reason can be connectivity. We may not have stable connectivity when we are on the move which makes it difficult to use the cloud. In all these cases, data needs to be processed locally, at the edge. When we are deploying AI at the edge, we have the challenge of feeding pre-trained machine learning models into computing resources that are much more limited than the ones we have in the cloud, for example, those in our mobile phones. Typically, at the edge, we only do inference, we don’t train the machine learning models there. But it is a big challenge to do inference at a reasonable speed and energy efficiency because we don’t want batteries to drain when we use AI.

How urgent is it to address these challenges?

It is very urgent given the added value that AI brings to both users and businesses. More and more companies are eager to deploy machine learning models for their services. For example, in audio and visual interfaces, face recognition or natural language translation in near real-time are services people are rapidly getting used to and increasingly expect to have. There is therefore a lot of pressure for businesses to have these services available right now. These challenges, therefore, need to be solved in the short rather than the long term.

You already mentioned edge computing which is one of the ways in which companies are trying to cut back on the time it takes them to process information. Is it enough though?

If you push computation to the edge, you eliminate the time required to communicate the data to a remote cloud, so you have lower latency. There are a lot of added benefits as well, which are increased resilience, privacy and scalability. There are a lot of problems with Cloud AI that Edge AI can help solve, but Edge AI comes with its own issues. We need to provide enough computation capabilities at the edge at a low enough power to be able to do something useful with it. And this comes down to hardware design.

Processing at the edge means processing in mobile phones, cameras, IoT devices. Many of these are not designed with powerful computation in mind. Does this limit what can be processed at the edge?

There is indeed a big gap there. Not only in the processing power, but also when it comes to energy consumption. Because battery and power delivery technologies at the edge are not evolving very rapidly. We won’t have batteries that are 100 times better than those we have today in a short time, but we do need to provide 100 times more computation power now.

In the last decades, software developments have been dominant in the industry. Will that be the case for this new AI boom as well or will hardware play a bigger role?

Hardware innovation will definitely play a bigger role. You can already see this in the large number of hardware developments that are taking place in the AI space. Many big companies like Google and Microsoft as well as startups are already developing hardware solutions for faster and more efficient AI. However, that doesn’t mean that software is not important. I think that the key answer is what we call co-design, which means being able to design software and hardware that work seamlessly together.

Axelera, a Dutch startup, is developing a novel hardware and software platform for AI that makes computations faster and 10 times more energy efficient than standard digital computing. You are a scientific advisor to them. How did you become involved with them?

When they were just starting out, Axelera’s CEO, Fabrizio Del Maffeo, approached me to ask my opinion on how their technology would respond to the expectations of the edge computing market. I thought the company was in a very good position, because it was covering a need in edge AI that I saw was growing. I therefore decided to help them out, both with advice, but also by connecting them to groups of researchers in the industry that were trying to do similar things. Many of them wound up joining Axelera. So I helped them on the recruiting side, as well as on the positioning side, but their technology was developed internally. They are not licensing technology that is coming from my research lab or anything.

What makes Axelera’s architecture unique? How does it serve the edge AI need that you spoke of?

When people want to use AI, they need to use accelerators which are machines specialized in running AI workloads. AI accelerators use architectures that are tightly matched with the type of computation we have to do today. AI is mostly neural networks and there are classical arithmetic circuits like adders and multipliers that can be used to perform the calculations needed for them. Most companies deploying AI nowadays use thousands of adders, multipliers and their combinations (called MACs) in parallel to perform the large number of operations needed for neural networks.

What Axelera does is slightly different. They are using very efficient techniques that come from memory design to perform the same calculations that can be performed by MACs. And thanks to these specialized memory-like circuits, also called in-memory computing (IMC) macros, they can achieve a much higher energy and error efficiency, which means many more multiplications and additions using much less power, and much less storage compared to the classical way to do it. Many research groups, including mine, have done research in this area, and there are many companies that are trying to come up with a good application for it. However, Axelera is already much further along. They have the possibility to be first in the market with a highly competitive product based on IMC macros.

Which markets do you think would benefit most from Axelera’s technology?

I think that probably the most interesting market for it and the one they will be targeting first is computer vision. That is anything related to high bandwidth sensors: video streams, camera streams, radar streams and so on. The most common high bandwidth sensors are high-resolution cameras that produce a lot of frames per second that need to be analyzed very quickly. They are usually used in markets where real-time visual analysis is essential, such as automotive, industrial production lines and home automation. It’s an area where you need a lot of computation in a short period of time and where Axelera’s technology could bring the greatest benefits.

There are many big chip companies, for example, Qualcomm and Nvidia. They have a lot of resources they can throw behind the development of new chips. Are they interested in the edge computing market? Can Axelera compete with them?

There is strong competition from this corner, but the advantage Axelera has is that they can really focus on the edge AI segment while these big companies address a much broader market. Qualcomm’s best market is mobile phones so they need to, first of all, provide a competitive solution for mobile phones. And then they can provide solutions for other markets like IoT. Nvidia has a very interesting product line for edge computing, but their focus is the cloud and they make most of their money from GPUs and big data centers. Axelera is smaller which makes it more agile and innovative and, by focusing on the edge AI market, it has the chance to bring its product to the market much faster than bigger players.

The semiconductor market is currently dominated by US and Chinese players. Is there a place for Europe at the table?

We hope so. This is actually a major priority for the European Union at the moment. You are probably familiar with the Chips Act and all the initiatives that are being pushed by European politicians to try to get Europe back into the production of chips. This means a lot of public money could be invested in these new companies that can bring back the capability of designing high-performance and energy-efficient chips to Europe. I think from the viewpoint of Axelera, this is an excellent time to be in this business, because a lot of public money will be available to help them grow.

At the same time, bringing chip production back to Europe is a very big challenge. Because it’s also a matter of the ecosystem. If you are in Silicon Valley, and you need 100 designers, if you have the money to pay them well, you can easily find them. In Europe, it’s a challenge because there are not many experienced people in this area, most of them have gone to China, Taiwan or the US. Companies in Europe will need to connect to universities and research labs to access their talent pools early. Axelera has already started doing this, opening positions in the Netherlands, Belgium, Switzerland and Italy, in cities near top universities. Overall, I would say, it’s a big challenge, but it’s a good time to try, because there is a lot of emphasis from politicians and from public funding to help companies in this area.