Thank you for joining me for this lecture on software challenges and AI accelerated co design.This presentation will expose the software bottlenecks that needs to be overcome to fully unleash the potential of AI accelerators. We'll cover the journey from initial concept to final deployment, highlight the key obstacles and solutions. The key is to make you familiar with topic less discussed within the AI buzz, which is the software that makes AI hardware, run. A little bit about myself. So my name is Anat Anath Heiper. I'm director of AI and software architecture at Intel. I have over twenty years of experience, in software development with the last ten years leading, AI accelerators and coauthored several patents in compile optimizations for this domain.

I directed, teams developing neural network compilers and run time systems. You can find more information about me in the LinkedIn, below, and feel free to ask questions. So this session, sorry, this session discusses the current hazards we have on AI domain, which started in 2012 with, breakthrough with, neural networks that's called AlexNet, a deep neural network that won ImageNet competition by a large margin, showing the deep learning, could dramatically outperform previous AI method in tasks like image recognition.

Alex's success shifted AI research towards deep learning and neural network processing. Now with deep learning architecture like VGGNet and ResNet kept pushing the boundaries, and AI systems soon plumed and powered everything as we see today from medical imaging to visual assistance. And to today, AI booms, of course, with ChargeePT and all the AI tools that surround us in our everyday life on generative AI. So all of this can be traced back traced back to AlexNet that started in 02/2012. But the main question is what why do we need it? The need for speed and efficiency. So the complexity of AI models is growing exponentially with larger and more sophisticated neuron networks requiring specialized hardware to run efficiently. Many real time applications like autonomous driving and video analysis need high performance and low latency, which general purpose processes often struggle to provide.

Traditional CPUs and GPUs have limitations in keeping up with computational demands of advanced AI. And this is setting the development of dedicated AI accelerators. These accelerators are designed to handle the unique focus and data of modern AI, delivering the speed and efficiency required for demanding real world applications. And as we can see in the diagram on the right side, the complexity of models is keep increasing in unprecedented pace, something that as me as a software developer over the last twenty plus years, I have never seen those challenges going to such demand in any other domain that I've been involved with.

Okay. So the rise of AI accelerators and the main reason that I wanted to have this discussion is to have some overview and understanding about the different hardware that architectures that are part of the AI domain. So as AI is rapidly evolving field, new methods and applications constantly emerging to handle the increasing computational demands, and specialized hardware called AI accelerators are authorized. These accelerators are designed to provide significant performance and efficiency gains over general purpose CPUs that we are all familiar with from our everyday life. So what are those AI accelerators that we keep hearing on the news from startups to those big, high-tech web start companies that are building? So there are mainly four types of accelerators around the the GPUs, FPGAs, and ASICs. GPUs are parallel processors that are well suited for graphics and AI workloads.

FPGAs are configurable hardware that can be customized for specific accelerator needs, and ASICs, are application specific chips that are built for maximum efficiency. And as we can see, from the graph here, there are trade off for each hardware that we can decide to use. CPUs are the most flexible ones. We can practically try to write whatever problems that we want to do. The downside for that that the flexibility comes with a price, of efficiency. So because you need to support all various types of operators, it won't be the most optimized one. And on the others, side of the spectrum, we have ASICs, application specific, accelerator, where you can design your hardware to write specific hardware, but it cannot run anything else. This is especially good if you have very compute demanding workloads such as AI that you can build specific hardware for that, but you cannot do any general purpose computing on that.

So the trade offs for for choosing those this architecture. Okay. So up until now, I've only talked about hardware why am I as a software architect, talking talking about software here? So the software is a key neighbor that bridges the AI models to hardware. It's usually not discussed when usually, when we thought think about AI, we think about the AI researcher or maybe about the hardware like NVIDIA, but we don't think about what's between those two, which is software, which is connecting everything, together. So software is a key network that bridges AI model to hardware. It translates the the algorithms into hardware instruction that the processor can understand and execute. Software optimizes the hardware, and it's unlocking the full potential and enabling powerful AI capabilities.

The software layers on the vertical AI software stack are critical for harnessing the power of AI and driving the digital transformation that we all experience. Without the software's bridge, m models cannot be effectively deployed on physical hardware. Another critical aspect, which becomes more and more important, is the ease of use and adoption. If we if it's not easy to program the device, no one would use it even if it would be the most efficient and maybe cheaper to use because we cannot we cannot program it and cannot make it work. So the ease of use is becoming more and more important in our decision on how to design software for those accelerators. It's a top priority. Okay. So how do we do that? I want to expose you in a nutshell what does it mean to do the codesign between hardware and software together because this is a imperative part of of designing such an accelerator.

So let's briefly discuss what it means for the hardware perspective. The three main components, components for that, concept, development, and deployment. The concept phase covers the initial phase of planning and design work, system architecture and functional design. The development phase focuses on building the hardware component, logic, circuit, and physical design. And last but not least, the deployment phase addresses the manufacturing and packaging of hardware. This is a very complicated, process of building a hardware from inception to deployment, and it's also very, very expensive. If there is any error in the concept design, concept phase, it's very difficult to fix it if anything, and it may require to do the whole process all over again, which results also in very expensive and and delays of the project. So what is the code design that, that I want to do between hardware and software? It's an approach where hardware software is developed in tandem rather in sequential manner.

In traditional approach, hardware is developed first, and the software is built on top of that to work with the, hardware, which is what I try to, show on the left, side, that traditional approach. Why it's more simpler to comprehend because the hardware process is difficult as it is, it comes with difficulties because it limited the the possibility for performance and optimization of hardware software co design and also delays the cycles, and time to meet, customers. And today, we just want to be more, effective with the come, come to market, which is a problem. We want to to speed up the development cycle as soon as possible. This is why the codesign approach is, what we aim to do and to do hardware and software in tandem, which means that we we do the design, in in parallel. It enables the seamless and optimized integration between the hardware and software components of the system.

And for that, there needs to be a collaboration, which means that me as a software architect and leader of software groups, I need to understand much more than before what are hardware requirements and dependencies, how to read the hardware specs and understand their limitations.

And on the other side, the hardware, team must come and include me in their design reviews and to be part of the decision making of what are the capabilities that, hardware needs to support such that we can do optimized and better solution, as part of the end product. Okay. So the power of synergy, there are key three key attributes of the fact of doing the hardware software integration together, which is performance, efficiency, and flexibility. Optimal performance is achieved through the tailored hardware software integration, allowing the system to operate at peak levels. This is very, very important, and much of the work that I had is leading those large scale software hardware accelerators is about doing those optimal performance. Some of the patterns that I've coauthored are around this topic of how, the compilers that is AI specific is able to optimize the hardware underneath it. The optim the efficiency is, again, in terms of better energy usage, which is a a hard limitation of the data centers today that are deploying the LMM models.

Memory management, as we know or heard, the models the LLM models are only getting bigger and bigger, so memory becomes a real scarcity in deployment. And the processing speed, we none of us would want to to wait for the Chargebee T for three minutes until it gets a response, so latency is a very critical issue. The solution should be also highly flexible and adaptable because the models are keep changing on almost daily basis. So when we design our hardware and software, we need to be able to somehow foresee what changes can be in the coming models and to be able to adapt to that and respond as soon as possible in order to be the best product for our future customers. Navigating the co design journey is a structured process, although it seems so complex, and it's it has unique challenges at each stage, that we discussed before, the the concept, development, and deployment. So in the concept set phase, the hardware and software architects collaborate as I just shared it, from my own experience, when we work on pieces in parallel and and to set the foundation for integration.

On the on the development phase, as much as possible, we try to build the pieces in parallel. Software tools, are built on top of hardware simulation along with hardware specification. On the deployment phase, it's about the optimization of the performance across diverse AI workloads and environments and putting it all to work. I want to deep dive a bit about the concept phase, because this is the most imperative phase of this the the workload. It's where where we lay the groundwork for hardware design and analysis to understand what what we need in the comp from computation perspective. What are the functionalities that are critical to be in hardware and what can be offload to software, whether because it's not as compute intensive or maybe we want to have allow allow more flexibility for future generation.

We this phase is very important, and we weigh the pros and cons for every possibility because there's no clear answer to that. There are many ways to implement a solution, and we need to to do compress, compromises from performance perspective, power, ease of use, and flexibility. All of those needs to be somewhat consolidated as part of the hardware software co design. After this is done through prototyping simulations, feedback from market, customers, if we have design customers, and also, through early prototyping wherever is possible. As I think we are running out of time, I would I'll jump on some of the slides here. Okay. So handling diversity is something that is also very important and difficult through the expression of of the architectural phase.

The when we build the hardware product, it's not enough to be good in one specific thing because the the deployment and development is very, very expensive. We try to build the product to be, relevant to as much models as possible, and here, I'm only highlighting some of of the AI models that we are using on everyday use. There's a lot of differences in the computation and and memory needs between the larger language models, computer vision models, recommendation system, and multimodal AI. And trying to an to address all of them in most efficient manner in the hardware and software side is an impossible task. And we need to consolidate and understand how to map it to be best for most use cases or or the most apparent, that will gain the most profit, which is very, very difficult. So, I may have highlight hinted for that before, but during my journey as software architect and leader on those projects, I suddenly feel like I'm trying I need to, like, live in the future and guess in the future.

There are many, many challenges. From the hardware perspective, when we build those accelerators, from the inception, to the times that we actually meet the market, it's it's a process that takes on best best case three years and, usually more than that. And in the crazy AI domain, this is very it's it's like ages. Everything can change in the industry until we meet the customers. So we need to somehow guesstimate, what would what would the market need. It's very, very complicated. This is on the hardware side. On the server side, the challenges are also similar in that respect. If you can think about the amount of, frameworks and, programming language and workloads that keep evolving And the softwares that and the planning that you do for the software support that you need can change tremendously within even six months, and you need to change and navigate the your software teams to develop some, and support other frameworks.

The the pace of change in the AI domain and AI infrastructure is just, unbelievable. I feel that we are out of time. So, unfortunately, I will skip the rest, of the slides, but I really hope that you were able to learn a bit about the challenges on the AI hardware. And I would be happy to answer questions now. So I I I will make if possible, I will make the slides available. So software bottlenecks, one of the main thing is, I think, is how to how to support the ecosystem and the software, because there's so much, framework that you need to support. And, of course, the the softwares that, NVIDIA has built and to be as a a new competitor is always a big challenge. So I think this is, like, the main challenge to be able to have this amount of support when introducing a new hardware. Any more questions? As we see it.

So, there is a question about environmental impact. I think that, from that respect, the the power consumption is a very huge deal, and, many, many hardware accelerators are trying to be as as efficient as possible in that respect. I think I I've answered, Ozga question in that respect. Production yes. Production so stability, for production is also something that we simulate and also various, batch sizes. This is also something that we we simulate, and it's also very important that there's a lot of implication about latency versus throughput, trend, trade offs that that are taken into consideration. It's, also a matter of how you split and break the models across accelerators. So definitely, this is something we take into consideration. So, regarding GenSu, so the convergence should should be as as soon as possible.

I mean, for the design and, both the architecture and the design phases because we have very strong simulation, tools, from the beginning. Like, for me as a software architect, from from, almost from the beginning because software is part of the product requirements. So, a software architecture is very vertical considerations, not only the high levels Python, infrastructure, of research. It's also, like, the and and the APIs. Yes. Software and hardware teams collab now have to collaborate together. This is what is done in in modern, development. How do I mention expectation? It's very difficult. There's no clear cut answer for that. You start from something and you just, when you see, for example, the trade offs and trends changes from, let's say, TensorFlow to Python or SG Lang or what whatever, framework thing, you just need to change on the go.

Those are, more easier changes because it's more on, like, the focusing of of software resources. This is no new workload that is, monumentally different in the, optimization that that's more difficult, to address. Technical standards, no, but it's a great idea. I don't think that we have. There are some benchmark for performance that, for example, ResNet has been a, a benchmark for a lot of time, for, CNNs and stuff like that, but no no standards yet. But thank you so much for the question. It's been, privileged to be here. And thank you, everyone.