On-device ML: Artificial Intelligence in your pocket
Priyanka Kulkarni21:592023
A few hundred megabytes of model can be the death knell of a mobile app
A typical deep learning model weighs a few hundred megabytes, and surveys show running out of storage and battery drain are the most common reasons people uninstall apps. Teams moving AI from the cloud to phones must win on memory, speed and security at once, often on mid to low range devices. Adobe senior machine learning engineer Priyanka Kulkarni maps that terrain: how do you shrink, accelerate and protect a model that lives in a customer's pocket?
Learn the compression, acceleration and watermarking strategies that make on-device ML practical
Priyanka Kulkarni, Senior Machine Learning Engineer at Adobe, explains what it takes to move deep learning models from cloud servers onto phones and other edge devices. She works through three challenges (model size, model efficiency and security) and covers compression, neural architecture search, hardware accelerators, model watermarking and federated learning as ways to meet them.
“A large app size is the death knell on mobile.”
“So keeping the mobile model size low and the latency low is really non-negotiable.”
What you will learn
Weigh the cost, latency, connectivity and privacy benefits of running models on device instead of in the cloud.
Choose mobile-friendly architectures such as EfficientNet, MobileNet or MobileViT and shrink them with pruning, knowledge distillation and quantization.
Use neural architecture search and Once-for-All style scalable networks to target several hardware environments without starting over each time.
Diagnose jank from GPU contention and adapt model layers to hardware accelerators such as XNNPACK or the Apple Neural Engine.
Protect deployed model IP with encryption, obfuscation, request limits and neural network watermarking.
Key takeaways from the speaker
Learn how to accelerate the journey from server to mobile for deploying deep learning neural networks
Understanding the risks and challenges that come with deploying neural networks on-device
Security implications of deploying neural networks on-device and which solutions and strategies to adopt to secure your intellectual property
Understand the fabric of mobile deployment platforms for both Android vs iOS
Push the boundaries of machine learning on mobile
Details
Level
Intermediate
Format
Talk
Technical depth
Practitioner
For
Mid-career, Senior individual contributor
Length
21:59
Published
June 2023
Questions this talk answers
Why run machine learning on the device rather than in the cloud?
On-device inference avoids data center and bandwidth costs and works regardless of connectivity. It gives users real-time results and keeps their data private because it never has to be sent to or stored on a server.
Why is model size such a critical constraint on mobile?
A typical deep learning model is a few hundred megabytes, and a large app size is the death knell on mobile, especially in emerging markets where mid to low range phones dominate. The speaker cited surveys showing lack of storage and battery drain are the most common reasons for uninstalls.
What is jank and how can on-device ML teams reduce it?
Jank is stuttering frames that appear when the GPU is handling UI rendering and model inference at the same time. Teams can profile it with iOS and Android developer tools, adapt layers to hardware accelerators and distribute layer computation across accelerators.
How can a company protect a model once it is deployed on a user's device?
Common practices include encryption, obfuscation and limiting the number of requests to curb adversarial attacks. Neural network watermarking adds hidden ownership signals that do not affect accuracy and resist removal.
How could a watermark work in a language model like ChatGPT?
One proposed technique selects specific tokens from a random set and softly promotes them during generation. Ordinary readers would not notice, but an investigator could trace the token selection back to prove the text came from the model.
About this talk
AI is moving from the servers to our pockets, making the technology on our phones faster and more private.
But how do you go from deploying a single machine learning model on the cloud to deploying millions of models around the world on every device in our customer's hands?
A mobile device comes with incredibly constrained resources and it requires running the triathlon of memory, speed and privacy to bring the technology to our phones.
Understand how to navigate this challenging journey and take the leap to on-device ML - because the future is at our fingertips.
Protecting AI models that live on user devices (63 s)
How a hidden watermark in ChatGPT could work (43 s)
All right. I guess we can get started. Hello, everyone. Hope you're all doing well. And thank you for attending this session. I'm Priyanka Kulkarni, and I specialize in building neural networks for on-device deployments at Adobe. Today, I'm going to talk about building on-device machine learning solutions to create, and I'm going to talk about building on-device machine learning solutions to create new technologies.
An AI-powered future that is fast, reliable, and secure. It's no surprise that mobile has become a predominant surface for AI interactions. We all want machine learning-powered magic at our fingertips everywhere we go. You all have probably used some form of AI on your phones for work, for travel, for shopping, or simply because you're bored. And there's tremendous demand rising for apps like digital assistants or face recognition and augmented reality.
An AI-powered technology, where people are reaching for their phones to engage with AI daily. An AI-powered technology. This has led to a huge uptick in the mobile AI market, now worth billions of dollars and rising. Recent studies have shown that applications that have advertised their association with OpenAI or ChatGPT, have seen their revenues jump dramatically in the past few months, as consumer demand around these technologies have skyrocketed. As more and more apps are incorporating machine learning into their offerings, This intelligence is moving closer to the edge where the computations are locally rather than in the cloud.
So in today's talk, I'll touch upon why on-device machine learning is important and relevant today more than ever, understanding the risks and challenges that come with the fabric of mobile platforms, including the security ones, and strategies that we can adopt to push the boundaries of neural network deployment across edge surfaces. The typical state-of-the-art neural networks are so large and computationally expensive that they require huge data centers with clusters of CPUs and GPUs to be able to run them.
They are too big to fit on edge devices and expensive to even run inference. So add to this bandwidth costs of transferring the data from the user's devices to the cloud, and you can tell that all of this adds up really quickly. On-device machine learning means that you can save all of these costs. This is also great for customers because now they can enjoy the benefits of access everywhere, regardless of connectivity, get instantaneous or real-time results, and assure data privacy.
Nobody likes the delay when the data is being streamed to the cloud, processed over there, and then sent back. And we can skip this whole hassle with on-device models. And because the data doesn't have to be sent to the server or stored there, there are fewer opportunities for cyber criminals to exploit any vulnerabilities, which also helps to preserve the sanctity of the data.
Now, integrating machine learning on the edge is nothing new, but the explosion of ML models in the world around us poses an increasingly difficult challenge to fit all of these intelligent systems onto resource-constrained devices like our mobile phones, which have limited compute, battery, memory, storage. Add to this mix, the generative AI race. I'm guessing you all have seen or heard about the mania around ChatGPT, DALI, mid-journey, and their mind-boggling abilities to write essays or create art.
I mean, I've been hooked to those apps for too many hours. And as generative AI is booming, so are the costs of running these models. They are way too large by smartphone standards and cannot be just taken off the shelf and run on our phones. So you can guess what the first challenge with on-device machine learning is, the model size and capacity.
We need a model that's small and fast, yet can provide high accuracy on the intended task. A large app size is the death knell on mobile. This is especially so in emerging markets around the world, where mid to low range smartphones are the majority. So we can't have our machine learning models driving up the app sizes by a few hundred megabytes, which is a typical size of a deep learning model.
A smaller size of the model also means we have fewer parameters than usual for the model to learn the task effectively. And in addition to this, we need to invest in accelerating the model inference so that we can provide users with low latency. These challenges are highlighted by innumerable surveys that show that a lack of storage space or battery running apps are the most common reasons for mobile app uninstalls.
So keeping the mobile model size low and the latency low is really non-negotiable. Finding this trade off between model accuracy and latency has become key in research as well. For instance, this chart tells us the relationship between a model's accuracy and its latency as we scale up the size. Each color here represents a specific architecture. And from looking at this, we know that the purple architecture gives us a really big boost for accuracy for a comparatively small increase in latency.
So it must be a really efficient model. And so happens that this model is also named efficient net. So it seems like a really good candidate for on-device models. In the bubble chart on the right, the size of each bubble represents how many parameters each model has. Here we see that the older models called BGG nets are the biggest models here and were created with a staggeringly high number of parameters.
Over time, though, research has taken machine learning models towards the direction of using fewer parameters and fewer operations while maximizing accuracy. And this is evident from the concentration of the models towards the top left section of the chart compared to where the old BGG models had started out. So it's great news that literature and industry is taking notice of how models can be made smaller as well as more efficient, which bodes well for on-device ML.
Typically, we've seen so far that convolutional neural networks remain the dominant choice for mobile deployments, where a majority of the deep learning models on mobile devices were some variant of mobile net. Mobile net architectures were developed with the intention to address the resource bottleneck seen on mobile. But over the past couple of years, the landscape of deep learning has been completely revolutionized with the advent of transformer based models.
These models have several advantages over CNNs, including introduction of attention layers for better information mapping and learning, as well as nuanced understanding of language. Following this, there's also been an interest of adapting transformers for mobile. However, they have been far more challenging to convert and run on device as compared to CNNs because they have more complex layers, higher floating point ops, and also exhibit much higher memory usage.
In attempts to optimize transformer models, so far we have seen that we've achieved varying degrees of mobile friendliness. This has raised the question if we can design architectures that are as mobile friendly as continents, but as efficient and accurate as transformers. A number of hybrid architecture designs have been proposed for combining advantages of CNNs with transformers for maximum efficiency, and mobile VIT is one of them where local processing done in convolutions are replaced with global processing using transformers.
Now, the selection of model architecture is crucial, but we also need to optimize the model further by shrinking it as much as we can. For this, we can use model compression techniques. This domain has seen various techniques developed over time and is widely being used already in literature and industry. These include things like network pruning to get rid of redundant or less important neurons, knowledge distillation from a teacher model to a smaller student model, as well as quantization to lower precision values.
As effective as these techniques have been, they are not that easy to automate, and it involves a lot of manual cumbersome tweaking and tuning to squeeze out every bit of optimization we can get, which is indispensable and goes a long way for mobile deployments. To combat manual tuning of models, you may have heard about the rise in popularity of neural architecture search.
So this is the process of automating the design of neural networks, where you search amongst various combinations of layers to create an architecture for your model requiring minimal human intervention. This is a fantastic way to find an optimal and efficient model where we can search for architectures that would work great for mobile. But what happens when we have diverse deployment scenarios on the edge like mobile browser or web.
These all present different resources as well as limitations for deployment. So we could deploy the same model across all of them, but then we'd be limited by the most constrained environment and end up being wasteful or inefficient in the others. So why not use neural architecture search to find a model tuned for each specific hardware environment. So this is a great idea.
But as these environments increase and scale, instead of rerunning neural architecture searches each time, we would want to find an effective way of walking the accuracy and latency curve for these models so that we can have greater flexibility and velocity to market. There has been interesting research in this field of scalable architectures, one research group at MIT has published a solution called once for all that attempts to tackle this problem.
And what they've proposed is training a single giant network, which we can then sample for smaller specialized sub networks of different sizes and different capacity, and then deploy them in various scenarios and target hardware. Also, not to forget that scalable architectures are like once for all ensure we're heading into the next generation of AI responsibly, because then these searches are targeted and optimized, so they are far better for the environment than traditional neural architecture search engines.
The second challenge with on-device ML is model efficiency. Due to resource limited environments on edge devices, our model needs to maximize its efficiency to consume the least amount of resources it can. This could be compute or memory. The on-device hardware landscape is also very diverse, especially on Android, where there are so many different manufacturers and hardware makers, leading to a spectrum of diverse resource limitations across all devices.
On mobile, this resource limitation often results in a contention problem, which if not handled properly can result in a symptom observed called jank. Consider a scenario where the GPU is simultaneously being called upon for UI graphics rendering requests, as well as running inference on machine learning models. This is common with real-time or interactive ML features where a user could be scrolling on his or her phone while the machine learning model is running simultaneously in the background.
In this case, you would get an experience like you see on the left-hand side, where there are more stuttering frames during the scroll. And this is called jank. While the right side presents a smoother scrolling experience for the user. There are now a lot of mobile developer tools available on iOS and Android to help us profile as well as measure jank.
We can mitigate against it. The CPU and GPU are not the only resources available on mobile. There are diverse set of hardware accelerators available across all platforms. One example is the XNN pack on TensorFlow Lite, which is a highly optimized library that enables faster CPU performance, including speed up of quantized models. Likewise, on iOS, the Apple neural engine is a special type of processor unit that you can leverage to make your ML models really fast by accelerating operations like convolutions and matrix multiplies.
There are a lot of other companies besides Apple developing their own AI accelerator chips, and you might have heard of Google's sensor processing unit as well. Even with all of these diverse hardware accelerators, there's a possibility that your ML model may not be able to use all of them. Many common layers seen in machine learning models may not be supported.
So we need to tailor our models to make the layers compatible with the accelerators that we intend to use. We can also parallelize the inference by developing an intelligent system to delegate and distribute the load of computing different layers using different accelerators. And this becomes even more critical when we have to run multiple ML models in parallel where they're all contending for the same device resources.
The third challenge of on-device ML is security. A significant amount of time, costs, data, and resources go into the development of each deep learning model, and it's critical to protect its intellectual property. When we deploy a deep learning model to the edge or on-client devices, they are far more vulnerable there than when they're queried using APIs in the cloud.
Thinking about the cybersecurity aspect of model protection is crucial. There are a lot of common best practices to safeguarding your deep learning model, like encryption, obfuscation, as well as limiting the number of requests to curb the potential for adversarial attacks. There's also research being done for embedding ownership signals into deep learning models directly, called watermarking. So this is similar to the watermarking techniques used to protect creator rights for digital images, video, and audio, and it has been a really effective technique to prove the ownership, authenticity, or integrity of these creations.
Neural network watermarking is a bleeding edge area of research to incorporate IP protection into deep learning models. These techniques aim at embedding ownership signals by inserting watermarks or backdoors into models that are so well disguised that it's hard to detect them. Aside from acting as the ownership proof, they do not affect the accuracy of the model and exhibit no other behavior.
And lastly, the watermark also persists against adversarial attacks, which makes it difficult or impossible to remove it. There's been some exciting research in extending this domain for watermarking generative AI models too. Watermarking gen AI is imperative not only to protect intellectual property, but also to prevent potential future malicious uses of language models like plagiarism, misattribution, misuse, or abuse of the outputs.
Recently, OpenAI also announced that they are planning to stop ChatGPT users from cheating on their homework or making social media bots by watermarking outputs produced by their models so that you can have a trace back to identify that they originated from ChatGPT. There are also recent research papers that have proposed frameworks for watermarking proprietary language models and already proven that the watermark can be embedded with negligible impact on the text quality in the outputs.
There are other challenges and security risks of generative AI models that need to be addressed. One is their vulnerability to training data extraction attacks. There is a potential risk that these models leak details about the data on which they've been trained. This point was raised in a research paper where they showed that if one prompts the GPT-2 language model with a particular prefix, it auto-completes a long block of text that contains sensitive and personally identifiable information, including contact details and the physical address of a particular person whose information was included in the model's training data.
There have been a lot of exciting developments in the field of optimizing and scaling down generative AI models, which will help pave the way for eventual on-device deployment. We're already seeing that these generative language models of different sizes and capacities can accomplish a variety of different tasks. So, while the larger models can pass college level exams or the bar exam, the smaller models can still achieve the ability of reading comprehension and logical deduction.
This is a great sign and helps us understand how to fine tune these models for business specific tasks or use cases and how we can optimize them for on-device. There's also a lot of promising evidence of running large scale AI with fewer parameters. For instance, the models like stable LM and Meta's Lama model have only around a few billion parameters.
This sounds like a lot, but these are vastly optimized models compared to the GPT models, which have hundreds of billions of parameters and have no hope in the world of running on smartphones. More recently, Web LLM is a project that brings large language models and chatbots to web browsers. So, everything is run inside the browser without any server support and it also leverages Web GPU acceleration.
Recently, Squeezebits also claim that they have optimized image generation models from stable diffusion to be able to run it on an Android phone, which is just astonishing. And why stop at only inference for on-device ML? We can also train and update these models locally without the data ever leaving the user's devices. Models can be fine-tuned to achieve better flexibility as well as offer personalization.
This concept can then extend to federated learning, which marries the advantages of on-device training with cloud training. So, you get the privacy benefit and the scale of on-device learning by keeping the data local closer to the user. And you get the benefit of learning from diverse data across users by having the cloud aggregate many locally trained models.
I'm so excited to see the proliferation of AI based services around us and the ability of running these lightweight personalized models locally on our phones will help us unlock the full potential. So, thank you all for listening. And with that, I'm happy to take any questions. Please feel free to drop any questions in the Q&A chat pod as well, and I can get to them offline too.
Okay, so we have a question from Sandra. What could be an example of a watermark in chat GPT? So, that's a great question, Sandra. One of the language models behind all of these chat GPT and prior technologies have a proposed watermarking technique, which the company isn't currently using, but the architecture behind these language models involve the use of transformers.
Which are composed of blocks that leverage tokens. And an example of a watermark is to select specific tokens from random set and softly promote the use of those specific tokens when generating the output. So, these watermarks would not be visible to a layman when chat GPT produces certain outputs. But if I'm someone who's a legal authority who would want to then investigate and trace it back, I can run reverse engineering using the token selection to be able to uncover and prove that this watermark originated from chat GPT.
But if I'm just a general user who's just going through a paragraph produced by chat GPT, it would be sort of masked from me. I have another question from Deborah. Can you explain a bit about the on-device and consolidated data with an example? Sure. So, you can think of it as each of us having smaller localized models with fewer parameters that learns to give us specific types of responses.
So, I might be having slightly different tolerance for bias or outputs of a language model compared to someone else. So, that would be really important. So, I think it would be very important. It would be very important to reflect my interests, my ethics, my values. Whereas, the larger models would get an aggregated version of all of our phones to reflect what general customers, you know, the mean average of customers would expect.
So, that helps to keep data local to me and a model that's personalized to my beliefs and my interests while also updating the larger models on server that reflects the aggregate data across all of the users without infringing privacy. I hope that helps answer your question. Thank you. And I think we're out of time. Thank you all for attending this session.
I really appreciate it. And please feel free to connect with me. All right. Have a great day, everyone.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
What it takes to run deep learning on a phone
Adobe's Priyanka Kulkarni breaks on-device machine learning into three challenges, size, efficiency and security, and the techniques that meet each one.
In short
Running AI on the device instead of in the cloud cuts server and bandwidth costs, removes latency and keeps user data private, but only if the model is small, efficient and protected. Priyanka Kulkarni, Senior Machine Learning Engineer at Adobe, frames on-device ML as a triathlon of memory, speed and privacy, and walks through the architectures, compression methods, hardware accelerators and watermarking techniques that make it work.
Key takeaways
Learn how to accelerate the journey from server to mobile for deploying deep learning neural networks.
Understand the risks and challenges that come with deploying neural networks on device.
Understand the security implications of deploying neural networks on device, and which solutions and strategies to adopt to secure your intellectual property.
Understand the fabric of mobile deployment platforms for both Android and iOS.
Push the boundaries of machine learning on mobile.
Why AI is moving off the server
Kulkarni opens with the observation that mobile has become a predominant surface for AI interactions, from digital assistants to face recognition and augmented reality. State-of-the-art neural networks, however, are built for data centers full of CPUs and GPUs. Add the cost of moving user data to the cloud and back, and the bill grows quickly.
On-device machine learning removes those costs and gives users something better: access regardless of connectivity, real-time results and stronger privacy. As she puts it, "Nobody likes the delay when the data is being streamed to the cloud, processed over there, and then sent back." (2:38) Because data does not need to be sent to or stored on a server, there are also fewer openings for attackers to exploit.
Challenge one: model size and capacity
Phones have limited compute, battery, memory and storage, and the generative AI boom has only made models bigger. Kulkarni is blunt about the consequences: "A large app size is the death knell on mobile." (3:50) She notes that a typical deep learning model is a few hundred megabytes, and that this matters most in emerging markets where mid to low range phones are the majority. As the speaker cited, surveys show lack of storage and battery drain are the most common reasons people uninstall apps. Her conclusion: "So keeping the mobile model size low and the latency low is really non-negotiable." (4:38)
Architecture choice is the first lever. Using research charts, she points to EfficientNet as a model that gains a lot of accuracy for a small increase in latency, and shows how the field has moved away from parameter-heavy VGG networks. MobileNet variants have dominated mobile so far, while transformers bring attention layers and better language understanding but are harder to run on device. Hybrid designs such as MobileViT try to combine the best of both.
Compression is the second lever: pruning redundant neurons, distilling knowledge from a large teacher model into a smaller student, and quantizing to lower precision. Because this tuning is manual and cumbersome, she highlights neural architecture search, which automates network design. When a model must run in several environments, such as mobile, browser and web, one model forces a compromise: "So we could deploy the same model across all of them, but then we'd be limited by the most constrained environment and end up being wasteful or inefficient in the others." (8:45) She describes Once-for-All, from an MIT research group, which trains one large network and samples smaller specialized subnetworks for each target, an approach she also calls better for the environment than repeated searches.
Challenge two: efficiency, jank and accelerators
The second challenge is getting the most out of scarce, highly varied hardware, especially across the many Android manufacturers. When the GPU is asked to render the interface and run inference at the same time, users see stuttering frames, known as jank. Developer tools on iOS and Android now help teams profile and measure it.
Kulkarni points to hardware accelerators beyond the CPU and GPU: XNNPACK in TensorFlow Lite for faster CPU performance and quantized models, the Apple Neural Engine for operations like convolutions and matrix multiplies, and Google's Tensor Processing Unit. Not every layer is supported everywhere, so "So we need to tailor our models to make the layers compatible with the accelerators that we intend to use." (12:21) Teams can also build systems that distribute layers across accelerators, which becomes critical when several models compete for the same device.
Challenge three: security and watermarking
Every model represents significant time, data and money, and shipping it to user devices exposes it. "When we deploy a deep learning model to the edge or on-client devices, they are far more vulnerable there than when they're queried using APIs in the cloud." (12:47) Standard defenses include encryption, obfuscation and limiting requests to curb adversarial attacks.
She then turns to neural network watermarking, a bleeding edge research area that embeds hidden ownership signals or backdoors into a model. A good watermark does not affect accuracy, shows no other behavior and survives adversarial attempts to remove it. The idea is being extended to generative AI, where it could help against plagiarism, misattribution and misuse. She mentions that OpenAI announced plans to watermark ChatGPT outputs, and a research paper showing that GPT-2 could be prompted to leak personal information from its training data. In the Q&A she explains one proposed technique: softly promoting specific tokens chosen from a random set, invisible to readers but traceable by an investigator.
Generative AI and federated learning on device
Kulkarni closes on progress in shrinking generative models. Models like StableLM and Meta's LLaMA have a few billion parameters, compared with hundreds of billions for GPT models; WebLLM runs language models in the browser using WebGPU; and SqueezeBits claims to run Stable Diffusion image generation on an Android phone. She then pushes further: "And why stop at only inference for on-device ML? We can also train and update these models locally without the data ever leaving the user's devices." (17:06) Federated learning combines this local, private personalization with cloud aggregation of many locally trained models, so the shared model learns from diverse users without collecting their data.
Watch the talk
"On-device ML: Artificial Intelligence in your pocket" by Priyanka Kulkarni, 22 minutes: https://www.womentech.net/video/device-ml-artificial-intelligence-in-your-pocket
About the speaker
Priyanka Kulkarni is a Senior Machine Learning Engineer at Adobe, where she works on next-generation AI innovation for mobile devices and builds neural networks for on-device and edge deployment. For the past five years she has applied deep learning to model training, compression, optimization, IP protection and mobile deployment for document image analysis and detection, and she holds a Master's degree from Carnegie Mellon University specializing in mobile and pervasive computing.
Questions this talk answers
Why run machine learning on the device rather than in the cloud?
Why is model size such a critical constraint on mobile?
What is jank and how can on-device ML teams reduce it?
How can a company protect a model once it is deployed on a user's device?
How could a watermark work in a language model like ChatGPT?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Neural architecture search
Once-for-All
Network pruning
Knowledge distillation
Quantization
Neural network watermarking
Federated learning
Resources mentioned
TensorFlow Lite XNNPACK
Apple Neural Engine
Once-for-All (MIT)
WebLLM
Cite this talk
Priyanka Kulkarni. "On-device ML: Artificial Intelligence in your pocket." WomenTech Network, 2023. https://www.womentech.net/video/device-ml-artificial-intelligence-in-your-pocket