0:0021:59

On-device ML: Artificial Intelligence in your pocket

Priyanka Kulkarni 21:59 2023

A few hundred megabytes of model can be the death knell of a mobile app

A typical deep learning model weighs a few hundred megabytes, and surveys show running out of storage and battery drain are the most common reasons people uninstall apps. Teams moving AI from the cloud to phones must win on memory, speed and security at once, often on mid to low range devices. Adobe senior machine learning engineer Priyanka Kulkarni maps that terrain: how do you shrink, accelerate and protect a model that lives in a customer's pocket?

Learn the compression, acceleration and watermarking strategies that make on-device ML practical

Watch on YouTube

In short

Priyanka Kulkarni, Senior Machine Learning Engineer at Adobe, explains what it takes to move deep learning models from cloud servers onto phones and other edge devices. She works through three challenges (model size, model efficiency and security) and covers compression, neural architecture search, hardware accelerators, model watermarking and federated learning as ways to meet them.

“A large app size is the death knell on mobile.”
“So keeping the mobile model size low and the latency low is really non-negotiable.”

What you will learn

  • Weigh the cost, latency, connectivity and privacy benefits of running models on device instead of in the cloud.
  • Choose mobile-friendly architectures such as EfficientNet, MobileNet or MobileViT and shrink them with pruning, knowledge distillation and quantization.
  • Use neural architecture search and Once-for-All style scalable networks to target several hardware environments without starting over each time.
  • Diagnose jank from GPU contention and adapt model layers to hardware accelerators such as XNNPACK or the Apple Neural Engine.
  • Protect deployed model IP with encryption, obfuscation, request limits and neural network watermarking.

Key takeaways from the speaker

  • Learn how to accelerate the journey from server to mobile for deploying deep learning neural networks
  • Understanding the risks and challenges that come with deploying neural networks on-device
  • Security implications of deploying neural networks on-device and which solutions and strategies to adopt to secure your intellectual property
  • Understand the fabric of mobile deployment platforms for both Android vs iOS
  • Push the boundaries of machine learning on mobile

Details

Level
Intermediate
Format
Talk
Technical depth
Practitioner
For
Mid-career, Senior individual contributor
Length
21:59
Published
June 2023

Questions this talk answers

Why run machine learning on the device rather than in the cloud?

On-device inference avoids data center and bandwidth costs and works regardless of connectivity. It gives users real-time results and keeps their data private because it never has to be sent to or stored on a server.

Why is model size such a critical constraint on mobile?

A typical deep learning model is a few hundred megabytes, and a large app size is the death knell on mobile, especially in emerging markets where mid to low range phones dominate. The speaker cited surveys showing lack of storage and battery drain are the most common reasons for uninstalls.

What is jank and how can on-device ML teams reduce it?

Jank is stuttering frames that appear when the GPU is handling UI rendering and model inference at the same time. Teams can profile it with iOS and Android developer tools, adapt layers to hardware accelerators and distribute layer computation across accelerators.

How can a company protect a model once it is deployed on a user's device?

Common practices include encryption, obfuscation and limiting the number of requests to curb adversarial attacks. Neural network watermarking adds hidden ownership signals that do not affect accuracy and resist removal.

How could a watermark work in a language model like ChatGPT?

One proposed technique selects specific tokens from a random set and softly promotes them during generation. Ordinary readers would not notice, but an investigator could trace the token selection back to prove the text came from the model.

About this talk

AI is moving from the servers to our pockets, making the technology on our phones faster and more private. But how do you go from deploying a single machine learning model on the cloud to deploying millions of models around the world on every device in our customer's hands? A mobile device comes with incredibly constrained resources and it requires running the triathlon of memory, speed and privacy to bring the technology to our phones. Understand how to navigate this challenging journey and take the leap to on-device ML - because the future is at our fingertips.

Topics

Related talks

Browse the library