In short
Deep learning models that forecast events such as crimes, disease outbreaks and viral content can be fooled by small, deliberate changes to their inputs and shaken by natural shocks like a pandemic. Samira Khorshidi argues that studying adversarial attacks, and building defenses against them, is central to trustworthy AI because it reveals how models really behave. Her research brings accuracy, cybersecurity and trust together to make deep point process models more robust, explainable and transparent.
Key takeaways
- Adversarial attacks and defense mechanisms against deep point processes are a matter of trustworthy AI and machine learning.
- Adversarial attacks are an important step towards understanding deep learning models.
Trust runs through the research
Khorshidi, who described a path from software developer and CTO in Iran to a master's degree in Germany and a PhD in the United States, opened by placing her work in the wider frame of trustworthy AI. Her research touches each of its wings: making machine learning models explainable, studying model fairness for marginalized groups, and testing robustness against adversarial attacks and natural shocks in data. She noted that the full work is highly technical, but chose to introduce the ideas rather than the mathematics.
"And I'm saying that because I believe that machine learning models should be like trustworthy and we should have a like a machine learning model that is explainable responsible for the decision making and also safe and robust." (2:06)
What point processes explain
To define her domain, she started with point pattern analysis, a field of mathematics that studies how points are arranged in a space, whether that space is time, physical location or both. A point process describes how such a pattern might have been generated. Point processes are used in criminology to analyze the locations of crimes and accidents, in epidemiology to follow how diseases such as COVID-19 spread, and in natural resource management for floods and wildfires.
"So basically what I'm talking about is the causality here right so point processes are before are explaining the cause behind the arrangements." (4:43)
Point processes can be deterministic or stochastic. Crimes, disease spread and wildfires are stochastic: random, but random in ways that can be explained. Her example map showed crimes in Indianapolis forming visible clusters. Her research focuses on clustered processes in which events are not independent. An initial infected person appears at random, secondary infections follow nearby and soon after, and those new cases infect others in turn. The same dynamic applies to crime and earthquake aftershocks. In earlier published work, she showed that the underlying network, such as the network of people in a society, can help explain how far a process spreads.
Why adversarial attacks matter
Khorshidi models these processes with deep learning, then crafts adversarial attacks against the models and simulates non-stationary changes, such as the effect COVID-19 had on crime in US cities. From there she proposes methods that make the models more robust, interpretable and transparent.
"And it's important because like the whole field the point processes is hard to understand itself and then when it comes to deep learning models, there is always the risk of adversarial attacks, regardless regardless of the performance and accuracy of these models." (8:56)
She explained adversarial examples with a familiar case: two stop sign images that look the same to people, where a tiny amount of noise is enough for a model to read one as a different sign. She asked the audience to imagine the same thing happening to a car driving on autopilot.
"And you know that like the adversarial attacks are the samples are the input to our models when the perturbation is not sensible to human eyes." (9:31)
For event data, her attacks are inspired by real dynamics: YouTube video views, Twitter sharing, viral marketing campaigns, the spread of computer viruses and gang retaliation. With social media data, one attack might change the text of posts while another changes the timing of new events, and each has to be considered separately. She also examines how model design affects sensitivity, for instance whether a deeper model or a larger input changes how robust the model is.
Defenses start with knowing your data
Khorshidi outlined defenses used in the literature and in practice. Adversarial training uses adversarial samples to train the model. Alternative architectures, such as convolutional layers instead of regular dense layers, may perform better. Generative adversarial networks offer another route, built on the idea that while people may not fully understand deep learning models, those models can be used to improve other deep learning models. She stressed that any robustness claim must be specific: is it attack-specific, or attack-agnostic and certifying the model against all possible attacks?
"Towards the robustness it's important to know your data for example here for me my data itself it's like random it has noisy." (13:06)
Noisy, random event data creates a real risk of overfitting, so practitioners also need to measure the uncertainty and sensitivity of their models.
The hidden cost of bigger models
Her final emphasis was cost. As networks grow larger, she observed, few discussions address how much CO2 large companies' data centers produce or how their models affect the environment.
"Not just in their local environment, but the global like climate change is affected by the big companies and the cost of their deep learning models." (14:15)
She closed by describing the work as a cycle of modeling attacks and defenses that must be studied both separately and together. The lessons reach beyond crime and disease: customer behavior analysis, promotion campaign management, supply chains and speech recognition systems all rely on similar deep learning models and similar data, and all are vulnerable to adversarial attacks.
Watch the talk
"Adversarial attacks and defense mechanisms against deep point processes" by Samira Khorshidi, about 16 minutes: https://www.womentech.net/video/adversarial-attacks-and-defense-mechanisms-against-deep-point-processes-samira-khorshidi
About the speaker
Samira Khorshidi, Research assistant, is a computer science PhD candidate and graduate research assistant at Indiana University-Purdue University Indianapolis (IUPUI), where her thesis focuses on adversarial attacks and defense mechanisms in trustworthy machine learning. She has seven years of industry experience in database and software development, has published several NSF-supported journal and conference papers, and co-founded the Lou-AI monthly meetup series.
Questions this talk answers
- What is a point process?
- What is an adversarial attack on a machine learning model?
- How can deep learning models be made more robust to adversarial attacks?
- Why does the nature of the data matter for robustness?
- What cost of large deep learning models is often overlooked?
Drafted with AI from the talk transcript. Editor review pending.