0:0015:11

Operationalising AI Ethics

Natalie Rouse 15:11 2022

AI ethics principles mean little until they live in your ML lifecycle

Discriminatory hiring models, facial recognition that performs poorly for some groups, deepfakes: the headlines have pushed many organisations to adopt AI ethics principles, yet there is still no standard way to build them into model development. Natalie Rouse, general manager of data science consultancy Eliiza and former lead of analytics and AI delivery at Qantas, maps where ethical review belongs. But how do you decide what an acceptable accuracy gap between groups looks like?

Learn where ethics reviews fit in the AI lifecycle and how they become production KPIs

Watch on YouTube

In short

Natalie Rouse, General Manager of Eliiza, explains how to move AI ethics from agreed principles into the day-to-day development lifecycle. Using the CRISP-ML process model, she shows where ethical impact assessments, subpopulation analysis and performance tolerances fit, and how they become KPIs monitored in production through MLOps.

“Please don't just copy and paste some ethical principles from the internet.”
“Doing something is infinitely better than doing nothing.”

What you will learn

  • Identify the ethical principles most organisations have converged on, such as fairness, transparency, accountability, contestability, privacy and security.
  • Extend a privacy impact assessment into an ethical impact assessment that lets low-risk projects proceed quickly.
  • Define subpopulations in your data to measure representation and model performance across groups.
  • Set realistic performance tolerances by linking accuracy differences to real-world outcomes for each group.
  • Embed ethics KPIs in an MLOps framework as alerts and retraining triggers visible to stakeholders.

Key takeaways from the speaker

  • The 5 commonly used ethical principles to apply to any AI project
  • The most important high-level considerations for embedding ethics into the AI project lifecycle
  • How and where to undertake ethical review tasks within the standard project development workflow
  • How undertaking an ethical review as part of the development process creates KPI's for ongoing monitoring in production

Details

Level
Intermediate
Format
Talk
Technical depth
Practitioner
For
Mid-career, Senior individual contributor, People manager
Length
15:11
Published
July 2022

Questions this talk answers

Why is agreeing on AI ethics principles not enough?

Principles do not automatically bridge the gap to the development lifecycle. Careful work is needed to embed them into development and deployment without adding prohibitive overhead.

Where in an AI project should ethical review happen?

At scoping through an ethical impact assessment and risk calculation, during model development through subpopulation representation and performance reviews, and at deployment through recommendations for use and a living report.

How can low-risk AI projects avoid unnecessary ethics bureaucracy?

Structure the ethical impact assessment so projects with little or no human or environmental impact drop out quickly and proceed, while riskier projects get adequate due diligence.

How do you decide whether accuracy differences between groups are acceptable?

Ask what the gaps mean in reality, whether a group falls below a critical level or a tipping point exists, and use those answers to set tolerances for monitoring.

What happens to ethics once a model is in production?

Ethics is not a set-and-forget exercise; performance across subpopulations should be monitored as ethics KPIs in the MLOps framework, with thresholds triggering alerts or retraining.

About this talk

The need for considering the ethical implications of AI projects is now widely agreed upon by technical practitioners and business leaders across industries. Much less understood is how to embed ethical principles into development and monitoring lifecycles - this talk will outline some practical ways to implement an ethical review process alongside AI development projects, and ensure adherence to ethical standards is visible for algorithms in production.

Topics

Related talks

Browse the library

Scale Enterprise AI with Trust

Aravinda Gollapudi Head of Platform and SVP of technology for Sage Intacct, Sage

LLMs give the likeliest answer, not the correct one, and enterprises pay for it

Intermediate AI and data 2025