I know where you wanna go next summer - Demand Forecasting at FlixBus
17:082021
A forecast nobody trusts creates no value, however good the model
OpenStreetMap covers small German towns in detail but leaves much of the US thin, and in a 2020 survey almost 90% of its contributors identified as male: gaps that flow straight into any forecast built on that data. Miriam, a former data scientist who now leads network planning products at FlixBus, forecasts demand across 30 million possible connections. What does it take for planners to stake a whole network on those forecasts?
Learn three hard-won lessons for turning machine learning forecasts into decisions people trust
Miriam, Domain Product Officer at FlixBus, explains how her teams forecast passenger demand between cities across a network of 30 million possible connections to plan bus supply a year ahead. She argues that successful applied machine learning depends on solid data groundwork, an experimental and iterative development process, and early involvement of the people who must trust and use the forecasts.
“Um, but basically like in general, if you feed like garbage data into your model, no matter how good that model is, you will get just a garbage forecast.”
“So it's not that you build one model, but you build hundreds of candidate models.”
What you will learn
Frame demand forecasting as one input to a larger human decision-making system rather than an end in itself.
Identify the external factors, such as population, distance and tourist attractions, that drive demand between locations.
Spot non-random gaps and bias in crowdsourced data before they produce biased forecasts.
Plan for an experimental process with many candidate models, dead ends and structural breaks such as the pandemic.
Build user trust by involving domain experts early, running feedback rounds and using explainable machine learning.
Details
Level
Intermediate
Format
Talk
Technical depth
Introductory technical
For
Mid-career, Senior individual contributor, People manager
Length
17:08
Published
June 2021
Questions this talk answers
Why does FlixBus need automated demand forecasting?
As the network grew to around 30 million possible connections with a limited number of buses, manual planning no longer scaled. Planners need a forecast in space and time to decide which connections to serve and how many buses to supply.
What factors influence demand between two cities?
The population of the two cities and the distance between them are core factors: the bigger and closer the cities, the more demand. Tourist attractions and many other factors add on top of that.
Why is data groundwork so important in machine learning projects?
Garbage data produces a garbage forecast no matter how good the model is. Crowdsourced sources like OpenStreetMap have non-random gaps, such as less detail in the US than in Germany, which must be handled to avoid biased forecasts.
How did the pandemic affect FlixBus's forecasting models?
The lockdown created a structural break in the travel data. The team has to account for that special period when training and evaluating models on historic data.
How do you get planners to trust and use a forecasting model?
Involve them early as part of the development process through feedback rounds, interviews and testing phases. Explainable machine learning methods also help show why a forecast has a certain value.
Machine learning is a mountain of unknown height (74 s)
No trust, no value from your model (74 s)
Speaker: Good morning, everyone. I'm very happy to be here and to talk about demand forecasting. My name is Miriam. I'm a Domain Product Officer at Flixbus. And yes, Anna said, I have a background in statistics and then worked many years as a data scientist before I joined Flixbus. And at Flixbus, I have been working with my teams on automating network planning in a smart way with data and machine learning and AI. And what's network planning? Basically, network planning is responsible to plan our network worldwide and to build the schedules for our buses and our train network.
Speaker: And I'm in the network planning domain. We have five product teams and we build products for network planning. So they are our internal customers. Speaker: And what I'd like to talk about today is demand forecasting and actually the title of my presentation, it's a bit misleading. And I'd like to explain why and what demand forecasting actually means in our context.
Speaker: So well, of course, it's not me, but it's a whole team driving the forecasting effort and we don't know, but we forecast and forecasting or every forecast is subject to uncertainty. Speaker: And also we are not interested in you as an individual, but we are interested in potential passenger flows or passenger flows between cities. So don't worry. We do not track you or anyone.
Speaker: Um, and next summer, well, that refers to our planning horizon and yes, our planning horizon is quite long. So we plan to like right now, we plan for next summer, but of course, not only for summer, but for all seasons. Speaker: And what I would like to cover in the following is like, what is the problem that we aim to solve with demand forecasting?
Speaker: Um, and then second, um, give you a glimpse at how we approach the problem. And then most importantly, I would like to share a few learnings, um, that we made on the way. And, um, my aim is here to create awareness. And, um, I hope that this might be useful for you because probably also in your context, on your company, um, you are faced with.
Speaker: You are faced with philomela challenges, um, like when applying on machine learning and I to real world problem. And although your use case might be different, I think there's a lot of like common patterns. Speaker: So what, why do we do demand forecasting, right? What is the problem that we aim to solve? Speaker: We went from just a few connections in Germany to Europe's law largest long distance bus network. And we're also operating in the US and Turkey and like starting off, you can still keep track of a few connections and optimize like manually and expand your network manually.
Speaker: You look at all the informations that you have at hand and then combine that with your planning expertise. Speaker: But as network growth complexity increases and we need to make a lot of decisions, like which connections should we serve and how many buses shall we plan to supply on a certain connection at a certain point in time.
Speaker: And what you see here in the next slide as an example is all the potential connections out of Florence. Speaker: So this is now only for one origin city and it gets quite overwhelming if you try to plot that for the whole network. Speaker: Um, but basically what you can see is that we need to make a lot of decisions on the supply side, um, for a network of like 30 million possible connections.
Speaker: And on the other side, there's of course only a number of buses that is limited and not all connections are equally attractive to our customers. Speaker: Um, so for instance, I would expect more customers going from Florence to Rome than going from Florence to Munich. Speaker: And also there's quite some fluctuation in demand over the course of a year or a week or even within a day.
Speaker: So what we need for purposes is we need a forecast in space and in time. Speaker: And we need given the size of their problem, we need an automated, scalable solution. Speaker: And then based on that, we can plan our supply in our network. Speaker: So what is demand forecasting demand for us? Speaker: It's the sum of our potential passengers that would like to travel between city A and city B by bus in a certain time period.
Speaker: And forecasting is predicting something in the future. Speaker: Now, of course, we cannot foresee the future, but what we can do is we can make a best guess taking all the information that we have at the current point in time into account. Speaker: All the data that we have right now and then make a forecast about the future.
Speaker: And in that way, demand forecasting, it's a tool for decision making, and it's like one piece in a larger decision making system. Speaker: So on one hand, we have demand on the other hand, we have buses, our supply, and we need to make a decision how with our supply meet demand. Speaker: And that decision needs to be made well in advance, way before the buses are on the road.
Speaker: And the demand forecast for next year, they feed into the decision making system. Speaker: And in that way, they contribute to efficient and effective network planning. Speaker: And they ultimately, they make a business objective happen. Speaker: So how do we approach the problem? Speaker: And here, I'll only give you a glimpse because 20 minutes are way too short for that.
Speaker: But we started off with a question. Speaker: What are the factors that influence demand in space and in time? Speaker: And now focusing on the space dimension, what are the factors to make a connection attractive to travelers? Speaker: And I will give you one example. Speaker: Where would you expect more passengers on a connection from Florence to Rome or a connection from Florence to Saint?
Speaker: At least I would prefer to go from Florence to Rome traveling three hours instead of going 19 hours to Saint in France. Speaker: And also, of course, Rome has slightly more tourist attractions than Saint. Speaker: They also have one amphitheater and a camping ground. Speaker: But a part of that, I think Rome is way more attractive.
Speaker: And in both cities, Florence and Rome, there are way more people who would like to travel than compared to Saint. Speaker: So this gives you an idea of a few factors that influence demand. Speaker: And I summarized the views that we touched on that slide. Speaker: So the population of two cities and the distance between the two cities, they are factors that influence demand.
Speaker: So the bigger the cities, the closer they're together, the more demand we expect. Speaker: And then tourist attraction is one of many other factors that come on top of that and influence demand. Speaker: So what we realized is that like many of those factors that influence demand, it's not data that we have at hand. Speaker: It's not data that people use ourselves, but it's data that we need to source externally.
Speaker: So we need to discover data sources and then collect the data. Speaker: And luckily, there's a lot of open source data, for instance, from government agencies and geospatial data is provided by OpenStreetMap, which is a crowdsource geospatial data. Speaker: And then once we have the data, we need to start with the modeling. Speaker: And here what we did was we kind of took that big problem and broke it down into like several sub problems.
Speaker: I already mentioned like the space and the time dimension. Speaker: So basically what we did is we built several models, not one, and combined them together in one forecasting system. Speaker: Like this is now like a very, very, very simplified overview of the approach. Speaker: But basically there's data and the data that can be factors that are externally, but can also be factors that are decision variables.
Speaker: Decision variables on our side and they feed into just like forecasting system and like this system of models. Speaker: And then that produces a forecast. Speaker: And now after this, like very simple, like glimpse of the approach, I don't want to go too much into the technical details. Speaker: Um, but what I'd like to focus on is to share some real life experience and some learnings of like this, this journey of applying machine learning to a, to a real life problem.
Speaker: And I would like to share with you what we have learned on that journey so far. Speaker: Now, first of all, what we found very important is the data groundwork. Speaker: Nobody wants to do the data work. Speaker: Everyone wants to do the fancy machine learning work, but actually the data work, it is super important. Speaker: And I would like to give you one, one example on why it's important.
Speaker: Um, but basically like in general, if you feed like garbage data into your model, no matter how good that model is, you will get just a garbage forecast. Speaker: Um, and then one example, um, that I, that I, that I have for you is, um, on, um, open, open street map. Speaker: So we leverage, um, data, um, from open street map and that data is crowdsourced.
Speaker: Um, and in that way, the data is reflection of the interests of those people who created data. Speaker: And now if you ask who creates it where like for this audience, probably very interesting. Speaker: There might be a slight gender bias, um, in a survey from 2020. Speaker: Um, they found that almost 90% of contributors identified as male.
Speaker: Um, and also, um, where we have, um, like a bias is in, um, like spatial distribution. Speaker: So the majority of contributors comes from Europe. Speaker: And that's actually something that we can see in the data. Speaker: So for instance, for Germany, um, a European country, even small cities, they are covered in great detail in open street map.
Speaker: However, if you then go to the US there, it's far less detailed. Speaker: And these these data gaps dismissing this. Speaker: That is not random. Speaker: And that's something that you have to deal with in the modeling because otherwise you'll get bias forecasts. Speaker: Now the second learning that I would like to share with you is on the experimental nature.
Speaker: And the iterative nature of the development process, um, which I think holds true for many machine learning and AI efforts. Speaker: And, um, I have like two metaphors for you. Speaker: It's. Speaker: It's like a research lab where you have a lot of iterations where you first you come up with ideas and hypotheses. Speaker: Then you test those ideas.
Speaker: Then you evaluate the ideas. Speaker: And if the evaluation shows that that idea improves your model, then you incorporate that into your forecasting system. Speaker: And you don't know in advance which factors have predictive power. Speaker: And you also do not know in advance which model type will work well for your particular problem. Speaker: So you have to try out many different factors and many different candidate models and evaluate them with respect to their forecast performance.
Speaker: So it's not that you build one model, but you build hundreds of candidate models. Speaker: And it's also a little bit like climbing a mountain where you don't know beforehand how high the mountain is, how high you have to climb. Speaker: And you have an idea on the routes that go up here, but you have to take the routes to then discover that some of them are dead ends.
Speaker: Or that some of them there is like some surprises behind the corner. Speaker: And I actually one surprise that hit us was the pandemic and the pandemic created a structural change with the lockdown, which had a massive impact on the travel industry. Speaker: And that impact manifests itself in the data as so-called structural break. Speaker: And that's now something that we have to deal with in our modeling.
Speaker: So when we train our models on historic data or when we evaluate our models on historic data, we have to take into account that there was this very special period. Speaker: The third learning that I would like to share with you is on taking our stakeholders along. Speaker: So in our case, and I think that has true for many other use cases, the model, it doesn't live in isolation.
Speaker: So there's an interaction of a machine that produces a forecast and a human. Speaker: So the forecasts, they are just one piece in a larger decision making system. Speaker: And humans look at forecasts for the decision making, and they might inject domain knowledge and strategic targets also either into the forecast or in the subsequent supply optimization.
Speaker: So ultimately, it is still humans that are responsible for the decision making, and they need to build up trust in the model. Speaker: They need to trust the model to be willing to base their decisions on the model's recommendation. Speaker: Otherwise they won't use it. Speaker: And if they don't use it, the forecast will not create any value.
Speaker: So what we found helpful to do was to involve our users. Speaker: And in our case, it's internal users. Speaker: So our network planners involve them early on to create acceptance and trust. Speaker: So we basically made them part of the development process. Speaker: And it's not only important to create a trust, but it also helps us to improve the model by incorporating their domain expertise.
Speaker: So what we found helpful to do is to, apart from the standard, rigorous model evaluation process, to also do user feedback rounds, interviews and testing phases with the users. Speaker: And it helps us to detect oddities. Speaker: So places where the model might might be off, and then with their domain knowledge, the experts, they can help to contribute ideas why a forecast may be off in certain areas.
Speaker: And a second point that we found important in order to kind of take our users and our stakeholders along is to leverage methods of, Speaker: like so-called explainable machine learning. Speaker: And that helps us to shed some light on the rather black boxy models and explain the user why forecast shows a certain value. Speaker: So. Speaker: To summarize, let me briefly restate the three learnings that I shared with you.
Speaker: First, the data groundwork is very important, at least as important as the modeling itself. Speaker: And then the experimental nature of the development process creates some additional challenges. Speaker: And that's why it's very important to take your stakeholders along in that in a journey and create trust. Speaker: But also learn from the expertise in the iterative development process.
Speaker: And I mean, yes, there are challenges, but they are also large opportunities. Speaker: And the journey is super exciting. Speaker: And I'm, yeah, I'm very happy to be part of, of the journey or of that bus ride and work with my teams on, on solving those challenges. Speaker: So yeah. Speaker: Thank you very much for your time.
Speaker.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
What makes a machine learning forecast good enough to plan a bus network on?
FlixBus's Miriam explains why data groundwork, patient experimentation and planner trust matter as much as the model itself.
In short
A demand forecast only creates value when the people making decisions trust it enough to use it. Miriam, who introduces herself as a Domain Product Officer at FlixBus with a background in statistics and data science, shares how her teams forecast passenger flows between cities across a network of 30 million possible connections. Her three lessons: invest in data groundwork, expect an experimental journey, and bring stakeholders along from the start.
Key takeaways
Frame demand forecasting as one input to a larger human decision-making system rather than an end in itself.
Identify the external factors, such as population, distance and tourist attractions, that drive demand between locations.
Spot non-random gaps and bias in crowdsourced data before they produce biased forecasts.
Plan for an experimental process with many candidate models, dead ends and structural breaks such as the pandemic.
Build user trust by involving domain experts early, running feedback rounds and using explainable machine learning.
Why a growing network needs forecasts
Miriam works in the network planning domain at FlixBus, where five product teams build tools for internal customers: the network planners who design schedules for buses and trains worldwide. She opens by explaining that her playful talk title is a little misleading. The team does not track individuals, it forecasts passenger flows between cities, and every forecast carries uncertainty.
The need comes from scale. FlixBus grew from a few connections in Germany to what Miriam describes as Europe's largest long-distance bus network, with operations in the US and Turkey as well. A handful of routes can be optimized by hand. A network of roughly 30 million possible connections, with a limited fleet and demand that fluctuates by year, week and even time of day, cannot. As she puts it, "So what we need for purposes is we need a forecast in space and in time." (4:29)
She is careful to place forecasting inside a wider system: "And in that way, demand forecasting, it's a tool for decision making, and it's like one piece in a larger decision making system." (5:21) Supply decisions have to be made well before any bus is on the road, so the forecast for next year feeds straight into network planning.
What drives demand between two cities
To show the modeling questions, Miriam compares a trip from Florence to Rome with a 19-hour journey from Florence to a small town in France. Shorter distance, bigger populations and more tourist attractions all point to Rome. These intuitions become modeling factors: "So the bigger the cities, the closer they're together, the more demand we expect." (7:33)
Many of these factors are not data FlixBus holds. The team has to discover and collect external sources, including open government data and geospatial data from OpenStreetMap. They then break the problem into subproblems along the space and time dimensions, build several models and combine them into one forecasting system that takes both external factors and FlixBus's own decision variables as inputs.
Learning one: the data groundwork
The first lesson is the least glamorous. "Everyone wants to do the fancy machine learning work, but actually the data work, it is super important." (9:25) Miriam illustrates why with OpenStreetMap. Because the data is crowdsourced, it reflects the interests of the people who create it. As the speaker cited, a 2020 survey found that almost 90% of contributors identified as male, and most contributors come from Europe.
That skew shows up in the data. Small German cities are mapped in great detail, while the US is far less detailed. "That is not random." (11:10) Teams must handle such gaps explicitly in modeling, or the forecasts will be biased.
Learning two: an experimental, iterative journey
Miriam offers two metaphors for applied machine learning. The first is a research lab: generate hypotheses, test them, evaluate them, and only add an idea to the system if it improves the forecast. Nobody knows in advance which factors have predictive power or which model type will work. "So it's not that you build one model, but you build hundreds of candidate models." (12:28)
The second is climbing a mountain of unknown height, where some routes turn out to be dead ends and others hide surprises. The biggest surprise was the pandemic. Lockdowns hit the travel industry hard and created a structural break in the data, which the team now has to account for whenever it trains or evaluates models on historic data.
Learning three: taking stakeholders along
The model does not live in isolation. Planners review forecasts and may inject domain knowledge and strategic targets, either into the forecast or into the later supply optimization. Humans stay responsible for the decisions, so trust is the deciding factor: "And if they don't use it, the forecast will not create any value." (14:13)
FlixBus involves its network planners early and makes them part of development. Alongside rigorous model evaluation, the team runs user feedback rounds, interviews and testing phases. These help detect oddities where a forecast might be off, and experts contribute ideas about why. The team also uses explainable machine learning methods to shed light on rather black-box models and show users why a forecast takes a certain value.
Miriam closes by restating her three learnings and noting that the challenges come with large opportunities: "First, the data groundwork is very important, at least as important as the modeling itself." (16:07)
Watch the talk
I know where you wanna go next summer - Demand Forecasting at FlixBus, by Miriam (about 17 minutes): https://www.womentech.net/video/i-know-where-you-wanna-go-next-summer-demand-forecasting-flixbus
About the speaker
Miriam is a Domain Product Officer at FlixBus, where she works with five product teams on automating network planning with data, machine learning and AI. She has a background in statistics and worked for many years as a data scientist before joining FlixBus.
Questions this talk answers
Why does FlixBus need automated demand forecasting?
What factors influence demand between two cities?
Why is data groundwork so important in machine learning projects?
How did the pandemic affect FlixBus's forecasting models?
How do you get planners to trust and use a forecasting model?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Explainable machine learning
Resources mentioned
OpenStreetMap
Cite this talk
WomenTech Network. "I know where you wanna go next summer - Demand Forecasting at FlixBus." WomenTech Network, 2021. https://www.womentech.net/video/i-know-where-you-wanna-go-next-summer-demand-forecasting-flixbus