Consuming, Contributing, Collaborating: Open Source from all sides by Ron Minsky
19:472022
A trading firm's build tool now has more outside contributors than inside
A trading firm with nearly 2,000 employees and no open source business model rewrote parts of Mercurial in Rust, runs its own OCaml fork and publishes around 240 packages to the outside world. Ron Minsky draws on Jane Street's experience to show why that investment pays off for a tech-heavy business. The open question is which way of engaging fits your goals, and what the tooling has to look like to make it work.
Learn four ways companies engage with open source and how to choose the right one
Ron Minsky draws on Jane Street's experience with Mercurial, OCaml, Dune and its exported libraries to show how a company whose business is not open source can still benefit from engaging deeply with open source communities. He argues that this work must align with business goals, that companies should choose their mode of engagement deliberately, and that tooling and developer experience shape whether it succeeds.
“Because I think an important thing to remember is contributing in open source is a repeated game, right?”
“Today, there are more people contributing to Dune outside of Jane Street than there are people inside of Jane Street.”
What you will learn
Align open source investment with your organization's goals so it can be sustained over time.
Choose deliberately between funding maintainers, filing issues, contributing patches, running a fork and launching your own projects.
Plan the tooling for hosting, exporting and importing contributions so friction stays low for internal and external developers.
Decide whether a project is internal-first with a public export or community-first hosted in the open.
Build trust with maintainers by proposing high-quality, well-engineered changes.
Details
Level
Intermediate
Format
Talk
Technical depth
Introductory technical
For
Mid-career, Senior individual contributor, People manager
Length
19:47
Published
June 2022
Questions this talk answers
Why would a company whose business is not open source invest in it?
Open source grows the ecosystem the company depends on, helps recruiting, and gives outside developers something to integrate with, which aligns external tools with internal needs.
When should a company stop relying on consultants for an open source dependency?
When it needs to move faster than consultants can, since they are not inside and cannot see the company's problems, repo or workflows. Jane Street then made changes itself while keeping consultants to generalize and upstream them.
How can a company keep an internal fork without breaking from the wider community?
Land changes internally first, iterate with internal users, then submit stabilized features upstream and argue for them as general additions to the language.
What makes an open source project attractive to outside contributors?
Using the community's tools rather than your own, as Jane Street did by hosting Dune on GitHub, so outside contributors have the same experience as anyone else.
What mindset helps when contributing to open source projects?
Treat contribution as a repeated game, think about what is best for the community, and take care over the engineering quality of every change so maintainers come to trust your contributions.
Speaker: Hi, everybody. So I am here today to talk about open source and in particular what it is like and how you should approach or how you can approach engaging in open source communities kind of from a kind of company and business perspective. And I just wanted to say folks should feel free to throw questions into the chat as we go. I'll be paying attention to that and try and pick off some questions later.
along the way to answer. So the whole talk is basically going to be from the perspective of my experience working at Jane Street. And so I want to spend a few minutes talking about who Jane Street is just kind of so you can. Whoops. Sorry, I lost my video there for a second. Talk about who Jane Street is just so you can get a sense of like where the talk is coming from the perspective that this all comes out of. So Jane Street is a trading firm and the basic business we have is that we are liquidity providers. We are people who are kind of middlemen. If you want to make an investment and you buy some security, then there's a good chance that we are on the other side of that purchase. If you decide to sell a security, there's a decent chance that we are on the other side there as well. And it's a kind of it's sort of like the moral equivalent of working in a grocery store. You know, you buy from one, you sell to the other, you try and make money from the gap between the price you buy and the price you sell. And that all kind of sounds, you know, pretty pedagogy.
But it's like a pedestrian. But it's actually really complicated. We have been doing this for 20 years. We have nearly 2000 employees. We operate in dozens of venues across many time zones and regulatory regimes, lots of different types of securities, tons of different strategies that we run. And the business is, as a result, enormously and intensely technological. We spend a lot of time and effort building software to simplify and scale, to optimize the delivery of data, to kind of make all of the things that we do more scalable and easier and make it easier for people to discover new ideas and build up new trades.
So that is not on the face of an open source business model, right? And so how does open source fit into that? Well, in some sense, open source fits into any serious technological enterprise. The foundations that we work on, the stones that we work on, the stones we stand on, are open source, right? From the operating system on up, we're using lots of open source software.
But that's just kind of the same as everybody else. In addition to that, we've spent a lot of time and effort over the years investing more deeply in various open source ecosystems, including supporting and engaging in pre-existing projects and creating and launching some of our own open source software. And that's really what I want to talk about, both how we've approached that and why we've done it. And I'll do that by walking through a few examples. So to start with, I want to talk about an example where we started out as a mere consumer of the software in question, and then moved on to kind of have a more deeply engaged role over time.
And this is with a piece of software called Mercurial. Mercurial is a version control system, a competitor to Git. It was actually designed at around the same time by almost the same people as Git. Both Git and Mercurial were built by Linux kernel developers. And Git is far more popular and pervasive than Mercurial. But Mercurial has an important niche, which is it's better suited to working with very large repos. And that's good for what we sometimes call a mono repo style workflow, where instead of, as you might do in a kind of common open source context, having lots of little repositories, one for each project that you have, you have one enormous repository that you put lots of different projects into.
And mono repos are useful, especially in large companies, because they simplify the versioning story. Instead of having to think, what version do I need of each project I want to include? Just every revision has a kind of complete and consistent decision as to what is all the software of all your dependencies. And a lot of big companies use mono repos, and a number of those, in fact, use Mercurial for some part of their version control to manage those mono repos, including Google, Meta, and Mozilla all use Mercurial materially.
So we started using Mercurial a long time ago, back in 2005. And as our use of it grew, I mean, we started, we just like downloaded it, it was software, we got it off the shelf, we used it, it was fine, it solved, it solved our problems. But we started doing more, growing the scope and scale of our efforts and writing more software.
And over time, we started hitting more stress points. And we didn't want to build out a big effort to work on Mercurial as a first class thing. So we tried to do something small, we hired some consultants. And this is kind of a common thing. Lots of open source projects have people who've taken on a role as consultants in that part of the ecosystem, who you can hire them to kind of help you out, help you with adoption, and also drive changes that are useful and necessary into the project itself.
And so we worked with those consultants both to advise us on how we approach things and also to kind of help build up more functionality in Mercurial as a whole to support the kind of uses we wanted. And that worked pretty well for a while. But eventually, as we continue to grow and continue to do more, we needed to do to move faster than the consultants could move on their own.
And the limitation for the consultants was mostly that they weren't inside. They didn't see our problems. They didn't see our repo. They weren't directly hooked into our workflows. And so we started getting more directly involved. And so we ended up rewriting significant bits of Mercurial and Rust. Most of Mercurial was implemented in Python. So for speed, we implemented some pieces in Rust. And we did it in a kind of narrow way that was fit to our needs.
But we kept the consultants on, and we worked with them to take that work, generalize it, improve it, and upstream it eventually. Because in the long run, we didn't want to have our own fork of the software. We wanted to be able to make the advances that we needed for our own use case, but also to kind of integrate nicely with open source.
So anyway, that's kind of one example of how we've engaged. Another case where we engaged more deeply, went from kind of starting out as kind of an engaged user and went on to become a primary stakeholder, is with OCaml. So OCaml is a programming language that's not used incredibly widely, but it's one that we're relatively well known for using.
We've used it for almost 20 years at this point. And we were pretty deeply embedded in the OCaml community from the beginning of our using of the language. But as with Mercurial, we started out as fairly naive users, at first just picking up the language and using it over time, submitting bug reports, submitting feature requests, and eventually starting to hire people to work on OCaml internally, to start submitting PRs and trying to get them accepted by the upstream people.
And that allowed us to kind of make a lot of important changes that made our internal lives better. But there was still a lot of friction there, because we went through the ordinary release process, and like, you know, it takes time to get your PR reviewed and accepted. And then there's the next stable release, which happens twice a year. And so it was a pretty long cycle.
And today, our engagement is much deeper. We have a really big team of compiler developers. We have about nine people working on that team. And we have our own fork of the language, where we're constantly making changes. Changes we make now land internally first. And then we iterate and refine and try and make those features better, quickly getting feedback from our internal users.
But we don't want to deviate away and break away from the larger OCaml world. So when those features stabilize, we try and submit them upstream and try and convince people that it makes sense as an addition to the language in general. And today, if you look at any given OCaml release, a large fraction of the improvements, surely not all of them, but a lot of the work that's done there comes from people who work at Jane Street.
So we're now really one of the primary people involved in that ecosystem. So Mercurial and OCaml are both cases where there's a pre-existing software ecosystem, and we were kind of joining into it. But that's not all we've done. We've also spent time taking our internal code and exporting it for the outside world. So if you go on to OCaml.org and do a search for things that are authored by us, you'll see 240-ish packages up there that came from Jane Street.
And that's all in almost a million lines of code. And that code is not a bunch of independent projects that are like sourced on GitHub and are done kind of fully out in the open. What's really going on is we're taking the work that we do internally, taking a subset of that work, and exporting it to GitHub, where other people can see and use it.
And the reason we do that is we get a lot of benefits from working in our internal world, working with our internal monorepo. But it just doesn't make any sense to take that monorepo and export its contents outside. First of all, it's got a lot of stuff that's private stuff that we don't want to export. Also, it would be kind of useless and unhelpful to people to have all of that stuff there.
Instead, what we do is we have a process that takes our internal software as it's developed and periodically, typically once or twice a week, exports it onto GitHub. Just the subset that we want to export broken into a bunch of individual repos for individual projects. And we tie things together in the other direction as well. When a new issue is filed or a PR is submitted, those are imported and kicked into our internal systems so developers can see that something is happening and respond to it in a reasonable way.
So this is all a lot of work. The tooling to hook these things up is not trivial. It's not stuff that people just kind of like have out there kind of in the outside world. It's stuff you have to build for yourself. And you might ask, why do we bother doing it? Because given that it's a lot of work.
And there's a couple of reasons. One is, from our perspective, it's really important to grow the OCaml ecosystem. We get a lot of benefit from all the software and work that's done on the outside in OCaml that we want to be able to use internally. And also from a kind of recruiting and hiring point of view, people here largely work in OCaml.
And we want that to be like an exciting ecosystem and exciting set of code for people to dive into. And so the more vibrant and healthy the community is, the better off we are. And so part of what we did in releasing all the software is to try to, you know, put out more stuff that makes OCaml more useful and encourages more people to use it and grow that ecosystem.
The other somewhat less obvious motivation for it is that by open sourcing all of this stuff, we provide something for other people to integrate with. There's lots of people on the outside who build useful libraries and useful tools and, in fact, extend the compiler in useful ways. And the more of our software that's out there for people to compile against and integrate with, the better those things will be matched to our needs.
And so in some ways by open sourcing all this stuff, we align the software that we build internally more closely with the external ecosystem. And that has a lot of knock-on benefits as well. Okay. So that's a case of like taking stuff that we have and like exporting it. But it's a clear like first class, second class thing.
The first class development experience is inside of our walls. And there's this kind of second class get export, which is like pretty good, but limited. There are also cases where we've decided to really go out and try and build whole communities around pieces of software. And maybe the most interesting example of this is Dune. So Dune is the, these days is the preeminent build system for OCaml.
So if you have like a big OCaml program and you want to do all the compilation and linking and pre-processing steps that are required, you can use Dune for doing that. And it gives you a nice, flexible, efficient system for that. It was developed at Jane Street. But we made a decision relatively early on to organize it as a kind of community focused project, which in part meant picking the community's tools over our own tools.
So instead of hosting it internally and exporting it to GitHub, we host it on GitHub and bring things, bring it internally periodically, right? So it changes where the frictions are. And it means that if you want to work on Dune and you're in the outside world, you have basically the same experience for doing that work as anyone else does.
And I think this was really important for making Dune a place that was attractive for other people to come and contribute. And it's really worked. Today, there are more people contributing to Dune outside of Jane Street than there are people inside of Jane Street. So I think of that as a pretty big success. And I think it also helped in the adoption.
Like if you look at OCaml's package manager, you'll see a larger and larger fraction of the packages over time are built using Dune. And that has other benefits. When we want to import someone's project internally, the fact that it's built in Dune, which is a build system that we understand very well, makes it way easier for us to pull it in to our internal systems.
Okay, so those are like three different projects, or actually four different projects, four different stories of things that we've done, open source work that we've done. They're not the only ones. There's a bunch of other neat things that you could look at. One of my favorite examples is a project called Magic Trace, which you go to magictrace.org.
You can see some beautiful analysis of exactly what Hello World does when you run it in a kind of elaborate detail that highlights that Hello World is way more complicated than you might ever have imagined. And that's another example of a piece of software that we've built as a kind of open source first piece of software. But anyway, these are examples, but there's lots more stuff that we've done over the years to kind of engage with the open source world.
So those are some examples. What are the takeaways from all that? What are things to remember? So the first one that I think is really important is if you're going to do open source work in the context of a business, in the context of a larger organization, I think it's really important to align that work with the kind of needs and goals and requirements of the organization.
So I want to be clear. Personally, I love open source software. It's kind of what I cut my teeth on, how I learned to be a software engineer. And I get a lot of personal satisfaction from doing stuff that contributes in this kind of public way. And both in terms of like contributing to existing projects and open sourcing our own stuff.
I find it enormously satisfying. But building open source software and doing it well and sustainably is really hard. And, you know, in some sense, the excitement that you might have around open source can get you a certain distance. But I think it's hard to justify a large and sustained level of investment unless you really think about how that open source work can line up with the goals of the company.
And I think part of what I think is interesting about the story here is even if your business model isn't fundamentally about open source, it's still the case that open source and active contribution and collaboration with open source communities can really benefit the business in a kind of useful and tangible way. The other thing that I hope these examples highlight is that there's a bunch of different ways that you can engage, right?
You can fund people to work on open source projects. You can communicate on forums and submit bug reports and write documentation. You can write your own patches and you can create your own freestanding open source projects to contribute to the world. And you should think of which approach you want as part of the open source strategy you're picking.
There are a ton of different choices and which choice you pick matters. It affects how you set things up. It affects what you can achieve. And so you really want to be intentional when you're making that choice. And the other thing is that, that's maybe less obvious, is that the tooling that you use matters an enormous amount.
Like every time you make a choice about where you're going to host it and how are you going to manage it and how are you going to manage contributions and things like that, there's a bunch of tooling implications. And that affects the friction, the friction for your internal developers and the friction for external people who are going to participate.
And so again, when you're thinking about what's your strategy for how to engage in open source, stopping and taking a deep breath and thinking, what is the user experience, what's the developer experience going to be like for people who are participating? And what should we do on the tooling side to make that as good as possible for the approach that we're taking?
I think it's a really important kind of fundamental question you should be asking early on in the process. Okay, so that's basically what I wanted to talk about. We have a few minutes left, and so if folk have questions, I would be happy to answer them. I see a lot of hellos, but not much in the way of questions, I think.
So yeah, if anyone has stuff to say, I would be happy to answer. Okay, and while we're doing that here, I'm going to... Ah, never mind. I was going to pull up a different one of our projects called Magic Trace to show people, but I think I don't have a moment. I think I can't quite do that now with how I have things set up.
All right. Ah, here. What are some best practices for open source contribution? That's a good question. Like, I'm not sure I have like a concise set of best practices, but I can say something about mindset, which is, I feel like when you engage in with open source communities, I think it's important to really think hard about what is best for that community.
Because I think an important thing to remember is contributing in open source is a repeated game, right? You're going in, you're not just trying to get one patch in or change one thing. You want people over time to trust and respect the things you're doing so that like when you go in to make contributions, people are actually willing to accept them.
So I think, like one thing I found to be really important is to not just like throw stuff up there without thinking, but to really take effort to think hard about the engineering and the quality of the changes you propose. And do that in a way that, so that people on the side of the kind of open source project come to think of your contributions as high quality things that they would be happy to accept.
Okay, and another question from you is, in terms of materials I can point you to. So there's a few things. So we actually have a tech blog that covers a lot of our work and is more focused on some of the open source stuff. So I think if you go to blog.jamesstreet.com, you'll see a bunch of posts that cover all sorts of different aspects of the work we do, but a lot of focus on the kind of open source side of the work.
You can also see actually all the concrete stuff we have. There's another website opensource.jamesstreet.com that has links to some of the major projects. And then github.com/jamesstreet is just our kind of our organization on GitHub that has all that there. All right. Well, I think we're at the end of our time. Thank you very much.
Transcribed automatically; names checked against the speaker record. Select a time to jump there.
Article based on this talk
Why a company without an open source business should still invest in open source
Ron Minsky uses Jane Street's work on Mercurial, OCaml and Dune to show the different ways a company can engage, and why tooling decides the outcome.
In short
A company does not need an open source business model to benefit from open source. Drawing on Jane Street's experience, Ron Minsky shows four ways a firm can engage, from paying consultants to building community-first projects, and argues that the work lasts only when it is aligned with business goals, chosen deliberately and supported by good tooling.
Key takeaways
Align open source investment with your organization's goals so it can be sustained over time.
Choose deliberately between funding maintainers, filing issues, contributing patches, running a fork and launching your own projects.
Plan the tooling for hosting, exporting and importing contributions so friction stays low for internal and external developers.
Decide whether a project is internal-first with a public export or community-first hosted in the open.
Build trust with maintainers by proposing high-quality, well-engineered changes.
A trading firm built on open source
Minsky opens by describing Jane Street as a liquidity provider: a middleman that buys from some market participants and sells to others. He compares it to a grocery store, but stresses that the business is complex and "enormously and intensely technological", operating across dozens of venues, time zones and regulatory regimes. As he put it, open source is not obviously part of that model, yet every serious technology business stands on it. "The foundations that we work on, the stones that we work on, the stones we stand on, are open source, right?" (2:05)
Beyond simply using open source like everyone else, the firm has invested in existing projects and launched its own. The rest of the talk walks through how.
From consumer to contributor: Mercurial and OCaml
The first story is Mercurial, the version control system Jane Street began using in 2005. Minsky explains that Mercurial handles very large repositories well, which suits a monorepo workflow where every revision fixes a consistent set of dependencies. As the firm grew, it hit stress points and first hired consultants who work in that ecosystem. That worked for a while, but the consultants were limited by distance. "And the limitation for the consultants was mostly that they weren't inside." (5:33)
Jane Street then rewrote performance-critical parts of Mercurial in Rust, narrowly for its own needs, and kept the consultants to generalize and upstream the work. "Because in the long run, we didn't want to have our own fork of the software." (5:58)
OCaml followed a similar arc, but went further. The firm moved from bug reports and feature requests to hiring people to submit pull requests, and eventually to a compiler team of about nine people maintaining its own fork. Changes land internally first, are refined with internal users, and are proposed upstream once they stabilize. Minsky says a large fraction of the improvements in a given OCaml release now come from Jane Street engineers.
Exporting internal code, and why it is worth the effort
Jane Street also publishes its own code: Minsky cites around 240 packages on OCaml.org, almost a million lines. Rather than open sourcing the whole monorepo, a process exports a selected subset to individual GitHub repositories once or twice a week, while issues and pull requests flow back into internal systems. He admits the tooling is not trivial and has to be built in-house.
He gives two reasons for doing it. A healthy OCaml ecosystem helps both the software the firm uses and its recruiting, since staff largely work in OCaml. Less obviously, published code gives outside developers something to build against. "And so in some ways by open sourcing all this stuff, we align the software that we build internally more closely with the external ecosystem." (11:20) Still, he is candid that this model is lopsided: "The first class development experience is inside of our walls." (11:38)
Dune: putting the community first
For the Dune build system, Jane Street reversed that arrangement. It hosts Dune on GitHub and brings it inside periodically, so outside contributors work the same way as anyone else. Minsky calls the result a big success. "Today, there are more people contributing to Dune outside of Jane Street than there are people inside of Jane Street." (12:49) Wider adoption also pays back internally: projects built with Dune are much easier for the firm to import. He mentions Magic Trace as another open-source-first project.
Three lessons for any organization
Minsky closes with takeaways. First, alignment: he loves open source personally, but enthusiasm only goes so far. "But building open source software and doing it well and sustainably is really hard." (14:58) Second, there are many modes of engagement, from funding people and writing documentation to launching freestanding projects, and the choice should be intentional. Third, tooling. "And the other thing is that, that's maybe less obvious, is that the tooling that you use matters an enormous amount." (16:07) Hosting and contribution choices determine friction for internal and external developers alike, so the developer experience deserves attention early.
In the Q&A, asked about best practices, he offers a mindset rather than a checklist. "Because I think an important thing to remember is contributing in open source is a repeated game, right?" (18:11) Thoughtful, high-quality changes build the trust that gets future contributions accepted. He points viewers to the Jane Street tech blog, opensource.janestreet.com and the firm's GitHub organization.
Watch the talk
Consuming, Contributing, Collaborating: Open Source from all sides by Ron Minsky, about 20 minutes: https://www.womentech.net/video/consuming-contributing-collaborating-open-source-all-sides-ron-minsky-0
About the speaker
Ron Minsky is the speaker named in the session record, which does not include a role, company or bio. In this talk he speaks from his experience working at Jane Street on its open source efforts.
Questions this talk answers
Why would a company whose business is not open source invest in it?
When should a company stop relying on consultants for an open source dependency?
How can a company keep an internal fork without breaking from the wider community?
What makes an open source project attractive to outside contributors?
What mindset helps when contributing to open source projects?
Drafted with AI from the talk transcript. Editor review pending.
Frameworks and ideas mentioned
Monorepo workflow
Internal-first export vs community-first hosting
Resources mentioned
blog.janestreet.com
opensource.janestreet.com
github.com/janestreet
magictrace.org
OCaml.org
Cite this talk
WomenTech Network. "Consuming, Contributing, Collaborating: Open Source from all sides by Ron Minsky." WomenTech Network, 2022. https://www.womentech.net/video/consuming-contributing-collaborating-open-source-all-sides-ron-minsky-0