The Real Costs of Open Source Sustainability
In March 2020 I presented “The Real Costs of Open Source Sustainability” to the CTO School meetup in Melbourne, Australia. This is a transcript of that talk.
You can watch the video of the talk and download the slides here on Internet Archive.
Click on the slide images in this transcript for a larger version of the image.
In 2016, a lovely individual named Nadia Eghbal released this: Roads & Bridges: the unseen labor behind our digital infrastructure. This is a 142 page report from the Ford Foundation, and it shines a light on the woeful state of many critical open source projects and their maintainers. | |
Now, prior to Nadia’s report, very few people outside of the immediate free and open source software world knew how few people maintain the software that underpins a very large part of the internet and the services that run on top of it. | |
However, after her report a whole lot of the software industry suddenly was vastly aware of just how bad things were. Speficically, they were aware of how bad things were with the projects on which their companies relied. Most of them were shocked, but that’s because they hadn’t been participating in free and open source software or contributing back. Had they been part of their communities they would have seen what the problems were, and actually they wouldn’t have been problems to begin with. But they weren’t. Whatever. They now know. Water under the bridge. We can now move on. | |
Being technologists, and being mostly technologists in the San Francisco Bay Area, they started doing what they’re most comfortable doing: making software to make money. | |
There has been a flood of newly launched endeavours around getting funding into the hands of open source maintainers. What you see on the screen here are just a few of them. There are one or two non-profits, such as the Mozilla MOSS grants. That’s from a non-profit, the non-profit side of Mozilla, but most of these are commercial and/or VC funded. | |
I do travel a lot. Like I said, I’m getting over something I picked up in Brussels. I’ll be in Singapore soon and Tahoe and various other places. So I travel a lot, and I get the opportunity to talk to a lot of folks. Because I am in this free and open source software world—I do business strategy, by the way, specifically around how your company can be more successful through free and open source software and how you can trounce your competitors. I’ve been in free and open source software for about thirty years now, so I’m pretty well known in that world. So people come up to me in all of these events that I’m at, and they say, “Vicky! I’ve got a great idea about how to get money into the hands of free and open source software maintainers!” They keep coming up with all of these ideas and running them by me, and some of them are kind of bizarre. A lot of them are kind of bizzare, but thankfully they haven’t launched yet, but they’re all in flight. It’s kind of like there are these software developers who think, “Well, they won’t let me use blockchain at work…so I’m gonna use it for this!” They often stray into that sort of Jurrasic Park space, where they’re so preoccupied about whether they could they don’t think of whether they should. So that’s a lot of this stuff that I’m hearing. | |
But however complex the solutions are that people come up with, at the end of the day it’s just people throwing money at the problem, to get money into the hands of open source maintainers. That’s all they want to do, just throw money at it. When you stop to think about it, it actually makes a fair bit of sense from a cultural perspective. Because we are, both by culture and experience, conditioned to equate money with stability, and money with assistance. Whenever there’s a crisis—say you’ve got a couple of little fires burning—whenever there’s a crisis people reach for their wallets. Whenever there’s a tsunami. Whenever there’s an earthquake. Whenever there’s something horrible, everybody reaches for their wallets. They want to do something about it. They feel otherwise powerless to lift a finger and to help these people in need, but they have to do something, so…they think, “let me at least donate money.” Because people are fundamentally good and people do want to help, when you get right down to it, and giving money makes them feel as though they’re doing that. But nobody really follows through after that. They don’t follow through to see whether those donations are being used, whether they’re eaten up by administrative fees, whether they went to a reputable group anyway, whether they were even needed. They’ve given money, they get that little dopamine hit of, “yay! I helped!”, and then they move on with their lives. I mean, it’s not bad to give money, to help, but people don’t really think it through, they just default to “money::good”. | |
In this specific situation, in many cases, throwing money at the open source sustainability issue is a misplaced good intention. Now, I do talk to a lot of people and I don’t only hear from people with really goofy-ass startup ideas. I also hear from project maintainers. That’s from whom I hear most of the time, because I’m in the open source world and I’m “open source advocate without portfolio.” It’s not like I identify as an Apache person or a Drupal person or, you know, a Python person or something like that. I work across all of the communities, so I hear from all of the communities. Maintainers will come up to me and go, “Omigosh, Vicky, it’s so great! People started giving us money through OpenCollective and now we have all this money!” and I’m like, “Wow, congratulations, good on ya mate…what are you gonna use that money for?” They weren’t expecting the money, so of course they don’t have plans for it. They weren’t expecting the money, so they don’t realise that you can’t simply put that money, for an entire community project, into your personal bank account. They weren’t expecting the money, and don’t don’t realise that there are tax implications for that money. I don’t know about ATO, but the IRS? They don’t look kindly on that sort of stuff. When I tell them this, and I ask them these questions like, “what are you going to do with it?” and “do you have an accountant to deal with the taxes?”, they kind of go white. They’re, like, “wait…this isn’t just free money?” And that’s a problem, because nobody bothered to ask them whether they wanted the money, let alone whether they needed the money. We’re just conditioned, again, to think, “money::good,” so we throw money at the open source problem. | |
This is despite Nadia’s report’s very clear focus on a multi-faceted approach to infrastructure maintenance. At no point point does she say, “simply throw money at the problem.” It’s a 142 page report. If it could be summed up by that line then, lemme tell ya, it would’ve been a helluva lot shorter. But people in our industry, in software, to us “sustainable” equals “funded,” and that’s not necessarily the case. | |
As is often the case in software, it feels as though we’re reinventing the wheel. We do this all the time. Because our problems are “special.” No one else across all of humanity or any other industry could possibly have had the same kinds of problems we have and in open source we’re particularly special. Rather than seeing how anyone else has done this, we’re just doing it our way. Which is a massive problem because there is prior art out there for dealing with sustainability. If the software industry—and open source in particular—but if the software industry, who are building their businesses on free and open source software… If you want your supply chain to be very stable, you have to look for prior art, and look at how you can do that. | |
What we need to do, surprisingly, is to look at the corporate world. This is something that most people don’t realise, something called Corporate Sustainability Planning. Specifically, we should look at this report. | |
It’s called Our Common Future: The world commission on environment and development, but it’s better known by this title, The Brundtland Report. This is a report published by the United Nations in 1987. They got a whole bunch of folks together, and the ones who initiated this were actually the corporations. Back in the 80s, they looked around at the world—how many of us were alive in the 80s aside from just me? probably most of us? how many of you remember the 80s? Big hair? Aquanet?—they looked at the world and what was going on: the cold war, nuclear scares, acid rain, ozone layer. They were like, “Wow…this is pretty rough. If someone doesn’t do something, we’re gonna lose all of our customers. Maybe we should do something about it.” So they actually went to the UN and were like, “What can we do about this?” The UN got together this commission that took three years. Ethicists and economists and corporate folks and scientists and politicians. They got a whole bunch of big-brained sort of folks together to talk about this. They came up with this report. What the report discovered is that it’s absolutely impossible to talk about growing an economy, of having a successful and sustainable corporate world, without also talking about sustaining the environment and everything inside of it. Now, again, this was kicked off by corporations, highly motivated to make it all about the money, and what they found was that you can’t do that. So the Brundtland Report focusses on environmental conservation and on the corporate responsibilities for human sustainability, and how these things are absolutely critical for corporate success, and not simply because you won’t have killed off all of your customers. | |
The Brundtland Report starts with something that none of these open source sustainability things have done, including SustainOSS, an entire conference all about open source sustainability. They never bother to define, “what do you mean by sustainability?” The Brundtland Report does that, and it does it of course for its specific audience: corporate suits and bigwigs and people like me. I’m Director of Open Source Strategy at Juniper Networks. It defines it as “development that meets the needs of the present without compromising the ability of future generations to meet their own needs.” It’s a very generalisable definition that I think we all should try to work with in open source software. | |
It recognises the need for the balance between economy and ecology, rather than focusing on the money. It spends a lot of time talking about ecosystems. It’s important to note that they don’t consider ecosystems simply to be things like trees and water. There are a lot of different elements to an ecosystem, and ecosystems are actually nested. So you have an ecosystem within an ecosystem within an ecosystem. For instance—I always say it wrong—when you look outside at the Yarra, it’s its own ecosystem. It has its own combination of plants and pollution and people using it for boating and various things like that. But it’s also part of the larger ecosystem of the Pacific Ocean, which is part of the larger ecosystem of all the oceans, which is part of the larger ecosystem of the Earth… It’s ecosystems all the way down, but each one can be considered separately. Each ecosystem has all of these constituent parts. If you do something to upset the balance of fish life in the Yarra, what’s that going to do to the rest of the ecosystem? It’s actually going to impact the entire thing and really screw up the river. If you do something to impact the plant life… Let’s say somebody starts dumping some sort of fertiliser and the plant life just takes off. That’s going to impact the entire ecosystem, and in a really negative way. Ecosystems like balance. All of these moving parts are intertwined. You can’t affect one without looking at the impact it has on all of the others and the greater ecosystem. | |
What the Brundtland Report found is that despite everything else and all the many moving parts, these are the three most important bits for corporate sustainability planning:
If you want corporate sustainability, if you want a better Earth, a better environment, a better ecosystem, you have to focus on these three things. Notice: none of these are “shareholder value.” None of these are “money”. They’re all about the bigger picture. Furthermore, in the words of the Brundtland Report, these are “interlocking crises.” Each is an individual crisis, but you can’t work on just one. What their studies found—and, again, they took years to do this—is that you can’t focus on just one, because you get absolutely no results. You have to focus on all three. It’s an all or nothing proposition. Either you focus on povery reduction and wealth redistribution and… I mean, you can’t just focus on gender equality and then say, “Yay! Diversity!” and think you’re going to have a corporate sustainability plan. It doesn’t work like that. So they’re interlocking crises. | |
The Brundtland Report has led to something called the Corporate Sustainability Movement. If you want to learn about it, MIT Sloan has done a lot of work on it and this is a good starting place. By the way, these slides are already available, I’ll give you a URL for that later or you can just look in the footer. If you want to learn more about corporate sustainability planning, this is a good place to start. It has really taken off in a lot of ways. Almost every Fortune 500 company will have a corporate sustainability plan in some way, including us at Juniper. | |
There are many elements to this, and they all have to be looked at simultaneously. None of them are simply throwing money at the problem. All of them do involve some sort of capital investment in order to work. You can’t redistribute wealth if you don’t get the wealth involved in some way, for instance. | |
None of these are simply throwing money at the problem and they all require a great deal of work. Planning for this and implementing your corporate sustainability plan is really difficult. It requires culture change. It requires product changes. It requires a lot of work. So…why would you do that? As a company, why would you spend all of that time? Is it just altruistic, because it’s the right thing to do to take care of the people and the environment? Well, yeah. I mean, some companies do that. Some are really good at that. Patagonia is amazing for their altruism as far as the environment and humanity. | |
But…good thing for the rest of those Fortune 500 companies, this is also really good for your bottom line. What studies show repeatedly—and almost all of them are somewhere in MIT Sloan studies, so you can do a little [citation needed] and just go to MIT Sloan—what it shows is that a well-executed corporate sustainability plan is very very good for profitability as well as for the planet and for humanity. | |
There are all these benefits that you get for a well-executed corporate sustainability plan. These are above and beyond, again, a healthier planet and being better for humanity. One thing that’s not listed on here is that recent studies are showing that you get better investment in your company if you have a well-executed corporate sustainability plan. Just about everyone who’s not a venture capitalist is going to look at that and think, “Wow, you’re looking at the long term. I would like to get a higher return on my investment over the long term, so therefore I’m going to invest in your company and not that other one.” It’s actually really good. | |
Now, I’ve been banging on about corporate sustainability for a while. Aside from the fact that it’s just really cool—I obviously do corporate strategy so I, like, dig this sort of stuff—but, it’s very relevant. I’m not buying the lede. We can learn a lot from the corporate sustainability movement. We really need to do the open source way, which is to build upon and iterate upon the work of those who came before us. That’s what we do in open source, but we’re not doing that. It’s really a shame that we’re reinventing the wheel. | |
We can learn a lot from the corporate sustainability plan, starting with the fact that it’s just not about the money. It’s about so much more than that. Let me be perfectly clear here: I am 100% for paying maintainers. Yes, absolutely. If they’re providing value, you should find some way to give that value back, to show them that you value what they’re providing to you. But…monetary support only is not going to create a sustainable free and open source software ecosystem. | |
Just as with corporate sustainability, there are interlocking crises here. Each one is part of the larger crisis in the open source ecology, a lot of which I’m not even going to go into here, because it’s a really big problem. Like with corporate sustainability, you can’t focus on just one—funding, for instance—and then call it good. You have to focus on all of them. | |
And, just like with corporate sustainabilty, it’s not about the money. Here are three elements of open source sustainabilty:
These are interlocking crises. As a company, if you want the projects on which you have built your company to be sustainable for the long term, so you’re not totally screwed and go Equifax on us… If you want these to be sustainable you have to focus on all three of these items. As a company, it is your responsibility. It’s malpractice not to make sure that your supply chain is sustainable. That’s bad business practice. | |
You have to work on all of these, and let’s go into them a bit more, starting with contributing back. Companies that use free and open source software but don’t contribute back in some way, they’re degrading the longevity and the success of free and open source software. These companies have been around since the very beginning of free software. Way back in the day, nearly 40 years ago, Richard Stallman nailed his articles to the MIT door, saying, no, I will not use that printer driver. As things were released, people focused on the “free,” because in English it’s a very ambiguous word. “Free? Oh, I don’t have to pay for that! Yay!” Unfortunately, it doesn’t make it sustainable if you take without giving back. We have a great deal of that going on in free and open source software and the corporate world. They’re not contributing back to the projects that have made them successful, and that leads to unstable projects that aren’t going to be maintained. It leads to maintainer burnout. | |
While money is almost always welcome by projects, whether they know what to do with it or not, there are other things that are equally needed. Contributing back can take so many forms. Tiffany Farriss, of the Drupal Association, laid this line of me last year. She’s a great gal, you should get to know her. “Time, Talent, and Treasure.” These are the three things that you have for contributing back. It doesn’t have to simply be treasure. You can volunteer your time and your company’s time, by doing things like security audits, or accessibility audits, or redesigning that piece of software so it’s actually usable, adding documentation, standardising an API, that sort of stuff. You can do that. You can also give your talent. You can do things like host itinerant free and open source software specialists in your office, thank you Fastmail very much for this. And yeah, you can throw money at the problem, you can donate money as well. It’s important to recognise that each one of these—time, talent, treasure—each one is an equally valid way to support and contribute back to free and open source software. Some people, for instance, can’t afford the treasure but they might be able to give just a little bit of time and that’s OK. As a matter of fact it’s better than OK, because contrary to what it might feel like when you can’t make your bills, there will always be more money. It will happen. But there will never be more time, so when you spend your time with a project it’s the most precious gift you can possibly give because you ain’t never getting it back. That’s really valuable, and it’s something that a lot of projects are in desperate need of, far more than your money. | |
For example, in my professional life I’ve worked with maintainers of vital free and open source software infrastructure, some of which was mentioned in Nadia’s report. These maintainers worked on these vital projects full time for the company that employed us. I know how much these maintainers were being paid—how much was in their take-home pay, what their bonuses were—because I was their direct supervisor. No amount of money, not a single red cent, could make up for the fact that these people were spending 60 to 80 hours a week on these projects and they had been for years, even before they joined this company, even before anyone was paying them to do it. They worked so hard, and they got so much abuse from their users. Now, I gladly would have paid them more money if I’d been able to, but I knew they were making Silicon Valley level salaries, and living in farflung places of the world. They were doing quite well from a salary point of view, but that’s not what they needed. They didn’t need money; they needed help. That’s what we were working on getting them, until the company laid us all off. | |
The next element in ensuring sustainable open source ecosystem is human and environmental diversity. Contributing back to the projects helps with this as well. For the sake of free and open source software, what we need is to get more and more varied people involved. It not only provides resources for additional development, and makes sure that things are sustainable and continually maintained, it also provides more innovation and more stability. A growing body of research, starting with this one from Harvard Business Review, shows that diverse teams consistently and hugely out perform homogeneous teams. Massively, and it’s really impressive. It’s not just this study, it’s lots of them now coming out and saying this. And not just in your business, where it really adds a lot of business value, but it’s also in open source. Diverse perspectives encourage more and better communication in order to be successful. That better communication and those diverse perspectives also lead to a lot more ideas being flung around, so a lot more innovation happens. This can only occur when you have a lot of people from different backgrounds, from different walks of life. You can’t do this otherwise. It works with open source as well as anywhere else. Most importantly, more people equals less risk that that maintainer is going to burn out, table flip, and walk away. This happens all the time, because people can only be pushed so far before they just have to leave. | |
Now diversity doesn’t simply mean gender or race, and I’m positive that’s where everyone went when I said the word “diversity,” because that’s all we talk about with diversity. “Oh, well, let’s get more women.” It’s really a much bigger picture than that. It means geographic and language diversity as well. Free and open source software is used all over the world. I’m based in Portland, Oregon and here I am in Melbourne. Last time I gave this talk it was at CERN in Geneva, where people from all over the world come to do the most incredible science. This is used everywhere, but the majority of documentation and support for a free and open source software project is in only one language: the language of the primary maintainer. And let’s be perfectly honest here, it’s almost always English. That works for me and that works for you, but that doesn’t work for billions of people all over the world. If you have someone on your team who speaks another language—Mandarin or Urdu or Spanish—and they’re using a free and open source software project, encourage them to participate in that project and be there as the language specialist. What you’ve just done is given that project on which you rely for your company this huge open door to millions of potential new contributors. If you can get new contributors from India or China or the Latin American world? Nigeria, by the way, new studies are showing that Nigeria has the most new contributors to free and open source software projects. You can capture that, just by embedding one person from your team on that project, to make people who speak that language feel welcome. They’ll stick around, and that’s really valuable and that’s something that we all have the power to do as companies. | |
There’s more to diversity than simply the human participants. Like I said for ecosystems, there are the cute little kangaroos that are hopping in front of your car and getting in your way on your drive to work, but you also have the trees and you have the air and you have the water. There are all of these other elements to an ecosystem that you need to look at. For free and open source software, that’s the tooling. I do business strategy. Before I joined Juniper Networks—I’m currently in a monogamous relationship with Juniper Networks—before that I was freelance and I helped a lot of companies be more successful through free and open source software. What you find, and what everyone talks about when they do talk about free and open source software from a company perspective, is they choose FOSS solutions because they don’t get vendor lock-in. A single vendor is a single point of failure. Therefore, if I’m using an open source solution—say OpenStack—and I get really pissed off at Red Hat, I can leave and I can go to another OpenStack provider, IBM or someone like that. I can do that because all of the APIs are the same and everything just works (sorta). I have the ability to do that at least. You avoid vendor lock-in. Vendor lock-in forms a monoculture. A monoculture is a business risk, and it’s something that you would like to avoid. In open source software, what we have right now is a growing monoculture. Let’s start with GitHub. I love GitHub. GitHub is great, but GitHub is not open source, and by that I do not mean its codebase. I don’t care what they do with their codebase. It’s not the end all be all of open source software. On those few times a year when they have infrastructure problems, what happens to open source software? What happens to all of those npm modules that your company relies on, that you’re trying to install from GitHub? You’re screwed, because it’s a monoculture and you can’t get those modules anywhere else unless you’re doing the smart thing and have some sort of archive internally, which you should do for compliance reasons anyway but that’s a different conversation. You’re screwed, because GitHub is a single point of failure and it’s not the only one in free and open source software. All these projects that have money, what are they going to do with it? They have to go somewhere. Those things are called fiscal sponsors. We have relatively few fiscal sponsors in free and open source software, and the one that everyone thinks about is biggest gorilla in the zoo, and that’s the Linux Foundation, which is growing by leaps and bounds every single year. A larger and larger percentage of the infrastructure and the projects on which your company relies are under this single umbrella. Now, whether they’re doing a good job or not is beside the point (just like with GitHub, who does a great job). It doesn’t matter. That’s beside the point, because this is a single point of failure and that’s a big problem for you and your company. It’s something that you should keep in mind as you’re using these free and open source software projects. We need more diversity in fiscal sponsors, in tooling. We need to do this, otherwise it’s very bad for free and open source software and it’s very bad for your companies. As an open source participant, rather than an onlooker, your company is in a position to do something about this. | |
The third thing we have to look at, the third element of open source sustainability is community safety. It’s not possible to build a diverse contributor base where people don’t feel safe. They’re just going to be, like, “Peace out, I don’t need this crap,” and then they’re going to leave. It happens, continually and in droves and it has for decades. Belittling comments, personally directed criticisms, sexualised language and imagery and references, bigoted, racist, sexist comments. All of these things happen in free and open source software, and it drives people away. It undermines the trust required to build a sustainable community. Without a sustainable community, you’re not going to have a sustainable project, and then the project on which your company relies goes away. Now trust is very important in any team, but it’s especially important in a team of open source contributors. They’re often globally distributed, and they may never meet each other. If you have behaviour in your community that undermines that trust, you will never be able to build a healthy community that will support the safety and the sustainability of your project. | |
These actions scare people away from contributing, or if they’re already a part of the project they’ll go away completely. As an open source participant, your company is in a position to do something about this. It can look at these actions in the community, and it can take some steps to stop them. It doesn’t have to be some big, dramatic, makes the front page of Hacker News sort of thing. It can be a simple, “You know what? We don’t do that here.” Just that can often be enough to start to set new community norms. The graph that we see here is a bit old, 2018 (I should update it). It shows the growth of the Python programming language. Years ago, led by Django…if you don’t know Django, it’s the Python frontend webdev stuff. It’s the Ruby on Rails of Python. I’m sure they wouldn’t like that, but whatever, that’s what it is. Python started looking at themselves, and that’s really hard. You ever look at yourself? And then you just go, “Shiiiiiiiiiit, oh man I am so broken.” That’s what they did. It started with Django and filtered into the whole Python programming language. They realised that the way their community was, was actually holding back the language from adoption. They started taking actions to fix that, right from the very stop. Guido has since stepped down as Benevolent Dictator for Life. Thankfully he didn’t die, so that “for Life” was apparently a big joke. So he stepped down, but at the time, from the very top, he said, “No. We don’t do that here.” While there are obviously other factors to this massive growth of Python, back here, when they started having these conversations… I’m almost positive, though it’s impossible to prove, that we wouldn’t see the continued growth. This line is your standard Bay Area VC up-and-to-the-right hockey stick. This is what Python is doing and is going to continue doing. A large part of that is because of the community. Rust looked at Python and said, “I like the cut of your jib. I like what you’re doing with your community. I think we can iterate on it and do better.” Which is, of course, the open source way. Rust, from the very beginning, has been a very welcoming community. It’s even a welcoming language. If you make a mistake, it tells you, “Oh, hey, look here. Here’s exactly what you did wrong and here’s how to fix it.” It’s brilliant. The usability of Rust, aside from its other technical merits, is far superior to most other languages. It’s really great. Swift does a similar thing. We can point back to Python here and say, “Yes, they did it right,” and it’s influencing a lot of projects now. | |
One way your company can help ensure community safety is to restrict your contributions to projects that have a code of conduct. The one on the screen here is the Contributor Covenant. It’s very popular. It’s under a Creative Commons license so you can fork it, use it, live it, love it, which is the open source way. Having a code of conduct is one thing, enforcing it is another. It’s very difficult to enforce codes of conduct. Because I’m a woman in technology, people come to me to fix their code of conduct problems. This is part of the emotional burden that you have to bear as a woman in open source. I unfortunately have had to deal with a lot of open source code of conduct enforcement, and helped a lot of projects go through this. What you learn is that people just don’t know how to do it. Also, as a manager—I lead teams at a VP level—as a manager, one of the hardest things you’re going to do is teach your team how to have critical conversations, and those difficult conversations. It’s the same sort of thing with code of conduct enforcement. It’s very difficult, but you can learn how to do it. One of the ways your company can contribute and give back to projects is by sponsoring that training. There are people out there who do that, and I’m happy to recommend them for you. Sage Sharp at OtterTech; they do a wonderful job with these trainings. They can come into any open source project and say, “OK, here’s what we’re going to do, here’s our training, we’re going to do it on Zoom, etc.” It’s great. You can sponsor that as a company, to help make the community of the project you rely on more sustainable. That helps your company. | |
So these are three elements of open source sustainability. | |
In light of all of these things, what actually is the cost of open source sustainability? What’s your company’s open source sustainability plan going to look like? | |
I don’t know. There is undoubtedly a very short list of people at your company who get to tell you how to use your time, talent, and treasure. Because I’m currently not freelance, I don’t think I’m on that list. So I can’t tell you what your sustainability plan will look like, but I can tell you that there’s no one right way to do this. There’s no one approach to do this. That’s because your company has its needs. Your company has its open source supply chain. That those projects in that supply chain have their own specific needs. So anyone who comes telling you, “Here read this book, this is the one way to do it,” is full of shit. There is no one way to do it. | |
There are, however, some places where you can start, and some questions you can ask yourself. Here are three steps. They look simple, but of course there’s that man behind the curtain pulling all the levers… It’s really complicated for each of these. Your company will have to do the very hard work of figuring out what it means for them. | |
Starting with figuring out what projects are strategically important for you. Do you actually know what your open source supply chain is? For the vast majority of companies I’ve worked with, the answer is “nope, I got no clue what we’re using.” Oh? Did you know that you have AGPL’d software in there? And now you have to release all of your code because you didn’t bother to do license compliance? “Whaaaat??” You have to look at your software supply chain. It matters. It doesn’t only matter for compliance, although that’s important because intellectual property law is nothing to mess with. It also matters because you can look at it and go, “Well, which of these pieces are important to us? If they were to go away, which ones would give us a helluva hard time? Which of these pieces are weak, and we have to find a way to support them?” The two answers to these might not be the same, and if they are they same then, wow, you’re in a world of hurt. You really need to start doing something about that now. But you’re not going to know the answers until you look at your open source supply chain. Figure out which ones are strategically important. Don’t just assume you’re going to support them all, because you’ll be screwed. There are millions and millions of these. One study I saw yesterday said there are 28 million open source projects. I think that number’s actually low. And, frankly, let’s be honest, at least 14 million of those are in npm. So you’re going to be screwed if you want to support everything. You have to use your money strategically, use your time stragically, use your talent strategically. | |
Once you figure out what those links are in the chain, that are most important for your company, don’t assume you know what you need to do to bolster them, to make them stronger. Go to that community. I guarantee they’re not going to get cranky, and if they are then that’s probably not a community or a project you want to work with anyway so you can start looking for an alternative and rearchitect stuff. Go to that community and say, “Hey! We love your stuff! We would love to help. Where do you need help? Security audits? Great, we’ve got that. You need UX? Great, we’ve got that. You need documentation? I will dedicate a tech writer purely to you.” Just see what they need, and what will make that project more successful…and then give it. But don’t simply force some support on them because it’s easy for you. Don’t simply throw money at the problem because it’s easy for you…although you might find that what they need is money. You never know until you ask. | |
Once you’ve done these things, then you have to follow through on them. That can be very difficult. Getting your board of directors to agree that it makes sense to raise headcount so you can support these open source projects. “Well, yeah, but then everyone else including our competitors gets it.” “Sure, but they already get it. We’re not giving away vital IP, we’re just fixing and bolstering something we’re already using.” Having those conversations can be quite difficult sometimes. | |
It can take a fair bit of time to implement this sort of plan. Just like with a corporate sustainability plan, an open source sustainability plan takes a lot of effort. Those three items I just pointed out, they were not simple items. Figuring out your open source supply chain alone is going to take you a while, and you’re really going to be pulling out the good whiskey for that one. It’s a pain once you start seeing what’s going on there. You can’t rush this. You can’t take short cuts. You have to look at the long view, but that’s what sustainability is all about. Just know this in advance, going in, eyes wide open, that this is going to be difficult and it’s going to take a long time, but it’s going to pay off over the long term. And that’s OK. Because it is so difficult, remember that you can’t copy someone else’s plans. Even the plans within your own company might be different. One team might be using Go, another team might be using Perl. These are completely different ecosystems, and will have completely different needs, so you can’t just cut and paste one plan across all of the teams, either. | |
So it will take time to do, but it’s totally worth it. You’ll find you’ll get a lot of benefit out of an open source sustainability plan. | |
These are some of those primary benefits. You may or may not notice, but these are the exact same benefits of a corporate sustainability plan. That’s not accidental. This is very good for your bottom line, for your company to have something like this. But ignoring the problem of the sustainability of the open source projects on which your company relies, is not. | |
So there we go. I am VM Brasseur, but we’re all friends here so you can call me Vicky. I’m Director of Open Source Strategy at Juniper Networks right now. I’m also the author of this, which is the first and only book about how to contribute to open source software. You can find me here on Twitter. These slides are already available here at Internet Archive, and the book at the URL at the bottom. Thank you to all you beautiful CTOs in training for being here. Thank you especially to Nicola for asking me to be here, and to Fastmail for hosting. And thanks to all of you. Snipped Nicola’s comments between the presentation and Q&A The microphone did not pick up the questions from attendees and I foolishly did not repeat the questions. Q: What happens to all those languages other than Python, the Javas, everything else that’s meandering along. What happens to those? A: It depends upon the language. Some of them… There’s actually a very strong and healthy community around Open Java, for instance. They’ve done a very nice job of saying, “No no no, we’re not Oracle,” and holding themselves separate and building a strong community. I know some of the leaders in that community. But it depends upon the language. If they don’t clean up their act, then… Now, no language is ever truly dead, but…if you’re not consistently attracting more people to your language, if you’re not consistently bringing more people to your mailing lists, to your chat channels and stuff like that because of the behaviour… And when people have consistently told you, “This is a problem,” your language will die. There are some languages out there like that; I’m not going to name and shame, but I have told them that this is a problem. Yeah, they’ll die, and that’s fine. That’s just ecology and evolution. Q: You were saying before that things like GitHub and stuff, you need to make sure you have a plan. Let’s say you don’t culturally abide by what they’re doing or something like that. But how many cases would there be for alternatives, if you’re using an open source library or something like that, or you’re wedded to a language. How many times are you going to have an alternative? Most open source projects wouldn’t have competitors, would they? A: There are actually open source projects with competitors. Do you use a noSQL database? Oh my god, how many of those are there? Javascript frameworks, there’s a new one every week. There are usually alternatives. For projects where you find where there’s one that’s strategically important for your company to use, and they use GitHub, then use GitHub. But if you release, say, a new library, if you’re releasing a new module or something like that, that ties into that larger ecosystem, you can expand it by releasing it elsewhere. You can use BitBucket. You can use GitLab. There are alternatives. You can even—although I don’t recommend this for most people—you can self-host your git server and allow people to just put stuff there. It’s kinda high overhead and I think it’s not a good idea. Q: That’s the example where there are alternatives, but I think there would be a very large number of open source projects where there aren’t. A: And that’s fine. You use what they have, but you yourself, when you’re releasing things, you have a choice. Are you going to continue perpetuating the monoculture, and the single point of failure, or are you going to expand the picture and release things elsewhere? We have a new project that we’re releasing soon that we’re working on with Red Hat and Intel. Where are we going to put it? Are we going to put it in GitHub? Or are we going to put in GitLab? There are valid reasons to do both. One of the most valid reasons, depending upon your particular need, is network effects. You go to GitHub because that’s where the people are, and that’s where everyone assumes they’re going to find it. But what we’re starting to see more and more is that, yeah, people will look at GitHub first but then they’ll go to your project’s website and they’ll click on the “Fork us!”. And then they’ll go to GitLab, or BitBucket, or if you put into a foundation they’ll go to the Apache website or what have you. So you have the alternative to expand or not, but you have to look at the reasons why you would do that, and that’s a conversation that we’re having with this new community right now. Where do we put it and why? It might turn out that because of network effects we do have to go to GitHub, but we’re not defaulting. We’re actually thinking about it. Q: It’s not so much where the code goes, it’s where you track your issues, really. A: Yeah, it’s where you track your issues. And frankly, it’s really irritating every time GitHub goes down. I’m on Twitter a lot, and every time GitHub goes down you see all the wags on Twitter going, “Uh huh. Distributed version control system, right? Uh huh, you can’t do any work because GitHub is down.” And git is a distributed version control system, but we all know there is one master, and you just go to the HEAD on GitHub. But you could just have that elsewhere. That’s perfectly fine. And then you can have it mirrored to GitHub and then just say, “No, we don’t take pull requests” and have a Hubot that closes stuff. You can do that, and that’s perfectly fine. That’s one of the alternatives that a lot of projects are doing right now, especially if they’re hosted elsewhere. Like the Linux Foundation often wants you to have stuff in their infrastructure. Those projects will do that and it’ll be their canonical repository, but they’ll be mirrored to where people will expect to see them. OpenStack does this. OpenStack has its own repositories and they’re under the OpenStack Gerrit—which is a horrible, terrible piece of software, but aside from that… They have everything under Gerrit, and then they mirror them to GitHub. These are options you can do. These aren’t stopped when GitHub goes down when everyone else is, since that’s just where you can go to do searching or what have you. Q: To add to that would be security, the idea that something comes up faster. Can fork it, have a look at it, there may be a security issue before it goes into a production system— A: —well, you should be doing that wherever it comes from. Q: Right, but the idea might be that something comes out of master that may have an issue that gets detected, and how does that…? A: That’s a build/release issue. I mean, where are you building your stuff from? Do you trust it? Are you using containers? Holy shit, that’s a nightmare for compliance and for security, and people are only just starting to figure out how bad it is, which is a shame. Do you get your Docker and manifest and build all your containers locally so you can control what’s in that and you can look at what in the licenses, and you can see what’s happening in the versions. Are you using a version of openSSL that’s too old in that container? You don’t know, because you haven’t opened your container, you’re just pulling it down from DockerHub and trusting it. Yeah, it’s like curling something to root, to su on your command line. You just get a random curl line from StackOverflow and you curl it to root and it’s perfectly fine. What could this possibly do to my computer? People aren’t necessarily aware of the risk profiles there. Compliance is one risk profile, security is another. It’s the sort of thing that a lot of companies don’t think about, which is why I have a job. Q: indistinct voice A: Yeah, again, it’s a risk profile. We’re so used to just grabbing something and using it, and we’ve allowed that in our developers for so long that, well, changing that is going to be very difficult but it’s something we’ll have to do and Docker is just making it so much worse. There is talk of the new Open Container spec version 2.0. People aren’t working on it yet but they’re starting to mumble around the edges that version 2.0 of the OCI spec is going to include more compliance stuff, more security stuff. So we can hopefully improve this going forward, but there’s still going to be all of this legacy software out there that are going to be little time bombs in your software supply chain. Q: You mentioned documentation around speaking, and you got me thinking about pushing open source concepts up to say indistinct, is that…? A: No…? And there’s no documentation out there for that. Some people would like to think they’ve got documentation and I’ve reviewed it all and it’s all super general and not useful. I’m on the Steering Committee of the TODO Group, which is the Linux Foundation sub-foundation—because that’s how they work, it’s foundations all the way down—for open source program offices. So if you run an open source program office, you join the TODO Group after giving the Linux Foundation $20,000, that’s bare minimum to join LF [\*]. Then you can show up and talk to other people who run open source program offices. They generate white papers and stuff. Some of them could potentially be used for this, but I haven’t found any of them to be particularly actionable. So no, it doesn’t exist yet. I’ve been speaking with my editor about potential next books, and it’s the sort of thing, like, business and open source is really something we need a book about so we can have these conversations and show people how to do these things. But I’m still healing from the first book. :-) So that’s down the road a bit. But it would be nice if there were some. [\*] Correction: This is the minimum for the size of company where I worked at the time of this talk. The membership fees vary by desired membership level and/or the number of employees in the company seeking membership. Here is an example of the LF Membership Agreement from 2017, which has numbers that are accurate as of the publication of this post. Q: The thing is, you mentioned, “give it to our competition.” A: There are plenty of blog posts out there about it, but it’s just stupid. You can’t…you know, I’m sorry, not every piece of intellectual property that you generate within your company is this precious golden nugget. It doesn’t matter. Most of the stuff you generate is just commodity crap. It’s automation. It’s utilities. It’s tooling. It’s not your actual critical intellectual property. It’s not your secret sauce. So it’s fine, most of the time, if you release it. It can be fairly strategic to do it, because you can start to build goodwill, and a good brand, and you can cut off your competitors who might otherwise provide a tool to make it easier to use their software, how about you provide a tool to make it use your software as well so people are more likely to do it and you can build an ecosystem around your software… There are really valid reasons to do it. There’s lots of reasons to do it, but it’s just a matter of sitting down and having these conversations with people, because you do have a lot of fear, uncertainty, and doubt (FUD). There’s a lot of FUD around free and open source software, particularly in the business world, thank you Microsoft of the 90s. They have, by the way, done a complete 180 and their change is real. It’s the most amazing thing to have watched happen. Satya Nadella has really done a wonderful job of changing the deep down to the bones culture of Microsoft around this. It’s because they figured out, “Oh…if we do things in open source and if we’re really sincere, legitimate, authentic open source citizens, we’re gonna sell a helluva lot more Azure.” And that’s why they do it. And it works! They sell a lot more Azure. I’m all for that, because it helps bring more open source into the world. They have some of the most brilliant minds in free and open source software working for them. It’s legit, it’s not just because they’re trying to make more money. It’s also because they’re trying to do the right thing and they see that doing the right thing makes them more money. Q: Adding onto that, would an idea of linking capabilities to business outcomes for open source fall in that devops mentality that accelerate devops indistinct voice regarding how you map these capabilities as far as versioning or security or CI/CD or culture or that sort of stuff, and the moving out of actual value so that when you go up to senior…? A: Yeah, you do have to speak their language. I mean, you can’t go in there… When I walk into a company, I don’t go in there wearing my Free Software hat. /me puts on a hat from behind the table. I bought this yesterday and I think it’s super cute, so… Anyway, so I don’t go in there wearing my Free Software hat. I personally tend more Free Software, but I’m very pragmatic rather than most Free Software folks who are very dogmatic. I go in there with my business hat and here’s what you need to accomplish and here’s how we can do it through open source software, and here’s the investment we need you make, and here’s what you’re going to get out of that…and you have the full presentation and stuff. But you do have to speak their language. That’s with anyone, right? The public speaking stuff: you figure out who your audience iis, and then you address things directly to the audience. You figure out, “What are your hot buttons? What’s important to you, and how can I weave that into my narrative so I can then ethically manipulate you to do what I need?” That’s really what it all is. You don’t use this power for evil, because that snaps the ethically off the manipulation, but that’s what you have to do, and you do that with any audience. Q: Would it be more along the line of the historics of, let’s say, a project? Where you say you’ve gone to a supplier that’s closed source versus something that’s open source where you can see parallels and you can see the history and you can say, “Hey, look how far we’ve gone with this open source project, because we’ve been able to develop these modules in-house, where we’ve had to go ahead and wait for this other company to provide us this function and by the time it gets to us it’s not what we were looking for to start off with.” A: Depending upon the project, that can make sense. You can say, “Well, switching to Postgres we dropped these two million dollars a year in Oracle fees so therefore we’re making a lot more money.” Or, for instance, paying someone to work fulltime on this one open source project—say embedding them with Fedora, in the Fedora community—and then switching all your servers to run Fedora rather than Red Hat Enterprise Linux. You’re paying the salary of that one person to be embedded in Fedora and become part of the community, but what you’re also doing is probably getting rid of that support contract from Red Hat, because you’ve got someone in-house who’s already super experienced. Or reducing the support contract. So there are various ways you can do this, depending upon the project. It just really…it depends. That’s going to be my answer to almost everything. I will almost never give you a straight answer, which is why I’m an executive. Q: I find if you can get more than one it’s even better. A: Yeah, I mean, like I said, it depends. It really will depend dramatically upon your specific use case and your needs and supply chain and stuff like that. Q: I was going to ask earlier about an example where someone indistinct voice and you pointed out Microsoft. Do you think, if you look at your sort of three areas of how indistinct voice is there anywhere they’ve done this really well and indistinct voice? A: Well, it’s relatively new, Microsoft doing the 180 and saying, “Yay! Open source!” We were all super skeptical when first we saw it, by the way. They’re doing a pretty good job across the board. They support a great deal of diversity efforts. All the projects that they work in they try very hard to make them safe, welcoming places. You have to remember, however, that it’s a massive company and just like any massive company (or even small groups), there are going to be bad actors. So you can’t say “Oh, they do everything right!” But across the board they do try very hard to do things properly, and that’s because they have people internally who are specialists in open source and specialists in community, who are keeping an eye on all of this stuff. They’re working internally to do a lot of training, “here’s how you engage in an open source community, please read Vicky’s book and you can engage in an open source community.” They’re keeping an eye on this stuff and trying to maintain consistent policies and procedures. A lot of that is around those three elements. They’re giving back in a consistent way in multiple ways, not simply code, not simply money, but they also do a lot of diversity stuff. They do a lot of —what was the third one? It’s my own talk, you’d think I would know; safety!—they do all of those things. It depends upon the project how they do that. They do have a lot of open source specialists who work on that. They invested. They saw that open source isn’t something that just anyone can do if you want to do it properly. There are people like me who have been doing this for like thirty years. There’s a reason why a specific job for this, because it’s a specific thing. They recognise that and are, like, “I’ll pay you a great deal of money to come work for me.” Q: And that’s why we see AWS and open source and being indistinct voice… A: That’s a different matter. That’s a completely different matter, and that’s all wrapped up in things like “open core.” Remember that secret sauce that Bron mentioned? Plenty of VCs out there think… Who else lived through the bubble in the 90s? I lived through the bubble in the 90s. “Oh, if you just give shit away for free then it’s a loss leader and people will come and they will pay for your stuff!” That was called “Freemium” in the 90s. Now it’s called “open core.” Because you release stuff, all of your software. It’s even worse than Freemium, because now you’re releasing all of your software as open source. Then you’re getting business strategy from VCs whose motivation is not the same as yours. They want any sort of exit whatsoever. They don’t care. So they’re giving really bad strategy, such as “release all of your code and then people will show up for the value adds.” But they’re not telling you how to run a company, they’re not giving you good ideas about pricing, they’re not giving you good ideas about product. So what you do is you release your secret sauce, and then a company like Amazon looks and they say, “Oh, it’s an open source project! That’s pretty cool! Let me take that take that open source and build a more compelling product offering around it.” That is perfectly valid. That’s not Amazon being bad at open source. That’s actually Amazon being good at open source. It’s these companies not knowing that they have to build communities around these things to help make a sustainable open source project. Then when Amazon takes it and builds a product it doesn’t matter because we still have our community, and our community is going to be great and now we can reach out to Amazon and say, “Amazon, please contribute back. How can we make it easier for you to contribute back? What can we do for you?” But nobody does that, so in all of these foot-stamping-little-princess startups saying, “Amazon stole my code!” No, you twit. You put it out there under a permissive license. Amazon is doing exactly what you told them they could do. Then they’re doing it better than you are. Somehow this is Amazon being bad? No, it’s you not knowing how to run a bloody business, that’s what that is. So Amazon is actually trying very hard to be a good open source citizen. Do they have far to go? Yeah, but we all have far to go. They’re trying very hard to do the right thing, it’s just that the larger conversation is, “Oh, Amazon is bad,” so we buy into that rather than looking at the actual details around it. Yeah, Amazon is a massive company and they can do more for open source. They’re aware of that, but changing the culture of a company that large, when each invididual team operates all on their own… It’s not like Microsoft, where the teams are actually fairly integrated, so you can have Satya at the top saying, “This is what we’re gonna do, mates, and you’re bloody well gonna fall in line.” It’s actually kinda difficult for Jeff to do that because of the way the Amazon teams are organised. You can’t shift the culture easily because you have millions of different little cultures, of each team doing their own thing. Yeah, they’re all very customer focused but that doesn’t necessarily mean they’re open source focused. So their open source program office—they actually have two open source offices—each of them has a very high hill to climb because of that. They are climbing it and they’re working really hard and I’m seeing some shifts, but it’s pretty difficult. It’s not Sisyphian; they will be able to make some progress and they are, but it’s going to take a while. So don’t just buy the Hacker News “OMG Amazon stole my code” story. Be analytical and look at it a little more. |