Don't fund Software that doesn't exist

I’ve been happy to see an increase in funding for open source software across research areas and across funding bodies. However, I observed that a majority of funding from, say, the NSF, goes to projects that do not exist yet, and where the funding is supposed to create a new project, or to extend projects that are developed and used within a single research lab. I think this top-down approach to creating software comes from a misunderstanding of the existing open source software that is used in science. This post collects thoughts on the effectiveness of current grant-based funding and how to improve it from the perspective of the grant-makers.

Instead of the current approach of funding new projects, I would recommend funding existing open source software, ideally software that is widely used, and underfunded. The story of the underfunded but critically important open source software (which I’ll refer to as infrastructure software) should be an old tale by now. If this is news to you, look at the history of the heartbleed bug which basically broke security on the internet and lead to the core infrastructure initiative, or read the insightful “Roads and Bridges” report by Nadia Eghbal. For a concrete example of a critical infrastructure project that is particularly scientific in scope, but still unfunded, see “The AstroPy Problem”.

There has been a lot of discussion surrounding the analysis of this problem, and drawing relationships between peer-production open-source infrastructure and Common Pool Resources. A great summary is provided again by Nadia Eghbal. In this particular writing I want to focus more directly on the impact and structure of grant funding in this context.

Before I explain why I think funding new projects is a bad idea, let me start by briefly providing some background on how many open source projects that became infrastructure have been created. Commonly, these were started by a single person, usually out of need (Python, NumPy, matplotlib, git, IPython/Jupyter) or curiosity (linux). Then, other people joined the project as they found it either interesting or useful themselves.

Most of these projects are not only “open source”, they became peer-production projects, in which a loosely assembled self-organizing community of volunteers collaborate to create software. In more historical terms, this could be characterized as the projects being “free software” not only “open source” - open source is a statement about the license attributed to the code, while free software also includes aspects of governance and community engagement (this is also related to the bazaar metaphor from the Cathedral and the Bazaar).

It seems to me that the important distinction between open source software and peer-production has been mostly lost on grant makers and funding agencies, as they are usually trying to create new projects, or plan big overhauls of existing projects. Both of these are in conflict with the peer-production (or bazaar) philosophy as I’ll describe below.

Was our infrastructure created from grants?

Given the amount of current funding, we can ask “How many software packages that are central to science were made as a result of grants?” I hope (and imagine) the answer is not zero, but I am not familiar with any examples. Looking at Jupyter or the scientific ecosystem (numpy, matplotlib, pandas, scipy, scikit-learn, scikit-image, seaborn, astropy, …)--none of these projects were started by grants (even though Jupyter benefited from substantial funding later on). From my conversations with people in the R community, my understanding was that the situation there is similar.

If you are aware of central pieces of open source infrastructure that were kick-started by a grant, please let me know, as I’m quite curious to see their development.

Do grants create infrastructure?

The reverse question is being worked on by Johanna Cohoon and James Howison. You can see preliminary results in Routes to Sustainable Software in Science: Transitioning to Peer Production. They followed packages that did get funding via the NSF program “Software Infrastructure for Sustained Innovation” (SI2), and investigated whether and how these projects were transitioned to a community-based model. As a side-note, the SI2 program is part of what funds my own work, so in a sense I’m quite lucky it exists. However Cohoon and Howison found that of the 23 projects they captured in their taxonomy, only one project transitioned from a single author project to a peer-production project, and none of the projects that were started within a tool group or a research lab transitioned to peer-production, i.e. to being a community project. The SI2 grant explicitly asks for a sustainability plan--which is usually peer-production. Still, only a single project was able to make this move. Given my experience with academic open source software, this is hardly surprising. Not transitioning to a community based project does not necessarily mean the project is not used, but it is a very strong indicator.

Several projects studied by Cohoon and Howison had no activity after the grant period ended, essentially meaning the project was abandoned, and the money and time wasted to create an artifact that was discarded immediately.

I should note that three of the projects examined were already peer-production projects (the study did not include my work on scikit-learn as the grant is ongoing), meaning the ratio of projects that are likely to die with the grant to those that will be useful is 4:19.

So we have seen that those projects that are central infrastructure were not established via grant money, and that those that are established via grant money do not become infrastructure, and in fact rarely become peer-production projects at all. What are possible explanations for these observations?

Do grant-based projects adhere to open source principles?

A more provocative version of this question might be “Are there fundamental reasons why creating infrastructure open source software via grants will fail?”. I think there are two potential reasons.

Incentives

To investigate this, let’s look at the manifesto of peer production software, “The Cathedral and the Bazaar” by Eric Steven Raymond. It states, as the very first lesson of open source software (!!):

1. Every good work of software starts by scratching a developer's personal itch.

I have found this to be true in my personal experience, and while Raymond doesn’t provide direct empirical data, I think this insight is largely accepted in the open source community and  for example repeated in Internet Success: A Study of Open-Source Software Commons, by Charles Schweik, which is well-summarized in Six Things to know about successful open-source software by Rich Gordon.

I want to argue that this first principle is contradicting the creation of open source projects as a result of a grant. If you need grant money to even start the project, it means the project is not useful enough to be worth your time otherwise. Writing software as a result of a grant means you didn’t actually need the software, otherwise you would have written it before. If even the author doesn’t need the software enough to incentivize them to write it, it’s unlikely that anyone else will, and it’s unlikely that a peer community will form that will will be incentivised to develop and maintain the software.

Let’s rephrase this last argument in a slightly different way. I think the impact a single developer can make is often inversely proportional to the size and maturity of an existing piece of software, and actually the personal return, both as a user, and in terms of community rewards, is similarly inversely related with project size and maturity.

That would imply that as a project grows, it will be harder to convince someone to voluntarily contribute to a project, because their trade-off gets worse and worse as the project grows.

If the project was started by grant money, that means the incentives for the very first developer were not strong enough to start the project out of self-interest. As the project grows (as a result of the funding), the incentive for someone else to come in and contribute will be even smaller than for that first developer. Given that the project wasn’t attractive enough for the first contributor, how will it ever attract anyone else? More discussion on developer motivations and the difficulties of attracting developers can be found in Cave or Community?: An Empirical Examination of 100 Mature Open Source Projects by Sandeep Krishnamurthy.  For a discussion of incentives in scientific open source software, see Incentives and Integration In Scientific Software Production by James Howison and Hersleb.

Natural Selection of Projects: Most open source projects fail

Another crucial aspect of peer-production projects is that there is a process of natural selection among projects. Most open source projects are not successful. There are thousands of open source projects related to science created every year, many of them aim at being infrastructure. Most of them never achieve any uptake or even become functional, and according to the study in Schweik’s Internet Success, having a source of funding is not related to becoming successful. Of the projects that Schweik studied, 17% were successful by their measure (not abandoned for at least three releases and fulfilled a clear need). While this number is not directly comparable to the numbers from the Cohoon and Howison study, this success rate over all projects on sourceforge (the dominating platform at the time of the study) is likely comparable or even higher than the number of successful projects that were funded by SI2.

The bazaar-style development model means that enough developers have been convinced the project is viable and interesting to commit their time. This sets a bar for the ability for a project to become a peer production project. Initiating a new project means creating one more proposal in the thousands of projects that might be started and wither. Funding a project that is already in the peer-production phase means the project has already cleared that bar. To me (and having been on review panels), it seems unlikely that a funding body can accurately judge whether a project will be successful or not, which is made evident by the numbers of the Cohoon and Howison study. However, this ability is implicitly assumed in the grant-making process if the long-term goal of the grant is peer-production.

Is funding existing software any better?

The observations above are not in conflict with asking for funding for existing, established projects. Often these have communities surrounding them that voluntarily contribute time and effort to support the project. However, the effort involved in developing and maintaining a project is usually proportional to the user base, and the developer community might not grow proportionally (NumPy has contributions from 46 developers over the last year according to GitHub, but is likely to have millions of users). Existing projects also suffer from bit-rot, and maintaining an existing project can often be seen as much less fun than writing something new from scratch. It also brings with it the burdens of legacy software, and the mixed bag of dependent users---who can be encouraging and motivating one day, and then inflammatory and aggressive the next.

To ensure central infrastructure will stay available to the scientific community, funding is necessary for large existing projects.

However, both the known dynamics of open source development, as well as data on existing grants and existing infrastructure show that money invested in new projects does not pay off.

Closing Notes

My recommendation for any future software funding would be to only fund software that already uses peer-production as its organizational principle, or absent that, to fund maintenance and enhancement of projects that are widely used. For now, ironically, the part that it is easiest to find volunteers for (implementing new features) is the easiest to fund, while the one that is basically impossible to fund (maintenance, though big props to the Gordon and Betty Moore Foundation Alfred P. Sloan foundation, in particular Josh Greenberg and Chris Mentzel, and Chan-Zuckerberg for starting to change that) is the one that it is hardest to find volunteers for.

Going even further, the requirement of milestones and deliverables that is often required for grant applications is in harsh contrast to the peer-production nature of open source infrastructure, and more closely resembles the waterfall model, that has long been abandoned in favor of agile software development. The more we can move towards community-driven decision making, the more advantageous the outcome for both scientific users and open source communities will be. In a sense, this is my response to a piece from Dan Katz, who was at the time an NSF director overseeing the SI2 program about maintenance-only grants, questioning their usefulness. I think maintenance-only grants are the way forward (or maybe even open-ended grants), not because we don’t need new development, but because it’s impossible to plan many important developments on the time-scale that grants are made (see agile vs waterfall) and because it’s much easier to find volunteers for new features than it is to find them for maintenance. Effective maintenance requires multi-year commitments that are much easier to make with long-term grants.

If you’ve read this far, I highly encourage you to read Six Things to know about successful open-source software by Rich Gordon and the “Roads and Bridges” report by Nadia Eghbal. James Howison has been studying the development of scientific open source, incentives and in particular its relationship with grant funding and I highly recommend checking out his work.