How Ruby Went Off the Rails

Hey there,

I write in my author bio that I love drama, and this story I have today is exactly why. What has been happening in the Ruby development and open source community over the last couple of weeks is interesting not only because it’s messy and dramatic, but because it reminds us how the internet actually works. The internet is made out of people who work with other people and make decisions about how to do things. You probably don’t know the vast majority of these people, but these personalities play a big role in maintaining the infrastructure we all rely on.

-Emanuel

For the past couple of weeks, a community of developers who use the programming language Ruby have been closely following a dramatic change in ownership of some of the most essential tools in its ecosystem with far reaching impacts for the worldwide web. 

If you’re not familiar with Ruby or the open source development community, you probably haven’t heard about any of this, but the tools in question serve as critical infrastructure for gigantic internet services like GitHub, Shopify, and others, so any disruption to them would be catastrophic to those companies, their users, and vast swaths of the internet. 

On September 19, Ruby Central, a nonprofit organization that manages RubyGems.org, a platform for sharing Ruby code and libraries, asserted control over several GitHub repositories for Ruby Gems as well as other critical Ruby open source projects that the rest of the Ruby development community relies on. A group of open source developers who had contributed to those projects and maintained them for years had their permissions suddenly revoked. When these developers announced on social media that their access was taken away, many Ruby developers saw the decision as a betrayal of their years-long contributions to the Ruby ecosystem and open source principles more generally. Others accused Ruby Central of succumbing to corporate pressure from companies like Shopify, which they claimed wanted more control over the project. 

In some ways, this whole affair is an example of why this stuff gets really messy when people start getting paid

I’ve spent the last week talking to people who had direct involvement with Ruby Central’s decision, the contributors who were ousted, and developers in the Ruby community. I’ve heard accusations of greed, toxic personalities, and stories about years-long feuds between people, at times in open disagreement, who ultimately govern some of these important open source tools. 

RubyGems.org and other critical Ruby tools have so far not been interrupted during this transition, but the incident sheds light on a basic truth about the internet and open source development: Much of the technology we use every day and take for granted is being maintained by a small number of developers who are not compensated for that work or get paid very little when compared to salaries at big tech companies. Open source development continues to make much of the internet possible, but as some of these tools become more important and financially valuable, they’re subject to more scrutiny and pressure from the community, organizations, and companies that rely on them. 

“In some ways, this whole affair is an example of why this stuff gets really messy when people start getting paid, and once you start introducing formal organizations and employees and nonprofits and lawyers and all this kind of complexity,” Mike McQuaid, developer of the popular package manager Homebrew, which is built with Ruby, told me. McQuaid has talked to and offered to mediate between Ruby Central and the ousted maintainers. “This is a textbook case of what happens when there's this conflict between what companies want, what nonprofit individuals want, how much responsibility people have when they take money, who gets control and when. How much democracy versus just ‘I have the power to do something, therefore I'm going to do it.’” 

With Ruby developers can download and use self-contained packages of code that add different functionalities to a Ruby project. These packages are called gems, and are distributed primarily via RubyGems.org, where developers can upload gems they’ve developed or download gems from other developers. 

The ability to download gems and plug them into different projects is very useful and convenient for Ruby developers, but can create complications. Different gems are developed by different teams and are updated at different times with bug fixes and new features, and might not necessarily be compatible or play well with one another as they evolve. 

This is where Bundler comes in. As its website explains, “Bundler provides a consistent environment for Ruby projects by tracking and installing the exact gems and versions that are needed.” So, for example, if a developer is building a Ruby project and wants to use gems X, Y, and Z, Bundler will pull the versions of those gems that are compatible with one another, providing developers an easy solution for what Bundler describes as “dependency hell.”

Bundler is an open source project that was initially developed by Yehuda Katz, but the GitHub repository for the project was created and was administrated by André Arko. In 2015, Arko also founded a nonprofit trade organization named Ruby Together, which raised funds from developers and companies that use Ruby in order to maintain Bundler and other open source tools. 

I will not mince words here: This was a hostile takeover

RubyGems.org, the site and service, is governed by Ruby Central, a nonprofit founded in 2001, which also organizes several Ruby conferences like RubyConf and RailsConf. In 2022, Arko’s Ruby Together and Ruby Central merged, “uniting the Ruby community’s leading events and infrastructure under one roof,” according to Ruby Central’s site. Bundler’s and RubyGems.org’s work often overlapped both in their goals and the developers who worked on them, but operated across two different GitHub organizations, each with its own repositories. To streamline development of these open source projects, Bundler also joined the Ruby Gems GitHub organization in 2022. 

In 2023, Ruby Central established the Open Source Software Committee, which according to its site oversees RubyGems, Bundler, and RubyGems.org, focusing on infrastructure stability, security, and sustainability.

A confusing and central point of disagreement between Ruby Central and the maintainers it ousted on September 19 is rooted in the merging of Ruby Together and Ruby Central and the difference between RubyCentral.org the service, essentially an implementation of the Ruby Gems codebase on an AWS instance, which both parties agree Ruby Central owns and operates, and the Ruby Gems the codebase that lives in the same GitHub organization as Bundler. 

According to a recording of a mid-September Zoom meeting which I obtained between Marty Haught, Ruby Central’s Director of Open Source, Arko, and the other ousted contributors, Ruby Central maintains that the codebase and GitHub organization became its responsibility when Ruby Central merged with Ruby Together in 2022. The ousted contributors’ position is that members of Ruby Central, like Haught, can be owners of the GitHub organization, but that ownership of the RubyGems codebase and other projects in the GitHub organization belong to the contributors, who don’t have a detailed governance model but historically have governed by consensus. 

Arko made this argument to me in a recent interview, but also outlined that argument in a blog post, where he also shared the merger agreement between Ruby Central and Ruby Together. It shows that Ruby Together would dissolve and that Ruby Central would be in charge of raising and allocating funds for development, but does not explicitly say Ruby Central takes ownership of the RubyGems and Bundler projects or the GitHub organization. 

To make matters even more complicated, Arko was at once a contributor to these open source projects, a contributor to RubyGems.org the service, an owner of the GitHub organization, and an advisor to Ruby Central’s Open Source Software Committee. 

In May, Arko resigned his position as an advisor to Ruby Central’s Open Source Software Committee, but continued his work as a contributor. Arko told me he resigned his advisory role because of Ruby Central’s last minute invitation of David Heinemeier Hansson, better known online as DHH, as a keynote speaker at RailsConf.

Arko told me he objected to that decision because of DHH’s “horrifying, racist, misogynist, politics” and DHH’s “personal vendetta” against him. In 2021, back at Motherboard, we reported that many employees at DHH’s company, Basecamp, quit after his decision to ban any discussion of politics at work, which many employees saw as squashing discussion about race, bias, and diversity. Arko told me that DHH’s “personal vendetta” against him stemmed from Arko not wanting to support a certain feature DHH wanted added to Bundler, after which DHH demanded Arko be removed from the Ruby Together board. 

The current controversy erupted on social media on September 19, when one contributor to the open source projects in the RubyGems and Bundler GitHub organization, Ellen Dash, announced that Haught, Ruby Central’s Director of Open Source, revoked GitHub organization membership for all admins on the RubyGems, Bundler, and RubyGems.org maintainer teams. At that moment, their permissions and access to the GitHub organization were revoked, meaning they could no longer make any changes or contributions to the code, and Haught, representing Ruby Central, took control. 

“I will not mince words here: This was a hostile takeover,” Dash said in a public “goodbye” letter she shared online. “I consider Ruby Central’s behavior a threat to the Ruby community as a whole. The forceful removal of those who maintained RubyGems and Bundler for over a decade is inherently a hostile action. Ruby Central crossed a line by doing this.”

The news was seen by many developers in the Ruby and open source community as betraying the dedication and labor that Dash, Arko, and other maintainers put into these tools for years. 

Ruby Central, meanwhile, describes the move as one centered around security. 

“With the recent increase of software supply chain attacks, we are taking proactive steps to safeguard the Ruby gem ecosystem end-to-end,” Ruby Central said in an explanation of its decision. “To strengthen supply chain security, we are taking important steps to ensure that administrative access to the RubyGems.org, RubyGems, and Bundler is securely managed. This includes both our production systems and GitHub repositories. In the near term we will temporarily hold administrative access to these projects while we finalize new policies that limit commit and organization access rights. This decision was made and approved by the Ruby Central Board as part of our fiduciary responsibility. In the interim, we have a strong on-call rotation in place to ensure continuity and reliability while we advance this work. These changes are designed to protect critical infrastructure that power the Ruby ecosystem, whether you are a developer downloading gems to your local machine [or] a small or large team who rely on the safety and availability of these tools.”

404 Media has covered the kind of recent supply chain attacks targeting open source projects that Ruby Central is referring to. Earlier this month, a critical JavaScript development tool Node Package Manager (NPM), was targeted by a similar supply chain attack. But not everyone in the Ruby development community bought the explanation that security was at the heart of the recent moves. One reason for that is a public statement from a Ruby Central board member and treasurer Freedom Dumalo. 

On Substack, Dumalo apologized for the sudden change and how it was communicated. 

“If Ruby Central made a critical mistake, it's here,” he wrote. “Could these conversations have been happening in public? Could the concerns we were hearing from companies, users and sponsors have been made more apparent? Probably. But I remind you we don't have a ‘communications team’, no real PR mechanism, we are all just engineers who (like many of you I'm sure) go heads down on a problem until it's solved.”

Dumalo reiterated that RubyGems and Bundler are critical infrastructure that are now increasingly under the threat of supply chain attacks, and said that the companies that rely on them “count” on Ruby Central do everything it can to keep them and their users safe.

However, Dumalo also said that Ruby Central was under “deadline” to make this change. 

“Either Ruby Central puts controls in place to ensure the safety and stability of the infrastructure we are responsible for, or lose the funding that we use to keep those things online and going,” Dumalo wrote. 

In a September 22 video message in response to criticism about its decision to remove maintainers, Ruby Central’s executive director Shan Cureton described a similar dynamic. She said “sponsors and companies who depend on Ruby tooling came to us with supply chain concerns” and that “Our funding and sponsorships are directly tied to our ability to demonstrate strong operational standards. Without those standards in place, it becomes harder to secure the support needed to keep maintainers paid, organize events, and provide resources for developers at every stage of their journey.”

Since Shopify is one of the primary sponsors and funders of Ruby Central, this led some in the Ruby community to believe that Shopify was exerting pressure on Ruby Central to make this change. 

“That is not how it happened, and I wish I had been more careful with my wording in that blog post,” Dumalo told me in a Linkedin message when I asked him if Ruby Central was under pressure from Shopify to make these changes.

I just don't think that there's any other plausible explanation than Shopify demanded this.

After I gave Dumalo my number so we could do a phone interview, I got an email from Cindi Sutera, who was recently brought on as a spokesperson for Ruby Central.

"Ruby Central’s mission is to keep the infrastructure that Rubyists rely on stable, safe, and trustworthy,” she told me. “As part of a routine review following organizational changes, we identified a small number of accounts whose privileges no longer matched current role requirements. The Board voted that it was imperative to align access with our privilege policy to keep the infrastructure that the Ruby community depends on stable. This is our mission.”

Sutera said that the board approved “a temporary administrative hold on certain elevated permissions” while it finalized operator agreements and governance roles.

“To move quickly and transparently, we imposed a clear deadline to complete operator agreements and close gaps,” she said. “We could have communicated earlier that we felt it necessary to move quickly and wish we could have given the community more time to prepare for this action. And now, here we are committed to completing this transition for the stability and security of the Ruby Gems supply chain. More updates are coming as we work through security protocols and stabilization efforts.”

“There’s literally only one company providing the money that is keeping Ruby Central open, and it is Shopify,” Arko told me. “And so I just don't think that there's any other plausible explanation than Shopify demanded this.”

When I asked Arko why he thought Ruby Central removed him, if it wasn’t for security reasons, Arko said: “totally unprovable speculation is Shopify’s CEO is best friends with DHH, who hates me.” DHH is also a Shopify board member. 

“Thanks for the invitation, but not my place to weigh in a lot on this while they're working through these changes,” DHH told me in an email when reached for comment. “But I support them taking steps to secure and professionalize the supply chain work they're doing.”

Shopify did not reply to a request for comment. 

As this episode spread on social media, I talked to several people associated with Ruby Central who told me the board was acting in the interest of the RubyGems and the Ruby community. Two sources who asked for anonymity for fear of retaliation said that Arko was difficult to work with, questioned how he used funds raised by Ruby Together, and claimed that a new Ruby version manager he’s working on, rv, means he has a conflict of interest with his work on RubyGems and Bundler. 

Arko acknowledged to me he heard he’s been difficult to work with in the past. He said that sometimes he’s been able to reach out to people directly and resolve any issues, and that sometimes he hasn’t. He rejected the other allegations, and said that Ruby Together’s financials have always been public.

“It has always been fully public, and the amount has been fixed at $150 an hour for 10 years,” he said, referring to the amount contributors got paid to work on Bundler. Arko added that nobody has ever been paid for more than 20 hours a week, and that the most he’s been able to raise in a single year is $300,000 to pay eight different contributors. “Nobody has gotten a raise for 10 years.”

"As a matter of policy, we don’t discuss individual personnel,” Sutera, the Ruby Central spokesperson, said when I asked if Arko was removed from the GitHub organization because of his previous behavior. “Our recent actions were organization-wide governance measures aimed at aligning access with policy. Our priority is maintaining a stable and secure Ruby Gems supply chain."

McQuaid, the developer of Homebrew and who followed the controversy, told me that even Arko’s harshest critics wouldn’t deny the contributions he’s made to the Ruby community over the years. 

Regarding Arko’s blog post about his removal, McQuaid told me it’s good that Arko is crediting other people for their contribution and that he’s following open source principles of community and transparency, but that “his ‘transparency’ here has been selective to things that benefit him/his narrative, he seems unwilling or unable to admit that he failed as a leader in being unwilling or unable to introduce a formal governance process long before this all went down or appoint a meaningful successor and step down amicably.”

The fundamental disagreement here is about who “owns” the GitHub organization that houses Bundler and RubyGems. Technically, Ruby Central was able to assert control because Hiroshi Shibata, a member of the Ruby core team and one of the contributors who has owner-level permissions on the GitHub, made Haught, who revoked the others’ access, an owner as well. Any owner can add or remove any other owner, but when Ruby Central’s board voted to make this change Haught acted immediately and removed Arko, Dash, and others. 

However, Arko fundamentally disagrees with the premise that Ruby Central has the right to govern the GitHub organization in any way, and believes that it has always belonged to the group of contributors who had access up until September 19.  

Arko said that even if Ruby Central gave him his permissions back, he would not consider the matter resolved until Ruby Central stopped claiming it owns Bundler “but I am definitely not going to hold my breath for that one.”

“When people really care, they're passionate and they're enthusiastic and they argue, and that often looks like drama,” McQuaid, the developer of Homebrew, said when I asked what he thinks this entire affair says about the state of open source development. “But if I had to pick between having the enthusiasm and the drama or losing both, then I'd probably pick the enthusiasm and the drama, because in some ways, the system is somewhat self correcting. Even the stuff that's going on right now, people are having essentially a very public debate about what role do large companies or nonprofits or individual maintainers have in open source. To what extent does someone's level of contribution matter versus what type of person they are? I think these are valuable discussions to be having, and we're having them in the open, whereas if it was in a company, this would all be in a meeting room or with an HR department or in a leadership offsite or whatever.”