GitHub Unveils Rubicon, a New Git Storage Architecture Built to Let Thousands of AI Agents Push to One Repository at Once

What happened

GitHub's official engineering blog says it announced a new Git storage architecture at GitHub Universe last week. GitHub says the architecture can handle thousands of writes per second, while read capacity can scale on demand. A repository is where a project's code and change history live; a push uploads local code changes to GitHub. GitHub says internal tests show the new architecture can support 5,000 agents working on a single repository at once, pushing 7,200 times per second, with read latency staying steady and low. All of these figures are test results published by GitHub itself.

GitHub explained why it is rebuilding the system. Today, the busiest repositories on GitHub.com see at most a few dozen writes per second, usually in short bursts. As agents work in parallel around the clock, GitHub expects those bursts to turn into sustained high write volume within a year. Each push also triggers more reads, for example when automated testing and deployment systems pull new commits. GitHub therefore set a target of at least two orders of magnitude (roughly 100 times or more) beyond today's scale.

How the new architecture works

GitHub says the key design observation is that Git objects (file contents, commit records and similar data) never change once written and are named by their content, so multiple writers can store them at the same time without coordinating. Only reference updates need coordination; a reference is a pointer such as which commit a branch currently points to. Because only references need coordination, storing data, serving requests and maintaining repositories can each scale independently.

• Objects: GitHub says packs (files in which Git bundles many objects together) and the acceleration data that helps reads are stored in the cloud object storage service Azure Blob Storage.
• References: references and the pack list are stored in a replicated log service called Ref-CAS. Ref-CAS stands for Reference Compare-and-Swap, meaning a reference is updated only if it still holds the value the writer expected. Each update takes effect only after a majority of replicas have recorded it.
• Virtual replicas: replaceable compute resources that handle reads and writes. Their local disks are only rebuildable caches, and more replicas are added when a repository gets busy.
• Compaction nodes: here, compaction means merging files, not deleting content. Compaction nodes work directly against the storage service in the background, merging many small packs into fewer, larger ones. All objects are kept, and the work does not consume the replicas that serve requests.

What it means for readers and developers

GitHub says that for developers, Git is still Git: pushing, fetching (downloading the latest changes) and branching work the same way, and protections remain in place. GitHub's goal is for every repository, from small personal projects to the busiest large repositories, to run on the same architecture with no special handling.

On timing, GitHub says the architecture is currently open only as a limited private preview, with a broader rollout expected early next year, so most users will not notice a direct change in the short term. GitHub also lists follow-up plans, including reading only the data that is needed, partial replicas, global replicas, an atomic commit API (submitting changes to multiple files at once, where either everything succeeds or the whole change is rolled back) and event streaming. These are all still just plans.

FAQ

Browse the latest news in this topic

• Tech news