What Gemini 4 Carbon's 2x, 10x and #4 numbers measure
Key takeaways
- Gemini 4 Carbon's 2x compares it with Argon, Google's own model rolled out two weeks earlier, so it measures progress inside one model family.
- The 10x is the Chrome team's claim about one job, generating high-quality patches, against non-Google models that the coverage doesn't identify.
- The #4 comes with a real gain: Carbon rose 89 Elo points to 1768 on WebDev Arena and still ranks fourth there.
- Independent numbers put Carbon a few points ahead of other frontier models, which fits with all of the headline claims being true at once.
- Before trusting any AI benchmark, ask who measured it, what it was compared against, which task it covers and whether you can try the model yourself.
What do the Gemini 4 Carbon headlines actually say?
Google announced Gemini 4 Carbon two weeks after it rolled out Gemini 4 Argon, and the coverage came with 2x, 10x and #4 attached. We'll take them one at a time and look at what each is measured against, because that decides what the number means.
Carbon has improved. Koray Kavukcuoglu, SVP at Google DeepMind, said it represents the company's strongest model yet, particularly in cybersecurity and long-running tasks. We're not disputing that. The question is how far each headline figure reaches.
- 2x: Google says Carbon is twice as likely as Argon to generate code free of vulnerabilities.
- 10x: Google's Chrome team says Carbon is 10 times better at generating high-quality patches than non-Google models.
- #4: Carbon jumped 89 Elo points to 1768 on WebDev Arena, which puts it fourth.
What does the 2x in Gemini 4 Carbon's launch compare?
Google says Carbon is twice as likely as Argon to generate code free of vulnerabilities, as Ars Technica reports it. Argon is Google's own model, and Ars Technica also notes that Google announced Carbon two weeks after rolling out Argon. So the 2x describes a step inside one model family.
Comparing a model with its predecessor is normal practice, and it's real progress. What the 2x can't do is tell you how Carbon stacks up against models from other labs.
There's a second gap, and it's plain arithmetic. Twice as likely means very different things depending on the starting rate. Twice a small chance of clean code is still a small chance. Twice a middling one is a big deal. A ratio on its own doesn't say which case you're in, so it's worth looking for Argon's own rate before deciding how impressed to be.
What does the 10x patch claim compare against?
The Chrome team, which is part of Google, claims Carbon is 10 times better at generating high-quality patches compared to non-Google models. That's the wording Ars Technica uses. The Register reports the same team saying Carbon performed more than 10x better than another lab's large frontier model.
The outlets word the comparison differently: Ars Technica says non-Google models, The Register says another lab's large frontier model. We can't tell from the coverage which is more exact. Either way, a comparison against a model you can't identify can't be checked from the outside.
The claim also covers one job, generating high-quality patches, meaning code changes that fix a problem. A team measuring its own model on a task it picked is a common way to produce a launch number. It's a useful signal about that task. Treat it as a thinner signal about everything else.
Is a #4 ranking good news or bad news?
Both. Google's blog says Carbon jumped 89 Elo points to #4, at 1768, on WebDev Arena. An 89-point gain is a real improvement. Fourth place is still fourth place. A large gain and a fourth-place finish can sit in the same launch post without either one being wrong.
Elo, in general terms, ranks models by head-to-head comparisons: models are matched in pairs, one comes out ahead, and the ratings move. On arena-style leaderboards the matchup is decided by which answer people prefer, which overlaps with correctness but isn't the same measure. We won't turn the 89 points into a win rate.
What the #4 leaves out is how close the models above Carbon are. It tells you where Carbon sits on one leaderboard right now, and it's a measure of preference on web-development tasks.
What do independent tests say about Gemini 4 Carbon?
Ars Technica says Carbon's gap over competing frontier models is just a few points, since every frontier model scores in a similar range. Artificial Analysis puts numbers on it. Carbon scores 54 on AA-Omniscience v1.1, 6 points ahead of Claude Opus 5.5 at its max setting, which scores 48.
On AA-Briefcase, Carbon (high) scores 1609 Elo, a 126-point improvement over Argon (high), which scores 1483. That's another Argon comparison, but from a third party, and it fits Google's story that Carbon is a real step up from Argon. Artificial Analysis also puts Carbon at $7.76 per Intelligence Index task at standard pricing, about 23% less than Claude Opus 5.5 at max.
So a large jump over Argon and a lead of a few points over the field can both be true. One asks how far Carbon moved from Google's previous model. The other asks how far ahead of everyone else it sits.
Fair objections apply. Artificial Analysis has its own methods and limits, like every benchmark, so we treat it as a second source and not the last word. And we'd argue a few points can matter a lot at the frontier, especially on security work. The figures here can't tell us how much.
Can anyone check Gemini 4 Carbon first-hand yet?
Not broadly. According to The Register, Google will release Carbon this week to select cybersecurity defenders and aims to make it more broadly available as soon as possible. Ars Technica adds that the release is staged and could take a while.
Nothing in this article is first-hand testing of Carbon. We're reporting what Google and independent testers have published, and we'd put any early claim of hands-on results through the same questions below. As access widens, more people can run their own tasks, and that's when these numbers get stress-tested.
What four questions should you ask of any AI benchmark?
In our view, four questions cover most of what a launch number leaves out. Carbon's 2x, 10x and #4 give different answers to them.
Who measured it? The 2x is Google's claim, the 10x is the Chrome team's, and the #4 is a WebDev Arena result reported in Google's post. Compared with what? The 2x is against Argon, the 10x is against non-Google models, and the #4 is against the other models on the leaderboard. Which task? Vulnerability-free code, then patch generation, then web development. Can you try it yourself? For now, mostly no.
The same questions help when you're picking a model for your own work. A benchmark tells you how a model did on someone else's task. The result that matters to you is how it does on yours, which is why we start from your own tasks in how we approach AI implementation.
- Who measured it?
- What was it compared against?
- Which task does it cover?
- Can you try the model yourself?
Frequently asked questions
Is Gemini 4 Carbon really twice as good at secure code?
Google says Carbon is twice as likely as Argon to generate code free of vulnerabilities. That's a comparison with Google's own earlier model, not with other labs' models. How much it matters depends on Argon's starting rate. Ars Technica says Carbon's lead over competing frontier models is just a few points.
What is Gemini 4 Argon, and why does it matter for the 2x claim?
Argon is Google's own model, rolled out two weeks before Google announced Carbon. The 2x claim compares Carbon with it, so it measures progress within one model family. That's a legitimate way to show improvement, but it doesn't say how Carbon compares with models from other labs.
Can I use Gemini 4 Carbon today?
Probably not yet. Google is releasing Carbon this week to select cybersecurity defenders and aims to make it more broadly available as soon as possible. Ars Technica reports the staged release could take a while, so wider access may be some way off.
How do I tell if an AI benchmark claim is hype?
Ask four things: who measured it, what it was compared against, which task it covers, and whether you can try the model yourself. A multiplier without a named comparison is hard to judge. With Carbon, the 2x is against Google's own Argon, the 10x is against non-Google models, and the #4 is a leaderboard rank.
Why do independent numbers show a smaller gap than the headlines?
Headline multipliers compare against a specific baseline, such as a previous model or an unnamed rival. Independent tests compare against the field. Ars Technica says every frontier model scores in a similar range, so Carbon's lead is a few points. Artificial Analysis has it 6 points ahead of Claude Opus 5.5 on AA-Omniscience v1.1.