We Benchmarked Closed vs. Open-Source AI Models by Having Them Build Games. Here's What We Found.

We Benchmarked Closed vs. Open-Source AI Models by Having Them Build Games. Here's What We Found.

If you're building AI-powered products right now, you're probably spending more than you should. Not because you're being careless — but because the default assumption in most engineering conversations is that the best-known model is the best model for the job. We decided to test that assumption in the most concrete way we could: by making AI models build games.

Small, bounded, complete games. The kind of task that's well-scoped, repeatable, and has a clear definition of done. And the results were sharp enough that we think every technical team should see them.

What We Actually Did

We ran a visual benchmark pitting closed-source models — Claude Opus, GPT-5.5 — against a set of open-weight models including MiniMax M3, Nemotron Ultra, and Kimi K2.7 Code. Each model received the same prompts and was asked to produce functional small games from scratch. We measured three things: speed, cost per run, and output quality.

The output quality findings were what most people expect to be the story. They weren't. The real story was cost.

First, Let's Clear Up the "Open-Source" Confusion

Before we get into the numbers, a quick definitional note that matters a lot for technical decision-makers.

"Open-weight" is not the same as "open-source," and neither means free.

• Open-weight models have publicly downloadable weights. Anyone can pull them, inspect them, and run them. But running them still costs GPU time — whether you're self-hosting on your own infrastructure or routing through a third-party inference provider.
• Open-source models go further, releasing training code and data. Fewer models qualify here.
• Closed models keep weights proprietary. You access them exclusively through the provider's API, at the provider's posted per-token pricing.

The key economic difference: closed models bill at the frontier model provider's margin. Open-weight models, when served through third-party inference providers, are priced closer to raw infrastructure cost. That gap compounds fast at scale.

And if you're thinking about self-hosting an open model entirely: that's genuinely viable, but it's not free either. DevOps, scaling, model versioning, and maintenance are real engineering costs. The economics only clearly favor self-hosting at significant inference volume.

The Cost Numbers Are Hard to Ignore

Here's what the benchmark showed on cost, and these figures are not rounding errors:

• Claude Opus was roughly 15x more expensive than MiniMax M3 for equivalent game-building tasks.
• GPT-5.5 was roughly 10x more expensive than Nemotron Ultra.
• Kimi K2.7 Code was roughly 7x cheaper than Claude Opus.

Let that sink in. For a well-scoped coding task — something with clear inputs, clear success criteria, and no ambiguous judgment calls — you could run roughly 15 open-weight model completions for the price of one Claude Opus completion.

This tracks with broader market analysis. The median open-weight model costs approximately 3x less per point of measured capability than the median closed model. Open models can run 5–10x cheaper per inference when accessed through competitive third-party providers, because those providers are competing on infrastructure efficiency rather than frontier R&D margins.

But What About Quality?

Here's the nuanced part, and we want to be precise about it.

For this category of task — bounded, well-defined code generation like building small games — the open-weight models produced output of comparable quality to the closed frontier models. The games worked. The code was readable. The outputs met the spec.

Closed models still lead on frontier reasoning: multi-step logic, ambiguous judgment calls, synthesizing conflicting information, navigating truly novel problems. If you're asking a model to architect a complex system, reason through edge cases in a legal or medical domain, or handle tasks where being wrong is expensive — that's where the premium on a Claude Opus or GPT-5.5 can still be justified.

But the key word is can. It's only justified if the task actually requires that capability. And most tasks in production AI systems don't.

The Practical Lesson: Match the Model to the Task

Most engineering teams are currently making model selection decisions at the brand level, not the task level. They pick one model, integrate it everywhere, and pay frontier pricing across all their workloads — including the ones that don't need frontier capability.

A better mental model looks like this:

• Narrow, repeatable, well-scoped tasks (code generation, formatting, templated responses, classification): open-weight models via competitive inference providers. Equivalent output, a fraction of the cost.
• Frontier reasoning, high-stakes judgment, novel synthesis: closed models from top providers. The premium is real, but so is the capability delta.
• High-volume production workloads at scale: evaluate self-hosting, but budget honestly for the engineering overhead.

This isn't about being cheap. It's about being precise. Paying 15x more for a task that doesn't need 15x more capability isn't a quality decision — it's a budgeting mistake.

How We Think About This at Anablock

At Anablock, we build AI agents and an AI-first CRM called Ana. We're an official Anthropic partner, and we use Claude models at the core of our stack for the reasoning tasks that genuinely warrant them. We've seen firsthand what Claude does that other models don't — and we respect that capability deeply.

But we also evaluate this exact cost-vs-capability tradeoff every time we're deciding which model powers which feature. Not every feature needs frontier reasoning. Some need reliable, fast, cheap execution of well-defined tasks. Getting that distinction right is part of what makes an AI-powered product viable at scale — not just technically impressive in a demo.

The game benchmark is a simplified illustration of a real architectural decision that technical teams face constantly. The takeaway isn't "closed models are overpriced" or "open models are always the answer." The takeaway is: the right model is the one that matches the task, not the one with the biggest name.

What This Means for Your Team

If you're a technical founder or engineering leader building with AI right now, here are three questions worth asking before your next model selection decision:

1. Is this task well-scoped? If yes, you probably don't need frontier reasoning. Test an open-weight alternative.
2. What's the cost at production volume? A 10x cost difference per completion compounds enormously at scale. Run the numbers before you commit to an architecture.
3. Are you paying for capability you're using, or capability you're assuming you need? Benchmark it. The answer might surprise you.

We benchmarked models by making them build games because games are fun, concrete, and easy to evaluate. But the framework applies everywhere: every feature you're building has a task profile, and that task profile should drive your model selection — not inertia, not brand familiarity, and not the assumption that the most expensive option is automatically the safest one.

Ready to build smarter AI systems? At Anablock, we help technical teams architect AI agents and workflows that use the right model for every layer of the stack. If you're evaluating AI infrastructure for a product you're building — or want to understand how we make these decisions inside Ana — schedule a demo with our team and let's talk through your specific use case.