beyond big AI

Building on my goodbye, big tech journey, and buoyed by philosophical, political, and practical thoughts on the matter, I've been thinking: can I replace Big AI in my workflow with performant, cost-effective, and energy-efficient alternatives?

I’ve been working with local models since they came out, and finally, they’re surprisingly good now.

- Vicki Boykis, Running local models is good now

This post is a deep dive on that thought, with the goal of exploring a couple of questions:

• where are open-weight models and harnesses good enough to be useful for software engineering, and where do I need Big AI frontier models?
• for open-weight models: should I self-host or use a non-Big AI provider?

As with my exploration of Big Tech alternatives, I'm also hoping to show you how you can reason through these questions for yourself.

Perhaps unsurprisingly, search results on anything related to LLMs and AI are rife with AI slop; it's rare to find deep human thought / exploration in this space. As an example: this Github repo claims to be a "curated list of resources" but just links to slop articles from a single site. By comparison, Awesome Local AI (same repo name, even!) appears to be curated by real humans.

TL;DR

Open-weight models are more than good enough for software engineering. If you're still paying the Anthropic + OpenAI duopoly, you're overpaying by 10-100x. This is a consequence of the frontier minus six dynamic, which continues to hold. We're now more than 6 months past the Nov 2025 release of Claude Opus 4.5, which is regarded as a meaningful inflection point in developer experience around AI tooling.

Open-source harnesses are likewise more than good enough. There's lots of high-quality options: pi, OpenCode, and loads of others. All your instruction files, skills, etc. will still work (though you might have to rename CLAUDE.md and copilot-instructions.md to AGENTS.md).

Self-hosting local models is not cost-competitive with open-weight model providers (e.g. OpenRouter, OpenCode Go). If your only goal is to find an alternative to Claude or Codex, $10-20 / month gets you more than enough tokens on top open-weight models. And you can't self-host those top models anyways unless you're willing to spend $40-50k.

That said, with NVIDIA entering the AI-ready hardware game and smaller local-ready models improving fast, I'd expect a shift towards self-hosting in the next couple of years, as it becomes more and more feasible to run models locally alongside normal workloads on consumer-grade machines. This reflects the frontier minus two years effect of local-ready models. If we continue to see diminishing returns on Big AI-scale training, Big AI may be short-lived as a market segment.

what's possible with open models?

Let's start with the first question above:

Where are open-weight models and harnesses good enough to be useful for software engineering?

When evaluating tech, I can't overstate the value of good enough to be useful to (insert user group / use case here). So many tech rollouts fail due to lack of clarity on what problem they're solving, which users they're solving that problem for, and what those users want. It's a bit easier when you're the user, but still worth some deliberate thought:

• good enough: especially useful in AI model selection, where using a closer-to-frontier-than-necessary model costs a lot more, often 5-10x.
• useful: my primary intended workload is inference for coding. I don't care if I can generate images, run agentic workflows in Excel, or train models from scratch. I do care about tool use, "thinking", and occasional screenshot upload.
• for software engineering: this is (so far) just for my own purposes and not for multi-user setups or resale, but I'd like to not be limited too much in the kinds of software engineering I can do.

open-weight models

One easy way to get started with open-weight models is OpenCode Go. For $10 / month, you get access to high-quality open-weight models with generous 5-hour, weekly, and monthly usage caps.

With some quick setup in opencode.json, we can configure the open-source harness OpenCode to use DeepSeek V4-Pro in plan mode, and DeepSeek V4-Flash in build mode:

{ 
  "model": "opencode-go/deepseek-v4-flash",
  "default_agent": "plan",
  "agent": {
    "build": {
      "model": "opencode-go/deepseek-v4-flash"
    },
    "plan": {
      "model": "opencode-go/deepseek-v4-pro"
    }
  }
}

Model switching is one of the best ways to reduce AI costs. If I wanted, I could define custom agents and use specific models for each, with fallback models for occasional provider outages, but this is enough to get started.

From here, I took these models for a spin on a few plan / build cycles on a couple of midsized personal projects. These models don't score quite as high on popular AI coding benchmarks as my work setup of Claude Opus 4.8 for plan / Claude Sonnet 4.6 for build, but in my quick testing I didn't notice any regression in developer experience. Perhaps this would be different for larger codebases 🤷‍♂️

local-ready models

I'm also curious about the possibility of self-hosting local-ready models at home, such as:

• qwen3.6-35b-a3b
• gemma-4-26b-a4b

Here I'm using local-ready to mean "will run at useful speeds on current high-end consumer-grade graphics cards" (as a rough benchmark: 32-64 GB VRAM).

Fortunately, you don't need to go all-in on a monster graphics card rig to test out local-ready models. OpenRouter offers Gemma 4 26B A4B for free, while Qwen 3.6 35B A3B is quite cheap; $10 worth of credits is more than enough to test these out. Let's configure the provider in OpenCode:

{
  "provider": {
    "openrouter": {
      "models": {
        "~google/gemma-4-26b-a4b-it:free": {},
        "~qwen/qwen3.6-35b-a3b": {}
      }
    }
  }
}

...and then adjust our model defaults for plan and build mode:

{ 
  "model": "openrouter/~qwen/qwen3.6-35b-a3b",
  "default_agent": "plan",
  "agent": {
    "build": {
      "model": "openrouter/~google/gemma-4-26b-a4b-it:free"
    },
    "plan": {
      "model": "openrouter/~qwen/qwen3.6-35b-a3b"
    }
  }
}

This post is long enough already, so I'll likely do a later follow-up on this investigation.

what about self-hosting?

a general note about local model economics

If you're considering self-hosting models, know that model hosting has a definite economy of scale. The main constraint is VRAM (i.e. graphics card memory), and VRAM usage scales sub-linearly in the number of users using a hosted model.

Why is that? When running a model, you use VRAM for two purposes: model weights and key-value cache (KV cache). The model weights only need to be stored once for all users, and the per-user KV cache footprint is typically much smaller than the model itself. For instance: using this VRAM calculator, with reasonable settings and a 128k context window qwen3.6-35b-a3b requires 25 GB VRAM to host for a single user, but only 30 GB VRAM to host for 8 users.

how much should I pay to self-host models?

If I were to think about this as a purely economic decision, what I'm proposing is a capital expenditure to own something I'd otherwise have to rent. In exchange for upfront spend on computer equipment plus ongoing spend on electricity, I save that rent.

And that rent can be considerable. From Apr-Jun 2026 I spent $1532.07 on tokens at Anthropic API rates through LEGO's internal model proxy, or roughly $500 / month. I've also got a Github Copilot enterprise seat through LEGO, which costs $39 / month.

My spend breaks down into four tiers of use: plan, build, suggest, review.

• Opus 4.8 (plan): $334
• Sonnet 4.6 (build): $146
• Haiku 4.5 (suggest): $20
• Github Copilot (review): $39

This has become a fairly widespread workflow for AI-assisted engineering: you use a heavyweight model to plan out changes, a midweight model to build those changes in code, and a lightweight model for smaller suggest tasks (e.g. inline autocomplete). The output then gets run through a heftier model again for first-pass code review, before going to normal human review.

We can also see the disproportionate cost of plan tier models, which is exactly why cost-conscious engineers and organizations encourage different models for different purposes. 60-70% of my cost is plan, even though this is maybe 20-30% of my usage.

Granted, this spend is small by some standards. Uber recently capped employee token spend at $1500 / month / tool, and well-known AI commentator Simon Willison estimates his own spend at $2000 / month ($1000 / month on each of OpenAI, Anthropic).

If I were, say, a senior engineer rather than an engineering leader, I'd be spending ~80% of my time on coding tasks rather than ~30-40%. So I'll assume 2x my current token spend as a reasonable benchmark for software engineering workloads:

• plan: $670
• build: $290
• autocomplete: $40
• review: $40

shopping around for paid alternatives

But wait. At work, I'm limited to a handful of costly frontier models from big, supposedly reputable providers. For my own purposes I'd be more likely to e.g. buy credit off of OpenRouter to use with their extensive library of programming models. Some quick price comparisons on a few plan tier models from the best open-source / open-weight models as of June 2026:

• Opus 4.8 (my baseline): $5/M input, $25/M output ($9/M assuming 80% input, 20% output)
• GLM 5.2: $0.95/M input, $3/M output ($1.36/M)
• MiniMax M3: $0.30/M input, $1.20/M output ($0.48/M)
• Kimi K2.6: $0.66/M input, $3.41/M output ($1.21/M)
• DeepSeek V4-Pro: $0.435/M input, $0.87/M output ($0.52/M)

So an engineer could cut 80-95% of their plan cost simply by picking a different model. (As we'll see below, the same is true for code and build, and the possible savings are even higher.)

In case you're wondering: you cannot run these locally. For instance, this VRAM calculator shows that you'd need ~2.8 TB of VRAM (i.e. graphics card memory) to run DeepSeek V4-Pro at its maximum context window of 1M tokens, which is roughly 100 high-end consumer-grade GPUs worth of VRAM:

What makes a model local-ready, i.e. suitable for self-hosting? As a rough ballpark for mid-2026: model + context can be hosted in 16-32 GB VRAM.

What about my build, suggest, and review usage? Again, I wouldn't personally pay Anthropic API rates, and would look for something cheaper on OpenRouter. If I use DeepSeek V4 Flash (most popular model on OpenRouter as of June 2026) for both build and suggest, and DeepSeek V4 Pro for plan, then with the 5.5% platform fee for OpenRouter I'd be paying:

• plan: $40
• build: $6.10
• suggest: $2.50

Finally, review tools are roughly $15 / month, for a total of $63.60 / month.

So we've already saved 94%, which is impressive! That said, I can get this even lower with OpenCode Go, as explored above. A single subscription likely won't cover normal engineering workloads, but tools like opencode-go-multi-auth allow you to switch seamlessly between multiple subscriptions. Let's assume 2 subscriptions are enough to start with, for a cost of $20 / month.

We can also get rid of that review cost: you can get surprisingly far with a CI pipeline that uses a custom review agent together with review-oriented .md instruction files, and perhaps a review skill to explain the intended process and structure. My team at work, for example, has a review skill that instructs Github Copilot to categorize review comments into severity / importance levels, and add clear text (e.g. 🟢 low, 🔴 critical) that build and plan agents can use to help in comment triage.

Final cost: $20 / month, or 50x cheaper than paying the duopoly.

depreciation

Unfortunately, computer hardware doesn't last forever. Tax codes account for this through depreciation: capital assets (like computers!) are assumed to lose value over a fixed period. For example, the IRS depreciates computer equipment over a period of 5 years. So let's assume we'd have to replace the whole thing in 5 years.

To break even, then, we should not pay more than $1200. This includes:

• hardware
• electricity costs
• rental costs for anything I can't run locally

a brief aside on intangible values

As mentioned, the most important reasons to explore local models have nothing to do with cost.

There's digital agency: the idea that my tools should work for me, not the other way around. For me, part of this is tool ownership: I don't want to rent my core professional tools from Big Tech, especially not when duopoly providers are looking to squeeze additional revenues out of their user, and especially not when those same providers are beholden to an administration that is perfectly willing to capriciously withhold access.

There's also the value of learning and growth, something I alluded to in my recent article. At this point, it seems undeniable that LLMs are powerful tools in the engineering toolkit. I'm curious to learn how to use them well, both for myself and for the engineers and teams I lead. More importantly: from my analysis here, we're not far off from the point where local-first inference becomes the default. So I'm also investing in learning what that means.

(And, of course, there's the geopolitical considerations right now, but I won't go too much into those here.)

Still, given the availability of model hosting providers outside Big AI, I'm more than happy to try out local-ready models through those providers before making the jump. As we'll see below, the point where that jump makes sense is likely coming soon.

picking hardware

Now that we've figured out the budget, let's approach this from the other end. What models do we want to run, and what hardware do we need to run them?

models

Let's revisit local model recommendations for June 2026:

• qwen3.6-35b-a3b
• gemma-4-26b-a4b

This article suggests that these models can easily replace build and suggest usage. My own exploration above shows that they can also replace some plan usage, and are good enough for review.

Assuming that we still have to keep 1 OpenCode Go subscription around for 50% of our plan usage, that means we have $10 / month in continued rental costs, or $600 over 5 years.

VRAM: model + context window

Looking back at that VRAM calculator: if we assume Q4_K_M model quantization and FP8 KV cache quantization plus 256K context window, we get:

• qwen3.6-35b-a3b: 31 GB VRAM
• gemma-4-26b-a4b: 23 GB VRAM

If I'm willing to reduce the context window to 128K, we could run either on a single RTX 5090 (32 GB VRAM) graphics card. The problem here is that RTX 5090 card prices are massively inflated right now due to - you guessed it - AI hype. Current prices in Denmark are over 30000 DKK ($4500), which is way outside the budget:

There's also the option of using 2x RTX 3090 / 4090 cards (48 GB VRAM). The 4090s are again too expensive here. Local Craigslist-style site DBA has listings for 16000 DKK ($2450):

This is also just the graphics card, mind you. I'd still need everything else: power supply, motherboard, RAM, SSD, etc.

Another option is Apple Silicon. These are much more power-efficient than RTX cards (which we'll see below). Here we can get an M4 Max for $2499; this has 36 GB of unified memory, which is used as both RAM and VRAM. This is upgradeable to 64 GB for a total cost of $3499.

NVIDIA is also entering the pre-built unified memory hardware market with their upcoming RTX Spark chip, which offers 128 GB unified memory.

electricity

My wife and I are customers of Vindstød, one of the many wind power providers here in Denmark. From my most recent bill, we pay 1.1120 DKK / kWh on average.

For the RTX build, I'd need 2x RTX cards. RTX 4080s draw 320W peak, for a total of 640W. If I assume these would run 6 hours a day, that's 128 DKK / month, or 7680 DKK ($1200) over the 5 year lifespan. That's equally as expensive as renting model access for the whole 5 years! Also a lot of heat in a small apartment 🥵

The M4 Max, by comparison, draws 145W peak. Under the same assumptions: 29 DKK / month, or 1740 DKK ($270) over 5 years.

RTX Spark target power draw is 110W peak, which is even lower: 19.80 DKK / month, or 1188 DKK ($180) over 5 years.

financial verdict: 🤔

M4 Max 36 GB unified memory comes in at $3369:

• computer: $2499
• electricity: $270
• continued rent: $600

The higher end of RTX Spark has a current estimated retail price at $2899, for an all-in cost of $3679. So these machines are 2-3x as expensive as simply renting model usage from non-Big AI providers.

RTX 30xx/40xx/50xx builds are significantly more expensive than pre-built unified memory machines. Not to mention that, once you somehow procure the card (probably second-hand), you still have to build the rest of the computer. I've built computers before, and it's fun, but it can also be a challenge to plan out builds around bulky components like cutting-edge consumer GPUs.

Plus the pre-built hardware is notably more power-efficient, which means lower electricity bills and less waste heat / noise. Unless you really like building highly specialized computers from scratch (which is a valid preference!) I'd probably recommend pre-built over custom-built.

But still: unless you're interested specifically in experimenting with local-ready model operation (e.g. to learn more about settings, internals, etc.), you can get the benefit of running much better open-weight models for much cheaper, and I'd recommend just waiting until model self-hosting capability becomes default on consumer machines (I'd estimate 2-3 years from now).

As I've previously mused, I'm deeply ambivalent about my growing use of AI. As a tool, LLMs can save us from toil and boilerplate, and make new approaches to early prototyping possible; short-sighted leaders can (and do) use them as an excuse to devalue creativity, craft, and learning. For many engineers, AI means frontier models hosted by Anthropic or OpenAI, sitting on hardware owned by a no-longer-trillionaire with a penchant for Nazi salutes.

In researching this post, I've seen that this duopoly is largely illusory: there are plenty of viable options outside Big AI that are much, much cheaper. Unless you have specific compliance reasons to stick with the duopoly (e.g. in risk-averse enterprise settings), you can switch now and get good enough quality at 1-10% the cost. That first-mover moat is looking real thin.