Open Source Models Close the Gap on Closed AI

What Running Your Own Model Costs, and When It Genuinely Makes Sense

Two years ago the gap between the best openly available model weights and the best commercial API was wide enough that the comparison barely mattered for serious work. That gap has narrowed sharply. Open-weight models now handle summarisation, extraction, classification, translation, and a substantial share of everyday coding at a quality most users cannot distinguish from a paid API.

This raises a practical question that many teams answer badly: should you run your own model? The honest answer depends on volume, latency requirements, privacy constraints, and how much of your own time an operations burden is worth. This article works through the arithmetic rather than the ideology.

What Open Weights Actually Means

Terminology in this area is used loosely, often deliberately. Three distinct things get called open, and the differences have real consequences for what you are permitted to do.

Category	What you get	What you can do
Fully open source	Weights, training code, and dataset documentation, under a recognised open licence	Anything, including commercial use, modification, and redistribution
Open weights	Downloadable weights under a custom licence; training data usually undisclosed	Usually commercial use with conditions, such as scale limits or naming requirements
Open access	API access only, sometimes with a free tier	Use the service under its terms; you cannot run it yourself

The practical distinction that matters for most people is simply whether you can download the weights and run them on hardware you control. If you can, you gain privacy, price predictability, and independence from a vendor's roadmap. If you cannot, none of the arguments in this article apply.

Where the Gap Has Closed, and Where It Has Not

Blanket statements in either direction are wrong. The gap is task-dependent, and knowing the shape of it lets you route work sensibly.

Task category	Open-weight quality	Verdict
Summarisation and rewriting	Essentially equivalent for most content	Use open weights
Structured extraction and classification	Equivalent, often faster after tuning	Use open weights
Translation into major languages	Very close; occasional idiom weakness	Use open weights, review sensitive output
Everyday code generation	Close for common languages and patterns	Use open weights
Long-document analysis	Behind, especially past long context lengths	Depends on document size
Multi-step reasoning and hard debugging	Noticeably behind the frontier	Use a frontier API
Agentic multi-tool workflows	Behind on reliability across many turns	Use a frontier API
Niche domain expertise	Behind, and hallucinates more confidently	Use a frontier API with retrieval

The interesting question stopped being which model is best and became which model is sufficient for each specific job. That is a much more useful question, and it usually has a cheap answer.

Tooliqo Editorial

The Real Cost of Self-Hosting

The pitch for self-hosting is that inference becomes free. This is true of the marginal token and false of the total cost, and the difference is where most projects go wrong. Here is what actually appears on the bill.

• Hardware or rental → a capable accelerator costs meaningful money to buy or several dollars an hour to rent. The rental clock runs whether or not requests are arriving, which is the detail that ruins most cost comparisons.
• Utilisation → API pricing is per token, so idle time is free. Self-hosting is per hour, so idle time is pure loss. Below roughly forty per cent utilisation, self-hosting is usually more expensive.
• Engineering time → serving infrastructure, batching, quantisation, monitoring, and upgrades. This is the cost people forget, and for a small team it typically dwarfs the compute.
• Opportunity cost of staleness → a hosted API improves without you doing anything. A self-hosted model stays exactly as capable as the day you deployed it until you invest effort in migrating.
• Reliability → you now own uptime. Providers have teams for this; you have yourself, at whatever hour it breaks.

A Concrete Break-Even Calculation

Suppose you process one million tokens of input and two hundred thousand tokens of output per day, a realistic figure for a moderately busy application. Compare an efficient hosted API tier against renting a single mid-range accelerator.

Option	Monthly compute	Monthly engineering	Effective total
Hosted API, small tier	~$40 to $90	Near zero	~$40 to $90
Rented accelerator, 24/7	~$700 to $1,400	5 to 15 hours of your time	~$1,200 to $3,000 equivalent
Rented accelerator, scaled to demand	~$200 to $500	15 to 30 hours plus autoscaling complexity	~$1,000 to $2,500 equivalent
Owned hardware, amortised	~$150 to $400 amortised	Setup plus ongoing maintenance	Competitive above high, steady volume

Quantisation and What Runs on Consumer Hardware

Model weights are stored as numbers, and those numbers can be stored at reduced precision with surprisingly little quality loss. Quantisation is what makes local inference practical on ordinary machines, and understanding it roughly is enough to make good choices.

Precision	Memory per billion parameters	Quality impact
16-bit	~2 GB	Reference quality
8-bit	~1 GB	Essentially indistinguishable in most use
5-bit	~0.7 GB	Slight degradation, acceptable for most tasks
4-bit	~0.55 GB	Noticeable on reasoning; fine for summarising and extraction
3-bit and below	~0.4 GB	Significant degradation; rarely worth it

The practical upshot: a machine with sixteen gigabytes of unified or graphics memory can comfortably run a well-quantised model in the seven-to-fourteen-billion parameter range, which is genuinely useful for drafting, summarising, extraction, and code completion. Thirty-two gigabytes opens up substantially more capable models. Beyond that you are into workstation and server territory.

• Start with 8-bit → if it fits, there is little reason to go lower.
• Prefer a smaller model at higher precision → over a larger model crushed to three bits. The smaller, less-quantised option usually wins in practice.
• Budget memory for context → a long context window consumes additional memory beyond the weights themselves, and this catches people out.
• Measure on your own tasks → published benchmarks correlate imperfectly with whether a model handles your specific work acceptably.

A Decision Framework

Four questions, answered honestly, will settle this for almost any project.

1. Can the data leave your infrastructure?If no, the decision is made and cost is irrelevant. Self-host, and budget accordingly rather than pretending it will be cheap.
2. What is your sustained volume and utilisation?Calculate tokens per day and what fraction of the day you would actually be using the hardware. Below roughly forty per cent utilisation, an API is almost certainly cheaper once your time is priced in.
3. Does the task sit in the closed part of the gap?Multi-step reasoning, agentic reliability, and very long context still favour frontier APIs clearly. Summarising, extracting, classifying, and routine code do not.
4. Who maintains it in six months?If the answer is nobody specific, self-hosting will decay into an unmaintained dependency running an increasingly outdated model. This question kills more self-hosting projects than cost does, usually a year after launch.

The Hybrid Pattern Most Teams End Up With

After trying one extreme or the other, most teams converge on the same arrangement, and it is worth adopting deliberately rather than discovering it slowly.

• A local or self-hosted open model for volume → classification, tagging, extraction, first-pass summarising, and anything touching sensitive data. High volume, low stakes, no per-token cost.
• A frontier API for the hard minority → complex reasoning, final drafts of important documents, debugging subtle problems, and any agentic workflow that must hold together across many steps.
• A cheap router in front → a small model or even simple heuristics deciding which path each request takes. This is the component that makes the arrangement pay.
• One abstraction layer → so that swapping either side is a configuration change rather than a rewrite. Given how fast this field moves, portability is worth more than any individual model choice.

This arrangement gets you privacy and cost control where volume is high, frontier capability where it matters, and the ability to change your mind when the landscape shifts again, which it will. The last property is the most valuable and the one teams most often neglect while optimising the other two.

Key Takeaways

• Open weights and open source are different things; read the actual licence.
• The gap has closed for summarising, extraction, translation, and routine code, and persists for reasoning and agentic reliability.
• Self-hosting saves money only at high, steady utilisation once your time is counted.
• Privacy and control are sound reasons to self-host; claimed savings usually are not.
• Prefer a smaller model at 8-bit over a larger one crushed to 3-bit.
• Most teams end up hybrid, and adopting that deliberately beats arriving there slowly.

Frequently Asked Questions

Which open model should I start with?

Whichever current release fits your memory budget in the seven-to-fourteen-billion parameter range, because that class has the best capability-per-gigabyte and the widest tooling support. Rather than chasing the specific name, which changes every few months, set up your evaluation on your own tasks first. Then swapping models becomes a twenty-minute experiment instead of a research project.

Is fine-tuning worth it?

Less often than people expect. Try a good prompt with a few examples first, then retrieval over your own documents, and only then fine-tuning. Fine-tuning helps most with output format and tone consistency at high volume, and helps least with adding knowledge, which retrieval does better and updates more easily.

Can I run a useful model on a laptop with no dedicated graphics card?

Yes, with realistic expectations. A recent laptop with unified memory or a reasonable amount of system RAM will run a small quantised model at a usable pace for drafting and summarising. It will feel slow compared with an API and it works offline, costs nothing per token, and keeps everything local, which for some work is exactly the trade you want.

Will open models catch up completely?

On the tasks that most people need most of the time, they are already close enough that the distinction has stopped mattering. At the genuine frontier of reasoning there is a structural lag, because the frontier is defined by whoever spent the most on the newest training run. Expect the useful-for-most-purposes gap to keep shrinking and the absolute-frontier gap to persist.

Keep Reading

• The Generative AI Revolution in 2026News
• The Edge Computing and Local AI RevolutionTechnology
• The Global Chip Race Is Redrawing the Technology MapNews
• The Best AI Tools for Creators and MarketersTools
• Using ChatGPT in Digital Content DevelopmentTechnology