AIThis post was created with the assistance of artificial intelligence (AI).

TL;DR

A report from The Pragmatic Engineer examines how software teams can move AI workloads from frontier labs’ APIs to open-weight models hosted by inference providers. It describes a progression from multi-tenant APIs to dedicated GPUs, with cost, latency, rate limits and workload needs shaping the choice.

The Pragmatic Engineer has published a report on inference engineering, the work involved in running AI models efficiently after they have been trained. Drawing on interviews with Modal, Fireworks AI and Baseten, the report describes how software teams can shift workloads from frontier labs’ APIs to open-weight models hosted by inference providers, and what operational trade-offs that move brings.

Before you orderOffer from Amazon

Get everyday helpers delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

Inference providers sell access to models they host on customers’ behalf, generally through APIs that support streamed responses and usage-based billing. The report distinguishes them from frontier AI labs: providers host open-weight models rather than training their own frontier models. Their services can serve as API replacements, though performance and operational details may differ.

The report identifies lower costs, latency control and fine-tuning as common reasons companies consider switching. It cites previously reported cost reductions of 50% at Uber, 90% at Pinterest and 56% at AT&T after moves to open models. Those figures are examples reported by the publication; the material does not provide their measurement periods, baselines or methods, so they should not be treated as a direct comparison across companies.

Teams can start with a multi-tenant API, which is generally the simplest way to test an alternative provider. At higher workloads, rate limits, inconsistent performance, cache-hit rates and costs may prompt a closer look at dedicated GPU deployments. Running dedicated hardware gives teams more control but adds decisions about model size, throughput versus interactive latency, key-value cache capacity and failover. The report also discusses speculative decoding as a possible way to use otherwise idle compute during token generation.

At a glance
reportWhen: Published recently; the source does not…
The developmentThe Pragmatic Engineer has published a report on inference engineering practices for software teams adopting open-weight models through specialized providers.

The Shift From API Calls to GPU Operations

The report’s practical point is that changing model vendors is not only an API integration task. Teams also need to measure how a model performs under their own workload and account for tail latency, caching and rate limits. A model that is cheaper per token may not meet an application’s response-time requirements or may need a different deployment setup.

The operational choices can affect both cost and reliability. A multi-tenant service reduces the work of managing hardware, while dedicated GPUs can offer more control over capacity and performance. The report presents the latter as a step to consider when usage patterns, limits or inconsistent results make shared deployments a poor fit—not as an automatic upgrade for every team.

For software engineers, this makes inference part of the application’s production architecture. Decisions about evaluation, caching, capacity and failover can shape the user experience as much as the choice of model. The publication frames these practices as increasingly useful for engineers working with deployed LLM features.

Products Worth Considering

Jackery Explorer 1000 v2 Portable Power Station, 1024Wh LiFePO4 Battery

Jackery Explorer 1000 v2 Portable Power Station, 1024Wh LiFePO4 Battery

As an affiliate, we earn on qualifying purchases.

Open Models and Provider Choices

The report describes two broad routes to hosting open models. Companies with substantial time, funding and infrastructure expertise can operate their own GPU systems. It cites Revolut as an example, saying the company runs 56 NVIDIA B200 units for inference. The alternative is contracting with a provider that manages a larger GPU fleet and offers a hosted service.

It profiles three providers with different emphases. Modal presents itself as programmable cloud infrastructure and relies heavily on open-source software for inference. Fireworks focuses on specialized AI workloads, including dedicated deployments and fine-tuning. Baseten describes an inference platform with additional tools for agent workloads. The report says it had no affiliation or paid placement involving the companies, and credits representatives from each for sharing information.

The source also says providers generally compete on price, efficiency and customization. It notes that training a frontier-level model can cost millions of dollars, while post-training an existing open-weight model is less expensive. That distinction helps explain why a provider can host and adapt models without being a frontier lab, though it does not establish that any particular hosted model will suit a given product.

“When you run code on Modal, you should be able to see all the code that’s running.”

— Richard Gong, inference lead at Modal

Costs and Performance Need Workload Tests

The report does not provide a common benchmark comparing Modal, Fireworks AI and Baseten, or establish which provider is cheapest or fastest for a particular workload. The cited company cost reductions lack measurement windows and comparison methods in the supplied material. Those figures cannot show how results would transfer to another organization.

It is also unclear what workload levels or latency targets should trigger a move from shared APIs to dedicated GPUs. Those decisions depend on model choice, traffic patterns, service requirements and provider terms. The report describes factors engineers should examine but does not give a universal threshold, nor does it detail the full cost of operating dedicated hardware.

Evaluate Before Committing GPUs

The report’s recommended path is to use evaluations tailored to the application to select an open model, compare vendors and decide whether a multi-tenant API is sufficient. Teams can begin with an API replacement, then monitor latency consistency, cache behavior, rate limits and spending as usage grows.

If those measures indicate that a shared service no longer fits, engineers can assess dedicated deployments and plan capacity, caching and failover around their requirements. The source does not announce a specific follow-up, vendor decision or deployment schedule; any move beyond the initial evaluation depends on each team’s results.

Key Questions

What does inference engineering mean?

It refers to engineering work involved in serving trained AI models, including latency, capacity, caching and reliability decisions.

How are inference providers different from frontier AI labs?

According to the report, inference providers host and serve open-weight models, while frontier labs train their own foundational models. Both can offer API access to customers.

When might a team move from a shared API to dedicated GPUs?

The report points to rate limits, inconsistent performance or high usage costs as reasons to evaluate dedicated hardware. It gives no universal threshold; teams need to test against their workloads.

Do the cited cost reductions predict savings for other companies?

No. The source cites reductions at Uber, Pinterest and AT&T, but the supplied material does not include measurement windows or methods. Results for another organization remain uncertain.

Source: newsletter.pragmaticengineer.com

This article is for informational purposes only and is not medical advice. Always consult a qualified healthcare professional about your specific situation.
You May Also Like

Trump Administration Advances Plan For Peptides Favored By RFK Jr.

HHS has sent a peptide interim access policy to the White House. People familiar with the plan say it could let pharmacies compound some peptides temporarily.

uBlock Origin Is Giving Up The Fight To Keep Ads Off Facebook

uBlock Origin has announced it will stop blocking ads on Facebook, citing technical challenges. The move impacts ad-blocking users and privacy advocates.

Spacex Surges In Global Coverage

SpaceX’s coverage has surged globally, with mentions increasing over fourfold, highlighting rising public and media interest in the company’s activities.

LG 55-Inch Class OLED AI 4K Discontinued? The Closest Current Alternatives

Compare LG OLED alternatives, judge Black Friday discounts, and find a screen that fits your room, gaming setup, and budget.