How GPU compute is priced.

On-demand, reserved and spot: what a GPU-hour buys, and what moves its price.

Carry Research6 min read

Most AI runs on rented hardware. Companies that train or serve models rarely own every GPU they use. They rent capacity from cloud providers, specialist GPU clouds and data center operators. That rent is one of the largest costs in AI, and the way it is quoted can be confusing at first.

The unit: a GPU-hour

Compute is usually priced per GPU per hour. Renting one GPU for one hour is one GPU-hour; renting a server with eight GPUs for a day is 192 GPU-hours.

A GPU-hour is not a standard product, though. Its price depends on:

  • The chip. Each generation does more work per hour than the last, and newer chips command higher prices.
  • The server around it. GPUs sit in servers with processors, memory and local storage. Most training runs on servers of eight GPUs joined by fast links.
  • The network between servers. Large training runs spread across hundreds or thousands of GPUs and need a fast network to connect them. Clusters built for this cost more than standalone servers.
  • The location. Where the hardware sits affects the price of power, the distance to users and where data may be stored.
  • What is included. Storage, data transfer, support and software may be bundled in or billed separately.

Two quotes for an hour on the same chip can describe quite different things. Comparing them starts with checking what each one includes.

What goes into the price of an hour

Behind every hourly rate is the cost of owning and running the hardware, spread over the hours it is expected to be rented.

  • The hardware. GPUs and the servers around them are the largest single cost, paid up front and recovered over the years the hardware is expected to earn.
  • Power and cooling. A modern AI server draws many kilowatts, around the clock, and the heat it produces has to be removed.
  • The building and network. Space, power connections, networking equipment and the people who run them.
  • Financing. The cost of the money borrowed to pay for all of the above.
  • Idle time. Hours that are not rented still cost money, so the rate has to cover them too.

Utilization is the lever that matters most. An operator that rents its GPUs 90% of the time can charge less per hour than one that rents them 60% of the time, and still earn the same return.

Three ways to buy

Most GPU capacity is sold in one of three ways.

  • On-demand. Pay by the hour, and start and stop at will. It is the most flexible way to buy and usually the most expensive, and popular chips are not always available on demand.
  • Reserved. Commit to a fixed amount of capacity for a set period, from weeks to several years, in exchange for a lower rate. Large AI labs and companies serving models at scale buy most of their capacity this way, often before the hardware is installed.
  • Spot. Use spare capacity at a discount, accepting that the provider can take it back at short notice. It suits work that can pause and resume, and not work that must run without a break.

Between these sit newer arrangements. Some cloud providers let customers book GPU capacity for a fixed block of time in advance, at prices that move with supply and demand. Marketplaces match buyers with operators that have capacity to spare.

What the contract says

For reserved capacity, the hourly rate is only one of the terms that matter.

  • Commitment. Many contracts are take-or-pay: the buyer pays for the reserved hours whether it uses them or not.
  • Reliability. Service levels set how much uptime the provider guarantees and what credits apply when it falls short. For a large training run, a failure in one part of a cluster can stall the whole job.
  • Delivery. When the capacity becomes available, and what happens if it arrives late.
  • Flexibility. Whether the buyer can resell or sublet the capacity, and whether it can move to newer chips.

Two contracts at the same hourly rate can carry very different risks. When comparing quotes, it helps to check five things side by side: the chip and its memory, the server and network configuration, what is included, the length of the commitment, and the service terms.

What moves the price

GPU rental prices change for familiar reasons: supply, demand and the cost of providing the service.

  • New chips. When a new generation arrives, it offers more work per hour, and the rent on older chips tends to fall as buyers move up.
  • Finished capacity. A GPU earns nothing until it is installed in a data center with power, cooling and networking. A shortage of any of these limits how much capacity reaches the market.
  • Training and inference demand. Large training runs can take huge blocks of capacity at once. Growing use of AI products adds steady demand for inference.
  • Power. Electricity is a large share of the cost of running a data center, and access to power increasingly decides where new capacity can be built.
  • Financing. Data centers and GPUs are often bought with borrowed money. The cost of that money, and how many years the hardware is expected to earn, shape the rates operators need to charge.
A GPU-hour is the price of a slice of a data center: the chip, the server, the network, the power and the place.

Training and inference buy differently

Training a large model needs thousands of GPUs working together for weeks or months, joined by a fast network. Buyers need large, tightly connected clusters, usually reserved well in advance, and they pay for the network as much as for the chips.

Inference is more forgiving. Requests can be spread across many smaller servers, in many places, and capacity can be added or removed as demand changes. Inference buyers care more about capacity close to their users, reliable service and the cost of each token served. As AI products grow, inference is taking a larger share of demand for compute, which changes the kind of capacity the market needs.

Why it matters beyond the invoice

For a buyer, the rental price is a cost to budget. For the operators and lenders behind the hardware, it is the income that repays the investment. A data center that buys GPUs today needs rental income for years, while the market rent for those hours may fall as newer chips arrive.

That makes the price of compute a risk that both sides carry for a long time, and one that grows as more capital flows into AI infrastructure. Managing the price of GPU compute looks at who carries that risk, and how forwards, futures and options can help manage it.

From compute to inference

Compute is the input; inference is what many buyers end up paying for. A company that calls a model through an API never sees a GPU-hour. It pays per token, and the provider turns GPU-hours into tokens. How efficiently that happens depends on the model, the software and how busy the hardware is kept.

The two prices are linked, but they are not the same, and they do not always move together. How AI inference is priced explains the token side. Carry Exchange brings markets for GPU compute and AI inference together in one place.

This article describes common pricing practices in general terms. It does not refer to any provider's prices; each provider's own terms apply.

Newsletter

New research, in your inbox.