The market for GPU cloud infrastructure is a mess.

‍

It is flooded with providers of various types and claimed value propositions. As a seller in this market, it’s challenging, which got me thinking about how challenging it must be for customers navigating this ecosystem. Not all clusters are real, not all timelines are honest, not all pricing is stable, and not all vendors own what they're selling.

‍

This guide is intended to help cut through the noise by providing perspective from the other side of the transaction. Perhaps by sharing what we see and experience, it will provide context that will help those looking for cloud GPUs more efficiently navigate, negotiate, and ultimately secure the right partner to deploy and operate your expensive and fragile infrastructure.

‍

What should companies look for when procuring GPU infrastructure?

‍

Let’s talk about it.

‍

‍

Vendor Credibility: What Value Are They Actually Adding?

The neo-cloud ecosystem comprises four general types of companies. There are those that offer value through on-demand services for early stage testing and experimentation. Others who offer value through novel software layers that simplify model deployment, orchestration, or other services that reduce overhead on your teams. There are some, like us, who focus on delivery of bare metal infrastructure optimized based on workload and security and/or compliance requirements. Finally there are brokers who do none of the previously mentioned things, but do promise to bridge the gap between customers and vendors.

‍

Where things start to get more complicated is when overlap occurs between these cohorts, intentionally or otherwise. Not long ago I was in a conversation with a partner about GPU capacity for one of his customers. He mentioned a similar conversation with another vendor and we quickly came to realize that the cluster we were discussing was the same cluster the other vendor was trying to sell. Two vendors, one cluster, one customer negotiating against themselves for infrastructure we own and operate. Not a great situation for anyone, including the entity trying to make margin on a finder’s fee tied to the number of GPUs/hr.

‍

We’ve been involved in conversations with five and more vendors on a single call. All of these folks are trying to figure out how they can claim their small piece of a very large pie. These layers result in added cost to the customer. For those who are doing the work, it means the chance of closing a deal diminishes with each new involved party incrementally adding cost per GPU/hr, but not adding value.

‍

We believe in partnership, co-selling, and mutually beneficial relationships with other neo-clouds, hardware providers, OEMs, data centers, and financing partners. We have a small set of trusted partners we work with regularly. The key traits of these organizations are transparency, a customer focused approach, and tangible value for the customer. When done right, partnering across the ecosystem has great outcomes for all involved.

‍

How can you tell if the vendor you are speaking with is credible?

‍

‍

‍

‍

Is the Cluster Real? How to Verify GPU Availability Before You Pay

We had a funny conversation with a customer recently. They were asking for an MSA covering a cluster deployment we’d been discussing with them. They were shocked by our response.

‍

“We won’t give you an MSA until we have the entire supply chain and power locked down.”

‍

That should not be a shocking reply. First calls with prospects more and more frequently start with requests to demonstrate whether or not what we are selling is real. Rightly, we are asked for data center agreements, POs with OEMs, anything that shows proof of cluster life (or near-future cluster life.)

‍

We refer to this as the ‘Phantom Cluster Phenomenon.’ There are many causes for phantom clusters. Heavy competition means infrastructure is here today and gone tomorrow, the aforementioned third-party selling of clusters, and in some cases people making commitments and hoping they can figure it out after the agreement is signed.

‍

How do you protect yourself from being haunted by a phantom cluster?

‍

‍

‍

A quick aside about delivery timelines. The above guidance also applies to ensuring your promised infrastructure timelines are legit. Your vendor should not be communicating a committed timeline until they have three things:

‍

‍

Any commitment before this point in time is an estimate, full-stop. Demand precision and transparency. We think you deserve it.

‍

‍

‍

Who Actually Owns and Operates Your Cluster?

This question is in contention for the first question you should ask on an introductory call:

‍

‘Who owns and operates this infrastructure?’

Here are a number of responses you can expect, listed in order from least to most favorable:

‍

1. Trust me bro.
2. I have a line of sight into some great providers with allocations and clusters that I can connect you with.
3. We know some people who we think will do a great job for you.
4. Our partner, who we can reveal to you at a later time.
5. Our partner, who we know and trust well enough to tell you their name right now because it’s not a weird relationship and we trust each other.
6. We do.

‍

The latter half of this list is viable. The first half should be viewed as unnecessary cost or some magic I frankly don’t understand. The last two bullets are ideal.

‍

Here’s the truth: There are no mysterious available clusters sitting around waiting for someone to reserve them. Demand is too high, and if there is a cluster that nobody wants to reserve, that’s a red flag.

‍

Direct ownership or tight partnership between two value-adding partners results in tighter SLAs, lower GPU/hr pricing, a clear line of communication if something goes wrong, and ultimately better ROI.

‍

‍

‍

Why GPU Pricing Changes, and How to Budget for It

The costs around GPU infrastructure are volatile at best, wildly volatile on a good day. About a year ago there was a big debate amongst pundits about the useful life-span of GPUs. The argument being made was that it was roughly three years and that the value of older-generation GPUs would depreciate rapidly. Fast forward to today, and guess what: the pundits were wrong

‍

H100s, the best marker we have for the value over time discussion around GPUs, are being reserved at or above their value when they first came on the market. Will this be the same for all generations of GPUs? Nobody knows for sure, but to date, prices are only going up. Scarcity is real, but it’s only one factor.

‍

The other primary factor is that the BOM (bill of materials) for a GPU cluster has more than one line item that says ‘GPU’ in column A. As demand increases, supply chain challenges across the stack cause significant fluctuations, typically upwards, month over month. As an example, earlier this year over a sixty-day period we saw server prices increase 13% in the first 30 days and 20% in the second 30 days, primarily driven by the increase in storage costs. Consider alongside this that NVMe and high-throughput storage tiers fluctuate independently. Wild.

‍

As a seller, we try our best to be transparent about pricing and how long our quotes are good for. We have seen many customers tell us our prices were too high, but then come back. Why? Because the quote they got from another vendor was indeed better, but that quote was no longer valid and now our prices are more closely aligned. Some OEM even have terms written in their quotes stating that the price can change up until and including the day of shipment. What a world.

‍

All that said, this is the reality of the market. Here are some tips to protect yourself from the market dynamics:

‍

1. Ask vendors for pricing change policies: How much notice? Is there a price-lock option?
2. Reserved/committed contracts offer more stability, and longer-term reservations offer better GPU/hr terms.
3. Negotiate on the pre-payment if that is something you can do.
4. Budget with a buffer and ask for price history to understand typical variance ranges
5. Be ready to act quickly to capitalize on the current price.

‍

‍

‍

Conclusion

If we use the 'Storm, Form, Norm' framework, this market is clearly still in the 'Storm' stage. The good news is that we’re going to get through this together. As the market settles and normalizes, things will continue to evolve for the better. For now, we hope that this post helps as you navigate your way towards your infrastructure goals.

‍

WhiteFiber may or may not be the right choice for your projects, but if we can be of help, we’d appreciate the chance to talk to you so you can see how we stack up against the criteria outlined above.

‍

Plus, our engineers are great, but don’t take my word for it.

‍

‍

‍

‍

FAQ

‍

How do I know if a vendor actually owns and operates the cluster they are selling?

Ask them directly and listen carefully to the answer. A vendor who owns their infrastructure can tell you the data center name, show you rack elevations and floor plans, and name their OEM partners without hesitation. If the answer involves phrases like "we have visibility into capacity" or "our partner will be introduced later in the process," you are likely talking to a broker adding cost without adding control. The cleaner the answer, the more direct the ownership.

‍

‍

What should be in an MSA to protect against missed deployment timelines?

Ask for financial penalties tied to missed delivery dates on the things the vendor can control, which include rack and stack, network configuration, and certification. A confident vendor will engage on this and explain what they can and cannot commit to. A vendor who deflects immediately to OEM delays without offering any accountability for their own process is showing you something important about how they operate under pressure.

‍

‍

Why do GPU prices change so much, and how far in advance should I budget?

The BOM (bill of materials) for a GPU cluster extends well beyond the GPU itself. Storage, networking components, and server hardware all fluctuate independently, sometimes significantly within a single quarter. Reserved contracts with longer terms offer more pricing stability than on-demand. If you are working from a quote, ask how long it is valid and whether there is a price-lock option. Budget with a buffer and ask vendors for their price history so you understand the typical range of variance before you commit.

‍

‍

At what point should a vendor be willing to give me a committed deployment timeline?

Not before they have three things confirmed: contracted data center space and power with a firm RFS (ready-for-service) date, engineered architectural designs including floor plans and topology diagrams, and a guaranteed allocation and delivery timeline from the OEM covering all hardware components. Any commitment before those three conditions are met is an estimate.