Before buying GPUs for your office, do this first

I am a fan of running models privately. You can control where the data goes, choose the model, integrate it into internal systems and, depending on usage, avoid paying a provider for every request. There are plenty of reasons to explore it.

But the first meeting about private AI tends to jump straight to GPUs. How many cards? Which server? How much VRAM? I would rather start with a laptop, some sample documents and five employees who actually want to use the thing.

Find the workload before sizing the box

Are people asking general questions, summarizing internal documents, writing code or using AI inside a business process? Those are different workloads. A model that feels fast in a one-person demonstration may become painfully slow when twenty people use it together. A model that sounds articulate may still be terrible at your specific documents.

Then there is access control. If an employee asks a question about an internal file, the assistant should not magically gain permission to retrieve everything on the file server. Hosting privately is not the same as designing secure permissions.

I would test response quality, concurrency and the actual amount of time people save. Only after that would I pick between a small on-premises deployment, larger dedicated GPU servers or a hybrid setup. Hardware is the easy thing to spend money on. Getting the workflow right is harder.

There is also a middle ground

One mistake is treating the choice as either a fully public AI service or a rack of expensive servers in the office. You can test a hosted model using approved data and sensible controls, run a smaller model privately for a narrow workflow, or keep the application internal while using an external model for certain requests. The right boundary depends on what your data actually contains and what your vendors allow.

For example, an internal assistant that searches policy documents needs more than a model. Someone has to keep the index current, handle access to documents, show sources and decide what happens when an answer is wrong. If those pieces are missing, private hosting just gives you an inaccurate assistant on hardware you own.

I would also write down who will use it before doing capacity math. Ten engineers occasionally asking questions have very different needs from a company generating thousands of requests a day as part of its product. The answer changes what you buy, what you host and how much of the system needs redundancy.

When a client really does need control over where the model runs, I like that kind of work. I just want them to buy the hardware after they've seen the workload, not before.