Using only a shared spreadsheet and our existing test setup, I’m trying to understand the terminology around AI scalability. I read that it means handling more users and data without losing performance. I doubt whether that includes mundane issues like requests sitting in a queue and my sheet reaching 37 rows. Does that definition hold up?
The missing detail is what do you mean by performance, response time, accuracy, cost, throughput, etc. Queued requests are a scalability problem if increases in the number of requests lead to increased delays. The fact that a spreadsheet can handle only a certain number of rows is a limitation not of the AI but of the workflow or test harness, but it is still a scalability limitation of the overall system.
A drawback of using a spreadsheet is that it may create an illusion of scalability by gradually increasing the load. By scalability, we usually mean the ability to handle more realistic simultaneous users within acceptable limits of response time, error rate, response quality, and costs. While your solution might well have some obvious bottlenecks, you cannot demonstrate its scalability before you subject it to such a load.
Do not think of scalability as a binary parameter. A system may scale in terms of processing volume but not in terms of message length and response token count, or retrieval may require scanning a much larger dataset. It can scale in throughput while becoming prohibitively expensive at higher loads, or produce lower-quality results under higher loads.
The spreadsheet you mentioned is not a problem; it can be used to organize the scenarios and their results, but the way you will have to structure the tests must involve overlapping stages rather than multiple sequential requests. For each level of load, measure the latency, number of failed or rate-limited requests, response quality, and overall cost. And identify the point at which the additional load creates a significant increase in one of these metrics.
So in practical terms, scalability means knowing how much useful work your current setup can handle before it violates limits you have defined. @not_a_hawk is right that realistic concurrency matters, but I would include input size and workload mix too. Fifty identical short prompts tell you much less than a mix of short requests, long documents, retrieval calls, and retries.
Write down two columns before doing anything else: load level and resources available. Then, repeat the same test workload after increasing a resource your system can scale, such as a worker processes, API quota, DB capacity, or retrieval replicas.
Capacity and scalability have to be distinguished here. Capacity is the ability of the current setup to handle a given load; scalability, in turn, is the ability to increase capacity when additional resources are invested. Something that handles a big load poorly is not necessarily less scalable than a smaller system that processes a smaller request surge. A small-scale setup, when given more workers, could process requests at a roughly comparable rate without unacceptable tail latency, errors, or monetary expense.
The spreadsheet can track configurations, timestamps, outcomes, and pass/fail thresholds. It cannot compensate for a test setup that sends requests sequentially, which is where I agree with @binarylogic2812. You need overlapping requests to expose contention. However, reaching a breaking point once is only a capacity result. To say something about scalability, change the available resources and see whether that breaking point moves as expected.
I would add recovery behavior to the list. Some systems look healthy through a gradual ramp but behave badly after a sudden burst: queues continue to grow, retries accumulate, and latency remains high after traffic has dropped. Testing the same scenario from a cold start, after warm-up, and right after an overload can help identify if autoscaling, caching, or queue management is beneficial or just masking the problem.
If the AI model is accessed via a hosted API, be specific about the claim. You are testing your application’s and workflow’s scalability and not the model provider’s infra’s scalability. While a valid point, a spreadsheet is still a valid tool for that.
The hidden bottleneck is the person reviewing the AI’s output. A system may well be returning results faster and faster while at the same time becoming progressively less useful due to an increasing number of responses requiring review, retry, or revision. The process is scaling, but not in a manner likely to be of interest to most businesses.
Concurrency is important, but I would not be tempted to declare it as the sole criterion for judgment. The nature of the output from your AI is likely to have considerable variation, so a single successful row in your spreadsheet is not worth getting excited about. Instead, try capturing results from critical scenarios several times, noting whether the result was immediately useful, required retry, needed editing, or had to be escalated to a human for review. You can then talk about the “useful throughput” of the system, and its relationship to total throughput.
Your spreadsheet can be used for this, by capturing different classes of response along with the associated retry and review effort. By batching these outcomes and comparing them to increasing levels of concurrency you may well discover that response time degrades gracefully while review time does not increase, or that longer inputs are required to produce acceptable results on the first try.
With the information you have I think you should aim to limit your claims to the proposition that you can recognize when your own process begins to encounter unacceptable response times, failures, costs, or reviewer load. You cannot demonstrate that the system will scale nicely under any conceivable workload, and you do not need to. The real-world implications are that you understand the point at which the ability to add resources stops keeping up with demand, and that you are not misleading users into thinking that “scale” means “hiding” these resource limits.
A system which gets slowed down for everyone when a user submits a big document is less scalable in practice than one which lets the big job wait a long time while allowing ordinary requests to proceed normally. Total throughput could be identical in the spreadsheet, but the user experience would not be.
That’s the missing angle here: isolation. AI workloads vary wildly in prompt size, output length, retrieval work, and retries. If your test uses a uniform batch, you could be hiding the ‘noisy neighbor’ problem where a few of your requests are eating up the queue, quota, memory, or database connections.
Your spreadsheet can still expose that. Separate requests into light and heavy groups, run them all together, and record results from each group separately. Do your light requests stay steady? Do your heavy ones cause failures elsewhere? Does the system serve one class of work ahead of the other, or does it do whichever comes first?
Avoid trusting an overall average, because that can mask a badly affected group entirely.
I’d be cautious in using the word ‘scalable’ unless the capacity really grows without unrelated users getting disproportionally hammered on. More work taking more resources is normal, but a small amount of really unwieldy work making the whole service unstable is a problem.
Consider two scenarios; one where a steady trickle of users are coming in, but the retrieval index has quietly ballooned from a handful to tens of thousands of documents, taking three times longer to return results at launch, and another getting swamped by a sudden influx of users but recovering cleanly after the peak. The former never failed a load test, but is proving to be problematic, the latter having never failed one is not necessarily the better system. This is the facet I feel this thread is underweighting.
Everyone is focusing on concurrency, isolation, reviewer load, capacity vs scalability, etc. But the original question literally asked about more users and data and almost everyone focused on users. The scalability problem is subtler with data, where your prompt size stays the same and traffic is flat but performance degrades because the corpus you are querying, the context you are stuffing in, or the history you are carrying is growing. A spreadsheet ramp test run in an afternoon will never detect the problem because the thing that took two months to accumulate is what takes down the system.
I mostly agree with @binarylogic2812 that scalability isn’t binary, and the point about scanning a larger dataset is the closest anyone got to what I mean. I’d just push it harder. Volume of stored data is its own axis, separate from request volume, and it moves on its own timeline.
So if you want your spreadsheet to say something honest about this:
- Run the same fixed workload against a small dataset, then a medium one, then a large one. Same prompts, same concurrency, only the underlying data changes.
- Log latency and cost per request at each size, not just pass or fail.
- Watch whether context length creeps up on its own as retrieval pulls in more matches, because that quietly inflates token cost even when nothing else changed.
The blunt version: the ability to handle a sustained load and to scale up with the growth are two separate questions, and a burst test defines one of them. If you only test at the crowd, you’ll ship an awesomeness that looked good at the week one and mysteriously crawls by the quarter’s end and no one will trace it back to the pile of data they kept adding. Define your limit, obviously, but not against a dataset you expect to have later but not the one that you have now.
Half this thread is arguing about how to test, but the original question was about a word, and the honest answer is that ‘scalable’ on its own means almost nothing until you attach it to a specific limit. @stacklab1415 nailed the part everyone skated past: data volume moves on its own clock, and no afternoon ramp test will ever show you the corpus that took months to pile up. So don’t wait for it. Fake it now. Seed your retrieval index with junk documents until it’s the size you expect next quarter, then run your normal workload against that. If latency and token cost creep even with flat traffic, you found the real problem before it found you. That one trick tells you more than any concurrency chart, and your spreadsheet handles it fine as long as you log per-request cost at each dataset size instead of a single pass/fail.