AI Broke Autoscaling: Rethinking Concurrency for LLM-Powered Systems

Large Language Models have fundamentally changed the latency profile of enterprise software. While traditional services spend milliseconds waiting for I/O, AI-powered workloads can remain blocked for tens of seconds on a single request. This shift exposes a critical weakness in conventional thread-per-request architectures and even in modern autoscaling mechanisms. Using a comparative study built with Quarkus and Java 21, this session explores how reactive programming and virtual threads cope with LLM-scale latency. We will analyze why a service running at only 3% CPU can become fully saturated, how Little’s Law accurately predicts concurrency collapse, and why a workload requiring 51 Kubernetes pods in a traditional model can be handled by a single pod using modern execution paradigms. Beyond performance benchmarks, we introduce a self-governing architecture that adapts concurrency, model quality, and resource consumption in real time. The result is an AI service capable of preserving throughput under pressure while significantly reducing infrastructure footprint and software carbon intensity.