Why Microseconds Matter in AI Infrastructure

I read “Every µs Matters: Achieving Near Speed-of-Light Latency in GPU Collectives” because of the unusual scale of the optimization presented. The paper is not attempting to make the model smaller or change the attention mechanism. Instead, the goal is to approach the theoretical minimum latency imposed by the memory system and interconnect. On four GB200 GPUs, they are able to reduce the latency of small messages for AllReduce from 11.0 µs for NCCL’s ring algorithm to 2.37 µs, or an 8.7% reduction in inter-token latency for Llama-3.1-70B. (1)

What I found more interesting than the individual result is the systems problem around it. Beyond the research that has gone into memory allocation, attention kernels, prefill/decode placement, and GPU communication, there have been efforts like vLLM to introduce PagedAttention to reduce waste in the KV-cache memory, DistServe to separate prefill and decode to different GPUs, FlashInfer to specialize attention kernels for different types of inference workloads, and MSCCL++ to introduce lower-level communication primitives that can be used with custom kernels to preserve performance. (2) (3) (4) (5)

These systems and the motivations behind them support my interpretation of LLM inference as a problem of coordinating computation, memory, and communication at very fine granularity. (1) (2) (3) (4) (5)

Latency can matter more than bandwidth

While data movement between GPUs is the usual topic of discussion around collective communication operations, the paper “Every µs Matters” investigates a scenario in which the data movement is minimal. Such conditions are common in long-context, small-batch tensor-parallel decoding: the KV cache grows large with sequence length and batch size is limited by available device memory. In such cases, the time required to start and complete collective communications can be more important than the bandwidth used during the operation. (1)

The implications of this finding have changed how I think about collective communication. Even with high bandwidth, there can be a significant fraction of time spent on synchronization in short collective communication operations. The authors show more than 1 µs of overhead for a global memory barrier. For a small-message AllReduce operation over four GPUs that takes about 5 µs, two memory barriers account for roughly 40% of the runtime of the operation. (1)

Their response is not simply to make a barrier faster. They propose AllReduce variants that eliminate the global barrier altogether by using techniques like low-latency flag packing, sentinel values, bidirectional communication, double buffering, and an LL128 atomic design. Each of these approaches has different costs. (1)

The protocol for low-latency communication places the data and a readiness flag into an atomic store. This avoids the need for a separate communication step to signal the readiness of the data, but at the cost of reducing the bandwidth available for data and increasing the use of scratch space to hold the data. The alternative sentinel method initializes the receiving buffer to some value, such as a negative NaN, and then repeatedly accesses the memory location until the value has changed. This method uses the full bandwidth of the communication link for the data but imposes the constraint that the sentinel value must not be used for transmitted values and requires managing the resetting of the receiving buffer to its initial value. (1)

The fact that the double-buffered design essentially replaces global coordination with local communication dependencies is also important. Each rank has access to the scratch buffers, so when one rank receives data from another, that exchange establishes when the other buffer can be reused for the next iteration of the reduction. Thus, multiple reductions can be performed without having to insert a global barrier to coordinate all of the processors, again assuming the communication constraints described by the authors. (1)

Defining the physical lower bound

One part of the paper that I liked is that they did not define the success of their algorithm only in comparison with some other library implementing some other algorithm. Instead, they defined a hardware “speed-of-light” lower bound for the operation based upon the size of the smallest piece of data that must move through the system: a single 128-byte cache line. The lower bound is defined as two L2 cache round trips plus the latency of one remote store:

L_{SoL}=2L_{L2_RTT}+L_{remote_store}

On two GB200 GPUs, they report that the L2 round-trip time is estimated at 0.306 µs and the remote store latency is estimated at 0.792 µs, providing a calculation of 1.404 µs for the lower bound on AllReduce time. This ignores latencies like instruction scheduling and reduction computation. (1)

Furthermore, they note that their fastest small-message one-shot kernels show around 7% overhead above this lower bound for two GPUs. For larger counts of GPUs, however, using multicast becomes more useful because NVLink and NVSwitch can perform portions of the AllReduce operation using multimem operations, rather than requiring the software to perform every reduction operation directly. (1)

I think this particular way of measuring performance is more useful than determining whether an implementation is better than the previous implementation alone. Comparing to the fastest existing library determines if an implementation is better than current software. However, comparing to the lower bound of the hardware determines if there is still an opportunity for further optimization of the system. While this does not ensure that the lower bound can be reached within the complete application, it does provide a target for those developing systems to aim for. (1)

This connects to a larger change in LLM serving

The paper made more sense to me when I connected it to the development of LLM serving systems over the last several years.

vLLM attacked the problem of memory management. By using blocks in its PagedAttention algorithm to manage the KV cache, it was able to achieve two to four times higher throughput than the systems it compared against. These gains were especially great with longer sequences of tokens, larger models, and more complex methods of decoding those tokens. (2)

DistServe attacked the problem of scheduling and placing the models on GPUs. It moved the prefill and decode steps to separate GPUs within the system. This architecture can serve more requests within the same amount of time while meeting both low time-to-first-token and time-per-output-token requirements than the systems it compared against. However, it must account for the communication between the separate GPUs resulting from this separation. (3)

FlashInfer attacks the problem of execution within attention itself. It provides customizable GPU attention kernels, block-sparse and composable KV-cache formats, and scheduling techniques to handle the variations in requests while remaining compatible with CUDA Graph execution. The evaluation of FlashInfer indicates that it can reduce both inter-token and long-context inference latency. (4)

MSCCL++ addresses the problem of communication abstraction. The authors state that the rapidly changing nature of AI hardware often leads to the implementation of application-specific communication libraries. These specialized libraries are often difficult to reuse or port to new hardware. MSCCL++ exposes low-level communication primitives to applications while maintaining higher-level interfaces. The implementation reports speedups in collective and end-to-end inference times compared to the baselines. (5)

Although these projects optimize for different resources, the same general design pattern is present in each. Each of these projects indicates a movement away from treating a single abstraction as applying to all instances of a process uniformly, and toward recognizing that not all uses of a given abstraction behave the same. Memory can be managed differently based on the KV-cache format. Prefill and decode can receive different resources. Attention can be performed in different ways based on the chosen attention kernel. Collective communication algorithms can be chosen based on the size and characteristics of the messages being sent. This inference is drawn from the combined work presented in these articles rather than from any one article alone. (1) (2) (3) (4) (5)

Communication is becoming part of GPU execution

Another connection is the movement away from orchestrating communication from the host toward device-initiated communication.

The work on GPU-Initiated Networking for NCCL describes Load/Store Accessible communication for NVLink or PCIe, multicast-oriented Multimem operations, and GPU-Initiated Networking for network RDMA. All of these aim to reduce the dependence of NCCL on the CPU to orchestrate GPU communication. (6)

NCCL EP applies this idea to the communication within Mixture-of-Experts models. It provides a separate implementation for low-latency communication and high-throughput communication. Low-latency communication is used during inference with small batches of tokens, whereas high-throughput communication is used during training and prefill with larger batches of tokens. The low-latency implementation targets batches of 1 to 128 tokens using all-to-all communication directly over RDMA and NVLink. The high-throughput implementation targets batches of 4096 tokens or more using hierarchical communication. (7)

Building on the same device-side direction for NCCL, Every µs Matters also considers how to apply these ideas to general collectives, such as AllReduce. The paper describes an experimental ncclLLBuffer abstraction that allows low-latency send, receive, reduction, broadcast, and related communication operations to be called directly from within kernels. (1)

I think that this idea is important. It allows kernels to participate directly in communication rather than relying entirely on host-managed communication calls between phases of GPU execution. This idea is reflected in several of the papers mentioned above: GPU-Initiated Networking, NCCL EP, MSCCL++, and Every µs Matters. (1) (5) (6) (7)

The same problem exists outside AI

The fact that the focus of much of the discussion has been on language model inference may make it seem like this is a problem created by language models, but the evaluation of cuSOLVERMp with a distributed generalized eigensolver on the Alps supercomputer also reveals performance improvements. The improvement is larger for the tested configuration because a larger fraction of the runtime is spent communicating. (1) Furthermore, some of the same researchers have published previously on related topics. For instance, LLAMP models the impact of network latency on HPC applications through application execution graphs, the LogGPS communication model, and linear programming. LLAMP tested applications like MILC, LULESH, LAMMPS, and ICON through MPI and found that the latency tolerance of HPC applications can vary dramatically. (8)

That is why I find this connection between AI and HPC interesting. AI systems are driving substantial investment in GPU interconnects and communication software, while HPC applications also contain execution phases that are sensitive to communication latency. Both areas are affected by latency-sensitive parallel communication. (1) (8)

What I learned from it

One of the main things I found interesting from this paper was that the bandwidth between GPUs did not eliminate the latency of GPU synchronization protocols. The GB200 NVL72 system features some of the highest bandwidth available for scale-up between GPUs, yet synchronization still occurs on the order of microseconds. The best performance improvements came from changing the protocols for synchronization and data movement between GPUs, not simply increasing the bandwidth between them. (1)

The second reason that a communication algorithm may become important is due to changes in the workload around that optimization. While models with long-context capabilities require more memory from the KV cache, the decode step performs relatively small collective operations repeatedly. Thus, a communication algorithm that is somewhat costly for processing large messages may become important for models with high requirements for low inter-token latencies. (1) This relates to both the focus of vLLM upon efficiently utilizing the memory of the KV cache, as well as the approach of DistServe to treat prefill and decode as two different steps of serving a request from user to model. (2) (3)

The third reason that there may not be a “best” algorithm for performing a collective operation is that the best choice can depend on the conditions of that operation. The authors conducted tests to determine which collective communication algorithms performed best for different message sizes, numbers of GPUs, scratch buffer sizes, synchronization methods, and multicast availability. As such, these empirical observations were used to create a system that selects between these available kernels, and using these observations to build a more accurate model for selecting such kernels is future work for the authors. (1)

That leads into the bigger picture of these systems being specialized for different workloads. DistServe makes decisions about model placement based on whether the workload is performing prefill or decode operations. FlashInfer specializes kernel execution based on the characteristics of the attention workload. NCCL EP exposes both low-latency and high-throughput communication APIs based upon the workloads of the models that are communicating. Finally, Every µs Matters selects between different collective kernels based upon the characteristics of the communication workload. (1) (3) (4) (7)

Where I think this is heading

I think in the future we will see even more optimization within the systems that already exist between model layers. For instance, existing systems have already led to memory managers influencing the execution of attention operations, serving architectures influencing the transfer of KV caches between GPUs, kernels performing communication operations directly between GPUs, and communication libraries exposing lower-level APIs directly to GPU kernels. (2) (3) (4) (5) (6)

I also expect performance models and runtime selection to become more important. While the current implementation in Every µs Matters relies on empirical measurements of kernel performance, accurate performance modeling is identified as future work. Additionally, MSCCL++ argues for creating abstractions that allow specialized implementations of the communication stack without having to maintain a separate communication stack for every application that uses the library. (1) (5)

The issue of portability is more difficult to address. Most of the optimizations discussed require specific features from the hardware. For instance, many of these features depend upon NVLink and NVSwitch, support for multicast operations, GPUs initiating RDMA requests, and support for atomic operations at the architectural level of those GPUs. MSCCL++ directly addresses both the need for this specialization of the hardware and the need for portability. NCCL EP and GPU-Initiated Networking expose these features for access by applications. (1) (5) (6) (7)

For me, that is the most useful idea from the paper. While it is certainly interesting to see that an AllReduce operation comes close to the theoretical lower bound for the performance of that operation between the GPUs, it is the remaining microseconds that are interesting and applicable to model serving systems. The fact that these microseconds are large enough to significantly impact the performance of an inference workload makes the low-level details of the communication library part of the performance of the model serving system. This idea is explicitly stated in the paper, established through its benchmarks and application results, and is additionally supported by other related systems for model serving and communication. (1) (2) (3) (4) (5) (6) (7)

Papers connected in this piece

Siyuan Shen et al., Every µs Matters: Achieving Near Speed-of-Light Latency in GPU Collectives, 2026. arXiv
Woosuk Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023. arXiv
Yinmin Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving, 2024. arXiv
Zihao Ye et al., FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving, 2025. arXiv
Aashaka Shah et al., MSCCL++: Rethinking GPU Communication Abstractions for AI Inference, 2025. arXiv
Khaled Hamidouche et al., GPU-Initiated Networking for NCCL, 2025. arXiv
Amos Goldman et al., NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL, 2026. arXiv
Siyuan Shen et al., LLAMP: Assessing Network Latency Tolerance of HPC Applications with Linear Programming, 2024. arXiv