The Challenge of Header Redundancy
In the transition from HTTP/1.1 to HTTP/2, the primary goal was to eliminate the performance bottlenecks caused by head-of-line blocking and inefficient text-based protocols. While binary framing addressed the transport layer, the overhead of repeated headers in successive requests remained a significant drag. Every HTTP request carries a set of headers—User-Agent, Cookie, Accept-Encoding—that are often identical across dozens of requests within a single session. HPACK was introduced as a specialized compression format designed to eliminate this redundancy without the security pitfalls of generic compression (like CRIME or BREACH).
How HPACK Works: Statefulness and Tables
HPACK functions as a stateful compression scheme that maintains a shared index between the client and the server. It uses two primary mechanisms to minimize data transmission:
The Static Table: A pre-defined list of 61 common header fields (e.g., :method: GET) that are always available, requiring no transmission overhead.
The Dynamic Table: A session-specific, evolving list that populates as new header key-value pairs are encountered. Once added, these headers are referenced by index rather than being sent as literal strings.
When a client sends a request, HPACK determines if a header exists in either table. If it does, the request only needs to encode the index integer. If the header is new or has a new value, HPACK encodes it using Huffman coding to further compress the raw string bytes before inserting it into the dynamic table for future reuse.
Implementation Trade-offs
The memory management of the dynamic table is a critical implementation detail. Since memory is finite, HPACK includes a mechanism for size constraints. When the table exceeds the configured limit, older headers are evicted, ensuring the state does not grow indefinitely. Engineers must be aware that while HPACK drastically reduces bandwidth usage, it requires both the sender and receiver to maintain synchronization. Any corruption in the header stream renders the entire connection invalid, as the state tables would diverge between nodes.
Ultimately, HPACK is a masterclass in protocol design, prioritizing security and deterministic performance over generic compression algorithms. By understanding how the dynamic table evolves during a connection, developers can better diagnose issues related to header bloating and optimize their application's interaction with HTTP/2 proxy layers.