Kestrel HTTP/2 SETTINGS Frame Ignoring Defaults Issue And Solution

Hey everyone, let's dive into an interesting issue we've encountered with Kestrel and HTTP/2, specifically concerning how it handles the initial SETTINGS frame. This came up when a customer was using the golang http module to make HTTP/2 requests to a service running on Kestrel.

The Heart of the Matter: MaxConcurrentStreams

The main keyword here is MaxConcurrentStreams and how different implementations handle its default value. In Kestrel, if you don't explicitly configure MaxConcurrentStreams in Http2Limits, it defaults to 100, as per the HTTP/2 specification. Now, here's where things get tricky. According to the Http2PeerSettings.GetNonProtocolDefaults in Kestrel, if MaxConcurrentStreams is set to this default value, it isn't included in the SETTINGS frame. This seems fine initially, but it leads to a mismatch with how some HTTP/2 clients behave.

The crux of the issue lies in the golang http/2 client. If this client doesn't receive a MaxConcurrentStreams value in the server's SETTINGS frame, it falls back to a default of 1000 streams! This is significantly higher than the protocol default of 100, and it can cause problems. To illustrate, consider HTTP.SYS, which does send the MaxConcurrentStreams value in the SETTINGS frame, regardless of whether it's the default or a custom value. This discrepancy in behavior between Kestrel and golang's http/2 client is the core of the issue we're addressing.

Imagine this scenario: Your Kestrel server is humming along, expecting a maximum of 100 concurrent streams per connection. Then, a golang-based client connects and, assuming it can use 1000 streams, starts bombarding your server with requests. This can lead to resource exhaustion, performance degradation, and potentially even service disruptions. This behavior is unexpected because the server intends to limit concurrent streams to 100, but the client interprets the absence of the MaxConcurrentStreams setting as permission to use a much larger number.

The impact of this mismatch can be substantial, especially in high-traffic scenarios. Load testing tools, like Bombardier (which we'll discuss shortly), can exacerbate this issue. When a client believes it can open many more streams than the server is prepared for, it can quickly overwhelm the server's resources. This is why understanding and addressing this discrepancy is so crucial for maintaining the stability and performance of your applications.

Expected Behavior: A Matter of Interpretation

The main keyword here is Expected Behavior, and it’s a bit of a philosophical question: Is this a bug in Kestrel, or is it a bug in golang's http/2 client? The golang client's decision to default to 1000 streams when the protocol default is 100 does seem a little odd. However, it's possible that other clients might behave similarly. This raises the question: should Kestrel simply send the default value in the SETTINGS frame to ensure consistency across different clients?

Think of it like this: Kestrel is adhering to the letter of the HTTP/2 specification by not sending the default value, but it's inadvertently creating a compatibility issue with clients that have their own interpretations of the absence of this value. It's a bit like two people speaking different dialects of the same language – they can communicate, but there's a risk of misunderstandings. In this case, the misunderstanding is about the maximum number of concurrent streams the server can handle.

From Kestrel's perspective, not sending the default value might seem like an optimization – why send information that's already implied by the protocol? However, in practice, this optimization can lead to unexpected behavior. By explicitly sending the default value, Kestrel could avoid these ambiguities and ensure that all clients have a clear understanding of the server's capabilities. This approach would prioritize interoperability and predictability over a minor optimization.

It’s also important to consider the broader ecosystem. If Kestrel is used in conjunction with various HTTP/2 clients, it's crucial to ensure that it behaves predictably with each of them. While it might be tempting to argue that clients should adhere strictly to the protocol and use the default value if none is provided, the reality is that different clients have different interpretations and implementations. To ensure robustness, Kestrel may need to accommodate these variations.

Steps to Reproduce: Seeing the Problem in Action

The main keyword here is Steps To Reproduce, to see this issue in action, it's pretty straightforward. First, make sure your Kestrel HTTP/2 MaxConcurrentStreams is configured to use the protocol-default value of 100 streams. This means you shouldn't have any explicit configuration setting a different value.

Next, you'll want to use a Go-based load testing tool. A great option here is Bombardier. It's a fast and easy-to-use tool that can generate a large number of HTTP requests. Bombardier is particularly well-suited for this scenario because it allows you to simulate a high-load environment where the MaxConcurrentStreams issue becomes apparent. It will create many connections to Kestrel which would, according to the expected behavior, use 100 concurrent streams each.

Here's the crux of the experiment: When Bombardier starts sending requests to Kestrel, it will attempt to use 1000 streams before making a new connection. Remember, this is because the golang http/2 client defaults to 1000 streams if it doesn't receive a MaxConcurrentStreams value in the SETTINGS frame. This behavior is contrary to what Kestrel expects, which can lead to performance issues.

Observing this behavior is key to understanding the problem. By running this test, you can see firsthand how the mismatch in MaxConcurrentStreams interpretation can lead to unexpected client behavior and potential server overload. This practical demonstration highlights the importance of aligning server and client expectations regarding HTTP/2 settings.

This example underscores the importance of thorough testing, especially when dealing with protocol-level configurations. By replicating the scenario in a controlled environment, developers can gain valuable insights into how their applications behave under load and identify potential bottlenecks or compatibility issues.

Exceptions and Versions: The Technical Details

In this particular case, there weren't any specific exceptions thrown, which makes the bug a bit sneaky. It's more of a behavioral issue than a crashing one. The reported .NET version where this was observed is 9.0.302. It's important to note the specific .NET version, as this can help narrow down the scope of the issue and identify any potential regressions or version-specific behaviors.

Additionally, the golang version used was 1.24. This is crucial information because the behavior of the golang http/2 client might have changed in different versions. Knowing the specific version helps in reproducing the issue and understanding its context.

Think of these version numbers as puzzle pieces. When investigating a bug, having the exact versions of the frameworks and libraries involved is essential for recreating the scenario and pinpointing the root cause. Each version can introduce subtle changes in behavior, and these changes can sometimes lead to unexpected interactions.

In this instance, the combination of Kestrel 9.0.302 and golang 1.24 is where this particular issue manifests. This doesn't necessarily mean that the issue is unique to these versions, but it does provide a starting point for further investigation. It's possible that the same issue exists in other versions, or that it has been fixed in more recent versions.

Understanding the technical context, including the absence of explicit exceptions and the specific versions involved, is a crucial step in addressing this type of subtle behavioral bug. It allows developers to focus their efforts and develop targeted solutions.

Final Thoughts: A Holistic View

So, what's the takeaway here? The main keyword here is Holistic View, the interaction between Kestrel and golang's http/2 client highlights the importance of understanding how different implementations interpret HTTP/2 specifications. While Kestrel's behavior is technically correct, it can lead to unexpected issues with clients that have different default behaviors.

Ultimately, the goal is to ensure smooth and predictable communication between servers and clients. In this case, Kestrel might benefit from sending the default MaxConcurrentStreams value in the SETTINGS frame to avoid ambiguity and ensure compatibility with a wider range of clients. This approach prioritizes interoperability and robustness, even if it means deviating slightly from a strict interpretation of the specification.

The real lesson here is that real-world software development often involves navigating these kinds of gray areas. It's not always about being 100% compliant with the standards; it's about making pragmatic decisions that lead to the best overall user experience. This means considering how different components interact and being willing to adapt to ensure compatibility.

By understanding the nuances of HTTP/2 and the specific behaviors of different clients, we can build more resilient and reliable applications. This issue serves as a reminder that even seemingly minor configuration details can have significant impacts on performance and stability. Keeping a holistic view of the system is essential for effectively troubleshooting and resolving these kinds of problems.