Increase the default number of IO queues on larger machines

[kouvel]

The number of IO queues is currently limited to 16. On machines with more processors, increasing the number of IO queues appears to improve throughput on some benchmarks.

[kouvel]

Some throughput below, on the arm64 Ampere machine with 80 procs. Before the change is with 16 IO queues, and after the change is with 40 IO queues. Looks like there would be some regressions and improvements with the change. Not sure if the regressions would be acceptable, but opening for consideration anyway.

Plaintext:

Plaintext regresses with any number of connections. This appears to be due to pipelining and the way IO queues work. Not sure if pipelining is an interesting scenario.

Json:

Slight regression at 256 connections, and slight improvement at the other connection counts.

Fortunes:

Looks mostly like no change.

[gfoidl]

Nit:

Suggested change

	int processorCount = Environment.ProcessorCount;
	var processorCount = Environment.ProcessorCount;

[kouvel]

Isn't the explicit type preferred in this case? I thought that was the case in the runtime repo anyway, not sure about the guidelines in this repo.

[danmoseley]

Even if var is allowed in this repo, how is it better here?

[gfoidl]

"var all the things" is convention here (personally I prefer the explicit type, but to be consistent w/ the codebase -> var).

[kouvel]

A consideration is that it seems plausible that in some cases fewer IOQueues could perform the relevant work more efficiently and sufficiently quickly for them not to be a bottleneck. Alternatives could be considered, some challenges are outlined in #41391.

[halter73]

Plaintext regresses with any number of connections. This appears to be due to pipelining and the way IO queues work. Not sure if pipelining is an interesting scenario.

I don't think the plaintext scenario is super interesting on its own, because as you note, it uses HTTP/1.1 pipelining which is rare to see in the wild outside of benchmarking.

However, poor HTTP/1.1 pipelining performance can be indicative of poor HTTP/2 performance since HTTP/2 can effectively pipeline 100 requests at a time with the default stream limit of 100. It would be nice to get some HTTP/2 numbers before changing this.

The main benchmark app takes a --threadCount parameter, so we can also use that to test this without needing to modify the socket transport. @sebastienros has done a lot of testing using non-default IOQueueCounts for different benchmarks. I know we've hard coded the --threadCount for some of those scenarios in the past, but I don't think we do currently.

A consideration is that it seems plausible that in some cases fewer IOQueues could perform the relevant work more efficiently and sufficiently quickly for them not to be a bottleneck.

This is why we clamped the value to begin with. We erred on the side of limiting parallelism where benchmark numbers didn't show a clear improvement which happened to be any count higher than 16 with our machines at the time.

But given we now have evidence that more parallelism can significantly help many-core machines at least on Linux, I'm all for trying new things. The way we came up with Math.Min(Environment.ProcessorCount, 16) wasn't super scientific to begin with. I think it's stayed that way mostly due to inertia and fear of regressing scenarios.

Alternatives could be considered, some challenges are outlined in #41391.

The challenges and tradeoffs are what make it interesting. I'm all for trying bigger changes too, but I don't see the harm of trying something like this first. We'll need to keep a close eye on how this affects all our benchmarks at https://aka.ms/aspnet/benchmarks, and be ready to revert if this causes significant regressions in important scenarios.

[halter73]

@sebastienros I can merge this if you don't have any objections.

[kouvel]

However, poor HTTP/1.1 pipelining performance can be indicative of poor HTTP/2 performance since HTTP/2 can effectively pipeline 100 requests at a time with the default stream limit of 100. It would be nice to get some HTTP/2 numbers before changing this.

I see that the https variant of plaintext still has pipeline: 16 here: https://github.com/aspnet/Benchmarks/blob/9c454b7357ac0b032c48535df236b8bbb9588052/scenarios/plaintext.benchmarks.yml#L62

Whereas the https variant of json does not: https://github.com/aspnet/Benchmarks/blob/9c454b7357ac0b032c48535df236b8bbb9588052/scenarios/json.benchmarks.yml#L60-L72

What's the difference there, and is that kind of pipelining different and interesting?

[Edit] Nevermind, I mistook one thing for another, these are not http/2 benchmarks

[kouvel]

What would be a good benchmark to run to measure HTTP/2 multiplexing?

[halter73]

One of the gRPC scenarios would be good. Make sure to set the stream count to something like 70 or 100. I think the default is 1.

[kouvel]

Grpc - This is running on the Ampere machine as server (80 procs) and citrine-amd as load (48 proc) with --scenario grpcaspnetcoreserver-grpcnetclient --variable scenario=serverstreaming --variable requestSize=0 --variable responseSize=0 --variable streams=70 --variable connections=28 --variable threads=28:

Maybe a slight regression, but a high error margin too and maybe not too significant. The test also appears to be more load-heavy than app-heavy and I'm not sure how representative it is.