Subscriptions Performance

[AlexanderWells-diamond]

I've been profiling a Strawberry server, and unfortunately it seems to perform less than adequately for my use. Is anyone able to offer any insights as to why it appears to be slow?

The test I'm seeing the problems with sets up a server, then a separate Python client that creates many subscriptions. Each subscription is to a process that in effect is just counting upwards at 10Hz. Knowing this, each client subscription can spot when updates have been missed. The issue I see comes at above 100 concurrent subscriptions. Everything seems stable up to that point, but above that I see a lot of dropped updates - this correlates to the point at which the single-threaded Python process hits 100% CPU usage.

I've attached a speedscope recording file here (it isn't a .txt, GitHub just made me name it that). From my understanding this shows that the program is spending at least 70% of its time inside Strawberry code, or its dependants:

• 29% in operation_task at /strawberry/subscriptions/protocols/graphql_transport_ws/handlers.py:274
• 18% in await_result at /graphql/execution/execute.py:528
• 14% in await_result at /graphql/execution/execute.py:527
• 7.7% in await_result at /graphql/execution/execute.py:531
• 4.5% in await_completed at /graphql/execution/execute.py:547

And I only see 8.0% of time in my subscription handler (coniql/src/coniql/strawberry_schema.py:184). This leads me to believe the vast majority of the time is being spent receiving/formatting/sending the GraphQL messages. I struggle to believe this though, as it would imply that Strawberry can only support ~1000 updates a second. This seems very low, so I'm hoping that I'm missing something!

Thank you for any input.

[patrick91]

hi @AlexanderWells-diamond 😊 would you be possible for you to write a benchmark that we can use to monitor this?

We also some updates on subscription, but I don't know if they will make things worse or better.

I'm personally very interested in improving the performance, but I haven't had time to do much on that. I'm sure we can make the Strawberry part faster, but I'm not 100% sure if we can do the same for GraphQL-core, at least not easily

[AlexanderWells-diamond]

Thank you for your reply. I'm not really sure what a benchmark would look like - I'm doing full system tests, with real servers and network requests aplenty. These are running on a dedicated Kubernetes node, to attempt to ensure stability. I can't see a simple way to directly use it as a Strawberry performance test.

Things get especially messy if you want to test what happens when you max out the CPU. In Kubernetes I can configure CPU limits and monitor it pretty carefully, but in a GitHub CI runner we wouldn't have that sort of ability.

Having said all this, I do note you have some benchmarking tests already, using pytest-codspeed. I don't know how clever that plugin is in regards to CPU and memory usage, or if it's simply just running a timer. Do you know if it can alleviate any of the concerns I've outlined? Thanks.

[patrick91]

@art049 might have some more info about codspeed 😊

@AlexanderWells-diamond would you be willing to do a call as well? mostly so I don't forget about this (I'm currently at EuroPython)

[AlexanderWells-diamond]

I've run some more tests, using a very basic Strawberry with a simple counter for subscriptions and that seems to behave very well. This makes me believe I've probably messed up something regarding the implementation of my subscription - I suspect I'm heavily over-using asyncio tasks and this is then causing the various await_* functions to show up heavily in the traces.

I'll do further investigation and refactoring of my subscriptions, to remove the potential bad code.

[art049]

Having said all this, I do note you have some benchmarking tests already, using pytest-codspeed. I don't know how clever that plugin is in regards to CPU and memory usage, or if it's simply just running a timer. Do you know if it can alleviate any of the concerns I've outlined? Thanks.

Hello, @AlexanderWells-diamond it's mainly measuring the CPU behavior while running the tests to get consistent performance results. Not much analysis is related to the memory (yet😅) but the metric is interesting to evaluate changes' impact over time.