Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Continuous Benchmarking #573

Open
5 of 8 tasks
avik-pal opened this issue Apr 4, 2024 · 0 comments
Open
5 of 8 tasks

Continuous Benchmarking #573

avik-pal opened this issue Apr 4, 2024 · 0 comments

Comments

@avik-pal
Copy link
Member

avik-pal commented Apr 4, 2024

Started initial setup in #572

  • Generate a minimal set of benchmarks
    • Some basic ones are present in https://github.com/FluxML/Flux.jl/tree/master/perf
    • Benchmark different AD backends -- use DifferentiationInterface.jl for this part.
      Main backends are Zygote.jl, Tracker.jl, Enzyme.jl, Tapir.jl and ReverseDiff.jl. Its working without DI, so I wont bother rewriting them. If anyone rewrites them I would be happy to merge it.
  • Comparative benchmarking against Flux.jl
  • Migrate to Buildkite to allow for CUDA benchmarks
    • The current benchmarks are very noisy and are running on too low threads and cores to make any reasonable sense of the numbers
  • Don't keep everything in memory, use setup to create the parameters and inputs
  • Ordering of the benchmarks are a bit screwed up rn, we could potentially fix it by migrating to a custom build step from a easier to use (for me) plotting library
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant