Roadmap to v1

[avik-pal]

Features

• Move Training utilities out of Experimental. One major recent push has been to clean up the training and ensure it is used everywhere in tutorials. It is about time to guarantee semvar for it.
• Preliminary XLA support. Please have a look at Meta Issue for Reactant Compilation of Lux Models #733 for a detailed discussion.
• Add a version of compute_gradients which doesn't need to return the loss. Allows us to switch to Enzyme ReversePrimalNoNeed.
• DispatchDoctor Add type-stability checks via DispatchDoctor.jl #683. Adding this directly to LuxCore. While not strictly breaking, I want to move this to 0.6
• Automatic Eltype warning using preferences. See Performance Pitfalls and How to Catch them #727
• Rethinking eltype conversions in Adaptors #684

Deprecations (need to be done)

• Returning stats from losses is a major PITA for Enzyme / Reactant use-cases. I want to purge that completely.
• Flattening of Chains Remove Auto-Flattening of Chains #682
• Remove unnecessary overloads of Base.keys
• Deprecate the TrainState construction using rng

Deprecations (already done)

• Older versions of preferences (mostly done in Performance Pitfalls and How to Catch them #727)
• cpu/gpu
• disable_stacktrace_truncation!
• Experimental symbols present outside Experimental
• Strings in DebugLayer/layer_map
• Experimental.@compact and Experimental.StatefulLuxLayer
• Not specifying true/false in StatefulLuxLayer.

Underlying packages that need to be updated to 1.0

• LuxCore. A bunch of these names came from when the eventual goal was to merge with Flux. That will never happen, so it might be good to drop those names.
  • initialparameters --> initial_parameters -- motivated with using more readable names (the latter is definitely cleaner)
  • initialstates --> initial_states
  • AbstractExplicitLayer --> AbstractLuxLayer
  • AbstractExplicitContainerLayer --> AbstractLuxContainerLayer
• LuxDeviceUtils
  • Rethinking eltype conversions in Adaptors #684
  • Remove Deprecations
• WeightInitializers
  • Test other RNGs WeightInitializers.jl#11
• LuxLib
  • Remove Deprecations

Documentation Updates

• Add a migration guide to 1.0
• Update wording on interface specs. Make it sound mandatory.
• Update WeightInitializers.jl documentation to highlight what RNGs are supported

[avik-pal]

Another consideration is whether we should do a 0.6 release or move directly to a 1.0 release.

I have read posts in the wild complaining about the rapid release, which led to the ..57 patches, which might give a (false) notion that we were fixing bugs, but most of them were features 😓