Generate public API in keras/api folder.

[sampathweb]

• Moves all sources to keras/src folder.
• Generates Public API in keras/api folder and imports them in keras/__init__.py.
• Added GitHub action (format section) to verify that user runs ./shell/api_gen.sh with the PR and there are no changes to API.

Now, Keras can be installed via pip install -e .. Users can continue to use python pip_build.py which builds the wheel in virtual env and installs it.

To generate public API, run - ./shell/api_gen.sh

@fchollet - One test failing in TensorFlow backend in CPU and GPU. This is because it expects keras.Functional API, but that doesn't exist in the API generated. It only exists in keras.src.Functional. This is true when we do pip_build.py as well and generate pip package. So need your help to sort this out.

FAILED keras/src/trainers/trainer_test.py::TestTrainer::test_trainer_with_raggeds_functional

    def get_functional(self):
        ExampleLayer = self.get_layer()
    
>       class ExampleFunctional(keras.Functional):
E       AttributeError: module 'keras' has no attribute 'Functional'

keras/src/trainers/trainer_test.py:971: AttributeError

Differences with Existing Keras Package Install:

• Installed in Colab to verify that users can install with pip install git+https://github.com/sampathweb/keras.git@api-gen3

>>> dir(keras)
['AbsMaxQuantizer', 'DTypePolicy', 'FloatDTypePolicy', 'Function', 'Initializer', 'Input', 'InputSpec', 'KerasTensor', 'Layer', 'Loss', 'Metric', 'Model', 'Operation', 'Optimizer', 'QuantizedDTypePolicy', 'Quantizer', 'Regularizer', 'Sequential', 'StatelessScope', 'Variable', '__all__', '__builtins__', '__cached__', '__dir__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', 'activations', 'applications', 'backend', 'callbacks', 'config', 'constraints', 'datasets', 'device', 'distribution', 'dtype_policies', 'export', 'initializers', 'layers', 'legacy', 'losses', 'metrics', 'mixed_precision', 'models', 'name_scope', 'ops', 'optimizers', 'preprocessing', 'quantizers', 'random', 'regularizers', 'saving', 'tree', 'utils', 'version']

Note that we hide src and api from intellisense (in keras/__init__.py), but they are available in keras.src (for private symbols) and keras.api (which is same as keras. symbols).

What we currently have for pip install in PyPi, the symbols match except that src, _tf_keras are listed here, but hidden in the new method above (via all) but still available. -

['AbsMaxQuantizer', 'DTypePolicy', 'FloatDTypePolicy', 'Function', 'Initializer', 'Input', 'InputSpec', 'KerasTensor', 'Layer', 'Loss', 'Metric', 'Model', 'Operation', 'Optimizer', 'QuantizedDTypePolicy', 'Quantizer', 'Regularizer', 'Sequential', 'StatelessScope', 'Variable', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_tf_keras', 'activations', 'applications', 'backend', 'callbacks', 'config', 'constraints', 'datasets', 'device', 'distribution', 'dtype_policies', 'export', 'initializers', 'layers', 'legacy', 'losses', 'metrics', 'mixed_precision', 'models', 'name_scope', 'ops', 'optimizers', 'preprocessing', 'quantizers', 'random', 'regularizers', 'saving', 'src', 'tree', 'utils', 'version']

Note that in either dirs, keras.Functional is not exposed directly, but is available in Keras Master branch GitHub keras/__init__.py and how the test for TensorFlow that's failing was trying to access it. Can we change the test to another way that would also pass in installed Keras public APIs?

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 60.39669% with 1198 lines in your changes are missing coverage. Please review.

Project coverage is 69.22%. Comparing base (61bbff5) to head (cf80ddb).
Report is 1 commits behind head on master.

Files	Patch %	Lines
keras/api/_tf_keras/keras/ops/__init__.py	0.00%	217 Missing ⚠️
keras/api/_tf_keras/keras/layers/__init__.py	0.00%	143 Missing ⚠️
keras/api/_tf_keras/keras/ops/numpy/__init__.py	0.00%	140 Missing ⚠️
keras/api/_tf_keras/keras/metrics/__init__.py	0.00%	68 Missing ⚠️
keras/api/_tf_keras/keras/applications/__init__.py	0.00%	57 Missing ⚠️
keras/api/_tf_keras/keras/losses/__init__.py	0.00%	44 Missing ⚠️
keras/api/_tf_keras/keras/utils/__init__.py	0.00%	38 Missing ⚠️
keras/api/_tf_keras/keras/initializers/__init__.py	0.00%	36 Missing ⚠️
keras/api/_tf_keras/keras/ops/nn/__init__.py	0.00%	32 Missing ⚠️
keras/api/_tf_keras/keras/activations/__init__.py	0.00%	23 Missing ⚠️
... and 70 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #19510      +/-   ##
==========================================
- Coverage   76.40%   69.22%   -7.18%     
==========================================
  Files         370      497     +127     
  Lines       41754    44211    +2457     
  Branches     8161     8206      +45     
==========================================
- Hits        31902    30605    -1297     
- Misses       8131    11790    +3659     
- Partials     1721     1816      +95     

Flag	Coverage Δ
keras	69.14% <60.33%> (-7.12%)	⬇️
keras-jax	63.94% <58.38%> (+3.44%)	⬆️
keras-numpy	?
keras-tensorflow	?
keras-torch	64.00% <59.86%> (+3.43%)	⬆️
keras.applications	80.42% <100.00%> (?)
keras.applications-jax	80.42% <100.00%> (?)
keras.applications-numpy	22.93% <100.00%> (?)
keras.applications-tensorflow	80.42% <100.00%> (?)
keras.applications-torch	80.15% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[fchollet]

Thanks for the PR!

@fchollet - One test failing in TensorFlow backend in CPU and GPU. This is because it expects keras.Functional API, but that doesn't exist in the API generated. It only exists in keras.src.Functional. This is true when we do pip_build.py as well and generate pip package. So need your help to sort this out.

This is because previously the "view" of the tests was the keras/ folder (rather than the public Keras API). Since the view has changed we can just update the path to keras.src.models.Functional.

We also need to run some integration checks with tf.keras. Check that the latest TF + this version of Keras work fine together and that the API reflected in tf.keras is the expected _tf_keras namespace, with the legacy entries included.

[sampathweb]

Closing this PR and incorporated the changes in #19530