kl divergence outputs confusing result when the label contains negative

[drewshark]

Here are some APIs implemented for kl divergence:

tf.losses.kullback_leibler_divergence
tf.keras.losses.KLDivergence
tf.keras.metrics.kl_divergence

However, when the y_true or y_pred is negative, these APIs output incorrect result which is inconsistent with the documentation (https://www.tensorflow.org/api_docs/python/tf/keras/losses/KLDivergence).

Here is the code to reproduce:

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
os.environ['OPENBLAS_NUM_THREADS'] = '1'
os.environ['CUDA_VISIBLE_DEVICES'] = ''
import numpy as np
np.random.seed(35)
import tensorflow as tf
y_true = tf.constant(np.random.randn(1), dtype='float32')
y_pred = tf.constant(np.random.randn(1), dtype='float32')
print(y_true, y_pred)
out1 = tf.losses.kullback_leibler_divergence(y_true,y_pred)
out2 = tf.keras.losses.KLDivergence(
    reduction='sum_over_batch_size', name='kl_divergence'
)(y_true, y_pred)
out3 = tf.keras.metrics.kl_divergence(y_true, y_pred)
print(out1, out2, out3)  # 0.0, 0.0, 0.0
print(f"Expected result: {y_true*(np.log(y_true/y_pred))}")  # Expected result following the equation: [-2.8711002]

I notice that the current code will silently clip the negative value close to zero:

keras/keras/losses/losses.py

Lines 1466 to 1467 in 61bbff5

	y_true = ops.clip(y_true, backend.epsilon(), 1)
	y_pred = ops.clip(y_pred, backend.epsilon(), 1)

If negative value is not allowed for these concerned APIs, maybe some descriptions on the documentation or validation check inside the source code can provide more information.

[fchollet]

y_true and y_pred are expected to be probability distributions, not logits, hence the clipping. We should make that clearer in the documentation.

[SuryanarayanaY]

Hi @drewshark ,

The documentation changed as mentioned in above comment in this commit.

Also updated the python example in the PR #19526

[drewshark]

Thank you for your fix.