Significant Performance Drop in GSM8k Evaluation with Updated SFT ckpt

[yinyueqin]

Hi,

Thank you for your work. We're re-evaluating experiments using an updated SFT ckpt from https://huggingface.co/alignment-handbook/zephyr-7b-sft-full and using lm-evaluation-harness v0.4.0 for evaluation. We've noticed a significant performance drop in GSM8k. We trained the model for 6 epochs in each iteration. Have you observed this issue or have insights into potential causes?

[junkangwu]

It could be related to the version of lm-evaluation-harness. For more details, see #12 (comment).

Additionally, after updating the SFT checkpoint from https://huggingface.co/alignment-handbook/zephyr-7b-sft-full, the relative improvement between iteration 0 and iteration 1 appears to be marginal. Are there any new parameter settings being recommended?

[yinyueqin]

I use lm-evaluation-harness v0.4.0 for evaluation, which is consistent with the evaluation version used by the author. In addition, the results displayed above are obtained using num_train_epochs=6 for training.

[AGTSAAA]

Hi @yinyueqin Have you reproduced the preformance? I also can not reproduce the preformance