Fix train data script

[natolambert]

Closes #153, makes it so token isnt needed (can use huggingface-cli), I tested this to retrain OLMo 1.7

[hamishivi]

Have you verified this is the same data?

[hamishivi]

LGTM so long as you've verified the data is the same as our released mixture.

[natolambert]

Code for comparing diffs:

from datasets import load_dataset

def load_and_compare_datasets(dataset_name1, dataset_name2, split='train'):
    # Load datasets
    dataset1 = load_dataset(dataset_name1, split=split)
    dataset2 = load_dataset(dataset_name2, split=split)

    # Create dictionaries indexed by 'id'
    dict1 = {row['id']: row for row in dataset1}
    dict2 = {row['id']: row for row in dataset2}

    # Find unique ids in each dataset
    unique_ids_to_set1 = set(dict1.keys()) - set(dict2.keys())
    unique_ids_to_set2 = set(dict2.keys()) - set(dict1.keys())

    # Print unique entries
    print("Entries unique to dataset 1:")
    for id in unique_ids_to_set1:
        print(dict1[id])

    print("\nEntries unique to dataset 2:")
    for id in unique_ids_to_set2:
        print(dict2[id])

# Example usage with dataset names and optional split specification
load_and_compare_datasets('ai2-adapt-dev/tulu2-tmp', 'allenai/tulu-v2-sft-mixture', split='train')

[natolambert]

There are minor differences, mostly via wizardlm taking data down and maybe slight changes to the dataset processing script from the v2 version on huggingface. New backups made:

• Tulu 2 as of now: https://huggingface.co/datasets/ai2-adapt-dev/tulu2-tmp
• WizardLM from original tulu 2 mix: https://huggingface.co/datasets/ai2-adapt-dev/wizardlm-backup

[hamishivi]

So now the script points to this backup wizardlm version that yields some small differences when you run the train prep script?

[natolambert]

@hamishivi, yes.
Backup was taken from your nfs raw train files. I can get the date it was last modified, but I think there are multiple things that have potentially changed since we uploaded tulu v2 sft mix

[cbfcbf]

I found a typo at line 86 reformat_dataset.py : "if num_few_shot_examples > 0" should be "if num_zero_shot_examples > 0"