Exception: variant of untagged enum PyPreTokenizerTypeWrapper

[susannapaoli]

Hello,
When trying to use mistralai/Mistral-7B-Instruct-v0.3 the following error arises

Traceback (most recent call last):
  File "/home/3139819/emotions/religion/./1_get_completions_simplegen_chat.py", line 177, in <module>
    fire.Fire(main)
  File "/home/3139819/anaconda3/envs/emotions/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/3139819/anaconda3/envs/emotions/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/3139819/anaconda3/envs/emotions/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/3139819/emotions/religion/./1_get_completions_simplegen_chat.py", line 74, in main
    generator = SimpleGenerator(
  File "/home/3139819/anaconda3/envs/emotions/lib/python3.10/site-packages/simple_generation/simple_generation.py", line 120, in __init__
    self.tokenizer = AutoTokenizer.from_pretrained(
  File "/home/3139819/anaconda3/envs/emotions/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 814, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/3139819/anaconda3/envs/emotions/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2029, in from_pretrained
    return cls._from_pretrained(
  File "/home/3139819/anaconda3/envs/emotions/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2261, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/3139819/anaconda3/envs/emotions/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 124, in __init__
    super().__init__(
  File "/home/3139819/anaconda3/envs/emotions/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 111, in __init__
    fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
Exception: data did not match any variant of untagged enum PyPreTokenizerTypeWrapper at line 6952 column 3

Library versions:

• simple-generation: 0.2.0
• transformers: 4.37.2
• datasets: 2.16.1

[g8a9]

Mmm it seems to be an issue related to the transformers library. Can you try to update it pip install -U transformers and try again?

[susannapaoli]

Yes, thank you. Updating solved the issue