What's Changed

This release candidate is focused on fixing AutoTokenizer, expanding the dynamic weight loading support, and improving performances with MoEs!

MoEs and performances:

• batched and grouped experts implementations by @IlyasMoutawwakil in https://github.com/huggingface/transformers/pull/42697
• Optimize MoEs for decoding using batched_mm by @IlyasMoutawwakil in https://github.com/huggingface/transformers/pull/43126

Tokenization:

The main issue with the tokenization refactor is that tokenizer_class are now "enforced" when in most cases they are wrong. This took a while to properly isolate and now we try to use TokenizersBackend whenever we can. #42894 has a much more detailed description of the big changes!

• use TokenizersBackend by @ArthurZucker in https://github.com/huggingface/transformers/pull/42894
• Fix convert_tekken_tokenizer by @juliendenize in https://github.com/huggingface/transformers/pull/42592
• refactor more tokenizers - v5 guide update by @itazap in https://github.com/huggingface/transformers/pull/42768
• [Tokenizers] Change treatment of special tokens by @vasqu in https://github.com/huggingface/transformers/pull/42903

Core

Here we focused on boosting the performances of loading weights on device!

• [saving] Simplify general logic by @Cyrilvallez in https://github.com/huggingface/transformers/pull/42766
• Do not rely on config for inferring model dtype by @Cyrilvallez in https://github.com/huggingface/transformers/pull/42838
• Improve BatchFeature: stack list and lists of torch tensors by @yonigozlan in https://github.com/huggingface/transformers/pull/42750
• Remove tied weights from internal attribute if they are not tied by @Cyrilvallez in https://github.com/huggingface/transformers/pull/42871
• Enforce call to post_init and fix all of them by @Cyrilvallez in https://github.com/huggingface/transformers/pull/42873
• Simplify tie weights logic by @Cyrilvallez in https://github.com/huggingface/transformers/pull/42895
• Add buffers to _init_weights for ALL models by @Cyrilvallez in https://github.com/huggingface/transformers/pull/42309
• [loading] Really initialize on meta device for huge perf gains by @Cyrilvallez in https://github.com/huggingface/transformers/pull/42941
• Do not use accelerate hooks if the device_map has only 1 device by @Cyrilvallez in https://github.com/huggingface/transformers/pull/43019
• Move missing weights and non-persistent buffers to correct device earlier by @Cyrilvallez in https://github.com/huggingface/transformers/pull/43021

New models

• Sam: Perception Encoder Audiovisual by @eustlb in https://github.com/huggingface/transformers/pull/42905
• adds jais2 model support by @sarathc-cerebras in https://github.com/huggingface/transformers/pull/42684
• Add Pixio pre-trained models by @LiheYoung in https://github.com/huggingface/transformers/pull/42795
• [Ernie 4.5] Ernie VL models by @vasqu in https://github.com/huggingface/transformers/pull/39585
• [loading][TP] Fix device placement at loading-time, and simplify sharding primitives by @Cyrilvallez in https://github.com/huggingface/transformers/pull/43003
• GLM-ASR Support by @zRzRzRzRzRzRzR in https://github.com/huggingface/transformers/pull/42875

Quantization

• [Devstral] Make sure FP8 conversion works correctly by @patrickvonplaten in https://github.com/huggingface/transformers/pull/42715
• Fp8 dq by @SunMarc in https://github.com/huggingface/transformers/pull/42926
• [Quantization] Removing misleading int8 quantization in Finegrained FP8 by @MekkCyber in https://github.com/huggingface/transformers/pull/42945
• Fix deepspeed + quantization by @SunMarc in https://github.com/huggingface/transformers/pull/43006

Breaking changes

Mostly around processors!

• 🚨 Fix ConvNeXt image processor default interpolation to BICUBIC by @lukepayyapilli in https://github.com/huggingface/transformers/pull/42934
• 🚨 Fix EfficientNet image processor default interpolation to BICUBIC by @lukepayyapilli in https://github.com/huggingface/transformers/pull/42956
• Add fast version of convert_segmentation_map_to_binary_masks to EoMT by @simonreise in https://github.com/huggingface/transformers/pull/43073
• 🚨Fix MobileViT image processor default interpolation to BICUBIC by @lukepayyapilli in https://github.com/huggingface/transformers/pull/43024

Thanks again to everyone !

New Contributors

• @ZX-ModelCloud made their first contribution in https://github.com/huggingface/transformers/pull/42833
• @AYou0207 made their first contribution in https://github.com/huggingface/transformers/pull/42863
• @wasertech made their first contribution in https://github.com/huggingface/transformers/pull/42864
• @preetam1407 made their first contribution in https://github.com/huggingface/transformers/pull/42685
• @Taise228 made their first contribution in https://github.com/huggingface/transformers/pull/41416
• @CandiedCode made their first contribution in https://github.com/huggingface/transformers/pull/42885
• @sarathc-cerebras made their first contribution in https://github.com/huggingface/transformers/pull/42684
• @nandan2003 made their first contribution in https://github.com/huggingface/transformers/pull/42318
• @LiheYoung made their first contribution in https://github.com/huggingface/transformers/pull/42795
• @majiayu000 made their first contribution in https://github.com/huggingface/transformers/pull/42928
• @lukepayyapilli made their first contribution in https://github.com/huggingface/transformers/pull/42934
• @leaderofARS made their first contribution in https://github.com/huggingface/transformers/pull/42966
• @qianyue76 made their first contribution in https://github.com/huggingface/transformers/pull/43095
• @stefgina made their first contribution in https://github.com/huggingface/transformers/pull/43033
• @HuiyingLi made their first contribution in https://github.com/huggingface/transformers/pull/43084
• @raimbekovm made their first contribution in https://github.com/huggingface/transformers/pull/43038
• @PredictiveManish made their first contribution in https://github.com/huggingface/transformers/pull/43053
• @pushkar-hue made their first contribution in https://github.com/huggingface/transformers/pull/42736
• @vykhovanets made their first contribution in https://github.com/huggingface/transformers/pull/43042
• @tanmay2004 made their first contribution in https://github.com/huggingface/transformers/pull/42737
• @atultw made their first contribution in https://github.com/huggingface/transformers/pull/43061

Full Changelog: https://github.com/huggingface/transformers/compare/v5.0.0rc1...v5.0.0rc2