New model additions

Llama 3

Llama 3 is supported in this release through the Llama 2 architecture and some fixes in the tokenizers library.

Idefics2

The Idefics2 model was created by the Hugging Face M4 team and authored by Léo Tronchon, Hugo Laurencon, Victor Sanh. The accompanying blog post can be found here.

Idefics2 is an open multimodal model that accepts arbitrary sequences of image and text inputs and produces text outputs. The model can answer questions about images, describe visual content, create stories grounded on multiple images, or simply behave as a pure language model without visual inputs. It improves upon IDEFICS-1, notably on document understanding, OCR, or visual reasoning. Idefics2 is lightweight (8 billion parameters) and treats images in their native aspect ratio and resolution, which allows for varying inference efficiency.

Add Idefics2 by @amyeroberts in #30253

Recurrent Gemma

Recurrent Gemma architecture. Taken from the original paper.

The Recurrent Gemma model was proposed in RecurrentGemma: Moving Past Transformers for Efficient Open Language Models by the Griffin, RLHF and Gemma Teams of Google.

The abstract from the paper is the following:

We introduce RecurrentGemma, an open language model which uses Google’s novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. It has a fixed-sized state, which reduces memory use and enables efficient inference on long sequences. We provide a pre-trained model with 2B non-embedding parameters, and an instruction tuned variant. Both models achieve comparable performance to Gemma-2B despite being trained on fewer tokens.

Add recurrent gemma by @ArthurZucker in #30143

Jamba

Jamba is a pretrained, mixture-of-experts (MoE) generative text model, with 12B active parameters and an overall of 52B parameters across all experts. It supports a 256K context length, and can fit up to 140K tokens on a single 80GB GPU.

As depicted in the diagram below, Jamba’s architecture features a blocks-and-layers approach that allows Jamba to successfully integrate Transformer and Mamba architectures altogether. Each Jamba block contains either an attention or a Mamba layer, followed by a multi-layer perceptron (MLP), producing an overall ratio of one Transformer layer out of every eight total layers.

Jamba introduces the first HybridCache object that allows it to natively support assisted generation, contrastive search, speculative decoding, beam search and all of the awesome features from the generate API!

Add jamba by @tomeras91 in #29943

DBRX

DBRX is a transformer-based decoder-only large language model (LLM) that was trained using next-token prediction. It uses a fine-grained mixture-of-experts (MoE) architecture with 132B total parameters of which 36B parameters are active on any input.

It was pre-trained on 12T tokens of text and code data. Compared to other open MoE models like Mixtral-8x7B and Grok-1, DBRX is fine-grained, meaning it uses a larger number of smaller experts. DBRX has 16 experts and chooses 4, while Mixtral-8x7B and Grok-1 have 8 experts and choose 2.

This provides 65x more possible combinations of experts and the authors found that this improves model quality. DBRX uses rotary position encodings (RoPE), gated linear units (GLU), and grouped query attention (GQA).

Add DBRX Model by @abhi-mosaic in #29921

OLMo

The OLMo model was proposed in OLMo: Accelerating the Science of Language Models by Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, Hannaneh Hajishirzi.

OLMo is a series of Open Language Models designed to enable the science of language models. The OLMo models are trained on the Dolma dataset. We release all code, checkpoints, logs (coming soon), and details involved in training these models.

Add OLMo model family by @2015aroras in #29890

Qwen2MoE

Qwen2MoE is the new model series of large language models from the Qwen team. Previously, we released the Qwen series, including Qwen-72B, Qwen-1.8B, Qwen-VL, Qwen-Audio, etc.

Model Details Qwen2MoE is a language model series including decoder language models of different model sizes. For each size, we release the base language model and the aligned chat model. Qwen2MoE has the following architectural choices:

Qwen2MoE is based on the Transformer architecture with SwiGLU activation, attention QKV bias, group query attention, mixture of sliding window attention and full attention, etc. Additionally, we have an improved tokenizer adaptive to multiple natural languages and codes. Qwen2MoE employs Mixture of Experts (MoE) architecture, where the models are upcycled from dense language models. For instance, Qwen1.5-MoE-A2.7B is upcycled from Qwen-1.8B. It has 14.3B parameters in total and 2.7B activated parameters during runtime, while it achieves comparable performance with Qwen1.5-7B, with only 25% of the training resources.

Add Qwen2MoE by @bozheng-hit in #29377

Grounding Dino

Taken from the original paper.

The Grounding DINO model was proposed in Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection by Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang. Grounding DINO extends a closed-set object detection model with a text encoder, enabling open-set object detection. The model achieves remarkable results, such as 52.5 AP on COCO zero-shot.

Adding grounding dino by @EduardoPach in #26087

Static pretrained maps

Static pretrained maps have been removed from the library's internals and are currently deprecated. These used to reflect all the available checkpoints for a given architecture on the Hugging Face Hub, but their presence does not make sense in light of the huge growth of checkpoint shared by the community.

With the objective of lowering the bar of model contributions and reviewing, we first start by removing legacy objects such as this one which do not serve a purpose.

Remove static pretrained maps from the library's internals by @LysandreJik in #29112

Notable improvements

Processors improvements

Processors are ungoing changes in order to uniformize them and make them clearer to use.

Separate out kwargs in processor by @amyeroberts in #30193
[Processor classes] Update docs by @NielsRogge in #29698

SDPA

re-introduced the fast path for sdpa by @fxmarty in #30070

Push to Hub for pipelines

Pipelines can now be pushed to Hub using a convenient push_to_hub method.

add push_to_hub to pipeline by @not-lain in #29172

Flash Attention 2 for more models (M2M100, NLLB, GPT2, MusicGen) !

Thanks to the community contribution, Flash Attention 2 has been integrated for more architectures

Adding Flash Attention 2 Support for GPT2 by @EduardoPach in #29226
Add Flash Attention 2 support to Musicgen and Musicgen Melody by @ylacombe in #29939
Add Flash Attention 2 to M2M100 model by @visheratin in #30256

Improvements and bugfixes

[docs] Remove redundant - and the from custom_tools.md by @windsonsea in #29767
Fixed typo in quantization_config.py by @kurokiasahi222 in #29766
OWL-ViT box_predictor inefficiency issue by @RVV-karma in #29712
Allow -OO mode for docstring_decorator by @matthid in #29689
fix issue with logit processor during beam search in Flax by @giganttheo in #29636
Fix docker image build for Latest PyTorch + TensorFlow [dev] by @ydshieh in #29764
[LlavaNext] Fix llava next unsafe imports by @ArthurZucker in #29773
Cast bfloat16 to float32 for Numpy conversions by @Rocketknight1 in #29755
Silence deprecations and use the DataLoaderConfig by @muellerzr in #29779
Add deterministic config to set_seed by @muellerzr in #29778
Add support for torch_dtype in the run_mlm example by @jla524 in #29776
Generate: remove legacy generation mixin imports by @gante in #29782
Llama: always convert the causal mask in the SDPA code path by @gante in #29663
Prepend bos token to Blip generations by @zucchini-nlp in #29642
Change in-place operations to out-of-place in LogitsProcessors by @zucchini-nlp in #29680
[quality] update quality check to make sure we check imports 😈 by @ArthurZucker in #29771
Fix type hint for train_dataset param of Trainer.init() to allow IterableDataset. Issue 29678 by @stevemadere in #29738
Enable AMD docker build CI by @IlyasMoutawwakil in #29803
Correct llava mask & fix missing setter for vocab_size by @fxmarty in #29389
rm input dtype change in CPU by @jiqing-feng in #28631
Generate: remove unused attributes in AssistedCandidateGenerator by @gante in #29787
replaced concatenation to f-strings to improve readability and unify … by @igeni in #29785
[cleanup] vestiges of causal mask by @ArthurZucker in #29806
Complete security policy with mentions of remote code by @LysandreJik in #29707
[SuperPoint] Fix doc example by @amyeroberts in #29816
[DOCS] Fix typo for llava next docs by @aliencaocao in #29829
model_summary.md - Restore link to Harvard's Annotated Transformer. by @gamepad-coder in #29702
Fix the behavior of collecting 'num_input_tokens_seen' by @YouliangHUANG in #29099
Populate torch_dtype from model to pipeline by @B-Step62 in #28940
remove quotes in code example by @johko in #29812
Add warnings if training args differ from checkpoint trainer state by @jonflynng in #29255
Replace 'decord' with 'av' in VideoClassificationPipeline by @Tyx-main in #29747
Fix header in IFE task guide by @merveenoyan in #29859
[docs] Indent ordered list in add_new_model.md by @windsonsea in #29796
Allow bos_token_id is None during the generation with inputs_embeds by @LZHgrla in #29772
Add cosine_with_min_lr scheduler in Trainer by @liuyanyi in #29341
Disable AMD memory benchmarks by @IlyasMoutawwakil in #29871
Set custom_container in build docs workflows by @Wauplin in #29855
Support num_attention_heads != num_key_value_heads in Flax Llama Implementation by @bminixhofer in #29557
Mamba slow_forward gradient fix by @vasqu in #29563
Fix 29807, sinusoidal positional encodings overwritten by post_init() by @hovnatan in #29813
Reimplement "Automatic safetensors conversion when lacking these files" by @LysandreJik in #29846
fix fuyu device_map compatibility by @SunMarc in #29880
Move eos_token_id to stopping criteria by @zucchini-nlp in #29459
add Cambricon MLUs support by @huismiling in #29627
MixtralSparseMoeBlock: add gate jitter by @lorenzoverardo in #29865
Fix typo in T5Block error message by @Mingosnake in #29881
[make fix-copies] update and help by @ArthurZucker in #29924
[GptNeox] don't gather on pkv when using the trainer by @ArthurZucker in #29892
[pipeline]. Zero shot add doc warning by @ArthurZucker in #29845
[doc] fix some typos and add xpu to the testing documentation by @faaany in #29894
Tests: replace torch.testing.assert_allclose by torch.testing.assert_close by @gante in #29915
Add beam search visualizer to the doc by @aymeric-roucher in #29876
Safe import of LRScheduler by @amyeroberts in #29919
add functions to inspect model and optimizer status to trainer.py by @CKeibel in #29838
RoPE models: add numerical sanity-check test for RoPE scaling by @gante in #29808
[Mamba] from pretrained issue with self.embeddings by @ArthurZucker in #29851
[ TokenizationLlama] fix the way we convert tokens to strings to keep leading spaces 🚨 breaking fix by @ArthurZucker in #29453
Allow GradientAccumulationPlugin to be configured from AcceleratorConfig by @fabianlim in #29589
[BC] Fix BC for other libraries by @ArthurZucker in #29934
Fix doc issue #29758 in DebertaV2Config class by @vinayakkgarg in #29842
[LlamaSlowConverter] Slow to Fast better support by @ArthurZucker in #29797
Update installs in image classification doc by @MariaHei in #29947
[StableLm] Add QK normalization and Parallel Residual Support by @jon-tow in #29745
Mark test_eager_matches_sdpa_generate flaky for some models by @ydshieh in #29479
Super tiny fix 12 typos about "with with" by @fzyzcjy in #29926
Fix rope theta for OpenLlama by @jla524 in #29893
Add warning message for run_qa.py by @jla524 in #29867
fix: get mlflow version from mlflow-skinny by @clumsy in #29918
Reset alarm signal when the function is ended by @coldnight in #29706
Update model card and link of blog post. by @bozheng-hit in #29928
[BC] Fix BC for AWQ quant by @TechxGenus in #29965
Rework tests to compare trainer checkpoint args by @muellerzr in #29883
Fix FA2 tests by @ylacombe in #29909
Fix copies main ci by @ArthurZucker in #29979
[tests] fix the wrong output in ImageToTextPipelineTests.test_conditional_generation_llava by @faaany in #29975
Generate: move misplaced test by @gante in #29902
[docs] Big model loading by @stevhliu in #29920
[generate] fix breaking change for patch by @ArthurZucker in #29976
Fix 29807 sinusoidal positional encodings in Flaubert, Informer and XLM by @hovnatan in #29904
[bnb] Fix bug in _replace_with_bnb_linear by @SunMarc in #29958
Adding FlaxNoRepeatNGramLogitsProcessor by @giganttheo in #29677
[Docs] Make an ordered list prettier in add_tensorflow_model.md by @windsonsea in #29949
Fix skip_special_tokens for Wav2Vec2CTCTokenizer._decode by @msublee in #29311
Hard error when ignoring tensors. by @Narsil in #27484)
Generate: fix logits processors doctests by @gante in #29718
Fix remove_columns in text-classification example by @mariosasko in #29351
Update tests/utils/tiny_model_summary.json by @ydshieh in #29941
Make EncodecModel.decode ONNX exportable by @fxmarty in #29913
Fix Swinv2ForImageClassification NaN output by @miguelm-almeida in #29981
Fix Qwen2Tokenizer by @jklj077 in #29929
Fix kwargs handling in generate_with_fallback by @cifkao in #29225
Fix probability computation in WhisperNoSpeechDetection when recomputing scores by @cifkao in #29248
Fix vipllava for generation by @zucchini-nlp in #29874
[docs] Fix audio file by @stevhliu in #30006
Superpoint imports fix by @zucchini-nlp in #29898
[Main CIs] Fix the red cis by @ArthurZucker in #30022
Make clearer about zero_init requirements by @muellerzr in #29879
Enable multi-device for efficientnet by @jla524 in #29989
Add a converter from mamba_ssm -> huggingface mamba by @byi8220 in #29705
[ProcessingIdefics] Attention mask bug with padding by @byi8220 in #29449
Add whisper to IMPORTANT_MODELS by @ydshieh in #30046
skip test_encode_decode_fast_slow_all_tokens for now by @ydshieh in #30044
if output is tuple like facebook/hf-seamless-m4t-medium, waveform is … by @sywangyi in #29722
Fix mixtral ONNX Exporter Issue. by @AdamLouly in #29858
[Trainer] Allow passing image processor by @NielsRogge in #29896
[bnb] Fix offload test by @SunMarc in #30039
Update quantizer_bnb_4bit.py: In the ValueError string there should be "....you need to set llm_int8_enable_fp32_cpu_offload=True...." instead of "load_in_8bit_fp32_cpu_offload=True". by @miRx923 in #30013
[test fetcher] Always include the directly related test files by @ydshieh in #30050
Fix torch.fx symbolic tracing for LLama by @michaelbenayoun in #30047
Refactor daily CI workflow by @ydshieh in #30012
Add docstrings and types for MambaCache by @koayon in #30023
Fix auto tests by @ydshieh in #30067
Fix whisper kwargs and generation config by @zucchini-nlp in #30018
doc: Correct spelling mistake by @caiyili in #30107
[Whisper] Computing features on GPU in batch mode for whisper feature extractor. by @vaibhavagg303 in #29900
Change log level to warning for num_train_epochs override by @xu-song in #30014
Make MLFlow version detection more robust and handles mlflow-skinny by @helloworld1 in #29957
updated examples/pytorch/language-modeling scripts and requirements.txt to require datasets>=2.14.0 by @Patchwork53 in #30120
[tests] add require_bitsandbytes marker by @faaany in #30116
fixing issue 30034 - adding data format for run_ner.py by @JINO-ROHIT in #30088
Patch fix - don't use safetensors for TF models by @amyeroberts in #30118
[#29174] ImportError Fix: Trainer with PyTorch requires accelerate>=0.20.1 Fix by @UtkarshaGupte in #29888
Accept token in trainer.push_to_hub() by @mapmeld in #30093
fix learning rate display in trainer when using galore optimizer by @vasqu in #30085
Fix falcon with SDPA, alibi but no passed mask by @fxmarty in #30123
Trainer / Core : Do not change init signature order by @younesbelkada in #30126
Make vitdet jit trace complient by @fxmarty in #30065
Fix typo at ImportError by @DrAnaximandre in #30090
Adding mps as device for Pipeline class by @fnhirwa in #30080
Fix failing DeepSpeed model zoo tests by @pacman100 in #30112
Add datasets.Dataset to Trainer's train_dataset and eval_dataset type hints by @ringohoffman in #30077
Fix docs Pop2Piano by @zucchini-nlp in #30140
Revert workaround for TF safetensors loading by @Rocketknight1 in #30128
[Trainer] Fix default data collator by @NielsRogge in #30142
[Trainer] Undo #29896 by @NielsRogge in #30129
Fix slow tests for important models to be compatible with A10 runners by @ydshieh in #29905
Send headers when converting safetensors by @ydshieh in #30144
Fix quantization tests by @SunMarc in #29914
[docs] Fix image segmentation guide by @stevhliu in #30132
[CI] Fix setup by @SunMarc in #30147
Fix length related warnings in speculative decoding by @zucchini-nlp in #29585
Fix and simplify semantic-segmentation example by @qubvel in #30145
[CI] Quantization workflow fix by @SunMarc in #30158
[tests] make 2 tests device-agnostic by @faaany in #30008
Add str to TrainingArguments report_to type hint by @ringohoffman in #30078
[UDOP] Fix tests by @NielsRogge in #29573
[UDOP] Improve docs, add resources by @NielsRogge in #29571
Fix accelerate kwargs for versions <0.28.0 by @vasqu in #30086
Fix typing annotation in hf_argparser by @xu-song in #30156
Fixing a bug when MlFlow try to log a torch.tensor by @etiennebonnafoux in #29932
Fix natten install in docker by @ydshieh in #30161
FIX / bnb: fix torch compatiblity issue with itemize by @younesbelkada in #30162
Update config class check in auto factory by @Rocketknight1 in #29854
Fixed typo in comments/documentation for Pipelines documentation by @DamonGuzman in #30170
Fix Llava chat template examples by @lewtun in #30130
Guard XLA version imports by @muellerzr in #30167
chore: remove repetitive words by @hugehope in #30174
fix: Fixed ruff configuration to avoid deprecated configuration warning by @Sai-Suraj-27 in #30179
Refactor Cohere Model by @saurabhdash2512 in #30027
Update output of SuperPointForKeypointDetection by @NielsRogge in #29809
Falcon: make activation, ffn_hidden_size configurable by @sshleifer in #30134
Docs PR template by @stevhliu in #30171
ENH: [CI] Add new workflow to run slow tests of important models on push main if they are modified by @younesbelkada in #29235
Fix pipeline logger.warning_once bug by @amyeroberts in #30195
fix: Replaced deprecated logger.warn with logger.warning by @Sai-Suraj-27 in #30197
fix typo by @mdeff in #30220
fix fuyu doctest by @molbap in #30215
Fix RecurrentGemmaIntegrationTest.test_2b_sample by @ydshieh in #30222
Update modeling_bark.py by @bes-dev in #30221
Fix/Update for doctest by @ydshieh in #30216
Fixed config.json download to go to user-supplied cache directory by @ulatekh in #30189
Add test for parse_json_file and change typing to os.PathLike by @xu-song in #30183
fix: Replace deprecated assertEquals with assertEqual by @Sai-Suraj-27 in #30241
Set pad_token in run_glue_no_trainer.py #28534 by @JINO-ROHIT in #30234
fix: Replaced deprecated typing.Text with str by @Sai-Suraj-27 in #30230
Refactor doctest by @ydshieh in #30210
fix: Fixed type annotation for compatability with python 3.8 by @Sai-Suraj-27 in #30243
Fix doctest more (for docs/source/en) by @ydshieh in #30247
round epoch only in console by @xdedss in #30237
update github actions packages' version to suppress warnings by @ydshieh in #30249
[tests] add the missing require_torch_multi_gpu flag by @faaany in #30250
[Docs] Update recurrent_gemma.md for some minor nits by @sayakpaul in #30238
Remove incorrect arg in codellama doctest by @Rocketknight1 in #30257
Update ko/_toctree.yml by @jungnerd in #30062
More fixes for doctest by @ydshieh in #30265
FIX: Fix corner-case issue with the important models workflow by @younesbelkada in #30212
FIX: Fix 8-bit serialization tests by @younesbelkada in #30051
Allow for str versions of dicts based on typing by @muellerzr in #30227
Workflow: Update tailscale to release version by @younesbelkada in #30268
Raise relevent err when wrong type is passed in as the accelerator_config by @muellerzr in #29997
BLIP - fix pt-tf equivalence test by @amyeroberts in #30258
fix: Fixed a raise statement by @Sai-Suraj-27 in #30275
Fix test fetcher (doctest) + Idefics2's doc example by @ydshieh in #30274
Fix SDPA sliding window compatibility by @fxmarty in #30127
Fix SpeechT5 forward docstrings by @ylacombe in #30287
FIX / AWQ: Fix failing exllama test by @younesbelkada in #30288
Configuring Translation Pipelines documents update #27753 by @UtkarshaGupte in #29986
Enable fx tracing for Mistral by @zucchini-nlp in #30209
Fix test ExamplesTests::test_run_translation by @ydshieh in #30281
Fix Fatal Python error: Bus error in ZeroShotAudioClassificationPipelineTests by @ydshieh in #30283
FIX: Fix push important models CI by @younesbelkada in #30291
Add token type ids to CodeGenTokenizer by @st81 in #29265
Add strategy to store results in evaluation loop by @qubvel in #30267
Upgrading to tokenizers 0.19.0 by @Narsil in #30289
Re-enable SDPA's FA2 path by @fxmarty in #30070
Fix quality Olmo + SDPA by @fxmarty in #30302
Fix donut token2json multiline by @qubvel in #30300
Fix all torch pipeline failures except one by @ydshieh in #30290
Add atol for sliding window test by @fxmarty in #30303
Fix RecurrentGemma device_map by @SunMarc in #30273
Revert "Re-enable SDPA's FA2 path by @ArthurZucker in #30070)"
Do not drop mask with SDPA for more cases by @fxmarty in #30311
FIX: Fixes unexpected behaviour for Llava / LLama & AWQ Fused modules + revert #30070 at the same time by @younesbelkada in #30317

Significant community contributions

The following contributors have made significant changes to the library over the last release:

@bozheng-hit
- Add Qwen2MoE (#29377)
- Update model card and link of blog post. (#29928)
@EduardoPach
- Adding Flash Attention 2 Support for GPT2 (#29226)
- Adding grounding dino (#26087)
@2015aroras
- Add OLMo model family (#29890)
@tomeras91
- Add jamba (#29943)
@abhi-mosaic
- Add DBRX Model (#29921)

New model additions

Llama 3

Llama 3 is supported in this release through the Llama 2 architecture and some fixes in the tokenizers library.

Idefics2

The Idefics2 model was created by the Hugging Face M4 team and authored by Léo Tronchon, Hugo Laurencon, Victor Sanh. The accompanying blog post can be found here.

Add Idefics2 by @amyeroberts in #30253

Recurrent Gemma

Recurrent Gemma architecture. Taken from the original paper.

The Recurrent Gemma model was proposed in RecurrentGemma: Moving Past Transformers for Efficient Open Language Models by the Griffin, RLHF and Gemma Teams of Google.

The abstract from the paper is the following:

Add recurrent gemma by @ArthurZucker in #30143

Jamba

Add jamba by @tomeras91 in #29943

DBRX

Add DBRX Model by @abhi-mosaic in #29921

OLMo

Add OLMo model family by @2015aroras in #29890

Qwen2MoE

Qwen2MoE is the new model series of large language models from the Qwen team. Previously, we released the Qwen series, including Qwen-72B, Qwen-1.8B, Qwen-VL, Qwen-Audio, etc.

Add Qwen2MoE by @bozheng-hit in #29377

Grounding Dino

Taken from the original paper.

Adding grounding dino by @EduardoPach in #26087

Static pretrained maps

With the objective of lowering the bar of model contributions and reviewing, we first start by removing legacy objects such as this one which do not serve a purpose.

Remove static pretrained maps from the library's internals by @LysandreJik in #29112

Notable improvements

Processors improvements

Processors are ungoing changes in order to uniformize them and make them clearer to use.

Separate out kwargs in processor by @amyeroberts in #30193
[Processor classes] Update docs by @NielsRogge in #29698

SDPA

re-introduced the fast path for sdpa by @fxmarty in #30070

Push to Hub for pipelines

Pipelines can now be pushed to Hub using a convenient push_to_hub method.

add push_to_hub to pipeline by @not-lain in #29172

Flash Attention 2 for more models (M2M100, NLLB, GPT2, MusicGen) !

Thanks to the community contribution, Flash Attention 2 has been integrated for more architectures

Adding Flash Attention 2 Support for GPT2 by @EduardoPach in #29226
Add Flash Attention 2 support to Musicgen and Musicgen Melody by @ylacombe in #29939
Add Flash Attention 2 to M2M100 model by @visheratin in #30256

Improvements and bugfixes

[docs] Remove redundant - and the from custom_tools.md by @windsonsea in #29767
Fixed typo in quantization_config.py by @kurokiasahi222 in #29766
OWL-ViT box_predictor inefficiency issue by @RVV-karma in #29712
Allow -OO mode for docstring_decorator by @matthid in #29689
fix issue with logit processor during beam search in Flax by @giganttheo in #29636
Fix docker image build for Latest PyTorch + TensorFlow [dev] by @ydshieh in #29764
[LlavaNext] Fix llava next unsafe imports by @ArthurZucker in #29773
Cast bfloat16 to float32 for Numpy conversions by @Rocketknight1 in #29755
Silence deprecations and use the DataLoaderConfig by @muellerzr in #29779
Add deterministic config to set_seed by @muellerzr in #29778
Add support for torch_dtype in the run_mlm example by @jla524 in #29776
Generate: remove legacy generation mixin imports by @gante in #29782
Llama: always convert the causal mask in the SDPA code path by @gante in #29663
Prepend bos token to Blip generations by @zucchini-nlp in #29642
Change in-place operations to out-of-place in LogitsProcessors by @zucchini-nlp in #29680
[quality] update quality check to make sure we check imports 😈 by @ArthurZucker in #29771
Fix type hint for train_dataset param of Trainer.init() to allow IterableDataset. Issue 29678 by @stevemadere in #29738
Enable AMD docker build CI by @IlyasMoutawwakil in #29803
Correct llava mask & fix missing setter for vocab_size by @fxmarty in #29389
rm input dtype change in CPU by @jiqing-feng in #28631
Generate: remove unused attributes in AssistedCandidateGenerator by @gante in #29787
replaced concatenation to f-strings to improve readability and unify … by @igeni in #29785
[cleanup] vestiges of causal mask by @ArthurZucker in #29806
Complete security policy with mentions of remote code by @LysandreJik in #29707
[SuperPoint] Fix doc example by @amyeroberts in #29816
[DOCS] Fix typo for llava next docs by @aliencaocao in #29829
model_summary.md - Restore link to Harvard's Annotated Transformer. by @gamepad-coder in #29702
Fix the behavior of collecting 'num_input_tokens_seen' by @YouliangHUANG in #29099
Populate torch_dtype from model to pipeline by @B-Step62 in #28940
remove quotes in code example by @johko in #29812
Add warnings if training args differ from checkpoint trainer state by @jonflynng in #29255
Replace 'decord' with 'av' in VideoClassificationPipeline by @Tyx-main in #29747
Fix header in IFE task guide by @merveenoyan in #29859
[docs] Indent ordered list in add_new_model.md by @windsonsea in #29796
Allow bos_token_id is None during the generation with inputs_embeds by @LZHgrla in #29772
Add cosine_with_min_lr scheduler in Trainer by @liuyanyi in #29341
Disable AMD memory benchmarks by @IlyasMoutawwakil in #29871
Set custom_container in build docs workflows by @Wauplin in #29855
Support num_attention_heads != num_key_value_heads in Flax Llama Implementation by @bminixhofer in #29557
Mamba slow_forward gradient fix by @vasqu in #29563
Fix 29807, sinusoidal positional encodings overwritten by post_init() by @hovnatan in #29813
Reimplement "Automatic safetensors conversion when lacking these files" by @LysandreJik in #29846
fix fuyu device_map compatibility by @SunMarc in #29880
Move eos_token_id to stopping criteria by @zucchini-nlp in #29459
add Cambricon MLUs support by @huismiling in #29627
MixtralSparseMoeBlock: add gate jitter by @lorenzoverardo in #29865
Fix typo in T5Block error message by @Mingosnake in #29881
[make fix-copies] update and help by @ArthurZucker in #29924
[GptNeox] don't gather on pkv when using the trainer by @ArthurZucker in #29892
[pipeline]. Zero shot add doc warning by @ArthurZucker in #29845
[doc] fix some typos and add xpu to the testing documentation by @faaany in #29894
Tests: replace torch.testing.assert_allclose by torch.testing.assert_close by @gante in #29915
Add beam search visualizer to the doc by @aymeric-roucher in #29876
Safe import of LRScheduler by @amyeroberts in #29919
add functions to inspect model and optimizer status to trainer.py by @CKeibel in #29838
RoPE models: add numerical sanity-check test for RoPE scaling by @gante in #29808
[Mamba] from pretrained issue with self.embeddings by @ArthurZucker in #29851
[ TokenizationLlama] fix the way we convert tokens to strings to keep leading spaces 🚨 breaking fix by @ArthurZucker in #29453
Allow GradientAccumulationPlugin to be configured from AcceleratorConfig by @fabianlim in #29589
[BC] Fix BC for other libraries by @ArthurZucker in #29934
Fix doc issue #29758 in DebertaV2Config class by @vinayakkgarg in #29842
[LlamaSlowConverter] Slow to Fast better support by @ArthurZucker in #29797
Update installs in image classification doc by @MariaHei in #29947
[StableLm] Add QK normalization and Parallel Residual Support by @jon-tow in #29745
Mark test_eager_matches_sdpa_generate flaky for some models by @ydshieh in #29479
Super tiny fix 12 typos about "with with" by @fzyzcjy in #29926
Fix rope theta for OpenLlama by @jla524 in #29893
Add warning message for run_qa.py by @jla524 in #29867
fix: get mlflow version from mlflow-skinny by @clumsy in #29918
Reset alarm signal when the function is ended by @coldnight in #29706
Update model card and link of blog post. by @bozheng-hit in #29928
[BC] Fix BC for AWQ quant by @TechxGenus in #29965
Rework tests to compare trainer checkpoint args by @muellerzr in #29883
Fix FA2 tests by @ylacombe in #29909
Fix copies main ci by @ArthurZucker in #29979
[tests] fix the wrong output in ImageToTextPipelineTests.test_conditional_generation_llava by @faaany in #29975
Generate: move misplaced test by @gante in #29902
[docs] Big model loading by @stevhliu in #29920
[generate] fix breaking change for patch by @ArthurZucker in #29976
Fix 29807 sinusoidal positional encodings in Flaubert, Informer and XLM by @hovnatan in #29904
[bnb] Fix bug in _replace_with_bnb_linear by @SunMarc in #29958
Adding FlaxNoRepeatNGramLogitsProcessor by @giganttheo in #29677
[Docs] Make an ordered list prettier in add_tensorflow_model.md by @windsonsea in #29949
Fix skip_special_tokens for Wav2Vec2CTCTokenizer._decode by @msublee in #29311
Hard error when ignoring tensors. by @Narsil in #27484)
Generate: fix logits processors doctests by @gante in #29718
Fix remove_columns in text-classification example by @mariosasko in #29351
Update tests/utils/tiny_model_summary.json by @ydshieh in #29941
Make EncodecModel.decode ONNX exportable by @fxmarty in #29913
Fix Swinv2ForImageClassification NaN output by @miguelm-almeida in #29981
Fix Qwen2Tokenizer by @jklj077 in #29929
Fix kwargs handling in generate_with_fallback by @cifkao in #29225
Fix probability computation in WhisperNoSpeechDetection when recomputing scores by @cifkao in #29248
Fix vipllava for generation by @zucchini-nlp in #29874
[docs] Fix audio file by @stevhliu in #30006
Superpoint imports fix by @zucchini-nlp in #29898
[Main CIs] Fix the red cis by @ArthurZucker in #30022
Make clearer about zero_init requirements by @muellerzr in #29879
Enable multi-device for efficientnet by @jla524 in #29989
Add a converter from mamba_ssm -> huggingface mamba by @byi8220 in #29705
[ProcessingIdefics] Attention mask bug with padding by @byi8220 in #29449
Add whisper to IMPORTANT_MODELS by @ydshieh in #30046
skip test_encode_decode_fast_slow_all_tokens for now by @ydshieh in #30044
if output is tuple like facebook/hf-seamless-m4t-medium, waveform is … by @sywangyi in #29722
Fix mixtral ONNX Exporter Issue. by @AdamLouly in #29858
[Trainer] Allow passing image processor by @NielsRogge in #29896
[bnb] Fix offload test by @SunMarc in #30039
Update quantizer_bnb_4bit.py: In the ValueError string there should be "....you need to set llm_int8_enable_fp32_cpu_offload=True...." instead of "load_in_8bit_fp32_cpu_offload=True". by @miRx923 in #30013
[test fetcher] Always include the directly related test files by @ydshieh in #30050
Fix torch.fx symbolic tracing for LLama by @michaelbenayoun in #30047
Refactor daily CI workflow by @ydshieh in #30012
Add docstrings and types for MambaCache by @koayon in #30023
Fix auto tests by @ydshieh in #30067
Fix whisper kwargs and generation config by @zucchini-nlp in #30018
doc: Correct spelling mistake by @caiyili in #30107
[Whisper] Computing features on GPU in batch mode for whisper feature extractor. by @vaibhavagg303 in #29900
Change log level to warning for num_train_epochs override by @xu-song in #30014
Make MLFlow version detection more robust and handles mlflow-skinny by @helloworld1 in #29957
updated examples/pytorch/language-modeling scripts and requirements.txt to require datasets>=2.14.0 by @Patchwork53 in #30120
[tests] add require_bitsandbytes marker by @faaany in #30116
fixing issue 30034 - adding data format for run_ner.py by @JINO-ROHIT in #30088
Patch fix - don't use safetensors for TF models by @amyeroberts in #30118
[#29174] ImportError Fix: Trainer with PyTorch requires accelerate>=0.20.1 Fix by @UtkarshaGupte in #29888
Accept token in trainer.push_to_hub() by @mapmeld in #30093
fix learning rate display in trainer when using galore optimizer by @vasqu in #30085
Fix falcon with SDPA, alibi but no passed mask by @fxmarty in #30123
Trainer / Core : Do not change init signature order by @younesbelkada in #30126
Make vitdet jit trace complient by @fxmarty in #30065
Fix typo at ImportError by @DrAnaximandre in #30090
Adding mps as device for Pipeline class by @fnhirwa in #30080
Fix failing DeepSpeed model zoo tests by @pacman100 in #30112
Add datasets.Dataset to Trainer's train_dataset and eval_dataset type hints by @ringohoffman in #30077
Fix docs Pop2Piano by @zucchini-nlp in #30140
Revert workaround for TF safetensors loading by @Rocketknight1 in #30128
[Trainer] Fix default data collator by @NielsRogge in #30142
[Trainer] Undo #29896 by @NielsRogge in #30129
Fix slow tests for important models to be compatible with A10 runners by @ydshieh in #29905
Send headers when converting safetensors by @ydshieh in #30144
Fix quantization tests by @SunMarc in #29914
[docs] Fix image segmentation guide by @stevhliu in #30132
[CI] Fix setup by @SunMarc in #30147
Fix length related warnings in speculative decoding by @zucchini-nlp in #29585
Fix and simplify semantic-segmentation example by @qubvel in #30145
[CI] Quantization workflow fix by @SunMarc in #30158
[tests] make 2 tests device-agnostic by @faaany in #30008
Add str to TrainingArguments report_to type hint by @ringohoffman in #30078
[UDOP] Fix tests by @NielsRogge in #29573
[UDOP] Improve docs, add resources by @NielsRogge in #29571
Fix accelerate kwargs for versions <0.28.0 by @vasqu in #30086
Fix typing annotation in hf_argparser by @xu-song in #30156
Fixing a bug when MlFlow try to log a torch.tensor by @etiennebonnafoux in #29932
Fix natten install in docker by @ydshieh in #30161
FIX / bnb: fix torch compatiblity issue with itemize by @younesbelkada in #30162
Update config class check in auto factory by @Rocketknight1 in #29854
Fixed typo in comments/documentation for Pipelines documentation by @DamonGuzman in #30170
Fix Llava chat template examples by @lewtun in #30130
Guard XLA version imports by @muellerzr in #30167
chore: remove repetitive words by @hugehope in #30174
fix: Fixed ruff configuration to avoid deprecated configuration warning by @Sai-Suraj-27 in #30179
Refactor Cohere Model by @saurabhdash2512 in #30027
Update output of SuperPointForKeypointDetection by @NielsRogge in #29809
Falcon: make activation, ffn_hidden_size configurable by @sshleifer in #30134
Docs PR template by @stevhliu in #30171
ENH: [CI] Add new workflow to run slow tests of important models on push main if they are modified by @younesbelkada in #29235
Fix pipeline logger.warning_once bug by @amyeroberts in #30195
fix: Replaced deprecated logger.warn with logger.warning by @Sai-Suraj-27 in #30197
fix typo by @mdeff in #30220
fix fuyu doctest by @molbap in #30215
Fix RecurrentGemmaIntegrationTest.test_2b_sample by @ydshieh in #30222
Update modeling_bark.py by @bes-dev in #30221
Fix/Update for doctest by @ydshieh in #30216
Fixed config.json download to go to user-supplied cache directory by @ulatekh in #30189
Add test for parse_json_file and change typing to os.PathLike by @xu-song in #30183
fix: Replace deprecated assertEquals with assertEqual by @Sai-Suraj-27 in #30241
Set pad_token in run_glue_no_trainer.py #28534 by @JINO-ROHIT in #30234
fix: Replaced deprecated typing.Text with str by @Sai-Suraj-27 in #30230
Refactor doctest by @ydshieh in #30210
fix: Fixed type annotation for compatability with python 3.8 by @Sai-Suraj-27 in #30243
Fix doctest more (for docs/source/en) by @ydshieh in #30247
round epoch only in console by @xdedss in #30237
update github actions packages' version to suppress warnings by @ydshieh in #30249
[tests] add the missing require_torch_multi_gpu flag by @faaany in #30250
[Docs] Update recurrent_gemma.md for some minor nits by @sayakpaul in #30238
Remove incorrect arg in codellama doctest by @Rocketknight1 in #30257
Update ko/_toctree.yml by @jungnerd in #30062
More fixes for doctest by @ydshieh in #30265
FIX: Fix corner-case issue with the important models workflow by @younesbelkada in #30212
FIX: Fix 8-bit serialization tests by @younesbelkada in #30051
Allow for str versions of dicts based on typing by @muellerzr in #30227
Workflow: Update tailscale to release version by @younesbelkada in #30268
Raise relevent err when wrong type is passed in as the accelerator_config by @muellerzr in #29997
BLIP - fix pt-tf equivalence test by @amyeroberts in #30258
fix: Fixed a raise statement by @Sai-Suraj-27 in #30275
Fix test fetcher (doctest) + Idefics2's doc example by @ydshieh in #30274
Fix SDPA sliding window compatibility by @fxmarty in #30127
Fix SpeechT5 forward docstrings by @ylacombe in #30287
FIX / AWQ: Fix failing exllama test by @younesbelkada in #30288
Configuring Translation Pipelines documents update #27753 by @UtkarshaGupte in #29986
Enable fx tracing for Mistral by @zucchini-nlp in #30209
Fix test ExamplesTests::test_run_translation by @ydshieh in #30281
Fix Fatal Python error: Bus error in ZeroShotAudioClassificationPipelineTests by @ydshieh in #30283
FIX: Fix push important models CI by @younesbelkada in #30291
Add token type ids to CodeGenTokenizer by @st81 in #29265
Add strategy to store results in evaluation loop by @qubvel in #30267
Upgrading to tokenizers 0.19.0 by @Narsil in #30289
Re-enable SDPA's FA2 path by @fxmarty in #30070
Fix quality Olmo + SDPA by @fxmarty in #30302
Fix donut token2json multiline by @qubvel in #30300
Fix all torch pipeline failures except one by @ydshieh in #30290
Add atol for sliding window test by @fxmarty in #30303
Fix RecurrentGemma device_map by @SunMarc in #30273
Revert "Re-enable SDPA's FA2 path by @ArthurZucker in #30070)"
Do not drop mask with SDPA for more cases by @fxmarty in #30311
FIX: Fixes unexpected behaviour for Llava / LLama & AWQ Fused modules + revert #30070 at the same time by @younesbelkada in #30317

Significant community contributions

The following contributors have made significant changes to the library over the last release:

@bozheng-hit
- Add Qwen2MoE (#29377)
- Update model card and link of blog post. (#29928)
@EduardoPach
- Adding Flash Attention 2 Support for GPT2 (#29226)
- Adding grounding dino (#26087)
@2015aroras
- Add OLMo model family (#29890)
@tomeras91
- Add jamba (#29943)
@abhi-mosaic
- Add DBRX Model (#29921)

huggingface/transformers

huggingface/transformers

v4.40.0

v4.40.0

New model additions

Llama 3

Idefics2

Recurrent Gemma

Jamba

DBRX

OLMo

Qwen2MoE

Grounding Dino

Static pretrained maps

Notable improvements

Processors improvements

SDPA

Push to Hub for pipelines

Flash Attention 2 for more models (M2M100, NLLB, GPT2, MusicGen) !

Improvements and bugfixes

Significant community contributions

New model additions

Llama 3

Idefics2

Recurrent Gemma

Jamba

DBRX

OLMo

Qwen2MoE

Grounding Dino

Static pretrained maps

Notable improvements

Processors improvements

SDPA

Push to Hub for pipelines

Flash Attention 2 for more models (M2M100, NLLB, GPT2, MusicGen) !

Improvements and bugfixes

Significant community contributions