Lm_eval static generation improved

[12010486]

Main changes as below:

Precision and device support improvements

• Added a mixed_precision_dtype parameter to the ModelAdapter class, allowing users to specify the desired precision for HPU autocasting. The torch.autocast context is now used during generation, improving performance and memory usage on HPUs.

Static shape and padding logic

• Improved static shape bucket management: the code now calculates the required left-padding for input contexts to fit the selected bucket size, ensuring correct input shapes for HPUs. This includes padding both the context and the attention mask.

Argument and API updates

• Updated the default input length buckets in run_lm_eval.py to remove the largest value. It might caused mismatch with previous legacy results
• Added new arguments (ignore_eos) and updated references for compatibility with the latest version of the underlying evaluation harness.

Note: ignore_eos is passed but not used, as it is decreasing accuracy results.

How to test the impact

The command below was producing an OOM error, now on G2 memory is ~ 27 GB

PT_HPU_LAZY_MODE=1 python run_lm_eval.py --model_name_or_path meta-llama/Llama-3.1-8B-Instruct \
--attn_softmax_bf16 --use_hpu_graphs --limit_hpu_graphs   --use_kv_cache --bf16 --sdp_on_bf16 --trim_logits \
--batch_size=4 --tasks gsm8k_cot_llama -o eval_gsm8k.json --num_fewshot=8 \
--fewshot_as_multiturn --apply_chat_template True 

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[12010486]

Checking this PR on other lm tasks I've seen a drop wrt to the numbers I was having before (on mbpp or humaneval, for example) so converting it to draft while I investigate more

[astachowiczhabana]

Hi @12010486
This code is to counter-effect tokenizer with hardcoded padding right?

[12010486]

Correct, we could have done it also modifying this function https://github.com/EleutherAI/lm-evaluation-harness/blob/v0.4.7/lm_eval/models/huggingface.py/#L858 but I wanted to avoid another function to patch

[astachowiczhabana]

is this change intentional?

[12010486]

Yes, but thanks for double checking. I've introduced it because on v1.19 there was this commit, for granite accuracy, so I wanted to be sure not to introduce back a known regression. 0222c48

[regisss]

LGTM