[Bug]: LLMC integration test failed on XPU

[chensuyue]

Problem Description

LLMC integration test failed on XPU,
https://dev.azure.com/lpot-inc/neural-compressor/_build/results?buildId=53245&view=logs&j=7b714897-4f52-5244-b55a-4a10c39218a3&t=12cd45b8-7610-578e-a9ec-a666072b64ea

link to task: #1149

Reproduction Steps

https://github.com/intel/auto-round/pull/1483/changes
pytest -vs test/test_xpu/test_llmc_integration.py

Environment Information

• Hardware: B60
• AR: Enhance llmc CI on GPU and XPU #1483

Error Logs

=========================== short test summary info ============================
FAILED test_xpu/test_llmc_integration.py::test_oneshot_application[\nquant_stage:\n    quant_modifiers:\n        AutoRoundModifier:\n            ignore: ["lm_head"]\n            iters: 10\n            config_groups:\n                group_0:\n                    targets:\n                        - "Linear"\n                    input_activations: null\n                    output_activations: null\n                    weights:\n                        num_bits: 4\n                        type: "int"\n                        symmetric: true\n                        strategy: group\n                        group_size: 128\n]
FAILED test_xpu/test_llmc_integration.py::test_oneshot_application[recipe1]
FAILED test_xpu/test_llmc_integration.py::test_oneshot_application[recipe2]
FAILED test_xpu/test_llmc_integration.py::test_oneshot_application[recipe3]
FAILED test_xpu/test_llmc_integration.py::test_oneshot_with_device_ids - torc...
FAILED test_xpu/test_llmc_integration.py::test_rtn_oneshot[recipe0] - torch._...
FAILED test_xpu/test_llmc_integration.py::test_rtn_oneshot[recipe1] - NotImpl...
================= 7 failed, 145 warnings in 119.11s (0:01:59) ==================

Additional Context

No response