Question about ultrafineweb dataset preparation for tequila/sherry

[koshostanford]

Hello,

I'm trying to reproduce the results of tequila/sherry. But I'm not seeing the accuracy reported by the paper (I attached the tequila results here, trained by me). I suspect it's the way I prepare the dataset is not the same as the authors did. Can you share how we can prepare the ultrafineweb dataset for the tequila/sherry QAT.

Thanks very much
Sho

[GeoMorax]

I have successfully reproduced them. Their performance is aligned with the reported accuracy. I used the top-scoring 10B data slice from Ultrafineweb. For my reproduction, the warmup phase at initial stage of training is important.

I have put reproduced Sherry 1B model on Hugging Face: https://huggingface.co/MoraxGeo/Sherry-1B-1.25bit-per-channel/
Below are my results:

Model	Quant type	arc_challenge	arc_easy	hellaswag	piqa	winogrande	Average
LLaMA3.2	-	0.313	0.654	0.477	0.742	0.603	0.558
Sherry (reported in paper)	per-channel	-	-	-	-	-	0.513
Sherry (reproduced)	per-channel	0.297	0.618	0.386	0.707	0.560	0.514
Sherry (reported in paper)	per-group	0.309	0.647	0.388	0.699	0.550	0.519
Sherry (reproduced)	per-group	0.292	0.648	0.390	0.706	0.556	0.518

[koshostanford]

@GeoMorax is it the warmup ratio parameter? May i ask what number should we set it as?

Thanks

[Little0o0]

@GeoMorax Thanks for reproduction !

[GeoMorax]

@koshostanford I set 20% warmup stage.

[TomLucidor]

@Little0o0 are there any way of turning Tequila/Sherry into MLX format? I find this tool very interesting for accelerating matrix operations! cubist38/mlx-openai-server#161 ml-explore/mlx-lm#812