Apple publishes set of optimized language models – OpenELM (for Open-source Efficient Language Model). Apple provides pretrained and instruction-tuned models with 270M, 450M, 1.1B and 3B parameters.
Our pre-training dataset contains RefinedWeb, deduplicated PILE, a subset of RedPajama, and a subset of Dolma v1.6, totaling approximately 1.8 trillion tokens. Please check license agreements and terms of these datasets before using them.
Apple
Models are available on HuggingFace.
HuggingFace: OpenELM
References and further reading: