Research News
Papers and technical reports from the Apertus project. This list is continuously expanded: please visit our đź“– Zotero group for other shared literature, and the News area for general announcements.
Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs
Shows that respecting robots.txt opt-outs causes minimal performance degradation.
Positional Fragility in LLMs: How Offset Effects Reshape Our Understanding of Memorization Risks
Research on memorization patterns and copyright risks in LLMs.
Quantifying Training Data Retention in Large Language Models: An Analysis of Pretraining Factors and Mitigation Strategies
Analysis of memorization and mitigation strategies applied in Apertus.
INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge
Multilingual evaluation benchmark across 44 languages.
Deriving Activation Functions Using Integration
xIELU activation function used in Apertus architecture.
Global MMLU: Multilingual Evaluation
Understanding addressing cultural, linguistic biases.
Towards Fully FP8 GEMM LLM Training at Scale
A new LLM architecture, enables unprecedented throughput gains.
Understanding and Minimising Outlier Features
Methods to reduce OFs and improve quantisation without slowing down convergence.
Scaling Laws and Compute-Optimal Training
How scaling experiments can be performed with reduced compute and GPU hours.
Training Dynamics of the Cooldown Stage
Performance impacts of the Warmup-Stable-Decay of learning rate schedulers.
The AdEMAMix Optimizer: Better, Faster, Older
A mixture of EMAs to improve performance in language and image tasks.
Benchmarking optimizers for large language model pretraining
A comprehensive evaluation of recent optimization techniques in LLM pretraining.
Quantile reward policy optimization
Alignment with pointwise regression and exact partition functions.
ConLID
Supervised Contrastive Learning for Low-Resource Language Identification.
Parity-aware byte-pair encoding
Improving cross-lingual fairness in tokenization.
Going over Fine Web with a Fine-Tooth Comb
Technical Report of Indexing Fine Web for Problematic Content Search and Retrieval.
Low-Perplexity LLM-Generated Sequences and Where To Find Them
Tracing texts back to training sources, revealing how data impacts model behavior.
Mixtera: A data plane for foundation model training
Define and dynamically adjust data mixtures during training without performance bottlenecks.
An Engineering Journey Training Large Language Models at Scale on Alps: The Apertus Experience
Details the challenges encountered in readying HPC infrastructure for training AI models.
Apertus LLM Family Expansion via Distillation and Quantization
Apertus-v1.1 is a distilled family of models trained on 1.7T permissive license tokens.
Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors
An optimizer modification to improve warmup, decay, training dynamics and performance across model scales.