This document outlines how to use the Transformers library from Hugging Face with Apertus.

🚧 Information for Apertus 1.5 - we are currently working on integrating changes from the latest release. Apertus 1.5 accepts interleaved text, image, and audio inputs and generates text. The model does not generate audio or images. The integration is not yet part of a released Transformers version (upstreaming is in progress). Until then, install transformers from our branch:

pip install "transformers[torch,vision,audio] @ git+https://github.com/swiss-ai/transformers.git@apertusv1p5-release-pre-upstream"

Transformers acts as the model-definition framework for state-of-the-art machine learning models in text, computer vision, audio, video, and multimodal models, for both inference and training. Visit the official documentation for further information.

Quickstart

Run a command like this first to install the library, as well as PyTorch, using a package manager:

pip install -U transformers torch

With this sample Python code, you can load and prompt Apertus:

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

MODEL_ID = "swiss-ai/Apertus-v1.5-8B"  # or "swiss-ai/Apertus-v1.5-70B"

processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForMultimodalLM.from_pretrained(
    MODEL_ID, dtype="auto", device_map="auto"
).eval()


def generate(messages, max_new_tokens=256, **template_kwargs):
    inputs = processor.apply_chat_template(
        messages,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
        **template_kwargs,
    ).to(model.device)
    with torch.inference_mode():
        output_ids = model.generate(**inputs, max_new_tokens=max_new_tokens)
    return processor.decode(
        output_ids[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True
    )

# prepare the model input
prompt = "Give me a brief explanation of gravity in simple terms."
messages = [
    {"role": "system", "content": "You are a concise and helpful assistant."},
    {"role": "user", "content": prompt}
]
print(generate(messages))

Tip:

  • Check the model card for further examples on using Image or Audio inputs.
  • We recommend setting temperature=0.8 and top_p=0.9 in the sampling parameters.
Edit this page