SGLang is a high-performance open source serving framework for large language models and multimodal models. It is designed to deliver low-latency and high-throughput inference across a wide range of setups, from a single GPU to large distributed clusters.

🚧 Information for Apertus 1.5 - we are currently working on integrating changes from the latest release. Please stay tuned for updated instructions here soon. Support work for SGLang compatibility is taking place in https://github.com/swiss-ai/sglang

SGLang is well supported by the engineering team of Apertus - our Scratchpad repo has an example configuration. You can also find usage guidelines from Hugging Face.

For practical examples and tutorials, open the SGLang quickstart.

Architecture diagram Diagram from Getting started with SGLang by Wilson Wu

Overview

For LLM developers, SGLang streamlines the deployment process with quick server setup, leveraging GPU resources efficiently. Detailed control over model parameters (e.g., max context length, temperature, batch size) allows us to fine-tune performance and behavior.

For production use, consider deploying SGLang with Kubernetes - for a basic example, see the official documentation. You can also use the OME operator or SkyPilot for cloud-scale setups.

To get started with SGLang:

  1. Install SGLang:

    pip install sglang huggingface-hub

  2. Start the Service:

    • Via CLI:
      python -m sglang.launcher --model swiss-ai/Apertus-8B-Instruct-2509 --host 0.0.0.0 --port 30000 --tp 1 --max-model-len 8192
    • Or with sglang command (if supported by your version):
      sglang serve -m swiss-ai/Apertus-8B-Instruct-2509 --host 0.0.0.0 --port 30000

  3. Test Inference:

    • Example with cURL:
      curl http://127.0.0.1:30000/v1/chat/completions\  
  -H "Content-Type: application/json"\  
  -H "Authorization: Bearer EMPTY"\  
  -d '{"model": "swiss-ai/Apertus-v1.5-8B",  
       "messages": [  
         {"role": "system", "content": "You are a helpful assistant."},  
         {"role": "user", "content": "Explain SGLang in 3 sentences."}  
       ],  
       "max_tokens": 128,  
       "temperature": 0.2  
     }'

Please visit the SGLang website and documentation for further deployment instructions.

Edit this page