SGLang is a high-performance open source serving framework for large language models and multimodal models. It is designed to deliver low-latency and high-throughput inference across a wide range of setups, from a single GPU to large distributed clusters.

It is recommended by the engineering team of Apertus, and our Scratchpad repo has an example configuration. You can also find usage guidelines from Hugging Face.

Please visit the official website and documentation for further deployment instructions. For practical examples and tutorials, open the SGLang quickstart.

Architecture diagram Diagram from Getting started with SGLang by Wilson Wu

Overview

For LLM developers on the Apertus team, SGLang streamlines the deployment process with quick server setup, leveraging GPU resources efficiently. Detailed control over model parameters (e.g., max context length, temperature, batch size) allows us to fine-tune performance and behavior.

For production use, consider deploying SGLang with Kubernetes, e.g., using the OME operator or SkyPilot for cloud-scale setups. You can use the Python SDK or API tools like cURL to interact with SGLang, maintaining compatibility with existing workflows.

To get started with SGLang:

  1. Install SGLang:

    pip install sglang huggingface-hub

  2. Start the Service:

    • Via CLI:
      python -m sglang.launcher --model swiss-ai/Apertus-8B-Instruct-2509 --host 0.0.0.0 --port 30000 --tp 1 --max-model-len 8192
    • Or with sglang command (if supported by your version):
      sglang serve -m swiss-ai/Apertus-8B-Instruct-2509 --host 0.0.0.0 --port 30000

  3. Test Inference:

    • Example with cURL:
      curl http://127.0.0.1:30000/v1/chat/completions\  
  -H "Content-Type: application/json"\  
  -H "Authorization: Bearer EMPTY"\  
  -d '{"model": "swiss-ai/Apertus-8B-Instruct-2509",  
       "messages": [  
         {"role": "system", "content": "You are a helpful assistant."},  
         {"role": "user", "content": "Explain SGLang in 3 sentences."}  
       ],  
       "max_tokens": 128,  
       "temperature": 0.2  
     }'

For more advanced use cases, explore the official documentation and community resources.

Edit this page