Yuki
November 17, 2026

AI with TabbyAPI

AI Linux

I followed the following guide: https://github.com/theroyallab/tabbyAPI/wiki/01.-Getting-Started#installing

  1. Clone this repository to your machine: git clone https://github.com/theroyallab/tabbyAPI
  2. Navigate to the project directory: cd tabbyAPI
  3. Run the appropriate start script (start.bat for Windows and start.sh for linux).
    1. Follow the on-screen instructions and select the correct GPU library.
    2. Assuming that the prerequisites are installed and can be located, a virtual environment will be created for you and dependencies will be installed.
  4. The API should start with no model loaded. Please read more to see how to download a model.

To download a model I did the following:

./start.sh download turboderp/Qwen3.8-27B-exl3 --revision SCP_4.00bpw

In this command, the turboderp/Qwen3.8-27B-exl3 is the Hugginface repository and SCP_4.00bpw is the specific branch for the model: https://huggingface.co/turboderp/Qwen3.8-27B-exl3/tree/SCP_4.00bpw

For a list of models: https://huggingface.co/collections/turboderp/exl3-models

Settings in TabbyAPI

I changed the following settings to make it load on my 3090:

Context and cache > Cache mode: Q4

Vision > Vision: Off

Draft model > Draft mode: MTP

API URL and API Key

When starting TabbyAPI it shows the API key and API URL in the terminal you started the script in. You can use this to configure your harness.

feed previous