AI with TabbyAPI
I followed the following guide: https://github.com/theroyallab/tabbyAPI/wiki/01.-Getting-Started#installing
- Clone this repository to your machine:
git clone https://github.com/theroyallab/tabbyAPI - Navigate to the project directory:
cd tabbyAPI - Run the appropriate start script (
start.batfor Windows andstart.shfor linux).- Follow the on-screen instructions and select the correct GPU library.
- Assuming that the prerequisites are installed and can be located, a virtual environment will be created for you and dependencies will be installed.
- The API should start with no model loaded. Please read more to see how to download a model.
To download a model I did the following:
./start.sh download turboderp/Qwen3.8-27B-exl3 --revision SCP_4.00bpw
In this command, the turboderp/Qwen3.8-27B-exl3 is the Hugginface repository and SCP_4.00bpw is the specific branch for the model: https://huggingface.co/turboderp/Qwen3.8-27B-exl3/tree/SCP_4.00bpw
For a list of models: https://huggingface.co/collections/turboderp/exl3-models
Settings in TabbyAPI
I changed the following settings to make it load on my 3090:
Context and cache > Cache mode: Q4
Vision > Vision: Off
Draft model > Draft mode: MTP
API URL and API Key
When starting TabbyAPI it shows the API key and API URL in the terminal you started the script in. You can use this to configure your harness.