Load models

Users in the Locai App only see models you have loaded.

Prerequisites

• You can sign in to the Admin Console.
• Locai One has enough free memory for the model you will Load. The bar on Models shows what is in use.

1. Open Models. The bar shows how much memory is in use on each GPU.
2. Select Add models. Choose Locai catalogue or Custom (Hugging Face, USB, or an external disk). Select Add to library to copy the model onto Locai One. Copying does not start it.
3. Return to Models. When the copy is complete, select Load on the model you want. Choose a GPU, set a Name (shown in chat and the Locai API), and the Parameters this instance will use. For a custom model, pick an engine image already on this Locai One, or paste a Docker Hub digest (preferred) or a digest from the signed Locai registry. Leave parameters empty to give this model the GPU. Each Load is a separate instance. If Load cannot be used, the model needs more memory than Locai One has.

The bar on Models shows what this Load reserved. When Load succeeds, the model is at the top of Models and in the Locai App model list.

Third-party models

Third-party models are separate from models loaded on the GPU.

1. Open Third-party.
2. Select the provider, paste the API key, and select Test and add. Locai One keeps the key. People never paste a key. To change it, paste a new key and select Update. Clear the key and select Update to turn that provider off.
3. Search at the top of the model list and select Add. Models you do not add stay off. Models that are on stay in the list when the search is empty. Use Previous and Next when the list is longer than one page.
4. Those models appear in the Locai App under Third-party models. Chat still goes to this Locai One. This Locai One calls the provider.

Locai One needs the internet for this. Check the office connection, then try again.

To stop a model, select Unload. The row stays as Unloading until the engine has stopped and its GPU memory is free, then a message confirms it. If the memory is not freed, the row says so: restart Locai One to clear it. Chat and the Locai API will not reply until another model is loaded.

Change a loaded model

Open the model from Models. Edit Parameters and select Apply and reload. A message confirms when the model is running with the new settings. If they do not start, Locai One goes back to the settings that last worked and tells you why.

To rename the model, add --served-model-name NewName to Parameters and select Apply and reload. The model page, Playground, and the Locai App then show locaione/NewName. The locaione/ part stays the same for every model on this Locai One. The engine restarts to pick up the name.

If a model fails to start, Locai One tries again after 2, 4, 8 and 16 minutes, then about once an hour. To try straight away, unload the model and load it again.

Copies from USB are scanned. They are not as trusted as a Locai catalogue model.

To try a reply, open Playground. If Load does not finish, open the model and read the log, or see Troubleshooting.