Triton

Obiguard provides a robust and secure platform to observe, govern, and manage your locally or privately hosted custom models using Triton.

Here’s the official Triton Inference Server documentation for more details.

Integrating Custom Models with Obiguard SDK

Expose your Triton Server

Expose your Triton server by using a tunneling service like ngrok or any other way you prefer. You can skip this step if you’re self-hosting the Gateway.

ngrok http 11434 --host-header="localhost:8080"

Install the Obiguard SDK

Python SDK

pip install obiguard

Initialize Obiguard with Triton custom URL

Pass your publicly-exposed Triton server URL to Obiguard with customHost
Set target provider as triton.

Python SDK

from obiguard import Obiguard

client = Obiguard(
    obiguard_api_key="sk-obg***",  # Your Obiguard API key
    provider="triton",
    custom_host="http://localhost:8000/v2/models/mymodel" # Your Triton Hosted URL
    Authorization="AUTH_KEY", # If you need to pass auth
)

More on custom_host here.

Invoke Chat Completions

Use the Obiguard SDK to invoke chat completions (generate) from your model, just as you would with any other provider:

Python SDK

completion = client.chat.completions.create(
    messages= [{ "role": 'user', "content": 'Say this is a test' }]
)

Next Steps

Explore the complete list of features supported in the SDK:

Ecosystem

AI Apps

Agents

LLMs

Vector Databases

Integrating Custom Models with Obiguard SDK

Next Steps

SDK

​Integrating Custom Models with Obiguard SDK

​Next Steps

SDK

Integrating Custom Models with Obiguard SDK

Next Steps