Agent on Home

Bringing Open WebUI to OpenShift

Sun, 31 May 2026 00:00:00 +0000

Open WebUI on OpenShift, connected to RHAIIS and Hermes Agent over the cluster-internal service network — AI generated

Introduction

In this post, I want to describe how to deploy Open WebUI on OpenShift and connect it to inference endpoints running on the same cluster. This is a continuation of two earlier posts: Running the Red Hat AI Inference Server on OpenShift, which covers the model-serving layer, and Deploying Hermes Agent on OpenShift, which adds an agent layer on top of it.

Both of those posts included a section on connecting to Open WebUI. In both cases, the connection went from an Open WebUI instance running in my homelab to the newly deployed OpenShift endpoints via their public routes. It works, but it requires the inference endpoints to be reachable from outside the cluster, and the API keys to travel across the public internet. Running Open WebUI on the same OpenShift cluster removes that requirement. The connection from Open WebUI to RHAIIS and Hermes goes over the internal cluster service network using Kubernetes DNS names. Neither backend needs a public Route for this to work.

Running the interface on OpenShift also makes practical sense beyond network topology. A homelab instance works for personal use, but once more than a small group needs access, a platform that handles scaling and TLS termination without additional configuration is easier to operate.

Architecture

Open WebUI runs in a dedicated open-webui namespace. It connects to two backends over internal cluster DNS:

RHAIIS, running in the rhaiis namespace, reachable at rhaiis-vllm.rhaiis.svc.cluster.local:8000
Hermes Agent, running in the hermes namespace, reachable at hermes.hermes.svc.cluster.local:8642

Both expose an OpenAI-compatible API. Open WebUI targets both through the OPENAI_API_BASE_URLS environment variable. The API keys in OPENAI_API_KEYS must be supplied in the same order as the URLs.

Open WebUI’s container image runs as UID 0 by default. OpenShift’s restricted SCC does not permit this, so the deployment uses a dedicated ServiceAccount bound to the anyuid SCC.

Prerequisites

The RHAIIS deployment from the previous post must be running in the rhaiis namespace with a deployment named rhaiis-vllm. The RHAIIS API key is needed in a later step.
The Hermes deployment from the Hermes post must be running in the hermes namespace. The Hermes API server key is needed in a later step. If you only want to connect RHAIIS, the Hermes parts can be skipped.

Deploying Open WebUI

All deployment files are available in the smichard/agent_on_ocp GitHub repository. The steps below apply them in sequence.

Clone the repository:

git clone https://github.com/smichard/agent_on_ocp.git
cd open_webui_on_ocp

Create the Namespace

oc new-project open-webui

Create the ServiceAccount and SCC

The anyuid SCC allows a container to run as a specific user ID outside the default range. It does not grant host access or elevated kernel capabilities, but it does allow the root UID that Open WebUI requires.

The scc.yaml file creates both the ServiceAccount and the ClusterRoleBinding in one apply:

apiVersion: v1
kind: ServiceAccount
metadata:
 name: open-webui
 namespace: open-webui
 labels:
 app: open-webui
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: open-webui-anyuid
subjects:
 - kind: ServiceAccount
 name: open-webui
 namespace: open-webui
roleRef:
 kind: ClusterRole
 name: system:openshift:scc:anyuid
 apiGroup: rbac.authorization.k8s.io

Apply the file to create the scc:

oc apply -f scc.yaml

Create the Secrets

Two secrets are needed. The first holds the API keys for the two backend connections. The keys must be ordered to match the backend URLs that will be set in the deployment.

Retrieve the RHAIIS and Hermes API keys:

export RHAIIS_API_KEY=$(oc get secret vllm-api-key-secret -n rhaiis \
 -o jsonpath='{.data.VLLM_API_KEY}' | base64 -d)
export HERMES_API_KEY=$(oc get secret hermes-api-secret -n hermes \
 -o jsonpath='{.data.API_SERVER_KEY}' | base64 -d)

Create the API keys secret with both values, semicolon-separated:

oc create secret generic open-webui-api-keys \
 --from-literal=API_KEYS="${RHAIIS_API_KEY};${HERMES_API_KEY}" \
 -n open-webui

The second secret holds the session key Open WebUI uses to sign cookies and session tokens:

oc create secret generic open-webui-secret \
 --from-literal=WEBUI_SECRET_KEY=$(openssl rand -hex 32) \
 -n open-webui

Create a PersistentVolumeClaim

Open WebUI stores its database, uploaded files, and user configuration on a persistent volume:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: open-webui-pvc
 namespace: open-webui
 labels:
 app: open-webui
spec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 5Gi

Apply the file to create the PVC:

oc apply -f pvc.yaml

Deploy Open WebUI

The Deployment sets OPENAI_API_BASE_URLS to the two internal service addresses, reads the API keys and session secret from the secrets created above, and mounts the PVC at /app/backend/data:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: open-webui
 namespace: open-webui
 labels:
 app: open-webui
spec:
 replicas: 1
 selector:
 matchLabels:
 app: open-webui
 template:
 metadata:
 labels:
 app: open-webui
 spec:
 serviceAccountName: open-webui
 securityContext:
 runAsUser: 0
 volumes:
 - name: open-webui-data
 persistentVolumeClaim:
 claimName: open-webui-pvc
 containers:
 - name: open-webui
 image: ghcr.io/open-webui/open-webui:main
 imagePullPolicy: Always
 ports:
 - name: http
 containerPort: 8080
 protocol: TCP
 env:
 - name: OPENAI_API_BASE_URLS
 value: "http://rhaiis-vllm.rhaiis.svc.cluster.local:8000/v1;http://hermes.hermes.svc.cluster.local:8642/v1"
 - name: OPENAI_API_KEYS
 valueFrom:
 secretKeyRef:
 name: open-webui-api-keys
 key: API_KEYS
 - name: WEBUI_SECRET_KEY
 valueFrom:
 secretKeyRef:
 name: open-webui-secret
 key: WEBUI_SECRET_KEY
 - name: ENABLE_OLLAMA_API
 value: "false"
 - name: ENABLE_OPENAI_API
 value: "true"
 volumeMounts:
 - name: open-webui-data
 mountPath: /app/backend/data
 resources:
 requests:
 cpu: "500m"
 memory: "500Mi"
 limits:
 cpu: "1000m"
 memory: "2Gi"
 restartPolicy: Always

Apply the file to create the deployment:

oc apply -f deployment.yaml

Create a Service and Route

Create a Service that maps port 8080 to port 8080 on the pod:

apiVersion: v1
kind: Service
metadata:
 name: open-webui
 namespace: open-webui
 labels:
 app: open-webui
spec:
 selector:
 app: open-webui
 ports:
 - name: http
 protocol: TCP
 port: 8080
 targetPort: 8080

Create a TLS-terminated Route to expose the Open WebUI interface outside the cluster:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
 name: open-webui
 namespace: open-webui
 labels:
 app: open-webui
spec:
 to:
 kind: Service
 name: open-webui
 port:
 targetPort: http
 tls:
 termination: edge
 insecureEdgeTerminationPolicy: Redirect

Apply both and retrieve the assigned hostname:

oc apply -f service.yaml
oc apply -f route.yaml
oc get route open-webui -n open-webui -o jsonpath='{.spec.host}'

Open the hostname from the previous step in a browser. On first launch, Open WebUI prompts for an admin account. The first account created becomes the administrator.

To verify that both backends are reachable, go to Settings > Connections in the admin panel. Each entry should show a green status indicator. If a connection shows an error, check that the corresponding deployment is running and that the API key in open-webui-api-keys matches the key currently in use on that backend.

Open WebUI Settings > Connections panel showing both the RHAIIS and Hermes Agent connections active

Once both connections are active, models from both backends appear in the model selector on the main chat screen. The list is fetched from each backend’s /v1/models endpoint.

Model selector in Open WebUI showing models from RHAIIS and Hermes Agent

Conclusion

This setup runs Open WebUI on the same OpenShift cluster as the model and agent backends. Traffic between Open WebUI and RHAIIS, and between Open WebUI and Hermes, stays on the cluster-internal service network. The public OpenShift Route is used only for the browser-facing Open WebUI interface. Neither RHAIIS nor Hermes requires a public Route for this to work, which avoids exposing inference endpoints outside the cluster.

The three deployments form a self-contained AI stack on OpenShift: a model server, an agent layer, and a browser interface, each in its own namespace and each reachable from the others through standard Kubernetes service discovery.

References

GitHub repository with deployment files - link
Running the Red Hat AI Inference Server on OpenShift - link
Deploying Hermes Agent on OpenShift - link
My Local AI Stack: Open WebUI, LiteLLM, SearXNG, and Docling - link
Open WebUI - project site - link
Open WebUI - GitHub repository - link

Deploying Hermes Agent on OpenShift

Sun, 24 May 2026 00:00:00 +0000

Hermes Agent on OpenShift: Private by default, cloud access when needed - AI generated

Introduction

In this post, I want to describe how to deploy the Hermes Agent on OpenShift and wire it to a self-hosted model endpoint running on the same cluster. This is a direct continuation of two earlier posts: Deploying OpenShift on AWS, which covers getting a cluster into place, and the post on running the Red Hat AI Inference Server on OpenShift, which covers the model serving layer that Hermes will talk to.

If you want background on what Hermes Agent is and why it is worth running, the companion post Hermes Agent: A Personal AI That Gets More Useful Over Time covers that in more detail. This post focuses on the mechanics of getting it running on OpenShift.

Architecture

The setup connects two namespaces on the same cluster. The Red Hat AI Inference Server (RHAIIS) runs in the rhaiis namespace and serves a model on port 8000. Hermes Agent runs in a separate hermes namespace and talks to the vLLM server over the internal cluster service network, using the DNS name rhaiis-vllm.rhaiis.svc.cluster.local:8000. No public route is involved in that hop.

OpenRouter is wired as an automatic fallback. If the vLLM server is unavailable or returns an error, Hermes falls back to a remote model through OpenRouter without requiring any manual intervention.

Externally, Hermes exposes an OpenAI-compatible API on port 8642, secured with a bearer token. An OpenShift Route with TLS termination handles the public endpoint.

Prerequisites

The RHAIIS deployment from the previous post must be running in the rhaiis namespace with a deployment named rhaiis-vllm.
An OpenRouter API key for the fallback model.

Deploying Hermes Agent

All deployment files are available in the smichard/agent_on_ocp GitHub repository. The steps below apply them in sequence.

Clone the repository:

git clone https://github.com/smichard/agent_on_ocp.git
cd hermes_on_ocp

Create the Namespace and ServiceAccount

oc new-project hermes

Hermes Agent runs as UID 10000. The default restricted SCC in OpenShift does not allow this, so the deployment needs a dedicated ServiceAccount with the anyuid SCC:

oc create serviceaccount hermes -n hermes

oc adm policy add-scc-to-user anyuid \
 -z hermes \
 -n hermes

Create Secrets

Three secrets are needed: one for the vLLM bearer token, one for the OpenRouter fallback key, and one for the Hermes API server key that clients must present.

vLLM bearer token:

Hermes reads the OPENAI_API_KEY environment variable for custom OpenAI-compatible endpoints. The vLLM API key from the rhaiis namespace is passed in under that name:

export RHAIIS_API_KEY=$(oc get secret vllm-api-key-secret -n rhaiis \
 -o jsonpath='{.data.VLLM_API_KEY}' | base64 -d)

oc create secret generic hermes-vllm-secret \
 --from-literal=OPENAI_API_KEY="${RHAIIS_API_KEY}" \
 -n hermes

OpenRouter fallback key:

oc create secret generic hermes-openrouter-secret \
 --from-literal=OPENROUTER_API_KEY=<your_openrouter_key> \
 -n hermes

Hermes API server key:

Clients calling the Hermes API must include this key as a bearer token. Generate a random value at creation time:

oc create secret generic hermes-api-secret \
 --from-literal=API_SERVER_KEY=$(openssl rand -hex 32) \
 -n hermes

Retrieve it later with:

oc get secret hermes-api-secret -n hermes \
 -o jsonpath='{.data.API_SERVER_KEY}' | base64 -d

Create the ConfigMap

The ConfigMap holds the Hermes Agent configuration file. It sets the primary model provider to the internal vLLM service and configures OpenRouter as the fallback:

apiVersion: v1
kind: ConfigMap
metadata:
 name: hermes-config
 namespace: hermes
 labels:
 app: hermes
data:
 config.yaml: |
 model:
 provider: "custom"
 base_url: "http://rhaiis-vllm.rhaiis.svc.cluster.local:8000/v1"
 default: "Qwen/Qwen3-Coder-30B-A3B-Instruct"

 fallback_model:
 provider: "openrouter"
 model: "anthropic/claude-sonnet-4-6"

 terminal:
 backend: "local"
 cwd: "/opt/data/workspace"
 timeout: 180
 lifetime_seconds: 300

 compression:
 enabled: true
 threshold: 0.50
 target_ratio: 0.20
 protect_last_n: 20

Adjust model.default to match the --served-model-name value used in the RHAIIS deployment. Adjust fallback_model.model to the OpenRouter model you want to use as a fallback.

oc apply -f configmap.yaml

Create a PersistentVolumeClaim

Hermes stores sessions, memories, and workspace data on a persistent volume:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: hermes-data
 namespace: hermes
 labels:
 app: hermes
spec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 5Gi

Apply the file to create the PVC:

oc apply -f pvc.yaml

Deploy Hermes Agent

The Deployment mounts the ConfigMap and the PVC, injects the three secrets as environment variables, and runs the container as the hermes ServiceAccount. Check the Hermes Agent repository for the current container image reference before applying:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: hermes
 namespace: hermes
 labels:
 app: hermes
spec:
 replicas: 1
 selector:
 matchLabels:
 app: hermes
 template:
 metadata:
 labels:
 app: hermes
 spec:
 tolerations:
 - key: nvidia.com/gpu
 effect: NoSchedule
 operator: Exists
 serviceAccountName: hermes
 securityContext:
 runAsUser: 10000
 runAsNonRoot: true
 fsGroup: 10000
 volumes:
 - name: hermes-data
 persistentVolumeClaim:
 claimName: hermes-data
 - name: hermes-config
 configMap:
 name: hermes-config
 containers:
 - name: hermes
 image: nousresearch/hermes-agent:latest
 imagePullPolicy: Always
 args: ["gateway"]
 env:
 - name: OPENAI_API_KEY
 valueFrom:
 secretKeyRef:
 name: hermes-vllm-secret
 key: OPENAI_API_KEY
 - name: OPENROUTER_API_KEY
 valueFrom:
 secretKeyRef:
 name: hermes-openrouter-secret
 key: OPENROUTER_API_KEY
 - name: API_SERVER_KEY
 valueFrom:
 secretKeyRef:
 name: hermes-api-secret
 key: API_SERVER_KEY
 - name: API_SERVER_ENABLED
 value: "true"
 - name: API_SERVER_HOST
 value: "0.0.0.0"
 - name: API_SERVER_PORT
 value: "8642"
 ports:
 - name: api
 containerPort: 8642
 protocol: TCP
 volumeMounts:
 - name: hermes-data
 mountPath: /opt/data
 - name: hermes-config
 mountPath: /opt/data/config.yaml
 subPath: config.yaml
 resources:
 requests:
 cpu: "500m"
 memory: "512Mi"
 limits:
 cpu: "2"
 memory: "2Gi"
 livenessProbe:
 tcpSocket:
 port: 8642
 initialDelaySeconds: 30
 periodSeconds: 30
 readinessProbe:
 tcpSocket:
 port: 8642
 initialDelaySeconds: 15
 periodSeconds: 10
 restartPolicy: Always

Apply the file to create the deployment:

oc apply -f deployment.yaml

The API server is ready when the logs show:

[api_server] Listening on 0.0.0.0:8642

Hermes Gateway starting up, with 83 skills bundled and the messaging platform scheduler ready to accept requests

Create a Service and Route

Create a Service that maps port 8642 to port 8642 on the pod:

apiVersion: v1
kind: Service
metadata:
 name: hermes
 namespace: hermes
 labels:
 app: hermes
spec:
 selector:
 app: hermes
 ports:
 - name: api
 protocol: TCP
 port: 8642
 targetPort: 8642

Create a TLS-terminated Route to expose the endpoint outside the cluster (optional):

apiVersion: route.openshift.io/v1
kind: Route
metadata:
 name: hermes
 namespace: hermes
 labels:
 app: hermes
spec:
 to:
 kind: Service
 name: hermes
 port:
 targetPort: api
 tls:
 termination: edge
 insecureEdgeTerminationPolicy: Redirect

Apply both and retrieve the assigned hostname:

oc apply -f service.yaml
oc apply -f route.yaml
oc get route hermes -n hermes -o jsonpath='{.spec.host}'

Testing the Endpoint

Store the hostname and API key in shell variables to keep the commands readable:

export HERMES_HOST=$(oc get route hermes -n hermes \
 -o jsonpath='{.spec.host}')
export HERMES_KEY=$(oc get secret hermes-api-secret -n hermes \
 -o jsonpath='{.data.API_SERVER_KEY}' | base64 -d)

Verify that the variables are populated before proceeding:

echo "HERMES_HOST : ${HERMES_HOST}"
echo "HERMES_API_KEY : ${HERMES_KEY}"

List available models:

curl -sS \
 "https://${HERMES_HOST}/v1/models" \
 -H "Authorization: Bearer ${HERMES_KEY}" | jq -r '.data[].id'

Send a chat completion request:

curl -sS \
 "https://${HERMES_HOST}/v1/chat/completions" \
 -H "Authorization: Bearer ${HERMES_KEY}" \
 -H "Content-Type: application/json" \
 -d '{
 "model": "Qwen2.5-1.5B-Instruct",
 "messages": [{"role": "user", "content": "What is OpenShift?"}]
 }' | jq -r '.choices[0].message.content'

A successful response confirms that Hermes is running, the API key is working, and the request reached the vLLM server over the internal cluster network.

To verify the fallback path, scale down the RHAIIS deployment temporarily and send the same request. Hermes should return a response via OpenRouter instead:

oc scale deployment rhaiis-vllm -n rhaiis --replicas=0
# send a request, observe fallback in hermes logs
oc scale deployment rhaiis-vllm -n rhaiis --replicas=1

Changing the Model

Update configmap.yaml and set model.default to any model name served by the vLLM instance. The value must match the --served-model-name argument used in the RHAIIS deployment. Apply the updated ConfigMap and restart the Hermes deployment to pick up the change:

oc apply -f configmap.yaml
oc rollout restart deployment/hermes -n hermes

Connecting to Open WebUI

Hermes Agent exposes a standard OpenAI-compatible API, which means Open WebUI can connect to it directly as an external provider. As described in the prvious cases it is very easy to add the Hermes endpoint to the existing stack.

In Open WebUI, go to Settings > Connections and add a new external connection. Set the URL to the Hermes route hostname with the /v1 suffix, add the Hermes API server key created in step 3 as a bearer token, set the provider type to OpenAI, and the API type to Chat Completions. Leave the model ID field empty so Open WebUI queries the /v1/models endpoint and discovers available models automatically.

Open WebUI external connection configured against the Hermes Agent endpoint

Once saved, the model appears in the model selector alongside any other configured providers. Requests go from Open WebUI through Hermes to the vLLM server running on the same cluster.

Hermes agent is available in the Open WebUI interface alongside the model served by RHAIIS

Conclusion

This setup places Hermes Agent inside the same OpenShift cluster as the inference server and routes all model traffic over the internal service network. The public Hermes API endpoint is secured with a separate bearer token, so the vLLM key never leaves the cluster. OpenRouter handles the fallback case without any changes to the application code. The result is a self-hosted agent that uses a self-hosted model for most requests and degrades gracefully when the local server is unavailable.

References

GitHub repository with eployment files - link
Deploying OpenShift on AWS with Automated Cluster Provisioning - link
Running the Red Hat AI Inference Server on OpenShift - link
Hermes Agent: A Personal AI That Gets More Useful Over Time - link
OpenRouter - link
OpenShift CLI (oc) - link
smichard/agent_on_ocp - GitHub repository - link
Hermes Agent - GitHub repository - link
Hermes Agent - Documentation - link
Nous Research - link

Agent on Home

Bringing Open WebUI to OpenShift

Open WebUI on OpenShift, connected to RHAIIS and Hermes Agent over the cluster-internal service network — AI generated

Introduction

Architecture

Prerequisites

Deploying Open WebUI

First Login and Verifying Connections

Open WebUI Settings > Connections panel showing both the RHAIIS and Hermes Agent connections active

Model selector in Open WebUI showing models from RHAIIS and Hermes Agent

Conclusion

References

Deploying Hermes Agent on OpenShift

Hermes Agent on OpenShift: Private by default, cloud access when needed - AI generated

Introduction

Architecture

Prerequisites

Deploying Hermes Agent

Hermes Gateway starting up, with 83 skills bundled and the messaging platform scheduler ready to accept requests

Testing the Endpoint

Changing the Model

Connecting to Open WebUI

Open WebUI external connection configured against the Hermes Agent endpoint

Hermes agent is available in the Open WebUI interface alongside the model served by RHAIIS

Conclusion

References