Kubernetes Deployment on Azure (AKS)

Run Studio on your own Azure Kubernetes Service cluster using vibedata. You build the Azure side (cluster, storage, secrets, HTTPS edge); the CLI installs Studio into the cluster over your kubeconfig and creates nothing in Azure.

The guide has three phases:

1. Prerequisites — the Azure resources you create before installing.
2. Installation — the single vibedata install kubernetes command.
3. After installation — wiring the HTTPS edge, first login, and verification.

Each step below is one command with a short explanation of what it does and why. Keep the same terminal open for the whole guide — the variables you set live for that session.

Choose your HTTPS edge tier first

Studio must be reached over HTTPS (single sign-on refuses plain HTTP), so an Azure Front Door edge is required — it is not optional. What you choose is the tier, which decides whether the cluster's single ingress load balancer is public or private:

Edge tier	Ingress origin	Cost	When to choose
Front Door Standard	Load balancer gets a public IP, fenced to Front Door's IP range	~$35/mo	Default; most installs
Front Door Premium + Private Link	Load balancer stays private, reached over Azure's internal network	~$330/mo	Regulated / no-public-IP requirement

Everything else — nodes, database, agent pods — is private in both tiers; only the one ingress load balancer differs. This guide gives the Standard steps in full and the Premium path in After installation. Pick one before you start.

          YOU (az CLI)                          vibedata CLI
  ┌──────────────────────────┐               ┌────────────────────────────┐
  │ Resource group           │               │ vibedata install kubernetes │
  │  ├─ AKS cluster ──────────┼── kubeconfig ─┤   → External Secrets,       │
  │  ├─ Azure Files NFS share │               │     ingress, Argo CD,       │
  │  ├─ Key Vault (secrets)   │               │     Studio + companions     │
  │  └─ Front Door edge       │               └────────────┬───────────────┘
  └──────────────────────────┘                            │ installs into
                                                           ▼
   browser → https://studio.you.com → Front Door (HTTPS) → ingress → Studio
                          (edge holds the TLS cert)

Install the vibedata CLI

Download the CLI binary for your machine:

curl -fsSL https://github.com/accelerate-data/vibedata-official/releases/latest/download/install.sh | sh

Confirm it's on your PATH:

vibedata version

Set your values

These are the only things you choose. Edit this one block, then paste every later command as-is — they all read these variables. The example values work unchanged, except the two marked globally unique (storage account and Key Vault names are shared across all of Azure — change the suffix if the name is taken).

export RG=studio-rg                # resource group
export LOCATION=eastus             # Azure region
export AKS=studio-aks              # AKS cluster name
export STORAGE=studiofiles01       # storage account — 3-24 chars, lowercase letters/numbers only, GLOBALLY UNIQUE
export SHARE=studio-data           # NFS share name
export BACKUP_SHARE=studio-backups # NFS share name for backup packages
export VAULT=studio-kv-01          # Key Vault name — 3-24 chars, GLOBALLY UNIQUE
export FD_PROFILE=studio-fd        # Front Door profile name
export FD_ENDPOINT=studio          # Front Door endpoint name (becomes the prefix of your free hostname)

---

Phase 1 — Prerequisites

You create these Azure resources yourself. The installer only reads their coordinates (URLs, identity id); it never sees a secret value and creates nothing in Azure.

1. Register the Azure services (first run only)

Sign in to Azure in your browser:

az login

Choose which subscription to use:

az account set --subscription "<your-subscription>"

Turn on the resource types AKS needs. This is free and creates nothing — a brand-new subscription has them off, and skipping this is the #1 first-time error.

az provider register --namespace Microsoft.ContainerService
az provider register --namespace Microsoft.Compute
az provider register --namespace Microsoft.Network
az provider register --namespace Microsoft.Storage
az provider register --namespace Microsoft.KeyVault

Wait until this prints Registered before moving on (repeat for the others if unsure):

az provider show --namespace Microsoft.ContainerService --query registrationState -o tsv

2. Create the AKS cluster

Create a resource group — the "folder" that holds every resource in this guide:

az group create --name $RG --location $LOCATION

Create the cluster itself. This is the slow one (a few minutes) and starts the node-VM billing. Use three nodes, not two. Studio runs its backend and its frontend as two replicas each, and each pair must sit on separate hosts, so a two-node cluster leaves one replica of each unschedulable. Three Standard_D2s_v3 nodes suit Studio plus a light agent load, and also fit --with-observability (the Grafana/LGTM bundle); measured on a fresh install, Studio and the platform ask for about 4.1 vCPU of reservations, which two such nodes cannot allocate at all.

Quota headroom for a second cluster is only needed for the break-glass path. A failed upgrade is normally rolled back on the same cluster (see "Backup and recovery"), which needs no extra capacity. Only a cluster that dies outright — nodes gone, disks unattachable — is recovered onto a fresh cluster, and that does need a second cluster's worth of vCPU free at that moment, in the VM size's own quota family, not just the regional total (a family at its cap refuses the create even when the region has room). Check with az vm list-usage --location $LOCATION -o table and keep an eye on both the cores row and your size's family row; the automation script's recover-cluster mode checks both before building anything, and RECOVERY_NODE_VM_SIZE lets it build the recovery cluster in a different family when yours is full.

Every container ships with an explicit CPU and memory reservation, so an undersized cluster fails loudly at scheduling time instead of degrading.

az aks create --resource-group $RG --name $AKS --node-count 3 --node-vm-size Standard_D2s_v3 --enable-managed-identity --generate-ssh-keys

Point kubectl at the new cluster (writes the cluster into your kubeconfig and switches to it):

az aks get-credentials --resource-group $RG --name $AKS

Confirm you can reach it — the nodes should show Ready:

kubectl get nodes

3. Create the storage share (Azure Files NFS)

Studio's backend and every agent pod read and write one shared folder (DATA_DIR), which needs an Azure Files NFS v4.1 share — not SMB, because agent pods run git and SMB's file locking can't carry concurrent git.

Create the storage account (Premium + FileStorage kind is required for NFS; secure-transfer is off because NFS 4.1 has no HTTPS):

az storage account create --name $STORAGE --resource-group $RG --location $LOCATION --sku Premium_LRS --kind FileStorage --https-only false

Create the share (the NFS protocol is fixed at creation and can't be changed later):

az storage share-rm create --storage-account $STORAGE --resource-group $RG --name $SHARE --enabled-protocols NFS --quota 100

Create a second share in the same storage account for backup packages. It has to live outside the cluster so it survives one, which is why it is a share and not a disk. Size it for two full copies of your data — 100 GiB is both the example above and the smallest a Premium share can be, so it is the floor either way; raise it with --quota if your data grows past half of it:

az storage share-rm create --storage-account $STORAGE --resource-group $RG --name $BACKUP_SHARE --enabled-protocols NFS --quota 100

An NFS share has no password — access is controlled by the network, so the next commands open it to your cluster's subnet and nothing else. Both shares are covered by those rules, and neither needs a key in the vault.

Look up your cluster's virtual network and subnet (AKS auto-created them in a separate resource group; these three lines find them so you don't have to):

NODE_RG=$(az aks show -g $RG -n $AKS --query nodeResourceGroup -o tsv)
VNET=$(az network vnet list -g "$NODE_RG" --query "[0].name" -o tsv)
SUBNET_ID=$(az network vnet subnet list -g "$NODE_RG" --vnet-name "$VNET" --query "[0].id" -o tsv)

Enable a storage "service endpoint" on that subnet — Azure requires it before the share will accept the subnet:

az network vnet subnet update --ids "$SUBNET_ID" --service-endpoints Microsoft.Storage

Allow that subnet on the storage account, then set the default to deny so only your cluster can reach the share:

az storage account network-rule add -g $RG -n $STORAGE --subnet "$SUBNET_ID"
az storage account update -g $RG -n $STORAGE --default-action Deny

4. Create the Key Vault and secrets

Studio reads its secrets in-cluster (via External Secrets Operator, which the installer sets up). The installer never sees a value — only the vault's URL.

Ask the CLI for the exact set your profile needs before you create anything. It needs no cluster, share or vault — only internet access, because it reads the set from the release's own chart, which is also what the install renders from. So this list can never be out of date:

vibedata install kubernetes --list-secrets                       # core: 9
vibedata install kubernetes --with-observability --list-secrets   # + Grafana: 11

Add --version <release> to ask about a release other than the one this binary installs by default.

Create the vault:

az keyvault create --name $VAULT --resource-group $RG

New vaults are RBAC-mode, so creating one doesn't let you write secrets yet — grant yourself the writer role first. Wait ~1–2 minutes after this for the role to propagate, or the next commands fail with ForbiddenByRbac:

az role assignment create --role "Key Vault Secrets Officer" --assignee "$(az ad signed-in-user show --query id -o tsv)" --scope "$(az keyvault show --name $VAULT --query id -o tsv)"

Now store the 9 core values (always required). Each line saves one under a fixed name:

az keyvault secret set --vault-name $VAULT --name pg-password         --value "$(openssl rand -hex 24)"
az keyvault secret set --vault-name $VAULT --name auth-secret         --value "$(openssl rand -base64 32)"
az keyvault secret set --vault-name $VAULT --name data-encryption-key --value "$(openssl rand -base64 32)"
az keyvault secret set --vault-name $VAULT --name data-encryption-key-id --value "k1"
az keyvault secret set --vault-name $VAULT --name data-encryption-retired-keys --value "{}"
az keyvault secret set --vault-name $VAULT --name obot-client-secret  --value "$(openssl rand -base64 32)"
az keyvault secret set --vault-name $VAULT --name obot-db-password    --value "$(openssl rand -hex 24)"
az keyvault secret set --vault-name $VAULT --name obot-tunnel-peer-token --value "$(openssl rand -base64 32)"
az keyvault secret set --vault-name $VAULT --name bootstrap-key       --value "$(openssl rand -base64 16)"

What each one is: pg-password = Studio's database password · auth-secret = signs login sessions · data-encryption-key = master key Studio encrypts data with · obot-client-secret = lets the agent service trust Studio login · obot-db-password = the agent service's own DB password · obot-tunnel-peer-token = lets the agent service's replicas trust each other · bootstrap-key = one-time key to create the first admin. (pg-password/obot-db-password use -hex because they go into a Postgres connection string and must avoid @ : / # ? &.)

The two immutable-keyring metadata values accompany the master key: data-encryption-key-id names the active key (k1 for a new installation), while data-encryption-retired-keys is a JSON map and begins as {}. Keep all three values unchanged after Studio has started. Studio stores only a one-way fingerprint in PostgreSQL and checks it before startup and upgrade mutation; if a vault value is replaced, restore the original vault value rather than choosing a new one.

Finally, let the cluster read the vault. Grab the cluster's identity:

CLUSTER_ID=$(az aks show -g $RG -n $AKS --query "identityProfile.kubeletidentity.clientId" -o tsv)

Grant that identity the read role (this same $CLUSTER_ID is also passed to the installer in Phase 2):

az role assignment create --role "Key Vault Secrets User" --assignee "$CLUSTER_ID" --scope "$(az keyvault show --name $VAULT --query id -o tsv)"

Only enabling observability? Also add the extra secrets your chosen flag needs — see Observability secrets at the end. Skip this if you're not using --with-observability.

5. Create the Front Door edge

Front Door gives every endpoint a free *.azurefd.net hostname with a Microsoft-managed, auto-renewed TLS certificate. Create it now — the hostname becomes --domain at install.

Create the Front Door profile (use Standard_AzureFrontDoor for this guide; for the private-origin path use Premium_AzureFrontDoor instead):

az afd profile create --profile-name $FD_PROFILE --resource-group $RG --sku Standard_AzureFrontDoor --origin-response-timeout-seconds 240

Don't drop --origin-response-timeout-seconds. It is how long Front Door waits for Studio to answer before returning 504. Azure defaults it to 30 seconds, which is shorter than opening an Intent: Studio clones the repository and starts an agent for you, which takes ~40 seconds warm and longer the first time on a node, while it downloads the agent image. At the default, Front Door gives up mid-way, and because the caller disconnected Studio discards the half-built agent and starts over on the retry. 240 is the Azure maximum and leaves room for the first, slowest open.

Already created the profile without it? Set it on the existing profile — no reinstall needed:

az afd profile update --profile-name $FD_PROFILE --resource-group $RG --origin-response-timeout-seconds 240

The change takes a few minutes to reach every Front Door edge location.

Create the endpoint — this prints your free HTTPS hostname (yours will differ, e.g. studio-ab12cd.z01.azurefd.net):

az afd endpoint create --endpoint-name $FD_ENDPOINT --profile-name $FD_PROFILE --resource-group $RG --query hostName -o tsv

Default domain vs custom domain. You get working HTTPS either way — the only difference is the address users type:

• Default (*.azurefd.net, recommended to start) — the hostname you just got already serves HTTPS. No purchase, no DNS. Phase 2 uses it as --domain automatically.
• Custom (studio.example.com, optional) — only if Studio must live at your own branded address. You own the domain, add it to Front Door, prove ownership, let Front Door issue a certificate, and add a CNAME record pointing your domain at Front Door. Functionally identical.

You wire Front Door's origin to the cluster after install (Phase 3).

---

Phase 2 — Installation

First, gather the four values the installer needs. These are all derived from your variables, so just run the block — the echo at the end lets you eyeball them:

VAULT_URL=$(az keyvault show --name $VAULT --query properties.vaultUri -o tsv)
STORAGE_URL="$(az storage account show -g $RG -n $STORAGE --query primaryEndpoints.file -o tsv)$SHARE"
FD_HOST=$(az afd endpoint show --endpoint-name $FD_ENDPOINT --profile-name $FD_PROFILE --resource-group $RG --query hostName -o tsv)
CLUSTER_ID=$(az aks show -g $RG -n $AKS --query identityProfile.kubeletidentity.clientId -o tsv)
echo "domain=$FD_HOST  storage=$STORAGE_URL  vault=$VAULT_URL  identity=$CLUSTER_ID"

Now run the install. Studio pulls public images and chart, so there's no registry login or image-pull Secret. Private or mirrored registries are not supported by this installer yet; do not add registry credentials or imagePullSecrets to a standard install. This takes several minutes (it installs the ingress, Argo CD, and waits for Studio to be healthy):

vibedata install kubernetes \
  --cloud azure \
  --domain "$FD_HOST" \
  --storage-url "$STORAGE_URL" \
  --vault-url "$VAULT_URL" \
  --vault-identity-client-id "$CLUSTER_ID"

Run with no flags for an interactive form that asks four things: kubeconfig, Studio domain, storage share URL, and vault URL.

Unattended runs (CI, scripts, agents) never see that form: with no terminal to answer on, the CLI uses your normal kubeconfig — whatever $KUBECONFIG points at, or ~/.kube/config if it is unset — and for anything else it still needs, stops and names the flag that supplies it. Passing --kube-context also skips the kubeconfig question at a terminal.

Optional flags:

Flag	Purpose
--with-observability	Add the LGTM/Grafana observability stack. Needs the grafana-* vault secrets.
--no-observability	Remove the observability stack. Deletes its stored trace history.
--kube-context <name>	Which context to install into (multi-cluster kubeconfig).
--storage-resource-group <rg>	Set if the storage account is in a different resource group than the nodes.
--timeout <seconds>	How long to wait for Studio to be healthy (default 600).
--kubeconfig <path>	Path to the kubeconfig. Defaults to $KUBECONFIG, or ~/.kube/config if unset.

Observability is off on a first install. The whole command is idempotent — safe to re-run to change your domain, storage, or anything else.

Re-running never removes observability by accident. If it's already installed and you re-run without a profile flag, the install stops before changing anything and prints the two commands to choose from: the one that keeps what you have, and the one that removes it. That guard exists because removal is destructive here — Argo prunes the monitoring apps and their stored traces go with them. To actually remove it, say so with --no-observability.

When it finishes it prints the Studio URL, the private ingress address, and where to find the bootstrap key. The ingress is private at this point — Phase 3 connects it to your edge.

---

Phase 3 — After installation (go live)

Choose the section matching your edge tier.

Option A — Front Door Standard (public origin)

The installer left the cluster's entry point (the "ingress load balancer") private, but Standard Front Door can only reach a public address. So you: make it public → lock it to Front Door → point Front Door at it.

A1 · Make the ingress public. This removes the "keep private" setting, so Azure gives the load balancer a public IP:

kubectl -n ingress-nginx annotate svc ingress-nginx-controller service.beta.kubernetes.io/azure-load-balancer-internal- --overwrite

The health check needs nothing from you. The installer already points Azure's probe at /healthz; probing the default / returns 404, which Azure reads as "dead."

A1 · Get the public IP. Wait ~1–2 min after the first command, then read the assigned IP into INGRESS_IP. If it prints a 10.x address it's still private — wait and re-run:

INGRESS_IP=$(kubectl -n ingress-nginx get svc ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}'); echo "$INGRESS_IP"

A2 · Find your cluster's firewall. AKS keeps it (an "NSG") in an auto-generated resource group; these two lines look up its name:

NODE_RG=$(az aks show -g $RG -n $AKS --query nodeResourceGroup -o tsv)
NSG=$(az network nsg list -g "$NODE_RG" --query "[0].name" -o tsv)

A2 · Allow Front Door in. Let traffic reach your ingress IP only from Azure Front Door's address range:

az network nsg rule create -g "$NODE_RG" --nsg-name "$NSG" -n allow-frontdoor-origin --priority 200 --direction Inbound --access Allow --protocol Tcp --source-address-prefixes AzureFrontDoor.Backend --destination-address-prefixes "$INGRESS_IP" --destination-port-ranges 80 443

A2 · Block everyone else. Deny the rest of the internet to that IP. Lower priority number wins, so Front Door (200) is allowed before this deny (300) applies — net result: only Front Door can use the public IP:

az network nsg rule create -g "$NODE_RG" --nsg-name "$NSG" -n deny-internet-origin --priority 300 --direction Inbound --access Deny --protocol Tcp --source-address-prefixes Internet --destination-address-prefixes "$INGRESS_IP" --destination-port-ranges 80 443

If you ever reinstall Studio (or anything else recreates the ingress load balancer), the ingress IP changes. Re-run A1 to get the new IP, update both rules above with it (az network nsg rule update … --destination-address-prefixes "$INGRESS_IP"), and re-point the Front Door origin. Rules left pinned to the old IP silently block all traffic — the edge returns 504.

A3 · Define the health check Front Door will run against your cluster — call GET /healthz every 30s, healthy after 3 good replies.

az afd origin-group create --origin-group-name studio-og --profile-name $FD_PROFILE --resource-group $RG --probe-request-type GET --probe-protocol Http --probe-path /healthz --probe-interval-in-seconds 30 --sample-size 4 --successful-samples-required 3 --additional-latency-in-milliseconds 50

Warm-up flap is normal. Each Front Door edge location converges on its own probe schedule, so right after go-live (or any origin/IP change) the site can alternate between working and 504 for a few minutes — one edge location is converged while another isn't. Don't chase it as an install bug: wait until Origin health (Front Door profile → Reports, OriginHealthPercentage) reads 100%, then verify. A shorter probe interval shrinks this window — at 30s it's usually under two minutes; at 100s it can stretch past ten.

A3 · Set the backend address — your public ingress IP. --origin-host-header "$FD_HOST" makes Front Door send your Studio hostname in the Host header, which is how the ingress knows to route to Studio (get this wrong → 404):

az afd origin create --origin-name studio-ingress --origin-group-name studio-og --profile-name $FD_PROFILE --resource-group $RG --host-name "$INGRESS_IP" --origin-host-header "$FD_HOST" --http-port 80 --https-port 443 --priority 1 --weight 1000 --enabled-state Enabled

A3 · Connect your hostname to that backend. This route forwards all paths, redirects HTTP→HTTPS, and attaches to your free azurefd.net hostname. Without it, Front Door has nowhere to send requests and returns 404:

az afd route create --route-name studio-route --profile-name $FD_PROFILE --resource-group $RG --endpoint-name $FD_ENDPOINT --origin-group studio-og --supported-protocols Http Https --forwarding-protocol HttpOnly --https-redirect Enabled --link-to-default-domain Enabled

Verify. Give it a few minutes, then check the app opens through Front Door and the raw IP is blocked:

curl -sS -o /dev/null -w "FD: %{http_code}\n" https://$FD_HOST/
curl -sS -o /dev/null -w "direct: %{http_code}\n" --max-time 10 http://$INGRESS_IP/ || echo "direct: blocked (good)"

FD: 200/302 with the direct hit blocked = live and fenced. Traffic path: browser → HTTPS to your Front Door hostname → Front Door (holds the cert) → public ingress IP (HTTP, fenced) → Studio. The certificate never enters the cluster.

A4 · Harden (recommended). The AzureFrontDoor.Backend range covers every Azure customer's Front Door, so the network fence alone still lets other tenants' Front Doors reach you. Pin it to your profile by rejecting requests whose X-Azure-FDID header doesn't match your Front Door ID. Get the ID:

az afd profile show -g $RG --profile-name $FD_PROFILE --query frontDoorId -o tsv

Then configure the ingress to reject requests without that value (a request-header match on the Studio ingress; depends on your ingress-nginx allowing it). Skip only for throwaway test installs; apply before real traffic.

Re-running the installer reverts A1. It re-adds the private annotation and releases the public IP. If you re-run install, redo A1 and re-point the Front Door origin at the new IP.

Option B — Front Door Premium + Private Link (private origin)

Keeps the load balancer private — nothing gets a public IP. Front Door reaches it over Azure's internal network through Private Link. Leave the ingress as the installer set it (do not run A1). The pieces:

• Front Door Premium — created in Phase 1 step 5 with --sku Premium_AzureFrontDoor. Only Premium supports Private Link.
• Private Link — a private connection from Front Door to the internal ingress; you approve it once on the cluster side.
• Origin + route — the origin targets the private ingress via Private Link, with the same /healthz probe and host header as Option A.

The exact Private Link approval commands vary by az version — confirm az afd origin create --enable-private-link / az network private-endpoint-connection approve against Microsoft's docs before running them. No NSG public-IP fence is needed (there's no public IP).

First login

Open the login page (open is macOS; on Linux use xdg-open, or just paste the URL in your browser):

open https://$FD_HOST/login

Read the one-time admin key from the vault:

az keyvault secret show --vault-name $VAULT --name bootstrap-key --query value -o tsv

Enter it to create the first admin. The key is then spent; the admin configures real SSO inside Studio (Org Settings → Identity Providers) and GitHub (Org Settings → GitHub). If you installed observability, Grafana (/grafana/) signs in with the same Studio identity.

Verify the install

Argo should report the app healthy:

kubectl -n argocd get applications

And the core pods should be running:

kubectl -n studio get pods

---

Observability secrets (optional)

Only if you install with --with-observability. Add these in Phase 1 step 4 before installing.

For --with-observability (LGTM/Grafana) — add 2:

az keyvault secret set --vault-name $VAULT --name grafana-client-secret  --value "$(openssl rand -base64 32)"
az keyvault secret set --vault-name $VAULT --name grafana-admin-password --value "$(openssl rand -base64 24)"

---

Upgrades

Studio never upgrades itself. A published release is noticed — the app reports it as available — and then waits, so that nothing rolls out before a backup exists. Ask for it when you are ready:

kubectl -n argocd annotate application studio-app argocd.argoproj.io/refresh=hard --overwrite
kubectl -n argocd patch application studio-app --type merge -p '{"operation":{"sync":{"prune":true,"syncStrategy":{"hook":{}}}}}'

The in-cluster safety gate runs on every sync: it blocks new logins, waits for in-flight agent work to finish, and only then lets the new version replace the old one.

Show the maintenance page while you upgrade

Your cluster runs a small maintenance pod at all times, separate from Studio. Point the public route at it before you sync, and users get the Studio maintenance page for the whole window instead of gateway errors. The Front Door route stays enabled — sending traffic to the maintenance pod is what drains Studio.

kubectl -n studio patch ingress studio --type=json -p '[{"op":"replace","path":"/spec/rules/0/http/paths/0/backend/service","value":{"name":"maintenance","port":{"number":8080}}}]'

Check it before you go further — this must report 503:

curl -sS -o /dev/null -w '%{http_code}\n' https://$STUDIO_DOMAIN/

Anything that re-applies the chart puts the route back to frontend, so re-run the patch after each sync while the window is open.

When the upgrade is done, put users back — but only after you have opened an Intent and seen it work, not just a green health endpoint:

kubectl -n studio patch ingress studio --type=json -p '[{"op":"replace","path":"/spec/rules/0/http/paths/0/backend/service","value":{"name":"frontend","port":{"number":8080}}}]'
curl -sS -o /dev/null -w '%{http_code}\n' https://$STUDIO_DOMAIN/    # expect 200

If the upgrade failed, leave the maintenance page up. It stays until you deliberately run the command above, which is what keeps users away from a half-upgraded cluster while you fix it or restore. The automation script below does this same switch for you, and leaves it on for the same reason.

Take a backup first (next section) — migrations have no down-path, so a backend that fails after migrating is recovered from a backup, not rolled back by hand. "Automating the whole upgrade" below does the backup, the per-component swaps, the health checks, and the rollback for you; prefer it over running the commands by hand.

Backup and recovery

Studio's database migrations have no down-path, so a backend that fails after migrating is not rolled back by swapping the old container in — the backup is restored over the same cluster underneath the old backend (vibedata restore --force), which is exactly what the automation script does on a failed backend swap. That makes the backup you take before an upgrade the thing that matters.

Restoring into a new cluster is the break-glass variant, for a cluster that can no longer run anything at all. There the broken cluster is left alone — it is the only place the failure can be reproduced.

Backups need the second file share from Phase 1 step 3. Every command below reads its address from your variables:

BACKUP_URL="$(az storage account show -g $RG -n $STORAGE --query primaryEndpoints.file -o tsv)$BACKUP_SHARE"

Take a backup

vibedata backup kubernetes --to "$BACKUP_URL"

This runs inside the cluster — a short-lived pod dumps the databases and tars the shared data files, with the backup share mounted for that one run and unmounted after, so nothing leaves the cluster on the way. The package folder is named pre-upgrade-<running version>-<timestamp> unless you pass --name (for example pre-upgrade-v1.2.3-20260811213045). The release is in the name so the folder says what it restores to; the time is in it because a package whose name already exists is reused rather than rewritten — reuse protects a retried upgrade from dumping a half-migrated database over good dumps, but it also means a fixed name silently stops taking new backups. The newest 2 packages are kept; older ones and any half-written folder are deleted.

A package holds a dump per database (Studio, the agent service, and Langfuse when it is installed), the DATA_DIR files that exist nowhere else, and a manifest.json written last — a folder without one is never restored from. Your az and gh sign-ins are deliberately not in it, so repeat those in Studio after any restore.

See what you can restore

vibedata restore kubernetes --list --from "$BACKUP_URL"

Prints the complete packages on the share, newest first, with the Studio version and date of each. The version is what you install in step 3 below.

Recover onto a new cluster (break-glass)

This is the path for a cluster that cannot serve even the previous release — nodes dead, disks unattachable, or the in-place restore itself failed. For an upgrade that merely shipped a bad release, prefer the in-place rollback the automation script performs (old renders back + restore over the same cluster): it needs no new infrastructure and no spare quota. The script's recover-cluster mode automates the steps below; they are listed for running by hand and for adapting to other clouds.

1. Take the edge offline and leave the broken cluster running. Deleting it destroys the only copy of the failure. Disable the route, not the origin — Azure refuses to disable the last origin in a route's origin group:

   az afd route update --route-name studio-route --endpoint-name $FD_ENDPOINT --profile-name $FD_PROFILE --resource-group $RG --enabled-state Disabled

2. Find the version to go back to with --list above. It is the version that was running before the upgrade — not the newest release, which is the one that just failed.

3. Create a new AKS cluster and a new data share, keeping the same resource group, Key Vault, storage account, and Front Door. Both clusters exist at once, so this needs a second cluster's worth of vCPU quota free (Phase 1 step 2) — check it before you need it, not during an outage. Reusing the Key Vault is not optional: the restored data is encrypted with the data-encryption-key in that vault, and a new vault means a new key, which means every stored credential is unreadable. Grant the new cluster's identity the Key Vault Secrets User role (Phase 1 step 4) and allow its subnet on the storage account (Phase 1 step 3).

4. Install that version, pinned:

   vibedata install kubernetes --cloud azure --domain "$FD_HOST" --storage-url "$STORAGE_URL" --vault-url "$VAULT_URL" --vault-identity-client-id "$CLUSTER_ID" --version v1.2.3

   --version pins the install to one release instead of the open auto-upgrade channel, so the cluster does not immediately pull the release that failed. Use the vibedata binary of that same version — CLI and Studio are paired 1:1.

5. Restore into it:

   vibedata restore kubernetes --from "$BACKUP_URL/pre-upgrade-v1.2.3"

   It prints what it is about to replace and asks for confirmation; --yes skips the prompt in a script. It refuses a cluster that already holds data — pass --force only when you mean to overwrite it, since the supported recovery is a fresh cluster. Studio no longer provisions Langfuse; --with-langfuse exists only to recover a pre-existing Langfuse deployment from an older backup, and only when its trace store survived — without its traces, restoring its dashboards would leave them pointing at nothing.

6. Repeat the platform sign-ins inside Studio (gh, and az if you use Fabric) — their token caches are not in the backup.

7. Point Front Door's origin at the new cluster (Phase 3, A1–A3), and verify.

8. Later, once the failure is understood, delete the quarantined cluster and its old data share.

Automating the whole upgrade

scripts/devops/upgrade-with-recovery.sh in the Studio repo is the reference script: drain users behind the edge, back up, then swap each component (frontend → obot → backend) to the new release's chart render, verifying each before the next. A component that fails is swapped back to the running release's render on the same cluster; a backend that fails additionally gets the backup restored underneath it. Copy it and adapt the az, Front Door, and trigger parts to your own infrastructure. A failure POSTs to a webhook URL if you set one, so it lands in your alerting.

It has three modes:

Mode	What it does	When
upgrade (default)	The swap flow above	Your cron / pipeline / by hand — safe to trigger blindly, it exits 0 when there is nothing to do
rehearse	backup → restore that backup → verify Studio comes back, then records when it passed	Optional, in a maintenance window — run it when you want to see recovery work before you have to rely on it
recover-cluster	The break-glass new-cluster recovery above	Only when the cluster itself is dead; requires typing the cluster name as confirmation plus RECOVER_STABLE_VERSION and RECOVER_PACKAGE

The upgrade target is the newest chart on the channel your cluster installs from — read with the same resolver Argo uses, so the target is installable by definition (GitHub release lists are a different artifact stream and are never consulted for the target). Before upgrade touches anything it confirms that chart is newer than what runs, that the app is not pinned to one chart version, that the vibedata binary matches the target version (it downloads the right one itself when not), and that your vault holds every secret the new version needs in the shape its consumer can parse. vibedata install kubernetes --list-secrets prints that contract as <name> <format> (text, json-object, base64-32, hex-64), and the upgrade checks the value of every key whose format is more than text — a vault holding [] where {} was required is caught here rather than by a crashlooping backend after everyone has been drained. Values are compared by shape and never printed. Any of those failing stops the run with Studio still up and serving — nobody is drained.

That last check exists because a release can introduce a new secret, and syncing the chart alone cannot deliver one: the chart does not own the objects that read your vault, so the cluster carries on fetching the old set and any pod wanting the new value never starts. Left to itself that looks like a failed upgrade, and the script would recover onto a fresh cluster over one missing string.

So it checks first, and only for the secrets your installed profile actually uses — a core install is never asked for Grafana's. Then one of three things happens:

What it finds	What it does
Every needed secret already in the cluster	Nothing extra — straight on with the upgrade. The normal case.
A secret the vault does not have	Stops and names it. Add it to the vault and run again.
In your vault, not yet in the cluster	Re-runs the installer to pick it up, then upgrades. No action needed.

Whoever runs the script therefore needs permission to read secret names from the vault (az keyvault secret show). Only names are read, never values. Without it every secret looks absent and the script refuses to start — safe, but it will not upgrade until the permission is there.

Set these before running it:

Variable	What it is	How to get it
VIBEDATA_BIN	Path to your vibedata binary	command -v vibedata
AKS_RESOURCE_GROUP	Resource group holding everything	the $RG you chose above
AKS_CLUSTER_NAME	Cluster to upgrade	the $AKS you chose above
STUDIO_DOMAIN	Public hostname; used for the health check	az afd endpoint show -g $RG --profile-name $FD_PROFILE --endpoint-name $FD_ENDPOINT --query hostName -o tsv
STORAGE_URL	Data share	az storage account show -g $RG -n $STORAGE --query primaryEndpoints.file -o tsv then append $SHARE
BACKUP_SHARE_URL	Backup share (a different share)	same, appending $BACKUP_SHARE
VAULT_URL	Key Vault — recovery must reuse it	az keyvault show -n $VAULT --query properties.vaultUri -o tsv
FRONT_DOOR_PROFILE	Front Door profile in front of the cluster	the $FD_PROFILE you chose above
STORAGE_RESOURCE_GROUP	Only if storage is in a different RG	that RG's name
VAULT_IDENTITY_CLIENT_ID	Only for a multi-identity cluster	az aks show -g $RG -n $AKS --query identityProfile.kubeletidentity.clientId -o tsv
DEMO_ARTIFACT_REPOSITORY	Optional Demo Pack source, owner/repo	the trusted public GitHub repository whose version-matched release carries Demo Pack bundles; leave unset to use Studio's own public release channel
RELEASE_REPO	Where vibedata CLI binaries download from	leave unset for the public releases repo; the upgrade target comes from your chart channel, not from here
UPGRADE_ALERT_WEBHOOK	Optional; where failures are POSTed	a Slack incoming-webhook URL (it POSTs {"text": "..."})
RECOVERY_NODE_VM_SIZE	recover-cluster only: recovery VM size	leave unset to copy the cluster's size; set it when that size's quota family is full
RECOVER_STABLE_VERSION	recover-cluster only: version to install	the version that was running before the failure, from --list
RECOVER_PACKAGE	recover-cluster only: package to restore	e.g. pre-upgrade-v1.2.3, from --list

Draining works by disabling the Front Door route, which takes the endpoint offline for the upgrade window — Azure refuses to disable the only origin in a route's origin group, so the route is the switch.

What it exits with:

Code	Meaning
0	Upgraded, rehearsed, or already on the latest release
2	Rolled back — users are being served again on the previous release, on the same cluster; the target release is broken and needs a human
1	Failed. The edge may still be disabled, so Studio is down until someone acts

Treat any non-zero as "act". The separate 2 exists so your pipeline can report a rollback as a success that needs follow-up rather than an outage — rollback working as designed should not look the same as nothing working.

Permissions the account running it needs

upgrade and rehearse need what an install needs, plus reading secret names from the vault (above) and the Front Door route toggle. The grants below are only for recover-cluster, because it creates a cluster and grants that cluster access. If a service principal runs it, give it these ($SP = its object id):

az role assignment create --assignee $SP --role Contributor --scope /subscriptions/$SUB/resourceGroups/$RG
az role assignment create --assignee $SP --role "Network Contributor" --scope /subscriptions/$SUB
az role assignment create --assignee $SP --role "User Access Administrator" --scope /subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.KeyVault/vaults/$VAULT

• Contributor on the resource group — create the cluster, create the recovery data share, add the storage network rule.
• Network Contributor at subscription scope — the new cluster's node resource group is named after the cluster, so it cannot be granted in advance. Use your own subnet (Phase 1 step 2) and you can skip this one entirely, because there is no new network to allow.
• User Access Administrator on the vault only — the new cluster's identity needs Key Vault Secrets User, and granting a role needs permission to grant.

Check with az role assignment list --assignee $SP --all -o table. The script fails clearly if a permission is missing, but only once an upgrade has already gone wrong.

Recovery also takes longer to finish serving than an upgrade: the new cluster is an origin the edge has never seen, so Front Door discovers it and waits for several good health probes first. That ran past 15 minutes on a real recovery, so the script allows much longer for that last check.

The install already leaves the cluster ready for this: the Studio app is arm-only — when a release publishes, Argo notices and reports it, but applies nothing until a sync is asked for. Studio never upgrades itself; the script (or a manual sync) is what starts an upgrade, and the script takes the backup first. The in-cluster safety gate still runs on every sync, unchanged.

Only a cluster installed before arm-only became the default needs this once:

kubectl -n argocd patch application studio-app --type merge -p '{"spec":{"syncPolicy":{"automated":null}}}'

Growing the database disk

The in-cluster PostgreSQL starts on a 5 GiB disk (a PVC — a persistent volume claim, the cluster's request for a disk — named data-postgres-0). Check usage any time:

kubectl -n studio exec postgres-0 -- df -h /var/lib/postgresql/data

To grow it, edit the claim — AKS's default managed-csi storage class expands disks online, no downtime:

kubectl -n studio patch pvc data-postgres-0 --type merge -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

Watch until the new size shows under CAPACITY (a minute or two):

kubectl -n studio get pvc data-postgres-0 -w

Notes:

• Grow only. Kubernetes disks can never shrink.
• The chart's built-in size (postgres.storage) only applies when a disk is first created, and GitOps never resizes an existing one — the PVC you patched is the source of truth for this install, and the mismatch with the chart default is harmless.
• If kubectl describe pvc reports the resize waiting on a pod restart (older storage drivers), restart the database during a quiet window: kubectl -n studio delete pod postgres-0. Deleting the pod directly is the supported move — the database is deliberately protected from automatic eviction (node drains block on it), so moving or restarting it is always an explicit operator action like this one.
• If your cluster uses a custom storage class without allowVolumeExpansion, the disk cannot grow in place: take a pg_dump backup, recreate the claim at the new size, and restore.

Cost and teardown

Only the node VMs stop cheaply. Front Door and Premium Files can't be stopped — only deleted — so az aks stop cuts the big cost but still leaves ~$1–2/day. Pick the level you need.

Pause overnight (fastest resume, keeps data)

Stop the node VMs — the main meter (the control plane is Free tier, $0):

az aks stop -g $RG -n $AKS

Next session, bring it back — ~5 min, data intact. The ingress IP and Front Door persist across stop/start, so nothing needs re-wiring:

az aks start -g $RG -n $AKS

While stopped you still pay ~$1–2/day: Front Door (~$35/mo), the two Premium shares (~$16/mo each), the public IP, and disks. Fine for a night or two.

Pause for several days (also drop Front Door)

Front Door is the biggest idle cost and can only be removed by deleting it:

az afd profile delete --profile-name $FD_PROFILE --resource-group $RG --yes

To resume, recreate it (Phase 1 step 5) and re-wire it (Phase 3 A3).

Go to $0 (delete per resource)

Delete the cluster (node VMs + public IP):

az aks delete -g $RG -n $AKS --yes --no-wait

Delete Front Door (~$35/mo):

az afd profile delete --profile-name $FD_PROFILE --resource-group $RG --yes

Delete the storage account (~$32/mo for both shares) — this erases your DATA_DIR data and every backup package:

az storage account delete -g $RG -n $STORAGE --yes

Delete and fully purge the Key Vault (soft-delete otherwise holds the name):

az keyvault delete --name $VAULT -g $RG
az keyvault purge --name $VAULT

Or delete everything in one shot — after this you rebuild from Phase 1 next time:

az group delete --name $RG --yes --no-wait

What costs what

Resource	Idle cost	Stop	Zero it
AKS node VMs	biggest	az aks stop	az aks delete
AKS control plane	$0 (Free tier)	—	—
Front Door Standard	~$35/mo	can't stop	az afd profile delete
Premium Files (2 × 100 GB)	~$32/mo	can't stop	az storage account delete (loses data)
Public IP / disks	small	—	released with the cluster
Key Vault	~$0	—	az keyvault delete + purge

Costs are approximate (East US). Overnight → az aks stop / start. Truly $0 → delete Front Door + storage + cluster.

Troubleshooting

• Front Door returns 504 / origin unhealthy. Usually the AzureFrontDoor.Backend NSG rule is missing — redo A2. Otherwise check the probe path Azure is actually using; it must be /healthz, not /:

  kubectl -n ingress-nginx get svc ingress-nginx-controller -o jsonpath='{.metadata.annotations.service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path}'

  Note that a curl from inside the cluster cannot detect this — it bypasses the load balancer and returns 200 either way.
• Front Door returns 404 (its own "Page not found", x-azure-ref header — not nginx). The origin points at the wrong address (e.g. a stale/internal 10.x IP instead of your public ingress IP), or the route wasn't deployed. Check and fix:

  az afd origin show --origin-name studio-ingress --origin-group-name studio-og --profile-name $FD_PROFILE --resource-group $RG --query hostName -o tsv
  az afd origin update --origin-name studio-ingress --origin-group-name studio-og --profile-name $FD_PROFILE --resource-group $RG --host-name "$INGRESS_IP"
  az afd route update --route-name studio-route --endpoint-name $FD_ENDPOINT --profile-name $FD_PROFILE --resource-group $RG --origin-group studio-og --link-to-default-domain Enabled

  Wait 2–3 min, then re-test. (Confirm the origin itself is fine first: curl -H "Host: <your-hostname>" http://$INGRESS_IP/healthz should return 200.)
• INGRESS_IP is a 10.x address. The load balancer is still internal — the annotation wasn't removed or reconciliation hasn't finished. Re-run A1's first command and wait 1–2 min.
• Anyone can reach the raw IP. The deny-internet-origin rule is missing or higher-numbered than the auto allow Internet rule. Check az network nsg rule list --nsg-name "$NSG" -g "$NODE_RG" -o table.
• Install stops with VAULT_SECRET_UNREADABLE. A vault secret is missing, misnamed, or unreadable. The message names the exact key and vault and lists every key your chosen profile needs — create or fix that one and re-run. If it names no key but reports an auth failure, the cluster identity lacks the Key Vault Secrets User role or the auth mode is wrong: pass --vault-identity-client-id "$CLUSTER_ID" at install. Check kubectl -n studio get externalsecret.
• obot pod in CrashLoopBackOff on a fresh install. Expected — not a failure. obot needs Studio's OIDC discovery, which isn't served until you finish First login and add an Identity Provider. Core Studio works without it; obot recovers on a later restart once discovery is reachable. If it still crashes after SSO is set, check kubectl -n studio logs deploy/obot.
• Grafana pods won't start. You passed an observability flag without creating the matching grafana-* secrets. The install now catches this up front by name; if pods are still stuck, add the secrets and re-run install.
• Storage/mount timeout at install. The NFS share isn't reachable from the cluster subnet — revisit Phase 1 step 3. Check az storage account show -g $RG -n $STORAGE --query networkRuleSet.
• MissingSubscriptionRegistration. You skipped Phase 1 step 1 — register the providers, wait for Registered, retry.

---

⚠️ Temporary workaround — first install hangs at the upgrade gate

Remove this section once a release containing the fix is published. The code fix is already merged (the gate now fails open when the DB is unreachable on a first install), but it is not yet in a published image/chart — so any cluster pulling the current published release still hits this. Known bug: on a brand-new install the upgrade-gate safety check (an Argo PreSync hook) runs before the database exists, can't reach a DB, and blocks the install. Symptom: install reports READINESS_FAILED: studio-app not healthy within 600s, and kubectl -n argocd get application studio-app shows OutOfSync / Missing. Until the fixed release ships, bring the database up first so the gate passes on its own.

Do not re-run vibedata install kubernetes — the install already created everything; it only got stuck at the gate. These steps push the existing Argo app past it.

Confirm you're on the AKS cluster, not a local one:

kubectl config current-context

Bring up only Postgres first, so the gate has a database to reach:

kubectl -n argocd patch application studio-app --type merge -p '{"operation":{"sync":{"resources":[{"group":"apps","kind":"StatefulSet","name":"postgres"},{"group":"","kind":"Service","name":"postgres"}]}}}'

Wait until postgres-0 shows 1/1 Running, then Ctrl-C:

kubectl -n studio get pods -w

Run a full sync — the gate now passes (Postgres is up) and everything else deploys. This requests one sync; it does not turn on automated sync, which would let releases roll out without a backup:

kubectl -n argocd patch application studio-app --type merge -p '{"operation":{"initiatedBy":{"username":"recovery"},"sync":{"prune":true,"syncStrategy":{"hook":{}}}}}'

Watch backend/frontend/obot come up (backend sits at 0/1 during migrations, then 1/1):

kubectl -n studio get pods -w

Confirm healthy — should read Synced / Healthy:

kubectl -n argocd get application studio-app -o jsonpath='{.status.sync.status} / {.status.health.status}{"\n"}'

Then continue with Phase 3 — After installation.