Techhalo · 360Voice Platform

Production architecture · voice360-prod-aks · Azure East US · Last updated June 2026

production release staging dev

Application services (AKS)

Managed SaaS (cloud providers)

In-cluster infrastructure

External vendors

CI/CD pipeline

Hardening backlog

System overview

ID card flow

CI/CD pipeline

DR status

Client ID card delivery flow

When a client needs to download their ID card, halo-task-manager-api triggers a Twilio SMS with a unique URL. The client taps the link and lands on the bic-sky.halo360.ai frontend (served by bic-agent), which renders the ID card and allows download — no login required.

halo-task API

Generates unique ID card link. Calls Twilio API to send SMS.

Twilio SMS

Sends SMS to client's phone number with the link.

Client's phone

Receives SMS, taps the unique link in message.

DNS resolves

bic-sky.halo360.ai → Azure LB 20.124.179.20 → ingress-nginx

bic-agent serves

Angular app renders ID card. Client downloads PDF.

Download

ID card downloaded to client device. Flow complete.

Environment hostnames

Production

bic-sky.halo360.ai

live

Release

release-sky.halo360.ai

pre-prod

Staging

staging-sky.halo360.ai

test

Dev

dev-sky.halo360.ai

dev

⚠ Notes (from ARCHITECTURE.md)

No explicit documentation was provided for the bic-sky ID card flow — the above is inferred from the architecture. The appconfig.json baked into each bic-agent image points at the correct api-halo-task-* host per environment. The 0.1.27-r119 release image is correctly configured; the 0.1.27-r121 prod image ships staging URLs and requires a postStart hook or image swap to fix.

Branch promotion flow

dev

release

main

production ✓

Branch policy enforced by GitHub Actions

Build & push jobs (dev / release / main)

git push to dev/release/main

↓

build-and-push · docker buildx · tag X.Y.Z-r<RUN>

↓

trivy-scan · CRITICAL+HIGH gate · SARIF → GitHub Security

↓

deploy · az aks get-credentials · kubectl create secret · helm upgrade --install --wait 5m

Production promotion (no rebuild)

push to production branch (from main)

↓

resolve-prod-image · picks newest non-latest tag from 360voiceprod ACR

↓

Trivy skipped (already scanned on main)

↓

deploy to voice360-prod-aks · letsencrypt-prod · 360voice.techhalo.ai

Build once, promote many
Production always runs the exact same image bytes that were scanned and soak-tested on main. No re-build = no new vulnerabilities introduced at promote time.

⚠ CI/CD gaps (from DEPLOY.md)

Multi-repo coordination
bic-agent and halo-task live in separate repos with their own CI. No coordinated "prod release manifest" listing all three image tags shipped together.

No smoke test
aks.yml does helm upgrade --wait but no post-deploy curl against the public hostname. Broken cert/ingress won't surface automatically.

Deployment targets per branch

Branch	ACR	Cluster	Namespace	Public host
dev	360voice	aplcommissions-aks	360voice-dev	360voice-dev.techhalo.ai
release	360voicerelease	aplcommissions-aks	360voice-release	360voice-release.techhalo.ai
main	360voiceprod	aplcommissions-aks	360voice-prod	360voice.techhalo.ai (legacy)
production ✓	360voiceprod	voice360-prod-aks	360voice-prod	360voice.techhalo.ai

RTO / RPO targets vs current reality

Component	RTO target	RPO target	Status

Hardening backlog (priority order)

Kafka HA

Move from 1 broker to 3 (replicationFactor=3, min.insync=2). Or migrate to Azure Event Hubs.

P1 · data loss risk

Cluster GitOps

Terraform/Bicep for AKS + Flux/ArgoCD for Helm. Eliminates 4-hour manual rebuild.

P2 · ops risk

Secret backup

Automate export to Azure Blob / Key Vault. Currently manual export to OneDrive.

P3 · security

Multi-AZ

Spread nodepool across AZs 1/2/3. Min 2 replicas per service with anti-affinity.

P4 · availability

Observability

Metrics + alerting + on-call rotation. Currently Sentry + AKS Container Insights only.

P5 · ops

App services

In-cluster infra (kafka ns)

Managed SaaS

External vendors

Needs attention

Healthy

Services & ports

Infrastructure

Secrets & config

Troubleshooting

voice360 — AI/Voice backend

Deploy: voice360-prod-360-deploy
Container: voice360 · port 8000
Image: 360voiceprod.azurecr.io/360-voice:0.1.94-r189
Replicas: 1 · ClusterIP 10.0.244.91:80→8000
Ingress: 360voice-release.techhalo.ai
TLS: voice360-tls (letsencrypt-prod)
Secret: voice360-env (envFrom)
Helm: voice360-prod rev 3 · chart 360-voice-0.1.0
Health: HTTP GET / on :8000

⚠ LiveCallConsumer degraded Kafka producer/consumer RabbitMQ subscriber ClickHouse writer

bic-agent — Angular frontend

Deploy: bic-agent-taskmanager-voice360-prod-bic-taskmanager-deploy
Container: bic-agent-taskmanager · port 4200
Image: 360voiceprod.azurecr.io/bic-agent-taskmanager:0.1.27-r121
Replicas: 2 · Ingress: halovoice-release.techhalo.ai
ID card: bic-sky.halo360.ai (same pod, different hostname)
TLS: bic-tls · Secret: bic-agent-taskmanager-env
Helm: bic-agent-taskmanager-voice360-prod rev 8
Config: appconfig.json baked into image (API URL: api-halo-task-release.techhalo.ai)

⚠ r121 ships staging URLs NGINX static appconfig.json

halo-task-manager-api — .NET API

Deploy: halo-task-manager-api-prod-halotask-backend-deploy
Container: backend-api · port 44301
Image: 360voiceprod.azurecr.io/backend-api:0.1.51-r110
Replicas: 1 · Ingress: api-halo-task-release.techhalo.ai
TLS: halo-task-tls · Secret: halo-task-manager-api-env
Helm: halo-task-manager-api-prod rev 5
Health: HTTP GET /health on :44301
In-process cache: Redis not configured (localhost fallback)

JWT/OpenIddict Stripe/PayPal Twilio/ClickSend Postgres S3 audio

Kafka (KRaft, in-cluster)

Namespace: kafka · StatefulSet: voice-kafka-controller
Image: bitnamilegacy/kafka:4.0.0-debian-12-r10
Storage: data-voice-kafka-controller-0 · 32Gi PVC
Internal: voice-kafka.kafka.svc.cluster.local:9092
External: kafka-prod-new.techhalo.ai:9094 → 4.156.62.101
Auth: PLAINTEXT (no auth on any listener)
Resources: 1 CPU / 2GiB RAM (250m req / 1 CPU lim)

⚠ Single broker · no replication KRaft (no ZK) bitnamilegacy

Topics: prod.voice.turn_resolution.v1 (3p,1r) · prod.voice.call_trace.v1 (3p,1r) + DLQs

Ingress + TLS

Controller: ingress-nginx chart ingress-nginx-4.15.1
Public LB: 20.124.179.20 · NodePort 30541
Cert-manager: v1.16.2 · ClusterIssuer: letsencrypt-prod
Certs: voice360-tls · bic-tls · halo-task-tls
Challenge: HTTP-01 via same ingress
Kafka ext LB: 4.156.62.101 (separate)

Let's Encrypt prodHTTP-01auto-renew

Cluster topology

Cluster: voice360-prod-aks · RG: voice360-prod-rg
Region: eastus · K8s: 1.34
Node SKU: Standard_D4s_v3 (4vCPU, 16GiB)
Autoscale: 3–6 nodes · Azure CNI
Pod CIDR: 10.244.0.0/16 · Svc CIDR: 10.0.0.0/16
DNS svc IP: 10.0.0.10 · Tier: Standard
API FQDN: voice360-p-…hcp.eastus.azmk8s.io
Kubelet ID: b29d9d53-d487-49f6-af99-fb2f820e2a17

⚠ Single AZManaged identityAcrPull on b29d9d53

External data services

RabbitMQ — CloudAMQP

Host: toucan.lmq.cloudamqp.com:5671 (TLS)
User/vhost: hvqhydzg
Exchanges: 360voice_exchange, call_steps_exchange
Queues: call_status, call_steps, call_transcripts

⚠ LiveCallConsumer not connectedSaaS · managed

ClickHouse Cloud

Provider: ClickHouse Cloud (GCP us-east1)
Host: un7hq7jnnr.us-east1.gcp.clickhouse.cloud:8443
User: voice_agent
DBs: voice (turns), analytics (call_trace)

Auto daily backupsHTTPS

Postgres — UpCloud

Host: public-teleport-backend-db-zbzzbqeytzmq.db.upclouddatabases.com:11550
User: upadmin
DB: halotasksapidb-release
Backup: automated vendor snapshots

RTO 1h · RPO 24hAuto-snapshots

AWS S3

Bucket: halovoiceaudiotranscripts
Region: us-east-1
Auth: IAM access key (in halo-task-manager-api-env)
Used by: halo-task for audio + transcripts

IAM key authus-east-1

Azure ACR (×2)

360voicerelease — release/main builds
360voiceprod — production promotions
Auth: kubelet managed identity (b29d9d53)
Attached 5/5/2026: az aks update --attach-acr

AcrPull via MSITrivy scanned

DNS / Cloudflare

All prod app hosts → 20.124.179.20
Kafka ext → 4.156.62.101
ID card host: bic-sky.halo360.ai → same LB IP
Kafka DNS: kafka-prod-new.techhalo.ai

Cloudflare

Namespaces

Namespace	Purpose	Workloads
`360voice-prod`	All app services + secrets	voice360, bic-agent, halo-task + Helm secrets
`kafka`	In-cluster Kafka broker	voice-kafka-controller-0 StatefulSet
`ingress-nginx`	Ingress controller	ingress-nginx-controller
`cert-manager`	TLS certificate issuance	cert-manager + webhook + cainjector

⚠ Security posture: All secrets stored as plain Kubernetes Secret objects. No Azure Key Vault integration (Configuration__AzureKeyVault__IsEnabled=false). Backup: manual export via agent-tools/export-secrets.ps1 to local OneDrive only.

voice360-env

Used by: voice360 pod (envFrom secretRef)
Contains: Kafka bootstrap, RabbitMQ creds, ClickHouse creds, OpenAI key, Sentry DSN, CORS origins, AWS creds, Stripe keys

Created by CI (aks.yml)From VOICE360_ENV_FILE_PROD

bic-agent-taskmanager-env

Used by: bic-agent pod
Contains: Angular runtime config overrides (if any). Most config baked into appconfig.json in the image.

Manual / separate CI

halo-task-manager-api-env

Used by: halo-task pod
Contains: Postgres conn string, RabbitMQ creds, S3 IAM key, Stripe/PayPal keys, Twilio/ClickSend, JWT signing keys, CORS origins, App__CorsOrigins

Manual / separate CICase-sensitive keys

TLS secrets

voice360-tls — voice360 cert
bic-tls — bic-agent cert
halo-task-tls — halo-task cert
Managed by cert-manager. Auto-renew via Let's Encrypt HTTP-01.

Auto-managedletsencrypt-prod

GitHub repo secrets

AZURE_CREDENTIALS_PROD — SP for prod ACR/AKS
VOICE360_ENV_FILE_PROD — env file content
VOICE360_ENV_FILE_RELEASE — release env
VOICE360_ENV_FILE — dev env

Repo-levelaz ad sp create-for-rbac

Known issues

PowerShell ConvertFrom-Json chokes on duplicate keys: RabbitMq__Connection vs RabbitMQ__Connection. Use export-secrets.ps1 (ordinal Dictionary) to avoid. Legacy IP 10.0.103.41 in any secret = stale, update to CloudAMQP.

Case sensitivity