What does ATC Ops actually monitor?
+
GPU temperature, power draw, ECC errors, VRAM fragmentation, model throughput and latency percentiles, queue depth, error rate, and dataset drift on production prompts.
How is predictive maintenance done?
+
GPU error patterns and thermal trends are modelled to predict imminent ECC or power failures hours to days before they occur, allowing proactive replacement during planned windows.
Does ATC Ops auto-scale inference?
+
Yes. Based on queue depth and target p99 latency, it scales vLLM workers up and down, drains traffic gracefully on shutdown, and warms new workers before adding to the load balancer.
Which alerting backends are supported?
+
PagerDuty, Opsgenie, VictorOps, Slack, Microsoft Teams, email, and webhook. Custom escalation policies per service tier and team.
Can ATC Ops auto-remediate incidents?
+
Yes for safe-listed incidents — restarting hung workers, rotating bad GPUs, draining nodes for kernel updates. Risky actions (failover, traffic shifts, capacity changes) require human approval.
What is the agent that maintains the agents?
+
A meta-agent on top of all the other BiltIQ agents in production. It treats each agent as an SLO target, watches its inputs and outputs, and triggers ATC Ops automation when SLOs degrade.
How does ATC Ops integrate with existing observability?
+
Prometheus, Grafana, Loki, OpenSearch, Datadog, and New Relic — by scrape, push, or OTLP. ATC Ops adds the AI-aware analysis layer on top of whatever stack you already run.
How is ATC Ops different from your DevOps & AI Ops service?
+
ATC Ops is a productised meta-agent that watches AI workloads (GPUs, models, queues) and is installable in days. The DevOps & AI Ops service is a broader engagement covering your CI/CD, IaC, hybrid-cloud architecture, and incident-response runbooks. ATC Ops is one component of what the service delivers.
