At three in the morning, a single fibre cut in a metro aggregation ring produced 11,400 alarms in nineteen minutes. Four engineers in the network operations centre spent the next three hours reading those alarms one screen at a time. The fault itself took eleven minutes to fix once someone found it. This was the operational reality at a leading telecommunications enterprise when they brought KriraAI into the conversation, and it was not an unusual night.
The operator was absorbing roughly 1.4 million alarms per month across 22,000 managed network elements. Internal audits showed that 82 percent of those alarms were redundant symptoms of a smaller number of real faults. Mean time to repair sat at four hours and twelve minutes, and most of that time was diagnosis, not repair. KriraAI is an AI solutions company that designs and ships production-grade systems for enterprise environments, and through our AI consultancy services, this was exactly the class of problem we exist to solve.
This blog covers the whole engagement. It explains the operational situation before we arrived, the system we designed and delivered, the complete architecture layer by layer, the technology choices and the reasoning behind each one, the implementation journey, including the parts that went wrong, and the measured results after go-live. It is written for engineers and operators who want to know how AI in telecom network operations actually works when it has to survive real traffic.
The operator ran a hybrid estate that had accumulated over two decades, the kind of complex environment KriraAI's telecommunications solutions are built to modernize. Optical transport, IP core, radio access, and a growing virtualised core all coexisted, each with its own element management system. Every one of those systems emitted alarms in its own dialect, on its own schedule, with its own severity semantics. The network operations centre was the place where all of that noise arrived at once.
A single physical fault does not announce itself once. It cascades upward through dependent layers, and every dependent element reports its own local view of the damage. A cut in the transport layer becomes hundreds of link-down events, thousands of service degradation notices, and a flood of threshold-crossing alerts. The engineer sees the flood, not the cut.
The correlation rules meant to control this had been written by hand over eleven years. There were 3,700 of them in production. Nobody on the current team had written more than a fraction of them, and the topology they assumed had changed many times since. Rule maintenance had quietly become a full-time job for two engineers who were needed elsewhere.
Because the rules were static and the network was not, coverage decayed continuously. New equipment arrived with no matching rules, so its alarms passed through uncorrelated. Old rules fired on topology that no longer existed, producing confident and wrong conclusions. Engineers had learned to distrust the correlation output entirely and read raw alarms instead.
The operator was not short of data. Every ticket ever closed carried a free text resolution note written by the engineer who fixed the fault. That archive held 340,000 tickets and roughly nine years of hard-won diagnostic reasoning. It was stored, indexed for compliance, and read by nobody.
Topology data was equally abundant and equally idle. The inventory system knew which port connected to which port, which service traversed which path, and which customer sat behind which node. That graph was the single most useful diagnostic asset in the building. It lived in a relational schema that nobody queried during an incident because it was too slow under pressure.
The human cost of ignoring both datasets was visible in the numbers. Roughly 31 percent of field dispatches were later classified as unnecessary because the fault was diagnosed incorrectly before the truck left. Each unnecessary dispatch costs the operator real money and real hours. Regulatory service level commitments were being missed often enough to trigger financial penalties in two markets.
Competitive pressure removed the option of doing nothing. Two rivals had publicly committed to autonomous network programmes. Enterprise customers were writing tighter availability clauses into renewals. The status quo was not stable; it was simply the thing that had not broken loudly enough yet.
KriraAI built a production system that ingests every alarm the network produces, collapses correlated symptoms into single fault hypotheses, identifies the probable root cause, and delivers a ranked diagnosis with a recommended action to the engineer within seconds. It replaced the 3,700 handwritten correlation rules entirely. It augments the engineer rather than removing them, because the final decision on a live network stays with a human. We explore further how AI agents are reshaping enterprise operations.
The system has three cooperating models rather than one. A graph neural network reasons over live network topology to determine which alarms are causally related. A transformer-based retrieval pipeline searches nine years of resolution history to propose what fixed this class of fault before. A gradient-boosted survival model estimates time to failure on degrading elements so that some faults never become outages at all.
The flow from raw alarm to actionable diagnosis follows a fixed path:
Alarms arrive from element management systems over SNMP traps, Kafka Connect source connectors, and a legacy syslog bridge, then land on a partitioned ingestion topic.
A Flink job normalises each event into a canonical schema, resolves the emitting entity against the topology graph, and stamps it with a stable element identifier.
The topology service builds a subgraph of the affected region, typically two hops in every direction from the emitting node, and attaches the recent state.
The graph neural network scores every alarm pair for causal relatedness and clusters the storm into a small number of candidate fault groups.
The retrieval pipeline embeds the fault group signature, searches the historical resolution index, and passes the top matches to a fine-tuned language model for synthesis.
The model returns a ranked root cause hypothesis with a confidence score, supporting evidence, and a recommended first action.
The diagnosis is written to the ticketing system, pushed to the operations dashboard, and, above a confidence threshold, triggers an automated remediation runbook.
The clustering step is where the operational leverage lives. An alarm storm of 11,400 events now presents to the engineer as two or three fault hypotheses. The engineer reads a paragraph instead of a scroll. That single change is what moved the diagnosis time, and it is the core of AI-powered root cause analysis in this deployment.
The language model does not invent explanations. It is constrained to synthesise from retrieved historical evidence and the current graph state, and every claim in its output carries a citation back to a ticket identifier or an alarm. When retrieval returns nothing sufficiently similar, the system says so and hands the engineer the raw clustered evidence instead. KriraAI designed that refusal path deliberately because a confident wrong diagnosis on a live network is worse than no diagnosis.
The architecture was designed around one constraint that shaped everything else. Diagnosis must arrive before a human finishes reading the first alarm, which we translated into a tight budget of one second at the ninety-fifth percentile. Every layer below was chosen to fit inside that budget while remaining auditable.
Ingestion runs on Apache Kafka with 48 partitions on the raw alarm topic, keyed by network region to preserve ordering where causality matters. Element management systems that speak SNMP terminate on a collector fleet that translates traps into Avro-encoded events. Legacy syslog sources pass through a Vector agent that handles the parsing horrors of vendor-specific formats.
Topology changes arrive through change data capture from the inventory database using Debezium. This matters more than it sounds. A stale topology graph makes the graph neural network confidently wrong, so we treat inventory mutations as first-class streaming events rather than nightly batch loads. Median lag from an inventory commit to ga raph update is 4.1 seconds.
Stream transformation runs on Apache Flink. The jobs perform schema normalisation across seven vendor dialects, entity resolution against the canonical element registry, temporal feature engineering over sliding windows, and embedding generation at ingestion time so that retrieval never pays that cost during an incident. Batch pipelines for training data assembly and nightly reconciliation are orchestrated as Dagster DAGs, chosen over Airflow because the asset-based model matched how we wanted to reason about feature lineage.
Features are served from Feast with Redis as the online store and Iceberg tables on S3 as the offline store. The dual-path design guarantees that training and inference read identical feature definitions. Training and serving skew was the failure mode we most wanted to eliminate, because it is the one that hides until production.
The correlation model is a relational graph attention network with three message-passing layers, operating over a heterogeneous graph of 22,000 nodes and roughly 61,000 typed edges. Node types include physical elements, logical ports, services, and customers. Edge types encode containment, connectivity, and service dependency, which lets attention weights learn that a transport failure propagates differently from a radio failure.
Training used contrastive learning on labelled historical incidents. Alarm pairs that shared a confirmed root cause were treated as positives, and pairs from concurrent unrelated faults as negatives. This mattered because concurrent unrelated faults are common on a network of this size, and a naive temporal proximity heuristic learns to merge them. Distributed training runs across an eight-GPU A100 node using PyTorch Geometric with DDP.
The diagnosis model is a Llama family model with eight billion parameters, subjected to supervised fine-tuning on 340,000 curated resolution notes. We then applied direct preference optimisation using 4,200 pairwise judgements collected from senior operations engineers. Those engineers rated candidate diagnoses for usefulness, not eloquence, and that preference signal is what taught the model to lead with the actionable finding.
Retrieval runs against Qdrant with HNSW indexing, configured with an m value of 32 and an ef construction of 256. Embeddings come from a bi-encoder fine-tuned on domain text because general-purpose embeddings could not distinguish a card failure from a card reseat in vector space. Serving uses vLLM with AWQ four-bit quantisation, which cuts the memory footprint enough to run the model on existing hardware while holding p95 generation latency at 380 milliseconds.
The operator ran a hybrid estate that had accumulated over two decades, the kind of complex environment KriraAI's telecommunications solutions are built to modernize. It emits a probability of failure within a seventy-two-hour horizon. That output feeds maintenance scheduling rather than the incident path, which keeps the two loops cleanly separated.
Integration was the layer that decided whether any of this reached a human. Outputs reach downstream systems over three distinct contracts, each chosen for a reason:
A gRPC service exposes correlation and diagnosis to internal consumers, chosen for low latency and strict Protobuf schema enforcement across service boundaries.
A versioned REST API serves the ticketing system and third-party integrations, with contract tests running in CI against recorded fixtures to catch breaking changes before release.
A Kafka outbound topic publishes every diagnosis as an immutable event, which lets new consumers subscribe later without touching the producing service.
Webhook triggers fire automated remediation runbooks in the existing orchestration platform when confidence exceeds the configured threshold and the action sits on the approved list.
The event-driven design was deliberate. The operator's ticketing system has scheduled maintenance windows, and a synchronous dependency would have made diagnosis unavailable whenever tickets were down. Diagnoses queue and drain instead, so an outage in a peripheral system never blinds the operations centre.
We instrumented the system on the assumption that it would degrade silently. Data drift on input feature distributions is tracked with a population stability index computed daily, with alerts at 0.15 and hard alerts at 0.25. Embedding distribution shift is tracked separately using KL divergence against a reference window, because new vendor equipment changes the text distribution before it changes numeric features.
Model performance is evaluated continuously against a held-out set of 2,000 labelled incidents refreshed quarterly. When correlation F1 drops below 0.88 or diagnosis top one accuracy drops below 0.85, an automated retraining pipeline triggers, and the candidate model enters shadow evaluation. Promotion to production requires the candidate to beat the incumbent on the frozen set and to survive seven days of shadow traffic.
Operational telemetry runs through OpenTelemetry into Prometheus and Grafana, with tracing in Tempo. We track latency at p50, p95, and p99 for every stage separately, because an end-to-end number hides which stage regressed. Evidently, AI handles the drift reporting surface, and MLflow holds the model registry and experiment lineage.
The entire system runs inside a private VPC with no public endpoints and no internet egress from inference nodes. Service-to-service authentication uses mutual TLS with certificates rotated through Vault. Model inputs and outputs are encrypted in transit and at rest, with KMS-backed keys held under customer control.
Access control is role-based with attribute-level masking applied at the query layer. Customer identifying attributes on the topology graph are masked from engineers whose role does not require them, and unmasking is itself an audited event. Every diagnosis, every human override, and every automated remediation is written to an append-only audit store, which was a direct requirement of the operator's regulatory obligations around service continuity reporting and lawful interception boundaries.
Data residency was enforced per market. Two of the operator's markets prohibit subscriber-adjacent data from leaving national borders, so training and inference are regionally partitioned with no cross-border feature sharing. KriraAI treats compliance as an architectural constraint at design time, not a document produced at the end.
Engineers work in a React-based operations console that presents fault hypotheses as cards rather than alarm rows. Each card shows the root cause statement, a confidence score, the collapsed alarm count, the affected service and customer footprint, and the supporting evidence with links to source tickets. The topology subgraph renders inline, so the engineer can see the blast radius without opening another tool.
Every card carries an accept, reject, or amend control. Those judgements flow back into the preference dataset, which means the system improves from the same work engineers already do. Adoption was designed into the interface rather than mandated, and KriraAI insisted on that feedback loop before any code was written.
Every choice below was made against the operator's existing environment, not against a greenfield ideal. They already ran Kafka at scale, already had Kubernetes, and already had an on-premises GPU estate they had paid for. Fighting those facts would have cost months and bought nothing.
Streaming and pipeline. Kafka was already the operator's event backbone, so the adoption cost was near zero. Flink was chosen over Kafka Streams because we needed genuine event time semantics with watermarks to handle collectors that lag by minutes during storms. Dagster beat Airflow on asset lineage, which mattered because auditors ask which data produced which model.
Modelling and serving. PyTorch Geometric was the only mature option for heterogeneous graphs at this scale with the attention variants we needed. An open-weight language model was mandatory because subscriber-adjacent data could not leave the operator's network under any commercial API arrangement. vLLM with AWQ quantisation was chosen over TensorRT because iteration speed mattered more than the last few percent of throughput on hardware where we already had headroom.
Storage and retrieval. Qdrant was selected over pgvector because filtered vector search across region and vendor attributes was a core query pattern, and HNSW recall held above 0.97 at the latency we needed. Iceberg on S3 gave us time travel for training set reproduction, which is not optional when a regulator asks how a decision was made. Redis served the online feature path because the millisecond budget left no room for anything else.
Platform. Everything runs on Kubernetes with KServe handling model serving and autoscaling, which lets the platform team operate AI services with the tooling they already know. That last point drove more of the stack than any benchmark did. A system that the client cannot operate after handover is a system that fails in month four.
Delivery ran fourteen weeks from the first workshop to production go-live, followed by six weeks of hypercare. KriraAI staffed the engagement with two ML engineers, one data engineer, one MLOps engineer, and an architect, working alongside four people from the operator. Joint staffing was a condition of the engagement because handover to a team that watched from outside did not work.
Weeks one to two, discovery. We sat in the network operations centre through two night shifts. That produced more useful requirements than the workshops did, because engineers describe their workflow differently from how they perform it. We left with a labelled sample of 600 incidents and a clear picture of which alarms engineers actually ignored.
Weeks three to four, architecture and data foundation. We froze the canonical event schema, stood up the Flink normalisation jobs, and built the topology graph service. The latency budget was set here and treated as a contract for the rest of the project.
Weeks five to nine: model development. Correlation and diagnosis models were built in parallel against the historical archive. Offline evaluation ran against the labelled incident set daily.
Weeks ten to twelve, shadow mode. The system consumed live traffic and produced diagnoses that nobody saw except us. We compared machine diagnosis against engineer resolution on every incident for three weeks.
Weeks thirteen to fourteen, phased go-live and handover. One region first, then national, with runbooks and joint on-call.
The topology data was wrong more often than anyone believed. Roughly 9 percent of inventory records disagreed with what the network actually reported. The graph neural network inherited every one of those errors and produced confident nonsense in those regions. We solved it by building a reconciliation job that cross-checks inventory against live link state discovery and quarantines conflicting nodes until a human resolves them.
Resolution notes were far dirtier than the sample suggested. Engineers had been pasting entire command outputs into free text fields for years. Naive fine-tuning taught the model to emit command dumps instead of diagnoses. We rebuilt the corpus with an extraction pass that separated narrative reasoning from tool output, which cut usable examples to 190,000 but raised diagnosis quality sharply.
Shadow mode exposed a failure we had not modelled. During genuine multi-fault events, the correlation model merged two independent faults into one cluster about 12 percent of the time. The fix was harder negative mining, specifically training on concurrent unrelated incidents drawn from the same historical windows. That single change took merge errors to 3.1 percent.
The last challenge was trust rather than code. Engineers rejected early diagnoses even when they were correct, because the output offered no evidence. We rebuilt the interface around citations and confidence, and acceptance rose from 34 percent to 79 percent in four weeks without changing the model at all.
The numbers below were measured across the six months following national go-live, compared against the twelve months preceding the engagement. Measurement was run by the operator's own performance team, not by KriraAI, using their existing reporting definitions. That independence was requested by us.
Mean time to repair fell from four hours and twelve minutes to seventy-one minutes, a 72 percent reduction, with diagnosis time falling from an average of 148 minutes to 9 minutes.
Alarm volume presented to engineers dropped by 94 percent after correlation, turning an average storm of 1,100 events into fewer than 70 reviewable items.
Root cause accuracy, measured as agreement with the engineer's final confirmed cause, reached 91.3 percent on top one and 97.6 percent within the top three hypotheses.
Unnecessary field dispatches fell by 38 percent, which alone accounted for the largest single line of cost saving in the operator's own model.
Service-impacting incidents caught before customer impact rose to 23 percent of the total, driven entirely by the survival model flagging degradation in advance.
Network operations cost per managed element fell 19 percent, with zero headcount reduction, because two engineers moved off rule maintenance and four moved onto proactive work.
Regulatory service level penalties in the two affected markets went to zero across two consecutive reporting quarters.
The result the operator's leadership talks about is not on that list. Overnight escalations to senior engineers fell by 61 percent, which means people sleep. That is what network downtime reduction with AI looks like from inside the building.
The architecture scales along the axis that matters, which is elements rather than alarms. Kafka partitions and Flink parallelism absorb alarm volume growth linearly, and we load-tested to 6.2 million events per day against a current peak of 1.9 million. Graph inference cost grows with subgraph size, not total graph size, because message passing is bounded to two hops.
New use cases attach to the existing foundation without rebuilding it. The canonical event stream, the topology graph service, the feature store, and the retrieval index are the shared substrate. Capacity forecasting, energy optimisation across radio sites, and customer impact prediction each require a new model and a new consumer, not a new platform. The operator has already shipped the first of those internally, which was the point of the handover discipline.
The roadmap for the next two to three years runs in three stages. Year one widens automated remediation from the current 14 percent of incidents to a target of 40 percent by expanding the approved action list as trust accrues. Year two extends the graph to include customer experience signals so that diagnosis ranks by business impact rather than technical severity. Year three targets closed-loop assurance in the virtualised core, where remediation and orchestration are the same system.
For other operators, the transferable lesson is not the model choice. It is that topology is the asset, and any AI implementation for telecom operators that treats alarms as independent text will plateau early. Invest in the graph, keep it current through change data capture, and the modelling problem becomes tractable.
Three insights carry beyond this engagement. Technically, the topology graph was the asset that made everything else possible, and treating inventory changes as streaming events rather than nightly batches was the decision that kept the models honest. Operationally, trust was an interface problem rather than a model problem, and acceptance moved from 34 percent to 79 percent by showing evidence rather than improving accuracy. Strategically, the platform mattered more than the first use case, because the shared substrate now carries models the operator built without us.
KriraAI brings this pattern to every engagement, which is production-first design, joint staffing through delivery, and measurement run by the client rather than by us. We build systems that survive contact with real traffic, real compliance regimes, and real teams who have to operate them after we leave. That discipline is why AI in telecom network operations moved from a slide to a system here in fourteen weeks.
If you are running an operations function that is drowning in signals and starved of answers, bring us the problem. KriraAI will tell you honestly whether AI is the right instrument before we quote you a line of work.
AI reduces mean time to repair primarily by compressing diagnosis, not repair. In this deployment, correlation models collapsed alarm storms averaging 1,100 events into fewer than 70 reviewable items, a 94 percent reduction in what an engineer reads. A retrieval-augmented language model then proposed a ranked root cause with supporting evidence from nine years of historical resolutions. Average diagnosis time fell from 148 minutes to 9 minutes, and overall mean time to repair dropped 72 percent, from four hours and twelve minutes to seventy-one minutes across six months of measurement.
Graph neural networks for telecom correlation work because network faults propagate along topology, and topology is a graph. Handwritten rules encode a snapshot of that topology and decay as the network changes, which is why the operator's 3,700 rules had become unreliable. A relational graph attention network reasons over live topology with 22,000 nodes and 61,000 typed edges, learning how failures propagate differently across transport, IP, and radio layers. It generalises to equipment for which it has not seen rules written, and it updates as the graph updates rather than requiring manual maintenance.
Three datasets carry most of the value. First, an accurate and current network topology, including containment, connectivity, and service dependency, because this is the reasoning substrate,e and 9 percent of records were wrong at the engagement start. Second, historical alarm streams with sufficient retention to cover seasonal and rare fault modes. Third, closed ticket resolution notes, which encode diagnostic reasoning that exists nowhere else. Data quality matters more than volume, and reconciliation between inventory and live network state should be treated as an engineering deliverable, not a prerequisite assumption.
Yes, and for most operators, it must. This system runs entirely inside a private VPC with no public endpoints and no internet egress from inference nodes, because subscriber-adjacent data could not leave the operator's network under regulation in two markets. An open-weight language model with eight billion parameters was fine-tuned in place and served with vLLM using four-bit quantisation, holding p95 generation latency at 380 milliseconds on existing GPU hardware. Data residency was enforced per market with regionally partitioned training and no cross-border feature sharing.
This engagement ran fourteen weeks from the first workshop to phased production go-live, followed by six weeks of hypercare and joint on-call. That timeline was achievable because the operator already ran Kafka and Kubernetes at scale, and because delivery was jointly staffed rather than handed over at the end. Three weeks of shadow mode against live traffic preceded any engineer seeing output, which is where the multi-fault merge error was caught. Timelines extend when topology data requires reconciliation first, which is the most common cause of slippage.
Founder & CEO
Divyang Mandani is the CEO of KriraAI, driving innovative AI and IT solutions with a focus on transformative technology, ethical AI, and impactful digital strategies for businesses worldwide.
