How We Built an AI Student Retention Platform: Case Study

Most online learning platforms lose more learners than they keep. Our client, a leading EdTech enterprise, watched this happen every single term. Only 41 percent of enrolled learners finished the courses they paid for. The remaining 59 percent disengaged quietly, week by week, until they were gone. By the time a human advisor noticed, the learner had already left.

This was the operational reality that brought us in. KriraAI is an AI solutions company that builds production grade systems for enterprises rather than pilots. The client engaged us to design an AI student retention platform that could see disengagement coming and act before a learner churned. It had to run at the scale of hundreds of thousands of active learners. It had to integrate with the systems the client already operated in production.

This case study walks through the full engagement. We cover the problem, the system we built, the architecture, the stack, the delivery journey, and the measured results. Everything below describes a hardened production system, not a proof of concept.

The Problem KriraAI Was Called In To Solve

The client ran a large self paced learning platform with roughly 220,000 active learners at any given time. Retention was the single largest threat to the business. Every learner who dropped out represented lost revenue and a lost reference. The economics of the business depended on completion, and completion was failing.

The core issue was timing. The data needed to predict disengagement already existed inside the platform. Every login, video watch, quiz attempt, forum post, and assignment submission was logged. The platform generated millions of these events per day. None of that signal was being used to intervene in real time.

Academic advisors worked from static weekly reports instead. Those reports were already stale when they landed. An advisor would see that a learner had missed two assignments after the window to help had closed. The team was reacting to history rather than reading the present. This reactive posture made meaningful intervention almost impossible.

The manual workflow also did not scale. Each advisor could realistically track a few hundred learners with any care. The platform had hundreds of thousands. So advisors triaged by gut feel and by whoever emailed them first. The learners most at risk were usually the ones who stopped reaching out entirely.

Support volume compounded the problem. Learners who felt stuck submitted tickets asking the same routine questions repeatedly. They asked where the next module was, why a quiz had not saved, and how to reset a path.

Human agents answered these one by one. The queue grew faster than the team could clear it. Every hour an agent spent on a routine question was an hour not spent helping a learner in real trouble.

The cost of all this was concrete and growing. Refund requests rose every term as frustrated learners gave up. Acquisition spend was effectively wasted on learners who never finished. Each percentage point of lost completion translated directly into lost recurring revenue.

The competitive pressure was sharp because newer platforms already marketed adaptive learning experiences. The client knew the status quo was not survivable. They needed a system that could predict, personalise, and act at machine speed.

What KriraAI Built

KriraAI built an AI student retention platform that unifies prediction, personalisation, and intervention in one production system. The platform ingests every learner event in near real time. It scores each learner continuously for disengagement risk. It then routes the right action to the right place, automatically.

At its core the platform combines three AI subsystems that share a common feature layer. The first is a student dropout prediction model that reads each learner as a sequence of events over time. The second is a retrieval grounded AI tutor that answers learner questions directly. The third is an adaptive recommendation engine that reshapes each learner path based on observed mastery.

The student dropout prediction model is a Temporal Fusion Transformer trained on learner event sequences. We chose a sequence aware architecture deliberately. Dropout is not a static property of a learner. It is a trajectory that bends over days and weeks. A model that reads order and timing detects that bend far earlier than a flat tabular classifier. The model emits a calibrated risk score and the top features driving that score.

The AI tutor is a retrieval grounded dialogue system rather than a raw chatbot. It runs an open weight large language model served with vLLM. Every answer is grounded in the client course catalogue and help content through a retrieval step. This grounding keeps answers accurate and tied to the learner actual course. The tutor handles the routine question volume that previously drowned human agents.

The adaptive recommendation engine treats learner mastery as a graph problem. It models relationships between learners, concepts, and content. When a learner struggles with a concept, the engine surfaces the prerequisite content that closes the gap. This keeps learners moving instead of stalling on a wall they cannot climb. The same engine also reorders upcoming modules so the path always reflects current mastery, which removes the manual replanning advisors once did by hand.

These subsystems do not operate in isolation. They feed a single intervention orchestration layer. When the dropout model flags a learner, the orchestration layer decides the response. A low risk nudge might be an automated email, while a high risk signal escalates to a human advisor with full context attached. A confused learner gets the tutor and a path adjustment together. The platform replaced guesswork with a continuous, evidence driven loop, and KriraAI designed every part of it to run without a human babysitting it.

Inside the AI Student Retention Platform Architecture

The AI student retention platform runs as six cooperating layers. Each layer owns one responsibility and communicates through explicit contracts rather than shared state. This separation let us scale, monitor, and replace components independently. The sections below cover each layer and its engineering rationale.

Data Ingestion and Pipeline Layer

The ingestion layer captures learner activity through two paths. Operational state arrives through change data capture, with Debezium streaming row level changes from the client PostgreSQL instances into Apache Kafka. This delivered enrolment, progress, and billing state without polling the production database.

High volume behavioural events flow through a separate streaming path. Clickstream and learning events publish directly to Kafka topics from the application. Apache Flink consumes them statefully and computes engagement windows, session boundaries, and sequence features within seconds of each event.

Batch enrichment runs alongside the streaming path. Nightly extracts from the client CRM and student information system land in an S3 data lake, orchestrated by Dagster DAGs with typed assets and lineage. We chose Dagster over Airflow for its asset model and testability. Schema normalisation and entity resolution resolve one learner to one identity. All features land in a Feast feature store whose matched online and offline paths remove training and serving skew.

The AI and Machine Learning Core

The machine learning core hosts three models. The student dropout prediction model is a Temporal Fusion Transformer in PyTorch, trained with supervised fine tuning on two years of labelled outcomes across a multi GPU cluster coordinated with Ray. It emits a calibrated risk score with per feature attributions so advisors trust the signal.

The AI tutor pairs retrieval with generation. We fine tuned a sentence embedding model using contrastive learning, indexed those embeddings into Qdrant with HNSW, and served an open weight large language model with vLLM. LangGraph drives the dialogue as an explicit state machine that escalates to a human below a confidence threshold. A graph neural network over the learner concept content graph scores the next best content by predicted mastery gain.

Integration Layer

The integration layer wires AI outputs to the systems that act on them. Internal services talk over gRPC, while external systems consume versioned REST and GraphQL contracts. Intervention triggers flow through an event driven design on Kafka, where the orchestrator emits events that consumers turn into emails, in app nudges, advisor queues, and path updates. Webhook subscriptions let existing client tools react in real time.

Monitoring and Observability

The monitoring layer treats model health as a production concern. We track per feature drift using population stability index and KL divergence, and compare live performance against held out sets on a schedule with Evidently. Operational telemetry flows through OpenTelemetry into Prometheus and Grafana, with latency tracked at p50, p95, and p99. Automated retraining triggers fire when quality crosses a degradation threshold, which keeps the student dropout prediction model accurate as behaviour shifts.

Security and Compliance

Education data carries strict obligations, so security was designed in from the start. Access uses role based access control with attribute level masking, so an advisor sees only their cohort and sensitive fields stay hidden. Model inputs and outputs are encrypted in transit and at rest, and the platform runs inside a private VPC with no public endpoints. The design aligns with FERPA obligations and writes every access and automated decision to an immutable append only audit log.

Educator and Learner Delivery Layer

The platform reaches people through two surfaces on a shared API. Advisors work from a React dashboard that ranks learners by risk, explains each score, and queues high risk cases with drafted actions. Learners meet an in app tutor and adaptive path widget that answers in context and reshapes the path when needed. Both surfaces share one GraphQL contract so behaviour stays consistent across the experience.

The Technology Stack and Why We Chose It

Every technology in this stack was selected against the client environment and scale. The client already ran on AWS, so we built natively there to avoid friction. We did not introduce a second cloud, because operational simplicity mattered more than novelty. The choices below each carry a clear engineering rationale.

Apache Kafka anchored the data backbone because the event volume was high and bursty. Kafka gave us durable replayable streams that a queue alone could not. We paired it with Apache Flink rather than batch Spark because retention needs fresh features. Flink computes sequence features in seconds, and seconds were the difference between catching and missing a learner.

We chose Dagster over Airflow for orchestration because of its asset and lineage model. Lineage mattered for an education system that needed defensible data provenance. Feast handled the feature store because training and serving skew had to be eliminated by design. A shared feature definition was safer than two parallel implementations drifting apart.

On the model side we used PyTorch for full control of the Temporal Fusion Transformer. We served the language model with vLLM because its paged attention raised throughput dramatically. Qdrant won the vector database choice for its HNSW performance and operational clarity. We picked an open weight large language model for cost control and data privacy. Running the model inside the client VPC kept learner data out of third party hands.

For observability we standardised on Evidently, Prometheus, Grafana, and OpenTelemetry. These are proven tools the client platform team could operate after handover. We deliberately avoided exotic monitoring stacks that would create dependence on KriraAI. The goal was a system the client could run, not one they would be locked into.

How We Delivered It, The Implementation Journey

KriraAI delivered the platform over a nine month engagement in six phases. We mapped every event source, every downstream system, and every compliance constraint before writing code. We defined what success meant in numbers before building a single model. This shared definition of done kept the project honest, and the phases ran in the following sequence.

1. Discovery and requirements ran for the first six weeks and produced a full data and systems map.

2. Architecture design followed and locked the six layer design and all interface contracts.

3. Development built the pipelines, models, and serving infrastructure in parallel tracks.

4. Testing and validation stress tested the system against historical and synthetic load.

5. Deployment rolled the platform out behind feature flags to a learner cohort first.

6. Handover transferred operational ownership with full documentation and runbooks.

The delivery was not friction free, and the real challenges are worth naming. The first surprise was data quality in the event stream. Event schemas had drifted across multiple platform versions over the years. The same action was logged three different ways depending on the client release. We built a normalisation layer in Flink that reconciled these variants into one canonical schema.

The second challenge was the cold start problem for new learners. A sequence model needs history, and new learners have none. We resolved this with a fallback model that scored early signals from the first sessions. The Temporal Fusion Transformer took over once enough sequence accrued.

The third challenge was a model performance gap in early validation. The first model had high recall but flooded advisors with false positives. We retrained with a cost sensitive objective and recalibrated the decision threshold.

That tuning matched the precision recall balance to real advisor capacity. The fourth challenge was integration with the legacy student information system. Its API was brittle and rate limited, so we moved that path to nightly batch and absorbed the latency where it did not hurt retention.

Validation and handover were treated as engineering work, not paperwork. We shadowed the model against historical cohorts before any learner saw a single nudge. We ran the platform behind feature flags on one cohort and compared outcomes against a holdout. Only after the numbers held did we widen the rollout. Handover included runbooks, on call playbooks, and live training for the client platform team. KriraAI builds for the client to own the system, so we left them able to operate and retrain it without us.

Results the Client Achieved

The results were measured across two full academic terms after go live. Course completion improved from 41 percent to 63 percent in that window. That is a 22 point absolute gain in the metric the business cared about most. The platform did exactly what it was built to do, which was improve course completion rates at scale.

The student dropout prediction model performed strongly in production. It identified 89 percent of eventual dropouts at least three weeks before disengagement. It reached a precision recall AUC of 0.81 on held out learners. Earlier detection gave advisors a real window to act rather than a postmortem.

The AI tutor changed the support economics immediately. It deflected 72 percent of routine learner questions without a human agent. It answered at a p95 latency of 1.8 seconds, which felt instant to learners. Support cost per learner fell by 38 percent within the measurement window.

Advisor productivity rose sharply once triage became automatic. Each advisor supported roughly three times more at risk learners than before. They spent their time on intervention rather than on searching reports. The before state was reactive, manual, and stale. The after state was predictive, automated, and current. These outcomes came from a completed engagement, not a projection.

The financial impact followed directly from the retention gain. The 22 point lift in completion cut churn driven revenue loss sharply. Because acquisition cost now spread across learners who finished, the effective cost per completed learner dropped. The client recovered the full engagement cost within the first two terms of operation.

What This Architecture Makes Possible Next

The platform was built to grow without a rebuild, and that was an intentional design goal. The streaming backbone scales horizontally as learner volume rises. Kafka partitions and Flink parallelism absorb more events by adding capacity, not by re architecting. The feature store and serving layer were sized for several times the current load. Growth is an operations task now, not an engineering project.

New use cases attach to the existing foundation rather than requiring fresh infrastructure. The same feature store and event backbone already support a fourth model with little new plumbing. The client is now extending the platform toward content quality scoring and instructor coaching. Each addition reuses the data, monitoring, and security layers already in place.

The roadmap over the next two to three years builds outward from this base. The client plans automated content generation grounded in the same retrieval layer. They also plan predictive enrolment planning that reuses the same time series forecasting core. Because the architecture separates concerns cleanly, each step adds value without destabilising the whole.

Other companies in education can apply the same pattern to their own situation. The lesson is that the signal usually already exists in the event logs. The work is turning that latent signal into timely action through a disciplined architecture. Any EdTech AI implementation that treats prediction, personalisation, and intervention as one loop can repeat this outcome. The building blocks are proven and available today, and the differentiator is engineering discipline rather than any single model or vendor.

Conclusion

This engagement produced three insights worth carrying forward. The technical insight is that dropout is a trajectory, so a sequence aware model detects it far earlier than a flat classifier. The operational insight is that prediction only matters when it triggers timely, automated intervention an advisor can trust. The strategic insight is that the signal already lived in the client event logs, waiting for the right architecture.

KriraAI brought the same engineering rigour to this project that we bring to every client engagement. We build production systems with real monitoring, security, and handover, not pilots that stall after a demo. The AI student retention platform we delivered lifted course completion from 41 percent to 63 percent and gave the client a foundation they can grow on for years. Every layer was chosen deliberately and built to be operated by the client team. If you are facing a retention, personalisation, or prediction challenge in education, bring it to KriraAI and let us design the system that solves it.