How to Build a Customer Health Score with Machine Learning
Move beyond gut-feel green/yellow/red dashboards. This guide walks through the engineering and methodology behind an ML-driven customer health score — from raw signals to production integration.
Most customer health scores are built by hand: a CS leader assigns weights to a handful of metrics, loads them into a spreadsheet, and calls it a model. The result is a score that reflects assumptions rather than evidence — and it degrades quietly as your product and customer base evolve.
A machine learning approach replaces static assumptions with a model trained on what actually predicts retention, expansion, and churn in your data. The build takes more upfront effort, but the output is a score that updates itself and improves over time.
Collect the Right Input Signals
Your model is only as good as its inputs. Gather signals across four categories — the more granular the event data, the better the feature engineering downstream.
Product Usage
Behavioral signals are the strongest predictors of retention. Capture at the event level, not just session counts.
- Daily / weekly active users (DAU/WAU)
- Core feature adoption rate
- Breadth of features used
- Time-to-value from onboarding
- Frequency & recency trends (RFM)
Support Tickets
Support data reveals friction and frustration before a customer ever tells their CSM.
- Open ticket count & age
- Ticket severity distribution
- First-response & resolution time
- Repeat issue rate (same category)
- Sentiment from ticket text (NLP)
NPS & Survey Signals
Survey scores are sparse but high-signal when available. Treat them as a feature, not a substitute for the full model.
- Most recent NPS score & trend
- CSAT per support interaction
- QBR / EBR sentiment rating
- Days since last response
Billing & Contract
Commercial signals flag at-risk accounts even when usage looks healthy on the surface.
- Invoice payment latency
- Failed payment attempts
- Seats contracted vs. seats active
- Days to renewal
- Expansion / contraction history
Engineer Features That Actually Predict Outcomes
Raw event counts are rarely useful on their own. Feature engineering transforms them into the patterns a model can learn from — and this step accounts for the majority of your model's final accuracy.
- Trend features: rolling 7-, 14-, 30-day change in usage (not just absolute values)
- Recency encoding: days since last login, days since last support ticket, days since last NPS response
- Ratio features: active seats ÷ contracted seats; tickets opened ÷ tickets resolved
- Lifecycle context: account age bucket, cohort, product tier — to avoid comparing new accounts to tenured ones
- Lag features: health score at T-30 and T-60 days as inputs, giving the model memory
- NLP features: sentiment score from ticket text, keyword flags ("cancel", "frustrated", "competitor")
Let the Model Determine Feature Weights
The biggest advantage of ML over a rule-based score is this: weights emerge from data, not opinion. Here's how different model families handle that, and where each fits.
| Model Type | How It Weights Features | Best For | Watch Out For |
|---|---|---|---|
| Logistic Regression | Explicit coefficients per feature — fully interpretable | First production model; CS team buy-in | Misses non-linear interactions |
| Random Forest | Feature importance scores from many decision trees | Tabular data with mixed feature types | Can overfit on small datasets (<500 accounts) |
| Gradient Boosting (XGBoost / LightGBM) | Iterative residual learning; SHAP for explanation | High accuracy on structured CS data | Requires careful tuning and more data |
| Survival Analysis (Cox PH) | Time-to-event coefficients | Predicting churn timing, not just churn risk | Less familiar to most CS teams |
Tip: Start with logistic regression to establish a baseline and build stakeholder trust. Graduate to gradient boosting once you have 1,000+ labeled renewal events.
Train, Validate, and Score
-
Label your historical data Define your target variable: churned, renewed, or expanded at renewal. Pull 12–24 months of account history with outcomes.
-
Split by time, not randomly Use a temporal train/test split — train on older cohorts, validate on recent ones. Random splits leak future data into training and inflate metrics.
-
Handle class imbalance Churn is a rare event in healthy books. Use SMOTE, class weights, or focal loss to stop the model from predicting "healthy" for everyone.
-
Evaluate on AUC-ROC and precision-recall Accuracy is misleading with imbalanced data. Optimize for the recall/precision tradeoff that matches your CS team's intervention capacity.
-
Calibrate probabilities to a score Convert raw model output (0–1 probability) to a 0–100 score using Platt scaling or isotonic regression so it reads naturally in your CS platform.
Integrate the Score with Your CS Platform
A model that only lives in a notebook doesn't drive action. The final step is surfacing the score where your CSMs actually work — and triggering the right plays automatically.
Score via CTAs & Scorecards
Write your ML score to a custom Company attribute via REST API. Feed it into Gainsight's Health Scorecard and trigger CTAs when the score drops below a configured threshold.
Custom Health Score Field
Push the score via ChurnZero's Account API to a custom field. Use Journey automation to assign a CSM task or playbook step when the score enters a risk tier.
Attribute Sync via Segment API
Post the score to a Totango custom attribute on each SuccessBloc. Combine with touchpoint data in Health Profiles to give CSMs full context in one view.
Salesforce / HubSpot Object
Write the score to a custom Account field in your CRM. Surface it on the Account page layout and create workflow rules that alert AEs on risk signals near renewal.
Common Pitfalls — and How to Avoid Them
Most ML health score projects stall or produce poor results for the same handful of reasons. Recognize them early.
Frequently Asked Questions
How much data do I need to build a customer health score with machine learning?
Which model should I start with — gradient boosting or logistic regression?
How often should a machine learning health score model be retrained?
How do I explain the health score to CSMs who aren't data scientists?
Can I build this without a dedicated data science team?
Ready to Build a Health Score That Actually Predicts Churn?
StratApps combines Customer Success expertise with applied data science to design and deploy ML-driven health scoring tailored to your product and customer base.
Explore Our Customer Success Solutions No commitment required — let’s start with a conversation.
