AI-Powered Predictive Maintenance — Case Study

Given a live snapshot of a machine's operating conditions — temperatures, speed, torque, tool wear — should a maintenance team pull it for inspection now, or is it safe to keep running? That's the decision this project builds toward: not a black-box stop/go, but a transparent, honestly-benchmarked early-warning signal, deliberately tuned to catch real failures before they happen even at the cost of a few extra inspections.

The data

Source: the AI4I 2020 Predictive Maintenance dataset — 10,000 machine records, each with a product quality Type (L / M / H) and five process readings: air temperature, process temperature, rotational speed, torque, and tool wear. The target, Machine failure, is a binary flag. It is a deliberately imbalanced problem: only 3.39% of machines fail (339 of 10,000) — which is exactly why headline accuracy is a trap here (a "never fails" model scores 96.6%) and why the whole project is built around recall and threshold choice instead. The dataset is synthetic, so this is a validation of the modelling approach, not a production maintenance controller — a distinction kept explicit rather than glossed over.

Target leakage, removed by design: the raw file ships five failure-mode flags — TWF, HDF, PWF, OSF, RNF — whose logical OR is the definition of the Machine failure target. Feeding any of them to the model would be near-deterministic leakage dressed up as skill, so all five were excluded from every model from the start. They are reserved for a possible future root-cause / multi-label study, never as inputs to the failure classifier.

Two controlled experiments, one clear answer

Following the same one-variable-at-a-time discipline as the Workforce Income and AEGIS Orbital Watch projects, feature engineering was tested as a controlled experiment. Experiment A (Baseline): the six raw inputs only. Experiment B (Engineered): the same six plus two derived features — Temperature_Difference (process minus air temperature, the thermal gradient the machine is working against) and Power (a speed–temperature load-interaction term). The result was the opposite of Project 6, where the raw set had won: here the two engineered features improved F1 for seven of the eight models — dramatically for the tree ensembles — and reshuffled the leaderboard, lifting Random Forest from 4th in Experiment A to champion in Experiment B.

Model	ROC-AUC (A)	ROC-AUC (B)	F1 (A)	F1 (B)
Random Forest	0.9668	0.9766	0.731	0.841
LightGBM	0.9748	0.9758	0.769	0.836
Extra Trees	0.9735	0.9774	0.582	0.816
XGBoost	0.9716	0.9755	0.769	0.807
HistGradientBoosting	0.9711	0.9651	0.739	0.794
CatBoost	0.9699	0.9746	0.707	0.780
Support Vector Machine	0.9468	0.9719	0.542	0.700
Logistic Regression	0.8994	0.8998	0.177	0.175

The eight-model benchmark (Experiment B)

Every model was tuned with a genuine hyperparameter search inside the same reusable AI Model Execution Framework used across Projects 5 and 6 — not fixed defaults. Ranked at the default 0.50 threshold on a held-out test set:

Rank	Model	Accuracy	Precision	Recall	F1	ROC-AUC
1	Random Forest	0.990	0.914	0.779	0.841	0.9766
2	LightGBM	0.990	0.944	0.750	0.836	0.9758
3	Extra Trees	0.989	0.895	0.750	0.816	0.9774
4	XGBoost	0.989	0.941	0.706	0.807	0.9755
5	HistGradientBoosting	0.987	0.862	0.735	0.794	0.9651
6	CatBoost	0.987	0.873	0.706	0.780	0.9746
7	Support Vector Machine	0.982	0.808	0.618	0.700	0.9719
8	Logistic Regression	0.967	0.583	0.103	0.175	0.8998

The champion: Random Forest, tuned for advance warning

Random Forest led the field on F1 and MCC while staying within a whisker of the best ROC-AUC. But the default 0.50 cutoff is the wrong operating point for an early-warning tool on a 3.4%-failure target — it under-calls real failures. Scanning the threshold, dropping it to 0.38 pushes recall past the 80% line while precision barely moves — the deliberate product choice for a system where a missed failure costs far more than an extra inspection.

0.9766

Test ROC-AUC

90.2%

Precision @ threshold

80.9%

Recall @ advance-warning threshold

Champion configuration: Random Forest · n_estimators=266 · max_depth=10 · criterion=entropy · min_samples_split=2 · min_samples_leaf=1 · bootstrap=True · operating threshold ≈ 0.38 (lowered from the default 0.50 to cross 80% recall: recall 80.9%, precision 90.2%, F1 0.853), evaluated on a held-out test set.

The application: a technician-facing early-warning tool

The trained model is served as a small Flask app — not a notebook, an actual working tool. A technician enters the six machine readings (the form pre-fills with typical "normal operation" values so a busy operator nudges from a sane baseline rather than typing six numbers cold), and the app computes the two engineered features server-side and returns a probability.

Crucially, the output is not a bare 0/1. It maps the probability onto a three-band, human-actionable verdict anchored on the model's tuned threshold — RED "schedule inspection", AMBER "monitor closely", GREEN "routine" — so a reading just under the alarm line reads differently from one nowhere near it. A JSON /api/predict endpoint and a /api/health check are included for programmatic use. The system surfaces the risk; the person decides.

Engineering rigor: honest choices on an imbalanced target

On a 3.4%-failure dataset the easy path is to report 99% accuracy and stop. This project does the opposite at every turn.

Leakage removed before modelling, not after: the five failure-mode flags that define the target were excluded up front — the single most important integrity decision on this dataset, and the difference between a real 0.98 ROC-AUC and a meaningless 1.0.

Accuracy deliberately ignored as the headline: selection was driven by F1, MCC, and a recall-first operating point, because on a 3.4% base rate accuracy rewards a model that never predicts a failure. The threshold was moved off the default 0.50 for exactly this reason.

The failure was reported, not hidden: Logistic Regression managed only 10.3% recall (F1 0.175) — it simply could not model this target. It stays in the published leaderboard in last place rather than being quietly dropped, because an honest benchmark includes the models that lost.

The shipped model was verified byte-for-byte: after packaging, the deployed random_forest_model.pkl was re-checked by SHA-256 against the trained artifact to confirm no corruption between training and deployment.

AI-Powered Predictive Maintenance Decision Support System

The data

Two controlled experiments, one clear answer

The eight-model benchmark (Experiment B)

The champion: Random Forest, tuned for advance warning

The application: a technician-facing early-warning tool

Try the Predictive Maintenance app

Engineering rigor: honest choices on an imbalanced target