Cracking Retail Fraud: An End-to-End EDA & Machine Learning Project

A full end-to-end data science project: exploratory data analysis and machine learning for real-world fraud detection in retail transactions.

---

Introduction: Why Fraud Detection Matters

Global payment fraud losses exceeded $40 billion in 2023 and are projected to surpass $50 billion by 2027 (Nilson Report). For retailers and e-commerce platforms, fraud is not just a financial hit — it erodes customer trust, increases operational costs, and creates regulatory exposure.

In this project, we work with the Retail Intelligence: Fraud Detection Dataset — a rich, 100,000-row dataset spanning 21 features including transaction behaviour, device context, location signals, and risk indicators. Our goals:

• Understand what patterns separate fraudulent from legitimate transactions
• Identify the most predictive risk signals
• Build and evaluate machine learning models capable of flagging fraud before it causes damage

---

Part 1: The Dataset

Schema Overview

The dataset contains 100,000 retail transactions with the following 21 columns across four categories:

Category	Columns	Description
Transaction Core	transaction_id, customer_id, transaction_timestamp, transaction_amount	Unique identifiers and base transaction details
Payment Context	payment_method, device_type, location, merchant_category, is_international	How, where, and on what the transaction was made
Behavioural Signals	transaction_frequency_24h, failed_attempts, velocity_score, transaction_count_30d, time_since_last_txn_hours	Recent activity patterns that deviate from norms
Risk Flags	is_high_risk_device, unusual_amount_flag, location_change_flag, account_age_days, avg_transaction_amount, anomaly_score	Pre-computed signals indicating elevated risk
Target	fraud_flag	1 = Fraud, 0 = Legitimate (our ML label)

One important characteristic: the dataset is imbalanced — only 9.56% of transactions are fraudulent. This is actually realistic (retail fraud rates typically run 2–15%), and it means we need to be careful about how we evaluate model performance — accuracy alone is misleading.

---

Part 2: Exploratory Data Analysis (EDA)

Figure 1: Overview — Fraud Rate & Numeric Distributions

Key observations:

• Transaction amounts are right-skewed — the majority of transactions cluster under $77, with a long tail extending to ~$1,548. Fraudulent transactions show a slightly different distribution, with more activity in the $50–$300 range.
• Anomaly score separates fraud well — fraudulent transactions have noticeably higher anomaly scores, suggesting this pre-computed feature is informative. However, there is significant overlap, meaning it alone is not sufficient.
• Failed attempts strongly predict fraud — the fraud rate jumps from ~5% for 0 failed attempts to over 60% for 4+ failed attempts. This is one of the strongest signals in the dataset.
• Velocity score — fraudulent transactions cluster at higher velocity scores, consistent with fraud rings making rapid sequential purchases.

Figure 2: Fraud Rates by Categorical Features

Key observations:

• Crypto payments carry the highest fraud rate — their pseudonymous nature makes them preferred by fraudsters. Regulatory gaps compound the risk.
• Mobile devices are riskier than Desktop — mobile transactions show a higher fraud rate, likely because mobile sessions are easier to hijack and harder to authenticate via hardware security keys.
• Travel and Electronics are hotspots — high-value, easily resaleable goods attract more fraud. Grocery and Healthcare show the lowest rates.
• Geographic variation is significant — some countries exhibit materially higher fraud rates. This could reflect differences in payment infrastructure maturity, cybersecurity legislation, or organised crime activity.

Figure 3: Temporal Patterns in Fraud

Key observations:

• Late-night fraud spikes — transactions between midnight and 5am show elevated fraud rates. This is a classic pattern: fraudsters operate during low-oversight hours when customer service teams are offline and victims are asleep.
• Weekend fraud is higher — weekends see more fraudulent activity, possibly because institutional fraud monitoring is reduced on those days.
• Monthly variation — fraud shows some seasonality. Peaks around major retail seasons (holidays, sales events) align with known industry patterns where fraudsters exploit high transaction volumes to hide in the noise.

Figure 4: Feature Correlation Matrix

Key observations:

• anomaly_score correlates with multiple risk flags — as expected, the composite anomaly score captures information from velocity_score, unusual_amount_flag, location_change_flag, and is_high_risk_device. It shouldn’t be used together with these features without regularisation.
• failed_attempts and fraud_flag have a strong positive correlation — confirming what we saw in Fig 1.
• account_age_days is negatively correlated with fraud — newer accounts are riskier. This is consistent with fraud rings creating synthetic identities.
• Features are largely independent — most pairs show low correlation, which is good for ML models as it means each feature adds distinct information.

Figure 5: Risk Factor Deep-Dive

Key observations:

• International transactions carry higher fraud risk — cross-border transactions are harder to verify and more attractive to fraud networks that exploit regulatory gaps between countries.
• New accounts (<1 month) are significantly riskier — the fraud rate for accounts under 30 days old is dramatically higher than for established accounts. Many fraud schemes involve “account farming” — creating fresh identities for a single fraudulent burst.
• Higher-value transactions have higher fraud rates — fraud rates rise with transaction amount, particularly above $200. Fraudsters chase higher-value goods (electronics, luxury items) once they gain access to compromised credentials.

---

Part 3: Machine Learning for Fraud Detection

The Challenge: Class Imbalance

With only 9.56% fraud, a naive model that predicts “legitimate” for every transaction would achieve 90.4% accuracy — yet catch zero fraud. This is why we focus on:

• ROC-AUC — measures the model’s ability to rank fraud above legitimate across all threshold settings
• PR-AUC (Average Precision) — more informative than ROC-AUC under class imbalance; captures precision-recall trade-off
• Fraud Recall — what fraction of actual fraud cases do we catch? Missing fraud is costly.
• Fraud Precision — of those we flag as fraud, how many actually are? False positives cause friction for legitimate customers.

Feature Engineering

Before training, we engineer several additional features:

# Extract temporal signals from timestamp
df["hour"]        = df["transaction_timestamp"].dt.hour
df["day_of_week"] = df["transaction_timestamp"].dt.dayofweek
df["is_weekend"]  = (df["day_of_week"] >= 5).astype(int)
df["is_night"]    = ((df["hour"] < 6) | (df["hour"] >= 22)).astype(int)

# Amount relative to customer's own baseline
df["amount_ratio"] = df["transaction_amount"] / df["avg_transaction_amount"]

The amount_ratio feature is especially important: a $500 transaction from a customer whose average is $50 is far more suspicious than the same transaction from someone who regularly spends $400.

Three Models Compared

We train and evaluate three models on an 80/20 stratified train-test split (80,000 training rows, 20,000 test rows):

Model	ROC-AUC	PR-AUC	Fraud Recall	Fraud Precision	Why it’s included
Logistic Regression	0.9448	0.6423	87.1%	41.0%	Fast, interpretable baseline; reveals linear relationships
Random Forest	0.9650	0.7654	89.9%	51.8%	Ensemble method; handles non-linearity, provides feature importance
✅ Gradient Boosting	0.9667	0.7841	59.5%	79.1%	Best ROC-AUC and PR-AUC; sequential correction of errors

Figure 6: Model Performance Comparison

Reading the ROC curve: All three models dramatically outperform random chance (the dotted diagonal). At a false positive rate of 5% — meaning 5% of legitimate transactions are incorrectly flagged — the Gradient Boosting model correctly identifies over 90% of fraud cases.

The precision-recall trade-off: Gradient Boosting achieves the best PR-AUC (0.784), but notice that it’s more conservative than Random Forest — it achieves higher precision at the cost of lower recall. This reflects its optimisation strategy. In practice, the right balance depends on business context: a bank processing $1B/day values high recall (catch all fraud); a small retailer might prefer high precision (minimise false alarms).

Figure 7: Gradient Boosting — Deep Dive

Confusion Matrix Analysis

On the 20,000-row test set:

• True Positives (fraud correctly caught): ~1,138 transactions
• True Negatives (legitimate correctly cleared): ~17,730 transactions
• False Positives (legitimate flagged as fraud): ~357 — these customers experience friction (declined transaction, manual review)
• False Negatives (fraud that slipped through): ~775 — each represents a real financial loss

Top Features by Importance

The model’s most influential features, in order:

1. failed_attempts — Most predictive feature. Even one failed attempt before success is a strong fraud signal.
2. is_international — Cross-border transactions carry inherently higher risk.
3. amount_ratio — Purchases well above the customer’s historical average stand out.
4. anomaly_score — The composite risk score computed upstream adds value even after including its components.
5. location_change_flag — A purchase from an unusual country or city is a key signal.
6. unusual_amount_flag — Binary flag for amounts far outside the customer’s norm.
7. is_high_risk_device — Transactions from flagged mobile devices.
8. transaction_frequency_24h — High frequency in a short window suggests velocity fraud.
9. velocity_score — Composite speed-of-activity metric.
10. time_since_last_txn_hours — Very rapid successive transactions are suspicious.

---

Part 4: Model Code Walkthrough

Step 1 — Data Loading & Preparation

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder

# Load the dataset
df = pd.read_csv("retail_fraud_detection_100k.csv")
df["transaction_timestamp"] = pd.to_datetime(df["transaction_timestamp"])

# Quick sanity check
print(f"Shape: {{df.shape}}")
print(f"Fraud rate: {{df['fraud_flag'].mean():.2%}}")
# → Shape: (100000, 21)
# → Fraud rate: 9.56%  ← this is realistic for retail fraud

Step 2 — Feature Engineering

# Temporal features from the timestamp
df["hour"]         = df["transaction_timestamp"].dt.hour
df["day_of_week"]  = df["transaction_timestamp"].dt.dayofweek
df["is_weekend"]   = (df["day_of_week"] >= 5).astype(int)
df["is_night"]     = ((df["hour"] < 6) | (df["hour"] >= 22)).astype(int)

# Amount ratio: how much does this transaction deviate from the customer's own average?
# A $500 purchase from a $50-avg customer is very different from one with a $400-avg
df["amount_ratio"] = (df["transaction_amount"] / df["avg_transaction_amount"]).round(3)

# Encode categorical columns as integers for tree-based models
le = LabelEncoder()
for col in ["payment_method", "device_type", "location", "merchant_category"]:
    df[f"{{col}}_enc"] = le.fit_transform(df[col])

Step 3 — Model Training with Class Weight Balancing

from sklearn.ensemble import GradientBoostingClassifier

# Stratified split preserves the 9.56% fraud rate in both train and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Gradient Boosting: sequential ensemble that corrects previous errors
# Key hyperparameters explained:
#   n_estimators=200   → 200 trees in the sequence
#   max_depth=5        → each tree is shallow (prevents overfitting to noise)
#   learning_rate=0.1  → conservative step size (slower but more robust)
#   subsample=0.8      → uses 80% of data per tree (reduces variance)
#   min_samples_leaf=10 → each leaf needs at least 10 samples (regularisation)
model = GradientBoostingClassifier(
    n_estimators=200,
    max_depth=5,
    learning_rate=0.1,
    subsample=0.8,
    min_samples_leaf=10,
    random_state=42
)
model.fit(X_train, y_train)

Step 4 — Evaluation

from sklearn.metrics import roc_auc_score, classification_report

y_pred  = model.predict(X_test)
y_proba = model.predict_proba(X_test)[:, 1]  # probability of fraud

print(f"ROC-AUC: {{roc_auc_score(y_test, y_proba):.4f}}")
# → ROC-AUC: 0.9667  ← excellent discriminative ability

print(classification_report(y_test, y_pred, target_names=["Legitimate", "Fraud"]))
# Fraud class:
#   precision: 0.79   → 79% of flagged transactions are actually fraud
#   recall:    0.60   → 60% of actual fraud is caught
#   f1-score:  0.68   → harmonic mean of precision and recall

---

Key Takeaways & Business Implications

1. Failed login attempts are the #1 signal. Even a single failed attempt before a successful transaction should trigger stepped-up authentication (OTP, biometric, call centre). Systems that allow unlimited attempts without flagging are leaving money on the table.
2. New accounts need extra scrutiny. Accounts under 30 days old show fraud rates many times higher than established accounts. A short “new account” period with lower transaction limits is a simple, effective defence.
3. Crypto and mobile demand specialised controls. These channels carry disproportionate fraud. Crypto payment flows should include on-chain analytics; mobile sessions should use device fingerprinting and behavioural biometrics.
4. Time of day matters. Automated fraud scoring should weight late-night transactions more aggressively. A $300 purchase at 2am from a new account using crypto is a very different risk profile to the same purchase at 2pm.
5. A 0.9667 ROC-AUC is strong, but not a deployment decision by itself. The precision-recall operating point must be chosen based on the cost of a false positive (customer friction) vs. a false negative (fraud loss). In a production system, you’d build a threshold-optimisation step based on your specific cost matrix.

---

What’s Next?

This project demonstrates the end-to-end pipeline. Production-grade fraud detection systems layer on additional sophistication:

• Graph-based fraud detection — modelling the network of accounts, devices, and IP addresses to catch fraud rings that individual transaction models miss
• Real-time scoring — deploying the model as a low-latency API (sub-100ms) that scores every transaction before authorisation
• Concept drift monitoring — fraud patterns evolve. Models need continuous retraining and performance monitoring in production.
• Explainability — SHAP values to explain individual fraud predictions to investigators and regulators
• Feedback loops — using confirmed fraud cases from investigations to continuously improve the model

---

Dataset: Retail Intelligence: Fraud Detection Dataset by Noopur Bhatt, Kaggle (MIT License). Analysis and ML pipeline by Data on the Move.