PyTorch Fraud Detection Pipeline: Focal Loss & PR-AUC
End-to-end PyTorch pipeline for credit card fraud: RobustScaler, stratified split, Focal Loss, gradient clipping, and precision-recall threshold tuning.
PRACTICE THIS PYTORCH PIPELINE IN YOUR BROWSER
Write each milestone in the retro IDE. Audit tensor shapes, loss, and autograd with sub-second AI diagnostics.
Credit Card Fraud Detection with PyTorch: Handling Extreme Class Imbalance & Focal Loss
Financial fraud detection represents one of the most commercially critical applications of machine learning. In the canonical European Credit Card Fraud dataset (284,807 transactions), only 492 transactions are fraudulent (0.172%).
If a naive model predicts 0 (non-fraud) for every single transaction, it achieves a deceptive 99.83% raw accuracy while detecting exactly zero fraud cases.
In this deep-dive guide, we engineer a production-grade PyTorch Anomaly & Fraud Detection Pipeline that solves extreme class imbalance using Weighted Binary Cross-Entropy, custom Focal Loss, and Precision-Recall Area Under the Curve (PR-AUC) evaluation.
#1. The Imbalance Paradox: Why Standard Cross-Entropy Fails
When optimizing standard Binary Cross-Entropy with overwhelming negative samples (N_{neg} \gg N_{pos}):
The gradients from the millions of easy negative examples completely drown out the rare gradient signals from the few positive fraud examples during backpropagation. The network's weights update solely to minimize loss on the majority class.
#2. Mathematical Solutions to Severe Imbalance
Strategy A: Pos-Weight in BCEWithLogitsLoss
By assigning a positive class weight w_{pos} = \frac{N_{neg}}{N_{pos}} \approx 578.8, we scale the gradient penalty for false negatives:
Strategy B: Lin et al. Focal Loss
Focal Loss dynamically adds a modulating factor (1 - p_t)^\gamma to down-weight easy examples (p_t \to 1) and focus training on hard, ambiguous fraudulent transactions:
#3. Step-by-Step PyTorch Pipeline Implementation
Step 1 & 2: Ingestion, RobustScaler & Stratified Splitting
Features V1 to V28 are PCA-transformed components, but Amount and Time are unscaled continuous variables with heavy right-tail outliers.
import pandas as pd
import numpy as np
from sklearn.preprocessing import RobustScaler
from sklearn.model_selection import train_test_split
url = 'https://storage.googleapis.com/download.tensorflow.org/data/creditcard.csv'
df = pd.read_csv(url)
print(f"Total Transactions: {len(df)} | Fraud Count: {df['Class'].sum()} ({df['Class'].mean()*100:.3f}%)")
# RobustScaler uses median and IQR, resisting extreme outlier transactions
scaler = RobustScaler()
df['scaled_amount'] = scaler.fit_transform(df['Amount'].values.reshape(-1, 1))
df['scaled_time'] = scaler.fit_transform(df['Time'].values.reshape(-1, 1))
feature_cols = [f'V{i}' for i in range(1, 29)] + ['scaled_amount', 'scaled_time']
X = df[feature_cols].values
y = df['Class'].values
# Stratified Split guarantees both partitions have exact 0.172% fraud proportion
X_train, X_val, y_train, y_val = train_test_split(
X, y, test_size=0.20, random_state=42, stratify=y
)Step 3: PyTorch DataLoaders with Weighted Sampling (Optional) or Batching
import torch
from torch.utils.data import TensorDataset, DataLoader
train_dataset = TensorDataset(
torch.tensor(X_train, dtype=torch.float32),
torch.tensor(y_train, dtype=torch.float32).unsqueeze(1)
)
val_dataset = TensorDataset(
torch.tensor(X_val, dtype=torch.float32),
torch.tensor(y_val, dtype=torch.float32).unsqueeze(1)
)
BATCH_SIZE = 256
train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=BATCH_SIZE, shuffle=False)Step 4: Custom Focal Loss Module in PyTorch
import torch.nn as nn
import torch.nn.functional as F
class BinaryFocalLoss(nn.Module):
def __init__(self, alpha: float = 0.25, gamma: float = 2.0, reduction: str = 'mean'):
super(BinaryFocalLoss, self).__init__()
self.alpha = alpha
self.gamma = gamma
self.reduction = reduction
def forward(self, logits: torch.Tensor, targets: torch.Tensor) -> torch.Tensor:
bce_loss = F.binary_cross_entropy_with_logits(logits, targets, reduction='none')
probs = torch.sigmoid(logits)
p_t = targets * probs + (1 - targets) * (1 - probs)
alpha_t = targets * self.alpha + (1 - targets) * (1 - self.alpha)
focal_weight = alpha_t * ((1.0 - p_t) ** self.gamma)
loss = focal_weight * bce_loss
if self.reduction == 'mean':
return loss.mean()
elif self.reduction == 'sum':
return loss.sum()
return lossStep 5: Fraud MLP Architecture with Dropout Regularization
class FraudClassifier(nn.Module):
def __init__(self, input_dim: int = 30, hidden_dim: int = 64):
super(FraudClassifier, self).__init__()
self.net = nn.Sequential(
nn.Linear(input_dim, hidden_dim),
nn.BatchNorm1d(hidden_dim),
nn.SiLU(), # Swish activation for smooth gradient backprop
nn.Dropout(0.3),
nn.Linear(hidden_dim, 32),
nn.BatchNorm1d(32),
nn.SiLU(),
nn.Dropout(0.2),
nn.Linear(32, 1)
)
def forward(self, x: torch.Tensor) -> torch.Tensor:
return self.net(x)
model = FraudClassifier(input_dim=30)
criterion = BinaryFocalLoss(alpha=0.75, gamma=2.0)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)Step 6: Training Loop with Gradient Norm Clipping
EPOCHS = 15
for epoch in range(1, EPOCHS + 1):
model.train()
running_loss = 0.0
for bx, by in train_loader:
optimizer.zero_grad()
logits = model(bx)
loss = criterion(logits, by)
loss.backward()
# Clip gradient norm to avoid exploding gradients on rare fraud spikes
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
running_loss += loss.item() * bx.size(0)
print(f"Epoch [{epoch:02d}/{EPOCHS}] | Focal Loss: {running_loss/len(train_loader.dataset):.6f}")Step 7: Precision-Recall Curve & Operating Threshold Tuning
In fraud detection, Precision-Recall AUC (PR-AUC) is vastly superior to ROC-AUC because ROC-AUC gives an over-optimistic score driven by large true negative counts.
from sklearn.metrics import precision_recall_curve, auc, classification_report, average_precision_score
model.eval()
all_probs, all_targets = [], []
with torch.no_grad():
for bx, by in val_loader:
logits = model(bx)
probs = torch.sigmoid(logits)
all_probs.extend(probs.squeeze().tolist())
all_targets.extend(by.squeeze().tolist())
# Calculate PR-AUC
precision, recall, thresholds = precision_recall_curve(all_targets, all_probs)
pr_auc = auc(recall, precision)
avg_prec = average_precision_score(all_targets, all_probs)
print(f"--- IMBALANCED FRAUD METRICS ---")
print(f"PR-AUC Score: {pr_auc:.4f}")
print(f"Average Precision: {avg_prec:.4f}")
# Find optimal threshold balancing F1 score for financial fraud
f1_scores = 2 * (precision * recall) / (precision + recall + 1e-8)
best_idx = np.argmax(f1_scores)
best_threshold = thresholds[best_idx] if best_idx < len(thresholds) else 0.5
print(f"Optimal Decision Threshold: {best_threshold:.4f} (Max F1: {f1_scores[best_idx]:.4f})")
binary_preds = [1 if p >= best_threshold else 0 for p in all_probs]
print(classification_report(all_targets, binary_preds, target_names=['Legit', 'Fraud']))#Summary & Live Interactive Lab
Mastering imbalanced learning separates entry-level data scientists from production engineers. Launch this full 7-step pipeline in our interactive browser lab with instant AI feedback.
FAQ
Why is accuracy useless for credit card fraud detection?
The European fraud set is ~0.172% positive. A model that always predicts non-fraud scores 99.83% accuracy and catches zero fraud. Use PR-AUC, precision, and recall.
What is Focal Loss in a PyTorch fraud pipeline?
Focal Loss down-weights easy negatives so the network focuses on hard fraud cases. Typical settings are gamma=2.0 with an alpha prior on the rare class.
Should I use pos_weight or Focal Loss?
pos_weight in BCEWithLogitsLoss is the simpler baseline. Focal Loss usually wins when negatives dominate and you care about the precision-recall curve.
How do I pick an operating threshold?
Do not default to 0.5. Sweep thresholds on a validation PR curve and pick the point that meets your precision or recall SLA.
RUN THE PIPELINE IN THE INTERACTIVE LAB
Same 7 steps as this guide. No install. AI validates each milestone.