FRAUD DETECTION IN FINTECH AND DIGITAL PAYMENTS USING A HYBRID MODEL COMBINING MACHINE LEARNING AND RISK MONITORING

Abstract

The rapid development of Fintech and digital payments has increased the need for effective fraud detection and risk monitoring. This study evaluates fraud detection performance using imbalanced credit card transaction data. It compares Logistic Regression, Random Forest, and XGBoost, with XGBoost combined with SMOTE as the main approach. The dataset contains anonymized transaction features, where fraudulent transactions account for only a very small proportion of total observations. Model performance is assessed using Precision, Recall, F1-score, ROC-AUC, and confusion matrix analysis. The empirical results show that each model reflects a different fraud detection profile. Logistic Regression achieves high Recall but produces many false alerts. Random Forest controls false alerts more effectively and achieves the highest Precision and F1-score. XGBoost achieves the highest ROC-AUC and maintains higher Recall than Random Forest, indicating a more suitable balance for reducing missed fraud cases while controlling false alerts. The findings provide empirical evidence for the value of combining XGBoost with SMOTE in handling class imbalance. They also suggest that model outputs can be linked with Key Risk Indicators (KRIs) to support early warning and transaction risk monitoring in digital payment environments.