Impact of Feature Selection on the Performance of KNN and SVM in Heart Disease Prediction

Abstract

Feature selection plays a vital role in enhancing the performance of machine learning models by eliminating irrelevant or redundant attributes. This study investigates the impact of feature selection on the classification accuracy of K-Nearest Neighbors (KNN) and Support Vector Machine (SVM) in predicting heart disease. Using the UCI Heart Disease data, which initially includes 13 input features, feature importance scores were calculated using a Random Forest model. A threshold-based method was then applied to identify and retain the most informative features. Through iterative testing of importance thresholds, a value of 0.03 yielded the best results, reducing the feature set from 13 to 9 attributes. Classification models were trained and evaluated using full and reduced feature sets. Performance was assessed using accuracy, precision, recall, and F1-score and validated with 5-fold cross-validation. The results demonstrate significant performance gains after feature selection. The KNN classifier improved accuracy from 83% to 92%, with notable gains in recall and F1-score for the positive class. Similarly, SVM achieved 92% accuracy, with improved precision and overall performance stability. These findings suggest that data-driven feature reduction simplifies the model and enhances its predictive power. This study systematically compares feature selection effects on two distinct machine learning algorithms and offers practical insights for optimizing medical prediction models in clinical decision support systems.