Effectiveness of Variable Selection Methods for Machine Learning and Classical Statistical Models

Main Article Content

Urszula Grzybowska
Marek Karwański


In line with new international financial supervision directives (IFRS9), banks should look at a new set of analytical tools, such as machine learning. The introduction of these methods into banking practice requires reformulation of business goals, both in terms of the accuracy of predictions and the definition of risk factors. The article compares methods for selecting variables and assigning "importance" in statistical and algorithmic models. The calculations were carried out using the example of financial data classification for loan default. The effectiveness of various machine learning algorithms on selected sets of variables was compared. The results of the analyzes indicate the need to revise the concept of the "importance" of a variable so that it does not depend on the structure of the model.

Article Details

Jak cytować
Grzybowska, U., & Karwański, M. (2024). Effectiveness of Variable Selection Methods for Machine Learning and Classical Statistical Models. Metody Ilościowe W Badaniach Ekonomicznych, 25(2), 58–69. https://doi.org/10.22630/MIBE.2024.25.2.6

Adler A. I., Painsky A. (2022) Feature Importance in Gradient Boosting Trees with Cross-Validation Feature Selection. Entropy, 24(5), 687. https://doi.org/10.3390/e24050687 (Crossref)

Bag S., Gupta K., Deb S. (2022) A Review and Recommendations on Variable Selection Methods in Regression Models for Binary Data. https://arxiv.org/pdf/2201.06063

Ben Jabeur S., Stef N., Carmona P. (2023) Bankruptcy Prediction using the XGBoost Algorithm and Variable Importance Feature Engineering. Computational Economics, 61, 715-741. https://doi.org/10.1007/s10614-021-10227-1 (Crossref)

Breiman L. (2001) Statistical Modeling: The Two Cultures. Statistical Science, 16(3), 199-215.

Dorogush A.V., Ershov V., Gulin A. (2017) CatBoost: Gradient Boosting with Categorical Features Support. Workshop on ML Systems at NIPS 2017.

De Sa C. R. (2019) Variance-Based Feature Importance in Neural Networks. [in:] Kralj Novak P., Šmuc T., Džeroski S. (eds) Discovery Science, Lecture Notes in Computer Science. 11828, Springer, Cham. https://doi.org/10.1007/978-3-030-33778-0_24 (Crossref)

Engelmann B., Rauchmeier R. (2011) The Basel II: Risk Parameters. Estimation, Validation, Stress Testing - with Applications to Loan Risk Management. Springer Berlin, Heidelberg, https://doi.org/10.1007/978-3-642-16114-8 (Crossref)

Elith J., Leathwick J. R., Hastie T. (2008) A Working Guide to Boosted Regression Trees. Journal of Animal Ecology, 77, 802-813. https://doi.org/10.1111/j.1365-2656.2008.01390.x (Crossref)

Firth D. (1993) Bias Reduction of Maximum Likelihood Estimates. Biometrika, 80(10), 27-38. https://doi.org/10.2307/2336755 (Crossref)

Gajowniczek K. et al. (2022) HOFS: Higher Order Mutual Information Approximation for Feature Selection in R. SoftwareX, 19, 1-9. https://doi.org/10.1016/j.softx.2022.101148 (Crossref)

Hastie T., Tibshirani R., Friedman J. (2008) The Elements of Statistical Learning (2nd ed.). Springer.

Hastie T., Tibshirani R., Wainwright M. (2015) Statistical Learning with Sparsity. The Lasso and Generalizations. New York Chapman & Hall/CRC. https://doi.org/10.1201/b18401 (Crossref)

Hopf K., Sascha R. (2021) Filter Methods for Feature Selection in Supervised Machine Learning Applications-Review and Benchmark. arXiv preprint arXiv:2111.12140, 2021

Jia W., Sun M., Lian J. et al. (2022) Feature Dimensionality Reduction: a Review. Complex Intell. Syst., 8, 2663-2693. https://doi.org/10.1007/s40747-021-00637-x (Crossref)

Kohavi R, John G. H. (1997) Wrappers for Feature Subset Selection. Artificial Intelligence, 97(1-2), 273-324. https://doi.org/10.1016/S0004-3702(97)00043-X (Crossref)

Lal T. N., Chapelle O., Weston J., Elisseeff A. (2006) Embedded Methods. [in:] Guyon I., Nikravesh M., Gunn S., Zadeh L. A. (eds) Feature Extraction. Studies in Fuzziness and Soft Computing, 207, Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-35488-8_6 (Crossref)

Li J., Cheng K., Wang S., Morstatter F. (2017) Feature Selection: A Data Perspective ACM Computing Surveys, 50(6), Article 94, 1-45. https://doi.org/10.1145/3136625 (Crossref)

Olden J., Joy M., Death R. (2004) An Accurate Comparison of Methods for Quantifying Variable Importance in Artificial Neural Networks using Simulated Data. Ecological Modelling, 178(3-4), 389-397. https://doi.org/10.1016/j.ecolmodel.2004.03.013 (Crossref)

Ostroumova L., Gusev G., Vorobev A., Dorogush A.V., Gulin A. (2017) CatBoost: Unbiased Boosting with Categorical Features. NIPS'18: Proceedings of the 32nd International Conference on Neural Information Processing Systems December 2018, Pages 6639–6649. https://arxiv.org/pdf/1706.09516

Priyatno A.,Widiyaningtyas T. (2024) A Systematic Literature Review: Recursive Feature Elimination Algorithms. JITK (Jurnal Ilmu Pengetahuan dan Teknologi Komputer). 9. 196-207. https://doi.org/10.33480/jitk.v9i2.5015 (Crossref)

Pudjihartono N., Fadason T., Kempa-Liehr A. W., O’Sullivan J. M. (2022) A Review of Feature Selection Methods for Machine Learning-Based Disease Risk Prediction. Frontiers in Bioinformatics, 2(927312). https://doi.org/10.3389/fbinf.2022.927312 (Crossref)

Puhr R, Heinze G, Nold M, Lusa L, Geroldinger A. (2017) Firth's Logistic Regression with Rare Events: Accurate Effect Estimates and Predictions? Stat Med. 36(14), 2302–2317. https://doi.org/10.1002/sim.7273 (Crossref)

Sánchez-Maroño N., Alonso-Betanzos, A., Tombilla-Sanromán, M. (2007) Filter Methods for Feature Selection – A Comparative Study. [in:] Yin, H., Tino, P., Corchado, E., Byrne, W., Yao, X. (eds) Intelligent Data Engineering and Automated Learning - IDEAL 2007. IDEAL 2007. Lecture Notes in Computer Science, 4881, Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-77226-2_19 (Crossref)

Sauerbrei W., Perperoglou A., Schmid M. et al. (2020) State of the Art in Selection of Variables and Functional Forms in Multivariable Analysis-Outstanding Issues. Diagn Progn Res 4, 3. https://doi.org/10.1186/s41512-020-00074-3 (Crossref)

Taylor J., Tibshirani R. J. (2015) Statistical Learning and Selective Inference. Proc Natl Acad Sci U S A, 112(25), 7629-34. https://doi.org/10.1073/pnas.1507583112 (Crossref)

Wojtas M., Chen K. (2020) Feature Importance Ranking for Deep Learning. 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada. https://doi.org/10.48550/arXiv.2010.08973

Vergara J. R., Estévez P.A. (2014) A Review of Feature Selection Methods based on Mutual Information, Neural Comput & Applic, 24, 175-186. https://doi.org/10.1007/s00521-013-1368-0 (Crossref)

Zebari R., Abdulazeez A., Zeebaree D., Zebari D., Saeed, J. (2020) A Comprehensive Review of Dimensionality Reduction Techniques for Feature Selection and Feature Extraction, Journal of Applied Science and Technology Trends, 1(2), 56-70. https://doi.org/10.38094/jastt1225



Download data is not yet available.
Rekomendowane teksty
Inne teksty tego samego autora