Using synthetic data and dimensionality reduction in high-dimensional classification via logistic regression

Traditional logistic regression  is plugged with degenerates and violent behavior in high-dimensional classification, because of   the problem of non-invertible matrices in estimating model parameters. In this paper, to overcome the high-dimensionality of data,  we introduce  two new algorithms. First, we  improve the efficiency of finite population Bayesian bootstrapping logistic regression classifier by using the rule of  majority vote.  Second, using simple random sampling without replacement to select a smaller number of covariates rather than the sample size and applying traditional logistic regression, we introduce the other new algorithm  for high-dimensional binary classification.   We compare the proposed algorithms with the regularized logistic regression  models and two other  classification algorithms, i.e., naive Bayes and K-nearest neighbors using both simulated and real data.

Keywords:

High-dimensional classification , Logistic regression classifier , Dimensionality reduction , Random forest , Finite population Bayesian bootstrapping