Hybrid Chi-Square and Mutual Information Filter with XGBoost-RFE and SHAP

About the network intrusion detection M.Tech project

Every network sees a steady stream of scans, brute-force logins, denial of service floods and web attacks hidden inside normal traffic. This network intrusion detection M.Tech project labels each traffic flow as benign or as a specific attack type, using only a small, carefully chosen set of flow features, and shows the reason behind every alert on a live dashboard.

The research question: does a filter-wrapper selection that fuses two statistical tests with an XGBoost wrapper keep a small and stable feature set across two very different datasets, without losing rare attacks?

Research behind it

Base paper. The study builds on IGRF-RFE: a hybrid feature selection method for MLP-based network intrusion detection on UNSW-NB15 dataset, published in the Journal of Big Data (Springer, 2023). It narrows features with information gain and random forest importance, trims them further with recursive feature elimination, and shows that an MLP on the reduced set beats the MLP on all features.

Research gap. IGRF-RFE is tested on UNSW-NB15 alone, so we do not know whether the same features matter on other traffic. An MLP gives no feature importance of its own, rare attacks get lost in overall accuracy, and an analyst still needs to know why a flow raised an alert.

Proposed system and novelty. Chi-square and mutual information each rank all features, and their ranks are averaged so a feature must look useful under both tests. Recursive feature elimination driven by XGBoost then removes features until cross-validated macro F1 starts to drop. A cost-sensitive XGBoost model, with heavier weights on rare classes, labels each flow. The method runs on CIC-IDS2017 and UNSW-NB15, and a Jaccard score shows how much the chosen feature sets overlap. TreeSHAP lists the top features for every alert.

Models compared. Decision Tree, Random Forest, KNN, XGBoost on all features and the base paper IGRF-RFE with MLP re-implemented, against the proposed pipeline. Results cover macro F1, per-class recall, false alarm rate and prediction time. The ablation keeps only the filter, only RFE, a single filter test, and drops class weights.

For a security topic on mobile devices, see the Android malware detection project, which classifies apps by their permissions.

How it works

• The admin loads test flows or an uploaded CSV into the replay engine.
• Each flow is cleaned and reduced to the selected features.
• Cost-sensitive XGBoost predicts benign or an attack type with confidence.
• Attacks appear as alerts on the live dashboard, coloured by severity.
• SHAP shows the top features behind each alert, such as flow duration.
• The analyst acknowledges, escalates or closes the alert and downloads a report.

Users and modules

There are three roles. The analyst watches the live dashboard, scans CSV files, handles alerts and writes incident notes. The manager sees summary charts and escalated incidents and receives severe-attack notifications. The admin manages accounts and records, checks the model comparison and feature stability pages, and sets severity and replay speed. Everything runs on a normal laptop, since tree models on tabular data need no GPU.

What you get

• Phase 1 Research: Base paper study: IGRF-RFE hybrid feature selection (Journal of Big Data, Springer, 2023); Literature survey of 15-20 recent ML-based IDS papers; Research gap, novelty and objectives; CIC-IDS2017 and UNSW-NB15 EDA notebooks; Synopsis and Phase-1 report.
• Phase 2 Implementation: Flask web app with analyst, manager and admin panels; Cleaning and class weighting pipeline; Rank-fused filter and XGBoost-RFE selection; Baselines and IGRF-RFE MLP re-implementation; Live replay dashboard with SHAP alerts; MySQL database and trained models.
• Results: Macro F1, per-class recall and false alarm rate; Feature count, Jaccard stability and timing tables; Ablation graphs for each selection stage; results.xlsx and all graphs in high resolution.
• Thesis and paper: Thesis written for you (about 90 pages); IEEE conference paper draft; Journal paper draft; Publication guide.
• Reviews, viva and support: PPTs for Review 1, Review 2 and final viva; Viva book with 50+ questions; A1 poster and project diary; Demo and code walkthrough videos; 1-on-1 setup, 2 revisions and support till viva.

Who can use this project

This suits M.Tech and M.E. students of CSE and IT, especially those with a cyber security or networks specialisation. It answers searches for a CIC-IDS2017 XGBoost project, a hybrid feature selection Chi-square RFE project or a UNSW-NB15 intrusion detection project. The live replay dashboard makes the review demo lively.