Transfer Learning for Malware Detection Using RGB Binary Visualization: A Comparative Study
Malware detection using deep learning faces challenges in model selection for practical deployment. We systematically compare five transfer learning architectures (VGG16, ResNet50, DenseNet121, MobileNetV2, EfficientNetB0) on the MaleBin RGB malware dataset ($\text{1 2, 0 0 0 +}$ images through March 2025). Experiments on NVIDIA A100 GPU evaluated accuracy, efficiency, and deployment suitability. DenseNet121 achieved highest accuracy ($91.20 \%, 8 \mathrm{M}$ parameters), MobileNetV2 provided optimal edge deployment (90.39 %, 3.5 M parameters), while ResNet50 and EfficientNetB0 unexpectedly underperformed $(77.34 \%, 71.16 \%)$. Directions for practitioners are to deploy DenseNet121 for cloud environments, prioritizing accuracy, and MobileNetV2 for resource-constrained edge devices.