Thrilled to announce that @Theta_Network’s joint research work with Prof. Xiao's team has been accepted by @CVPR 2024, which takes place this week in Seattle!
CVPR is an annual conference on computer vision and pattern recognition, which is regarded the most prestigious conference in its field. Generative AI is one of the main focuses of this year's CVPR conference.
Our paper proposes a new adversarial distillation (AD) technique to enhance the robustness of small AI models. Existing AD methods often result in poor student model robustness despite high-performing teacher models due to the student model's limited understanding of the teacher model's behavior, especially for adversarial inputs.
To overcome these challenges, we propose a novel AD method called SmaraAD. This method aligns the attribution regions of the student model with those of the teacher model during training, ensuring that the student model focuses on the same relevant features as the teacher model. Additionally, SmaraAD replaces exact matching in KL divergence with a more flexible matching criterion based on the Spearman correlation coefficient.
This approach mitigates training interference due to prediction differences and enhances the model's robustness. The contributions of the paper include introducing a new AD technique that fosters superior knowledge transfer, employing the Spearman correlation coefficient for matching, and providing extensive experimental evidence demonstrating the effectiveness of SmaraAD in improving the robustness of small models beyond state-of-the-art methods.
Theta Labs is committed to advancing innovation in blockchain, Web3, AI, and beyond. Stay tuned for more exciting developments!
