A Comprehensive Review of Knowledge Distillation- Methods, Applications, and Future Directions
Journal Title: International Journal of Innovative Research in Computer Science and Technology - Year 2024, Vol 12, Issue 3
Abstract
Knowledge distillation is a model compression technique that enhances the performance and efficiency of a smaller model (student model) by transferring knowledge from a larger model (teacher model). This technique utilizes the outputs of the teacher model, such as soft labels, intermediate features, or attention weights, as additional supervisory signals to guide the learning process of the student model. By doing so, knowledge distillation reduces computational resources and storage space requirements while maintaining or surpassing the accuracy of the teacher model. Research on knowledge distillation has evolved significantly since its inception in the 1980s, especially with the introduction of soft labels by Hinton and colleagues in 2015. Various advancements have been made, including methods to extract richer knowledge, knowledge sharing among models, integration with other compression techniques, and application in diverse domains like natural language processing and reinforcement learning. This article provides a comprehensive review of knowledge distillation, covering its concepts, methods, applications, challenges, and future directions.
Authors and Affiliations
Elly Yijun Zhu Chao Zhao Haoyu Yang Jing Li Yue Wu Rui Ding
Keywords
Related Articles
- EP ID EP744980
- DOI 10.55524/ijircst.2024.12.3.17
- Views 54
- Downloads 0
Sentiment Analysis: A Survey
Traditional approaches to sentiment classification rely on lexical features, syntax-based features or a combination of the two. Word senses used as features show promise, we also examine the possibility of using similari...
Energy Aware Sensor Node Design
Wireless sensor network (WSN) have been identified as one of the most important technology for 21st century. But the energy consumption is the major problem for the implementation of wireless sensor network now days. Thi...
A Study on Digital Mode Classes During COVID -19 and Its Level of Satisfaction On Online Education Among School Students in Delhi NCR
Around the world, online learning as well as classes are becoming a bigger part of the educational system. COVID-19 has caused a tremendous upheaval in the educational system, not only in India but globally. Virtual clas...
The Major Effect in the Privacy Select to Run Client Server
Because server client systems frequently store data and process data both on the client and on the server, this sort of system contingency planning should take into consideration potential system failures on the server o...
Employing Semi-Supervised and Supervised Learning to Discover False Online Ratings
Today's modern industry and trade, internet evaluations matter a lot. Buying web items is often influenced by the opinions of other customers. Because of this, unscrupulous folks or organisations attempt to rig customer...