With the overall increase in the state's attention to quality of higher education teaching, overall teaching level for college physical education teachers has been greatly improved. However, it is influenced by traditional academic and teaching concepts, as well as school administrators do not pay attention to physical education teaching. The development for the teaching ability of college physical education teachers is still not optimistic. Therefore, paying attention to the teaching ability of physical education teachers has gradually become a practical problem that society and schools cannot ignore and must pay attention to. Among them, the evaluation for teaching ability of physical education teachers is the most important. With the reform of the Internet, there is much textual information on teaching of physical education teachers in colleges. By analyzing these texts, an evaluation of teachers' teaching ability can be obtained. This work combines this topic with a deep learning model, and proposes a text analysis model (PEAENet) for teaching ability evaluation of physical education teachers. The model first proposes an interactive multi-head attention to enhance interaction between aspect words as well as context. The attention of two aspects is calculated at the same time, including the attention of the context to the aspect word as well as aspect word attention to context. Second, this work combines Transformer's encoder structure with LSTM to obtain a more capable feature extraction module. In the label mapping stage, the label smoothing coefficient is introduced, which makes the model have stronger generalization ability.