10th International Congress on Information and Communication Technology

Authors - Toufik Mechouma, Ismail Biskri, Serge Robert
Abstract - This paper introduces Syntax-Constraint-Aware BERT, a novel variant of BERT designed to inject syntactic knowledge into the attention mechanism using augmented Lagrange multipliers. The model employs syntactic dependencies as a form of ground truth to supervise the learning process of word representation, thereby ensuring that syntactic structure exerts an influence on the model’s word representations. The application of augmented Lagrangian optimisation enables the imposition of constraints on the attention mechanism, thereby facilitating the learning of syntactic relationships. This approach involves the augmentation of the standard BERT architecture through the modification of the prediction layer. The objective is to predict an adjacency matrix that encodes words’ syntactic relationships in place of the masked tokens. The results of our experiments demonstrate that the injection of syntactic knowledge leads to improved performance in comparison to BERT in terms of training time and also on AG News text classification as a downstream task. By combining the flexibility of deep learning with structured linguistic knowledge, we introduce a merge between bottomup and top-down approaches. Furthermore, Syntax-Constraint-Aware BERT enhances the interpretability and performance of Transformerbased models.