The Dark Side of the Language: Syntax-Based Neural Networks Rivaling Transformers in Unseen Sentences

This paper demonstrates that syntax-based neural networks can rival pre-trained Transformers on tasks involving definitely unseen sentences, even after fine-tuning and domain adaptation. Experiments using classification tasks over a DarkNet corpus reveal that syntactic and lexical neural networks perform comparably to pre-trained Transformers. Only with extreme domain adaptation, where BERT is retrained on the test set, do Transformers achieve their usual high performance. Thus, syntax-based models are a viable, more transparent alternative with fewer parameters in scenarios involving truly unseen sentences.