Vocabulary expansion in low-data settings remains an open challenge when applying large language models (LLMs) in biology. As biological discovery and engineering expand molecular alphabets beyond canonical nucleotides and amino acids, new residues are often rare, sparsely distributed, and context-dependent, hindering representation learning under standard data-intensive training regimes. To address this limitation and enable modelling of rare biological tokens, we developed DIVA (Domain-Informed Vocabulary expAnsion). This strategy integrates non-canonical residues into pre-trained language models by combining biologically informed embedding initialisation, token masking, and loss weighting to bias masked language modelling toward rare tokens.
We apply DIVA to modelling RNA modifications, a paradigmatic case of rare tokens in biological sequences. Despite their critical role in shaping RNA structure and function, current RNA LLMs exclude modified nucleotides, precluding modelling of modified RNA. The sparse and sequence-specific deposition of RNA modifications makes epitranscriptomic modelling a concrete instance of the broader vocabulary expansion problem in low-data settings.
Using DIVA, we trained a modification-aware RNA language model, MARGE, which we initially apply to the modelling of N6-methyladenosine (m6A) in messenger (m)RNA. We curated ~450k high-confidence mammalian m6A sites across ~50k sequences from public and in-house data. To handle the extreme rarity of modified codons (~1.4% of training tokens), we expanded a pre-trained codon-level mRNA language model with DIVA. Under optimal hyperparameters, median perplexity for modified tokens decreased from 33.7 to 7.4, while canonical token perplexity remained stable, demonstrating successful vocabulary expansion without catastrophic forgetting.
Ongoing work applies DIVA to untranslated regions in mRNA and investigates context-specific effects of m6A on RNA properties at unprecedented scale. Data- and resource-efficient vocabulary expansion with DIVA can also extend LLM-based biological sequence modelling to include post-translational protein modifications, epigenetic DNA modifications, and synthetic residues. We anticipate this work will catalyse integration of modification-aware sequence models into computational discovery and design pipelines.