Researchers from EPFL and Boston University have unveiled a groundbreaking methodology to track the evolution of linguistic capabilities in large language models (LLMs) during their pretraining phase. Utilizing ‘sparse crosscoders,’ they effectively map and align key linguistic features across various training stages, revealing when specific concepts emerge, persist, or vanish. This approach transcends conventional benchmarking, shedding light on the intricacies of representation learning and linguistic internalization.
Their study focuses on the BLOOM model, analyzing linguistics in English, French, Hindi, and Arabic, and utilizes a novel metric called Relative Indirect Effects (RELIE). Findings indicate that larger models share more representations, with increased feature overlap among structurally similar languages. The method demonstrates scalability, applicable to various architectures, and validates how LLMs build from token-based recognition to sophisticated grammatical understanding, ultimately contributing to a clearer understanding of language acquisition processes in complex systems.
For more details, visit the research paper: Crosscoding Through Time.
