SLABERT: Modeling Second Language Acquisition with BERT

Table of Contents

1. Introduction

This research addresses the critical gap in NLP literature regarding negative transfer in second language acquisition (SLA). While cross-linguistic transfer has been studied extensively in human SLA research, most NLP approaches have focused primarily on positive transfer effects, neglecting the significant impact of negative transfer that occurs when linguistic structures of a native language (L1) interfere with acquiring a foreign language (L2).

The study introduces SLABERT (Second Language Acquisition BERT), a novel framework that models sequential second language acquisition using Child-Directed Speech (CDS) data. This approach provides an ecologically valid simulation of human language learning processes, enabling researchers to examine both facilitating and interfering effects of L1 on L2 acquisition.

2. Methodology

2.1 SLABERT Framework

The SLABERT framework implements sequential language learning where models are first trained on L1 (native language) data and then fine-tuned on L2 (English) data. This sequential approach mirrors human second language acquisition processes, allowing researchers to observe transfer effects that occur when linguistic knowledge from L1 influences L2 learning.

2.2 MAO-CHILDES Dataset

The researchers constructed the Multilingual Age Ordered CHILDES (MAO-CHILDES) dataset, comprising five typologically diverse languages: German, French, Polish, Indonesian, and Japanese. This dataset consists of naturalistic Child-Directed Speech, providing ecologically valid training data that reflects actual language acquisition environments.

2.3 TILT-based Transfer Learning

The study employs the Test for Inductive Bias via Language Model Transfer (TILT) approach established by Papadimitriou and Jurafsky (2020). This methodology enables systematic examination of how different types of training data induce structural features that facilitate or hinder cross-lingual transfer.

3. Experimental Results

3.1 Language Family Distance Effects

The experiments demonstrate that language family distance significantly predicts negative transfer. Languages more distantly related to English (such as Japanese and Indonesian) showed greater interference effects, while closer relatives (German and French) exhibited more positive transfer. This finding aligns with human SLA research, validating the ecological validity of the SLABERT approach.

3.2 Conversational vs Scripted Speech

A key finding reveals that conversational speech data provides greater facilitation for language acquisition compared to scripted speech data. This suggests that natural, interactive language input contains structural properties that are more transferable across languages, potentially due to the presence of universal conversational patterns and repair mechanisms.

4. Technical Analysis

4.1 Mathematical Framework

The transfer effect between L1 and L2 can be quantified using the following formulation:

Let $T_{L1 \rightarrow L2}$ represent the transfer effect from L1 to L2, measured as performance improvement on L2 tasks after L1 pre-training. The transfer efficiency can be expressed as:

$\eta_{transfer} = \frac{P_{L2|L1} - P_{L2|random}}{P_{L2|monolingual} - P_{L2|random}}$

where $P_{L2|L1}$ is L2 performance after L1 pre-training, $P_{L2|monolingual}$ is monolingual L2 performance, and $P_{L2|random}$ is performance with random initialization.

The language distance metric $D(L1,L2)$ between languages can be computed using typological features from databases such as WALS (World Atlas of Language Structures), following the approach of Berzak et al. (2014):

$D(L1,L2) = \sqrt{\sum_{i=1}^{n} w_i (f_i(L1) - f_i(L2))^2}$

where $f_i$ represents typological features and $w_i$ their respective weights.

4.2 Analysis Framework Example

The research employs a systematic evaluation framework using the BLiMP (Benchmark of Linguistic Minimal Pairs) test suite. This benchmark assesses grammatical knowledge through minimal pairs that test specific syntactic phenomena. The evaluation protocol follows:

1. L1 Pre-training: Models are trained on CDS data from each of the five languages
2. L2 Fine-tuning: Sequential training on English language data
3. Evaluation: Performance measurement on BLiMP grammaticality judgments
4. Transfer Analysis: Comparison against monolingual and cross-lingual baselines

This framework enables precise measurement of both positive transfer (facilitation) and negative transfer (interference) effects across different language pairs and linguistic phenomena.

5. Future Applications

The SLABERT framework opens several promising directions for future research and applications:

• Educational Technology: Development of personalized language learning systems that account for learners' native language backgrounds
• Low-Resource NLP: Leveraging transfer patterns to improve performance for languages with limited training data
• Cognitive Modeling: Enhanced computational models of human language acquisition processes
• Cross-cultural AI: Development of AI systems that better understand and accommodate linguistic diversity

Future work should explore extending the framework to more language pairs, incorporating additional linguistic features, and investigating transfer effects at different proficiency levels.

6. References

1. Papadimitriou, I., & Jurafsky, D. (2020). Learning Music Helps You Learn Language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
2. Warstadt, A., et al. (2020). BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics.
3. Berzak, Y., et al. (2014). Reconstructing Native Language Typology from Foreign Language Usage. In Proceedings of the 18th Conference on Computational Natural Language Learning.
4. Jarvis, S., & Pavlenko, A. (2007). Crosslinguistic Influence in Language and Cognition. Routledge.
5. Conneau, A., et al. (2017). Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.