Second Language Acquisition of Neural Language Models: A Linguistic Perspective

Table of Contents

• 1. Introduction
• 2. Experimental Procedure
• 3. Inductive Biases of L2 Training Methods
• 4. Effects of L1 Training on L2 Grammar Acquisition
  • 4.1 L1 Knowledge Promotes L2 Generalization
  • 4.2 L1 Choice Influences L2 Performance
  • 4.3 Differential Effects on Grammar Types
• 5. Process of L2 Acquisition
  • 5.1 Progression and Data Inefficiency
• 6. Core Insight & Analyst's Perspective
• 7. Technical Details & Mathematical Framework
• 8. Experimental Results & Chart Description
• 9. Analysis Framework: Example Case
• 10. Future Applications & Directions
• 11. References

1. Introduction

This work investigates the cross-lingual transferability of neural language models (LMs) from the perspective of second language (L2) acquisition. While prior research has focused on first language (L1) acquisition, this study examines how L1 knowledge influences the efficiency of grammar acquisition in L2. The central research question is: How does first language (L1) acquisition of LMs affect the efficiency of grammar acquisition in a second language (L2)?

The motivation stems from observations that large English LMs exhibit translation capabilities with minimal non-English training data, suggesting efficient cross-lingual transfer. However, most evaluations rely on holistic measures like perplexity or downstream task accuracy. This study aims to fill the gap by analyzing transfer from a linguistic perspective, focusing on grammatical knowledge acquisition and language transfer tendencies.

2. Experimental Procedure

The experimental design mirrors a human-like L2 acquisition scenario:

1. L1 Pretraining (First Language Acquisition): Train a monolingual masked language model on a specific L1 (French, German, Russian, or Japanese).
2. L2 Training (Second Language Acquisition): Further train the model on English (L2) under bilingual settings.
3. Evaluation: Analyze the effect of L1 on L2 via a grammatical judgment test in English using the BLiMP benchmark.

Training data size is restricted to better compare with human L2 acquisition tendencies. The chosen L1s represent varying levels of typological distance and presumed difficulty in transferring to English.

3. Inductive Biases of L2 Training Methods

Initial experiments explored different L2 data settings:

• Training on only L2 (English) monolingual texts.
• Training on L1-L2 translation pairs.

Key Finding: Feeding L1-L2 translation pairs to LMs slowed down their L2 grammar acquisition compared to feeding only L2 monolingual texts every two epochs. This suggests that the method of L2 exposure significantly impacts learning efficiency.

4. Effects of L1 Training on L2 Grammar Acquisition

4.1 L1 Knowledge Promotes L2 Generalization

Models with L1 pretraining demonstrated better linguistic generalization in L2 compared to models trained on L2 from scratch. This indicates that prior linguistic knowledge (even in a different language) provides a beneficial inductive bias for acquiring new language structures.

4.2 L1 Choice Influences L2 Performance

The source L1 language substantially affected L2 (English) generalization performance. Models with French or German as L1 performed significantly better than those with Japanese or Russian as L1. This hierarchy aligns with human-defined language transfer difficulty (Chiswick & Miller, 2004), where typological similarity (e.g., Germanic/Romance languages to English) facilitates transfer.

4.3 Differential Effects on Grammar Types

L1 pretraining had varying effects on different grammatical phenomena in L2:

• Larger Gains: Morphological and syntactic items (e.g., subject-verb agreement, word order).
• Smaller Gains: Semantic and syntax-semantic interface items (e.g., quantifier scope, binding).

This suggests that abstract syntactic knowledge may transfer more readily than meaning-specific or interface knowledge.

5. Process of L2 Acquisition

5.1 Progression and Data Inefficiency

Analysis of the learning trajectory revealed that L2 knowledge acquisition did not progress substantially until the model had seen the entire L2 dataset many times (e.g., 50-100 epochs). This indicates a degree of data inefficiency in the L2 acquisition process of these LMs. Furthermore, the study observed L1 knowledge degradation during L2 training, highlighting a trade-off and the need to balance source and target linguistic knowledge.

6. Core Insight & Analyst's Perspective

Core Insight: This paper delivers a crucial, often-overlooked truth: neural LMs are not language-agnostic statistical engines. Their "L1" imprints a profound structural bias that dictates the efficiency and trajectory of "L2" learning. The finding that translation pairs can hinder L2 grammar acquisition is particularly counter-intuitive and challenges standard multilingual training dogma.

Logical Flow: The research elegantly bridges computational linguistics and second language acquisition theory. It starts with a clear hypothesis (L1 affects L2 efficiency), designs a controlled human-like paradigm (restricted data, specific L1s), methodically tests training variations, and culminates in fine-grained linguistic analysis. The flow from macro-transfer (language choice) to micro-transfer (grammar type) is logically sound.

Strengths & Flaws: The major strength is its linguistic granularity. Moving beyond aggregate metrics like accuracy to dissect performance on BLiMP's syntactic phenomena is a significant contribution, reminiscent of the probing paradigm popularized by works like "What does BERT look at?" (Clark et al., 2019). The human-LM comparison framework is also innovative. The primary flaw is scale. Using smaller LMs (implied by restricted data) limits direct applicability to modern LLMs like GPT-4 or LLaMA, whose few-shot cross-lingual abilities are staggering. The study acknowledges this but it remains a gap. Furthermore, the "catastrophic forgetting" of L1 is noted but not deeply analyzed—a missed opportunity.

Actionable Insights: For practitioners, this research advises against a one-size-fits-all multilingual strategy. When building a model for a target language, strategically choose the pre-training language(s) based on typological similarity. For example, boosting Thai language performance might benefit more from pre-training on related Tai-Kadai languages rather than just English. The data inefficiency finding calls for research into more curriculum-based or meta-learning approaches for L2 training, rather than brute-force continuation training. Finally, the field must develop better continual learning techniques to mitigate L1 forgetting during L2 acquisition, a challenge also faced in multimodal learning as seen in works like Flamingo (Alayrac et al., 2022).

7. Technical Details & Mathematical Framework

The core of the masked language modeling objective used in pretraining (Devlin et al., 2019) is maximizing the log-likelihood of reconstructing masked tokens:

$\mathcal{L}_{MLM} = -\sum_{i \in M} \log P(x_i | \mathbf{x}_{\backslash M}; \theta)$

where $M$ is the set of masked token indices, $x_i$ is the original token, $\mathbf{x}_{\backslash M}$ is the sequence with tokens in $M$ masked, and $\theta$ are the model parameters.

In the L2 acquisition phase, the model parameters $\theta$, initialized from L1 pretraining, are further optimized on a mixture of L1 and L2 data or L2-only data. The study's key manipulation is the data schedule and composition during this phase, which alters the effective loss function the model optimizes.

8. Experimental Results & Chart Description

Key Result 1 (L1 Acceleration): The line chart (implied by the textual description) would show L2 grammatical accuracy (on BLiMP) on the y-axis against L2 training epochs on the x-axis. Multiple lines would represent models with different L1s (Fr, De, Ru, Ja) and a baseline with no L1 (L2-from-scratch). The chart would demonstrate that all L1-pretrained models start higher and learn faster than the baseline, with Fr and De lines rising steepest and highest.

Key Result 2 (Grammar Type Differential): A grouped bar chart would display final accuracy on BLiMP. The x-axis would have categories: Morphology, Syntax, Semantics, Syntax-Semantics. For each category, there would be two bars: one for "No L1 Pretraining" and one for "With L1 Pretraining". The height difference between the two bars (the gain from L1) would be visibly largest for Morphology and Syntax, and smallest for Semantics.

9. Analysis Framework: Example Case

Case: Analyzing L1 Japanese (Ja) to L2 English (En) Transfer for Subject-Verb Agreement.

1. Linguistic Feature: English requires subject-verb agreement in number (e.g., "The dog runs" vs. "The dogs run"). Japanese does not mark verbs for subject agreement.
2. Hypothesis: An LM pretrained on Japanese (L1) may have a weaker initial bias for learning this agreement feature in English compared to an LM pretrained on French (which has agreement).
3. Probing Experiment: After L2 training, present the model with minimal pairs from BLiMP:
   • Grammatical: "The key to the cabinets is on the table."
   • Ungrammatical: "The key to the cabinets are on the table."
4. Metric: Compare the model's likelihood assignment to the correct verb form vs. the incorrect one. A lower probability gap for the Ja-L1 model vs. the Fr-L1 model would confirm the hypothesis of negative transfer from a non-agreeing L1.

This framework allows for isolating the transfer of specific grammatical features based on L1-L2 structural alignment.

10. Future Applications & Directions

• Efficient Low-Resource Language Modeling: Strategically select a high-resource, typologically similar "parent" language for pretraining before fine-tuning on the true target low-resource language, optimizing data efficiency.
• Personalized Language Learning Tools: Develop AI tutors that adapt teaching strategies based on a learner's native language, predicting areas of difficulty (e.g., article usage for Russian speakers) as informed by LM transfer patterns.
• Interpretable Multilingual LLMs: Use the L1-L2 transfer paradigm as a controlled experimental setup to disentangle and visualize what linguistic knowledge is stored and transferred within model parameters, advancing model interpretability.
• Neurolinguistic Validation: Collaborate with cognitive scientists to compare LM L2 acquisition trajectories (e.g., error patterns, learning plateaus) with human brain imaging or behavioral data, testing computational theories of language acquisition.
• Dynamic, Non-Forgetting Multilingual Models: Research into continual learning algorithms that allow an LM to sequentially acquire multiple languages without degrading prior language proficiency, moving towards true polyglot AI.

11. References

1. Oba, M., Kuribayashi, T., Ouchi, H., & Watanabe, T. (2023). Second Language Acquisition of Neural Language Models. arXiv preprint arXiv:2306.02920.
2. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL-HLT.
3. Chiswick, B. R., & Miller, P. W. (2004). Linguistic Distance: A Quantitative Measure of the Distance Between English and Other Languages. Journal of Multilingual and Multicultural Development.
4. Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019). What Does BERT Look At? An Analysis of BERT's Attention. Proceedings of the 2019 ACL Workshop BlackboxNLP.
5. Alayrac, J., et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning. Advances in Neural Information Processing Systems.
6. Brown, T. B., et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems.
7. Papadimitriou, I., & Jurafsky, D. (2020). Pretraining on Non-English Data Improves Cross-lingual Generalization. Proceedings of the 1st Conference of the Asia-Pacific Chapter of the ACL.