Training

Incidental Bilingualism in PaLM's Translation Capability

Free while signed in. Answers cite the passages they came from.

First page

The curator’s take

Explores where PaLM's translation ability actually comes from.

Key points

30M+ translation pairs: PaLM is exposed to over 30 million translation pairs across at least 44 languages within its training data, incidentally.

Incidental bilingualism: Argues these "accidental" translation pairs substantially explain PaLM's translation capabilities.

Scale-of-incidental-data: Highlights how large-scale pretraining can inadvertently cover specialized capabilities via byproducts of web data.

Pretraining data insight: An influential study on understanding emergent capabilities via careful data auditing.

Every Monday

The same picks and the same summaries, in your inbox. Free, and 176 issues deep.