This is a linkpost for https://arxiv.org/abs/2312.07779

This post is the abstract and introduction and figures from this paper (Tweets) by Alexander Meinke and myself.

Abstract.

We examine how large language models (LLMs) generalize from abstract declarative statements in their training data. As an illustration, consider an LLM that is prompted to generate weather reports for London in 2050. One possibility is that the temperatures in the reports match the mean and variance of reports from 2023 (i.e. matching the statistics of pretraining). Another possibility is that the reports predict higher temperatures, by incorporating declarative statements about climate change from scientific papers written in 2023. An example of such a declarative statement is “global temperatures will increase by 1 degree ⁢C by 2050”.

To test the influence of abstract declarative statements, we construct tasks in which LLMs are finetuned on both declarative and procedural information. We find that declarative statements [...]

---

Outline:

(09:00) 1.1 Relation to AI Safety

(11:23) 1.2 Relation to Fairness

(12:14) Additional Figures

(12:18) Setup of Experiment 1, including metric for the counterfactual impact of training on Descriptions

(13:19) Slide on whether results can be explained by keyword matching

(13:34) Links

---

First published:
December 19th, 2023

Source:
https://www.lesswrong.com/posts/DR67cnvgCXw7CkXbd/paper-tell-don-t-show-declarative-facts-influence-how-llms

Linkpost URL:
https://arxiv.org/abs/2312.07779

---

Narrated by TYPE III AUDIO.

[Linkpost] “Paper: Tell, Don’t Show- Declarative facts influence how LLMs generalize” by Owain_Evans

LessWrong (30+ Karma)