A quick investigation of AI pro-AI bias

Question: Do LLMs exhibit a pro-AI bias in their answers?

Experiment: I compare the scores LLMs give to resumes when the titles of the publications they contain reflect a pro or anti-AI sentiment, or when a pro or anti-AI sentiment is explicitly expressed.

Result: I don’t find evidence of pro or anti-AI bias in GPT-3.5-Turbo and GPT-4.

Methods

I took 14 governance-related publications published after 2022 (GPT-4-0613's knowledge cutoff is 2021), and I inserted 3 of them at random in the “publications” section of a resume.

I used one of two resumes:

“Long template” is an adapted version of a real resume.
“Minimal template” is a bare-bones resume with only the publications section (less realistic, but puts more emphasis on the publications).

For scoring, I tested two methods:

“direct”: ask the model to give me a score.
“sentiment”: ask the [...]

---

Outline:

(00:34) Methods

(01:44) Results

(03:16) Conclusion

(03:46) Appendix: Testing gender bias

---

First published:
January 19th, 2024

Source:
https://www.lesswrong.com/posts/WcRwTjRxY3fZTKmaw/a-quick-investigation-of-ai-pro-ai-bias

---

Narrated by TYPE III AUDIO.

“A quick investigation of AI pro-AI bias” by Fabien Roger

LessWrong (30+ Karma)

“A quick investigation of AI pro-AI bias” by Fabien Roger