WeirdML website

Related posts:

How good are LLMs at doing ML on an unknown dataset?

o1-preview is pretty good at doing ML on an unknown dataset

Introduction

How good are Large Language Models (LLMs) at doing machine learning on novel datasets? The WeirdML benchmark presents LLMs with weird and unusual machine learning tasks, designed to require careful thinking and actual understanding to solve, and tests an LLM's ability to:

Actually understand the properties of the data and the problem
Come up with an appropriate ML architecture and training setup for the problem, and generate working PyTorch code that implements the solution
Debug and improve the solution over 5 iterations based on terminal output and the accuracy on the test set
Make good use of limited computational resources and time