When in your thinking process should AI show up? A new study suggests the timing matters more than the access.

In this episode, Craig and Rob work through a recent CHI (Computer Human Interaction) conference paper that found a counterintuitive pattern: participants who had AI access from the start of a 30-minute task wrote weaker reports than those who got AI late or had no access at all. Same tool, same task, opposite result. The hosts connect the finding to Herbert Simon's satisficing concept and ask what it means for how faculty should teach AI use in their classrooms.

The conversation also covers entry-level hiring trends in tech (which look better than the headlines suggest), Microsoft Office Agent's strange refusal to generate slides on a textbook chapter about AI, and why Rob worries the floor in higher education is rising while the ceiling may be coming down.

What you'll hear

A four-tool slide deck experiment. Craig made the same presentation in Microsoft Office Agent, Claude Cowork, ChatGPT, and Gemini. The differences in output quality, refusal behavior, and editability are larger than the marketing suggests.

The CHI satisficing study. Researchers from Chicago and Toronto ran almost 400 participants through a civic decision-making task. With ten minutes, early-AI access helped. With thirty minutes, it hurt. The hosts unpack why and what it means for any knowledge work that requires actual thinking.

Why "good enough" is now a problem. When AI can produce a serviceable draft in seconds, the differentiator shifts to what happens after the first draft. Craig and Rob discuss why the floor is rising for entry-level work and why the ceiling may not be rising with it.

Entry-level hiring data. Recent IEEE Spectrum reporting suggests entry-level tech roles are growing in some categories, contradicting the prevailing narrative. The hosts walk through which roles and what the trend means for university programs preparing students for those jobs.

AI sycophancy in the wild. Rob shares why the tools' tendency to agree with the user's framing is more dangerous in high-stakes situations than in low-stakes ones, and what that means for how we should be using them.

Why timing matters more than access

The dominant question in higher education has been whether students should use AI. The CHI study suggests that's the wrong fight. The better question is when AI should appear in a student's thinking process.

Participants with late AI access in the study produced the same number of arguments as those without AI, but with more balanced pro-and-con reasoning. The tool became a counterweight to their own thinking rather than a substitute for it. That's a different mental model than the one most faculty (and most knowledge workers) default to, and it has practical implications for course design, assignment structure, and how we coach students to work with these tools.

Episode highlights

(approx. 5:41) Craig on the four-tool slide experiment: "A lot of times with AI, 50% is better than 100%, because you can get the 50% really quickly."
(approx. 17:10) Craig on what entry-level workers need now: "The solution to helping our students get jobs is to show them how to lean into their humanity."
(approx. 19:55) Rob on the floor-and-ceiling tension: "The floor is going up, but the fear is that our ceiling is coming down."
(approx. 31:15) Craig on the satisficing finding: "It's not literally where you stopped — it's where your engagement stopped."
(approx. 37:15) Rob's end-of-semester challenge to faculty: "Pick one thing. One thing that you're going to engage with over the summer."

Links and references

Computer Human Interaction (CHI) conference paper on AI access timing and decision quality (researchers from Chicago and Toronto): https://dl.acm.org/doi/pdf/10.1145/3772318.3791796
IEEE Spectrum reporting on entry-level technology hiring trends: https://spectrum.ieee.org/ai-effect-entry-level-jobs
Herbert Simon's concept of satisficing (1956)
Microsoft Office Agent, Claude Cowork (Design feature), ChatGPT, Gemini

For faculty: questions worth sitting with

Where in your course design does AI currently show up, and would your students be better off if it appeared later in their process?
How would you redesign one assignment so that students engage with the problem cold before the AI shows up?
What does excellence look like in your discipline now that "good enough" is trivially achievable? How will you recognize it, and how will you teach students to reach for it?

About the show

AI Goes to College is a podcast for higher education professionals trying to make sense of artificial intelligence in their classrooms, their research, and their institutions. Co-hosted by Craig Van Slyke and Rob Crossler, the show focuses on practical, evidence-based perspectives on AI in higher education without the hype.

Subscribe and listen: https://www.aigoestocollege.com/

Newsletter: https://aigoestocollege.substack.com/

Mentioned in this episode:

AI Goes to College Newsletter

When You Bring AI to the Party Matters More Than Whether You Bring It

AI Goes to College

When You Bring AI to the Party Matters More Than Whether You Bring It