Day 1 - Session 2: The ‘Confident’ Assistant

System prompts and the drunk tutor problem

Author

Dr Brian Ballsun-Stanton

Published

September 8, 2025

Welcome Back!

Where We’re Going

Control output style with system prompts
Discover the “drunk tutor” problem
Build our jargon-busting glossary
Start our annotation practice

What We’ll Learn This Session

By the end of today, you will be able to:

Demonstrate: How system prompts change style but not accuracy
Explain: Why AI confidence doesn’t signal correctness
Identify: Marketing fluff vs. technical necessity in AI terminology

Exercise: System Prompts and Copy Editing (35 min)

Your Task

Find a page of your own writing (email, report, anything)
Ask Claude to copy edit it three times with different system prompts
Compare what changes and what doesn’t

System Prompt Variations

You are a highly meticulous copy-editor specialising in German and English texts. You do not need to summarise or compliment me. You are a highly accurate copy-editor. It is more important to be useful than to be complimentary. The user’s primary language is German. You will take extra time and be extremely thoughtful in your discussions and considerations. Before responding to the user, make notes to yourself in tags.

User prompt

Please take the role of a highly meticulous copy-editor. You will take extra time and be extremely thoughtful in your discussions and considerations. The user, above all else, requires thoughtful and reasoned editing and discussion. > I would like you to highlight issues rather than making corrections. Then, for each issue, please thoughtfully discuss the issue. Make sure any suggested contributions (rare) are in bold so that we may quickly assess them. Your first task is to make sure the language is clear, consistent, smooth, and academically appropriate throughout the task. Our venue is an academic journal, so please thoughtfully and slowly ensure that our language is appropriate for that. Second: proceed slowly and thoughtfully paragraph by paragraph. We need to double-check that each paragraph is not only merely “grammatically correct” but that it has appropriate flow and academic use of speech. Beyond that, we also need to do a “thinko” check as well as the “typo” check – identifying places where we use terms or jargon before we describe them, or places that should have citations but do not. Third: Find places where we have been unclear, or have repeated ourselves (especially in distant paragraphs) – this paper has been edited multiple times and we want to make sure that it has effective academic flow. Make sure to think through your advice step by step. You do not need to summarise or compliment me. You are a highly accurate copy-editor. It is more important to be useful than to be complimentary. I’m going to paste in a long paper. My objective is for us to highlight issues with flow, clarity, and consistency, and for you to function as a highly attentive copy editor. Attached is a copy of the work I would like you to edit. To begin, please give me a readback of my instructions (with functional decomposition) and the headers of the document.

Put up a green sticky when you’ve completed all three variations. Pink sticky if you need a hand.

What Did You Notice?

Style Changes

Tone and register shift dramatically
Emoji density varies
Formality adjusts

What Stays Same

Factual claims (right or wrong)
Core suggestions
Confidence level

The Drunk Tutor Problem

“Always confident, usually correct”

What This Means

No correlation between tone and truth
Professional language can mask errors
Confidence remains constant regardless of knowledge

Why This Matters

You cannot use style as a proxy for accuracy.

Confabulation in action

(cross-reference LLM lecture slides “General Purpose Transformer” example) [https://osf.io/54sgp]

Jargon Busting (20 min)

Let’s Build Our Glossary

On the Conceptboard, add terms you’ve encountered: - LLM, API, token, context window… - Hallucination, temperature, fine-tuning… - AGI, singularity, emergence…

Together we’ll sort: ✓ Technical necessities
✗ Marketing buzzwords
? Contested concepts

Your First Annotation Task

Tonight’s Homework

Read: “ChatGPT is Bullshit” (Hicks 2024)
- [https://link.springer.com/article/10.1007/s10676-024-09775-5]
Use the following prompt to help with the paper:

Hi Claude. Today is the first day of class. My teacher assigned me ChatGPT is Bullshit by Hicks et al to read. Can you help me work through this text, section by section, and talk with me about each section. My teacher said to get you to force me to explain what the point of each section is before we continue. Specifically, ask me one pointed and specific question at a time until you think that I understand the section at hand. My first language is German, so please provide definitions in both languages for key terms and concepts, especially where it looks like I don’t understand. To begin, functionally decompose this task and give me a readback of our interaction patterns.

Print and annotate your prompts:
- Pink highlighter: what didn’t work
- Blue highlighter: what was effective
- Pen: notes on why

Tomorrow Morning

We’ll share these in “Show me your prompts!”

Looking Ahead

Tomorrow Morning: Basic Prompting

Your first “Show me your prompts” session
Rules of thumb for effective prompting
Building context windows intentionally

Tomorrow Midday: Metaprompting

Getting AI to write its own prompts
The epistemic humility problem
Why AI can’t judge its own capabilities

Answering the day’s question

Can we control AI output?

We can control AI output by: * Being mindful of our prompts * Giving it scaffolding

Control exists - but only over style and approach
Confidence is constant - regardless of correctness
Your judgment matters - AI won’t tell you when it’s wrong
Scaffolding - Make sure the AI knows what you want

Remember

The goal isn’t to memorize perfect prompts. It’s to develop intuition about what works and why.

Sticky Note Feedback

On your green sticky, write one specific thing we did well today
On your pink sticky, write one specific thing we can improve for tomorrow

Resources for Tonight

Paper link: https://link.springer.com/article/10.1007/s10676-024-09775-5
Annotation example: https://zenodo.org/records/15583013 page 11
Annotation guide: Focus on YOUR prompts, not AI output
Questions? Post in Conceptboard for tomorrow

See you at 9:00 tomorrow with your annotated prompts!

--- title: "Day 1 - Session 2: The 'Confident' Assistant" subtitle: "System prompts and the drunk tutor problem" author: - "Dr Brian Ballsun-Stanton" date: "8 September 2025" format: revealjs: slide-number: true chalkboard: true theme: [default, ../reveal-custom.scss] footer: "AI Summercamp 2025 - Basic Course" preview-links: auto transition: slide background-transition: fade highlight-style: github navigation-mode: vertical --- ## Welcome Back! ### Where We're Going - Control output style with system prompts - Discover the "drunk tutor" problem - Build our jargon-busting glossary - Start our annotation practice --- ## What We'll Learn This Session By the end of today, you will be able to: - **Demonstrate**: How system prompts change style but not accuracy - **Explain**: Why AI confidence doesn't signal correctness - **Identify**: Marketing fluff vs. technical necessity in AI terminology --- ## Exercise: System Prompts and Copy Editing (35 min) ### Your Task 1. Find a page of your own writing (email, report, anything) 2. Ask Claude to copy edit it three times with different system prompts 3. Compare what changes and what doesn't ### System Prompt Variations > You are a highly meticulous copy-editor specialising in German and English texts. You do not need to summarise or compliment me. You are a highly accurate copy-editor. It is more important to be useful than to be complimentary. The user's primary language is German. You will take *extra* time and be extremely thoughtful in your discussions and considerations. Before responding to the user, make notes to yourself in <think></think> tags. ### User prompt > Please take the role of a highly meticulous copy-editor. You will take *extra* time and be extremely thoughtful in your discussions and considerations. The user, above all else, requires thoughtful and reasoned editing and discussion. > I would like you to highlight issues rather than making corrections. Then, for each issue, please thoughtfully discuss the issue. Make sure any suggested contributions (rare) are in bold so that we may quickly assess them. > Your first task is to make sure the language is clear, consistent, smooth, and academically appropriate throughout the task. Our venue is an academic journal, so please thoughtfully and slowly ensure that our language is appropriate for that. > Second: proceed *slowly* and *thoughtfully* paragraph by paragraph. We need to double-check that each paragraph is not only merely "grammatically correct" but that it has appropriate flow and academic use of speech. Beyond that, we also need to do a "thinko" check as well as the "typo" check -- identifying places where we use terms or jargon before we describe them, or places that should have citations but do not. > Third: Find places where we have been unclear, or have repeated ourselves (especially in distant paragraphs) -- this paper has been edited multiple times and we want to make sure that it has effective academic flow. Make sure to think through your advice step by step. > You do not need to summarise or compliment me. You are a highly accurate copy-editor. It is more important to be useful than to be complimentary. I'm going to paste in a long paper. My objective is for us to highlight issues with flow, clarity, and consistency, and for you to function as a highly attentive copy editor. > Attached is a copy of the work I would like you to edit. > To begin, please give me a readback of my instructions (with functional decomposition) and the headers of the document. Put up a green sticky when you've completed all three variations. Pink sticky if you need a hand. --- ## What Did You Notice? ### Style Changes - Tone and register shift dramatically - Emoji density varies - Formality adjusts ### What Stays Same - Factual claims (right or wrong) - Core suggestions - Confidence level --- ## The Drunk Tutor Problem > "Always confident, usually correct" ### What This Means - No correlation between tone and truth - Professional language can mask errors - Confidence remains constant regardless of knowledge ### Why This Matters You cannot use style as a proxy for accuracy. --- ## Confabulation in action (cross-reference LLM lecture slides "General Purpose Transformer" example) [https://osf.io/54sgp] --- ## Jargon Busting (20 min) ### Let's Build Our Glossary On the Conceptboard, add terms you've encountered: - LLM, API, token, context window... - Hallucination, temperature, fine-tuning... - AGI, singularity, emergence... Together we'll sort: ✓ Technical necessities ✗ Marketing buzzwords ? Contested concepts --- ## Your First Annotation Task ### Tonight's Homework 1. Read: "ChatGPT is Bullshit" (Hicks 2024) - [https://link.springer.com/article/10.1007/s10676-024-09775-5] 2. Use the following prompt to help with the paper: > Hi Claude. > Today is the first day of class. My teacher assigned me ChatGPT is Bullshit by Hicks et al to read. Can you help me work through this text, section by section, and talk with me about each section. My teacher said to get you to force me to explain what the point of each section is before we continue. Specifically, ask me one pointed and specific question at a time until you think that I understand the section at hand. My first language is German, so please provide definitions in both languages for key terms and concepts, especially where it looks like I don't understand. > To begin, functionally decompose this task and give me a readback of our interaction patterns. 3. Print and annotate your prompts: - Pink highlighter: what didn't work - Blue highlighter: what was effective - Pen: notes on why ### Tomorrow Morning We'll share these in "Show me your prompts!" --- ## Looking Ahead ### Tomorrow Morning: Basic Prompting - Your first "Show me your prompts" session - Rules of thumb for effective prompting - Building context windows intentionally ### Tomorrow Midday: Metaprompting - Getting AI to write its own prompts - The epistemic humility problem - Why AI can't judge its own capabilities --- ### Answering the day's question **Can we control AI output?** We can control AI output by: * Being mindful of our prompts * Giving it scaffolding 1. **Control exists** - but only over style and approach 2. **Confidence is constant** - regardless of correctness 3. **Your judgment matters** - AI won't tell you when it's wrong 4. **Scaffolding** - Make sure the AI knows what you want ### Remember The goal isn't to memorize perfect prompts. It's to develop intuition about what works and why. --- ## Sticky Note Feedback * On your green sticky, write one specific thing we did well today * On your pink sticky, write one specific thing we can improve for tomorrow --- ## Resources for Tonight - Paper link: https://link.springer.com/article/10.1007/s10676-024-09775-5 - Annotation example: https://zenodo.org/records/15583013 page 11 - Annotation guide: Focus on YOUR prompts, not AI output - Questions? Post in Conceptboard for tomorrow See you at 9:00 tomorrow with your annotated prompts!