IntroductoryEconometrics · Chapter 1About 10 min

An AI tutor promises twelve more points,and you ask who chose to use it

Twelve points

March. Final year at school on Halcyon. Exams in June.

Half your class uses Lumi, an AI tutoring app. It explains homework at midnight and never gets tired. At the bus stop, its advert covers the whole shelter.

1. A
   AriTwelve points! That’s a whole grade. I’m subscribing tonight.
2. You
   YouTwelve points more than who?
3. A
   AriThan people who don’t use it. It says so right there.
4. You
   YouSo, people who chose to use it. Who are they?
5. A
   AriYou’re going to make this complicated, aren’t you.

The small print: last year’s average exam score (out of 100) of 3,000 Lumi users, against everyone else.

Predict first

Every student on Halcyon starts using Lumi next year. How much does the average score rise?

Who chose the app

You want one thing: each student’s score with Lumi and without it. You only ever see one of the two.

So the advert compares two groups. Nobody assigned them. Students chose. Step through the diagram.

1. The advert: Lumi → +12 points.
2. But students differ in motivation.
3. Motivated students pick Lumi.
4. They also score higher anyway.
5. A lottery breaks that link. Lumi’s real effect: +4.

In symbols, a student’s score is:

\text{score} = \beta\,T + \gamma\,m + e

• T = 1 if the student uses Lumi.
• \beta = Lumi’s real effect.
• m = motivation. Nobody measured it.
• \gamma = how much motivation raises the score.

If Lumi users are more motivated by a gap \Delta m, the advert’s comparison gives:

\bar y_{\text{Lumi}} - \bar y_{\text{others}} = \beta + \gamma\,\Delta m

Motivation is a confounder (or omitted variable). Separating \beta from \gamma\,\Delta m is what econometrics is about.

The bench

Set Lumi’s real effect, how much motivation matters, and the motivation gap. Press Play to watch the comparison grow with the gap.

Model bench — what the comparison measures

Challenge. Make the comparison tell the truth: get it to equal Lumi’s real effect.

Illustrative numbers for a fictional app on a fictional island.

At the start: Lumi adds 4 points. The comparison says 12.

Drag the gap below zero. Now struggling students chose Lumi. The comparison drops below the real effect.

An AI’s answer

Many students would just ask an AI. Here is the kind of answer they get. Fluent. Mostly right. Wrong in one place.

Spot the error

Question to an AI assistant Lumi users score 12 points higher than other students. Does Lumi raise exam scores by 12 points?

Answer (written for this exercise)

¹ The figure compares the average scores of Lumi users and non-users. ² It covers 3,000 Lumi users, so random noise in the averages is small. ³ Because the sample is so large, the 12 points measure the effect of using Lumi. ⁴ A trial that gave Lumi out by lottery would be another way to measure the effect.

Which sentence is wrong?

Sentence 1 Sentence 2 Sentence 3 Sentence 4

A fair test

Let chance decide who gets the app.

Say the schools have 500 free Lumi licences and 1,500 applicants. Draw lots. Winners and losers differ only by luck. The gap in their scores is Lumi’s real effect.

That is how new medicines are tested. Economists look for cases where something like a lottery happened on its own.

You write to the evening paper: The 12 points compare two kinds of students. A lottery would show what Lumi adds. Next morning, same bus stop:

1. A
   AriI read your letter. So Lumi is useless?
2. You
   YouNo. It might help — a few points, maybe. The advert just can’t tell how many.
3. A
   AriAnd a lottery could?
4. You
   YouYes. Or anything that makes the two groups alike to start with.
5. A
   AriFine. I’m still subscribing. But now I know what I’m paying for.

Your prediction

You did not make a prediction at the start of the chapter. Here is what it asked:

Every student on Halcyon starts using Lumi next year. How much does the average score rise?

The 12 points describe two groups. How much of it Lumi causes depends on who chose Lumi.

Check yourself

Which design makes Lumi users and non-users alike in motivation?

Compare everyone who chose Lumi with everyone who didn’t Give Lumi by lottery among students who apply Ask students whether Lumi helped them

Real effect 3 points. One s.d. of motivation adds 5 points. Lumi users are 0.6 s.d. more motivated. What does the naive comparison show, in points?

Your answer

Struggling students are the most likely to use Lumi. The naive comparison will:

Overstate Lumi’s effect Understate Lumi’s effect Get it right

Take it with you

This chapter on two pages

The ideas, the formula, a worked example, four traps — and six questions to try on your own, with answers.

Download the sheet (PDF) 2 pages · 160 KB

Take it to your AI

Want to go further? Paste this into the AI you use. It asks the AI to quiz you, not to hand you answers.

I am learning the difference between correlation and causation. An advert says: "Students who use an AI tutoring app score 12 points higher in their final exams than students who do not."
Please act as my tutor. Do not give me the answers. Ask me one question at a time to help me work out:
1. why this comparison may not measure the effect of the app,
2. in which direction it could be wrong, and
3. how a study could measure the effect properly.
Wait for my answer after each question. At the end, tell me what I got right and correct anything I got wrong.

Key points

• A gap between two groups is a fact. Whether one thing caused it is a different question.
• When people choose their group, something else can drive both the choice and the result. That is a confounder.
• Naive comparison = \beta + \gamma \times (gap in the confounder).
• A bigger sample makes a comparison more precise. It does not make it less biased.
• A lottery makes the groups alike. Then the gap is the real effect.

The evening paper prints your letter. The editor replies: then how do you measure an effect from data? To be continued — Chapter 2: simple linear regression.