05: Hypothesis Testing
Hypothesis Testing
Transcript
Every test in this lecture follows these same five steps, so use this slide as a map. Step one is to state the null hypothesis, H zero, and the alternative, H one. The null is the claim you subject to a test; the alternative describes the departure that would make you reject it. You must specify both before looking at the result, because the alternative determines whether one tail or both tails count as evidence.
Step two is the intellectual core. Choose a test statistic and find its sampling distribution under the assumption that H zero is true. The red phrase matters twice on this slide. We are not asking for the statistic’s distribution under every possible state of the world. We temporarily impose the null so that we have a reference distribution against which an observed result can be judged. If you do not know that null distribution, you do not yet have a formal test.
In step three, calculate the statistic from your sample and regression output. In step four, choose the significance level, such as five percent. That is the long-run Type I error rate you are willing to tolerate, and it determines a rejection region or critical value. Finally, in step five, ask how unusual the statistic from step three would be in the null distribution from step two. If it is sufficiently extreme, you reject H zero in favor of H one. If it is not, you fail to reject H zero. You do not prove the null true. Keep these five steps in view, because the z, t, and F tests later differ mainly in the statistic and reference distribution they use.
Here is the general step of any hypothesis testing:
Step 1: specify the null (H_0) and alternative (H_1) hypotheses
Step 2: find the distribution of the test statistic if the null hypothesis is true
Step 3: calculate the test statistic based on the data and regression results
Step 4: define the significance level
Step 5: check how unlikely that you get the actual test statistic (found at Step 3) if indeed the null hypothesis is true
Transcript
This is a deliberately artificial example, and the artificiality lets us see the testing logic without having to estimate a variance first. The goal at the top says that x is Normally distributed and that we want to test whether its population expected value is one. I will call the unknown population mean alpha. The phrase “or not” makes this a two-sided question: values of alpha either above or below one would count against the null.
The state of knowledge gives us unusually strong information. We know the population distribution is Normal and we somehow know its variance is four, so its standard deviation is two. In real applications, a population variance is almost never handed to us. Later, when we replace an unknown standard deviation with an estimate, the reference distribution changes from Normal to t. For now, knowing the variance keeps that complication out of the way.
Our estimator is theta, the sample mean. Read the formula as theta equals the sum of x sub i from i equals one through J, divided by J. Here J is the number of observations, x sub i is observation i, alpha is the population mean we want to learn about, and theta is the random estimator computed from whichever sample we happen to draw. From the earlier lecture, you know theta is unbiased, so its expected value is alpha. But one observed sample mean will almost never equal alpha exactly. The testing question is therefore not simply whether theta equals one. It is whether theta is far enough from one, relative to its sampling variation, that the discrepancy would be hard to explain if alpha really were one. Flip to the distribution tab to quantify that sampling variation.
Goal
Suppose you want to test if the expected value of a normally distributed random variable (x) is 1 or not.
State of Knowledge
We do know x follows a normal distribution and its variance is 4 for some reason.
Your Estimator
Your estimator is the sample mean: \theta = \sum_{i=1}^J x_i/J
Transcript
Follow the two math asides carefully, because together they produce the statistic we will test. Math Aside 1 says that if you multiply a random variable X by a constant a, its variance is multiplied by a squared. This is why standard deviations scale by the absolute value of a. Math Aside 2 says that subtracting a constant shifts a Normal distribution’s mean but does not change its variance. Thus, if x follows a Normal distribution with mean a and variance b, x minus a has mean zero and variance b. Dividing that centered variable by root b, its standard deviation, gives a standard Normal variable with mean zero and variance one.
Now apply those facts to theta, the sample mean of J independent observations. Each x sub i follows Normal with mean alpha and variance four. A sum of independent Normal variables is Normal, and dividing the sum by J keeps it Normal. Its mean is alpha. For the variance, theta is one over J times each observation, so each term contributes one over J squared times four. There are J independent terms, so the variances add: J times one over J squared times four equals four over J. That justifies both the distribution shown near the top, theta follows Normal with mean alpha and variance four over J, and the first bullet at the bottom.
The standard deviation of theta is the square root of four over J, which is two over root J. Standardize theta by subtracting its mean alpha and dividing by two over root J. Dividing by a fraction is the same as multiplying by its reciprocal, so the expression becomes root J over two, times theta minus alpha. The last line says this transformed quantity follows the standard Normal distribution. Alpha is still unknown at this stage, so we know the mathematical distribution but cannot yet calculate the statistic from data. The next tab shows how imposing the null supplies the missing value.
So, we know that \theta \sim N(\alpha, 4/J) (of course \alpha is not known).
So, \frac{x-a}{\sqrt{b}} \sim N(0, 1) (combined with Math Aside 1)
This means,
Since \theta = \sum_{i=1}^J x_i/J and x_i \sim N(\alpha, 4),
- Var(\theta) = J \times \frac{1}{J^2}Var(x) = 4/J
- \frac{\sqrt{J}}{2} \cdot (\theta - \alpha)\sim N(0, 1).
Transcript
Here is where the null hypothesis actually enters the calculation. On the previous tab we established that root J over two, times theta minus alpha, follows a standard Normal distribution. Alpha is the unknown population mean, so that expression cannot be evaluated directly. The null hypothesis supplies a particular value: H zero says alpha equals one. If that claim is true, substitute one for alpha. The computable test statistic is therefore root J times theta minus one, divided by two.
Read the verbal recipe in the left column in exactly that order. Start with the sample mean theta. Subtract the value asserted by the null, which is one. The red phrase is there because students often forget that centering step and test the magnitude of the estimate rather than its distance from the null value. Multiply the resulting gap by the square root of the sample size J, then divide by the known population standard deviation, two. Equivalently, divide theta minus one by the standard error of the sample mean, two over root J. The statistic therefore measures how many standard errors the estimate lies from the null value.
Now look at the figure in the right column. The horizontal axis contains possible values of this standardized statistic, and the vertical height is the standard Normal density. The curve is centered at zero and symmetric. Values near zero are common under H zero, while values far into either tail are rare. Because the goal was to test whether alpha is one “or not,” large positive and large negative values would both count as evidence. The curve is not the distribution of the raw x values or of theta itself. It is the distribution of the transformed statistic, conditional on the null being true. The next two tabs put actual sample means into this formula.
We established that \frac{\sqrt{J}}{2} \cdot (\theta - \alpha)\sim N(0, 1).
The null hypothesis is \alpha = 1.
If \alpha = 1 is indeed true, then \sqrt{J} \times (\theta - 1)/2 \sim N(0, 1).
In other words, if you take the sample mean, subtract the value under the null, multiply by the square root of the number of observations, and divide by 2, the result follows the standard Normal distribution like below.
Transcript
Now substitute the numbers from Case 1. The sample contains J equals one hundred observations, and its sample mean theta is two. Under H zero, the comparison value is one. Root one hundred is ten, the difference two minus one is one, and dividing ten times one by two gives a test statistic of five. In other words, the observed sample mean is five standard errors above the mean asserted by the null, because the standard error here is two over root one hundred, or zero point two.
Look at the standard Normal curve in the right column. Its horizontal axis shows possible values of the test statistic and its vertical height shows density. The displayed range ends around four, so a value of five is not even visible. It lies extremely far into the positive tail. Under a standard Normal distribution, an observation five standard deviations from zero is extraordinarily rare. For this two-sided question, we would count an equally extreme negative result as well.
The logic is a comparison between two explanations. One possibility is that alpha truly equals one and our sample happened to produce an extraordinarily unusual statistic. The other is that alpha does not equal one, so centering the statistic at one was inappropriate and the extreme value reflects a false null. Hypothesis testing controls how unusual a result must be before we choose the second interpretation and reject. This case clearly provides strong evidence against H zero, but we have not yet established a formal cutoff. The significance-level section will define exactly how much tail probability we are willing to accept.
Suppose you have obtained 100 samples (J = 100) and calculated \theta (sample mean), which turned out to be 2.
Then, your test statistic is \sqrt{100} \times (2-1)/2 = 5.
How unlikely is it to get the number you got (5) if the null hypothesis is indeed true?
Transcript
Case 2 uses exactly the same statistic and null distribution, but different data. Here J is four hundred and the sample mean theta is one point zero two. Root four hundred is twenty. The sample mean exceeds the null value by only zero point zero two. Multiply twenty by zero point zero two and divide by two, and the test statistic is zero point two. Equivalently, the sample mean’s standard error is two over root four hundred, or zero point one, so the observed mean lies zero point two standard errors above one.
Find zero point two on the horizontal axis of the standard Normal curve. It sits very close to zero, in the high, central part of the density. Values around zero point two, and values even farther from zero, occur routinely when the null is true. This sample therefore gives us no compelling reason to doubt alpha equals one. The correct conclusion is to fail to reject the null, not to prove or accept it. A different sample could lead to a different estimate and decision.
Compare this tab with Case 1. The sample here is four times as large, which cuts the standard error in half, but the raw discrepancy from the null is only zero point zero two rather than one. A larger sample makes a given raw gap more statistically noticeable, yet it cannot manufacture evidence when the observed gap is tiny. The test statistic combines both ingredients: the size of the estimate’s departure from the null and the precision with which that departure was estimated. This is why looking only at a coefficient or sample mean, without its standard error, is never enough.
Suppose you have obtained 400 samples (J = 400) and calculated \theta (sample mean), which turned out to be 1.02.
Then, your test statistic is \sqrt{400} \times (1.02-1)/2 = 0.2.
How unlikely is it to get the number you got (0.2) if the null hypothesis is indeed true?
Transcript
This aside explains why we standardize even though standardization is not logically required. In the left column, the alternative statistic is simply theta minus alpha, the sample mean’s raw estimation error. Because theta follows Normal with mean alpha and variance four over J, subtracting alpha gives a Normal distribution with mean zero and variance four over J. That is a perfectly valid null reference distribution. The complication is that it changes with J.
The figure on the right uses J equals one hundred. The horizontal axis is the raw difference theta minus alpha, and the vertical height is the density of a Normal distribution centered at zero with variance four over one hundred, which is zero point zero four, and standard deviation zero point two. That is why the curve is much narrower and taller than the standard Normal curve. Most raw differences should fall close to zero when the null is true.
Now reconsider Case 1 below the figure. Theta was two and the null value of alpha was one, so the raw statistic is two minus one, or one. On this distribution, one is five standard deviations above zero because one divided by zero point two equals five. That is exactly the standardized statistic we calculated before. The conclusion cannot depend on whether we measure the gap in original units or standard-error units, as long as we compare it with the matching distribution.
The callout gives the practical reason for normalization. Root J times theta minus alpha, divided by two, always follows the same standard Normal distribution, regardless of sample size. The unnormalized theta minus alpha requires a different variance, four over J, for every J. Standardization lets you reuse one reference distribution and its quantiles. Later, regression t-statistics do the same thing: subtract the null value and divide by an estimated standard error so coefficients measured in completely different units can be judged against a common t distribution.
Note that you do not really need to use \sqrt{J} \times (\theta - \alpha)/2 as your test statistic.
You could alternatively use \theta - \alpha. But, in that case, you need to be looking at N(0, 4/J) instead of N(0, 1) to see how unlikely you get the number you got.
For example, when the number of observations is 100 (J = 100), the distribution of \theta-\alpha looks like the figure on the right.
Reconsider the case 1
Suppose you have obtained 100 samples (J = 100) and calculated \theta (sample mean), which turned out to be 2.
Then, your test statistic is 2-1 = 1.
Is it unlikely for you to get 1 if the null hypothesis is true?
The conclusion would be exactly the same as using \sqrt{J} \times (\theta - \alpha)/2 because the distribution under the null is adjusted according to the test statistic you use.
Transcript
This summary separates what a hypothesis test requires from what we usually do to meet that requirement. First, we need a test statistic whose probability distribution is known when the null hypothesis is true. A test statistic is a random quantity before the sample is drawn and a number after we insert our data. The null distribution tells us which realized numbers are ordinary and which are unusual under H zero. The examples listed here, a t distribution and a Normal distribution, are reference distributions we can use to assign tail probabilities or find critical values.
Second, we often have to transform the raw random variable we care about. In the example, the raw object was theta, the sample mean. Its distribution depended on the unknown alpha and on the sample size. We centered it by subtracting the null value and scaled it by its standard error. Under H zero, that transformed statistic followed the standard Normal distribution. In regression, the raw object will be a coefficient estimate. We will again subtract the coefficient value under the null and divide by a standard error, producing a t-statistic.
Once the statistic has a known null distribution, compute its realized value and ask whether a draw at least that extreme is plausible under H zero. “Extreme” must match the alternative: the upper tail for a positive one-sided alternative, the lower tail for a negative one-sided alternative, and both tails for a two-sided alternative. The parenthetical note in the last bullet identifies the missing piece. We have not formally defined how small a probability must be before we reject. That threshold is the significance level, which later converts the general phrase “unlikely” into a reproducible decision rule.
What do we need?
- test-statistic of which we know the distribution (e.g., t-distribution, Normal distribution) assuming the null hypothesis
What do we (often) do?
- transform (most of the time) a raw random variable (e.g., sample mean in the example above) into a test statistic of which we know the distribution assuming that the null hypothesis is true
- e.g., we transformed the sample mean so that it follows the standard Normal distribution.
- check if the actual number you got from the test statistic is likely to happen or not (formal criteria has not been discussed yet)
Transcript
Your turn. This exercise has the same structure as the worked example, but the numbers change. Let x denote the annual-salary difference between graduates of University A and University B. The population model says x follows a Normal distribution with unknown mean alpha and known variance sixteen, so its population standard deviation is four. The substantive question is whether the average salary difference is ten, which makes the null H zero: alpha equals ten. Because the wording asks whether it is ten rather than whether it is greater or smaller, treat departures on either side as relevant.
For part two, let theta be the sample mean of four hundred independent observations. Its mean is alpha and its variance is the individual variance divided by the sample size: sixteen over four hundred, not sixteen. Its standard deviation, or standard error, is root sixteen over four hundred, which is four over root four hundred, or zero point two.
For part three, turn the sample mean into standard-error units. Subtract alpha and divide by four over root four hundred. That is the same as multiplying theta minus alpha by root four hundred over four. Under the Normal model, the result follows a standard Normal distribution. Under the null, replace alpha with ten so the statistic is computable.
Part four gives an observed sample mean of ten point two. Convert that to the standardized statistic, then use pnorm to find an upper-tail probability. Be precise about the question: it asks for the probability of observing a number greater than ten point two, not the two-sided probability of being at least zero point two away from ten. Try all four parts before opening the Answer tab. Finally, read the callout. In real work, the variance is not magically known, and we still need a formal rule for rejecting or failing to reject. Those two complications motivate the t distribution and significance levels in the rest of the lecture.
You have collected data on annual salary for those who graduated from University A and B. You are interested in testing whether the difference in annual salary between the universities (call it x) is 10 on average. You know (for unknown reasons) that the difference is distributed as N(\alpha, 16).
- What is the null hypothesis?
- Under the null hypothesis, what is the distribution of the sample mean when the number of observation is 400?
- Normalize the test statistic so that the transformed version follows N(0, 1).
- The actual difference you observed is 10.2. What is the probability that you observe a number greater than 10.2 if the null hypothesis is true? Use
pnorm().
Transcript
Check each line against your solution. Line one states the null hypothesis: alpha equals ten. An alternative is not printed in the answer, but the original goal of testing whether the average is ten would normally pair it with alpha not equal to ten.
Line two gives theta’s sampling distribution as Normal with mean alpha and variance sixteen over four hundred. The variance comes from adding the variances of four hundred independent observations and then scaling their sum by one over four hundred. Numerically, sixteen over four hundred is zero point zero four, so the standard error is its square root, zero point two.
Line three standardizes theta. Read root four hundred over sixteen, times theta minus alpha. Root of the fraction is root four hundred divided by root sixteen, which is twenty divided by four, or five. That multiplier is the reciprocal of the standard error. After centering and scaling, the statistic follows the standard Normal distribution. Under H zero, alpha is replaced by ten.
Line four inserts the observed sample mean ten point two. Five times ten point two minus ten equals five times zero point two, which is one. The code cell calculates one minus pnorm of one. pnorm(1) is the standard Normal probability to the left of one, so subtracting from one gives the probability to the right. Running it returns about zero point one five nine, or fifteen point nine percent. Thus, if alpha truly equals ten, a sample mean greater than ten point two occurs about sixteen percent of the time. That is roughly one sample in six, so the upper-tail result is not unusual. If we were conducting the two-sided test suggested by “ten or not,” we would also count the equally extreme lower tail and double this symmetric tail probability, giving about thirty-one point seven percent. The exercise explicitly asked only for the probability above ten point two, which is why the displayed code uses one tail.
- \alpha = 10
- \theta \sim N(\alpha, 16/400)
- \sqrt{\frac{400}{16}}\cdot (\theta - \alpha)
- The test statistic is 5 \times (10.2 - 10) = 1
Details of hypothesis testing on \beta_j
Transcript
The courtroom analogy is useful because it makes the asymmetry between the two hypotheses explicit. Imagine that you are an attorney arguing that a defendant committed a crime. The null hypothesis corresponds to “the defendant is not guilty,” the position that remains unless the evidence crosses a prespecified threshold. The alternative corresponds to “the defendant is guilty,” the claim for which you are presenting evidence.
In a statistical application, the null is often a benchmark such as beta j equals zero, meaning no partial effect. The alternative states what would contradict that benchmark, such as beta j not equal to zero or beta j greater than zero. We compute how compatible the observed evidence is with the null. If the evidence would be sufficiently rare under H zero, we reject H zero in favor of H one.
Notice what the procedure does not say. A court that does not convict has not proved innocence; it has concluded that the evidence did not satisfy the burden of proof. Likewise, a statistical test that does not reject H zero has not proved the null. The proper phrase is “fail to reject,” not “accept.” Failure can arise because the null is true, because the real effect is too small for the current sample to detect, or because the estimate is too imprecise.
The analogy also previews the two errors discussed later. Rejecting a true null is like convicting an innocent defendant, a Type I error. Failing to reject a false null is like failing to convict a guilty defendant, a Type II error. The significance level controls the first risk, not the probability that the null itself is true. Keep that burden-of-proof logic in mind whenever an insignificant coefficient tempts you to say “there is no effect.”
Statistical hypothesis testing involves two hypotheses: Null and Alternative hypotheses.
Pretend that you are an attorney who indicted a defendant who you think committed a crime.
Null Hypothesis
Hypothesis that you would like to reject (defendant is not guilty)
Alternative Hypothesis
Hypothesis you are in support of (defendant is guilty)
Transcript
This tab contrasts two shapes of alternative hypothesis. In the one-sided example, H zero says beta j equals zero and H one says beta j is greater than zero. The alternative is directional, so only large positive t-statistics support it. A large negative statistic may contradict equality, but it points in the direction opposite to the stated alternative and therefore is not in this upper-tail rejection region. If the alternative were beta j less than zero, we would instead examine the lower tail.
In the two-sided example, H zero again says beta j equals zero, but H one says beta j does not equal zero. Now sufficiently large departures in either direction count as evidence. A positive t-statistic indicates an estimate above zero and a negative statistic indicates one below zero, so the rejection region occupies both tails of the t distribution.
The choice affects critical values. Suppose the total significance level is five percent. An upper-tailed one-sided test puts the entire five percent in the upper tail, giving a critical value near the ninety-fifth percentile. A two-sided test splits the five percent across two tails, two point five percent in each, so its positive boundary is the ninety-seven point five percentile and its negative boundary is the two point five percentile. Because less tail area lies beyond each boundary, the absolute two-sided critical value is farther from zero. This means a one-sided test has more power in its prespecified direction, but it cannot be chosen after seeing which sign the estimate has.
Read the callout at the bottom. A critical value is simply a quantile of the null distribution. In Lecture 03, qnorm(0.95) returned about one point six four, the ninety-fifth percentile of the standard Normal. Here we use qt instead of qnorm because estimating sigma puts the statistic on a t distribution, and we also supply its degrees of freedom. The following significance-level tabs turn these tail choices into formal decision rules.
one-sided alternative
H_0: \beta_j = 0 H_1: \beta_j > 0
You look at the positive end of the t-distribution to see if the t-statistic you obtained is more extreme than the level of error you accept (significance level).
two-sided alternative
H_0: \beta_j = 0 H_1: \beta_j \ne 0
You look at both ends of the t-distribution to see if the t-statistic you obtained is more extreme than the level of error you accept (significance level).
Transcript
The significance level formalizes how unlikely a result must be before we reject. The definition callout says it is the probability of rejecting H zero when H zero is actually true. That is a Type I error, analogous to convicting an innocent defendant. If you choose a five percent level and repeatedly apply the procedure in situations where the null is true, the procedure is designed to reject about five percent of the time in the long run.
A lower level requires stronger evidence. Moving from five percent to one percent pushes the critical value farther into the tail, so fewer test statistics enter the rejection region when H zero is true. This reduces the long-run Type I error rate. It does not mean there is a one or five percent probability that the null is true. Frequentist significance treats the parameter and hypothesis as fixed and describes the behavior of the testing procedure across hypothetical repeated samples.
Now read the table by rows and columns. If H zero is true and you reject it, you make a Type I error. If it is true and you do not reject it, the decision is correct. If H zero is false and you reject it, you correctly detect a real departure. If it is false and you do not reject it, you make a Type II error. The probability of Type II error depends on the actual alternative value, the sample size, noise, collinearity, and the chosen significance level.
The bullets explain the trade-off. Holding the design and sample size fixed, making rejection harder lowers Type I error but generally increases Type II error, so more real effects go undetected. One minus the probability of Type II error is power, the chance that the test rejects when a specified real effect exists. This is why “not statistically significant” does not mean “no effect.” It may mean that the study had too little power to distinguish the effect from sampling noise. The next tabs show how a chosen level marks a rejection area on the null distribution.
A lower significance level requires stronger evidence before rejecting the null and reduces the long-run probability of a Type I error. It does not tell us the probability that the null hypothesis is true.
But lowering it is not free
There are two ways to be wrong, and they trade off against each other:
| H_0 is true | H_0 is false | |
|---|---|---|
| You reject H_0 | Type I error (probability = significance level) | correct |
| You do not reject H_0 | correct | Type II error |
- the significance level is a choice about how much Type I error you will tolerate
- pushing it down from 5% to 1% moves the critical value further out, so you reject less often — which means more Type II error: real effects you fail to detect
- 1 - P(\text{Type II error}) is called the power of the test: the chance of detecting an effect that is genuinely there
- this is why “not statistically significant” does not mean “no effect”. It may just mean your test had too little power to see it
Transcript
This picture implements a one-sided upper-tail test at the five percent significance level. The quantity in the first bullet is beta j hat minus beta j, divided by its estimated standard error. Under the null beta j equals zero, so the numerator becomes beta j hat and, under the classical assumptions, the statistic follows the t distribution shown on the right with five hundred and twenty-two residual degrees of freedom.
On the figure, the horizontal axis contains possible t-statistics and the vertical height is the t density. The curve is centered at zero and symmetric, although this test uses only its positive tail because H one is beta j greater than zero. The red ribbon is the upper five percent of the total area. Its left edge, about one point six five for this many degrees of freedom, is the ninety-fifth percentile of the t distribution. That boundary is the critical value.
The decision callout says to reject H zero if the observed t-statistic is greater than that critical value. If H zero is true, only five percent of repeated samples generate a statistic in the shaded region. Therefore the rule has a five percent Type I error probability. A statistic in the red region can still occur under a true null, so rejection is not certainty. It means the result crossed the evidence threshold we chose.
Keep the two red terms below the figure distinct. Five percent is a probability and is the significance level, chosen before the test. About one point six five is a value on the t-statistic scale and is the critical value implied by that choice and by the degrees of freedom. With different degrees of freedom, the same five percent level produces a different critical value. With a lower-tail alternative we would use the negative counterpart; with a two-sided alternative we would shade both tails.
Figure on the right presents the distribution of \frac{\widehat{\beta}_j-\beta_j}{\widehat{se(\widehat{\beta}_j)}} if \beta_j = 0 (the null hypothesis is true).
The probability that you get a value larger than 1.65 is 5% (0.05 in area).
If you follow the decision rule, then you have a 5% chance that you are wrong in rejecting the null hypothesis of \beta_j = 0. Here,
- 5% is the significance level
- 1.65 is the critical value above which you will reject the null
Transcript
This tab keeps the same one-sided hypothesis and the same t distribution but changes the significance level from five percent to one percent. The statistic is still beta j hat minus beta j, divided by its estimated standard error, and under H zero beta j equals zero. The curve on the right therefore has exactly the same shape as on the previous tab.
What changes is the shaded rejection region. The red area now contains only the upper one percent of the distribution. Its left boundary is the ninety-ninth percentile of t with five hundred and twenty-two degrees of freedom, about two point three three. The decision rule is consequently stricter: reject only if the observed t-statistic exceeds about two point three three. Compare that with the roughly one point six five boundary at the five percent level.
If H zero is true, this rule rejects only one percent of the time in repeated samples, so its Type I error probability is one percent. Requiring a statistic farther from zero makes false rejection less frequent, but also makes a false null harder to detect. Holding everything else fixed, the one percent test has less power than the five percent test against the same positive alternative.
Again, distinguish the level from the critical value. One percent is the chosen tail probability. About two point three three is the t-statistic cutoff created by that probability and these degrees of freedom. A result between one point six five and two point three three would reject at five percent but fail to reject at one percent. That is not a contradiction. It describes intermediate strength of evidence, which a p-value will summarize more precisely. No significance level is universally correct; five percent is a convention, and the cost of the two error types should guide the choice when stakes differ.
- The figure on the right presents the distribution of \frac{\widehat{\beta}_j-\beta_j}{\widehat{se(\widehat{\beta}_j)}} if \beta_j = 0 (the null hypothesis is true).
- The probability that you get a value larger than 2.33 is 1% (0.01 in area).
If you follow the decision rule, then you have a 1% chance that you are wrong in rejecting the null hypothesis of \beta_j = 0. Here,
- 1% is the significance level
- 2.33 is the critical value above which you will reject the null
Transcript
Now the alternative is two-sided, so both positive and negative departures from zero count against H zero. The statistic and its null distribution have not changed: beta j hat minus beta j, divided by the estimated standard error, follows a t distribution with five hundred and twenty-two degrees of freedom under beta j equals zero. The parenthetical note emphasizes that every figure in this sequence uses the same t curve. The hypothesis changes the rejection region, not the statistic’s null distribution.
The total significance level is still five percent. Because evidence in either direction matters, split that probability evenly: two point five percent in the left tail and two point five percent in the right tail. The positive boundary is the ninety-seven point five percentile of the t distribution, about one point nine six, and symmetry puts the negative boundary at about minus one point nine six. Those are farther from zero in absolute value than the one-sided five percent boundary because each tail receives less area.
Look at the figure. The horizontal axis is the t-statistic, the vertical height is density, and both extreme ends are shaded red. The unshaded center contains ninety-five percent of the null distribution. The annotation in the middle identifies the tail allocation, while the number near the right edge marks the positive critical value. The left edge is its negative counterpart even if only one label is printed.
The decision callout expresses both boundaries compactly: reject if the absolute value of the observed t-statistic is greater than about one point nine six. Taking the absolute value treats equally strong positive and negative evidence the same way. Under a true null, the probability of landing in either shaded region is five percent in total, which is the Type I error rate. Five percent is the significance level; one point nine six is the absolute critical value for these degrees of freedom. Do not halve the critical value or compare a signed statistic only with the positive boundary without taking its absolute value.
The figure on the right presents the distribution of \frac{\widehat{\beta}_j-\beta_j}{\widehat{se(\widehat{\beta}_j)}} if \beta_j = 0 (the null hypothesis is true).
The probability that you get a value more extreme than 1.96 or -1.96 is 5% (0.05 in area combining the two areas at the edges).
(Note: irrespective of the type of test, the distribution of the t-statistic is the same — every figure on this slide is the same t_{522} curve. Only the shaded region changes.)
If you follow the decision rule, then you have a 5% chance that you are wrong in rejecting the null hypothesis of \beta_j = 0. Here,
- 5% is the significance level
- 1.96 is the critical value beyond which you will reject the null
Transcript
The p-value packages the same tail comparison into a single number. The definition callout gives two equivalent readings. It is the smallest significance level at which the observed result would lead you to reject H zero. It is also, assuming H zero is true, the probability of drawing a test statistic at least as extreme as the one actually observed. “At least as extreme” must be interpreted using the alternative hypothesis, so one-sided and two-sided tests generally have different p-values for the same signed t-statistic.
This slide uses a two-sided example with an observed t-statistic of two point one six. The left column states that result. On the figure, the horizontal axis is the t-statistic and the vertical height is the t density under H zero. The right-tail boundary is two point one six, and the symmetric left boundary is minus two point one six. Both tails beyond those values are shaded because a statistic equally far from zero in either direction would be at least as extreme for a two-sided alternative.
The combined shaded area is about zero point zero three one, or three point one percent. That is the two-sided p-value. If you selected a significance level of exactly three point one percent, the statistic would sit at the critical boundary. At any conventional level larger than the p-value, such as five percent, the rejection region is wider and contains the observed statistic, so you reject. At a smaller level, such as one percent, the rejection boundary lies farther out and you fail to reject.
The decision callout summarizes the comparison: reject when the p-value is below the chosen significance level. This always agrees with comparing the statistic to the matching critical value, provided the same distribution, degrees of freedom, and sidedness are used. A p-value is not the probability that H zero is true, not the probability that the result arose “by chance,” and not a measure of effect size. It measures how incompatible the observed statistic is with the null reference distribution. The advantage is that it shows the strength of evidence without forcing every reader to use the same significance threshold.
Suppose the t-statistic you got is 2.16. If the null hypothesis is true, the probability of observing a test statistic at least this extreme is 3.1%.
Thus, 3.1% is the smallest significance level at which the null would be rejected, which is the definition of the p-value.
This decision rule of course results in the same test results as the one we saw that uses a t-value and critical value.
Transcript
Work through this example from the model to the decision. The estimated log-wage equation includes education, experience, and tenure. The coefficient on experience is zero point zero zero four one. Because the dependent variable is log wage and experience is measured in years, that coefficient is approximately the proportional change in wage associated with one more year of experience, holding education and tenure fixed. The slide focuses on statistical evidence rather than the exact percentage interpretation.
The estimated standard error of the experience coefficient is zero point zero zero one seven. We test H zero: beta exper equals zero against the one-sided alternative beta exper greater than zero. Under H zero, the t-statistic is the estimate minus zero, divided by its estimated standard error. That gives zero point zero zero four one divided by zero point zero zero one seven, or about two point four one. The statistic says the estimate is about two point four one standard errors above the null value.
The model has n equals five hundred and twenty-six observations and k equals three explanatory variables, excluding the intercept. Residual degrees of freedom are therefore five hundred and twenty-six minus three minus one, or five hundred and twenty-two. Because the alternative is positive and the significance level is one percent, use an upper-tail critical value rather than splitting the probability across two tails. qt(0.99, 522) returns about two point three three, the ninety-ninth percentile of the appropriate t distribution.
Finally compare on the same scale. Two point four one is greater than two point three three, so the observed statistic lies in the one-percent upper-tail rejection region. We reject H zero in favor of a positive experience effect at the one percent level. Notice that statistical significance is not determined by the coefficient’s raw numerical size. Zero point zero zero four one looks small, but the standard error is smaller, so the estimate is precise relative to the null. Practical importance requires interpreting units and magnitude separately from this test.
Estimated Model
The impact of experience on wage:
- log(wage) = 0.284+0.092\times educ+0.0041\times exper + 0.022 \times tenure
- \widehat{se(\widehat{\beta}_{exper})} = 0.0017
- n = 526
Hypothesis
- H_0: \beta_{exper}=0
- H_1: \beta_{exper}>0
Test
t-statistic = 0.0041/0.0017 = 2.41
The critical value is the 99% quantile of t_{526-3-1}, which is 2.33 (it can be obtained by qt(0.99, 522))
Since 2.41 > 2.33, we reject the null in favor of the alternative hypothesis at the 1% level.
Confidence Interval (CI)
Transcript
The definition callout gives the repeated-sampling meaning of a ninety-five percent confidence interval. Imagine repeatedly drawing new random samples using the same design and, for each sample, calculating an interval with the same formula. The endpoints change from sample to sample because the estimate and its standard error change. In the long run, ninety-five percent of those intervals will cover the fixed true parameter and five percent will miss it, provided the assumptions behind the procedure hold.
Confidence therefore describes the coverage rate of the interval-generating procedure. Before sampling, the endpoints are random and the procedure has a ninety-five percent probability of producing an interval that covers the parameter. Once a specific sample has been observed and its interval calculated, the endpoints are fixed. The population parameter is also fixed. That realized interval either contains the true parameter or it does not.
The lower half identifies the common incorrect interpretation: “there is a ninety-five percent probability that this particular realized interval includes the true parameter.” In the frequentist framework used here, that is not what the statement means. We do not assign a probability distribution to the fixed beta after seeing the data. The ninety-five percent refers to what the method would do over repeated samples.
In ordinary speech, economists often say they are “ninety-five percent confident” that a displayed interval contains beta. Treat that as shorthand for using a procedure with ninety-five percent coverage, not as a literal posterior probability. Also remember that coverage is only as good as the model, standard-error estimator, and sampling assumptions. The following tabs show how central probability ranges from a known reference distribution become intervals for regression coefficients.
What confidence interval is not
The probability that a realized CI calculated from specific sample data includes the true parameter
Transcript
This is the general central-interval recipe for a statistic x with an assumed probability distribution. An A percent interval places A percent of that distribution’s probability in the middle and leaves one hundred minus A percent outside. For a two-sided central interval, divide the omitted probability equally between the lower and upper tails.
The lower endpoint is the open parenthesis one hundred minus A close parenthesis, divided by two, percent quantile of x. The upper endpoint is one hundred minus that same tail percentage. A quantile is the value with the stated proportion of the distribution at or below it. Thus the lower endpoint leaves half of the omitted probability below, and the upper endpoint leaves the other half above.
Pay attention to the parentheses. For A equals ninety-five, calculate one hundred minus ninety-five first, obtaining five, then divide by two to obtain two point five. Writing or reading it as one hundred minus A divided by two would instead produce fifty-two point five, which is completely different. The upper percentile is one hundred minus two point five, or ninety-seven point five.
This tab describes the interval of possible values for the statistic’s reference distribution. To obtain a confidence interval for an unknown regression coefficient, we will start with such a central range for a standardized t-statistic and algebraically solve for beta j. The next tab illustrates the quantile calculation with a standard Normal distribution before we do that inversion.
For the assumed distribution of statistic x, the A\% confidence interval of x is the range with
- lower bound: (100 - A)/2 percent quantile of x
- upper bound: 100-(100 - A)/2 percent quantile of x
Transcript
Now insert A equals ninety-five into the general procedure. One hundred minus ninety-five leaves five percent outside the central interval. Split that evenly, so the lower endpoint is the two point five percentile and the upper endpoint is the ninety-seven point five percentile. The formulas in parentheses in the left column show both calculations explicitly.
For this example, x follows the standard Normal distribution, with mean zero and variance one. Its two point five percentile is approximately minus one point nine six, and its ninety-seven point five percentile is approximately positive one point nine six. Therefore the central ninety-five percent of possible x values lies between minus one point nine six and one point nine six.
Look at the figure in the right column. The horizontal axis shows values of x, the vertical height shows the standard Normal density, and the curve is centered at zero. The red ribbons occupy the two tails outside the interval. Each tail has area zero point zero two five, so together they contain zero point zero five, leaving zero point nine five unshaded in the middle. The annotation near the positive boundary marks about one point nine six; symmetry places the negative boundary at minus one point nine six.
Recognize the same number from the two-sided five-percent test. That is not a coincidence. A two-sided test rejects a zero null when a standardized statistic falls outside the central ninety-five percent region. A ninety-five percent confidence interval contains exactly the parameter values that would keep the corresponding statistic inside that region. For regression coefficients we use t rather than standard Normal quantiles because the error standard deviation is estimated, but the central-tail logic is identical.
For the 95% CI (A = 95),
- lower bound: 2.5 \big((100-95)/2\big) percent quantile of x
- upper bound: 97.5 \big(100-(100-95)/2\big) percent quantile of x
If x follows the standard normal distribution (x \sim N(0, 1)), then,the 2.5% and 97.5% quantiles are -1.96 and 1.96, respectively. So, the 95% CI of x is [-1.96, 1.96].
Transcript
This slide derives a confidence interval for beta j from the t-statistic. Under MLR one through MLR six, beta j hat minus beta j, divided by the estimated standard error of beta j hat, follows a t distribution with n minus k minus one degrees of freedom. Beta j hat is the sample coefficient estimate, beta j is the fixed population coefficient, the denominator is the estimated sampling standard deviation, n is the sample size, and k counts explanatory variables other than the intercept.
For an A percent central interval, define Q sub l as the open parenthesis one hundred minus A close parenthesis over two percentile of that t distribution. Define Q sub h as the one hundred minus open parenthesis one hundred minus A close parenthesis over two percentile. Because the t distribution is symmetric around zero, Q sub l is negative and Q sub h is the corresponding positive value.
Those quantiles initially bound the standardized statistic, but our target is beta j. Solve the displayed statistic for beta j. If t denotes a possible value of the standardized expression, multiply by the estimated standard error to get beta j hat minus beta j, then rearrange: beta j equals beta j hat minus t times the estimated standard error. The minus sign reverses which t quantile maps to which beta endpoint. Symmetry makes the familiar final result beta j hat plus a negative critical amount for the lower bound and beta j hat plus a positive critical amount for the upper bound.
That is what the final bullets state. Multiply Q sub l by the estimated standard error and add beta j hat for the lower bound. Multiply Q sub h by the same standard error and add beta j hat for the upper bound. Since Q sub l is negative and Q sub h positive, the interval is centered on beta j hat and extends the same number of standard errors in both directions. A larger standard error makes the interval wider, signaling less precision. More residual degrees of freedom bring the t quantiles closer to Normal quantiles and slightly narrow the interval. The next tabs calculate a ninety-percent interval for the education coefficient.
Under the assumption of MLR.1 through MLR.6 (which includes the normality assumption of the error), we learned that
\frac{\widehat{\beta}_j - \beta_j}{\widehat{se(\widehat{\beta}_j)}} \sim t_{n-k-1}
So, following the general procedure we discussed in the previous slide, the A% confidence interval of \frac{\widehat{\beta}_j - \beta_j}{\widehat{se(\widehat{\beta}_j)}} is
- lower bound: (100 − A)/2% quantile of the t_{n-k-1} distribution (let’s call this Q_l)
- upper bound: 100 - (100 − A)/2% quantile of the t_{n-k-1} distribution (let’s call this Q_h)
But, we want the A% CI of \beta_j, not \frac{\widehat{\beta}_j - \beta_j}{\widehat{se(\widehat{\beta}_j)}}. Solving for \beta_j,
\beta_j = \widehat{\beta}_j - t_{n-k-1}\times \widehat{se(\widehat{\beta}_j)}
So, to get the A% CI of \beta_j, we scale the CI of \frac{\widehat{\beta}_j - \beta_j}{\widehat{se(\widehat{\beta}_j)}} by \widehat{se(\widehat{\beta}_j)} and then shift by \widehat{\beta}_j. Because the t distribution is symmetric, the minus sign above just swaps which quantile lands on which end:
- lower bound: Q_l \times \widehat{se(\widehat{\beta}_j)} + \widehat{\beta}_j
- upper bound: Q_h \times \widehat{se(\widehat{\beta}_j)} + \widehat{\beta}_j
Note that Q_l is negative and Q_h is positive.
Transcript
This tab reconstructs the wage-level regression for the confidence-interval example. The first cell calls feols with wage on the left of the tilde and education, experience, and tenure on the right. data = wage1 supplies the dataset loaded earlier. The fitted object is assigned to wage_reg. Because the cell has autorun: true, it is available to later interactive cells as soon as the page loads, and assignment alone produces no printed model summary.
The second cell applies broom::tidy to wage_reg and assigns the returned tibble to wage_reg_coef. Parentheses around the assignment cause the table to print as well as be stored. This cell also runs automatically. The table has one row for the intercept and each explanatory variable, with columns for the estimate, estimated standard error, default zero-null t-statistic, and p-value.
Focus on the educ row. Running the code gives an estimated coefficient of about zero point five nine nine and an estimated standard error of about zero point zero five one three. The model has five hundred and twenty-six observations and three non-intercept regressors, so its residual degrees of freedom are five hundred and twenty-two. Those three pieces, estimate, standard error, and degrees of freedom, are sufficient to form the usual confidence interval for beta educ under the stated classical assumptions.
The slide also contains a hidden R chunk that computes these objects for the rendered inline values. Its comment explains why the degrees-of-freedom object is called df_wage: base R already has a function named df for the F density, so a descriptive name prevents an unassigned bare df from resolving to that function. You do not need to interact with the hidden chunk, but the next tab shows the values it supplies.
Run OLS and extract necessary information
Applying broom::tidy() to wage_reg,
Transcript
We want a ninety-percent confidence interval for beta educ, the population coefficient on education in this wage-level regression. The displayed t result says beta educ hat minus beta educ, divided by the estimated standard error of beta educ hat, follows a t distribution with n minus k minus one degrees of freedom, assuming MLR one through MLR six. That result is the bridge from the regression output to an interval.
The three bullets collect the required inputs. Beta educ hat is about zero point five nine nine, shown to four decimal places on the slide. This is the center of the interval. Its estimated standard error is about zero point zero five one three, which determines the scale or width. Smaller uncertainty would produce a tighter interval around the same estimate.
The final input is five hundred and twenty-two residual degrees of freedom. The dataset supplies n equals five hundred and twenty-six observations. There are k equals three explanatory variables, education, experience, and tenure, and the intercept costs one additional degree of freedom. Thus n minus k minus one equals five hundred and twenty-six minus three minus one, or five hundred and twenty-two.
Notice that the rendered values are calculated from the fitted model rather than manually typed. That is a good reproducibility habit. If missing values, filters, or regressors changed the estimation sample, a hardcoded degrees-of-freedom number could become stale while the document still rendered. The next tab uses these exact inputs. For a ninety-percent central interval, it will find the five percent and ninety-five percent quantiles of t with five hundred and twenty-two degrees of freedom, multiply them by the standard error, and add the coefficient estimate.
We are interested in getting the 90% confidence interval of the coefficient on educ (\beta_{educ}). Under all the assumptions (MLR.1 through MLR.6), we know that in general,
\frac{\widehat{\beta}_{educ} - \beta_{educ}}{\widehat{se(\widehat{\beta}_{educ})}} \sim t_{n-k-1}
Specifically for this regression,
\widehat{\beta}_{educ} = 0.599
\widehat{se(\widehat{\beta}_{educ})} = 0.0513
n - k - 1 = 522 (degrees of freedom)
Transcript
Now calculate the ninety-percent interval. Ninety percent in the center leaves ten percent outside, split evenly into five percent in each tail. We therefore need the five percent and ninety-five percent quantiles of a t distribution with five hundred and twenty-two degrees of freedom. The first interactive cell calls qt(0.05, df = 522) and qt(0.95, df = 522). The first argument is the cumulative probability and the named df argument selects the degrees of freedom. Running it returns symmetric values of about minus one point six four eight and positive one point six four eight.
The left column then states the central ninety-percent interval for the standardized statistic. Using the rounded values displayed in the fraction, the numerator is zero point five nine nine minus beta educ and the denominator is about zero point zero five one. That ratio lies between the two t quantiles in ninety percent of repeated samples under the model assumptions.
To translate back to beta educ, scale by the estimated standard error and shift by the estimate. The lower bound is zero point five nine nine plus zero point zero five one times minus one point six four eight. The upper bound is zero point five nine nine plus zero point zero five one times positive one point six four eight. Using the underlying unrounded values, the bounds are approximately zero point five one four and zero point six eight three. The negative lower quantile moves below the estimate; the positive upper quantile moves above it.
Look at the figure in the right column. Its horizontal axis is the t-statistic, its vertical height is the t density, and the red ribbons mark symmetric tail regions. The curve is symmetric and, with five hundred and twenty-two degrees of freedom, nearly indistinguishable from standard Normal. For the ninety-percent calculation on this tab, use the five and ninety-five percent quantiles printed by the interactive cell.
The final sentence hands you to the In practice tab. There, broom requests the same confidence level directly. Its conf.low and conf.high values should match the hand calculation. Doing this algebra once makes clear that software is taking an estimate, adding and subtracting a t critical value times its standard error, not producing an interval by magic.
Now, we need to find the 5% \big((100-90)/2\big) and 95% \big(100-(100-90)/2\big) quantiles of t_{522}.
So, the 90% CI of \frac{0.599 - \beta_{educ}}{0.051} \sim t_{522} is [-1.648, 1.648]
By scaling and shifting, the lower and upper bounds of the 90% CI of \beta_{educ} are:
lower bound: 0.599 + 0.051 \times -1.648 = 0.514
upper bound: 0.599 + 0.051 \times 1.648 = 0.683
Confirm these against the In practice tab, which asks broom for the same interval.
The distribution of t_{522}:
Transcript
This is the practical way to obtain the interval from a fitted model. broom::tidy receives wage_reg as its first argument. Setting conf.int = TRUE asks broom to add confidence-interval endpoints to the usual coefficient table. Setting conf.level = 0.9 requests ninety-percent coverage rather than the common default of ninety-five percent. The decimal zero point nine is a proportion, not the number ninety.
The pipe sends the resulting tibble to dplyr::relocate(term, conf.low, conf.high). relocate does not change any calculations or rows. It simply moves the term name and the two interval columns to the front so they are easier to compare. The remaining columns, including estimate, standard error, statistic, and p-value, remain in the table after them.
Run the cell and find the educ row. conf.low is approximately zero point five one four and conf.high approximately zero point six eight three. These agree with the by-hand calculation on the previous tab, up to rounding. Both endpoints are above zero, so the corresponding two-sided test of beta educ equals zero would reject at the ten-percent significance level. More generally, every value inside this ninety-percent interval is a null value that the matching two-sided ten-percent test would fail to reject.
In applied work, report an interval alongside the coefficient rather than reporting only a significance star. The estimate gives a central effect, the interval displays precision and the range of values reasonably consistent with the data and model, and the confidence level states the procedure used. A star only reports whether one particular null crossed one conventional threshold. The interval gives readers enough information to judge both statistical and substantive importance.
You can just use broom::tidy() with conf.int = TRUE, conf.level = confidence level like below:
Multiple Linear Restrictions: F-test
Transcript
This baseball example introduces a joint hypothesis involving three restrictions. The dependent variable is log salary, where salary is measured in 1993. The model includes years in the league, average games played per year, career batting average, home runs per year, and runs batted in per year. Beta zero is the intercept and beta one through beta five multiply those regressors in the order shown.
The variable list supplies the substantive meanings. years measures career length, and gamesyr measures how regularly a player appears. We treat those as controls. The remaining variables measure performance: bavg is career batting average, hrunsyr is home runs per year, and rbisyr is runs batted in per year.
The question asks whether the three performance measures have any effect on salary collectively after holding years and games per year fixed. The word “collectively” is essential. H zero sets beta three, beta four, and beta five all equal to zero at the same time. Under that restriction, none of the three variables adds explanatory power conditional on the controls. There are q equals three restrictions.
H one says H zero is not true. That means at least one of those three coefficients is nonzero; it does not require all three to be nonzero, and it does not specify their signs. This is one joint hypothesis about a vector of parameters, not three independent yes-or-no questions. Separate coefficient t-tests do not directly control the error rate or measure the combined contribution of the group. The next tab asks why we need a new testing device for this collective claim.
Model
log(salary) = \beta_0 + \beta_1 years + \beta_2 gamesyr + \beta_3 bavg + \beta_4 hrunsyr + \beta_5 rbisyr + u
- salary: salary in 1993
- years: years in the league
- gamesyr: average games played per year
- bavg: career batting average
- hrunsyr: home runs per year
- rbisyr: runs batted in per year
Hypothesis
Once years in the league and games per year have been controlled for, the statistics measuring performance ( bavg, hrunsyr, rbisyr) have no effect on salary collectively.
H_0: \beta_3=0, \beta_4=0, and \beta_5=0
H_1: H_0 is not true
Transcript
How should we test the joint null from the previous tab? The first bullet restates its logical structure: H zero holds only when beta three, beta four, and beta five are all zero. If even one is nonzero, the joint null is false. That makes it tempting to run an ordinary t-test for each coefficient and combine the three conclusions.
Pause before accepting that approach. Separate t-tests answer three separate questions: whether each coefficient is individually distinguishable from zero after controlling for all the others. They do not directly answer whether the variables as a group improve the model. Correlated regressors can make individual standard errors large because the regression has difficulty allocating a shared contribution among them. Each t-statistic may then be small even when removing the whole group causes a major loss of fit.
There is also a multiple-testing issue. If you run several unadjusted five-percent tests under a joint null, the probability of at least one false rejection can exceed five percent. The exact family-wise probability depends on how the statistics are related, so simply declaring the joint null false whenever one individual p-value is below zero point zero five does not generally produce a five-percent joint test.
We need one statistic with a known null distribution that evaluates all three restrictions simultaneously. The F-test does this by comparing an unrestricted model, which includes all three performance variables, with a restricted model that imposes the null by excluding them. Before deriving that test, the next tabs show what the individual coefficient tests say in these data and why their answer is misleading for the collective question.
How do we test this?
- H_0 holds only if \beta_3, \beta_4, and \beta_5 are all zero.
- Conduct t-test for each coefficient individually?
Transcript
This cell loads the baseball data and fits the unrestricted model. data("mlb1", package = "wooldridge") loads mlb1 from the Wooldridge package. The model formula places log(salary) on the left and years, gamesyr, bavg, hrunsyr, and rbisyr on the right. The logarithm is applied within the formula. data = mlb1 supplies the observations, and the fitted fixest object is stored as mlb_reg.
The call to broom::tidy(mlb_reg) converts the coefficient results into a tibble and prints it. The cell has autorun: true, so the data and model objects are created for later tabs automatically. Ignore the comment’s spelling and focus on the model itself: this is the full, unrestricted specification because none of the three performance coefficients is forced to zero.
Read the output rows for bavg, hrunsyr, and rbisyr. Their estimates are approximately zero point zero zero zero nine eight, zero point zero one four four, and zero point zero one zero eight. Their estimated standard errors are roughly zero point zero zero one one zero, zero point zero one six one, and zero point zero zero seven one seven. Dividing each estimate by its standard error gives t-statistics of about zero point eight nine, zero point nine zero, and one point five zero.
The corresponding two-sided p-values are about zero point three seven six, zero point three six nine, and zero point one three four. These values concern each coefficient separately, conditional on all other regressors remaining in the model. Before moving on, ask whether those individual results logically establish that the three coefficients are jointly zero. The next Question and Answer tabs make the distinction explicit.
Transcript
The same tidy output appears again so you can inspect it without flipping back. Go row by row through bavg, hrunsyr, and rbisyr. For each one, compare its two-sided p-value with ten percent, zero point one zero. Batting average has a p-value around zero point three seven six, home runs per year around zero point three six nine, and runs batted in per year around zero point one three four.
None is below ten percent. Equivalently, none of the absolute t-statistics exceeds the two-sided ten-percent critical value. Thus every individual test fails to reject that coefficient equals zero while controlling for the other performance measures and the two career controls.
Before opening the Answer tab, separate three statements. First, an estimate can be positive without being statistically distinguishable from zero. Second, failure to reject is not proof that the population coefficient is exactly zero. Third, three individual failures to reject do not logically imply failure of the joint test that all three equal zero.
Ask why the individual estimates may be imprecise. Baseball performance measures can carry overlapping information. For example, players with many home runs also tend to have many runs batted in. When correlated variables enter together, OLS may estimate their combined contribution well but have difficulty deciding which variable receives how much. That raises individual standard errors. The Answer tab states what the separate tests found and points toward a model-fit comparison that can answer the collective question.
What do you find?
Transcript
The result is deliberately surprising: none of the coefficients on batting average, home runs per year, or runs batted in per year is individually statistically significant, even at the ten-percent level. “Individually” means each t-test asks whether one coefficient is zero while the other two performance variables remain in the regression. The data do not estimate any one variable’s unique partial effect precisely enough to reject zero at that threshold.
Do not convert those three decisions into the statement that performance collectively has no effect on salary. Failure to reject each coefficient separately is not evidence that all three population coefficients are exactly zero. The estimates may be correlated, and the group can carry substantial information even when the model cannot allocate that information precisely among its members.
The joint null is more demanding: beta bavg, beta hrunsyr, and beta rbisyr must all equal zero simultaneously. To evaluate it, compare the unrestricted model containing the three variables with a restricted model that imposes H zero by removing them. If the group contributes little, forcing the coefficients to zero should barely worsen the fit. If the group matters, the restricted model’s residuals should become substantially larger.
The slide points specifically to SSR, the sum of squared residuals, or equivalently a comparison based on R squared. The F-test scales the loss of fit by the number of restrictions and the unrestricted model’s estimate of residual noise, producing a statistic with a known F distribution under H zero. The next tab introduces the two models, and the tabs after that build the comparison step by step.
None of the coefficients on bavg, hrunsyr, and rbisyr is statistically significantly different from 0 even at the 10% level!!
So, does this mean that they collectively have no impact on the salary of MLB players?
If you were to conclude that they do not have statistically significant impact jointly, you would turn out to be wrong!!
SSR (or R^2) turns out to be useful for testing their impacts jointly.
In doing an F-test of the null hypothesis, we compare sum of squared residuals (SSR) of two models:
Unrestricted Model
log(salary) = \beta_0 + \beta_1 years + \beta_2 gamesyr + \beta_3 bavg + \beta_4 hrunsyr + \beta_5 rbisyr + u
Restricted Model
log(salary) = \beta_0 + \beta_1 years + \beta_2 gamesyr + u
The coefficients on bavg, hrunsyr, and rbisyr are restricted to be 0 following the null hypothesis.
Transcript
The slide above this tab defines the two models used by the F-test. The unrestricted model contains years, games per year, batting average, home runs per year, and runs batted in per year. The restricted model keeps only years and games per year. Dropping the three performance variables imposes the joint null that their coefficients are all zero.
The question asks which model should have the larger sum of squared residuals if we fit both to the same observations. Recall the OLS objective: choose coefficient values that minimize the sum of squared residuals. The unrestricted model may choose any values for all five slope coefficients. The restricted model is forced to choose beta three, beta four, and beta five equal to zero.
The unrestricted parameter space therefore contains every fit available to the restricted model plus additional possibilities. If zero truly is optimal for the three coefficients in this sample, the two models can tie. Otherwise, the extra flexibility lets the unrestricted model reduce SSR. Taking options away cannot produce a lower minimized objective value.
Notice that this ordering is a mathematical property of nested OLS fits, not something that only holds probabilistically when the null is true. What the null affects is how large a difference we should expect from sampling noise alone. Think through the constrained-minimization logic, then open the Answer tab.
If the null hypothesis is indeed true, then what do you think is going to happen if you compare the SSR of the two models? Which one has a bigger SSR?
Transcript
The restricted model’s SSR must be at least as large as the unrestricted model’s SSR. The unrestricted OLS fit chooses coefficients from a larger set. It could reproduce the restricted fit exactly by setting the three performance coefficients to zero, so it can never do worse. It will do better whenever nonzero choices reduce the residual sum of squares.
The slide phrases this as the restricted model having smaller explanatory power, which is the intuition. More precisely, the restricted SSR is greater than or equal to the unrestricted SSR. Equality is possible if the restrictions do not change the sample fit. Therefore SSR sub r minus SSR sub u is always nonnegative for these nested models estimated on the same data with the same dependent variable.
This fact alone does not tell us whether the null is false. Adding regressors almost always improves in-sample fit at least slightly, even when their population coefficients are zero, because OLS can exploit random sample variation. We need to decide whether the observed improvement is large relative to what chance fitting would produce under H zero.
That is why the F-statistic scales the SSR gap rather than rejecting whenever the gap is positive. It divides the improvement per restriction by an estimate of the unrestricted model’s residual variance. A modest difference can be ordinary noise, while a large standardized difference is evidence that forcing the coefficients to zero meaningfully damages the model.
SSR from the restricted model should be large because the restricted model has a smaller explanatory power than the unrestricted model.
SSR of the unrestricted model: SSR_u
SSR of the restricted model: SSR_r
Transcript
The code above this tab has fitted both nested models and printed their sums of squared residuals. The unrestricted fit includes all five regressors and stores its residuals in res_u. sum(res_u$residuals^2) squares every residual and adds the squares to obtain SSR sub u. The restricted fit includes only years and games per year, and the analogous expression obtains SSR sub r.
As predicted, the restricted number is larger. The question asks what the difference SSR sub r minus SSR sub u measures. Compare the formulas for the two regressions. They use the same dependent variable, same observations, same controls, and same OLS criterion. The only change is whether batting average, home runs per year, and runs batted in per year are allowed to enter.
Consequently, the reduction in SSR achieved by the unrestricted model measures the extra in-sample explanatory contribution of those three variables as a group. It is the fit lost when their coefficients are jointly forced to zero. The difference cannot tell us how the contribution should be divided among correlated variables, but that is precisely why it is useful for a collective question.
Answer in those terms before opening the next tab: the gap is the group’s joint contribution to reducing squared prediction errors, conditional on the controls that appear in both models.
What does SSR_r - SSR_u measure?
Transcript
SSR sub r minus SSR sub u is the reduction in residual sum of squares produced by adding the three excluded performance variables to the restricted model. In that sense, it measures their joint contribution to explaining log salary after years and games per year are already controlled for.
It is not the unique contribution of any single performance variable, and it is not generally the sum of three separate one-variable improvements. The regressors may overlap in the information they carry. The unrestricted model chooses all three coefficients together, so the SSR difference captures their combined ability to fit the dependent variable, including contributions that cannot be cleanly allocated when the variables are correlated.
This is the feature the individual t-tests miss. Suppose two regressors contain nearly the same signal. Holding one fixed, it may be difficult to show that the other has a distinct partial effect, giving each a large standard error and small t-statistic. But removing both can discard a great deal of predictive information. The SSR comparison asks what the group accomplishes as a group.
The gap is still only a raw, sample-dependent quantity. It tends to grow with the scale of the dependent variable, the sample size, and the noise level. The next Question tab asks whether its magnitude is large enough to count as evidence, which requires a reference distribution rather than interpretation of the raw number alone.
The contribution from the three excluded variables in explaining the dependent variable.
Transcript
Now that we know what SSR sub r minus SSR sub u measures, ask whether the observed contribution is large enough to conclude that the excluded variables matter in the population. A positive gap by itself is not sufficient. The unrestricted model has more parameters and will almost always fit the sample at least a little better, even when the restrictions are true.
Try to judge the printed gap directly and notice the problem. Its magnitude depends on the units and transformation of the dependent variable, the number of observations, the number of added regressors, and the overall residual noise. If salary rather than log salary were used, every residual and squared residual would be on a different scale. A raw SSR improvement that looks numerically large in one application could be trivial in another.
We need to compare the improvement per restriction with a measure of unexplained variation, then determine how that ratio behaves under H zero. In other words, return to step two of the general testing recipe: find a statistic with a known distribution when the null is true.
Do not answer based only on the sign or raw size. Ask what reference distribution would tell us how often a gap this large arises from the extra flexibility of three irrelevant regressors. The following Answer tab states why we cannot yet decide, and the F-statistic tab supplies the normalization.
Is the contribution large enough to say that the excluded variables are important?
Transcript
We cannot tell at this point because we do not know the null distribution of the raw difference SSR sub r minus SSR sub u. Computing a sample number completes only the calculation step. Without a reference distribution, words such as “large” or “unlikely” have no formal meaning.
This returns us to step two of the five-step recipe. We must ask what values an appropriately standardized improvement in fit would take if the three population coefficients were all zero. Under H zero, extra regressors can still reduce sample SSR by fitting random noise, so a positive improvement is expected. The question is whether the improvement is unusually large relative to that chance behavior.
The F-statistic performs the required scaling. It first divides the SSR gap by q, the number of restrictions, producing an average improvement per added coefficient. It then divides by SSR sub u over n minus k minus one, the unrestricted residual mean square and estimator of the error variance. The resulting ratio is unit-free.
Under the classical assumptions and H zero, that ratio follows an F distribution with q numerator degrees of freedom and n minus k minus one denominator degrees of freedom. Once we have that distribution, we can choose a significance level, find an upper-tail critical value, and decide whether the observed joint contribution is too large to attribute plausibly to sampling noise.
Cannot tell at this point because we do not know the distribution of the difference!
Setup
Consider a following general model:
y = \beta_0 +\beta_1 x_1 + \dots+\beta_k x_k +u
Suppose we have q restrictions to test: that is, the null hypothesis states that q of the variables have zero coefficients.
H_0: \beta_{k-q+1} =0, \beta_{k-q+2} =0, \dots, \beta_k=0
When we impose the restrictions under H_0, the restricted model is the following:
y = \beta_0 +\beta_1 x_1 + \dots+\beta_{k-q} x_{k-q} + u
F-statistic
If the null hypothesis is true, then,
F = \frac{(SSR_r-SSR_u)/q}{SSR_u/(n-k-1)} \sim F_{q,n-k-1}
- q: the number of restrictions
- n-k-1: degrees of freedom of residuals
Transcript
Before using the general F formula above, check its sign. The statistic is open parenthesis SSR sub r minus SSR sub u close parenthesis divided by q, all divided by SSR sub u over n minus k minus one. Here q is the positive number of restrictions, and n minus k minus one is the positive residual degrees of freedom.
The denominator contains SSR sub u, a sum of squared residuals. Except for a perfect fit it is positive, and dividing by positive residual degrees of freedom preserves that sign. It estimates the error variance in the unrestricted model.
The numerator contains the difference between restricted and unrestricted SSR. For nested OLS models fitted on the same observations, the unrestricted model’s feasible coefficient choices include the restricted choices. Therefore the minimized unrestricted SSR cannot exceed the restricted SSR. The difference is nonnegative, and division by q leaves it nonnegative.
So what sign can F have? Recall that equality of the two SSR values is possible when the restrictions make no difference to the sample fit. Answer with that boundary case in mind, then open the Answer tab.
Is the above F-statistic always positive?
Transcript
The F-statistic is nonnegative. The restricted sum of squared residuals can never be below the unrestricted sum for nested OLS models estimated on the same sample, so SSR sub r minus SSR sub u is greater than or equal to zero. The remaining quantities in the ratio are positive under an ordinary nondegenerate regression.
If the restriction does not change the fitted values at all, the two SSR values are equal and F is exactly zero. Otherwise, the restricted fit is worse and F is positive. This is the constrained-minimum argument from the previous tabs, not an empirical pattern that happens only most of the time.
The sign restriction determines the test’s shape. A t-statistic can be positive or negative, so one-sided direction and two-sided absolute values matter. An F-statistic lives on zero and the positive real line. Evidence against the joint null appears only as an unusually large positive value because larger F means the restrictions caused a greater standardized loss of fit.
Therefore an F-test uses the upper tail of the F distribution. There is no negative tail to inspect and no reason to split the significance level across two tails. A very small F means the restricted and unrestricted fits are similar, which is compatible with H zero; it is not evidence against the null in an opposite direction.
Yes, because SSR_r-SSR_u is always non-negative: the restricted model can never fit better than the unrestricted one.
Transcript
Read the F formula one more time to determine its direction. The numerator is the loss of fit from imposing q restrictions, averaged over those restrictions. The denominator is the unrestricted residual mean square, which measures the remaining noise per residual degree of freedom.
Suppose the q excluded variables make a large joint contribution to explaining y. Removing them from the model should make the restricted residuals much larger, so SSR sub r rises well above SSR sub u. That increases SSR sub r minus SSR sub u and therefore increases the numerator.
Holding the denominator and number of restrictions fixed, a larger numerator makes the F-statistic larger. The ratio interprets the fit lost through restriction relative to the background error variance. If excluding the variables costs little compared with the noise, F stays near its ordinary null range. If it costs a great deal, F moves into the upper tail.
Choose between “greater” and “smaller” from the formula itself. This direction is the reason the decision rule will reject only for F greater than a positive critical value.
The greater the joint contribution of the q variables, the (greater or smaller) the F-statistic?
Transcript
Greater. When the q variables contribute more jointly, removing them increases the restricted model’s residual sum of squares by more. The gap SSR sub r minus SSR sub u becomes larger. After dividing by q and by the unrestricted residual mean square, that larger standardized loss produces a larger F-statistic.
The reasoning gives you the decision rule without memorization. A large F means the restrictions hurt the model’s fit substantially relative to residual noise. Those restrictions are exactly the joint null, so a sufficiently large value is evidence against H zero. We reject when F lies beyond an upper-tail critical value from the appropriate F distribution.
A small F has the opposite interpretation. It says the restricted and unrestricted models fit similarly after accounting for the number of restrictions and noise. Dropping the variables cost little, so the data are compatible with their coefficients being jointly zero. We then fail to reject H zero.
Because F is nonnegative and only large values contradict the restrictions, every conventional F-test here is upper-tailed. The significance level is placed entirely in the upper tail. The next slide plots F distributions and lays out the complete step-by-step procedure.
Greater.
Transcript
This slide puts the F reference distribution beside the testing recipe. Start with the figure on the left. The horizontal axis shows nonnegative F values and the vertical axis shows probability density. The two curves are different F distributions: F with two numerator and one hundred denominator degrees of freedom, and F with four numerator and four hundred denominator degrees of freedom. Both are right-skewed and live only on zero and positive values, consistent with the SSR-ratio construction. Their shapes differ because an F distribution requires two degrees-of-freedom parameters, so there is no single universal F curve.
Now read the five steps on the right. First define H zero precisely, including every restriction. Second estimate the unrestricted model and the restricted model on the same observations, then obtain SSR sub u and SSR sub r. The unrestricted model contains all coefficients; the restricted model imposes H zero. Third calculate F as the average SSR increase per restriction divided by the unrestricted residual mean square.
Fourth choose a significance level and find the matching upper-tail critical value from F with q numerator degrees of freedom and n minus k minus one denominator degrees of freedom. Here q is the number of restrictions and k counts regressors in the unrestricted model excluding its intercept. Both values matter to qf; interchanging them changes the answer.
Fifth compare the observed statistic with the critical value. Reject H zero if F is greater, because only a large standardized loss of fit is evidence that the restrictions are false. Otherwise fail to reject. This is the same general five-step logic from the first slide, specialized to a statistic that tests several linear restrictions at once. The next tabs implement each calculation for the baseball model.
F-distribution
F-test steps
- Define the null hypothesis
- Estimate the unrestricted and restricted models to obtains their SSR
- Calculate F-statistic
- Define the significance level and corresponding critical value according to the F distribution with appropriate degrees of freedoms
- Reject if your F-statistic is greater than the critical value, otherwise do not reject
Transcript
Step one fits the two nested models and obtains their sums of squared residuals. In the unrestricted call, feols regresses log salary on years, games per year, batting average, home runs per year, and runs batted in per year, using mlb1. The line break after gamesyr + is only formatting; the plus sign continues the formula. The fitted object is stored as reg_u.
reg_u$residuals extracts the vector of unrestricted residuals. Raising it to the second power squares each element, and sum adds those squares. The result is stored as SSR_u. This is the minimized residual sum of squares when all five slope coefficients are free.
The restricted call regresses the same log salary outcome on only years and games per year. It uses the same dataset and estimator but omits bavg, hrunsyr, and rbisyr, thereby imposing the three zero restrictions in H zero. The fitted object is reg_r, and squaring and summing reg_r$residuals gives SSR_r.
Compare the formulas carefully. The dependent variable, observations, controls, intercept treatment, and OLS method are identical. Only the three jointly tested variables differ. That nested structure is required for interpreting SSR sub r minus SSR sub u as the fit lost by imposing the null. Both model cells use earlier-loaded data, and autorun: true ensures the objects exist for Step 2, where the raw gap is converted into an F-statistic.
Estimate the unrestricted and restricted models
Transcript
Step two assembles the F-statistic from the objects created in Step 1. df_q <- 3 records q, the number of restrictions: one each for batting average, home runs per year, and runs batted in per year. It is not a residual degree of freedom; it will be the numerator degrees of freedom of the F distribution.
degrees_freedom(reg_u, "resid") asks the unrestricted model for its residual degrees of freedom and stores the answer as df_ur. The unrestricted model is the relevant one because its SSR estimates the remaining error variance after all coefficients are freely estimated. With the available baseball observations and five regressors plus an intercept, this value is n minus five minus one.
F_stat_num <- (SSR_r - SSR_u) / df_q calculates the numerator mean square. Subtracting measures the fit lost under the restrictions, and dividing by three expresses the average loss per restriction. F_stat_denom <- SSR_u / df_ur calculates the unrestricted residual mean square. Under homoskedastic classical assumptions, this is sigma hat squared, the estimator of the error variance.
Finally, F_sta <- F_stat_num / F_stat_denom divides the restriction mean square by the residual mean square, and the last line prints F_sta. The ratio is unit-free because both pieces are measured in squared dependent-variable units. A value near the ordinary null range means the fit loss is modest relative to noise; a large value means the restrictions caused an unusually costly loss. Step 3 will use qf to decide how large is large at the chosen significance level.
Calculate F-stat
Transcript
Steps three and four choose the threshold and make the decision. alpha <- 0.05 sets the significance level to five percent. In this upper-tail test, the rejection region occupies the largest five percent of F values, so the critical value has ninety-five percent of the null distribution to its left.
qf(1 - alpha, df1 = df_q, df2 = df_ur) computes that boundary. qf is the F quantile function. Its first argument is zero point nine five. df1 receives the three numerator degrees of freedom, one per restriction, and df2 receives the unrestricted residual degrees of freedom. The result is assigned to c_value and printed. Do not divide alpha by two, because F is nonnegative and only the upper tail is a rejection region.
The second cell evaluates F_sta > c_value. This logical comparison returns TRUE for the baseball data, meaning the observed F-statistic lies beyond the five-percent critical value. Equivalently, its upper-tail p-value is below zero point zero five. We reject the joint null that all three performance coefficients are zero.
Read the Observation callout carefully. The variables have statistically significant impacts jointly, even though none of their individual two-sided t-tests rejected at ten percent. That is not a logical contradiction. Individual tests ask whether each variable has a precisely identifiable unique contribution while the correlated others stay in the model. The F-test asks whether removing the whole group causes a statistically large loss of fit. The result tells us the group matters, and the later multicollinearity tabs explain why its members can look weak one at a time.
Find the critical value
Is F-stat > critical value?
Transcript
This is the practical interface for linear restrictions. car::linearHypothesis receives a fitted unrestricted regression and a description of H zero, then constructs the appropriate Wald test using the model’s coefficient vector and variance-covariance matrix. For a classical OLS model and restrictions that set a group of coefficients to zero, the test corresponds to the restricted-versus-unrestricted comparison developed by hand.
The syntax block has two arguments. regression is the fitted model object containing the unrestricted specification. You do not pass the restricted fit. hypothesis is text describing one or more linear equations that must all hold under H zero. Because the code chunk has eval = F, it only illustrates syntax and does not try to execute placeholder names.
The example c("x1 = 0", "x2 = 1") uses c to create a character vector with two strings. The first restriction says the coefficient named x1 equals zero. The second says the coefficient named x2 equals one. Together they define q equals two restrictions. Quotation marks matter because the equations are parsed by the function rather than evaluated as ordinary R assignments.
The function is not limited to zero restrictions. A coefficient can be tested against one or another specified value, and relationships such as x1 = x2 or x1 - 2*x2 = 0 can express linear combinations. You still need to read what test statistic and reference distribution the method reports for the model class. Flip to the Demonstration tab to apply the function to the three baseball restrictions.
You can use the car::linearHypothesis() function from the car package.
Syntax
regression: the name of the regression results of the unrestricted modelhypothesis: text of the null hypothesis. For example,
c("x1 = 0", "x2 = 1") means the coefficients on x1 and x2 are 0 and 1, respectively
Transcript
This code applies the convenience function to the baseball regression. mlb_reg is the unrestricted model containing years, games per year, batting average, home runs per year, and runs batted in per year. The second argument is a character vector made with c, containing "bavg=0", "hrunsyr=0", and "rbisyr=0". The three strings jointly impose exactly the null stated at the start of the section.
car::linearHypothesis reads the coefficient names from the model, forms the three-restriction contrast, obtains its estimated covariance, and reports the joint test. With this fixest object, the displayed method reports a Wald chi-squared statistic of about twenty-eight point seven with three restrictions and a p-value around zero point zero zero zero zero zero two seven. That p-value is far below five percent, so the joint null is rejected.
Compare the conclusion with the by-hand F calculation. Both show that the performance variables matter collectively. Depending on the method used for the fitted object, the printed Wald table may use a chi-squared reference rather than the exact finite-sample F reference from the SSR derivation, so inspect the output labels rather than assuming every linearHypothesis call prints the same statistic. With a large residual degrees of freedom, the decisions here are the same.
This function is what you will usually use because it handles the coefficient covariance algebra and accepts general linear restrictions. The hand calculation remains important. It tells you that the test asks how much imposing H zero worsens model fit relative to residual noise. If a result is surprising, check the unrestricted specification, the exact restrictions, the estimation sample, the covariance estimator, and the reported reference distribution.
The following code test if \beta_{bavg} = 0, \beta_{hrunsyr} = 0, and \beta_{rbisyr} = 0.