Let’s start with why we are doing this at all. In the univariate model on the top half of the slide, you explain an outcome, y, with only one independent variable, x. Everything else that affects y but is not represented by x is collected in the error term, u. The crucial condition for an unbiased OLS slope is that the expected value of that error, conditional on x, is zero. In words, once you specify a value of x, the omitted influences on y must average to zero. Outside a randomized experiment, that is often hard to believe. People, firms, counties, and countries differ along many dimensions, and some omitted dimension can easily move together with x.
The multivariate model in the lower half responds by including additional independent variables. Each included variable is moved out of the error term and represented explicitly by its own coefficient. That gives us a chance to compare observations that have the same values of those other included factors. It can therefore remove a particular source of omitted-variable bias. But read the word in blue carefully: the endogeneity problem becomes less severe, not automatically solved. Adding regressors only helps with relevant factors that you actually observe, include in the model, and specify correctly. Other omitted factors remain inside u, and any one of them may still be related to the included x variables. So multivariate regression improves the comparison, but causal interpretation will still require a zero conditional mean assumption for the new, smaller error term. We will make that condition precise in the next set of tabs.
Univariate
The most important assumption E[u|x] = 0 (zero conditional mean) is almost always violated (unless your data comes from randomized experiments) because all the other variables are sitting in the error term, which can be correlated with x.
Multivariate
More independent variables mean fewer factors left in the error term, which makes the endogeneity problem less severe
Here is the move written out with wages. Look at the first line. Wage is modeled as beta-zero plus beta-one times education plus u-one. The parenthesis tells you what u-one contains in this example: u-two plus beta-two times experience. So experience really affects wages, but the univariate equation hides that effect inside its error term.
Why is that a problem for the estimated education coefficient, beta-one-hat? Experience tends to raise wages, and education and experience are generally related because an additional year in school changes the years available for work and because workers with different schooling histories can follow different career paths. When an omitted wage determinant is correlated with education, OLS cannot tell which part of their joint movement should be credited to education. The education coefficient can then absorb some of the omitted experience effect. It is unbiased only in the special case that experience is uncorrelated with education, along with the other conditions we have discussed.
Now look at the second line. Experience has been pulled out of u-one, placed explicitly in the equation, and assigned its own slope, beta-two. The remaining error is called u-two. In this bivariate model, beta-one measures the change in wage associated with one more unit of education while experience is held fixed. Likewise, beta-two measures the change associated with one more unit of experience while education is held fixed. That is the meaning of the blue phrase, “holding experience fixed.” The red phrase gives our standard vocabulary: experience is controlled for. We are comparing workers at the same modeled experience level, rather than mixing an education comparison with systematic differences in experience. This removes experience as this particular source of omitted-variable bias, although anything still inside u-two can remain a problem.
Uni-variate vs. bi-variate
\begin{align*} \text{Uni-variate}\;\; wage = & \beta_0 + \beta_1 educ + u_1 (=u_2+\beta_2 exper)\\ \text{Bi-variate}\;\; wage = & \beta_0 + \beta_1 educ + \beta_2 exper + u_2 \end{align*}What’s different?
uni-variate: \widehat{\beta}_1 is biased unless experience is uncorrelated with education because experience was in error term
bi-variate: able to measure the effect of education on wage, holding experience fixed because experience is modeled explicitly ( We say exper is controlled for. )
Same structure, now applied to a policy question you will meet in real debates: does spending more per student raise standardized test scores? In the first line, average score equals beta-zero plus beta-one times expenditure per student plus u-one. If average family income also affects scores but you leave it out, its contribution is inside u-one. The parenthesis makes that bookkeeping explicit: u-one equals u-two plus beta-two times average income.
The danger is that district income is plausibly related to both sides of the question. Higher-income districts may have a larger tax base and spend more per student. Their students may also score differently because of home resources, tutoring, nutrition, parental education, neighborhood conditions, and other channels that are not caused by school expenditure. A regression of score on expenditure alone sees spending and income move together. It can credit spending with score differences that are partly associated with income, so beta-one-hat need not isolate the effect of spending.
In the second line, average income is included explicitly. Beta-one now describes how predicted average score changes with expenditure when average income is held fixed, while beta-two describes how it changes with income when expenditure is held fixed. We say that the model controls for average income. This has exactly the same shape as the wage and experience example, and the same logic as the firefighters example from lecture zero: a third factor can make two variables move together even when the simple regression does not represent the causal relationship you care about. Controlling for income removes that measured channel from the error, but it does not control for every other district characteristic. The remaining u-two still has to satisfy the relevant mean independence condition.
The impact of per student spending (expend) on standardized test score (avgscore) at the high school level
Now replace the names in those examples with a general bivariate model. The equation says y equals beta-zero plus beta-one times x-one plus beta-two times x-two plus u. Here y is the dependent variable we want to explain. X-one and x-two are the two included independent variables. U contains all determinants of y that the equation does not represent explicitly.
Beta-zero is the intercept. It is the conditional mean predicted by the linear model when both x-one and x-two equal zero. Whether that is substantively meaningful depends on whether zero is a possible or useful value for those variables, but the intercept is still needed to position the regression plane.
Beta-one is the change in the model’s predicted y associated with a one-unit increase in x-one, holding x-two fixed. You can see that directly by comparing two observations or two hypothetical cases with the same x-two and x-one values one unit apart. The beta-two x-two term is unchanged, beta-zero is unchanged, and the predicted value differs by beta-one. Beta-two has the symmetric interpretation: it is the change in predicted y for a one-unit increase in x-two while x-one is fixed. These are partial effects because each slope describes movement along one regressor while the other included regressor is held constant.
The phrase “holding other factors fixed” refers to the other factors included in the equation. It does not mean that every feature of the world has somehow been fixed. Including x-two creates a conditional comparison and prevents its modeled contribution from being mixed into the x-one slope. It does not automatically make beta-one causal. For that, the remaining error u must have conditional mean zero given both x-one and x-two. Later, the Frisch-Waugh-Lovell section will show mechanically how OLS constructs this holding-fixed comparison by removing the variation explained by the other regressors.
More generally,
\begin{align*} y=\beta_0+\beta_1 x_1 + \beta_2 x_2 + u \end{align*}The condition for unbiasedness generalizes from one regressor to the full set of regressors. On the univariate side, the model is y equals beta-zero plus beta-one x plus u, and the assumption is that the expected value of u conditional on x equals zero. This means that within every value of x, the positive and negative unobserved influences average out. It is not a claim that every observation has u equal to zero. Individual errors can be positive or negative. The claim is about their conditional average.
On the bivariate side, the model includes x-one and x-two, so the condition becomes: the expected value of u conditional on x-one and x-two equals zero. Read the verbal statement on the slide carefully. For any combination of x-one and x-two values, the average of all unobserved determinants of y must be zero. Imagine dividing the population into cells defined jointly by x-one and x-two. Within each cell, there must be no systematic positive or negative component left in u.
This is a joint conditional statement. It is not enough to say that u has zero unconditional mean, or even merely that u has zero pairwise correlation with each regressor. A nonlinear or interaction pattern could make the conditional mean depend on the pair while those simpler correlations are zero. Adding x-two can help because its effect is moved out of u and into the modeled portion of y. But after that move, unbiasedness still depends on what remains in u. The assumption is now stated conditional on both included variables because OLS is using variation in each variable after accounting for the other.
Uni-variate
y = \beta_0 + \beta_1x + u,
E[u|x]=0
Bi-variate
y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + u,
Here is the joint mean independence condition applied to the wage model. The equation says wage equals beta-zero plus beta-one times education plus beta-two times experience plus u. Beta-one is the partial association between education and wage at a fixed experience level, and beta-two is the partial association between experience and wage at a fixed education level. For the OLS estimators of those coefficients to be unbiased, the displayed condition requires the expected value of u, given education and experience, to equal zero.
Translate that into the groups OLS is comparing. Among people with a particular education level and a particular amount of experience, the factors left inside u must average to zero. As you move to another education-experience combination, the average omitted component must still be zero. Suppose innate ability affects wages and is not observed in the data, so it remains part of u. The verbal sentence says the condition would hold if ability were, on average, unrelated to education and experience after conditioning on their values. Say that aloud and ask whether it is credible. If people with greater innate ability tend to obtain more education, then the conditional average of the omitted ability component changes with education. Beta-one-hat can then combine the education association with an ability association.
Now read the callout, because it states the honest limit of controlling for things. Adding experience moved experience out of the error term and gave it its own coefficient, so omitted experience can no longer bias the education coefficient through that channel. But ability is still in u. No number of other regressors can move ability itself out if ability was never observed or represented by a valid measured proxy. Multivariate regression shrinks the list of omitted determinants; it does not empty the error term. This is why choosing controls requires substantive knowledge, and why a longer regression is not automatically a credible causal design.
In the following wage model,
\begin{align*} wage = & \beta_0 + \beta_1 educ + \beta_2 exper + u \end{align*}Mean independence condition is
\begin{align*} E[u|educ,exper]=0 \end{align*}Verbally:
This condition would be satisfied if innate ability of students is on average unrelated to education level and experience.
The limit of controlling for things
Notice what just happened. Adding exper to the model moved experience out of the error term, so it can no longer bias us. But ability is still in there, and no amount of extra regressors will move it out, because you cannot control for something you do not observe . Multivariate regression shrinks the error term. It does not empty it.
The extension from two independent variables to k independent variables contains no new conceptual step, only more terms. Read the model from left to right: y equals beta-zero, the intercept, plus beta-one times x-one, beta-two times x-two, continuing through beta-k times x-k, plus the error u. The index j can label any slope from one through k. Beta-j is the change in the model’s conditional mean of y associated with a one-unit increase in x-j while all of the other included x variables are held fixed. U collects determinants of y not represented by those k regressors and the chosen functional form.
The slide uses beta-OLS as shorthand for the collection of OLS estimators of all these beta coefficients. For those estimators to be unbiased under the other multiple linear regression assumptions, the expected value of u conditional on the entire vector x-one through x-k must equal zero. For every joint combination of the regressors, the unobserved component must average to zero. The verbal line uses “uncorrelated” as an informal description. Strictly, the displayed zero conditional mean restriction is stronger than requiring only zero pairwise correlation between u and each x. Zero pairwise correlations do not rule out a systematic nonlinear conditional pattern.
If joint zero conditional mean holds, repeated random samples center each beta-j-hat on its true beta-j. It does not say every estimate equals the truth in one sample, because sampling variation remains. If the condition fails, one or more coefficient estimators may be biased. Which ones, in what direction, and by how much depends on how the omitted component is related to the included regressors and how those regressors relate to one another. This is why adding a control can change several estimated coefficients, not just the coefficient attached to the new variable.
Model
\begin{align*} y=\beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_k x_k + u \end{align*}Mean independence assumption?
\beta_{OLS} (OLS estimators of \betas) is unbiased if,
\begin{align*} E[u|x_1,x_2,\dots,x_k]=0 \end{align*}Verbally: this condition would be satisfied if the error term is uncorrelated with any of the independent variables, x_1,x_2,\dots,x_k.
Here is the general model implemented in R. Start at the top. The library line loads the fix-est package, which supplies fee-ols, the estimation function used throughout this course. Set-seed with twenty twenty-four fixes the pseudorandom number sequence, so every student who runs the cell gets the same simulated sample and therefore the same regression results. Capital N is set to one hundred, our sample size.
The three calls to r-norm each draw one hundred independent standard normal values. X-one and x-two are the independent variables, and u is the error term. The next line constructs y as one plus x-one plus x-two plus u. Because we wrote the data-generating process ourselves, we know the population parameters: the intercept is one, the slope on x-one is one, and the slope on x-two is one. Data-frame then combines y, x-one, and x-two into a rectangular data set with one hundred rows. Notice that u is not included in data-sim. In real data the error is unobserved, and the regression has to infer residuals after fitting the model.
The estimation line calls fee-ols. To the left of the tilde is the dependent variable y. To the right, x-one plus x-two tells R to include both regressors; an intercept is included by default. The data argument tells fee-ols to find those variable names inside data-sim. That small formula change is the syntactic move from a univariate to a multivariate regression, even though the interpretation and assumptions require much more thought.
The final line prints reg. When I run this exact code, the estimates are about one point zero seven five for the intercept, zero point nine two six for x-one, and zero point nine nine two for x-two. They are close to, but not exactly, the true values of one because this is one random sample of only one hundred observations and u creates sampling noise. Unbiasedness does not promise exact recovery in every sample. It says that if we repeatedly generated samples this way, the estimates would average to the true coefficients. Run the cell, compare each estimate with one, then flip to the next tab to see how to present the same fitted model properly.
Note the true values: the intercept is 1, and both slopes are 1. Compare them with what OLS returns.
This is a practical reporting tab. When you hand in an assignment or write your final paper, do not paste the raw console printout from fee-ols. A regression table lets a reader see estimates, uncertainty, sample size, and model information in a standard layout. On the left is the code that produces the table, and on the right is its rendered result.
The first line loads model-summary, the package that supplies m-summary. The next fee-ols call recreates the regression from the previous tab: y is on the left of the tilde, x-one plus x-two is on the right, and data-sim supplies the variables. Then m-summary receives the fitted model object reg. If you later have several models, this function can display them as separate columns, which makes specifications easy to compare.
The stars equals true argument adds conventional significance markers. Read them using the key at the bottom of the table: three stars correspond to a p-value below zero point zero zero one, two below zero point zero one, one below zero point zero five, and the plus sign below zero point one. A star is not an effect size and is not evidence that an estimate is substantively important. It is only a compact statement about a hypothesis test under the reported standard error.
The g-o-f omit argument uses the pattern shown to suppress selected goodness-of-fit rows whose names contain items such as I-C, Log, Adj, F, Pseudo, or Within. The vertical bars inside the quoted pattern mean “or,” so a row matching any one of those pieces is omitted. Keep that option as written for course work unless you have a reason to report a different set. It does not change the regression; it only changes which summary rows are printed.
Now read the right-hand table. Column one reports the same fitted model as the previous tab. Each coefficient estimate appears first, with its standard error directly underneath in parentheses. The intercept is about one point zero seven five with a standard error of zero point zero nine one, x-one is about zero point nine two six with a standard error of zero point zero eight nine, and x-two is about zero point nine nine two with a standard error of zero point zero eight nine. All three receive three stars in this simulation. Below them, the table retains one hundred observations, R-squared of zero point seven two zero, R-M-S-E of zero point eight nine, and the label I-I-D for the standard errors. This is the form readers expect: a clear estimate, its uncertainty, and enough model information to understand what was fitted.
When you are asked to present regression results in assignments or your final paper, use the msummary() function from the modelsummary package.
| (1) | |
|---|---|
| + p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001 | |
| (Intercept) | 1.075*** |
| (0.091) | |
| x1 | 0.926*** |
| (0.089) | |
| x2 | 0.992*** |
| (0.089) | |
| Num.Obs. | 100 |
| R2 | 0.720 |
| RMSE | 0.89 |
| Std.Errors | IID |
The derivation is the same least-squares problem as lecture one, part two, now with more unknown coefficients. The first sentence states the goal: find the combination of beta estimates that minimizes the sum of squared residuals. A residual is the observed outcome minus the outcome predicted by the fitted equation.
Theta-hat is introduced as shorthand for the whole collection of estimates: beta-zero-hat, beta-one-hat, continuing through beta-k-hat. There are k slope coefficients because there are k independent variables, plus one intercept, so theta-hat contains k plus one unknown numbers. The word “min” under theta-hat means that we choose all k plus one of them jointly, not one at a time while treating the others as fixed estimates.
Now look inside the square brackets. For observation i, y-i is the observed dependent variable. The expression in parentheses is its fitted value: beta-zero-hat plus beta-one-hat times x-one-i, beta-two-hat times x-two-i, continuing through beta-k-hat times x-k-i. Observed y-i minus that fitted value is u-i-hat, the residual for observation i. Squaring prevents positive and negative residuals from cancelling and penalizes large misses more heavily. The summation from i equals one to n adds those squared misses across all n observations.
OLS chooses the coefficients that make this total as small as possible. Compared with the univariate problem, the criterion has not changed at all. The bracket simply has k regressor terms instead of one, and we optimize over k plus one unknowns instead of two. If you followed the two-parameter derivation, the next tab applies the same differentiation rule to a longer fitted value.
OLS
Find the combination of \betas that minimizes the sum of squared residuals
So,
Denoting the collection of \widehat{\beta}s as \widehat{\theta} (=\{\widehat{\beta}_0,\widehat{\beta}_1,\dots,\widehat{\beta}_k\}),
\begin{align*} Min_{\widehat{\theta}} \sum_{i=1}^n \Big[ y_i-(\widehat{\beta}_0+\widehat{\beta}_1 x_{1,i} + \widehat{\beta}_2 x_{2,i} + \dots + \widehat{\beta}_k x_{k,i}) \Big]^2 \end{align*}Exactly the univariate problem from lecture 01-2, with k+1 unknowns instead of two: an intercept and k slope coefficients.
To find an interior minimum, partially differentiate the sum of squared residuals with respect to every element of theta-hat and set each derivative equal to zero. There are k plus one coefficients, so there are k plus one first-order conditions.
Start with beta-zero-hat. Each residual contains minus beta-zero-hat, so its derivative with respect to the intercept is minus one. Applying the chain rule to a squared residual gives two times the residual times minus one. Summing across observations produces minus two times the sum of the residuals. Set that equal to zero and divide by minus two. That leaves the first displayed equation: the residuals sum to zero.
For beta-one-hat, the residual contains minus beta-one-hat times x-one-i, so the derivative is minus x-one-i. The chain rule therefore gives minus two times the sum of x-one-i times the residual. After setting it to zero and dividing by minus two, we get the second equation. The same logic for beta-two-hat yields the sum of x-two-i times the residuals equal to zero. Continue this pattern through beta-k-hat, which gives the last displayed equation with x-k-i. The vertical dots mean that every intervening regressor has its own condition of exactly this form.
Look at the shared bracket in every line. It is y-i minus the fitted value, so it is u-i-hat. The intercept condition says the fitted residuals have sample mean zero because their sum is zero. Each slope condition says the sample cross-product of that regressor with the residual is zero. Since the residual mean is zero and the model includes an intercept, this is also zero sample covariance. OLS makes residuals orthogonal to every included regressor by construction.
This resembles the population zero conditional mean assumption, but keep the logical direction straight. The population assumption is a claim about the unobserved true error and is used to establish unbiasedness. The sample orthogonality conditions here are mechanical properties of the fitted residuals, even if the model is endogenous. Seeing these sums equal zero in estimated output does not test or prove that the population assumption holds.
Partially differentiate the objective function (sum of squared residuals) with respect to each of \widehat{\theta} (=\{\widehat{\beta}_0,\widehat{\beta}_1,\dots,\widehat{\beta}_k\}) and set each to zero. Dividing each by -2 leaves k+1 equations:
\begin{align*} \sum_{i=1}^n \Big[ y_i-(\widehat{\beta}_0+\widehat{\beta}_1 x_{1,i} + \widehat{\beta}_2 x_{2,i} + \dots + \widehat{\beta}_k x_{k,i}) \Big] = & 0 \;\; (\widehat{\beta}_0) \\ \sum_{i=1}^n x_{i,1}\Big[ y_i-(\widehat{\beta}_0+\widehat{\beta}_1 x_{1,i} + \widehat{\beta}_2 x_{2,i} + \dots + \widehat{\beta}_k x_{k,i}) \Big]= & 0 \;\; (\widehat{\beta}_1) \\ \sum_{i=1}^n x_{i,2}\Big[ y_i-(\widehat{\beta}_0+\widehat{\beta}_1 x_{1,i} + \widehat{\beta}_2 x_{2,i} + \dots + \widehat{\beta}_k x_{k,i}) \Big]= & 0 \;\; (\widehat{\beta}_2) \\ \vdots \\ \sum_{i=1}^n x_{i,k}\Big[ y_i-(\widehat{\beta}_0+\widehat{\beta}_1 x_{1,i} + \widehat{\beta}_2 x_{2,i} + \dots + \widehat{\beta}_k x_{k,i}) \Big]= & 0 \;\; (\widehat{\beta}_k) \end{align*}This tab replaces the long bracket on the previous tab with its name, u-i-hat, so you can see the structure of the normal equations. The first line is the intercept condition: sum u-i-hat from i equals one to n is zero. With an intercept, the fitted residuals therefore average to zero.
Every remaining line corresponds to one slope. The sum of x-one-i times u-i-hat is zero for beta-one-hat; the sum of x-two-i times u-i-hat is zero for beta-two-hat; the pattern continues through the sum of x-k-i times u-i-hat for beta-k-hat. In compact matrix language, the columns of the regressor matrix are orthogonal to the residual vector. You do not need that notation yet, but you should remember the verbal result: fitted OLS residuals are uncorrelated in the sample with every included regressor when an intercept is present.
Count the system. One intercept condition plus k slope conditions gives k plus one equations. The unknowns are the intercept estimate plus k slope estimates, also k plus one. Having the same number of equations and unknowns is necessary, but by itself it is not enough for a unique solution. The columns of the regressor matrix must also have full column rank. In practical language, no included regressor can be an exact linear combination of the others. That is the no-perfect-collinearity condition introduced shortly.
The final sentence connects this derivation to the variance estimator later in the deck. We estimate k plus one coefficients from n observations, and those fitted coefficients impose k plus one independent restrictions on the residuals. The residual sum of squares therefore has n minus k minus one residual degrees of freedom. That is why the estimator of sigma squared divides by n minus k minus one rather than by n or n minus two.
Every bracket above is just the residual \widehat{u}_i, so the whole system is:
\begin{align*} \sum_{i=1}^n \widehat{u}_i = & 0 \;\; (\widehat{\beta}_0) \\ \sum_{i=1}^n x_{i,1}\widehat{u}_i = & 0 \;\; (\widehat{\beta}_1) \\ \sum_{i=1}^n x_{i,2}\widehat{u}_i = & 0 \;\; (\widehat{\beta}_2) \\ \vdots \\ \sum_{i=1}^n x_{i,k}\widehat{u}_i = & 0 \;\; (\widehat{\beta}_k) \end{align*}k+1 equations in k+1 unknowns. That count is what costs you k+1 degrees of freedom later.
The callout gives the conclusion for this section: OLS estimators in a multivariate model are unbiased if four multiple linear regression conditions are satisfied. These are the same basic conditions from lecture one, part three, now labeled M-L-R because the model has multiple regressors.
Condition one, M-L-R one, says the model is correctly specified. The outcome must be represented in a form that is linear in the unknown beta parameters, with the relevant regressors and appropriate transformations included. If the true relationship uses a log, a square, or an omitted interaction and your equation leaves that structure in u, the error can vary systematically with x. Calling the model correct is a substantive modeling claim, not something OLS can guarantee for you.
Condition two, M-L-R two, is random sampling. The n observations should be drawn randomly from the population of interest, so the sample carries the population relationship rather than a selection rule tied to the outcome or error. Dependence across observations, such as repeated observations on one person or spatially linked counties, requires methods beyond this simple sampling condition.
Condition three, M-L-R three, is no perfect collinearity. None of the independent variables can be an exact linear combination of the others, and each regressor must have usable variation after the others are accounted for. This generalizes the univariate requirement that x varies. Without it, the data cannot separately identify all the slopes.
Condition four, M-L-R four, is zero conditional mean: expected u given x-one, x-two, through x-k equals zero. For every joint combination of the included regressors, the omitted component must average to zero. This is the condition that rules out endogeneity and gives OLS its unbiasedness. The earlier conditions are often more directly addressed through design, data structure, and specification. This fourth condition is the one empirical researchers can spend a career defending because the relevant error is unobserved. Adding regressors changes exactly what it asks: the mean of u must be zero conditional on all of them at once. These four conditions concern unbiasedness. Homoskedasticity is not on this slide because it is not required for unbiased OLS coefficients; it will enter when we derive the conventional variance formula.
Important
OLS estimators of multivariate models are unbiased if the following conditions are satisfied.
Condition 1
Your model is correct (Assumption MLR.1)
Condition 2
Random sampling (Assumption MLR.2)
Condition 3
No perfect collinearity (Assumption MLR.3)
Condition 4
Zero Conditional Mean (Assumption MLR.4)
E[u|x_1,x_2,\dots,x_k]=0
No perfect collinearity is the genuinely new-looking condition in the multivariate list. The slide states that an independent variable cannot be a linear function of the other variables. More precisely, no regressor column can be an exact linear combination of the remaining regressor columns, including the intercept. The word perfect matters. High correlation can make an estimate imprecise, but it does not make OLS algebraically impossible. Exact dependence does.
The displayed wage equation gives a deliberately silly example. It includes education once under the abbreviated name pronounced “ed-yook,” and again as three times education. Whenever education rises by one year, the second regressor rises by exactly three. There is no observation in which those two variables move independently. The fitted contribution is beta-one times education plus beta-two times three education, which can be rewritten as beta-one plus three beta-two, all multiplied by education. The data can identify that combined coefficient, but it cannot determine a unique beta-one and beta-two separately. Infinitely many pairs give the same fitted values and the same residual sum of squares.
Software has to resolve this before estimation. Fix-est generally detects the collinear column, removes one redundant term, and reports that removal rather than estimating two separate effects. That is not evidence that the dropped variable has no effect. It means its coefficient is not separately identified under this specification.
This becomes practically important when we study dummy variables. If an intercept and an indicator for every mutually exclusive category are included, the category indicators add exactly to the intercept column. One category must serve as the omitted reference group, or the intercept must be removed. We will return to that dummy-variable trap later. For now, remember the identification question: after holding all other regressors fixed, must there be some independent variation left in x-j to estimate beta-j?
No Perfect Collinearity (MLR.3)
Any variable cannot be a linear function of the other variables
Example (silly)
\begin{align*} wage = \beta_0 + \beta_1 educ + \beta_2 (3\times educ) + u \end{align*}( More on this later when we talk about dummy variables)
Here is endogeneity written formally, although we have been circling the idea since lecture zero. The callout says that the expected value of u conditional on x-one through x-k equals some function of those regressors and is not zero. In other words, once you know the included x values, you can systematically predict part of what the model placed in the error. That violates M-L-R four. OLS then treats some movement associated with the error as if it belonged to the included regressors, so its coefficient estimators need not be unbiased or causal.
The first listed cause is functional-form misspecification. Read the two wage equations line by line. In the true equation, x-one affects wage through log x-one, while x-two enters linearly. In your equation, you mistakenly enter x-one itself. To see where the mistake goes, equate the two right-hand sides. The shared intercept and beta-two x-two terms cancel. Your error u-two must then equal the true error u-one plus beta-one times the bracket, log x-one minus x-one. That extra bracket is a deterministic function of x-one. Therefore the conditional mean of u-two changes with x-one even if the original u-one had conditional mean zero. The wrong functional form creates endogeneity by construction. Of course, log x-one also requires x-one to be positive, which must make sense for the application.
The second cause is omitting a variable that is correlated with any included regressor. The wage examples showed the mechanism: if ability belongs in the wage equation but remains in u and moves with education, the education coefficient can absorb some of the ability relationship. Adding observed controls can address specific omissions, but unobserved confounders remain.
The final blue bullet warns that omitted variables and functional form are not the only sources. Later in the course we will encounter issues such as simultaneous causality, measurement error, and selection, each of which can make the conditional error depend on regressors. So endogeneity is a family of violations summarized by the same mathematical condition, not the name of one particular mistake.
Endogeneity: Definition
E[u|x_1,x_2,\dots,x_k] = f(x_1,x_2,\dots,x_k) \ne 0
What could cause endogeneity problem?
Condition five is homoskedasticity. The displayed assumption says that the variance of u conditional on x-one through x-k equals sigma squared. Sigma squared is one constant number, so the spread of the unobserved determinants around their conditional mean must be the same at every joint combination of regressor values. The errors themselves can differ across observations, and homoskedasticity does not say they are all small or normally distributed. It says their conditional variance does not systematically expand or contract with the x variables.
Under M-L-R one through M-L-R five, and conditional on the observed sample values of the independent variables, the variance of beta-j-hat is sigma squared divided by S-S-T-j times one minus R-squared-j. The numerator, sigma squared, measures the noise remaining in y. More noise makes the coefficient estimate vary more from sample to sample.
Now unpack the denominator. S-S-T-j is the sum, over i from one to n, of x-j-i minus x-j-bar, squared. It measures the total sample variation in regressor x-j around its sample mean. More variation in x-j gives OLS more information for estimating its slope, so a larger S-S-T-j lowers the variance.
R-squared-j comes from an auxiliary regression in which x-j is the dependent variable and all the other independent variables, plus an intercept, are the regressors. It measures the fraction of x-j’s sample variation that those other regressors explain. Therefore one minus R-squared-j is the unexplained fraction, and S-S-T-j times one minus R-squared-j is the variation in x-j left after partialling out the others. That is the independent variation OLS can use to identify beta-j.
Read the note at the bottom as a limiting argument. If R-squared-j is close to zero, the other regressors explain little of x-j, and most of its variation remains useful. As R-squared-j approaches one, one minus R-squared-j approaches zero, the denominator becomes small, and the variance of beta-j-hat becomes very large. At exactly one, x-j is perfectly collinear and the coefficient is not separately estimable. Near one, it is estimable but imprecise. That is multicollinearity, the subject of lecture four. The formula also previews the Frisch-Waugh-Lovell theorem: coefficient precision depends on the part of x-j that the other regressors cannot explain.
Condition 5
Error term is homoskedastic (Assumption MLR.5)
\begin{align*} Var(u|x_1,\dots,x_k)=\sigma^2 \end{align*}Under conditions MLR.1 through MLR.5, conditional on the sample values of the independent variables,
Variance of \widehat{\beta}_{OLS}
where
Why R_j^2 is in there
R_j^2 measures how much of x_j the other regressors already explain. As it approaches 1, the term 1-R_j^2 approaches 0 and the variance explodes . Intuitively: once the others account for nearly all of x_j’s movement, there is almost no independent variation left to identify \beta_j from. That is multicollinearity, and it is the whole of lecture 04.
The variance formula on the previous tab contains sigma squared, the conditional variance of the unobserved population error. We cannot observe u, so we cannot calculate sigma squared directly. As in univariate regression, we estimate it using the fitted residuals, u-i-hat. Each residual is observed y-i minus its OLS fitted value. Squaring them and adding across observations gives the residual sum of squares.
In the univariate model, there is one slope and one intercept. Estimating those two coefficients imposes two normal-equation restrictions on the residuals, so the residual degrees of freedom are n minus two. The displayed estimator divides the sum of squared residuals by n minus two, not by n.
In the multivariate model on the lower half, k counts the independent variables, and the intercept is an additional coefficient. We estimate k plus one parameters in total. The residual degrees of freedom are therefore n minus open-parenthesis k plus one close-parenthesis, which is n minus k minus one. The estimator is the sum of u-i-hat squared divided by that quantity. Under the stated classical assumptions, this degrees-of-freedom correction makes sigma-hat squared unbiased for sigma squared. Dividing by n would tend to understate the error variance because the fitted regression has already chosen coefficients that make these in-sample residuals as small as possible.
The sentence at the bottom connects the count to the first-order conditions. The residual vector has n entries, but the intercept and k regressor orthogonality equations impose k plus one independent linear restrictions when the regressor matrix has full column rank. Thus only n minus k minus one residual directions are free. Equivalently, with a suitable full-rank subset fixed, those restrictions determine the remaining k plus one components. Every parameter estimated costs one residual degree of freedom. When k grows while n is fixed, fewer degrees of freedom remain for estimating the noise, another reason not to add controls without thought.
Just like uni-variate regression, you need to estimate \sigma^2 if you want to estimate the variance (and standard deviation) of the OLS estimators.
uni-variate regression
\begin{align*} \widehat{\sigma}^2=\sum_{i=1}^n \frac{\widehat{u}_i^2}{n-2} \end{align*}multi-variate regression
A model with k independent variables with intercept.
\begin{align*} \widehat{\sigma}^2=\sum_{i=1}^n \frac{\widehat{u}_i^2}{n-(k+1)} \end{align*}You solved k+1 simultaneous equations to get \widehat{\beta}_j (j=0,\dots,k). So, once you know the value of n-k-1 of the residuals, you know the rest.
Now put the two previous tabs together. The population variance formula for beta-j-hat contains the unknown sigma squared. Replace it with the residual-based estimator sigma-hat squared. The result is an estimator of the variance of the OLS estimator, which is why the word estimator is red and why there is a hat over the word variance in the callout.
Read the first equality. Estimated variance of beta-j-hat equals sigma-hat squared divided by S-S-T-j times one minus R-squared-j. S-S-T-j and R-squared-j are calculated from the observed x values. Sigma-hat squared is calculated from the OLS residuals. Everything on the right is therefore observable or computable from the sample.
The second equality substitutes the formula for sigma-hat squared explicitly. Inside the first parentheses, sum the squared residuals and divide by n minus k minus one. Then multiply by one over S-S-T-j times one minus R-squared-j. This makes the sources of uncertainty visible. Larger residual noise raises estimated variance. More total variation in x-j lowers it. More overlap between x-j and the other regressors raises it because R-squared-j grows and the usable fraction, one minus R-squared-j, shrinks.
The standard error reported underneath beta-j-hat in a regression table is the positive square root of this estimated variance. We use that standard error to construct t statistics, confidence intervals, and hypothesis tests. Notice how many assumptions and corrections lead to the number: M-L-R one through M-L-R four give unbiased coefficients, M-L-R five gives this conventional homoskedastic conditional variance formula, and the n minus k minus one divisor corrects for the coefficients estimated before measuring residual variation. If homoskedasticity fails, the coefficient estimator may still be unbiased under the first four assumptions, but this particular standard-error formula is generally wrong. Later we will use heteroskedasticity-robust standard errors for that case.
Using the estimator of \sigma^2 in place of \sigma^2, we have the estimator of the variance of the OLS estimator.
Estimator of the variance of the OLS estimator
This section is optional, but I encourage you to follow it because the Frisch-Waugh-Lovell theorem explains what “holding other factors fixed” means mechanically. The model at the top has an intercept, three regressors, and an error: y-i equals beta-zero plus beta-one x-one-i plus beta-two x-two-i plus beta-three x-three-i plus u-i. Suppose beta-one is the only coefficient you care about. It measures the partial association between x-one and y while x-two and x-three are held fixed.
Method one is regular multiple regression. Regress y on x-one, x-two, and x-three with an intercept, and estimate beta-zero through beta-three simultaneously. The coefficient attached to x-one is beta-one-hat. This is the procedure you would normally run.
Method two reaches that one coefficient in three regressions. First, regress y on x-two and x-three with an intercept, and save the residual for every observation. Call those residuals u-y-hat. They are the part of y left after its linear projection on the intercept, x-two, and x-three. Second, regress x-one on x-two and x-three with an intercept and save u-x-one-hat. These residuals are the part of x-one that the same controls do not linearly explain. Third, regress u-y-hat on u-x-one-hat. The displayed equation writes its slope as alpha-one and its new error as v-three. Because both residual series have sample mean zero, an included intercept in this final regression is estimated as zero, so omitting it from the written equation does not change the slope.
The theorem says alpha-one-hat from that third regression is exactly equal to beta-one-hat from method one. This is an algebraic identity for the same sample and the same set of controls, not an approximation and not an asymptotic claim. Method one is convenient for estimation. Method two reveals the comparison OLS is making: it relates the part of y not explained by the controls to the part of x-one not explained by those controls. The next slide lets you run both routes and verify the equality.
Consider the following simple model,
\begin{align*} y_i = \beta_0 + \beta_1 x_{1,i} + \beta_2 x_{2,i} + \beta_3 x_{3,i} + u_i \end{align*}Suppose you are interested in estimating only \beta_1.
Let’s consider the following two methods,
Method 1: Regular OLS
Regress y on x_1, x_2, and x_3 with an intercept to estimate \beta_0, \beta_1, \beta_2, \beta_3 at the same time (just like you normally do)
Method 2: 3-step
Frisch-Waugh–Lovell theorem
Methods 1 and 2 produce the same coefficient estimate on x_1
\widehat{\beta}_1 = \widehat{\alpha_1}
Rather than take the theorem on faith, run all four steps on the same simulated data. In step one, the library line loads fix-est and makes fee-ols available. Set-seed with four eight nine one fixes the random draws, and capital N equals two hundred sets the number of observations. X-two and x-three are independent standard normal draws. X-one is then built as one-half times x-two minus three-tenths times x-three plus a fresh normal draw. This construction deliberately makes x-one share variation with both controls, so partialling them out has real work to do.
The y line sets the true data-generating process: an intercept of one, a slope of two on x-one, a slope of three on x-two, a slope of negative one on x-three, and a new normal error. Data-frame collects the four observed variables into d. We know the true x-one effect is two, but sampling noise means the estimate in this particular data set need not equal two exactly.
Step two implements regular OLS. The formula y tilde x-one plus x-two plus x-three regresses y on all three variables with the default intercept. The data equals d argument tells fee-ols where to find them. The coefficient function applied to m-one extracts the coefficient vector, and brackets containing the quoted name x-one select only the x-one estimate. Running this exact cell returns about two point zero four six six seven two.
Step three implements partialling out. The first inner fee-ols call regresses y on x-two and x-three, and the resid function extracts its residuals into a new column called u underscore y. The second does the same with x-one as the dependent variable, saving u underscore x-one. Then fee-ols regresses the y residuals on the x-one residuals. Its x-one-residual slope, selected by applying the coefficient function to m-two and taking the entry named u underscore x-one, is also about two point zero four six six seven two. The final regression includes an intercept by default; it is essentially zero because both residual vectors have mean zero.
Step four uses the c function to combine the method-one and method-two coefficients side by side under descriptive labels. Look at the printed values after running it. They agree to every displayed digit. With this exact seed, method one minus method two is negative four point four four zero eight nine two times ten to the minus sixteen. That tiny discrepancy is floating-point rounding at machine precision, not an econometric difference. These are not two unrelated procedures that happen to be close. They are the same OLS coefficient computed through two algebraically equivalent routes.
Do not take it on faith. Run both methods on the same simulated data and compare.
Step 1: generate the data. x1 is built from x2 and x3, so they really do share variation.
Step 2: Method 1. Regress y on everything at once.
Step 3: Method 2. Partial x_2 and x_3 out of both y and x_1, then regress one set of residuals on the other.
Step 4: compare.
Not merely close: identical to machine precision. The difference is about -4\times 10^{-16}, which is rounding, not disagreement.
Now read the three steps for meaning rather than code mechanics. In step one, regress y on x-two and x-three with an intercept. For each observation, u-y-hat is observed y minus the value predicted from those controls. It is the part of y that is orthogonal in the sample to the intercept, x-two, and x-three. The slide calls it void of their impact. Interpret that as removing their linear fitted component. By itself, this projection is not a causal claim about the true effects of x-two and x-three.
In step two, regress x-one on the same controls with an intercept. U-x-one-hat is observed x-one minus the part predicted by x-two and x-three. It contains the remaining variation in x-one that has zero sample covariance with each of those controls. For example, if x-two predicts that an observation should have x-one equal to five but its actual x-one is seven, this residual contributes the unexplained difference of two. This is the variation in x-one available after holding the controls fixed in the linear regression sense.
Step three regresses the y leftovers on the x-one leftovers. Any variation in y or x-one that x-two and x-three linearly account for has been removed before this comparison. The resulting slope asks how residual y changes with residual x-one. The Frisch-Waugh-Lovell theorem proves that this slope is identical to the coefficient on x-one from regressing y directly on x-one, x-two, and x-three together.
This is the partialling-out interpretation of a multiple-regression coefficient. OLS does not estimate beta-one from every bit of x-one’s variation. It estimates beta-one from the component of x-one that cannot be linearly predicted by the other included regressors. That is also why multicollinearity raises variance: if the controls explain nearly all of x-one, u-x-one-hat has very little variation left from which to estimate a slope.
Step 1
Regress y on x_2 and x_3 with an intercept and get residuals, which we call \widehat{u}_y
Step 2
Regress x_1 on x_2 and x_3 with an intercept and get residuals, which we call \widehat{u}_{x_1}
Step 3
Regress \widehat{u}_y on \widehat{u}_{x_1}, which produces an estimate of \beta_1 identical to the one you get from regressing y on x_1, x_2, and x_3
Here is the main lesson from the whole optional section. When you regress y on x-one, x-two, and x-three, the coefficient on any one variable can be understood as a univariate regression after the linear contributions of all the other included variables have been partialled out of both that variable and y. For beta-one-hat, remove from y what x-two and x-three predict, remove from x-one what x-two and x-three predict, and relate the two residual series. The resulting slope is exactly the multiple-regression coefficient.
That is what “holding the other included variables fixed” means in OLS mechanics. The estimate uses comparisons in x-one that are not explained by the included controls. If education is the variable of interest and experience is a control, OLS uses the part of education variation not linearly predicted by experience and compares it with the part of wage variation not linearly predicted by experience.
Now read the second bullet, which gives the practical payoff. Including a relevant variable can prevent its modeled relationship with y from being confused with the relationship you want to attribute to x-one. In the spending example, controlling for average income removes the linear income-related component from both school spending and test scores before estimating the spending coefficient. That is more informative than a simple raw association.
But carry forward the caveat from the beginning of the lecture. Partialling out works only for variables actually included in the model and only for the functional form used. It does nothing to remove an unmeasured ability, an omitted family characteristic, or an incorrectly modeled nonlinear relationship. Nor does the algebra itself prove causality. A causal interpretation still requires the remaining error to have zero conditional mean given all included regressors. Multivariate OLS is a precise way to control for observed modeled factors, not a machine that automatically eliminates endogeneity.
Regressing y on all explanatory variables (x_1, x_2, and x_3) in a multivariate regression is as if you are looking at the impact of a single explanatory variable with the effects of all the other variables partialled out
In other words, including variables beyond your variable of interest lets you control for (remove the effect of) other variables, avoiding confusing the impact of the variable of interest with the impact of other variables.