Let’s start with a concrete data set, because everything from here on is done on real numbers. The data section says we have observations on five hundred and forty-six houses. For every house, we observe its sale price in dollars and its lot size. The subscript i labels a particular house, so price-i and lot-size-i belong to the same observation.
Now look at the model. Price-i equals beta-zero, plus beta-one times lot-size-i, plus u-i. Beta-zero is the intercept, the baseline component when lot size is zero and u is zero. That value helps position the line, although a zero-sized lot is not economically meaningful here. Beta-one is the slope of the model’s systematic component. It tells us how that component changes when lot size rises by one unit. Interpreting the line as the conditional expectation, or the slope as a causal effect, requires restrictions on u-i. The objective at the bottom is to estimate that slope, which the slide calls the impact of lot size on price.
Notice how much work u-i is doing. It contains every determinant of this house’s price that is not explicitly represented by lot size, including bedrooms, bathrooms, condition, neighbourhood, and the year of sale. It can also contain measurement error and purely idiosyncratic influences. That matters because lot size may be related to some of those omitted factors. Today we learn how OLS mechanically estimates beta-one. Whether that estimate can be interpreted as a causal, everything-else-fixed effect requires assumptions about u-i, which come next in the course. Flip to the R-code tab and we will bring these observations into the browser.
Data
Observations of house price and lot size for 546 houses.
Model
price_i = \beta_0 + \beta_1 lotsize_i+u_i
Objective
Estimate the impact of lot size on house price
Here is the code that gets you the data. This is a live R session running inside your browser, so you can edit the code and press Run Code. The autorun option means the cell should also execute when it becomes ready. The first time can take a moment because the browser has to initialize R and load the packages.
Read the code from the top. Library, with A-E-R inside the parentheses, attaches the AER package for this session. That package supplies the HousePrices data. The next line calls data with HousePrices, which loads that named data set into memory as an object also called HousePrices. Loading a package makes its functions and data definitions available, while data actually creates the data object we want to inspect.
The last line uses head, which displays the first six rows. Inside head, HousePrices bracket comma c of price and lotsize selects columns from the data frame. Leaving the position before the comma empty means keep every row. The c function combines the two quoted column names, so we keep only price and lotsize, in that order. The quotation marks matter because those are names, not objects to evaluate.
Run it and read the result. The first house has a price of forty-two thousand dollars and a lot size of five thousand eight hundred fifty; the next rows begin thirty-eight thousand five hundred with four thousand, and forty-nine thousand five hundred with three thousand sixty. The complete data has twelve columns, but these are the two variables used on every slide today. This preview is a basic check that the data loaded, the names are correct, and the units look plausible. Now flip to Data visualization, because inspecting rows is not enough to reveal the relationship.
Before you estimate anything, look at your data. The horizontal axis is lot size, and the vertical axis is house price in dollars. Each dot represents one of the five hundred and forty-six houses, positioned using that house’s lot size and sale price. Two features matter. First, the point cloud generally rises as you move right, so larger lots tend to be associated with higher prices. That positive pattern tells you to expect a positive estimate of beta-one. Second, look at the large vertical spread among houses with similar lot sizes. Around a lot size of four thousand, for example, prices range from roughly thirty thousand dollars to well above one hundred thousand. Lot size is informative, but it plainly cannot explain price by itself.
That unexplained vertical variation is where the error term shows up in the data. It is not literally a set of observed u-i values, because we never observe the true population regression function and therefore cannot calculate the true errors. But it makes the idea concrete: houses with the same or similar lot size can have very different prices because bedrooms, location, condition, and other factors differ.
The code builds the figure in layers. G-g-plot, with data equal to HousePrices, establishes the data source and saves the unfinished plot as g-h-p. Geom-point adds the observations. Inside aes, y equals price and x equals lotsize map those columns to the two axes; size equal to zero point six is outside aes because every dot should have the same small size. Y-lab and x-lab supply the readable axis titles. Theme-lecture applies the deck’s common appearance. The final line prints g-h-p. We save that plot because later tabs add different candidate lines to the same point cloud. Our next question is which line should be added and why.
So here is the goal as both a picture and two bullets. We want a straight line through the cloud of house prices and lot sizes. Its vertical intercept will estimate beta-zero, and its slope will estimate beta-one, the change in predicted price for a one-unit increase in lot size. The line shown rises from left to right, matching the positive association we saw in the scatterplot.
The code starts with g-h-p, the saved scatterplot from the previous tab, and uses a plus sign to add geom-smooth. Inside aes, y is price and x is lotsize, so the new layer uses the same two variables. Method equal to quote l-m tells R to fit an ordinary linear model rather than a flexible curve. S-e equal to false suppresses the shaded confidence band, leaving just the fitted line so we can focus on the mechanics.
The second bullet names the method: ordinary least squares, or OLS. You will use it throughout the course. But notice that merely drawing a plausible line has not solved the problem. There are infinitely many possible intercepts and slopes. A line could be a little steeper, flatter, higher, or lower and still pass through the cloud. We therefore need a rule that ranks candidate lines. Flip through the examples next, where two visibly plausible lines imply very different slopes.
Let’s take the choice of a line seriously by trying candidates. The heading gives this one’s two coefficients: beta-zero-hat equals twenty thousand and beta-one-hat equals seven. The hat tells you these are proposed estimates, not the unknown population parameters. The red line therefore has the equation predicted price equals twenty thousand plus seven times lot size. Its intercept says the line would predict twenty thousand dollars at a lot size of zero. Its slope says one additional unit of lot size is associated with seven additional dollars of predicted price, so one thousand additional units would correspond to seven thousand dollars.
The code adds geom-abline to the saved scatterplot. Intercept equal to twenty thousand fixes where the line crosses the vertical axis, slope equal to seven fixes its rise per horizontal unit, and color equal to red distinguishes this candidate from the points. On screen, lot size is still on the horizontal axis and house price is on the vertical axis. The line is not absurd; it passes through the broad middle of the cloud. But visual plausibility alone does not prove it is best. Keep this slope of seven in mind, then flip to the second example.
Here is the rival candidate. Its beta-zero-hat is seventy thousand and its beta-one-hat is three point eight, so the equation is predicted price equals seventy thousand plus three point eight times lot size. Compared with the previous line, this one begins fifty thousand dollars higher at zero and rises more slowly. One additional unit of lot size corresponds to only three dollars and eighty cents of predicted price, or three thousand eight hundred dollars for an additional thousand units.
The code again adds geom-abline to g-h-p. Intercept equal to seventy thousand and slope equal to three point eight define the line, while color equal to red makes it easy to see against the scatter. Flip back and forth between this tab and Example 1. Both red lines pass through substantial parts of the point cloud, and neither is obviously impossible by eye. Yet their economic implications differ sharply: one values an extra unit of lot size at seven dollars, the other at less than four. If you were advising someone about buying more land, that gap would matter. Eyeballing cannot resolve it consistently, so the next tab asks us to define what “best” means.
The callout collects the problem raised by the two examples. The first bullet asks which pair, among all possible values of beta-zero-hat and beta-one-hat, is best. Each pair defines one entire line, so this is not a choice of the intercept and slope separately. Moving either coefficient changes the fitted values for the observations.
The second bullet is logically prior: what criterion will define “best”? The data do not label one line as the winner. We must choose a loss function, a numerical score that translates the gaps between a candidate line and the observed points into one measure of fit. Then every possible line receives a score, and “best” can mean the coefficient pair with the smallest score. A clear criterion makes the result reproducible. Two researchers using the same data and rule will choose the same line, rather than defending different visual impressions.
The next tab introduces the individual gaps that the criterion will summarize. Those gaps are residuals. After that, OLS will square and add them, turning the search for a line into a minimization problem.
Question
Here is the raw material for our criterion. Once you pick beta-zero-hat and beta-one-hat, observation i receives the modeled, or fitted, value beta-zero-hat plus beta-one-hat times x-i. The equation defines u-i-hat as y-i minus that fitted value. Y-i is the observed dependent variable for observation i, x-i is its explanatory variable, and the two beta-hats define the candidate line. A positive residual means the point lies above the line because actual y exceeds fitted y. A negative residual means the point lies below it. The residual’s absolute magnitude is the vertical miss, measured in the units of y, which here are dollars.
Keep residual and error distinct. The population error u-i is the gap from the unknown population relationship, so it is unobservable. A residual u-i-hat is the gap from a line estimated or proposed using the sample, so we can compute it. Change either candidate coefficient and the fitted line moves, which changes every residual.
Now look at the figure. Lot size is on the horizontal axis and house price in dollars is on the vertical axis. The code uses set-seed with five hundred forty-three so the same random subset appears every time. Sample chooses twenty-five row numbers from one through n-row of HousePrices, and the brackets keep those rows and every column in h-p-few. L-m estimates price tilde lotsize on that subset as fit-few.
G-g-plot then uses h-p-few and maps lotsize to x and price to y. Geom-segment draws each red residual: x and x-end are both that house’s lot size, so the segment is vertical; y begins at the observed price, and y-end equals fitted of fit-few, the point on the estimated line. Geom-point draws the observations with size one point two. Geom-smooth, with method l-m and s-e false, draws the fitted straight line without a confidence band. The labels and theme finish the plot. Every red segment therefore connects one observed point to its fitted value. Long segments are large misses. The next tab explains how OLS converts all these signed misses into one score.
For particular values of \widehat{\beta}_0 and \widehat{\beta}_1 you pick, the modeled value of y for individual i is \widehat{\beta}_0 + \widehat{\beta}_1 x_i.
Then, the residual for individual i is:
\widehat{u}_i = y_i - (\widehat{\beta}_0 + \widehat{\beta}_1 x_i)
That is, the residual is the observed value of the dependent variable less the modeled value. For different values of \widehat{\beta}_0 and \widehat{\beta}_1, you have a different value of residual.
The vertical red segments below are the residuals: each one runs from an observed point to the line.
Now we can state the OLS criterion. The prose says to find beta-zero-hat and beta-one-hat that minimize the sum of the squared residuals. In other words, evaluate how far every observation is from a candidate line, square those distances, add them, and choose the coefficient pair with the smallest total. The name ordinary least squares is literal: choose the least possible sum of squares.
Read the mathematical line aloud. Minimize, with respect to beta-zero-hat and beta-one-hat, the sum from i equals one to n of u-i-hat squared. The index i runs across all n observations, so every house affects the criterion. The text after “where” substitutes the definition of a residual: u-i-hat equals y-i minus the quantity beta-zero-hat plus beta-one-hat times x-i. Thus the objective is entirely a function of observed x and y values and the two coefficients we are choosing.
Squaring makes every contribution nonnegative, so a residual of minus ten and a residual of plus ten contribute equally, one hundred, rather than canceling. It also gives larger misses disproportionate weight. The criterion is measured in squared units of y, squared dollars here, but its absolute unit is less important than which line makes it smallest. This slide defines the task; the Visualization tab asks why squares are the chosen loss, and the Derivation tab solves the minimization.
Idea of OLS (Ordinary Least Squares)
Let’s find the values of \widehat{\beta}_0 and \widehat{\beta}_1 that minimize the sum of the squared residuals!
Mathematically
Solve the following minimization problem:
Min_{\widehat{\beta}_0,\widehat{\beta}_1} \sum_{i=1}^n \widehat{u}_i^2, \text{where} \;\; \widehat{u}_i=y_i-(\widehat{\beta}_0+\widehat{\beta}_1 x_i)
This slide asks you not to take the word “squares” on faith. First orient yourself in the synthetic figure. The horizontal axis is x and the vertical axis is y. The code creates twenty x values with two hundred times run-if, so they lie between zero and two hundred. It creates y as one hundred plus zero point five times x plus random normal noise scaled by thirty. Data-table stores those vectors as columns y and x, and l-m estimates a line whose coefficients are saved in coefs. G-g-plot draws the points, geom-abline draws that fitted intercept and slope, and the horizontal and vertical reference lines mark zero. The red horizontal and curved vertical marks illustrate a horizontal change and the corresponding rise along the fitted line, which is how slope connects changes in x to changes in fitted y. The names ending in “demo” keep this illustration separate from the house-price objects used later.
Now read the first question. If we simply sum raw residuals, observations above the line contribute positive values and observations below contribute negative values. Large errors can cancel even though the line fits poorly. In fact, once a regression includes an intercept, the OLS residuals sum to zero by construction, so the raw sum cannot tell you the quality of that fitted line at all. Squaring prevents cancellation because every squared residual is nonnegative. It also makes a residual twice as large contribute four times as much, so OLS penalizes a few very large misses heavily.
The second question asks why not use absolute values, which also remove signs without squaring. We can. Minimizing the sum of absolute residuals gives least absolute deviations, closely related to median regression, and it is less sensitive to extreme residuals. OLS is especially convenient because squared residuals are smooth and differentiable. That lets us obtain closed-form coefficient formulas and later derive sampling variances cleanly. So the loss function is a modeling choice with consequences, not a law of nature. Flip to Derivation to see the calculus that squares make possible.
Questions
Why do we square the residuals, and then sum them up together? What’s gonna happen if you just sum up residuals?
How about taking the absolute value of residuals, and then sum them up?
Here is the calculus, and you should follow every line once even if software will do it afterward. The minimization problem at the top chooses beta-zero-hat and beta-one-hat to minimize the sum, from i equals one to n, of y-i minus beta-zero-hat minus beta-one-hat x-i, all squared. The slide then gives that objective a short name, S of beta-zero-hat comma beta-one-hat. S is one number for any proposed coefficient pair.
Because there are two coefficients, we need two first-order conditions. For beta-zero-hat, differentiate each squared bracket using the chain rule. The derivative of the square supplies two times the bracket, and the derivative of the bracket with respect to beta-zero-hat is minus one. That produces minus two times the sum of the residuals. At a minimum, set it equal to zero. For beta-one-hat, the derivative of the bracket is minus x-i, so the second condition is minus two times the sum of x-i times u-i-hat, again set equal to zero.
Dividing by minus two leaves the two conditions displayed next. The sum of all OLS residuals is zero. Also, the sum of x-i times its residual is zero. These are not behavioral assumptions about the population. They are mechanical consequences of fitting OLS with an intercept. The first says positive and negative residuals balance. Combined with the definition of the sample mean, it also implies the average fitted value equals y-bar. The second says the sample residual vector is orthogonal to x, which is the geometry behind the OLS solution.
To see how the coefficient formulas follow, start with the first condition and substitute u-i-hat. Sum of y-i minus n beta-zero-hat minus beta-one-hat times sum of x-i equals zero. Divide by n and solve to obtain beta-zero-hat equals y-bar minus beta-one-hat x-bar. That is why the fitted line passes through the sample-means point, x-bar comma y-bar.
Insert that intercept expression into the second condition and rewrite variables as deviations from their sample means. Solving gives beta-one-hat equal to the sum of x-i minus x-bar times y-i minus y-bar, divided by the sum of x-i minus x-bar squared. The numerator measures how x and y move together; the denominator measures the sample variation in x. The denominator must be positive, so x cannot be identical for every observation. Finally, beta-zero-hat is recovered from y-bar minus beta-one-hat x-bar. The bottom line defines y-bar as the sum of the y values divided by n and x-bar analogously. These formulas are the analytical OLS estimators. The next tab explains why “estimator” is not the same word as “estimate.”
Minimization problem to solve
Min_{\widehat{\beta}_0,\widehat{\beta}_1} \sum_{i=1}^n [y_i-(\widehat{\beta}_0+\widehat{\beta}_1 x_i)]^2
Steps
Min_{\widehat{\beta}_0,\widehat{\beta}_1} \sum_{i=1}^n [y_i-(\widehat{\beta}_0+\widehat{\beta}_1 x_i)]^2
Write the objective as S(\widehat{\beta}_0,\widehat{\beta}_1) = \sum_{i=1}^n [y_i-(\widehat{\beta}_0+\widehat{\beta}_1 x_i)]^2.
FOC
\def\sumn{\sum_{i=1}^{n}} \begin{align*} \frac{\partial S}{\partial \widehat{\beta}_0}=& -2 \sumn [y_i-(\widehat{\beta}_0+\widehat{\beta}_1 x_i)] = -2\sumn \widehat{u}_i = 0 \\\\ \frac{\partial S}{\partial \widehat{\beta}_1}=& -2 \sumn x_i\cdot [y_i-(\widehat{\beta}_0+\widehat{\beta}_1 x_i)] = -2\sumn x_i\cdot \widehat{u}_i = 0 \end{align*}
Dividing each by -2 leaves the two conditions OLS always satisfies:
\def\sumn{\sum_{i=1}^{n}} \sumn \widehat{u}_i = 0 \;\;\;\text{and}\;\;\; \sumn x_i \widehat{u}_i = 0
OLS estimators: analytical formula
\def\sumn{\sum_{i=1}^{n}} \begin{aligned} \widehat{\beta}_1 & = \frac{\sumn (x_i-\bar{x})(y_i-\bar{y})}{\sumn (x_i-\bar{x})^2},\\\\ \widehat{\beta}_0 & = \bar{y}-\widehat{\beta}_1 \bar{x}, \\\\ \text{where} & \;\; \bar{y} = \sumn y_i/n \;\; \text{and} \;\;\bar{x} = \sumn x_i/n \end{aligned}
This is a small vocabulary distinction with a large payoff later, so let’s be precise. The first block defines an estimator as a rule or formula to apply once you receive data. Before sampling, it is a random quantity because its inputs, the sample observations, are random. The second block defines an estimate as the realized number obtained after plugging one particular data set into that rule.
Look at the two bullets under “In this lecture.” The fraction with the sum of x-i minus x-bar times y-i minus y-bar in the numerator and the sum of x-i minus x-bar squared in the denominator is the slope estimator. It tells you what calculation to perform for any sample with variation in x. When that rule is applied to these five hundred and forty-six HousePrices observations, it returns six point five nine nine after rounding to three decimal places. That number is the slope estimate.
Why insist on the distinction? Imagine drawing a different sample of houses. We would use exactly the same estimator, but the sample x values, y values, and their means would change, so the resulting estimate would generally change. Across repeated samples, beta-one-hat has a sampling distribution. Its center, spread, and shape determine whether our estimate is reliable and how uncertain we should be. That variability is the subject of the next lecture. First, the R demonstration will show that the formula and regression software produce the same estimate.
Estimators
Specific rules (formula) to use once you get the data
Estimates
Numbers you get once you plug values (your data) into the formula
In this lecture
HousePrices data, is the estimate Feed the same estimator a different sample and you get a different estimate. That is the whole subject of the next lecture.
Let’s compute beta-one-hat by hand, straight from the formulas at the top, so you can see there is no magic in the regression command. The slope is the sum of the products of each x and y deviation from their means, divided by the sum of squared x deviations. Once the slope is known, the intercept is y-bar minus beta-one-hat times x-bar. The code on this slide implements the slope formula; it does not yet calculate the intercept.
Read each line. HousePrices dollar-sign price extracts the entire price column and assigns that vector to y. HousePrices dollar-sign lotsize does the same for lot size and assigns it to x. The dollar sign selects a named component from a data frame. Because the rows stay aligned, x at position i and y at position i still describe the same house.
For b-one-num, mean of x calculates x-bar and mean of y calculates y-bar. R subtracts each mean from every element of its vector. The two centered vectors are multiplied element by element, so each house contributes x-i minus x-bar times y-i minus y-bar. Sum adds those products to form the numerator. For b-one-denom, x minus mean x is squared element by element, and sum adds the squared deviations. Dividing b-one-num by b-one-denom gives b-one, and the final line prints it.
Run the cell. The result is approximately six point five nine eight seven six eight, which rounds to six point five nine nine. Its units are dollars of house price per unit of lot size. The denominator is large when lot sizes have substantial variation; without any variation in x it would be zero and the slope could not be estimated. Note this number, then flip to the better-way tab. The software there should return exactly the same slope because it carries out this calculation internally.
OLS Estimator Formula
\def\sumn{\sum_{i=1}^{n}} \begin{aligned} \widehat{\beta}_1 & = \frac{\sumn (x_i-\bar{x})(y_i-\bar{y})}{\sumn (x_i-\bar{x})^2}\\\\ \widehat{\beta}_0 & = \bar{y}-\widehat{\beta}_1 \bar{x} \end{aligned}
R code
Here is the short way you will actually use. Library of fixest attaches the package that provides the feols function. Then uni-reg gets the result of feols with two main arguments. The first is the formula price tilde lotsize, read as “price modeled by lot size.” The variable to the left of the tilde is the dependent variable; the variable to the right is the explanatory variable. R includes an intercept by default, so this formula represents the same beta-zero plus beta-one times lotsize model used throughout the deck. The second argument, data equal to HousePrices, tells feols where to find both named columns.
The returned model object is assigned to uni-reg. The final line prints it. Run the cell and the header identifies price as the dependent variable and reports five hundred and forty-six observations. The coefficient table gives an intercept of about thirty-four thousand one hundred thirty-six point one nine and a lotsize coefficient of about six point five nine eight seven seven. That slope matches the hand calculation on the previous tab. The printout also shows standard errors, t statistics, p-values, the root mean squared error, and adjusted R-squared. Those inference columns are previewed here but developed later.
The comment shows the base R equivalent, l-m with the same formula and data arguments. For this simple regression, l-m and feols give the same OLS coefficient estimates. We use feols because fixest provides convenient tools for the more complicated models later in the course. The intercept “comes along for free” only in the sense that feols computes it automatically; mathematically it is y-bar minus the estimated slope times x-bar. Now open post-estimation to see what else the uni-reg object stores.
We can use the feols() function from the fixest package.
Lots of information is stored in the regression results (here, uni_reg), which is of class list. Each tab below pulls out one piece of it.
What comes back from feols is not just the two displayed coefficients. Uni-reg is a list-like fitted-model object containing the inputs, results, and intermediate quantities the regression computed. The code calls l-s on uni-reg. Here l-s lists the names of the components inside that object in alphabetical order; it does not print the component values themselves.
Run it and scan the output. Call records the function call, and f-m-l and f-m-l-all preserve the formula. Coefficients stores the intercept and slope estimates, while coeftable combines estimates with their standard errors and test statistics. Fitted dot values contains one prediction for every observation, and residuals contains one observed-minus-fitted gap for every observation. N-obs records the sample size, n-params the number of estimated parameters, and s-s-r the sum of squared residuals that OLS minimized. S-e, sigma-squared, the Hessian, scores, and covariance components support the uncertainty calculations introduced later. Other entries record model settings, possible collinearity information, and the original observation selection.
You do not need to memorize all these names. The point is that an R model object is a reusable container, not merely a printout. Functions such as summary and predict read components from it, and you can extract particular pieces when you need them. The next four tabs demonstrate the components that matter today: coefficients, fitted values, residuals, and a formatted summary. Start with Coefficients.
Apply ls() to see its elements:
The first component you will often want is the coefficient vector. The code uses uni-reg dollar-sign coefficients. The dollar sign reaches into the fitted model object by name, and coefficients is the component containing the OLS estimates. There are no function arguments here; the line simply extracts and prints that vector.
Run it and two labeled numbers appear. The value labeled intercept is about thirty-four thousand one hundred thirty-six point one nine. That is beta-zero-hat, the fitted price at a lot size of zero. It positions the line, but be cautious about giving it a practical interpretation because the observed lots range from one thousand six hundred fifty to sixteen thousand two hundred, nowhere near zero. The value labeled lotsize is about six point five nine eight seven six eight. That is beta-one-hat, the slope estimate we calculated by hand.
Read the slope as a sentence with units: in this sample, one additional unit of lot size is associated with about six dollars and sixty cents more in predicted house price. For a one-thousand-unit increase, linearity scales that to about six thousand five hundred ninety-nine dollars. “Associated” is the right word at this stage. Calling it a causal effect requires the omitted factors in u to satisfy the relevant zero conditional mean condition, among other assumptions. The labels are useful because they let you extract a coefficient by name later, as the exercise does. Flip to Fitted values to see what the two coefficients predict for each observed house.
Fitted values are the model’s predictions at the explanatory-variable values actually observed in the estimation sample. For house i, y-i-hat equals the estimated intercept, thirty-four thousand one hundred thirty-six point one nine, plus the estimated slope, about six point five nine eight eight, times that house’s lot size.
The code extracts uni-reg dollar-sign fitted dot values. The square brackets then subset that vector. One colon five creates the integer sequence one through five, so the display contains only the first five predictions rather than all five hundred and forty-six. Run it and you get approximately seventy-two thousand seven hundred thirty-eight point nine eight, sixty thousand five hundred thirty-one point two six, fifty-four thousand three hundred twenty-eight point four two, seventy-eight thousand eighteen, and seventy-six thousand one hundred four point three five dollars. Each corresponds, in order, to one of the first five houses shown in the data preview.
There is one fitted value per estimation observation, so the complete component has five hundred and forty-six entries. These predictions all lie on the fitted line. The first house, for example, has a lot size of five thousand eight hundred fifty, which the equation maps to about seventy-two thousand seven hundred thirty-nine dollars. Its actual price was forty-two thousand dollars. Actual minus fitted gives a negative residual of about thirty thousand seven hundred thirty-nine dollars. The next tab extracts those residuals directly.
Predicted values at the observation points:
Here are the residuals as numbers, the same quantity represented by the red vertical segments earlier. For every house, u-i-hat equals its actual price minus its fitted price. The code extracts the residuals component with the dollar sign and then uses square brackets one colon five to print the first five entries.
Run it and you get approximately minus thirty thousand seven hundred thirty-eight point nine eight, minus twenty-two thousand thirty-one point two six, minus four thousand eight hundred twenty-eight point four two, minus seventeen thousand five hundred eighteen, and minus fifteen thousand one hundred four point three five dollars. All five displayed residuals happen to be negative, so every one of these first five houses sold for less than the model predicted. For the first observation, forty-two thousand minus seventy-two thousand seven hundred thirty-eight point nine eight equals minus thirty thousand seven hundred thirty-eight point nine eight, exactly matching the output.
Across the full sample there are both positive and negative residuals. Because this OLS model includes an intercept, their full-sample sum is zero apart from numerical rounding, as the first-order condition showed. Do not infer that a short subset must sum to zero, and do not confuse “sum equals zero” with “small errors.” The magnitudes here are in dollars, and several are tens of thousands. That is a reality check: lot size helps explain price, but a one-variable model leaves much of the variation to location, house characteristics, and the other contents of u. Now flip to summary for the conventional compact report.
If you want a conventional regression report rather than one component, use summary. The code passes uni-reg as the sole argument, so summary dispatches the method designed for a fixest model and prints a structured overview.
Run it and read from the top. “Dep. Var.: price” identifies the dependent variable, and “Observations: 546” confirms the sample size. “Standard-errors: IID” says the displayed standard errors use the independent and identically distributed error formula. That assumption and alternatives to it are later topics.
The coefficient table has one row for the intercept and one for lotsize. Estimate is the coefficient itself: about thirty-four thousand one hundred thirty-six point one nine for the intercept and six point five nine eight eight for lotsize. Std. Error measures estimated sampling uncertainty. T value divides each estimate by its standard error, and Pr greater than absolute t is the two-sided p-value for testing that coefficient against zero. The stars summarize p-value thresholds. Do not use the stars as a substitute for understanding the estimand and assumptions; lectures five and six develop this inference machinery.
At the bottom, the root mean squared error is about twenty-two thousand five hundred twenty-five point seven dollars, a typical residual size expressed back in price units. Adjusted R-squared is about zero point two eight five eight; it modifies R-squared for the number of regressors. The unadjusted R-squared used later is about zero point two eight seven one. For now, focus on the estimate column and recognize the other output. The final section of this deck takes apart exactly what R-squared measures.
You can have a nice quick summary of the regression results with the summary() function:
Now that we have coefficient estimates, we can write the fitted relationship explicitly. This is the relationship you would typically report with the regression results. The model to be estimated at the top is price equals beta-zero plus beta-one times lotsize plus u. Beta-zero and beta-one are unknown population parameters, and u collects the other influences on an individual house’s price.
The estimated line below replaces the unknown coefficients with the OLS estimates. Read it as fitted, or predicted, price equals thirty-four thousand one hundred thirty-six plus six point five nine nine times lot size. Thirty-four thousand one hundred thirty-six is the fitted intercept. Six point five nine nine is the fitted slope, so increasing lot size by one unit changes the line’s predicted price by about six dollars and sixty cents; increasing it by one thousand units changes predicted price by about six thousand five hundred ninety-nine dollars. The individual residual is not included in this prediction because it is the gap between a particular observed price and the line.
The slide calls this fitted line the sample regression function, or S-R-F. It is estimated from this particular sample. Its population target is the population regression function, or P-R-F, written as the expected value of price conditional on lot size. “Conditional on” means we compare the average price among houses at a given lot size, not claim that every such house has the same price.
Read the important callout carefully. OLS predicts an expected value of the dependent variable conditional on the explanatory variables. It does not tell you the exact sale price of an individual house. A house can lie far above or below the line because its residual includes location, bedrooms, condition, and everything else. The second bullet says beta-one-hat estimates how changing x changes expected y. Mechanically that is the change along the fitted line. Interpreting it as a causal, everything-else-fixed effect also requires the assumptions introduced next in the course. The following tabs show how to obtain predictions at observed and newly chosen lot sizes.
Model to be estimated
price = \beta_0 + \beta_1 lotsize + u
Estimated Model
This is the estimated version of the expected value of y conditional on x.
price = 34136 + 6.599 \times lotsize
This is called sample regression function (SRF) , and it is an estimation of E[price|lotsize], the population regression function (PRF).
Important
OLS regression predicts the expected value of the dependent variable conditional on the explanatory variables.
\widehat{\beta}_1 is an estimate of how a change in x affects the expected value of y.
This is the fitted-values idea again, now presented as the tool you would use after estimating the model. Uni-reg dollar-sign fitted dot values reaches into the regression object and extracts the predictions at the observed lot sizes. The square brackets one colon five retain the first five entries, and autorun asks the browser to execute the cell when ready.
Run it and the first five predicted prices are approximately seventy-two thousand seven hundred thirty-nine, sixty thousand five hundred thirty-one, fifty-four thousand three hundred twenty-eight, seventy-eight thousand eighteen, and seventy-six thousand one hundred four dollars. Each is calculated as thirty-four thousand one hundred thirty-six point one nine plus six point five nine eight eight times that observation’s lot size. These are conditional mean predictions from the fitted line, not guaranteed sale prices.
The full fitted-values vector has one entry for each of the five hundred and forty-six houses used to estimate the regression. That row alignment makes it useful for comparing actual against predicted prices, plotting fit diagnostics, or identifying houses with especially large residuals. Notice the restriction stated in the slide prose: this component gives predictions only at lot sizes already present in the estimation data. To ask for a predicted price at a specifically chosen lot size, including one not observed in the sample, flip to New predictions.
You can access the predicted values at the observed points by looking at the fitted.values element of the regression results.
Predicting at chosen x values is a two-step move you will use constantly. Step one creates a new data frame. Inside data-frame, lotsize equal to c of three thousand, twelve thousand, and fifteen thousand creates one column with three rows. The c function combines those three numbers into a vector. The column must be named lotsize exactly, because that is the explanatory-variable name stored in the regression formula. The assignment saves the result as new-data. The surrounding parentheses make R both assign and print it, so the output lets you verify the three requested rows.
Step two calls predict. Its first argument is uni-reg, the fitted model that supplies the intercept and slope. Newdata equal to new-data supplies the explanatory-variable values where predictions should be evaluated. Run it and the three predicted prices, in the same row order, are about fifty-three thousand nine hundred thirty-two point four nine dollars at a lot size of three thousand, one hundred thirteen thousand three hundred twenty-one point four zero at twelve thousand, and one hundred thirty-three thousand one hundred seventeen point seven one at fifteen thousand.
Predict substitutes each lot size into the same line, thirty-four thousand one hundred thirty-six point one nine plus six point five nine eight eight times lotsize. It does not refit the regression. All three requested values lie within the sample’s observed range of one thousand six hundred fifty to sixteen thousand two hundred, although fifteen thousand is near the upper edge. Predictions far outside the observed range are extrapolations. The formula will still return a number, but the data provide little evidence that the linear relationship continues there. The exercise on the next tab uses the same slope to evaluate a change in lot size.
To calculate the predicted value at arbitrary values of x,
data.frame with values of x of your choice.predict() to the data.frame using the regression results.Here is a decision problem to work through before opening the solution. Your current lot size is three thousand units. You can add one thousand units, holding everything else fixed, and the expansion costs five thousand dollars. The economic question is whether the predicted gain in house value exceeds that cost.
You already have uni-reg in the browser, and the empty code cell is for your calculation. Start from the estimated slope, about six point five nine nine dollars of predicted price per additional unit of lot size. Multiply that marginal change by one thousand units to obtain the predicted gross benefit, then subtract five thousand dollars to obtain the predicted net benefit. A positive net value supports expanding under this simple calculation; a negative value does not.
The note points out the deliberate trap. The current lot size of three thousand is a red herring. In a linear model, the fitted slope is constant, so the difference between the prediction at four thousand and the prediction at three thousand is beta-one-hat times one thousand. The intercept and starting value cancel. This constant marginal effect is convenient, but it is a restriction of the straight-line specification, not a universal economic truth. When the course allows curved relationships in lecture seven, the effect of an additional thousand units can depend on where you start. Try extracting the lotsize coefficient by name and calculating the net benefit, then compare your result with the Solution tab.
Your current lot size is 3000. You are thinking of expanding your lot by 1000 (with everything else fixed), which would cost you 5,000 USD. Should you do it? Use R to figure it out.
One thing to notice
The answer does not depend on the 3000. In a linear model the effect of one more unit of x is \widehat{\beta}_1 wherever you start from, so the current lot size is a red herring. That stops being true the moment we allow the relationship to bend, which is lecture 07.
Here is the calculation. The first executable line, uni-reg dollar-sign coefficients, prints the complete coefficient vector so you can confirm the available names and values. It shows the intercept and the lotsize slope. The commented class line does not run because it begins with a hash mark.
The assessment line again extracts coefficients, then uses square brackets with the quoted name lotsize to select only that coefficient. Named selection is safer and clearer than relying on its numerical position. Multiplying by one thousand converts the slope from dollars per one unit of lot size into the predicted dollar gain from one thousand additional units. Subtracting five thousand accounts for the expansion cost.
Run it. Six point five nine eight seven six eight times one thousand is about six thousand five hundred ninety-eight dollars and seventy-seven cents. After subtracting five thousand, the predicted net benefit is about one thousand five hundred ninety-eight dollars and seventy-seven cents. It is positive, so under the fitted linear relationship and the stated cost, the mechanical answer is yes, expand. The starting lot size never enters because the constant slope makes the predicted change equal to beta-one-hat times the change in lotsize.
Now the honest caveat. This is an association estimated from a regression with one explanatory variable and a great deal left in u. The phrase “with everything else fixed” gives the decision a causal interpretation, but the regression warrants that interpretation only if the required identification assumptions hold. I would not stake my own five thousand dollars on this estimate without investigating omitted factors, uncertainty, and whether the expansion truly changes sale value. Still, the mechanics are exactly right: coefficient times change gives the model’s predicted gross change, and comparing it with cost gives the predicted net benefit.
The last topic today is R-squared, a goodness-of-fit measure. Start with the comparison emphasized in red. R-squared does not evaluate prediction in isolation. It asks how much better the fitted regression predicts the dependent variable than a benchmark that ignores the explanatory variables and uses the sample average of y for every observation.
For our application, the benchmark predicts the same value, y-bar, for every house, regardless of lot size. That may sound lazy, but among all possible constant predictions, the sample mean minimizes the sum of squared prediction errors. The regression using lot size is allowed to move its prediction along a line instead. R-squared measures the proportional reduction in squared prediction error achieved by that line relative to the mean-only benchmark. Equivalently, it describes the share of the sample variation in price accounted for by fitted-price variation.
This is a statement about in-sample fit. It does not automatically tell you how well the model predicts new data, whether the straight-line specification is sensible, or whether the lotsize coefficient is causal. A higher R-squared means the fitted values track the observed y values better according to this squared-error comparison. Everything in the next tabs builds that statement from the decomposition of one observation to the sums of squares and finally the formula.
R^2 is a measure of how good your model is in predicting the dependent variable (explaining variations in the dependent variable) compared to just using the average of the dependent variable as the predictor.
Before the R-squared formula, look at the exact decomposition on which it rests. By the definition of a residual, u-i-hat equals y-i minus y-i-hat. Rearranging gives the first line: observed y-i equals fitted y-i-hat plus residual u-i-hat. The slide also defines the fitted value as beta-zero-hat plus beta-one-hat times x-i. Every symbol refers to observation i except the two coefficient estimates, which define the common fitted line.
The next line subtracts the sample average y-bar from both sides. Starting from y-i equals y-i-hat plus u-i-hat, subtract y-bar to get y-i minus y-bar equals y-i-hat minus y-bar plus u-i-hat. Nothing statistical has been assumed in that step; it is algebra.
Now connect the equation to the three bullets. Y-i minus y-bar is the actual, or total, deviation from the mean. It tells you how far observation i’s actual outcome lies above or below the common mean prediction. Y-i-hat minus y-bar is the explained deviation: how far the regression moves its prediction away from the mean when it uses x-i. U-i-hat is the residual, the remaining gap between actual and fitted y. These are signed deviations, so direction matters: a negative residual means actual y is below fitted y.
With an OLS intercept, the residuals sum to zero and the average fitted value equals y-bar. Those properties let the sample-wide squared total deviation separate cleanly into explained and residual sums of squares. R-squared will compare those totals. Flip to Visualization to see the same identity for one actual house.
You can decompose observed value of y into two parts: fitted value and residual
y_i=\widehat{y}_i +\widehat{u}_i, \;\;\text{where}\;\; \widehat{y}_i = \widehat{\beta}_0+\widehat{\beta}_1 x_i
now, subtracting \bar{y} (sample average of y),
y_i-\bar{y}=\widehat{y}_i-\bar{y}+\widehat{u}_i
Here is the decomposition drawn for one real house. Lot size is on the horizontal axis, and house price in dollars is on the vertical axis. The background dots are the HousePrices observations. The blue regression line gives fitted price at each lot size, and the red horizontal line is y-bar, the sample mean price, about sixty-eight thousand one hundred twenty-one dollars and sixty cents.
The code first fits price tilde lotsize with l-m and saves it as fit-all. Mean of HousePrices dollar-sign price calculates y-bar. The next line chooses one house for which both decomposition pieces are visible. Abs of fitted minus y-bar measures the explained component’s length for every observation, and abs of resid measures the residual length. P-min takes the smaller of those two lengths observation by observation. Which-max selects the row whose smaller component is largest, ensuring neither arrow is merely a stub. The following lines extract that house’s lot size, actual price, and fitted price.
The selected observation is house three hundred sixty-five. Its lot size is fifteen thousand six hundred, its actual price is eighty-four thousand nine hundred dollars, and its fitted price is about one hundred thirty-seven thousand seventy-seven dollars. G-h-p supplies the original scatterplot. Geom-smooth maps price to y and lotsize to x, uses method l-m for the straight OLS line, suppresses the standard-error band with s-e false, and sets line width to zero point eight. Geom-h-line places the red mean line at y-bar.
The first annotate segment draws the green explained deviation at x-i. Its x and x-end are identical, making it vertical; y begins at y-bar and y-end is y-hat-i. Dark green and line width zero point eight style it. Arrow with length zero point zero one five in normalized parent coordinates and ends equal to both puts arrowheads at both endpoints. The two green annotate curves are curved leaders from the interval’s endpoints toward its label. Their chosen x and y endpoints, opposite curvatures, color, and width keep the label readable without obscuring the interval.
The purple annotate segment runs vertically from y-hat-i to actual y-i, so it is the residual. Here the residual is negative, about minus fifty-two thousand one hundred seventy-seven dollars, which means the purple arrow goes downward from the fitted line to the black observed point. The next annotate call redraws that point at size three point two so the selected house stands out.
The text annotations label the green interval as y-hat-i minus y-bar and the purple interval as u-i-hat. Parse equal to true renders those strings as mathematical notation; color matches each arrow, size controls readability, and h-just positions the text. The final text annotation labels y-bar in red near a clear part of the horizontal line.
Now read the bullets as signed arithmetic. Green, y-hat-i minus y-bar, plus purple, u-i-hat, equals y-i minus y-bar, the total deviation. For this house, the fitted value is far above the mean, but the negative residual pulls actual price back down. The model explains movement from the mean to the fitted line; the residual is what it misses. The next tab squares and sums each kind of deviation across all houses.
The three pieces, drawn for a single observation. The red line is \bar{y}; the blue line is the fitted model.
This slide turns the three observation-level pieces into three sample-wide totals. Take the definitions slowly because the acronyms are less reliable than the formulas.
Total sum of squares, S-S-T, is the sum from i equals one to n of y-i minus y-bar squared. Each term measures one observation’s squared distance from the sample mean. S-S-T therefore records the total sample variation in the dependent variable around its mean. It depends only on the observed y values, so every model fitted to the same sample is compared with the same total.
Explained sum of squares, called S-S-E on this slide, is the sum of y-i-hat minus y-bar squared. These are the squared green components from the previous figure. It measures how much the model’s fitted values vary around the mean, or how much of the total variation the regression accounts for.
Residual sum of squares, called S-S-R here, is the sum of u-i-hat squared. These are the squared purple components, the distances between actual and fitted y. You have already met this exact quantity: OLS chooses the coefficients that minimize it. Smaller S-S-R means the fitted line lies closer to the observations under squared-error loss.
With an intercept, the OLS orthogonality conditions make the cross-product between explained deviations and residuals equal zero. Therefore S-S-T equals S-S-E plus S-S-R. Squaring also makes all three totals nonnegative. One warning: many textbooks use S-S-R for “regression,” meaning explained variation, and S-S-E for “error,” meaning residual variation, exactly reversing the final two labels used here. Always read the formula and full name rather than trusting the acronym. The next tab uses the identity among these three sums to define R-squared.
total sum of squares (SST)
SST\equiv \sum_{i=1}^{n}(y_i-\bar{y})^2
explained sum of squares (SSE) SSE\equiv \sum_{i=1}^{n}(\widehat{y}_i-\bar{y})^2
residual sum of squares (SSR) SSR\equiv \sum_{i=1}^{n}\widehat{u}_i^2
The callout gives the definition: R-squared equals one minus residual sum of squares divided by total sum of squares. S-S-R is the squared error left after fitting the regression. S-S-T is the squared error from predicting every outcome with y-bar. Their ratio is the fraction of benchmark variation that remains unexplained, so subtracting it from one gives the fraction removed by the regression.
The three aligned lines show the equivalent explained-share formula. First, total sum of squares equals explained sum of squares plus residual sum of squares. Subtract S-S-R from both sides to get S-S-E equals S-S-T minus S-S-R. Divide every term by S-S-T. Then S-S-E over S-S-T equals one minus S-S-R over S-S-T, which is R-squared. So you can read R-squared either as the proportional reduction in squared error relative to the mean-only model or as the explained share of total sample variation.
The stated zero-to-one bound relies on the model including an intercept and on a nonconstant dependent variable so S-S-T is positive. OLS cannot have a larger residual sum of squares than the intercept-only benchmark because that benchmark is one of the coefficient choices available to it, so R-squared is at least zero. Since residual sum of squares is nonnegative, R-squared cannot exceed one. Without an intercept, these identities and the usual bound need not hold.
The next block contrasts two models. In y equals beta-zero plus u, the prediction is only the mean. Adding beta-one x lets predictions vary with x. R-squared measures the improvement in fitting y that this added information achieves. It is about the dependent variable, not directly about the quality or causal validity of beta-one.
Finally, the code calls r-two with uni-reg and the quoted type r-two. Uni-reg supplies the fitted model, and the second argument requests ordinary R-squared rather than an adjusted version. Run it and the value is about zero point two eight seven zero seven seven. Thus lot size alone accounts for roughly twenty-eight point seven percent, about twenty-nine percent, of the sample variation in house prices relative to the mean benchmark. Roughly seventy-one point three percent remains in the residual variation, reflecting bedrooms, location, condition, sale year, nonlinearities, noise, and other contents of u. The Caveat tab explains why even this correctly calculated fit measure cannot validate the coefficient we care about.
Definition
R^2 = 1 - SSR/SST
Where did it come from?
\begin{align*} & SST = SSE + SSR \\ \Rightarrow & SSE = SST - SSR \\ \Rightarrow & SSE/SST = 1 - SSR/SST = R^2\\ \end{align*}The value of R^2 always lies between 0 and 1 as long as an intercept is included in the econometric model.
What does it measure?
R^2 is a measure of how much improvement in predicting the dependent variable you’ve made by including independent variable(s) (y=\beta_0+\beta_1 x+u) compared to when simply using the mean of dependent variable as the predictor (y=\beta_0+u).
For our model
That is how much of the variation in house price lot size alone accounts for. The rest is everything in u — bedrooms, location, condition, the year it sold.
This warning is the most important lesson in the R-squared section. Read the “Important” block first. R-squared tells you how closely fitted outcomes track observed outcomes relative to the mean benchmark. It tells you nothing by itself about whether beta-one-hat is close to the causal, ceteris paribus effect beta-one. A model can predict y well for the wrong reasons, and a model with modest R-squared can still estimate a causal effect well if its identification strategy and assumptions are sound.
The phrase ceteris paribus means holding other determinants of y fixed. In this one-variable regression, those determinants are placed in u. If lot size is correlated with omitted features such as location, house size, or neighbourhood quality, the slope may absorb their influence. That can bias beta-one-hat even if the fitted line explains a substantial share of price variation. Conversely, inherently noisy outcomes can produce a low R-squared even when x is exogenous and the slope is centered on the causal effect.
The second bullet makes the economist’s distinction. We predict outcomes, such as an individual house price, but we estimate parameters, such as beta-one. Many applied economic questions care primarily about the parameter because it describes a policy effect, behavioral response, or ceteris paribus relationship. Prediction can still matter in economics, but it is a different objective and should not be confused with identification.
Now read the “Problem” block. We observe the dependent variable, otherwise we could not run the regression, so we can compare actual y with fitted y and calculate residuals, sums of squares, and R-squared. But the true population beta-one is not observed. In a real application, nobody hands us the causal coefficient so we can calculate beta-one-hat minus beta-one directly. That means the sample cannot provide a direct test that our point estimate is close to the truth.
We therefore evaluate estimation indirectly, through the research design and assumptions that connect the sample regression to the population parameter. We ask whether the model is correctly specified, whether x has variation, how the sample was drawn, and especially whether the error has mean zero conditional on x. Later we quantify sampling uncertainty with standard errors and confidence intervals, but those calculations also rest on assumptions. So do not select a causal model because it has the largest R-squared. The next topic turns from fit, which we can observe, to the assumptions needed for credible estimation of beta-one, which is what we actually care about.
Important
Problem