Regression and Correlation

Regression and Correlation

Simple and multiple linear regression, correlation coefficients, least squares method, and residual analysis.

Engineers frequently need to predict the value of one variable based on the value of another. For example, predicting the compressive strength of concrete (

y

) based on its curing time (

x

), or estimating a river's peak flow rate (

y

) based on rainfall intensity (

x

While correlation measures the strength and direction of a linear relationship, regression provides the mathematical equation to make actual predictions.

Correlation

Quantifying the strength of the linear relationship.

Scatter Plots

A graphical tool used to plot two quantitative variables on a Cartesian coordinate system. Scatter plots provide the first visual indication of whether a linear relationship, a non-linear relationship, or no relationship exists between

X

and

Y

. They are essential before proceeding with correlation or regression calculations.

Pearson Correlation Coefficient ( $r$ )

A unitless measure that ranges from -1 to +1, describing how closely data points fall along a straight line.

r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}}

$r \approx 1$ : Strong positive linear relationship (as $x$ increases, $y$ predictably increases).
$r \approx -1$ : Strong negative linear relationship (as $x$ increases, $y$ predictably decreases).
$r \approx 0$ : Weak or no linear relationship (the data appears as a random scatter). Note: A value of $r=0$ does not mean there is no relationship at all; there could be a strong non-linear (e.g., quadratic) relationship.

Correlation does not imply causation

Just because two variables are highly correlated (e.g., asphalt sales and ice cream sales) does not mean one causes the other. Both might be driven by a lurking third variable (summer weather).

Simple Linear Regression

Modeling the relationship with a straight line.

The Regression Model

We hypothesize that the true relationship is a straight line, plus some random, unobservable error (

\epsilon

y_i = \beta_0 + \beta_1 x_i + \epsilon_i

$y_i$ : The dependent (response) variable we want to predict.
$x_i$ : The independent (predictor or explanatory) variable.
$\beta_0$ : The true $y$ -intercept (the value of $y$ when $x=0$ ).
$\beta_1$ : The true slope (the change in $y$ for a one-unit change in $x$ ).
$\epsilon_i$ : The random error term. We assume these errors are normally distributed with a mean of 0 and a constant variance ( $\sigma^2$ ).

The Method of Least Squares

Finding the "line of best fit."

Because we only have a sample, we estimate the true parameters (

\beta_0, \beta_1

) with sample statistics (

b_0, b_1

Estimated Regression Equation

\hat{y} = b_0 + b_1 x

The "hat" on

\hat{y}

indicates it is an estimated or predicted value, not an actual observed value.

Least Squares Principle

The line of best fit is the one that minimizes the Sum of Squared Errors (SSE). The error (or residual) is the vertical distance between an observed data point (

y_i

) and the predicted value on the line (

\hat{y}_i

SSE = \sum (y_i - \hat{y}_i)^2 = \sum e_i^2

Using calculus to minimize SSE yields the formulas for the slope (

b_1

) and intercept (

b_0

b_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2}

b_0 = \bar{y} - b_1\bar{x}

Assessing the Model and Residual Analysis

How good is our prediction?

Before relying on a regression equation for engineering decisions, we must verify that the model is appropriate.

Coefficient of Determination ( $R^2$ )

The proportion of the total variation in the dependent variable (

y

) that is explained by the regression model (the independent variable

x

R^2 = \frac{SSR}{SST} = 1 - \frac{SSE}{SST}

$0 \le R^2 \le 1$ .
An $R^2 = 0.85$ means that 85% of the variation in concrete strength can be explained by the variation in curing time.
For simple linear regression (one predictor), $R^2 = r^2$ .

Standard Error of the Estimate ( $s_e$ )

A measure of the typical distance that observed data points fall from the regression line. It estimates the standard deviation of the error term (

\sigma

s_e = \sqrt{\frac{SSE}{n-2}}

Residual Analysis (Validating Assumptions)

A critical step. We plot the residuals (

e_i = y_i - \hat{y}_i

) against the predicted values (

\hat{y}_i

) or the predictor (

x_i

). For the linear model to be valid, the residual plot should show a random, structureless horizontal band around zero.

Non-linearity: If the residuals show a curved pattern (like a U-shape), a straight line is not appropriate; a polynomial regression is needed.
Heteroscedasticity: If the spread of the residuals increases (a fan shape), the variance of the errors is not constant. This violates a key assumption and requires data transformation (e.g., taking the log of $y$ ).

Multiple Linear Regression

Using more than one predictor variable.

Engineers rarely predict an outcome based on a single variable. Concrete strength depends on water-cement ratio, curing time, temperature, and aggregate type.

Multiple Regression Model

\hat{y} = b_0 + b_1 x_1 + b_2 x_2 + \dots + b_k x_k

Each $b_i$ represents the change in the estimated $y$ for a one-unit change in $x_i$ , holding all other predictor variables constant.
Adjusted $R^2$ : Unlike regular $R^2$ (which always increases when you add a variable), Adjusted $R^2$ penalizes adding variables that do not significantly improve the model, preventing "overfitting."

Hypothesis Testing in Regression (ANOVA Approach)

Testing if the model is statistically significant.

Hypothesis Test for the Slope ( $\beta_1$ )

In simple linear regression, testing whether a linear relationship exists is equivalent to testing the null hypothesis

H_0: \beta_1 = 0

(the true slope is zero). We use a t-test with

n-2

degrees of freedom. If we reject

H_0

, there is sufficient evidence that

X

provides information in predicting

Y

F-Test for Overall Significance

Tests whether the regression model as a whole is better than simply predicting the mean of

y

(

\bar{y}

$H_0: \beta_1 = \beta_2 = \dots = \beta_k = 0$ (The model is useless).
If the resulting P-value is very small, at least one predictor is significantly related to $y$ .

t-Tests for Individual Coefficients

If the overall model is significant, we run t-tests on each individual slope (

b_i

) to determine which specific variables are actually contributing.

$H_0: \beta_i = 0$ (This specific predictor is useless, assuming all others are already in the model).

Key Takeaways

Correlation ( $r$ ): Measures the strength of a linear relationship (-1 to 1). Does not imply causation.
Least Squares: The mathematical method used to find the line that minimizes the sum of squared errors (SSE).
$R^2$ : The percentage of variation in $y$ explained by the model.
Residual Analysis: Crucial for validating the assumptions of linearity and constant variance. Residual plots should look like random noise.
Multiple Regression: Uses multiple predictors. $b_i$ coefficients represent the effect of $x_i$ holding all other variables constant.
ANOVA F-Test: Determines if the overall regression model is statistically significant.

PreviousTests of Hypotheses - Examples & Applications

Quiz Me

NextRegression and Correlation - Examples & Applications

Correlation

Scatter Plots

Pearson Correlation Coefficient (rrr)

Correlation does not imply causation

Simple Linear Regression

The Regression Model

The Method of Least Squares

Estimated Regression Equation

Least Squares Principle

Assessing the Model and Residual Analysis

Coefficient of Determination (R2R^2R2)

Standard Error of the Estimate (ses_ese​)