Understanding Regression Coefficients: Standardized vs Unstandardized

CHIRAG GOYAL 11 Jan, 2024 • 8 min read

Regression coefficients: standardized versus unstandardized. Two sides of the same coin, each with its own unique identity. Like a pair of mismatched socks, they bring confusion and clarity to linear regression. This article unravels the enigma behind these coefficients and explores their distinctive characteristics. Get ready to dive into standardized vs unstandardized regression coefficients as we decipher their roles, significance, and implications. You’ll better understand these key players in statistical modeling by the end.

Learning Objectives

  • Understand what standardized and unstandardized regression coefficients are.
  • Find out the use cases of standardized regression coefficients.
  • Learn to calculate regression coefficients.

This article is a part of Data Science Blogathon.

Quiz Time

Challenge yourself with questions about Standardized and Unstandardized Regression Coefficients and their interpretation in regression analysis.

What are Regression Coefficients?

Regression coefficients are numerical values that represent the strength and direction of the relationship between variables in a regression model.

Regression coefficients, also known as regression parameters, are the estimated values depicting the relationship between independent variables and the dependent variable in a regression model. They quantitatively capture the impact of each independent variable, indicating both direction and extent. In linear regression, these coefficients signify the slope of the line, providing insight into the rate of change in the dependent variable per unit change in the independent variable. For different types of regression models, such as multiple regression, coefficients convey the alteration in the dependent variable for a one-unit shift in the corresponding independent variable, while keeping other variables unaltered. These coefficients play a crucial role in understanding and interpreting the significance of variables within the regression framework.

Formula for Regression Coefficient

The formula for calculating regression coefficients in simple linear regression is:

β = (Σ((X – X̄)(Y – Ȳ))) / Σ((X – X̄)²)

Where:

  • β is the regression coefficient (slope)
  • X is the independent variable (input)
  • Y is the dependent variable (output)
  • X̄ is the mean of the independent variable
  • Ȳ is the mean of the dependent variable
  • Σ represents the sum of

The regression coefficients formula is essential in calculating the slope of the line that optimally represents the relationship between the independent and dependent variables. It quantifies the change in the dependent variable with each unit change in the independent variable. This coefficient, whether positive or negative, naturally indicates both the direction and magnitude of the relationship. Understanding this formula is fundamental to grasping the dynamics of linear relationships in statistical analysis

Unstandardized Regression Coefficients

Unstandardized regression coefficients, also known as raw coefficients, represent the change in the dependent variable associated with a one-unit change in the corresponding independent variable, while holding other variables constant. They are expressed in the original units of the variables and provide a direct measure of the effect size and direction of the relationship between variables in a regression model.

Unstandardized Regression coefficients are those that the linear regression model produces after its training using the independent variables, which are measured in their original scales, i.e., in the same units in which we are taken the dataset from the source to train the model.

An unstandardized coefficient should not be used to drop or rank predictors (aka independent variables) as it does not eliminate the unit of measurement.

For Example, let’s take a hypothetical multiple regression example where we want to predict the income(in rupees) of a person based on their age (in years), height(in cm), and weight(in kg). So, here inputs for our regression analysis are age, height, and weight, and the output(response variable) is income. Then,

Income(rupees)=a0+a1*age(years)+a2*height(cm)+a3*weight(kg)+e                (eqn-1)

How to Interpret Unstandardized Regression Coefficients?

These regression coefficients naturally interpret the effect of each independent variable on the outcome (response/output). Their interpretation is straightforward and intuitive. All other variables held constant; a 1 unit change in Xi (predictors) implies there is an average change of ai units in Y (outcome). Understanding these regression coefficients is crucial for gaining insights into how individual predictors contribute to the overall change in the outcome variable.

In the above example of multiple linear regression, if a1=0.3, a2=0.2, and a3=0.4 (and assume all are statistically significant), then we interpret these coefficients as follows:

Getting 1 year older is associated with an increase of 0.3 in income, assuming other variables are constant (which means there is no change in height and weight). Similarly, we can interpret the coefficient for other independent variables as well.

It represents the amount by which dependent variable changes if we change independent variable by one unit keeping other independent variables constant.

Limitations of Unstandardized Regression Coefficients

Unstandardized coefficients are great for interpreting the relationship between an independent variable X and an outcome Y. However, they are not useful for comparing the effect of an independent variable with another one in the model.

For Example, which variable has a larger impact on Income? Age, Height, or weight?
We can try to answer this question by looking at equation-1 and again assume that a1=0.3, a2=0.2, and a3=0.4, we conclude that :

“An increase of 20 cm in height has the same effect on the weight increases 10 times” Still, this does not answer the question of which variable affects Income more.

Specifically, the statement that “the effect of the increase of weight by 10 times = the effect of the increase in the height by 20 cm” is meaningless without specifying how hard it is to increase height by 20 cm, specifically for someone who’s not familiar with this scale.

So, at last, we conclude that a direct comparison of the regression coefficients for any of the pair of independent variables is not making sense or is not useful as these independent variables are on different scales (age in years, weight in kg, and height in cm).

It turns out that the effects of these variables can be compared by using the standardized version of their coefficients. And that’s what we’re going to discuss next.

Standardized Regression Coefficients

Standardized regression coefficients, also known as beta coefficients, represent the change in the dependent variable in terms of standard deviations for a one-standard-deviation change in the corresponding standardized independent variable. They allow for direct comparison of the relative importance of different variables and help assess the impact of predictors while accounting for differences in scale and units.

The concept of standardization or standard regression coefficients is used in data science when independent variables or predictor variables for a particular model are expressed in different units. For Example, let’s say we have three independent features of a woman: height, age, and weight. Her height is in inches, her weight in kilograms, and her age in years. If we want to rank these predictors based on the unstandardized coefficient (which directly comes when we train a regression model), it would not be a fair comparison since the units for all the predictors are different.

The standardised regression coefficients are obtained by training(or running) a linear regression model on the standardized form of the variables.

The standardized variables are calculated by subtracting the mean and dividing by the standard deviation for each observation, i.e., calculating the Z-score. It would make mean 0 and standard deviation 1. For this, they also need to follow the normal distribution. Then, they don’t represent their original scales since they have no unit.

For each observation “j” of the variable X, we calculate the z-score using the formula:

z-score formula

Which variables do we have to standardize for finding the standardized regression coefficients, i.e., both predictor and response or either one of them?

Yes, we standardize both the dependent(response) and the independent(predictor) variables before running the linear regression model (as this is the widely accepted practice when we want to find the standardized form of the variables).

How to Interpret the Standardized Regression Coefficients?

The interpretation of standardized regression coefficients is non-intuitive compared to their unstandardized versions: For example, a 1 standard deviation unit increase in X will result in β standard deviation units increase in y.

A change of 1 standard deviation in X is associated with a change of β standard deviations of Y.

If there is a categorical variable in place of a numerical variable in our analysis, then its standardized coefficient cannot be interpreted as it does not make sense to change X by 1 standard deviation. In general, this is not a problem for our model since these coefficients are not meant to be interpreted individually but to be compared to one another in order to get a sense of the importance of each variable in the linear regression model.

The standardized coefficient is measured in units of standard deviation. A beta value of 2.25 indicates that of one standard deviation increase in the independent variable results in a 2.25 standard deviations increase in the dependent variable.

What Is the Real Use of Standardized Coefficients?

They are mainly used to rank predictors (or independent or explanatory variables) as they eliminate the units of measurement of independent and dependent variables). We can rank independent variables with an absolute value of standardized coefficients. The most important variable will have the maximum absolute value of the standardized regression coefficient.

For example:

Y = β0 + β1 X1 + β2 X2 + ε

If the standardized coefficients β1 = 0.5 and β2 = 1, we can conclude that:

X2 is twice as important as X1 in predicting Y, assuming that both X1 and X2 follow roughly the same distribution and their standard deviations are not that different.

Limitations of Standardized Regression Coefficients

The standardized Regression coefficients are misleading if the variables in the model have different standard deviations means all variables are having different distributions.

Take a look at the following linear regression equation:

Income($) = β0 + β1 Age(years) + β2 Experience(years) + ε

Because our independent variables, Age and Experience, are on the same scale (years) and if it is reasonable to assume that their standard deviations differ a lot, then in this case:

  • Their unstandardized coefficients should be used to compare their importance/influence in the model.
  • Standardized these variables would, in fact, cause them to be on a different scale (different standard deviations or follows different distribution)

Calculation of Standardized Coefficients

For Linear Regression

(Another approach as we see one approach in the above part of the article)

Multiplying the unstandardized coefficient by the ratio of the independent and dependent variable standard deviations gives standardized coefficient.

STANDARDIZED vs UNSTANDARDIZED for linear regression formula

For Logistic Regression

STANDARDIZED UNSTANDARDIZED logistic regression

 We calculate them using various software like spss, sas, R, and Python.

Standardized vs Unstandardized Regression Coefficients

Check out the difference between Standardized vs Unstandardized regression coefficients here:

Standardized Regression CoefficientsUnstandardized Regression Coefficients
InterpretationMeasures the change in the dependent variable in terms of standard deviations per unit change in the independent variable.Measures the change in the dependent variable per unit change in the independent variable.
ScaleDimensionless, with a mean of 0 and a standard deviation of 1.In the original scale of the dependent variable.
ComparabilityCan be directly compared across different independent variables.Cannot be directly compared across different independent variables due to differences in their scales.
ImportanceUseful when comparing the relative influence of different independent variables on the dependent variable.Useful when interpreting the magnitude and direction of the effect of an independent variable on the dependent variable.
ApplicationHelpful when the scales of independent variables differ significantly or when comparing variables with different units.Useful when the focus is on understanding the direct impact of an independent variable on the dependent variable.

Conclusion

This article covered some basic but necessary concepts that come in handy while working on real-life projects in Machine Learning and Artificial Intelligence. Towards the end of this article, we’ve looked into the Mathematics behind these concepts and also learned to calculate regression coefficients. Not that both standardized and unstandardized coefficients have their own separate use cases and you should choose the one that matches your data set and need.

Key Takeaways

  • Training a linear regression model using the independent variables, measured in the same units as the source or raw data set gives unstandardized coefficients.
  • You can find the standardized coefficients of regression by training a linear regression model on the standardized form of the variables.
  • Subtracting the mean and dividing the answer by the standard deviation for each observation gives standardized variables.

Frequently Asked Questions

Q1. Should I use standardized or unstandardized coefficients in regression?

A. Both of them have their use cases. So, it’s better to calculate both and interpret them accordingly.

Q2. How do I find the standardized coefficient of a variable?

A. Multiplying the unstandardized coefficient by the ratio of the independent and dependent variable standard deviations to derive standardized coefficient. 1 standard deviation change in feature results in coefficient times standard deviation change in the y variable.

Q3. What is the difference between B and β in simple linear regression?

A. B corresponds to the unstandardized coefficients, while β corresponds to the standardized coefficients.

CHIRAG GOYAL 11 Jan 2024

I am currently pursuing my Bachelor of Technology (B.Tech) in Computer Science and Engineering from the Indian Institute of Technology Jodhpur(IITJ). I am very enthusiastic about Machine learning, Deep Learning, and Artificial Intelligence. Feel free to connect with me on Linkedin.

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

Girma
Girma 18 Feb, 2022

I read your article and it is very nice, thanks for that. I have one question facing when I am doing my masters thesis. I used four independent variables & one dependent variables to test the significant effect of independent variables on dependent variable with ordinal Likert scales (measured 1 up to 5 rank questionnaries). However, SPSS analysis output shows one independent variable is redundant and deleted from it but I need this variable to test the hypothesis. How can correct it and is it a multicollinarity issues that I faced, how may I avoid the problem please? Besides, the result says accept the null hypothesis which states there is no effect of independent on dependent variables though in reality this is a general truth as it has a relationship between the two variables, would you please advise me why this result arises from the analysis? Lastly, my hypothesis is as follows : Major hypothesis; H0: There is no significant effect of strategic leadership on investment opportunities. H1: There is a significant effect of strategic leadership on investment opportunities. Sub hypothesis a) H0a: There is no significant effect of organizational creativity on investment opportunities. H1a: There is a significant effect of organizational creativity on investment opportunities. b) H0b: There is no significant effect of business development on investment opportunities. H1b: There is a significant effect of business development on investment opportunities. c) H0c: There is no significant effect of client/customer centricity on investment opportunities. H1c: There is a significant effect of client/customer centricity on investment opportunities. d) H0d: There is no significant effect of operational efficiency on investment opportunities. H1d: There is a significant effect of operational efficiency on investment opportunities. as my sample size is 70 & ordinal data, can I use parametric test or nonparmetric test as it has some normal distribution? How I test the major hypothesis above, can I use interval data instead of ordinal and use parametric test; like, Pearson's correlation or t-test? In general, which test is more appropriate for the above hypothesis please? Though it is a long questions and make you busy, hoping I get your nice expertise soon. Please email me. Thanks,

Raza
Raza 28 Jan, 2023

Thank you. Would you please elaborate on, whether can we report the standardized regression coefficients in terms of %-Percentage change? i.e., Percentage change in X on Y. E.g., Y = -0.20.X (interest rate) Here can we interpret that a 1% Decrease in X (interest rate) would lead to a 20% increase in Y?