This article was published as a part of theÂ Data Science Blogathon

Optimization provides a way to minimize the loss function. Optimization aims to reduce training errors, and Deep Learning Optimization is concerned with finding a suitable model. Another goal of optimization in deep learning is to minimize generalization errors. In this article, we will discuss linear models.

The Linear Model is one of the simplest models in machine learning, but linear models are the building blocks for deep neural networks. There are two main classes in supervised learning problems, regression and classification. In reversal, the target value is the actual value. For example, if we have job description data and want to predict how much salary will be given for this job, this is a regression task because the salary is a real value. Or for example, if we have drug stock data and have demand (x_{1}) and usage (x_{2}) variables and the variable to be predicted (dependent) is supply (y). This is also a problem that regression can solve because the inventory variable is a real value.

Whereas if the number of targets is limited, it is a classification task. For example, if we want to recognize an object in an image, say, we want to know if there is a motorcycle, a car, a fence, or perhaps a building or a bicycle in the picture. This is an object recognition task. Since the number of objects is limited, classification can solve this. Or for example, we are analyzing an article and want to know the topic, whether it is about Machine Learning or Computer Vision, or Deep Learning, then that is also a classification task because, once again, the number of target values â€‹â€‹is limited.

Vector Notation:

For a sample X:

This linear model is the point product of the weight vector and the feature vector X., And if we want to apply the model to the entire training set, then we have a Matrix X, which has L rows and d columns. This multiplication produces a vector of size L, and each component predicts the linear model. Then how to measure the model error or know the quality of the train or test set?

Mean Squared Error:

The Mean Squared Error is one of the popular choices in regression for the loss function. A particular example, for example, x_{i}. Calculating the model prediction for this example is the product of w and x_{i}, then subtracted from the target value (y_{i}). It then calculates the deviation of the target value from the predicted value, then takes its square and averages the square of the deviation across all training sets. It measures how well our model fits the data. The smaller the mean square error, the better the model provides the data. The mean square error is written in vector form.

Fitting Model:

The essence of machine learning is optimizing losses to find the best model. So the last function we have, measures how well the model fits the data by minimizing it. So, the goal is to find the set of parameters w that gives this minor mean squared error for the train.

Exact Solution:

If we do the derivative and solve the equation, then we will have an analytical solution to the optimization problem. But it involves inverting and matrices, and highly complex operation and very difficult to find the inverse matrix if it has more features (say more than 100). The linear model for regression is straightforward but valuable (very) for deep neural networks.

How to adapt linear methods to classification problems? Logistics Regression. Logistics Regression is a regression model that can be used for classification problems from the simplest classification, namely binary classification, which has only two values â€‹â€‹on the target (say minus one and one), negative and positive, yes/no, success/failure even with multiple classification problems classes such as strongly agree, agree, strongly disagree and disagree.

Let’s start by solving a 2D classification problem with synthetic data using Python programming to better understand how the Algorithm works.

import numpy as np

from sklearn.datasets import make_moons

from matplotlib import pyplot

from pandas import DataFrame

# generate 2d classification dataset

X, y = make_moons(n_samples=1000, noise=0.5)

# scatter plot, dots colored by class value

df = DataFrame(dict(x=X[:,0], y=X[:,1], label=y))

colors = {0:’red’, 1:’blue’}

fig, ax = pyplot.subplots()

grouped = df.groupby(‘label’)

for key, group in grouped:

group.plot(ax=ax, kind=’scatter’, x=’x’, y=’y’, label=key, color=colors[key])

pyplot.show()

The output of the above code:

It cannot be separated linearly. That’s the state of the data above that we can see. Then we have to add features or use a non-linear model. Because the decision line between the two classes is circular, we can add a quadratic part to make the problem in the data above separated linearly, as shown below.

The next step is to add features. The expansion allows linear models to make non-linear separations.

def expand(X): X_expanded = np.zeros((X.shape[0], 6)) X_expanded[:, 0] = X[:, 0] X_expanded[:, 1] = X[:, 1] X_expanded[:, 2] = X[:, 0] ** 2 X_expanded[:, 3] = X[:, 1] ** 2 X_expanded[:, 4] = X[:, 0] * X[:, 1] X_expanded[:, 5] = 1 return X_expanded

The above function works like: For each sample (row in matrix), compute an expanded row: [feature0, feature1, feature0^2, feature1^2, feature0*feature1, 1]

Next, let’s look at the Logistics Regression section. When classifying the thing, we will obtain the probability that the object belongs to class ‘1’. Linear models and logistic functions as below are used to predict chances.

Logistic Function:

def probability(X, w): return 1 / (1 + np.exp(-np.dot(X, w))) dummy_weights = np.linspace(-1, 1, 6)

predict_prob = probability(X_expanded[:1, :], dummy_weights)[0]

The probability is approximatelyÂ 0.8678884252629746

Next is to calculate the loss in Logistic Regression with cross-entropy. In logistic regression the optimal parameters w are found by cross-entropy minimization.

Loss for one sample:

Loss for many sample:

To calculate the loss, we use the function:

def compute_loss(X, y, w): l = X.shape[0] p = probability(X, w) return -(1.0/l) * np.sum(y * np.log(p) + (1-y) * np.log(1-p)) cross_ent = compute_loss(X_expanded, y, dummy_weights)

The cross-entropy value is aboutÂ 1.0523716363491382

In the Regression problem, we discussed a squared error, a loss function, and an analytical solution, but it isn’t easy to calculate.Â In Logistic Regression, the optimal parameters are found by cross-entropy minimization.

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Become a full stack data scientist
##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

##

Understanding Cost Function
Understanding Gradient Descent
Math Behind Gradient Descent
Assumptions of Linear Regression
Implement Linear Regression from Scratch
Train Linear Regression in Python
Implementing Linear Regression in R
Diagnosing Residual Plots in Linear Regression Models
Generalized Linear Models
Introduction to Logistic Regression
Odds Ratio
Implementing Logistic Regression from Scratch
Introduction to Scikit-learn in Python
Train Logistic Regression in python
Multiclass using Logistic Regression
How to use Multinomial and Ordinal Logistic Regression in R ?
Challenges with Linear Regression
Introduction to Regularisation
Implementing Regularisation
Ridge Regression
Lasso Regression

Introduction to Stacking
Implementing Stacking
Variants of Stacking
Implementing Variants of Stacking
Introduction to Blending
Bootstrap Sampling
Introduction to Random Sampling
Hyper-parameters of Random Forest
Implementing Random Forest
Out-of-Bag (OOB) Score in the Random Forest
IPL Team Win Prediction Project Using Machine Learning
Introduction to Boosting
Gradient Boosting Algorithm
Math behind GBM
Implementing GBM in python
Regularized Greedy Forests
Extreme Gradient Boosting
Implementing XGBM in python
Tuning Hyperparameters of XGBoost in Python
Implement XGBM in R/H2O
Adaptive Boosting
Implementing Adaptive Boosing
LightGBM
Implementing LightGBM in Python
Catboost
Implementing Catboost in Python

Introduction to Clustering
Applications of Clustering
Evaluation Metrics for Clustering
Understanding K-Means
Implementation of K-Means in Python
Implementation of K-Means in R
Choosing Right Value for K
Profiling Market Segments using K-Means Clustering
Hierarchical Clustering
Implementation of Hierarchial Clustering
DBSCAN
Defining Similarity between clusters
Build Better and Accurate Clusters with Gaussian Mixture Models

Introduction to Machine Learning Interpretability
Framework and Interpretable Models
model Agnostic Methods for Interpretability
Implementing Interpretable Model
Understanding SHAP
Out-of-Core ML
Introduction to Interpretable Machine Learning Models
Model Agnostic Methods for Interpretability
Game Theory & Shapley Values

Deploying Machine Learning Model using Streamlit
Deploying ML Models in Docker
Deploy Using Streamlit
Deploy on Heroku
Deploy Using Netlify
Introduction to Amazon Sagemaker
Setting up Amazon SageMaker
Using SageMaker Endpoint to Generate Inference
Deploy on Microsoft Azure Cloud
Introduction to Flask for Model
Deploying ML model using Flask