What Is Gradient Descent in Machine Learning?

Key takeaways

Gradient descent is an algorithm you can use to train models in both neural networks and machine learning.

• The “gradient” in the gradient descent formula represents the slope of the cost function at a specific point. 

• Convergence in gradient descent is the point where the algorithm has minimized the cost function enough that parameter updates are almost negligible.

• You can strengthen your math skills to better understand and implement gradient descent. 

Find out what gradient descent is, why it’s vital for machine learning, its main types, and potential limitations. Afterward, if you’re ready to build foundational skills in machine learning, consider enrolling in DeepLearning.AI’s Mathematics for Machine Learning and Data Science Specialization. Along with machine learning methods, this program offers guidance on data preprocessing, statistical analysis, applied mathematics, A/B testing, and more. 

What is gradient descent?

Gradient descent is an algorithm you can use to train models in both neural networks and machine learning. It uses a cost function to optimize its parameters, showing the accuracy of a machine learning model under training at each parameter. Gradient descent existed as a mathematical concept before the emergence of machine learning. 

A gradient in vector calculus is similar to the slope but applies when you have three or more variables. It becomes the vector of partial derivatives for all independent variables and is denoted f for the maximum gradient increase or -∇f for the maximum gradient decrease of the function. 

Gradient descent requires knowledge of calculus for its implementation in machine learning. 

How does gradient descent work?

Gradient descent works with convex functions and finds the fewest and most accurate number of steps toward the lowest point of a curve, optimizing the path. Let’s go over a couple of terms that inform gradient descent before examining how they work:

• Parameters: The coefficients of the function that minimize the cost

• Cost function: Also called the “loss function” in machine learning, it is the difference between the actual and predicted value at the present position. A model stops learning once this function gets as close as possible to 0.0 

• Learning rate: Sometimes referred to as the step rate or alpha, it is the magnitude of the steps the function takes as it minimizes the cost 

The primary function of gradient descent is to find the parameters that best minimize the cost by having the coefficients make the cost equal to or as close to 0.0 as possible 

How is the “gradient” used in the gradient descent formula?

In the gradient descent formula, the “gradient” is the slope of the cost function at a specific point. It shows the direction and rate of the steepest descent, using it to move toward the minimum of the function.

Basics of performing gradient descent

The goal is to have the cost = 0.0 or the closest acceptable minimum. To calculate this, start by writing the cost function. 

1. Write the cost function as cost =f(x) with x as the coefficient. 

2. Use a starting coefficient of 0.0 or any small number. 

3. Take the derivative, or partial derivative if multiple variables are present, to find the gradient to know which direction to move in on the curve. 

4. Once you have a gradient (the derivative of the cost function) and know which way to move, use your learning rate to tell how much the value of the coefficient changes every calculation. 

5. Repeat until the cost is zero or as close as it can reach. 

Gradient descent involves knowledge of calculus, but its implementation is always the same series of steps. 

What is convergence in gradient descent​?

Convergence in gradient descent is the point where the algorithm has minimized the cost function enough that parameter updates are almost negligible. This is essentially when the model reaches the “best” solution, and more iterations are unlikely to improve findings. 

What is gradient descent in machine learning? Two types to explore

Machine learning uses two main types of gradient descent: 

• Batch gradient descent (BGD): Provides updating for the machine learning model after each training epoch by averaging the error of predictions and actual outcomes of the cost function at each iteration. 

• Stochastic gradient descent (SGD): Calculates the error for every sample in the data set, needing a prediction for every iteration of the training epoch, recalculating each coefficient for every instance. 

Comparing batch gradient descent and SGD

Batch gradient descent and stochastic gradient descent have unique advantages and disadvantages when calculating gradient descent in machine learning. Let’s take a look at each:

Batch gradient descent is a common approach to machine learning, but stochastic gradient descent performs better on larger data sets. 

Mini-batch gradient descent

If you need to use aspects of both batch gradient descent and SGD, consider using a method called mini-batch gradient descent (MBGD) that combines them. It still uses batches but breaks up a data set into small batches, each providing the updates from SGD as they perform gradient descent. It makes the learning of each batch quick while also remaining computationally fast. This method is standard in machine learning, the training of neural networks, and deep learning applications. 

Common gradient descent problems

While gradient descent is an efficient way to optimize machine learning algorithms, you will find some common problems the algorithm runs into that may leave you with models that aren’t fully optimized. With graphs that are not entirely convex parabolas, points other than the global minimum can make the cost function equal to 0.0. These two points are:

• Local minima: These give a slope of 0.0 and seem like global minimum points to the algorithm, but are just local minimum points before the cost function increases again before reaching the global minimum

• Saddle points: Give a slope of 0.0 at a series of points where the cost function stops steadily decreasing before it continues its descent to the global minimum 

In applying gradient descent to deep learning neural networks, two issues arise:

• Vanishing gradients: Occur during the backpropagation of neural networks, making the gradient too small, leading to an eventual coefficient of zero, resulting in the network stopping learning 

• Exploding gradients: Occur when a model becomes unstable due to a gradient that is too large, leading to coefficients that are no longer calculable because of a complex machine learning algorithm 

Read more: Neural Network Weights: A Comprehensive Guide

Check out our free resources on machine learning

Subscribe to Career Chat on LinkedIn to stay current with the latest trends in machine learning. Continue your learning journey with our other free digital resources:

• Explore certifications: 6 machine learning certificates + how to choose the right one for you

• Watch on YouTube: Machine Learning Classification | Python Diabetes Prediction Model

• Plan your career trajectory: Machine Learning Career Paths: Explore Roles & Specializations

Accelerate your career growth with a Coursera Plus subscription. When you enroll in either the monthly or annual option, you’ll get access to over 10,000 courses.

Build job-ready skills with Coursera Plus

Start 7-day free trial

• 
• 
• 
• 

Start 7-day free trial