Gradient descent and stochastic gradient descent (SGD) are fundamental optimization algorithms in machine learning, often replacing closed-form solutions like the normal equation for linear regression when handling large datasets. While linear regression can theoretically be solved using matrix inversion to find optimal slope and intercept parameters, this approach becomes computationally prohibitive for massive datasets due to the high cost of calculating the inverse of the matrix $X^TX$. The normal equation, derived by minimizing the mean squared error through differentiation, provides a direct solution. However, as the number of features and observations grows, the memory and processing requirements for matrix inversion scale poorly. To address this limitation, optimization algorithms like gradient descent were adopted. Instead of solving for parameters directly, gradient descent initializes random values and iteratively moves toward the minimum of the loss function by following the negative gradient. This process relies on the learning rate, a hyperparameter that determines the step size; if too small, convergence is slow, and if too large, the algorithm may overshoot the minimum. The standard implementation, known as batch gradient descent, calculates the gradient using the entire training dataset before updating parameters. Although this ensures a smooth path toward the optimal solution, it is inefficient for datasets with millions of records, as the full computation must be repeated for every single update. Stochastic gradient descent was introduced to solve this scalability issue. Unlike the batch approach, SGD selects a single random data point at a time to compute the gradient and update the parameters immediately. This significantly reduces the computational burden per iteration, allowing the model to learn much faster. However, because each update is based on a single noisy data point, the path to the minimum is erratic and zig-zagging rather than smooth. Variations like mini-batch gradient descent have since emerged, utilizing small batches of data to balance the speed of SGD with the stability of batch gradient descent. Despite linear regression having a closed-form solution, gradient descent remains indispensable in deep learning and large-scale machine learning. In complex neural networks, closed-form solutions are often mathematically impossible to derive, making iterative optimization the only viable path. Furthermore, for modern applications involving massive datasets, the efficiency of SGD or mini-batch methods far outweighs the computational overhead of matrix inversion. These algorithms form the backbone of training contemporary AI models, enabling the handling of vast amounts of data that were previously unmanageable.