Gradient Descent Update Rule Practice Problem

This data science coding problem helps you practice Linear Regression, gradient descent update rule, and implementation skills. Read the problem statement, write your solution, and strengthen your understanding of Linear Regression.

• Problem ID: 2
• Problem key: 2-gradient-descent-update-rule
• URL: https://datacrack.app/solve/2-gradient-descent-update-rule
• Difficulty: easy
• Topic: Linear Regression
• Module: Introduction to Machine Learning

Problem Statement

# Gradient Descent Update Rule  

---

### 🎯 Goal  
Implement the gradient descent algorithm for **linear regression**, using the gradients of the loss function to iteratively update the model’s weights and bias.  
This problem teaches how a model actually “learns” by minimizing the Mean Squared Error (MSE) step by step.

---

### 🔍 Explanation of Symbols  

| Symbol | Meaning | Shape / Type |
|:-------:|:--------|:-------------|
| **$X$** | Input features | $(n, d)$ |
| **$y$** | True target values | $(n,)$ |
| **$\hat{y}$** | Predictions | $(n,)$ |
| **$w$** | Weights | $(d,)$ |
| **$b$** | Bias | scalar |
| **$\alpha$** | Learning rate | float |
| **$L$** | Mean Squared Error loss | scalar |

---

### 📥 Input / 📤 Output  

**Inputs:**  
- `X`: NumPy array of shape $(n, d)$ — training data  
- `y_true`: NumPy array of shape $(n,)$ — true labels  
- `w`: NumPy array of shape $(d,)$ — initial weights  
- `b`: float — initial bias  
- `alpha`: float — learning rate  
- `iterations`: int — number of update steps  

**Output:**  
- Updated $(w, b)$ after gradient descent optimization  

---

### 💻 Task  

Implement **`gradient_descent_step(X, y_true, w, b, alpha, iterations)`** that:  
1. Computes predictions $ \hat{y} = Xw + b $  
2. Calculates gradients — *use the formulas you derived in the previous problem* (**“Gradients in Linear Regression”**) for  
   $ \frac{\partial L}{\partial w} $ and $ \frac{\partial L}{\partial b} $.  
3. Updates parameters for a given number of iterations:  
   $$
   w := w - \alpha \cdot \frac{\partial L}{\partial w}, \quad
   b := b - \alpha \cdot \frac{\partial L}{\partial b}
   $$
4. Returns the final $(w, b)$ values.

---

### 🧩 Starter Code  

```python
import numpy as np

def gradient_descent_step(X, y_true, w, b, alpha=0.01, iterations=10):
    """
    Perform gradient descent updates for linear regression.
    """
    n = len(y_true)

    for i in range(iterations):
        # TODO: Compute predictions y_pred
        # TODO: Compute gradients (see "Gradients in Linear Regression")
        # TODO: Update weights (w) and bias (b)
        pass

    return w, b
````

---

### 💡 Example + Expected Output

```python
X = [[1], [2], [3], [4]]
y = [3, 5, 7, 9]
w_init = [0.0]
b_init = 0.0

w_final, b_final = gradient_descent_step(X, y, w_init, b_init, alpha=0.01, iterations=1000)
print(w_final, b_final)
```

**Expected Output:**

```
([1.7572225886493895], 0.6071152031917301)
```

---

Starter Code