Regularized Loss Function Practice Problem

This data science coding problem helps you practice Regularization for Linear Regression, regularized loss function, and implementation skills. Read the problem statement, write your solution, and strengthen your understanding of Regularization for Linear Regression.

• Problem ID: 129
• Problem key: 129-regularized-loss-function
• URL: https://datacrack.app/solve/129-regularized-loss-function
• Difficulty: medium
• Topic: Regularization for Linear Regression
• Module: Introduction to Machine Learning

Problem Statement

# 🧩 Regularized Loss Function

---

### 🎯 Goal

Understand how **regularization** modifies the standard loss function by adding a **penalty term** that discourages large weights, helping prevent **overfitting**.

---

### 💻 Task  

You are given input features $X$, targets $y$, a weight vector $w$, a regularization type (`l1` or `l2`), and a regularization strength $\lambda$.

Steps:

1. Compute the **base MSE loss**: $L_{\text{base}} = \frac{1}{2N}\sum_{i=1}^{N}(y_i - \hat{y}_i)^2$
2. Compute the **penalty**:
   - L2: $P = \frac{\lambda}{2N}\sum_j w_j^2$
   - L1: $P = \frac{\lambda}{N}\sum_j |w_j|$
3. Compute the **total loss**: $L_{\text{total}} = L_{\text{base}} + P$
4. Return the tuple `(base_loss, penalty, total_loss)`, each rounded to 6 decimal places.

---

### 🔍 Explanation of Symbols

| Symbol | Meaning | Shape / Type |
| :----: | :------ | :----------- |
| $X$ | Input feature matrix | $(N, d)$ |
| $y$ | Target values | $(N,)$ |
| $w$ | Weight vector | $(d,)$ |
| $\lambda$ | Regularization strength | float |
| $N$ | Number of samples | integer |
| $d$ | Number of features | integer |

---

### 📖 Background

In standard linear regression, we minimize the **Mean Squared Error (MSE)**:
$$L_{\text{base}} = \frac{1}{2N}\sum_{i=1}^{N}(y_i - \hat{y}_i)^2$$

However, models with many features can **overfit** — they learn noise instead of patterns. **Regularization** adds a penalty to discourage large weights:

- **L2 (Ridge):** $P = \frac{\lambda}{2N}\|w\|_2^2$ → shrinks weights smoothly toward zero
- **L1 (Lasso):** $P = \frac{\lambda}{N}\|w\|_1$ → can push weights exactly to zero (feature selection)

---

### 📥 Input / 📤 Output

* **Input:** `X`, `y`, `weights`, `reg_type` (`'l1'` or `'l2'`), `lambda_param`
* **Output:** Tuple `(base_loss, penalty, total_loss)` — each a float rounded to 6 decimals

---

### 🧩 Starter Code

```python
import numpy as np

def compute_regularized_loss(X, y, weights, reg_type, lambda_param):
    """
    Compute the regularized loss = base MSE loss + penalty.

    Args:
        X: input features, shape (N, d)
        y: target values, shape (N,)
        weights: weight vector, shape (d,)
        reg_type: 'l1' or 'l2'
        lambda_param: regularization strength
    Returns:
        tuple: (base_loss, penalty, total_loss)
    """
    # TODO: Compute base loss, penalty, and total loss
    pass
```

---

### 💡 Example

```python
X = [[1, 2], [3, 4], [5, 6]]
y = [5, 11, 17]
weights = [1.0, 2.0]
result = compute_regularized_loss(X, y, weights, 'l2', 0.1)
print(result)
```

**Expected Output:**
```
(0.0, 0.083333, 0.083333)
```

---