Information Gain Practice Problem

This data science coding problem helps you practice Decision Trees, information gain, and implementation skills. Read the problem statement, write your solution, and strengthen your understanding of Decision Trees.

• Problem ID: 210
• Problem key: 210-information-gain
• URL: https://datacrack.app/solve/210-information-gain
• Difficulty: medium
• Topic: Decision Trees
• Module: Supervised Learning

Problem Statement

# 🧩 Information Gain

---

### 🎯 Goal

Measure how much a decision-tree split reduces uncertainty.

Information gain compares the entropy before a split with the weighted entropy after the split. This problem builds on entropy: instead of measuring uncertainty for one node, we measure how much uncertainty decreases after a split.

---

### 📖 Introduction

A decision tree grows by asking questions. Each question splits one parent node into two child nodes.

A good question should make the child nodes cleaner than the parent node.

Remember, entropy measures uncertainty inside one node:

$$
Entropy = -\sum_{j=1}^{k} p_j\log_2(p_j)
$$

Information gain measures this improvement:

Information gain measures this improvement:

$$
Gain = Entropy_{parent} - Entropy_{children}
$$

The child entropy is weighted by child size:

$$
Entropy_{children}=\frac{n_L}{n}Entropy_L+\frac{n_R}{n}Entropy_R
$$

where:

- $n_L$ is the number of samples in the left child.
- $n_R$ is the number of samples in the right child.
- $n$ is the number of samples in the parent.

---

### 💻 Task

Implement `information_gain(parent_counts, left_counts, right_counts)`.

For this app, use only **one top-level function**. If you want a helper for entropy, define it **inside** `information_gain`.

Your function should return a dictionary with:

- `parent_entropy`
- `left_entropy`
- `right_entropy`
- `weighted_child_entropy`
- `information_gain`

Round every value to 6 decimals.

---

### 📥 Input / 📤 Output

**Input**
- `parent_counts`: class counts before the split
- `left_counts`: class counts in the left child
- `right_counts`: class counts in the right child

**Output**
- dictionary containing entropy values and information gain

---

### 🧩 Starter Code

```python
import math

def information_gain(parent_counts, left_counts, right_counts):
    def entropy_from_counts(class_counts):
        # Helper lives inside the main function
        # Your entropy code here
        pass

    # Your information gain code here
    pass
```

---

### 💡 Example

```python
information_gain([3, 3], [3, 0], [0, 3])
```

Expected Output:

```python
{
    "parent_entropy": 1.0,
    "left_entropy": 0.0,
    "right_entropy": 0.0,
    "weighted_child_entropy": 0.0,
    "information_gain": 1.0
}
```

---

### ⚠️ Common Mistakes

- Averaging left and right entropy without weighting by child size.
- Subtracting in the wrong order.
- Treating high child entropy as good. Lower child entropy is better.

Starter Code

import math

def information_gain(parent_counts, left_counts, right_counts):
    def entropy_from_counts(class_counts):
        # Helper lives inside the main function
        # Your entropy code here
        pass

    # Your information gain code here
    pass