Information gain ratio

Let ${\displaystyle Attr}${\displaystyle Attr} be the set of all attributes and ${\displaystyle Ex}${\displaystyle Ex} the set of all training examples, ${\displaystyle value(x,a)}${\displaystyle value(x,a)} with ${\displaystyle x\in Ex}${\displaystyle x\in Ex} defines the value of a specific example ${\displaystyle x}${\displaystyle x} for attribute ${\displaystyle a\in Attr}${\displaystyle a\in Attr}, ${\displaystyle H}${\displaystyle H} specifies the entropy. The information gain for an attribute ${\displaystyle a\in Attr}${\displaystyle a\in Attr} is defined as follows:

${\displaystyle IG(Ex,a)=H(Ex)-\sum _{v\in values(a)}{\frac {|\{x\in Ex|value(x,a)=v\}|}{|Ex|}}\bullet H(\{x\in Ex|value(x,a)=v\})}${\displaystyle IG(Ex,a)=H(Ex)-\sum _{v\in values(a)}{\frac {|\{x\in Ex|value(x,a)=v\}|}{|Ex|}}\bullet H(\{x\in Ex|value(x,a)=v\})}

The information gain is equal to the total entropy for an attribute if for each of the attribute values a unique classification can be made for the result attribute. In this case the relative entropies subtracted from the total entropy are 0.

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "http://localhost:6011/en.wikipedia.org/v1/":): {\displaystyle IV(Ex,a)= -\sum_{v\in values(a)} \frac{|\Ex}{|Ex|}}

${\displaystyle IGR(Ex,a)=IG/IV}${\displaystyle IGR(Ex,a)=IG/IV}

Information gain ratio biases the decision tree against considering attributes with a large number of distinct values. So it solves the drawback of information gain (information gain applied to attributes that can take on a large number of distinct values might learn the training set too well. For example, suppose that we are building a decision tree for some data describing a business's customers. Information gain is often used to decide which of the attributes are the most relevant, so they can be tested near the root of the tree. One of the input attributes might be the customer's credit card number. This attribute has a high information gain, because it uniquely identifies each customer, but we do not want to include it in the decision tree: deciding how to treat a customer based on their credit card number is unlikely to generalize to customers we haven't seen before.)

• Wikipedia introduction cleanup from November 2008
• Articles covered by WikiProject Wikify from November 2008
• Articles lacking sources from November 2008

• Articles with invalid date parameter in template
• All pages needing cleanup
• Pages missing lead section
• All articles covered by WikiProject Wikify
• All articles lacking sources
• Articles with math errors
• Articles with math render errors