NOISE

Title: NOISE

Writer: Daniel Kahneman, Olivier Sibony, Cass R. Sunstein

 

I went on a trip to Japan, and in the bookstore I found this book. As I couldn't read Chinese characters well, I rather chose this English book. There, I just skimmed over this book, but still, I could find out that this book is amazing. This book distinguishes noise from bias. Highly biased means that the average score deviates a lot from the ideal score, while highly noisy means that scores scatter all over the average score. As one of the students currently studying Artificial Intelligence, I believe applying this area well to human society is one of the keys to solve the problem of noise. I felt that knowing what noise is and how to deal with it would be really helpful for my AI study. So I bought this book. While reading this book almost every day, I'm trying to summarize some main points here.

 

 

 

MAIN POINT I

NOISE IS FOUND EVERYWHERE

 

1. Noise is found in a courtroom. "Two men, neither of whom had a criminal record, were convicted for cashing counterfeit checks in the amounts of $58.40 and $35.20, respectively. The first man was sentenced to fifteen years, and the second to 30 days." Systematically speaking, this happened because there was a substantial variation in the length of prison terms recommended(i.e., 0-25 years to a bank robbery). Shocked by this situation, Edward M. Kennedy pressed the Congress, and Congress enacted the Sentencing Reform Act of 1984. In the following year, guidelines which established a restricted range for criminal sentences were made. Due to the guidelines, sentence variations across judges became much smaller. Interestingly, the guidelines went into fire in critics, stating that the complexities of the individual case should be taken account for. Which one is the right thing, equal fairness or unequal leniency?
2. Noise is found in an insurance system. Most executives of the insurance company guessed 10% or less variation in similar expert judgments. However, a noise audit found out that the median difference ratio in underwriting was 55% and the median difference ratio for claims adjusters was 43%. In some cases where selection works, we need variation. At the same time, there is an unwanted variability. We need to know that "in noise systems, errors do not cancel out, but add up." To alleviate this effect, we can average out measures made by several professions. Of course, it costs a lot. Can you weigh the two important factors in human life, justice and cost, and find out the best value within?
3. We can regard a singular decision as a recurrent decision that is made only once. Even though it is not possible to measure how massive the noise is regarding a singular decision, still by treating a singular decision as a recurrent decision, noise can be reduced.

 

MAIN POINT II

MEASUREMENT BEGETS NOISE

 

1. We make a judgment after getting an internal signal of judgment completion in which we feel right enough to decide the answer. There are two kinds of judgments. One is a predictive judgment, where the observers can see if the judgment or prediction was right or wrong in the future. The other is an evaluative judgment, where we'll never know the answer to our judgment. In the first case, the judgment is evaluated by both the outcome and the process. But, in the second case, the judgment can be only evaluated by the process.
2. Here's a maxim. "bias and noise play the same role in the calculation of overall error."
   So, $Overall Error(MSE)$ = $Bias^2$ + $Noise^2$
   In the equation, the negative value of the noise and the positive value of the noise are treated equally, as they are squared. However, in reality, underestimating or overestimating the maximum load of an elevator play different roles. Underestimation is costly, but overestimation could be catastrophic.
3. If you analyze noise, you will find out that
   $System Noise$ =  $Level Noise^2$ + $Pattern Noise^2$
   where level noise is variability in the average level of judgments by different judges
   and pattern noise is variability in judges' responses to particular cases.
4. Even though you rate the same thing twice in 2 weeks interval, the score will be different. It is because you're influenced by your mood, by the preceding accidents you experienced that day, and even by the weather. You are not the same person at all times. One interesting thing here: by just averaging the independent judgments of different people, accuracy improves. This is called wisdom-of-crowds effect. And it can be applied even in the case when you judge one accident twice, and average out your opinions! Regarding AI models, we know that decisions made by several models are better than the decisions made by a single model. 
5. Sadly, crowds effect is not always positive. If the judgments are shared by the members of the group. Then usually early popularity decides most of the decisions. This effect is applied to all the areas, including political area and industrial area. Be aware of it!

 

MAIN POINT III

NOISE IN PREDICTIVE JUDGMENTS

 

1. When making predictive judgments, linear regression is always better than clinical predictions. Even, linear equation made by human is better than clinical predictions. Human applies complex method to compute the prediction for some cases but not for the others, thus, inevitably makes some noises.
2. According to the extent of the complexity of the models for judgments, models can be classified into four groups, simple rules, improper linear models, linear regression models, and machine learning models. Due to the poor representation of the real world made by the data we have, sometimes simpler models like simple rules or improper linear models achieve similar performances to those of the regression models. Machine learning models are able to capture more information and thus produce better outputs most of the time. However, if data itself contain some biases, then models will inevitably have some biases. Still, we have to be aware that human judgments contain more biases and mistakes.