There are several distinct failure modes for overoptimization of systems on
the basis of metrics. This occurs when a metric which can be used to improve a
system is used to an extent that further optimization is ineffective or
harmful, and is sometimes termed Goodhart's Law. This class of failure is often
poorly understood, partly because terminology for discussing them is ambiguous,
and partly because discussion using this ambiguous terminology ignores
distinctions between different failure modes of this general type. This paper
expands on an earlier discussion by Garrabrant, which notes there are "(at
least) four different mechanisms" that relate to Goodhart's Law. This paper is
intended to explore these mechanisms further, and specify more clearly how they
occur. This discussion should be helpful in better understanding these types of
failures in economic regulation, in public policy, in machine learning, and in
Artificial Intelligence alignment. The importance of Goodhart effects depends
on the amount of power directed towards optimizing the proxy, and so the
increased optimization power offered by artificial intelligence makes it
especially critical for that field.