AKA Goodharting
If you use a metric to optimize something the metric is gonna be targeted by other actors and with time becomes less good metric to measure something
If you pay your employees by hours of work, incentives are set such people will tend to stay at work longer doing less,
Incentives in education
The ordinary-life equivalent is “teaching to the test”. The system’s programmers, eg. the Department of Education have an objective: Children should learn. They delegate that objective to mesa-optimizers (the teachers) via a proxy objective (children should do well on the standardized test) and a correlated reward function (teachers get paid more if their students get higher test scores). The teachers can either pursue the base objective for less reward (teach children useful skills), or pursue their mesa-level objective for more reward (teach them how to do well on the test). An alignment failure!
The idea of “What’s measured gets managed”. If you want to be healthier and say focus on improving markers, like V02 max or length of sleep, you may end up rising VO2 max by exercising more and sleeping longer. I think Goodharting often applies more to situation where it relates to the parent markers that are not in your interest. Imagine you are hired to layer bricks and are paid by amount of bricks laid. You are looking for money and company is looking for finished buildings you will be incentivized to build rooms with very high ceiling .