G8 Guru / Judging Someone You Cannot Check

Ask what would change their mind, and listen to the answer

The reply to that one question sorts people faster than almost anything else you can ask.

Ask someone what evidence would make them abandon the position they are advocating. A good answer is specific and describable: if this measurement came back at that value, if this happened by that point, if this alternative explanation held up, I would drop it. The answer proves the person has a model with edges, has considered the possibility of being wrong, and can imagine the world in which they are. It also gives you something concrete to watch for later, which is useful whether they turn out to be right or not.

The failure modes are informative in different ways. Some people say nothing would change their mind, which is at least honest and tells you the claim is not really an empirical one. Others give an answer that is unfalsifiable in practice, setting a condition that could never be observed. Others become irritated by the question itself, treating it as an attack rather than a normal request, and that reaction is often the most revealing response of the three.

The question is also worth turning on yourself before you ask it of anyone else, because the same standard applies. If you cannot say what would move you off your current view, you are not weighing the expert's evidence, you are shopping for agreement, and you will find it, since somebody credentialled can be found for almost any position. Being able to state your own conditions makes you a harder audience to fool and a fairer one to the careful speaker, who is usually the one most willing to answer.