When Should We Control For Factors?
Conditioning for Causality
Simpson’s Paradox
There is a well known statistical “paradox” Simpson’s Paradox, which discusses the ability for a trend observed in your data to reverse when groups of interest are aggregated. The basic idea is often illustrated with the UC Berkeley Gender Bias example. If you took a look at the admissions data for 1973, you would note that 44% of men who applied, and only 35% of women, were accepted into their respective programs. However, four of the six departments at the school, women were accepted at higher rates than men. The issue is that, on the whole, women were applying to more competitive departments (such as English), where mentended to apply to less competitive disciplines (Engineering and Chemistry). This creates a scenario where there is no explicit discrimination at any department level, but it appears as though there is when aggregated.
The lesson from the above, statistically, tends to be condition on relevant quantities. That is, when we are conducting statistical analysis, we ought to “control for” the effects of possibly influential variables; if the above analysis had initially looked at acceptance rates controlling for department of application, a more correct conclusion would have been reached. Seems easy enough - so what is the issue?
Berkson’s Paradox
(Prepare for Deja Vu) There is a well known statistical “paradox” Berkson’s Paradox, which discusses the ability for a correlation observed in your data to reverse based on the set of individuals observed. The basic idea is often illustrated with Why Are Handsome Men Such Jerks? example. If you imagine scoring the handsomeness, and the niceness, of men numerically, then perhaps Jim will only date a guy who has some niceness plus handsomeness score that exceeds a threshold. We may say that, in the general population, nicer men also tend to be more handsome (who knows?) so that these two traits are positively correlated. However, among the men that Jim dates, he will observe that nicer men are on average less handsome (and vice-versa). Why? Well, if there is an exceedingly nice man, in order to cross the dating threshold for Jim, this guy does not need to be very handsome at all - and as such, there will be more observations that are extreme than would be expected in the general public.
The lesson from the above, statistically, tends to be do not condition on irrelevant quantities. That is, when we are conducting statistical analysis, we ought not “control for” the effects of irrelevant variables; if the above analysis had inititally looked at the pool of men that Jim dates, then we would erroneously conclude that handsomeness and niceness are negatively correlated.
Well, now what?
Without having a subject-matter understanding of the causal structure at play, it is not obvious as to whether a factor is a confounder (e.g. in the first example) or a collider (e.g. in the second example). Confounders should always be conditioned on in a causal analysis, where colliders should never be conditiond on. Doing this incorrectly will lead to incorrect conclusions regarding the presence of a causal effect. This ultimately means that we should not “control for” absolutely every, possible factor; it also ultimately means that we ought to “control for” every relevant factor. Because this cannot be empirically informed, causal inference necessitates importing assumptions from a subject matter expert.