Relative vs. absolute $\chi^2$

On this week’s workshop on $B$ decays in Edinburgh, which unfortunately I was not able to attend, apparently there have been many interesting talks and lots of fruitful discussion. An interesting point was raised on the slides of the talk by N. Mahmoudi regarding the statistical approach taken in my recent paper with Wolfgang Altmannshofer. It concerns the allowed regions for new physics contributions to the Wilson coefficients as obtained from a global fit to experimental data. In our paper, we used the $\chi^2$ function defined as
$$\chi^2(\vec C^{\rm NP})=\left[\vec O_{\rm exp}-\vec O_{\rm th}(\vec C^{\rm NP})\right]^T\left[C_{\rm exp}+C_{\rm th}\right]^{-1}\left[\vec O_{\rm exp}-\vec O_{\rm th}(\vec C^{\rm NP})\right]$$
where $\vec C^{\rm NP}$ are the new physics contributions to the Wilson coefficients, $\vec O_{\rm th}$ are the observables, and $C_\text{exp,th}$ the experimental and theoretical covariance matrices, respectively. The latter encode the experimental and theoretical uncertainties as well as their correlations.

Now, when we present plots with constraints on Wilson coefficients such as this one, we proceed as follows.

The $\Delta\chi^2$ method

• Make a hypothesis which two Wilson coefficients receive new physics contributions (assuming all others are SM-like),
• Determine the minimum value of $\chi^2$ under variation of these two coefficients,
• Plot contours of $\Delta\chi^2=2.3$ and $6$, where $\Delta\chi^2$ is the difference of the $\chi^2$ with respect to the minimum value in the previous item.

The numbers are chosen because $F_{\chi^2}(2.3, 2) \approx 0.68$, $F_{\chi^2}(6, 2) \approx 0.95$, where  $F_{\chi^2}(x, 2)$ is the cumulative distribution function (CDF) of the $\chi^2$ distribution with 2 degrees of freedom. That is, the regions we plot are the 68% and 95% credibility regions for the Wilson coefficients under the hypothesis that new physics resides only in these two coefficients.

The “absolute $\chi^2$” method

Now, in the talk mentioned above, an alternative method is proposed, coined “absolute $\chi^2$ method”. In this method, one plots contours of the absolute $\chi^2$ value, rather than $\Delta\chi^2$, again assuming that new physics affects two Wilson coefficients. Conversely, the $\Delta\chi^2$ is claimed to be “NOT appropriate … to claim no physics”. Several plots are shown to demonstrate that the “absolute” method leads to looser constraints on Wilson coefficients $C_9$ and $C_{10}$.

Comparing the two

For a fair comparison, I reproduced the plots of $C_9$ and $C_{10}$ using the numerics of our paper. First, we need to determine the $\chi^2$ values corresponding to the “1 and 2$\sigma$ regions” in the “absolute” method. This can be determined as  $F_{\chi^2}^{-1}(\alpha, \nu)$, where $F_{\chi^2}^{-1}(x,\nu)$ is the inverse CDF of the $\chi^2$ distribution with $\nu$ degrees of freedom, $\nu=N-2$ for $N$ observables and 2 Wilson coefficients, and $\alpha=0.68$ or $0.95$.

In our nominal fit described in the paper, we have $N=88$, thus for the “$1\sigma$” region, $F_{\chi^2}^{-1}(0.68, 86) =$ 91.7. The minimum (i.e. best-fit) $\chi^2$ when varying $C_9$ and $C_{10}$ is given by 102.7, while $\chi^2$ value at the SM point is 116.9. Comparing the two methods,

• Using $\Delta\chi^2$, the best-fit scenario improves over the SM by 14.2. Assuming everything is normally distributed, this can be roughly translated to a “number of sigmas” by equating the CDFs of the $\chi^2$ distribution for $\nu=2$ with a standard Gaussian, and the result is $3.6\sigma$.
• Using the “absolute $\chi^2$”, there is a paradox: In the entire plane, the “$1\sigma$” value cannot be attained since $91.7 < 102.7$. The “$2\sigma$” region does contain the best-fit point, but not the SM point.

Graphically: Left, the $\Delta\chi^2$ regions, right the “absolute $\chi^2$” regions. Note that the similarity of the “$2\sigma$” regions is a numerical coincidence, the “$1\sigma$” region in the right-hand plot is even completely gone.

How ist this possible? Well, the reason is very simple. Even if the theoretical model describes nature perfectly, the data have statistical as well as systematic uncertainties leading to irreducible constant contributions to $\chi^2$. For instance, the two measurements of $F_L$ in $B\to K^*\mu^+\mu^-$ by ATLAS and LHCb at low $q^2$ are not compatible with each other within 1 standard deviation, which contributes a constant positive contribution to $\chi^2$ when they are treated as independent.

Using only “sensitive” observables

On the other hand, in the talk at hand, it is also mentioned that one should remove an observable from the fit if it is “relatively insensitive to the variation of the Wilson coefficients”. Could this help to solve the paradox? To determine which these observables are, I computed
$$\delta_i = \left[O_i \left( \vec C^{\rm NP} \right) – O_i \left( \vec 0 \right) \right] / \sqrt{\sigma_{{\rm th},i}^2+\sigma_{{\rm exp},i}^2}$$
i.e. the relative variation of an observable under variation of the Wilson coefficients, normalized to the combined experimental and theoretical uncertainty. Requiring that $|\delta|>1$ for the benchmark points $C_9^{\rm NP}=-1.5$ and $C_{10}^{\rm NP}=+1.5$, we are left with just $N=37$ observables.

Let’s repeat the previous game: we get$F_{\chi^2}^{-1}(0.68, 35) =38.4$, $F_{\chi^2}^{-1}(0.95, 35) =50.3$, $\chi^2_{\rm min}=55.9$, $\chi^2_{\rm SM}=67.9$. Again, the “$1\sigma$” value cannot be attained, and this time, not even the best fit point is within the “$2\sigma$” region!

Graphically: The left plot, using the “traditional” $\Delta\chi^2$ method, is now a bit looser since less observables are included. With the “absolute $\chi^2$”, the regions disappear completely.

Upshot

The problem of the “absolute $\chi^2$” method can be understood with a simple gedanken experiment. Let’s assume experiment X measures one of the “sensitive” observables but, due to a measurement error, is very far off the “true” value, while the other experiments got it right. In the $\Delta\chi^2$ method, this will have no impact, since it will simply shift the $\chi^2$ by a virtually constant amount. In the absolute $\chi^2$ method instead, this can lead to a drastic shrinking of the “allowed regions”. And in fact this is what happens taking the numerics of our paper that includes a large number of observables.

To summarize, I don’t think the “absolute $\chi^2$” plots can be used to judge the significance of a possible new physics contribution (or of an underestimation of SM uncertainties that mimicks new physics).

I would be happy to hear your opinions.