Unresolved mass

This page explains unresolved mass, the average share of refit outcomes that falls outside every group of the full fit. A refit can count a row fully or in part.

The question it answers

How much of the table do the refits fail to place in a group that the full fit also has?

A refit draws a row when the row is in its random 80 percent subset. For one refit, give a row 1 if the refit called it noise. Otherwise give it the share of its refit group that the full fit calls noise. The unresolved share of the row is the average of that number over the refits that drew it. Unresolved mass is the mean of the unresolved share over all rows that at least one refit drew.

A picture

Take a row r with four refits (toy numbers).

refit 1: r in a refit group that the full fit never calls noise   -> 0.0
refit 2: r in a refit group, 2 of its 10 rows are noise in the full fit -> 0.2
refit 3: r is noise in the refit                                  -> 1.0
refit 4: r in a refit group that the full fit never calls noise   -> 0.0

The unresolved share of r is (0 + 0.2 + 1 + 0) / 4 = 0.3. Averaging such shares over all rows gives the unresolved mass.

The middle panel below shows the unresolved share of every row of the recorded run on the scikit-learn digits dataset. The dashed line is the mean. The recorded runs quoted on this page ship in the repository under notebooks/runs/digits and notebooks/runs/apogee. Notebooks 01 and 02 write them, and starfold.load reads them.

Middle panel: histogram of the unresolved share of every digits row, with the mean (the unresolved mass) as a dashed line.

Look at the tall bar near 0 and the dashed line next to it. Nearly all rows are placed by almost every refit, so the mass is small.

Range and reference values

  • Range: 0 to 1.

  • Perfect result: 0. No refit calls a drawn row noise, and no refit cluster holds rows that the full fit calls noise. It does not show that refits keep the groups apart, so check Support and own support.

  • Pure chance: not applicable.

How to read a value

The 0.5 mark is the threshold of the high_unresolved_mass flag in the code. The other bands are a rule of thumb from this project, not from a published source.

Unresolved mass

Reading

Decision

Below 0.1

Refits place almost every row

Read the other numbers as usual

Between 0.1 and 0.5

A noticeable share of outcomes is noise

Check which groups lose rows with Support and own support

Above 0.5

Most refit outcomes fall outside every full-fit group

Report the flag high_unresolved_mass. Report the full fit’s share of grouped rows separately, and check world_fit_failures

What misleads

  • A low value does not make the groups true. It says the refits agree on where noise is not.

  • The value counts the full fit’s own noise rows. A table with a large diffuse background can have a high value although the groups are stable. The full fit’s noise share is in (result.labels < 0).mean().

  • Refits that fail count as all noise, so failures raise the mass. Check result.worlds.world_errors.

  • A mean hides structure. One group can lose many rows while the mean stays low.

Where Starfold prints it

The attribute is result.unresolved_mass (same value as result.worlds.unresolved_mass). The per-row values are in result.worlds.unresolved. The notebook notebooks/01_digits.ipynb printed the recorded line below, from 32 refits.

print(f"unresolved mass: {result.unresolved_mass:.3f}")

unresolved mass: 0.044

The recorded run on the APOGEE DR19 example table (notebooks/02_apogee.ipynb) gave 0.154, which the notebook reads as about 15 percent of a star’s refit outcomes being “no group”.

Formula

For row \(i\) with \(W_i\) the refits that drew it, let \(u_{w,i} = 1\) if refit \(w\) gives row \(i\) noise. Otherwise \(u_{w,i} = s_{k0}\), the share of its refit cluster \(k\) that the full fit calls noise. Then

\[ \text{unresolved}[i] = \frac{1}{|W_i|}\sum_{w\in W_i} u_{w,i}, \qquad \text{unresolved mass} = \frac{1}{|S|}\sum_{i \in S} \text{unresolved}[i], \]

where \(S\) is the set of rows drawn by at least one refit. For every \(i \in S\), \(\sum_g \text{support}[i,g] + \text{unresolved}[i] = 1\).

Learn more