cca_zoo.model_selection¶

Cross-validation, hyperparameter search and significance testing for multiview models.

GridSearchCV ¶

Bases: _MultiviewSearch, GridSearchCV

Exhaustive grid search over a multiview estimator's parameters.

The multiview form of :class:~sklearn.model_selection.GridSearchCV, with its parameters, attributes and methods. fit, transform and score take a list of views; parameter names are the estimator's own, with name__<view> setting one view's value of a per-view parameter; best_estimator_ is the fitted multiview estimator; and a scoring callable is called as scoring(estimator, views).

Examples:

>>> import numpy as np
>>> from cca_zoo.linear import RidgeCCA
>>> from cca_zoo.model_selection import GridSearchCV
>>> rng = np.random.default_rng(0)
>>> X1 = rng.standard_normal((50, 5))
>>> X2 = rng.standard_normal((50, 4))
>>> grid = {"shrinkage__0": [0.0, 0.1], "shrinkage__1": [0.0, 0.5]}
>>> gs = GridSearchCV(RidgeCCA(), param_grid=grid, cv=2).fit([X1, X2])
>>> sorted(gs.best_params_.items())
[('shrinkage__0', 0.0), ('shrinkage__1', 0.5)]

RandomizedSearchCV ¶

Bases: _MultiviewSearch, RandomizedSearchCV

Randomized search over a multiview estimator's parameters.

The multiview form of :class:~sklearn.model_selection.RandomizedSearchCV, with its parameters, attributes and methods. fit, transform and score take a list of views; parameter names are the estimator's own, with name__<view> setting one view's value of a per-view parameter; best_estimator_ is the fitted multiview estimator; and a scoring callable is called as scoring(estimator, views).

Examples:

>>> import numpy as np
>>> from scipy.stats import loguniform
>>> from cca_zoo.linear import RidgeCCA
>>> from cca_zoo.model_selection import RandomizedSearchCV
>>> rng = np.random.default_rng(0)
>>> X1 = rng.standard_normal((50, 5))
>>> X2 = rng.standard_normal((50, 4))
>>> space = {"shrinkage__0": loguniform(1e-3, 1.0)}
>>> rs = RandomizedSearchCV(
...     RidgeCCA(), space, n_iter=5, cv=2, random_state=0
... ).fit([X1, X2])
>>> sorted(rs.best_params_)
['shrinkage__0']

HalvingGridSearchCV ¶

Bases: _MultiviewSearch, HalvingGridSearchCV

Successive-halving grid search over a multiview estimator's parameters.

The multiview form of :class:~sklearn.model_selection.HalvingGridSearchCV, with its parameters, attributes and methods. fit, transform and score take a list of views; parameter names are the estimator's own, with name__<view> setting one view's value of a per-view parameter; best_estimator_ is the fitted multiview estimator; and a scoring callable is called as scoring(estimator, views).

Examples:

>>> import numpy as np
>>> from cca_zoo.linear import RidgeCCA
>>> from cca_zoo.model_selection import HalvingGridSearchCV
>>> rng = np.random.default_rng(0)
>>> X1 = rng.standard_normal((60, 5))
>>> X2 = rng.standard_normal((60, 4))
>>> grid = {"shrinkage__0": [0.0, 0.1, 0.5]}
>>> hgs = HalvingGridSearchCV(RidgeCCA(), grid, cv=2, random_state=0)
>>> sorted(hgs.fit([X1, X2]).best_params_)
['shrinkage__0']

HalvingRandomSearchCV ¶

Bases: _MultiviewSearch, HalvingRandomSearchCV

Successive-halving randomized search over a multiview model's parameters.

The multiview form of :class:~sklearn.model_selection.HalvingRandomSearchCV, with its parameters, attributes and methods. fit, transform and score take a list of views; parameter names are the estimator's own, with name__<view> setting one view's value of a per-view parameter; best_estimator_ is the fitted multiview estimator; and a scoring callable is called as scoring(estimator, views).

Examples:

>>> import numpy as np
>>> from scipy.stats import loguniform
>>> from cca_zoo.linear import RidgeCCA
>>> from cca_zoo.model_selection import HalvingRandomSearchCV
>>> rng = np.random.default_rng(0)
>>> X1 = rng.standard_normal((60, 5))
>>> X2 = rng.standard_normal((60, 4))
>>> space = {"shrinkage__0": loguniform(1e-3, 1.0)}
>>> hrs = HalvingRandomSearchCV(RidgeCCA(), space, cv=2, random_state=0)
>>> sorted(hrs.fit([X1, X2]).best_params_)
['shrinkage__0']

OptunaSearchCV `module-attribute` ¶

OptunaSearchCV = _search.OptunaSearchCV

cross_val_score ¶

cross_val_score(
    estimator: BaseEstimator,
    views: list[ArrayLike],
    **kwargs: Any,
) -> np.ndarray

Score a multiview estimator by cross-validation.

:func:sklearn.model_selection.cross_val_score on a list of views.

Parameters:

Name	Type	Description	Default
`estimator`	`BaseEstimator`	A multiview estimator.	required
`views`	`list[ArrayLike]`	Arrays of shape (n_samples, n_features_i), one per view.	required
`**kwargs`	`Any`	Passed to :func:`sklearn.model_selection.cross_val_score`, such as `cv`, `groups` and `n_jobs`. A `scoring` callable is called as `scoring(estimator, views)`.	`{}`

Returns:

Type	Description
`ndarray`	The score of each fold, by default the mean canonical correlation.

Examples:

>>> import numpy as np
>>> from cca_zoo.linear import CCA
>>> from cca_zoo.model_selection import cross_val_score
>>> rng = np.random.default_rng(0)
>>> X1, X2 = rng.standard_normal((50, 5)), rng.standard_normal((50, 4))
>>> cross_val_score(CCA(), [X1, X2], cv=3).shape
(3,)

Source code in cca_zoo/model_selection/_validation.py

def cross_val_score(
    estimator: BaseEstimator, views: list[ArrayLike], **kwargs: Any
) -> np.ndarray:
    """Score a multiview estimator by cross-validation.

    :func:`sklearn.model_selection.cross_val_score` on a list of views.

    Args:
        estimator: A multiview estimator.
        views: Arrays of shape (n_samples, n_features_i), one per view.
        **kwargs: Passed to :func:`sklearn.model_selection.cross_val_score`,
            such as ``cv``, ``groups`` and ``n_jobs``. A ``scoring`` callable
            is called as ``scoring(estimator, views)``.

    Returns:
        The score of each fold, by default the mean canonical correlation.

    Examples:
        >>> import numpy as np
        >>> from cca_zoo.linear import CCA
        >>> from cca_zoo.model_selection import cross_val_score
        >>> rng = np.random.default_rng(0)
        >>> X1, X2 = rng.standard_normal((50, 5)), rng.standard_normal((50, 4))
        >>> cross_val_score(CCA(), [X1, X2], cv=3).shape
        (3,)
    """
    wrapper, X = _wrap(estimator, views)
    scores: np.ndarray = skms.cross_val_score(
        wrapper, X, scoring=_unwrapped_scoring(kwargs.pop("scoring", None)), **kwargs
    )
    return scores

cross_validate ¶

cross_validate(
    estimator: BaseEstimator,
    views: list[ArrayLike],
    **kwargs: Any,
) -> dict[str, Any]

Evaluate a multiview estimator by cross-validation.

:func:sklearn.model_selection.cross_validate on a list of views.

Parameters:

Name	Type	Description	Default
`estimator`	`BaseEstimator`	A multiview estimator.	required
`views`	`list[ArrayLike]`	Arrays of shape (n_samples, n_features_i), one per view.	required
`**kwargs`	`Any`	Passed to :func:`sklearn.model_selection.cross_validate`, such as `cv`, `return_train_score` and `return_estimator`. A `scoring` callable, or each in a dict, is called as `scoring(estimator, views)`.	`{}`

Returns:

Type	Description
`dict[str, Any]`	sklearn's result dictionary; with `return_estimator`, the fitted
`dict[str, Any]`	multiview estimators.

Source code in cca_zoo/model_selection/_validation.py

def cross_validate(
    estimator: BaseEstimator, views: list[ArrayLike], **kwargs: Any
) -> dict[str, Any]:
    """Evaluate a multiview estimator by cross-validation.

    :func:`sklearn.model_selection.cross_validate` on a list of views.

    Args:
        estimator: A multiview estimator.
        views: Arrays of shape (n_samples, n_features_i), one per view.
        **kwargs: Passed to :func:`sklearn.model_selection.cross_validate`,
            such as ``cv``, ``return_train_score`` and ``return_estimator``.
            A ``scoring`` callable, or each in a dict, is called as
            ``scoring(estimator, views)``.

    Returns:
        sklearn's result dictionary; with ``return_estimator``, the fitted
        multiview estimators.
    """
    wrapper, X = _wrap(estimator, views)
    results: dict[str, Any] = skms.cross_validate(
        wrapper, X, scoring=_unwrapped_scoring(kwargs.pop("scoring", None)), **kwargs
    )
    if "estimator" in results:
        results["estimator"] = [fitted.estimator_ for fitted in results["estimator"]]
    return results

cross_val_predict ¶

cross_val_predict(
    estimator: BaseEstimator,
    views: list[ArrayLike],
    **kwargs: Any,
) -> list[np.ndarray]

Out-of-fold scores of a multiview estimator.

:func:sklearn.model_selection.cross_val_predict with method="transform" on a list of views: each sample is transformed by the model fitted without its fold. Each fold's model fixes its own sign and order of components, so compare views within a component (e.g. with :func:cca_zoo.metrics.pairwise_correlations) rather than scores across folds.

Parameters:

Name	Type	Description	Default
`estimator`	`BaseEstimator`	A multiview estimator.	required
`views`	`list[ArrayLike]`	Arrays of shape (n_samples, n_features_i), one per view.	required
`**kwargs`	`Any`	Passed to :func:`sklearn.model_selection.cross_val_predict`, such as `cv`, `groups` and `n_jobs`.	`{}`

Returns:

Type	Description
`list[ndarray]`	Arrays of shape (n_samples, n_components), one per view.

Examples:

>>> import numpy as np
>>> from cca_zoo.linear import CCA
>>> from cca_zoo.model_selection import cross_val_predict
>>> rng = np.random.default_rng(0)
>>> X1, X2 = rng.standard_normal((50, 5)), rng.standard_normal((50, 4))
>>> [s.shape for s in cross_val_predict(CCA(n_components=2), [X1, X2], cv=3)]
[(50, 2), (50, 2)]

Source code in cca_zoo/model_selection/_validation.py

def cross_val_predict(
    estimator: BaseEstimator, views: list[ArrayLike], **kwargs: Any
) -> list[np.ndarray]:
    """Out-of-fold scores of a multiview estimator.

    :func:`sklearn.model_selection.cross_val_predict` with ``method="transform"``
    on a list of views: each sample is transformed by the model fitted without
    its fold. Each fold's model fixes its own sign and order of components, so
    compare views within a component (e.g. with
    :func:`cca_zoo.metrics.pairwise_correlations`) rather than scores across folds.

    Args:
        estimator: A multiview estimator.
        views: Arrays of shape (n_samples, n_features_i), one per view.
        **kwargs: Passed to :func:`sklearn.model_selection.cross_val_predict`,
            such as ``cv``, ``groups`` and ``n_jobs``.

    Returns:
        Arrays of shape (n_samples, n_components), one per view.

    Examples:
        >>> import numpy as np
        >>> from cca_zoo.linear import CCA
        >>> from cca_zoo.model_selection import cross_val_predict
        >>> rng = np.random.default_rng(0)
        >>> X1, X2 = rng.standard_normal((50, 5)), rng.standard_normal((50, 4))
        >>> [s.shape for s in cross_val_predict(CCA(n_components=2), [X1, X2], cv=3)]
        [(50, 2), (50, 2)]
    """
    wrapper, X = _wrap(estimator, views)
    # sklearn's validation insists on ``predict`` even with method="transform".
    with sklearn.config_context(skip_parameter_validation=True):
        scores = skms.cross_val_predict(wrapper, X, method="transform", **kwargs)
    return np.split(scores, len(views), axis=1)

learning_curve ¶

learning_curve(
    estimator: BaseEstimator,
    views: list[ArrayLike],
    **kwargs: Any,
) -> tuple[np.ndarray, ...]

Cross-validated scores of a multiview estimator against training set size.

:func:sklearn.model_selection.learning_curve on a list of views.

Parameters:

Name	Type	Description	Default
`estimator`	`BaseEstimator`	A multiview estimator.	required
`views`	`list[ArrayLike]`	Arrays of shape (n_samples, n_features_i), one per view.	required
`**kwargs`	`Any`	Passed to :func:`sklearn.model_selection.learning_curve`, such as `train_sizes` and `cv`.	`{}`

Returns:

Type	Description
`tuple[ndarray, ...]`	sklearn's `(train_sizes, train_scores, test_scores, ...)`.

Source code in cca_zoo/model_selection/_validation.py

def learning_curve(
    estimator: BaseEstimator, views: list[ArrayLike], **kwargs: Any
) -> tuple[np.ndarray, ...]:
    """Cross-validated scores of a multiview estimator against training set size.

    :func:`sklearn.model_selection.learning_curve` on a list of views.

    Args:
        estimator: A multiview estimator.
        views: Arrays of shape (n_samples, n_features_i), one per view.
        **kwargs: Passed to :func:`sklearn.model_selection.learning_curve`,
            such as ``train_sizes`` and ``cv``.

    Returns:
        sklearn's ``(train_sizes, train_scores, test_scores, ...)``.
    """
    wrapper, X = _wrap(estimator, views)
    curve: tuple[np.ndarray, ...] = skms.learning_curve(
        wrapper,
        X,
        None,
        scoring=_unwrapped_scoring(kwargs.pop("scoring", None)),
        **kwargs,
    )
    return curve

validation_curve ¶

validation_curve(
    estimator: BaseEstimator,
    views: list[ArrayLike],
    param_name: str,
    param_range: ArrayLike,
    **kwargs: Any,
) -> tuple[np.ndarray, np.ndarray]

Cross-validated scores of a multiview estimator across one parameter's values.

:func:sklearn.model_selection.validation_curve on a list of views. param_name may set one view's value of a per-view parameter, e.g. "shrinkage__0".

Parameters:

Name	Type	Description	Default
`estimator`	`BaseEstimator`	A multiview estimator.	required
`views`	`list[ArrayLike]`	Arrays of shape (n_samples, n_features_i), one per view.	required
`param_name`	`str`	Parameter to vary.	required
`param_range`	`ArrayLike`	Values to try.	required
`**kwargs`	`Any`	Passed to :func:`sklearn.model_selection.validation_curve`, such as `cv`.	`{}`

Returns:

Type	Description
`tuple[ndarray, ndarray]`	`(train_scores, test_scores)`, each of shape (n_values, n_folds).

Examples:

>>> import numpy as np
>>> from cca_zoo.linear import RidgeCCA
>>> from cca_zoo.model_selection import validation_curve
>>> rng = np.random.default_rng(0)
>>> X1, X2 = rng.standard_normal((50, 5)), rng.standard_normal((50, 4))
>>> train, test = validation_curve(
...     RidgeCCA(), [X1, X2], "shrinkage__0", [0.0, 0.5, 1.0], cv=3
... )
>>> test.shape
(3, 3)

Source code in cca_zoo/model_selection/_validation.py

def validation_curve(
    estimator: BaseEstimator,
    views: list[ArrayLike],
    param_name: str,
    param_range: ArrayLike,
    **kwargs: Any,
) -> tuple[np.ndarray, np.ndarray]:
    """Cross-validated scores of a multiview estimator across one parameter's values.

    :func:`sklearn.model_selection.validation_curve` on a list of views.
    ``param_name`` may set one view's value of a per-view parameter, e.g.
    ``"shrinkage__0"``.

    Args:
        estimator: A multiview estimator.
        views: Arrays of shape (n_samples, n_features_i), one per view.
        param_name: Parameter to vary.
        param_range: Values to try.
        **kwargs: Passed to :func:`sklearn.model_selection.validation_curve`,
            such as ``cv``.

    Returns:
        ``(train_scores, test_scores)``, each of shape (n_values, n_folds).

    Examples:
        >>> import numpy as np
        >>> from cca_zoo.linear import RidgeCCA
        >>> from cca_zoo.model_selection import validation_curve
        >>> rng = np.random.default_rng(0)
        >>> X1, X2 = rng.standard_normal((50, 5)), rng.standard_normal((50, 4))
        >>> train, test = validation_curve(
        ...     RidgeCCA(), [X1, X2], "shrinkage__0", [0.0, 0.5, 1.0], cv=3
        ... )
        >>> test.shape
        (3, 3)
    """
    wrapper, X = _wrap(estimator, views)
    train_scores, test_scores = skms.validation_curve(
        wrapper,
        X,
        None,
        param_name=_PARAM_PREFIX + param_name,
        param_range=param_range,
        scoring=_unwrapped_scoring(kwargs.pop("scoring", None)),
        **kwargs,
    )
    return train_scores, test_scores

permutation_test_significance ¶

permutation_test_significance(
    estimator: BaseEstimator,
    views: list[ArrayLike],
    n_permutations: int = 1000,
    random_state: int | RandomState | None = None,
    n_jobs: int | None = None,
) -> PermutationTestResult

Permutation test of the canonical correlations and factor loadings.

Refits estimator on data with the rows of every view but the first shuffled independently, which keeps each view's covariance and breaks their correspondence. Each permuted fit's loadings are aligned to the observed ones by orthogonal Procrustes over all views before comparison, since permuted fits can rotate or reflect near-tied dimensions.

Only the first dimension's p-value is a valid test. Each later dimension is compared with the permuted fits' dimension of the same rank, which were fitted without removing the earlier dimensions' signal; Winkler et al. (2020) test them step-down, residualising each earlier dimension first.

Parameters:

Name	Type	Description	Default
`estimator`	`BaseEstimator`	An unfitted multiview estimator.	required
`views`	`list[ArrayLike]`	Arrays of shape (n_samples, n_features_i), one per view.	required
`n_permutations`	`int`	Number of permutations. Default is 1000.	`1000`
`random_state`	`int \| RandomState \| None`	Seed for the permutations. Default is None.	`None`
`n_jobs`	`int \| None`	Number of parallel jobs. Default is None.	`None`

Returns:

Type	Description
`PermutationTestResult`	The observed and null statistics.

Raises:

Type	Description
`ValueError`	If `n_permutations` is not positive.

References

Winkler, A. M., Renaud, O., Smith, S. M., & Nichols, T. E. (2020). Permutation inference for canonical correlation analysis. NeuroImage, 220, 117065.

Examples:

>>> import numpy as np
>>> from cca_zoo.linear import CCA
>>> from cca_zoo.model_selection import permutation_test_significance
>>> rng = np.random.default_rng(0)
>>> z = rng.standard_normal((40, 1))
>>> X1 = z @ rng.standard_normal((1, 5)) + 0.1 * rng.standard_normal((40, 5))
>>> X2 = z @ rng.standard_normal((1, 4)) + 0.1 * rng.standard_normal((40, 4))
>>> result = permutation_test_significance(
...     CCA(), [X1, X2], n_permutations=49, random_state=0
... )
>>> float(result.p_values[0])
0.02

Source code in cca_zoo/model_selection/_significance.py

@validate_params(
    {
        "estimator": [HasMethods(["fit", "transform"])],
        "views": [list],
        "n_permutations": [Interval(Integral, 1, None, closed="left")],
        "random_state": ["random_state"],
        "n_jobs": [Integral, None],
    },
    prefer_skip_nested_validation=False,
)
def permutation_test_significance(
    estimator: BaseEstimator,
    views: list[ArrayLike],
    n_permutations: int = 1000,
    random_state: int | np.random.RandomState | None = None,
    n_jobs: int | None = None,
) -> PermutationTestResult:
    """Permutation test of the canonical correlations and factor loadings.

    Refits ``estimator`` on data with the rows of every view but the first
    shuffled independently, which keeps each view's covariance and breaks
    their correspondence. Each permuted fit's loadings are aligned to the
    observed ones by orthogonal Procrustes over all views before comparison,
    since permuted fits can rotate or reflect near-tied dimensions.

    Only the first dimension's p-value is a valid test. Each later dimension
    is compared with the permuted fits' dimension of the same rank, which
    were fitted without removing the earlier dimensions' signal; Winkler et
    al. (2020) test them step-down, residualising each earlier dimension
    first.

    Args:
        estimator: An unfitted multiview estimator.
        views: Arrays of shape (n_samples, n_features_i), one per view.
        n_permutations: Number of permutations. Default is 1000.
        random_state: Seed for the permutations. Default is None.
        n_jobs: Number of parallel jobs. Default is None.

    Returns:
        The observed and null statistics.

    Raises:
        ValueError: If ``n_permutations`` is not positive.

    References:
        Winkler, A. M., Renaud, O., Smith, S. M., & Nichols, T. E. (2020).
        Permutation inference for canonical correlation analysis.
        NeuroImage, 220, 117065.

    Examples:
        >>> import numpy as np
        >>> from cca_zoo.linear import CCA
        >>> from cca_zoo.model_selection import permutation_test_significance
        >>> rng = np.random.default_rng(0)
        >>> z = rng.standard_normal((40, 1))
        >>> X1 = z @ rng.standard_normal((1, 5)) + 0.1 * rng.standard_normal((40, 5))
        >>> X2 = z @ rng.standard_normal((1, 4)) + 0.1 * rng.standard_normal((40, 4))
        >>> result = permutation_test_significance(
        ...     CCA(), [X1, X2], n_permutations=49, random_state=0
        ... )
        >>> float(result.p_values[0])
        0.02
    """
    arrays = validate_views(views)
    n_views = len(arrays)

    fitted = clone(estimator).fit(arrays)
    true_corr, true_loadings = _correlations_and_loadings(fitted, arrays)
    true_stack = np.vstack(true_loadings)  # (sum(n_features_i), k)
    split_points = np.cumsum([loading.shape[0] for loading in true_loadings[:-1]])

    seeds = check_random_state(random_state).randint(
        np.iinfo(np.int32).max, size=n_permutations
    )

    def _one_permutation(seed: int) -> tuple[np.ndarray, np.ndarray]:
        local_rng = np.random.default_rng(seed)
        permuted = [arrays[0]] + [local_rng.permutation(v, axis=0) for v in arrays[1:]]
        model = clone(estimator).fit(permuted)
        corr, loadings = _correlations_and_loadings(model, permuted)
        stack = np.vstack(loadings)
        rotation = scipy.linalg.orthogonal_procrustes(stack, true_stack)[0]
        aligned_stack: np.ndarray = stack @ rotation
        return corr, aligned_stack

    results = Parallel(n_jobs=n_jobs)(delayed(_one_permutation)(seed) for seed in seeds)
    null_correlations = np.stack([corr for corr, _ in results])  # (n_perm, k)
    null_stacks = np.stack([stack for _, stack in results])  # (n_perm, sum_p, k)
    null_loadings = list(np.split(null_stacks, split_points, axis=1))

    p_values = (1 + (null_correlations >= true_corr).sum(axis=0)) / (1 + n_permutations)
    loading_p_values = [
        (1 + (np.abs(null_loadings[i]) >= np.abs(true_loadings[i])).sum(axis=0))
        / (1 + n_permutations)
        for i in range(n_views)
    ]

    return PermutationTestResult(
        correlations=true_corr,
        null_correlations=null_correlations,
        p_values=p_values,
        loadings=true_loadings,
        null_loadings=null_loadings,
        loading_p_values=loading_p_values,
    )

PermutationTestResult `dataclass` ¶

PermutationTestResult(
    correlations: ndarray,
    null_correlations: ndarray,
    p_values: ndarray,
    loadings: list[ndarray],
    null_loadings: list[ndarray],
    loading_p_values: list[ndarray],
)

Result of :func:permutation_test_significance.

Attributes:

Name	Type	Description
`correlations`	`ndarray`	Observed canonical correlation per dimension, shape (k,).
`null_correlations`	`ndarray`	Permuted correlations, shape (n_permutations, k).
`p_values`	`ndarray`	P-value per dimension, shape (k,).
`loadings`	`list[ndarray]`	Observed factor loadings, shape (n_features_i, k) per view.
`null_loadings`	`list[ndarray]`	Permuted loadings aligned to `loadings`, shape (n_permutations, n_features_i, k) per view.
`loading_p_values`	`list[ndarray]`	P-value per feature and dimension, shape (n_features_i, k) per view.

cca_zoo.model_selection¶

GridSearchCV ¶

RandomizedSearchCV ¶

HalvingGridSearchCV ¶

HalvingRandomSearchCV ¶

OptunaSearchCV module-attribute ¶

cross_val_score ¶

cross_validate ¶

cross_val_predict ¶

learning_curve ¶

validation_curve ¶

permutation_test_significance ¶

PermutationTestResult dataclass ¶

OptunaSearchCV `module-attribute` ¶

PermutationTestResult `dataclass` ¶