Linear Methods¶

All linear CCA methods in cca_zoo.linear share the same fit/transform/score interface and are sklearn.base.BaseEstimator subclasses. They find linear projections of the input views.

Two-view methods¶

These methods operate on exactly two views.

CCA — Standard Canonical Correlation Analysis¶

When to use: The default choice for two balanced, moderately-sized views.

CCA finds directions \(\mathbf{w}_1, \mathbf{w}_2\) that maximise the Pearson correlation between projected views:

\[ \max_{\mathbf{w}_1, \mathbf{w}_2} \; \mathbf{w}_1^\top X_1^\top X_2 \mathbf{w}_2 \quad \text{s.t.} \quad \mathbf{w}_i^\top X_i^\top X_i \mathbf{w}_i = 1 \]

The solution uses PCA whitening followed by an SVD of the cross-covariance matrix, giving a numerically stable result even for high-dimensional views.

from cca_zoo.linear import CCA

model = CCA(n_components=2).fit([X1, X2])
z1, z2 = model.transform([X1, X2])
print(model.score([X1, X2]))  # mean canonical correlation

RidgeCCA — Regularised CCA¶

When to use: CCA breaks down when \(n < p\) (more features than samples). RidgeCCA adds a ridge penalty to stabilise the covariance matrices.

The parameter shrinkage controls the regularisation strength:

shrinkage=0 → equivalent to CCA
shrinkage=1 → equivalent to PLS
0 < c < 1 → interpolates between the two

from cca_zoo.linear import RidgeCCA

model = RidgeCCA(n_components=2, shrinkage=0.1).fit([X1, X2])

PLS — Partial Least Squares¶

When to use: When you want to maximise covariance rather than correlation. PLS is more robust to noise and does not require invertible covariance matrices.

PLS is a special case of RidgeCCA with shrinkage=1:

\[ \max_{\mathbf{w}_1, \mathbf{w}_2} \; \mathbf{w}_1^\top X_1^\top X_2 \mathbf{w}_2 \quad \text{s.t.} \quad \|\mathbf{w}_i\|_2 = 1 \]

from cca_zoo.linear import PLS

model = PLS(n_components=2).fit([X1, X2])

Multiview methods (≥2 views)¶

These methods generalise CCA to three or more views.

MCCA — Multiset CCA¶

When to use: Multiple views, interpretable pairwise-sum objective.

MCCA maximises the sum of pairwise correlations across all view pairs. It solves a generalised eigenvalue problem on block matrices:

\[ A \mathbf{v} = \lambda B \mathbf{v} \]

where \(A\) contains the cross-view covariances and \(B\) the regularised within-view covariances.

from cca_zoo.linear import MCCA

model = MCCA(n_components=2, shrinkage=0.1).fit([X1, X2, X3])

GCCA — Generalised CCA¶

When to use: Multiple views with potentially different numbers of features; best when you want a single shared low-dimensional representation.

GCCA finds a common latent variable \(G\) (of shape \(n \times k\)) such that each view can be reconstructed from it:

\[ \min_{G, W_i} \sum_{i} \mu_i \|X_i W_i - G\|_F^2 \]

This is equivalent to maximising the sum of squared canonical correlations between each view and the shared projection.

from cca_zoo.linear import GCCA

model = GCCA(n_components=2, shrinkage=0.01).fit([X1, X2, X3])

TCCA — Tensor CCA¶

When to use: When you want to capture higher-order (beyond pairwise) correlations among multiple views.

TCCA builds a joint cross-moment tensor of the whitened views and finds its best rank-\(k\) PARAFAC decomposition:

\[ T = \frac{1}{n} \sum_{j} \tilde{\mathbf{x}}^{(1)}_j \otimes \cdots \otimes \tilde{\mathbf{x}}^{(M)}_j \]

from cca_zoo.linear import TCCA

model = TCCA(n_components=2, shrinkage=0.01, random_state=0).fit([X1, X2, X3])

EY-loss methods¶

These methods replace the full eigendecomposition with the unconstrained Eckart-Young (EY) objective (see cca_zoo.tree and cca_zoo.deep for the same objective applied to tree and neural-network encoders), making them practical for very high-dimensional data. No manifold projection step and no upfront whitening pass over the full dataset are needed: the EY loss's quadratic penalty term drives the weights towards the canonical directions on its own.

Class	Description
`PLSEY`	Eckart-Young PLS objective, full-batch L-BFGS-B
`CCAEY`	Eckart-Young CCA for 2 or more views, full-batch L-BFGS-B, ridge-blended with `PLSEY` via `shrinkage`
`HuberCCA`	Bounded-influence (Huber-style) EY-CCA, full-batch L-BFGS-B

For datasets too large to fit comfortably in memory, see cca_zoo.stochastic.StochasticCCAEY, which fits the same objective as CCAEY with mini-batch SGD instead.

CCAEY's shrinkage parameter (default 0) blends its loss towards PLSEY's (shrinkage=1) — in fact PLSEY is implemented as CCAEY with shrinkage fixed at 1. Optimising the raw, unregularised (shrinkage=0) objective can be poorly conditioned when the number of samples doesn't outnumber the number of features by a healthy margin; if you see nan weights, increase shrinkage (0.1-0.3 is usually enough).

from cca_zoo.linear import CCAEY

model = CCAEY(n_components=2, max_iter=200)
model.fit([X1, X2])

HuberCCA — bounded-influence EY-CCA¶

CCAEY's cross- and auto-covariance statistics weight every sample equally, so a handful of high-leverage points (their contribution grows with the square of their magnitude) can hijack the fit. HuberCCA reweights each sample by a Huber-style factor of its own leverage before forming those statistics — the same bounded-influence mechanism sklearn.linear_model.HuberRegressor uses against outliers, applied to the EY loss's own statistics. Every sample still contributes something (smooth downweighting, never exactly zero). Fit by the same full-batch L-BFGS-B as CCAEY, so it shares that class's nan-on ill-conditioned-data caveat above; it has no ridge-blend shrinkage of its own. The delta parameter sets the cutoff as a multiple of the dataset's own median sample leverage (self-calibrating, so it doesn't need re-tuning per n_components); values below 1 downweight the majority of the data and are not recommended.

from cca_zoo.linear import HuberCCA

model = HuberCCA(n_components=2, delta=4.0, max_iter=200)
model.fit([X1, X2])

Robust methods¶

HuberCCA above guards against high-leverage contamination: points whose combined magnitude across views dominates the covariance statistics simply by being large. That leaves a different failure mode untouched — a subset of rows whose cross-view relationship is wrong (mismatched, corrupted, or drawn from an unrelated pattern) while remaining completely ordinary in magnitude within each view. Nothing about such a row's norm flags it as unusual, so leverage-based reweighting can't see it, and in practice can even make things slightly worse.

RANSACCCA is the multiview-CCA analogue of sklearn.linear_model.RANSACRegressor: it repeatedly fits a fast closed-form MCCA on a random subset of rows, scores each candidate by how much of the full dataset actually agrees with it, and keeps the best-supported candidate's consensus set for a final refit. Since it measures agreement with a candidate direction rather than raw magnitude, it catches exactly the contamination HuberCCA can't:

from cca_zoo.linear import RANSACCCA

model = RANSACCCA(n_components=2, min_samples=0.25, random_state=0)
model.fit([X1, X2])
inliers = model.inlier_mask_  # boolean array over the training rows

min_samples (a fraction or an absolute count) trades off two things: smaller subsets are more likely to be drawn free of contamination, but need shrinkage (a small ridge, default 0.1) to stay numerically well-posed. residual_threshold defaults to 0 — the natural zero point of the per-sample agreement score under no real relationship — rather than anything estimated from the data. Like MCCA, this isn't convex, and the random subset draws add their own instability on top: when the "wrong" relationship is supported by close to half the data, different random_state seeds can land on different consensus sets — see the class's own tests for a worked example of where this helps and where the problem becomes too ambiguous for any method to resolve reliably.

RANSACCCA's random-subset search is exactly where it struggles too: near the ~50% contamination breakdown point, a random min_samples-sized draw becomes close to a coin flip on being usably clean, however many trials are tried. TrimmedCCA takes a different approach borrowed from Rousseeuw's Least Trimmed Squares / Minimum Covariance Determinant: rather than gambling on a lucky small draw, it starts from a large random subset of support_fraction * n rows and alternates concentration steps — rank every row by its own contribution to CCAEY's exact loss and keep the best h, then re-fit on exactly those rows — each step provably non-increasing in the real loss. With support_fraction set close to the true clean fraction, this holds up where RANSACCCA's search degrades:

from cca_zoo.linear import TrimmedCCA

model = TrimmedCCA(support_fraction=0.55, n_init=40, random_state=0)
model.fit([X1, X2])
inliers = model.inlier_mask_  # boolean array over the training rows

support_fraction is a prior on the contamination rate, not something fit from the data — set it too high and good rows get discarded for nothing; set it too low and contaminated rows get forced into every fit once true contamination exceeds 1 - support_fraction. TrimmedCCA supports any number of views (2 or more) but only n_components=1: the selection rule's closed-form derivation relies on CCAEY's penalty being the square of a single linear functional of the selection, which holds for any number of views but not past one latent dimension — with k > 1 the same penalty becomes a genuine matrix-valued quadratic form (rank up to k(k+1)/2) that the same single-multiplier bisection can't solve. Away from the ~50% breakdown regime, or when more than one latent dimension is needed, RANSACCCA matches or beats it directly.

HuberCCA, RANSACCCA and TrimmedCCA all still route through a covariance (or covariance-like per-sample loss) statistic of the projected views. ProjectionPursuitCCA does not: it searches directly over candidate unit-norm projection directions for the pair maximising a robust bivariate correlation measure (the projection index) between the resulting scores, never forming a covariance matrix at all.

from cca_zoo.linear import ProjectionPursuitCCA

model = ProjectionPursuitCCA(n_components=2, random_state=0)
model.fit([X1, X2])

The projection index is Spearman's rank correlation, insensitive to an outlier's exact magnitude, only its rank. Each direction is parametrised by unconstrained hyperspherical angles and found by derivative-free search (n_init random restarts of scipy.optimize.minimize with Powell's method — a rank-correlation-based objective is non-smooth, so gradient-based solvers don't apply). Unlike the other three estimators here, there is no inlier_mask_: its robustness comes from the projection index's own insensitivity to extreme values, not from singling out individual bad rows, so it has nothing analogous to report.

Related work. Robust CCA is a much older problem than these estimators: Filzmoser, Dehon & Croux (2000) and Branco, Croux, Filzmoser & Oliveira (2005) already reweight an alternating-regression CCA fit by robust-distance-based weights ("Robust Alternating Regression"), the same bounded-influence idea HuberCCA applies to the EY loss's own statistics instead; Croux & Dehon (2002) plug the minimum covariance determinant estimator into the classical covariance-matrix eigenproblem, the same high-breakdown idea TrimmedCCA applies via concentration steps on the EY loss directly instead. Branco et al. (2005)'s comparative study also covers the projection-pursuit paradigm ProjectionPursuitCCA implements here (maximising a robust bivariate correlation measure directly over projection directions), tracing back to Huber's original projection pursuit (1985). Alfons, Croux & Filzmoser (2017) put the same paradigm on firmer statistical footing (efficiency, breakdown point, a wider family of projection indices) and ship it as the R package ccaPP (Alfons, Croux & Filzmoser, 2016), the reference implementation ProjectionPursuitCCA's Spearman projection index follows.

Filzmoser, P., Dehon, C., & Croux, C. (2000). Outlier resistant estimators for canonical correlation analysis. In COMPSTAT: Proceedings in Computational Statistics 2000 (pp. 301-306). Physica-Verlag.

Croux, C., & Dehon, C. (2002). Analyse canonique basee sur des estimateurs robustes de la matrice de covariance. Revue de Statistique Appliquee, 50(2), 5-26.

Branco, J. A., Croux, C., Filzmoser, P., & Oliveira, M. R. (2005). Robust canonical correlations: A comparative study. Computational Statistics, 20(2), 203-229.

Huber, P. J. (1985). Projection pursuit. The Annals of Statistics, 13(2), 435-475.

Alfons, A., Croux, C., & Filzmoser, P. (2017). Robust maximum association estimators. Journal of the American Statistical Association, 112(517), 436-445.

Alfons, A., Croux, C., & Filzmoser, P. (2016). Robust maximum association between data sets: The R package ccaPP. Austrian Journal of Statistics, 45(1), 71-79.

GPUs and the Array API¶

CCA, RidgeCCA, PLS, MCCA and GCCA compute in the namespace of their inputs under scikit-learn's Array API dispatch, so PyTorch or CuPy arrays on a GPU are fitted and transformed on the GPU. Their cost is the covariance products and one eigendecomposition, both of which a GPU accelerates. scipy must be told before it is imported:

import os

os.environ["SCIPY_ARRAY_API"] = "1"

import sklearn
import torch
from cca_zoo.linear import MCCA

views = [torch.asarray(X1, device="cuda"), torch.asarray(X2, device="cuda")]
with sklearn.config_context(array_api_dispatch=True):
    model = MCCA(n_components=2, shrinkage=0.1).fit(views)
    z1, z2 = model.transform(views)  # torch tensors on the GPU

The other models compute with numpy and raise a TypeError on other arrays under dispatch.

Choosing a method¶

Scenario	Recommended
\(n \gg p\), two views	`CCA`
\(n < p\) or ill-conditioned	`RidgeCCA` (tune `shrinkage`)
Maximise covariance, not correlation	`PLS`
Three or more views	`MCCA` or `GCCA`
Higher-order cross-view structure	`TCCA`
Sparse weights needed	`cca_zoo.sparse.PMDCCA` or `IPLSCCA`
Very large \(p\)	`CCAEY`, `PLSEY`
Dataset too large for full-batch gradients	`cca_zoo.stochastic.StochasticCCAEY`
A few high-magnitude outlier samples	`HuberCCA`
A subset of rows with a wrong (mismatched/corrupted) relationship	`RANSACCCA`
Heavy contamination (near ~50%), with a known contamination-rate prior	`TrimmedCCA`
Robustness without ever forming a covariance matrix	`ProjectionPursuitCCA`
Nonlinear relationships	See Nonparametric Methods