NN7: Blind Separation: Discovering Modular Decompositions Without Target Labels

The Energy Separation Theorem guarantees that when a system decomposes into independent degrees of freedom, the loss can separate exactly across modules. The separable architectures paper demonstrated this principle when target labels for each module are known—a setting called supervised separation. But in most real-world tasks, the decomposition into subtasks is not given. The modules exist, but we do not know which module should learn what. This paper addresses the blind separation problem: discovering the modular decomposition from data alone, without target labels for individual modules. The primes needed no algorithm—their independence is built into the fundamental theorem of arithmetic. Our networks are not so fortunate. What this paper provides: · A formal definition of the blind separation problem. Given input-output pairs (x, y) where y = \sum_i f_i(x) for unknown component functions f_i, recover the decomposition—assign each component to a module and train each module on its assigned component—without access to the individual f_i(x) values during training. The constraint of vanishing cross-derivatives \partial^2 \mathcal{L} / \partial W_i \partial W_k = 0 for i \neq k is the operational definition of independence.· Four algorithms for blind separation, each derived from the principle that modules should learn independent, additive contributions: 1. Data-Driven Clustering: Partition the input space into regions where different components dominate. Each module trains on data from its assigned region. Works when components are spatially localized; fails for global functions. 2. Functional Spectral Decomposition: Diagonalize the interaction matrix between module gradients and assign modules to orthogonal subspaces. Requires computable and diagonalizable interaction matrix; computationally expensive for large models. 3. Emergent Specialization: Initialize modules identically and add a repulsive regularization term \sum_{i \neq k} \|W_i - W_k\|_F^{-1} that penalizes similarity. Pushes modules toward orthogonal functions without explicit target assignment. Sensitive to hyperparameter \lambda and requires joint training. 4. Residual Decomposition: Train modules sequentially. Module 1 learns the full target y. Module 2 learns the residual y - g_1(x). Module 3 learns y - g_1(x) - g_2(x). Each module learns the component of the target that previous modules could not capture. Cross-derivatives vanish by construction. No gradient communication between modules.· Benchmark on a compositional regression task where the target is the sum of four unknown component functions (\sin(2\pi x_1) + \cos(2\pi x_2) + \sigma(x_3) + 0.5x_4^2). Residual decomposition achieves component correlation of 0.92 \pm 0.05 (vs 0.62 \pm 0.18 for clustering, 0.71 \pm 0.15 for spectral, 0.78 \pm 0.12 for emergent). Cross-derivatives are identically zero for residual decomposition; other algorithms show nonzero cross-terms.· Benchmark on multi-task MNIST without task labels. The network receives only input images and three binary targets, without knowing which target corresponds to digit classification, even/odd, or greater-than-4. Residual decomposition achieves 97.8%, 97.1%, and 95.8% accuracy on the three tasks (vs supervised upper bound of 98.2%, 97.9%, 96.6% and single shared network baseline of 97.5%, 96.8%, 95.2%). Modules naturally specialize: the easiest task is learned first, then the residual, then the hardest.· Analysis of why residual decomposition works. It imposes a sequential order on learning. The greedy strategy does not guarantee recovery of the true decomposition—but it guarantees additive separation, requires no communication between modules, and works well in practice when component functions have different magnitudes or complexities. The order matters; the modules are not learning the "true" components in any particular order, but a valid additive decomposition.· Open challenges: Permutation invariance (true decomposition is unique only up to permutation; developing permutation-invariant evaluation metrics is open), unknown number of modules K (must be chosen or discovered), and non-additive compositions (where components combine multiplicatively or through more complex interactions remain an open frontier). Why this matters: The Energy Separation Theorem guarantees that a decomposition exists. The primes are the existence proof. The blind separation problem is to find it. Residual decomposition is the bridge between the mathematical guarantee and the engineering reality: sequential training on residuals, zero cross-derivatives, no communication overhead, and near-supervised accuracy without task labels. Keywords: blind separation, modular decomposition, Energy Separation Theorem, residual decomposition, additive components, vanishing cross-derivatives, compositional regression, multi-task learning, permutation invariance, independent modules, Canvas Model