Emergent Sparsity in Frozen Random CNN Feature Extractors for Deep Reinforcement Learning

Computer Science > Machine Learning
[Submitted on 15 May 2026]
Title:Emergent Sparsity in Frozen Random CNN Feature Extractors for Deep Reinforcement Learning
View PDF HTML (experimental)Abstract:We report a striking phenomenon: deep reinforcement learning agents trained with frozen, randomly initialized CNN feature extractors spontaneously develop extremely sparse fully-connected representations, without any sparsity-inducing objective. In the first fully-connected layer (FC1, $3{,}136 \to 64$), agents compress task-relevant information through as few as 1-3 neurons out of 64 for deterministic Pong (5-11 for stochastic Pong), while trainable CNNs activate 55-64 neurons under matched conditions. We establish four principal findings. First, FC1 sparsity scales with task complexity: 1-11 for Pong, 19-26 for Breakout, and $\sim$42 for Space Invaders. Width-scaling confirms this reflects task structure rather than a fixed capacity fraction. Second, within-game scaling emerges: three identical Pong seeds produce 5, 7, and 11 active neurons. The 5-neuron seed plateaus at $+14$ reward, while the others reach expert performance ($+18.4$, $+18.7$), suggesting the random projection's usable dimensionality bounds achievable performance. Third, ablation confirms necessity: removing these active neurons crashes performance across two PPO implementations and four games. Fourth, the information bottleneck commits early: a sweep shows the active set locks by 15-30M steps, while reward turns positive 35-105M steps later. A complementary finding in Breakout shows frozen and trainable CNNs reach competitive rewards via structurally different bottlenecks: frozen agents use 17-25 active neurons (participation ratio $\sim$10-14), while trainable agents use 51 (participation ratio $\sim$3.6). Finally, wherever input dimensionality dwarfs intrinsic task dimensionality, gradient descent on a frozen random projection may reveal the effective rank of the underlying problem without explicit sparsity machinery.
Current browse context:
cs.LG
References & Citations
Loading...
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
IArxiv Recommender
(What is IArxiv?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Facts Only

* Deep reinforcement learning agents developed sparse fully-connected representations with frozen, randomly initialized CNN feature extractors.
* For deterministic Pong, FC1 compressed information using 1-3 neurons out of 64; for stochastic Pong, this was 5-11 neurons.
* Trainable CNNs activated 55-64 neurons under matched conditions.
* FC1 sparsity scaled with task complexity: 1-11 for Pong, 19-26 for Breakout, and $\sim$42 for Space Invaders.
* Three identical Pong seeds resulted in active neurons of 5, 7, or 11.
* The 5-neuron seed plateaued at a $+14$ reward.
* Removing active neurons caused performance crashes across two PPO implementations and four games.
* Frozen agents reached competitive rewards in Breakout using 17-25 active neurons (participation ratio $\sim$10-14).
* Trainable agents used 51 active neurons in Breakout (participation ratio $\sim$3.6).
* Gradient descent on frozen random projection may reveal the effective rank of the problem without explicit sparsity machinery.

Executive Summary

Deep reinforcement learning agents trained using frozen, randomly initialized CNN feature extractors spontaneously develop extremely sparse fully-connected representations without explicit sparsity objectives. In the first fully-connected layer (FC1), agents compress task information effectively with few neurons; for deterministic Pong, this is 1-3 out of 64, and for stochastic Pong, it is 5-11 out of 64. This contrasts with trainable CNNs, which activate 55-64 neurons under matched conditions. The observed sparsity scales with task complexity: 1-11 neurons for Pong, 19-26 for Breakout, and approximately 42 for Space Invaders. Furthermore, specific game seeds exhibited emergent scaling, where three identical Pong seeds produced active neuron counts of 5, 7, or 11, with the 5-neuron seed correlating to a $+14$ reward, while other seeds reached expert performance ($+18.4$ and $+18.7$). Ablating these active neurons resulted in performance crashes across different PPO implementations and games. The information bottleneck also shows differential effects: frozen agents utilize 17-25 active neurons (participation ratio $\sim$10-14) in Breakout, while trainable agents use 51 (participation ratio $\sim$3.6).

Full Take

The research demonstrates that information bottlenecks can emerge naturally through training dynamics when regularization is absent, suggesting that latent structure within the randomly initialized feature space dictates achievable representations. The contrast between the highly sparse, task-specific projections learned by frozen agents and the denser activations in trainable models points toward a fundamental difference in how representation capacity is utilized: one optimizes for minimal necessary information transfer dictated by the reward landscape, while the other optimizes for general feature extraction within a fixed dimension. The emergence of specific neuron counts tied to game outcomes suggests that the random initialization acts as an implicit mechanism guiding the learning process toward the most efficient latent manifold relevant to the task, effectively revealing the intrinsic dimensionality without manual constraint. This implies that sparsity is not merely a regularization technique but an emergent property linked to the underlying complexity of the problem itself. The findings suggest a shift in perspective: instead of imposing external sparsity mechanisms, researchers might investigate how task structure inherently sculpts feature representations when learning from random initializations.

Sentinel — Human

Confidence

This text exhibits strong characteristics of a genuine technical research abstract, demonstrating focused argumentation and specific empirical findings typical of academic machine learning literature.

Signals Detected

High density of specialized technical terminology and precise mathematical notation.

The flow is highly logical, moving from observation to scaling to ablation, characteristic of a formal experimental summary.

The structure follows standard academic reporting (Abstract/Findings), suggesting adherence to established ML paper conventions.

The specific, granular findings (e.g., 1-3 neurons for Pong, 55-64 neurons for CNNs) appear highly specific and technical, typical of genuine experimental results.

Human Indicators

The use of specific, context-dependent scaling relationships (e.g., '1-11 for Pong', 'active set locks by 15-30M steps') suggests direct reporting from an experimental run rather than generalized synthetic statement.

The juxtaposition of results across different tasks (Pong, Breakout, Space Invaders) demonstrates the kind of deep comparative analysis a human researcher performs.