Input
A measured feature or an upstream neuron activation.
OPEN SOFTWARE · PEER-REVIEWED METHOD
Linear Lens
Inspect what neurons learn and how those meanings flow through a neural network—without perturbing inputs, patching activations, or changing trained parameters during the explanation.
Python · PyTorchNon-interventionalNeuron + layer explanationsGitHub-backed evidence
THE IDEA FIRST
A neuron is understandable before its inputs are summed and transformed.
A dense neuron receives values, multiplies each by a learned weight, adds them, then applies a nonlinear activation. Linear Lens looks at the feature × weight contributions in the original forward computation, summarizes their distribution, and assigns a layer-relative functional role.
Contribution
How strongly that input enters a particular neuron before summation.
Concentration
Entropy asks whether influence is focused on one source or spread across several.
Meaning
Specialized, distributed, diffuse, or—deeper in the network—composed from earlier meanings.
Non-interventional means observation, not manipulation.
The explanation is computed from the model's existing weights, inputs, pre-activations, and activations. Linear Lens does not need to replace activations or perturb the model to discover these neuron/layer roles.
Input representation matters.
Linear Lens explains the processed values that actually enter the network. If the trained model receives standardized features, the explanation is in that standardized representation. If one feature enters on a radically different raw scale, its feature×weight contribution changes accordingly. This is why preprocessing provenance is shown with every browser model.
FORMULA, STEP BY STEP
Every symbol has a visible purpose.
The first five steps describe a first hidden layer. The same influence → entropy → layer-relative decision pattern can then be applied to deeper layers using the previous layer's activations as inputs.
Read the neuron's pre-activation
xi,j is feature j for sample i; wk,j is the learned connection from feature j to neuron k; bk is the neuron's bias; zi,k is the value before the activation function.
Purpose: stay at a directly traceable point in the model's actual computation.
Measure feature-to-neuron influence
μk,j is the average absolute contribution of input j to neuron k over N analyzed samples. Absolute magnitude prevents positive and negative contributions from cancelling each other in this global role summary.
Purpose: turn raw feature-weight interactions into comparable evidence for each neuron.
Normalize influence into a distribution
pk,j is the share of neuron k's total measured influence assigned to input j. ε is a tiny stability constant. Each neuron's influence row sums to approximately 1.
Purpose: express “what this neuron listens to” as a probability-style profile.
Ask whether the profile is focused or diffuse
Hk is Shannon entropy. Low entropy means influence is concentrated; higher entropy means it is distributed more evenly.
Purpose: reduce a full influence profile to one concentration statistic without changing the model.
Interpret entropy relative to the layer
H̄ is mean neuron entropy in the layer and sH is its standard deviation. The paper uses a one-tailed 90% standard-normal decision region; c = Φ−1(0.90) ≈ 1.282.
Zk < −c → monosemantic−c ≤ Zk ≤ c → polysemanticZk > c → dead/flat
Purpose: make the labels layer-relative rather than pretending one absolute entropy threshold fits every architecture.
Validate distributed neurons
For a polysemantic neuron, Sk is its selected influential feature set. The paper illustrates selection with normalized influence pk,j > 0.15; the browser demonstrator uses that illustrated threshold. A regression model tests whether the selected set collectively explains the observed pre-activation. R² is validation evidence, not a new neural-network prediction.
Summarize the whole layer with QSM
QSM = neurons × semantic inputs
The Qualitative Symbolic Matrix converts dense numeric influence into a human-readable layer fingerprint. This website keeps the exact numeric influence available while using text labels and strength bands for accessibility.
Trace deeper meanings toward the output
raw features → L1 roles → L2 roles → … → output
In deeper layers, raw features are replaced by upstream activations. A focused deeper unit is unimodal; a distributed unit is multimodal; a diffuse high-entropy unit is muted. The final output can be described as a composition of previously traced semantic paths.
Symbol guide — what does every symbol mean?
isample index
j, rinput-feature or upstream-unit indices
kneuron index
Nnumber of analyzed samples
dnumber of inputs to the layer
xi,jinput j for sample i
wk,jlearned weight from input j to neuron k
bkbias of neuron k
zi,kpre-activation before nonlinearity
μk,jmean absolute contribution
pk,jnormalized influence share
Hkentropy / influence dispersion
H̄mean entropy of the layer
sHentropy standard deviation
Zklayer-relative entropy z-score
Φ−1inverse standard-normal CDF
εsmall numerical-stability constant
R²regression validation goodness-of-fit
ROLE DICTIONARY
What do the labels actually mean?
Role names are shorthand for observed influence structure. They should not be interpreted more strongly than the evidence supports.
One source dominates
Influence is unusually concentrated relative to other neurons in the same first layer.
Several sources matter
Influence is distributed across multiple raw features. The feature set can be regression-validated.
No clear semantic concentration
Important: this does not necessarily mean an inactive or useless neuron. In Linear Lens it denotes an unusually diffuse influence profile under the analyzed data.
One upstream meaning dominates
A deeper neuron is primarily driven by one previously explained semantic path.
Several upstream meanings combine
A deeper neuron integrates multiple earlier units or semantic pathways.
No focused upstream source
The deeper neuron's influence is comparatively diffuse under the observed activation distribution.
LIVE GITHUB VERIFICATION
Fetch a dataset and trained model. Recompute Linear Lens here.
The browser downloads a CSV and exported dense model directly from udanish50/LinearLens, verifies both SHA-256 hashes against the repository manifest, applies the recorded training preprocessing, and recomputes neuron roles without a server-side analysis API.
45public verification datasets
180trained-model analyses
9,720archived neuron records
36controlled recovery scenarios
1 · Choose evidence
Six canonical scikit-learn datasets plus 39 deterministic controlled fixtures.
No dataset is silently substituted with local page data.
Waiting for the public manifest.
2 · Read the model
Start with the summary, then inspect neurons, QSM, and one semantic path.
Choose a datasetThe result will show hash provenance, model performance, layer roles, dominant inputs, QSM, and a trace toward the output.
ANALYZE YOUR OWN MODEL
A dataset alone is not enough—you also need the trained model being explained.
Linear Lens analyzes internal neural computation, so this browser workspace asks for both the numeric CSV and a supported model JSON exported from the Python package. Both files remain on this device.
LOCAL WORKSPACENo uploaded CSV or model JSON is sent to umairdanish.com by the Linear Lens analyzer.
1 · Load files
Dense feed-forward browser support; use Python adapters for broader architectures.
from linearlens.web import export_dense_model
export_dense_model(model, "model.json", feature_names=features)Load both files to begin.
2 · Explanation report
Global layer roles plus exportable neuron evidence.
Ready for your modelAfter analysis, download the neuron table, QSM-compatible matrix, and JSON report.
PUBLIC SOFTWARE EVIDENCE
Many datasets, several checks—and the negative cases stay visible.
The public suite is additional software verification. It does not replace the paper's original ten confidential energy datasets.
Evidence tables load only when requested.
PAPEROriginal peer-reviewed evaluation
Published article ↗Ten real-world energy consumption datasets; MLP, LSTM and Transformer architectures; statistically validated user study with N = 400. The publication reports the underlying energy data as confidential.
METHOD COMPARISON
Do not force unlike explanation methods into one fake leaderboard.
Linear Lens explains neuron/layer organization. Local attribution, perturbation importance, activation patching, and learned dictionaries answer different questions. The comparison below makes those scopes visible instead of implying that one scalar score establishes universal superiority. In the live model result, the same verified network is also shown through a local Gradient × Input view and global permutation importance when a target column is available.
| Method | Main question | Changes model/inputs for explanation? | Typical output | Best viewed as |
|---|---|---|---|---|
| Linear Lens | What role does each neuron/layer play, and how do meanings propagate? | No intervention in the explained forward computation | Influence profiles, entropy roles, QSM, semantic pathways | Mechanistic / behavioral audit |
| Gradient × Input | Which input dimensions locally affect this output? | No parameter change; uses derivatives at the sample | Local attribution vector | Local sensitivity attribution |
| Integrated Gradients | How does attribution accumulate from a reference to this input? | Evaluates a path of interpolated inputs | Local attribution vector | Baseline-dependent attribution |
| Permutation importance | How much does predictive performance drop when a feature is disrupted? | Yes—permutes input values | Global feature importance | Performance-based importance |
| Activation patching / causal tracing | Which internal components causally affect a behavior under an intervention? | Yes—replaces/patches internal activations | Causal component/path evidence | Interventional mechanistic test |
| Sparse autoencoder feature analysis | Can activations be decomposed into a sparse learned feature dictionary? | Trains an auxiliary representation model | Learned sparse features | Representation decomposition |
Choose based on the decision you need to make.
If you need a local feature attribution, use an attribution method. If you need to inspect neuron specialization, distributed representations, layer organization, and semantic propagation without altering the explained computation, Linear Lens is designed for that question.
HUMAN-CENTERED INTERACTION
The interface is designed to teach before it asks for trust.
These are implementation choices aligned with accessible/usability principles—not a claim of formal accessibility certification.
Progressive disclosure
Mental model first, equations second, live evidence third, raw neuron tables only on demand.
Recognition over recall
Every role and symbol is defined where it first appears and again in compact references.
Visible system status
Downloads, hash checks, parsing, analysis, and evidence loading report state through aria-live regions.
No color-only meaning
Role names and text labels accompany visual accents; tables remain interpretable without color.
Evidence provenance
LIVE, ARCHIVED, and PAPER evidence are visibly separated so visitors know what was computed now versus previously reported.
Error prevention
The upload workflow explains why both data and a trained model are required and rejects mismatched feature schemas before analysis.
SOURCE & REPRODUCIBILITY
Inspect the implementation—not only this interface.
The repository contains the PyTorch package, tests, architecture adapters, method documentation, public evidence suite, exported browser models, reproducibility material, and citation metadata.