Introduction

I often use LLMs, like ChatGPT or Claude, for getting around new cities and countries. I've found these models to be really effective for restaurant suggestions, walking routes, and even planning full itineraries, and I'm often left wondering how they internally understand space and spatial relationships. In this post, I investigate one part of this puzzle: space neurons.

Prior work has found that geographic coordinates can be recovered with linear probes from the hidden representations of language models and that there are even individual neurons whose activations vary geographically (Gurnee and Tegmark, 2024).

I set out to reproduce the key results with the open-weight model Qwen3.6-27B (4-bit quantized version), then test whether the identified space neurons generalize to new places and remain stable across prompt variations.

Linear representation hypothesis

A well-known hypothesis in interpretability research, the linear representation hypothesis (see Mikolov, Yih, and Zweig (2013)) states that high-level concepts, like sentiment, are encoded as directions in the activation space of an LLM and that these features can be extracted from the activations with a linear map. The process of training a small model for predicting a label from the activations is called probing.

I use two spatial datasets by Gurnee and Tegmark for training and evaluating a simple probe on the activations of Qwen3.6-27B. The two datasets contain names and coordinates of 39,585 and 29,997 places worldwide and in the United States, respectively. These datasets are further split into training (80%) and test (20%). To extract model activations, I feed the model the prompt "What are the lat/lon coordinates of {name}" for each place in the dataset. Then, a ridge regression model is fit to predict latitude and longitude. Finally, the probe is evaluated on the test set.

Qwen3.6-27B linear probes (layer 38): true vs predicted coordinates

The final probes perform well on both test sets. At layer 38 of 64, mean test-set $R^2$ is about $0.86$ for world places and $0.76$ for US places, which is consistent with Gurnee and Tegmark's findings for Llama models. This shows that coordinate information is, indeed, linearly decodable from the activations.

Probe performance by layer

A linear probe is fit on the activations of a specific layer in the model, and its performance varies across layers. Above, I picked a sufficiently deep layer at 60% of the model depth (38/64). By training probes for all layers in the model, I can investigate how linearly decodable coordinate information changes at different depths.

Probe R² by layer depth on the world and US datasets

Linearly decodable spatial information appears early and then plateaus on both sets. Unsurprisingly, the first few layers seem to contain less of this information. On the world set, the best-performing layer is near the end (layer 58, $R^2 \approx 0.88$), while on the US dataset performance peaks earlier (layer 24, $R^2 \approx 0.76$).

These results show that coordinate information is linearly accessible in the model's activations, but the linear probe combines all 5,120 dimensions of a layer. Is it possible to find individual neurons whose activations carry spatial information?

Space Neurons

In their 2024 paper, Gurnee and Tegmark introduced space neurons, individual nodes in Llama-2 family models whose preactivations track certain spatial properties. Even though they did not find a single neuron tracking global latitude or longitude, they found neurons for more specific features like “US East Coast longitude” or “negative global latitude.”

Here, I use space neuron to mean an individual gate or up-projection preactivation channel in one of Qwen's feed-forward layers. Qwen3.6-27B is a 64-layer model with a mix of Gated DeltaNet and full-attention layers, plus a dense 17,408-channel SwiGLU FFN at every layer.

I scanned Qwen's activations for several spatial patterns: continents, hemispheres, signed and absolute coordinates, East and West Coast subsets, and others, scoring each pattern only on relevant places. For the neuron search, I repartitioned the datasets into 50/50 training and test sets, separate from the 80/20 split used for the linear probes.

At the final token of the prompt, I evaluated both the gate and up-projection preactivation for every channel in every layer, about 2.2 million possibilities for each target. The candidates were then ranked by the absolute Spearman correlation between preactivation and the target on the training places, and finally evaluated on the test set.

Neurons selected on the training set and evaluated on the test set

The scan identified many neurons that have a high correlation with a specific spatial feature on both the training and test sets. I hand-picked four interesting and well-performing neurons from the results (shown above). On the world dataset, absolute latitude and longitude reach correlation values of $0.81$ and $0.76$. On the US set, overall longitude is about $0.72$ and East Coast longitude about $0.73$.

The plots are noisier than the earlier linear probes, which is expected—and consistent with Gurnee and Tegmark's findings on the Llama-2 family models. These results show that the Qwen3.6-27B model, too, contains certain neurons whose activations strongly correlate with geographic features.

Visual verification

To further analyze the results, I plotted these four neurons' activations on the test set. The activations are standardized using the results from the training set, so the test set does not influence neuron selection or the color scale.

Selected neuron activations across test places

The plotted activations have a strong spatial correlation across the map with a noticeably small number of outliers. Some discontinuous jumps align with country and state borders, suggesting these activations may mix continuous coordinate data with other discrete information. Overall, the plotted activations show that generally the predictions hold up well.

Prompt variations

The original prompt ("What are the lat/lon coordinates of {name}") might actively cause the model to echo coordinates learned in pretraining. To test that, I evaluated the same neurons using two additional prompts: "Tell me about {name}" and "I recently travelled to {name}". I then stored the neuron activations at the final token of the prompt for each data point in the test set.

Geographic correlation and activation-rank agreement across prompts

All four neurons continue to correlate strongly with their respective geographic features using the new prompts, with correlations between $0.64$ and $0.81$. Across the dataset, Spearman correlations between each neuron's alternate-prompt and original-prompt activations range from $0.77$ to $0.96$. Together, these results suggest that the observed correlations are robust to prompt changes and not merely an artifact of asking for coordinates.

Further work

I found that spatial information is decodable from the model's activations, but this does not show that the model uses this information during generation. Interventions could test whether the identified neurons and directions are necessary or sufficient for spatial behavior, for example, by steering or ablating them and measuring changes in model output (Tehenan et al., 2025)—similar to Gurnee and Tegmark's work on Llama-2.

The analyses here used linear probes and individual-neuron correlations. Recent work on superposition (Elhage et al., 2022) argues that models compress many features into fewer dimensions, mixing unrelated concepts in individual neurons. Sparse autoencoders (Bricken et al., 2023) are one way to separate those features, which could reveal cleaner spatial features than those found here.

Conclusion

Latitude and longitude are strongly linearly decodable from Qwen3.6-27B's activations. Furthermore, the model contains individual neurons whose activations track more specific spatial properties like "longitude in the US" or "absolute latitude worldwide." These neurons retain their correlation on new places and across prompts that do not ask for coordinates, suggesting that the signals are more general than a specific prompt. Together, these results suggest that the model represents geographic information both in activation space directions and in individual neurons. However, while this information can be extracted from the activations, decodability alone does not show whether or how the model uses it in generation.