Jarrett Vickers

Cosmos Institute - Compression Integrity Bench

I recently received a grant from the Cosmos Institute to build the Compression Integrity Bench.

Decentralized AI is premised on open-weight models running on local hardware. But almost nobody runs them at full precision. LLM deployments are implemented under various inference optimization regimes, such as quantization. Quantized LLM deployments are evaluated on task performance, but not on epistemic behavior. Careless quantization does more than just degrade performance - it disproportionately affects low-magnitude weights and the tails of the output distribution. If hedging, alternative hypotheses, and dissent live in those tails, then quantization could cause epistemic degradation. If this compromises the ability of open-weight models to maintain epistemic integrity, then the promise of decentralized AI may remain unrealized.

LLM epistemic integrity - such as calibrated uncertainty, (resistance to) sycophancy, willingness to perform Bayesian updates, use of references, and steelmanning opposition - are all implemented late in the LLM training process, in post-training. Research has shown that post-trained behaviors degrade under quantization, which suggests that epistemic integrity may be compromised. LLM epistemic integrity under quantization has never been rigorously researched and documented. You should be able to know if the local model you chose behaves faithfully to the one you think you're running.

Principles of Machine Learning

I teach machine learning at Network School in Astana, Kazakhstan.

Quantization Study

I ran a quantization study on an open-source coding model, DeepSeek-Coder-V2-Lite-Instruct.

The goals of this project were to 1) teach myself inference engineering by jumping into the deep end of the pool, and 2) extensively document this process in a series of notebooks so that I could use them to teach inference engineering to others.

The intent was to quantize the model as heavily as possible without degrading performance too badly. Quantizing from 16-bit to 8-bit integers had minimal impact on the HumanEval results, while substantially reducing model size and increasing token/sec. We see more model errors at 4-bit quantization, which I was able to mitigate by running sensitivity analyses on each layer and identifying which layers degenerated performance the most under 4-bit quantization. This led me to create a mixed-precision policy, in which four layers were left at 8-bit quantization while the others were set to 4-bit. The result was a model that was only marginally larger than the pure 4-bit version, while reclaiming much of the lost HumanEval Pass@1 performance.

Does GRAM's Knowledge Isolation Survive Quantization?

In July, AE Studios and Anthropic joint published a new model architecture for technical alignment called Gradient Routed Auxiliary Modules (GRAM). The idea is to separate knowledge of dual-use technologies (such as nuclear, biotech, etc.) into separate modules during pretraining, which can then be ablated at inference.

Many technical alignment solutions have been shown to degrade under quantization and other inference optimizations, so I tested whether this was the case for GRAM. They didn't publish the weights of the model they trained, but they did publish the exact methodology they used to train it. It's a small 26m param model, so I was able to train my own model that performed the same as theirs.

I ran some evals to establish a baseline, then started quantizing. From the results:

At int4, signed capability recovery was −77.48% and signed isolation erosion was 1.97%, both below the pre-registered 20% thresholds. Mean retained-topic loss rose 5.46%, below the 10% general-degradation guard. Negative recovery means the quantized, ablated model moved farther from the FP32 active reference; it is not a claim that quantization improved knowledge removal.

Robotics Hackathon at Network School

At Network School, I taught a robotics class with my co-host Lana. We were able to take a room of complete robotics novices and had them assemble and program a series of SO-ARM100 robotic arms in just a few hours. Our students also learned how to operate a 3D printer.

OpenAI-to-Z Challenge

My team's submission for the OpenAI-to-Z challenge. Process large amounts of satellite data to scan the Amazonian rainforest for undiscovered ruins and earthworks. A Jupyter notebook that utilizes essential data science tools such as numpy, pandas, matplotlib, and more niche tooling such as geopandas and rasterio. Large-scale image classification.