less than 1 minute read

I have updated my Artificial Analysis graph to v4.2. What interests me is how quickly the index changes. Metrics used to be something models worked around, but once models learn how to do that, the metrics have to change too.

That creates an awkward problem. The new metric has to chase a moving target, using some proxy for what we care about, or perhaps just a well-informed gut feeling. I am not sure there is a final answer here, but every new index deserves a fresh look.

You May Also Enjoy

OpenCode Go’s quota is the price

2 minute read

I am a big fan of OpenCode Go. For $10 a month after the first month, it gives developers access to a useful set of coding models, and the price feels almost...

PII is becoming a propensity

less than 1 minute read

I suspect the definition of personally identifiable information will become stricter as AI lowers the cost of piecing together an identity. A recent paper, L...

The surplus hidden by gold imports

1 minute read

I keep coming back to Brad Setser’s work on Chinese economic data because he shows how much interpretation sits behind an apparently simple metric. In his la...

AI and the history that was still hidden

less than 1 minute read

The report that GPT-6 Astra helped decode a 1918 German ADFGVX message is another reason I expect interesting historical research from AI. The result still n...