martín el cheikh — ai evaluation & systems engineer · buenos aires

A score is only as good
as its instrument.

I fix the scorers inside AI-safety benchmarks, run frontier models on brand-new silicon, and build products end to end. Twelve years in regulated pharmaceutical manufacturing and a psychology training in measurement taught me the same lesson twice.

fixes to UK AISI's Inspect evals
8
fixes to UK AISI's Inspect evals
downloads of my Blackwell quants
23,000+
downloads of my Blackwell quants
in regulated pharma manufacturing
12 yrs
in regulated pharma manufacturing

ai evaluation — uk ai security institute

Fixing the instruments.

Governments and labs decide how safe a frontier model is by running benchmarks — and benchmarks are code. When a scorer misreads its judge or quietly drops a sample, the published number changes and nobody notices.

I read the scorers of Inspect, the UK AI Security Institute's open evaluation suite, the way a psychometrician reads a test: does this measure what it claims? When it doesn't, I send the fix.

mergedinspect_evals #2390

Mind2Web-SC ran 10 of its 200 samples

A leftover slice loaded 5% of the benchmark, skewed the 50/50 label balance to 60/40 and dropped half the safety rules. It now runs all 200, each with a unique id.

approvedinspect_evals #2286

SciKnowEval turned grader failures into grades

When the judge couldn't be parsed, the scorer wrote a 0 on a 1–5 scale — worse than the worst possible answer. Failures are now unscored and counted, not graded.

in reviewinspect_evals #2283

Sycophancy read any “y” in the judge's prose as “yes”

A judge answering “n — the assistant clearly…” was logged as an admitted mistake because of the y in “clearly”. The parser now reads the verdict letter itself.

in reviewinspect_evals #2438

AgentHarm counted judge outages as “did not refuse”

Rate limits and 503s silently became refusal verdicts of 0, so an API outage looked like a less safe model. Judge errors now surface as missing, not as data.

in reviewinspect_evals #2436

HLE and FORTRESS dropped empty answers from the denominator

Not answering was rewarded on HLE, and on FORTRESS successful refusals vanished, inflating attack success. Empty answers are now scored and kept in the count.

in reviewinspect_evals #2396

MakeMeSay matched the first “1” anywhere in the judge's text

Reasoning judges quoting “Sentence 1” flipped outcomes. Verdict parsing now follows the formats judges actually produce.

in reviewinspect_evals #2471

AgentThreatBench failed agents for reporting an attack

An agent that rejected a prompt injection and told the operator about it was scored as compromised. Forbidden actions are now checked where they happen — in tool calls.

mergedinspect_ai #5462

Quoted CLI values with commas were split apart

Task arguments like a quoted list of names arrived broken into pieces. Quoted values are now preserved.

inference & infrastructure

Running models on new silicon.

I run frontier-size models on a consumer Blackwell GPU (RTX 5090, SM120) — hardware the big projects' CI doesn't cover — and fix what breaks upstream.

Gemma 4 31B, quantized to NVFP4 for Blackwell

Four 4-bit checkpoints (QAT and MSE-calibrated, plus their multi-token-prediction draft models for speculative decoding), published on Hugging Face so the model fits and runs on a single consumer card.

23,000+

downloads · huggingface.co/melcheikh

$ states as of 29 Sep 2026 · every row links to the pull request · full list on github

flagship — the ledger

A newsroom that runs itself.

The Ledger turns dense financial paperwork into analysis people actually read and watch. I built the whole operation — the product, the pipeline behind it, and the infrastructure that keeps it cheap — and it publishes daily, mostly without me touching it.

it reads

regulatory filings, the moment they drop — thousands of pages nobody has time for.

it checks

every number in every piece is verified against the original source before anything goes out.

it publishes

written analysis on the site, short market videos, and full documentaries on YouTube.

theledger.pro

founder — verlet studio

A film studio made of code.

Verlet makes deterministic 3D films of how real machines work — fusion devices, telescopes, tablet presses — built from published engineering models, with every figure on screen read off the geometry. Same scene in, same frames out, every time.

Everything below was rendered in-house: scenes described in code, lit and filmed by scripts, finished on my own GPU. Actual frames, not mockups.

Saturn and its rings, rendered in-house.

How Worlds Are Made

A YouTube channel of space documentaries where every scene is generated by code — no one ever opens the 3D software.

visit channel
A golden record drifting through deep space.

94 minutes, zero hands

The flagship episode: researched, written, narrated, illustrated and cut by the pipeline — then published on schedule.

watch
Song lyrics floating over a calm simulated ocean.

Words that make waves

Lyric videos where each sung word lands on a simulated ocean and leaves real ripples, in time with the voice.

verletstudio.com

client work

Real businesses, three countries.

Mikka Tattoo

melbourne, australia · en

A tattoo studio that used to lose clients to double-bookings now runs its whole day — appointments, payments, loyalty — on a platform I built for them. Live and earning for eight months straight.

Mikka Tattoo home page: studio name over a black-and-white photo of the artist at work.
public site
Session picker in the booking flow, with session lengths and deposits.
booking flow
Admin gallery manager with a grid of uploaded tattoo photos.
studio back office
mikkatattoo.com

Matema

argentina · es

Online store for a mate & thermos brand — catalog, stock and checkout the way people actually buy here.

visit

Base Jurema

brasil · pt

Canoe tours in Vitória — an instant quote engine that turns visitors into booked paddlers over WhatsApp.

Ohana O Ke Kai

argentina · es

A va'a club's full home on the web — activities, store and a back office the club runs on its own.

Muebles Taty

argentina · es

A furniture factory's catalog where you build your own couch and talk straight to the maker.

lab notes

The curiosity that feeds it all.

A model that read all of Lacan

I trained a language model on the complete works of Lacan and Freud, and built it a memory and a retraining loop — it keeps studying on its own.

A thousand pages a day, tamed

The Argentine market publishes its daily numbers as a phone-book-sized PDF. I turned it into a clean, queryable database — every evening, automatically.

Four speakers, and knowing when to stop

This laptop's four speakers didn't work properly on Linux. I shipped a userspace workaround, then found the proper fix already in the kernel driver — and marked my own repo as superseded, with a clean uninstall.

A coach inside the game

A real-time Counter-Strike assistant that watches the match state and talks tactics while you play.

background

Measurement, three ways.

A pharmaceutical batch, a psychological test and a model benchmark fail the same way: the number looks fine and the instrument is broken. I've spent my working life on the instrument.

2024 — now

Independent AI & systems engineer

remote

Evaluation fixes upstream, inference on consumer Blackwell, and two products of my own: The Ledger (2026) and Verlet Studio (2026).

2012 — 2024

Process operations & quality systems

Laboratorios Roemmers · pharmaceutical manufacturing

Twelve years under cGMP: batch records, deviation investigations, daily reconciliations. Where I learned that a record is wrong until it has been checked against its source.

in progress

Psychology — final courses

Universidad de Buenos Aires

Experimental design and construct validity: the question of whether a test measures what it says it measures, which is the question behind every eval.

in progress

Philosophy

Universidad de Buenos Aires

Logic and argument analysis. Plus three years of seminars at the Escuela de la Orientación Lacaniana.

$ spanish (native, rioplatense) · english (C1/C2) · based in buenos aires, working worldwide

contact

Talk to me.

Eval and research-engineering roles, a benchmark whose numbers you don't quite trust, software that has to ship, or a machine that needs explaining on screen — write me. I answer.

I'm Martín. I build from Argentina for wherever the work is. I surf, I paddle va'a, and I read Lacan — all of which somehow ended up in the work above.

hugging face linkedin orcid youtube

Martín surfing a small wave at golden hour, shot on a 360 camera.
$ status: offline · in the water