Results

Source: evals/results.json · provider gemini · model gemma-4-26b-a4b-it · ran on: laptop

24tasks

20accepted

4rejected

0errors

function	category	status	attempts	oracle cases	speedup	lower bound
`linear_score`	float	rejected	3	206	1.41x	1.37x
`sigmoid_approx`	float	accepted	2	206	1.65x	1.60x
`collatz_steps`	int	accepted	1	209	22.68x	21.79x
`digital_root`	int	accepted	1	209	3.32x	3.24x
`popcount`	int	accepted	1	209	4.99x	4.72x
`reverse_digits`	int	accepted	1	209	2.19x	2.16x
`sum_digits_sq`	int	accepted	1	209	3.01x	2.83x
`triangular`	int	accepted	1	209	47.14x	46.97x
`bucket_hash`	list	accepted	2	204	2.23x	2.17x
`count_peaks`	list	accepted	1	204	2.13x	2.02x
`longest_run`	list	accepted	2	204	1.64x	1.54x
`max_subarray`	list	accepted	1	204	1.72x	1.56x
`median_doubled`	list	rejected	3	204	1.38x	1.35x
`prefix_max_sum`	list	rejected	3	204	1.46x	1.41x
`sum_squares`	list	accepted	1	204	1.74x	1.66x
`caesar_shift`	string	accepted	1	225	11.51x	11.48x
`count_vowel_runs`	string	accepted	1	225	5.96x	5.28x
`count_words`	string	accepted	1	225	7.36x	6.74x
`is_palindrome_norm`	string	accepted	1	225	2.40x	2.39x
`longest_alpha_run`	string	accepted	1	225	6.77x	6.30x
`normalize_spaces`	string	rejected	3	225	1.22x	1.20x
`reverse_words`	string	accepted	1	225	1.61x	1.58x
`snake_to_camel`	string	accepted	1	225	2.66x	2.49x
`top_word`	string	accepted	2	225	4.96x	4.74x

Speedups — measured, not modeled

The gold mark is the conservative lower bound — the acceptance rule reads it, not the median.

triangular

47.14x

collatz_steps

22.68x

caesar_shift

11.51x

count_words

7.36x

longest_alpha_run

6.77x

count_vowel_runs

5.96x

popcount

4.99x

top_word

4.96x

digital_root

3.32x

sum_digits_sq

3.01x

snake_to_camel

2.66x

is_palindrome_norm

2.40x

bucket_hash

2.23x

reverse_digits

2.19x

count_peaks

2.13x

sum_squares

1.74x

max_subarray

1.72x

sigmoid_approx

1.65x

longest_run

1.64x

reverse_words

1.61x

False-accept gate

Wrong rewrites are rejected

13/13 deliberately wrong rewrites rejected by the meta-eval (plus a 16/16 tuple battery). The oracle catches wrong values, wrong exceptions, and panics — so a plausible-looking candidate cannot slip through.

Source: scripts/meta_eval.py · full corpus: evals/results.json

CLI reference

Commands, flags and exit codes. The help text is a captured snapshot of smelt --help; regenerate it with python3 site/scripts/gen_cli_ref.py --refresh.

Commands


smelt run <file.py:fn> [--provider p] [--model m] [--candidate lib.rs] [--cases N] [--json]
smelt capture <file.py:fn> [--cases N] [--json]      # freeze an oracle, no generation
smelt resume <run-dir> [--json]                       # continue an interrupted run
smelt report <run-dir>                                # print the run report (markdown)
smelt scan <path> [--json]                            # candidates with tier + skip reasons
smelt apply <run-dir>                                 # VIEW ONLY diff + fallback shim
smelt doctor [--probe] [--json]                       # environment + provider checks
smelt evals [--provider p] [--model m] [--max-calls N] [--pause-s S] [--out-tag T]
smelt furnace                                         # launch the Go TUI if installed

Exit codes

code	meaning
0	accepted — verified, speedup clears the conservative 1.5x lower bound
2	config error (bad target, dotted qualname, unsupported annotation on a required path)
3	refused — impure target or unsupported-but-understood shape (typed, with a hint)
4	rejected — no acceptable rewrite within budget (a correct, honest outcome)
5	harness/internal error (report the run dir)

Live `smelt --help`