
TL;DR
We recently shared Muse Ember's results in agentic pentesting. We wanted to put it to the test in Arnica's AI SAST harness, where the goal is to identify vulnerabilities in source code.
When a new model is released, we put it through rigorous evaluations and benchmarking before considering it for production. That's especially important in AI SAST, which is broadly deployed and protects our customers' source code every day.
Across multiple benchmark screenings and runs, Muse Ember at High reasoning outscored GPT-6-Luna at Max reasoning while using approximately 80% less total scan time.
Better quality at a fraction of the time.
We initially screened six model-and-reasoning configurations against roughly a quarter of our benchmark cases. This narrowed the field to three top contenders:
GPT-6-Luna High was eliminated during the qualifying round.
Each of the three finalists received two additional executions on the same inputs, followed by three executions across the full testing suite.
The evaluation covered four public and internal benchmarks, weighted at 35%, 35%, 20%, and 10%, respectively.
Findings count represents distinct labels found across any run. Scan time and cost are summed across all runs, while benchmark scores are averaged.
Ember High found more real CVE labels than Luna Max in approximately 80% less time and at about one-sixth of the cost.
Muse Ember Max achieved the highest overall benchmark score, but Muse Ember High delivered the strongest balance of cost, speed, and findings. That makes it our top choice for implementation in AI SAST.
At Arnica, a model earns its place the same way: tested repeatedly, against real code, using the actual scanner.
This time, the highest-reasoning setting wasn't the best choice. A more cost-efficient configuration delivered better results against our real-world evaluation criteria.
Muse Ember High is going into AI SAST!
>>> See Arnica AI SAST in action
AI SAST (AI-powered Static Application Security Testing) uses artificial intelligence and large language models to analyze source code and identify potential security vulnerabilities. It helps security teams detect risks in code earlier in the development lifecycle.
Muse Ember was evaluated using Arnica's AI SAST benchmarking harness across four public and internal benchmarks. We compared multiple model and reasoning configurations, measuring weighted benchmark scores, real CVE detection, public-benchmark true positives, scan time, and estimated cost.
Muse Ember High achieved an 87.21% weighted benchmark score, compared to 85.38% for GPT-6-Luna Max. It also identified 25 of 26 real CVEs compared to Luna Max's 23, while using approximately 80% less total scan time and costing about one-sixth as much.
Although Muse Ember Max achieved the highest overall benchmark score at 89.56%, Muse Ember High offered a stronger balance of vulnerability detection, speed, and cost efficiency. It identified more real CVEs than either alternative while requiring less scan time and lower estimated costs.
Scan time directly impacts how quickly security teams can identify and address vulnerabilities. Faster analysis can help teams scale security testing across more repositories and development workflows without introducing unnecessary delays or costs.
Not necessarily. Higher reasoning settings can improve benchmark performance, but they may also increase processing time and cost. Arnica's testing demonstrated that a lower reasoning configuration can deliver better practical results when evaluated across detection quality, speed, and cost.
Arnica AI SAST uses AI to identify security vulnerabilities in source code, helping teams detect and address risks earlier in the software development lifecycle. Combined with Arnica's broader application security platform, teams can prioritize findings and integrate security into developer workflows.
Integrate Arnica ChatOps with your development workflow to eliminate risks before they ever reach production.