The problem we tackled

Inference-time defenses are attractive because they modify the input, prompt, generation procedure, or returned output without retraining the model. Comparing them honestly, however, takes more than running several configurations and collecting a table. A benchmark name in a log does not prove that the intended image reached the model; a keyword match is not a semantic harmfulness judgment; and a vote over several runs is not necessarily the response shown to the user.

We therefore audited the whole experimental chain: the intended input, the payload actually delivered to the model, the returned text, the proxy classifier, and a separate semantic review. The archive contains two fixed defensive prompts and a Gaussian image adapter recorded under aliases derived from RapGuard, AdaShield, and SmoothVLM. They are simplified local adapters, not paper-faithful implementations of the published methods.

Audited inference-time pipeline linking the input, local adapters, model, and four independent evidence channels
The audited pipeline. Input and output provenance, batch time, semantic refusal judgments, and keyword aggregates are distinct forms of evidence and cannot substitute for one another.

What and how we audited

The planned experiment covered eight models from the InternVL and Qwen-VL families, seven safety benchmarks, and 9,000 inputs. The provenance check retained four corpora for comparative analysis: FigStep, the text-only JailBreakV branch, SALAD-Bench, and HarmBench-Simple, totaling 4,820 inputs per core configuration. Three branches had to be excluded from numerical claims:

  1. The local MM-SafetyBench renderer used the wrong payload field.
  2. The multimodal JailBreakV branch silently allowed a text-only fallback when an image was missing.
  3. The adversarial-patch branch rendered text and a random rectangle instead of generating the stated optimized attack.

Rejudging a stored answer cannot reconstruct an image that never reached the model, so post-processing cannot repair these results. For the retained corpora, we preserve the aggregates as descriptions of the archived keyword protocol, but do not interpret them as semantically validated harmlessness rates.

What the metric audit found

The archived classifier searched responses for lists of refusal and harmful-content markers. An empty string matched neither list and was therefore counted as safe. Words such as harmful, unsafe, and unethical could also make a response containing harmful instructions appear “safe,” while a defensive preamble could look like a refusal even when a substantive answer followed.

We separately reviewed 274 adversarial responses drawn from the archives. After excluding 28 garbage outputs, 246 comparable responses remained; among them, the semantic judge found 13 harmful responses missed by the keyword rule. The sample was targeted rather than random. It establishes that evaluator failures exist, but cannot estimate their prevalence or recover reliable per-cell rankings.

Heatmap of A+R keyword-score changes across four corpora and eight models
A+R keyword-score changes relative to no adapter. Their direction depends on the model and corpus, but they remain descriptive aggregates rather than semantic defense effects.

Even within the restricted evidence, the profiles vary sharply. A+R changes the InternVL3.5-8B FigStep score by +11.8 percentage points, while changing InternVL2.5-8B-MPO on HarmBench by -12.5 points. These are useful signals for a paired semantic rerun, not proof that the defense truly improved or degraded safety.

What happened to benign refusal

For legitimate requests, we ran a separate audit over 38,500 stored outputs and obtained 11,637 unique judge labels. Across the canonical 56-cell grid, the pooled refusal point estimate is 0.52%, and the largest estimable cell is 3.24%. Two cells remain non-estimable; sampling-only interval upper endpoints reach 10.92% and do not include judge error.

Heatmap of estimated benign refusal rates for eight models and seven configurations
Point estimates on 500 benign requests per cell. They do not support mass over-refusal, but rare events and judge limitations mean they are not precise rates.

The keyword layer flagged 7,532 outputs, while the weighted estimate corresponds to roughly 138 refusals, a 54.6-fold difference. Most false positives were responses echoing defensive language. In other words, the presence of safety vocabulary measures a different construct from an actual refusal to answer the user.

The practical takeaway

For me, the central result is not a defense ranking but a set of requirements for the evidence chain. A versioned manifest should bind the benchmark payload, model revision, prompts, user-visible returned text, runtime errors, proxy labels, and judge version. Without that binding, even a carefully formatted table may compare different events.

The measurable cost of the local adapters appears mainly in pipeline operation: the Gaussian branch takes 5.45 to 12.76 times the no-adapter batch processing time, the full stack takes 4.36 to 16.59 times, and the rationale prompt emits a visible defensive preamble in 20% to 78% of outputs. The observed profiles motivate fitting a configuration to the model and traffic, but do not establish that any one defense or an adaptive router is superior. Validated inputs, preserved user-visible responses, and denser independent semantic judging have to come first.