One-sentence Summary

Tencent and Xiamen University researchers propose YOLO-PEFT, a structure-aware framework that formulates parameter-efficient fine-tuning adapter placement as an auditable constraint-planning problem, enabling automated target-module selection with reason codes and a Refuse capability, and achieving mAP50-95 scores of 0.7138 on YOLO11s and 0.7307 on YOLO12s while cutting peak training memory by 43.9%.

Key Contributions

• YOLO-PEFT formulates adapter placement for real-time detectors as an auditable constraint-planning problem that assigns operator and semantic roles, evaluates explicit validity checks, and returns a budgeted target-module plan or a refusal with reason codes before training.
• Under the VOC07+12 trainval-to-VOC07 test protocol, planner-selected RS-LoRA achieves 0.7138 and 0.7307 mAP50-95 on YOLO11s and YOLO12s, respectively, outperforming full fine-tuning (0.6428 and 0.6662). On RT-DETR-L, all evaluated LoRA configurations trigger a predefined catastrophic threshold, supporting a calibrated Refuse-to-Full-SFT decision within the tested coverage.
• The framework replaces manual target-module trial and error with explicit, inspectable planning and preserves verified train-save-merge-export paths. A controlled YOLO11 audit shows LoRA reduces peak training memory by 43.9% with 1.72× longer training time.

Introduction

Deploying real-time detectors across diverse vocabularies, domains, and hardware backends requires recurring adaptation that full fine-tuning makes prohibitively expensive. Parameter-efficient fine-tuning (PEFT) addresses this cost, but standard PEFT methods designed for homogeneous Transformer stacks break down on heterogeneous detector graphs: modern YOLO-style architectures mix convolution types, distribution-aware losses, deformable attention, text–image fusion, and mixture-of-experts routing, causing indiscriminate adapter insertion to degrade accuracy or fail entirely. Prior detector-specific PEFT approaches are tied to particular architectures or tasks, lack a pre-training feasibility check, and offer no unified lifecycle for training, merging, and export, so failure is discovered only through trial and error. The authors recast detector PEFT as multi-constraint planning over a heterogeneous computation graph and introduce YOLO-PEFT, a structure-aware framework that parses operator semantics, filters unsafe targets, allocates ranks within a budget, estimates risk, refuses infeasible plans before training, and lowers accepted plans into a complete train–save–merge–export runtime.

Method

The authors frame adapter placement in YOLO detectors as a constrained decision problem that cannot rely solely on module-name matching, because the heterogeneous operators and task-specific dense heads demand structural and semantic awareness. The pipeline proceeds in four stages: parsing the detector into a role-aware graph, resolving a constraint-valid placement, lowering the plan into a YOLO-compatible runtime, and applying optional training strategies that stabilize optimization. A request is accepted only if a trainable, deployable plan exists; otherwise the framework explicitly refuses, distinguishing unsupported configurations from those predicted to be inaccurate.

A detector is represented as a directed acyclic graph $G = (V, E)$G=(V,E) where vertices are module instances and edges carry tensor flow. A PEFT request $R = (p, T, L, K)$R=(p,T,L,K) specifies a variant $p$p, an optional target set $T$T, an optional layer interval $L$L, and a set of allowed ranks $K$K. Given a parameter budget $B$B, the framework produces a placement $\pi: V_{\mathrm{cand}} \to \{0\} \cup K$π:Vcand​→{0}∪K, with $\pi(i)=0$π(i)=0 indicating a frozen layer and $\pi(i)=r$π(i)=r assigning a rank-$r$r adapter. A valid plan must jointly satisfy operator validity, detection-head semantic safety, graph-interface compatibility, parameter-budget feasibility, and deployment compatibility. The overall output is a triple $\mathcal{P}(G, R, B) = (d, \pi, \mathcal{J})$P(G,R,B)=(d,π,J) where $d \in \{\mathrm{ACCEPT}, \mathrm{REFUSE}\}$d∈{ACCEPT,REFUSE} and $\mathcal{J}$J is an ordered rejection log that explains every exclusion.

To make these decisions auditable, the GraphParser assigns each module two independent roles. The operator role captures the computational contract: dense convolutions admit ordinary low-rank factors, grouped convolutions require group-local factors, and depthwise convolutions are considered unsafe because they cannot mix channels. Normalization, activation, and unknown operators are treated conservatively. The semantic role encodes detector-specific meaning: fixed DFL projections, geometry-sensitive regression paths, and MoE routers are permanently excluded, since even small updates can disturb calibrated outputs or expert assignment. Role assignment combines graph position with recognized architectural idioms, including YOLO12 Area-Attention blocks, RT-DETR deformable-attention modules, YOLO-World text fusion, and MoE router/expert structures. Unrecognized modules receive an unknown role and are never adapted, making unsupported structure explicit. The parser also computes a bounded fingerprint $\phi(G) \in \mathbb{R}^{10}$ϕ(G)∈R10 whose core dimensions measure the relative presence of attention, text fusion, MoE experts, depthwise convolution, and dense/grouped convolution (e.g., $\phi_{\mathrm{attn}} = |\mathrm{attention}| / |V_{\mathrm{role}}|$ϕattn​=∣attention∣/∣Vrole​∣), while five additional statistics encode depth, width, head-parameter ratio, residual density, and normalization type. The core dimensions drive stability analysis and variant-level calibration, and neither fingerprint dimension is validated on unseen detector families.

Given the role-aware graph, the Planner resolves placement in a fixed sequence: filter operators, filter semantics, apply architecture-conditioned policies, allocate ranks, then validate reliability. Validity always precedes efficiency. Operator filtering removes depthwise, normalization/activation, and unknown operators. For grouped convolutions with $G$G groups, the total rank is distributed as $r = \sum_{g=1}^{G} r_g$r=∑g=1G​rg​, and balanced allocation requires $G \mid r$G∣r. Semantic filtering then eliminates fixed DFL projections, MoE routers, and geometry-sensitive regression heads. User-supplied targets $T$T and layer interval $L$L are intersected only after these mandatory filters, so an explicit request cannot re-enable an unsafe module.

Architecture-conditioned policies encode empirical safeguards. For example, an RT-DETR-L profile with $\phi_{\mathrm{attn}} > 0.7$ϕattn​>0.7 refuses LoRA-family configurations below a calibrated threshold, and unrecognized Transformer detectors remain unsupported. On attention-heavy YOLO12, DoRA is converted to LoRA and safe-attention training is enabled; the evaluated profile admits $r \in \{8, 16, 32\}$r∈{8,16,32} with $r=16$r=16 as the Pareto default. YOLO-World text-fusion requests can be routed from LoRA to LoHa, while CNN-only graphs disable attention targets. Module-level rules are deliberately conservative: YOLO12 excludes attn.{qkv,proj,pe} and internal Area-Attention MLP convolutions; RT-DETR excludes grid-initialised sampling_offsets and zero-initialised attention_weights; and .dfl. projections are always frozen. The authors illustrate with an end-to-end example on YOLO12s, where a rank-16 RS-LoRA request over all convolutional and linear modules is first expanded into backbone, neck, and decoupled-head nodes. Operator and semantic filtering remove depthwise hosts, DFL, and geometry-sensitive regression nodes; the YOLO12 policy removes attn.{qkv,proj,pe} and internal attention MLP convolutions; graph-interface checks keep shape-preserving backbone/neck convolutions. The budget solver then assigns $r=16$r=16 to the surviving dense convolution targets until $B$B is met, and the emitted target list is accompanied by an excluded-name/reason map and the realised parameter cost. Changing the user range $L$L or budget $B$B changes only the final intersection and rank assignment, never the mandatory exclusions.

After filtering, the planner assigns ranks by solving

where for LoRA-style adapters $c_{\mathrm{linear}} = b r (d_{\mathrm{in}} + d_{\mathrm{out}})$clinear​=br(din​+dout​) and $c_{\mathrm{conv}} = b r (C_{\mathrm{in}} k_h k_w + C_{\mathrm{out}})$cconv​=br(Cin​kh​kw​+Cout​), with $b$b bytes per trainable parameter. Grouped convolution uses $b \sum_g r_g ( (C_{\mathrm{in}}/G) k_h k_w + C_{\mathrm{out}}/G )$b∑g​rg​((Cin​/G)kh​kw​+Cout​/G). In rule-only mode the utility $u$u combines operator, semantic, range, and rank utilities minus a cost penalty $\lambda c_p$λcp​; candidate pairs with non-positive utility are discarded. If every rank is zero or the realized cost exceeds $B$B, the plan is refused.

For candidate pairs not settled by hard constraints, a calibrated reliability module estimates $\Delta\mathrm{mAP} \approx \beta_0 + \beta_1 \phi_{\mathrm{attn}} + \beta_2 \phi_{\mathrm{text}} + \beta_3 \phi_{\mathrm{dw}} + \beta_4 \xi_p$ΔmAP≈β0​+β1​ϕattn​+β2​ϕtext​+β3​ϕdw​+β4​ξp​, where $\xi_p$ξp​ is a variant coefficient fitted on a canonical matrix. If the prediction falls below a catastrophe threshold, the plan is refused. Calibrated mode is evaluated on observed architectures only; rule-only mode remains the default.

Once a plan is accepted, the Contract lowers each $(i, r)$(i,r) while preserving four invariants: base-checkpoint loading is unchanged; adapter checkpoints store only adapter tensors and runtime metadata; merging restores ordinary YOLO modules; and ONNX and TensorRT exporters see an export-compatible graph. The runtime exposes one interface over two backend families. HuggingFace PEFT handles LoRA, RS-LoRA, DoRA, LoHa, LoKr, AdaLoRA, IA$^3$3, OFT, BOFT, and HRA. A narrower in-repository backend supplies plain convolutional LoRA when PEFT is unavailable. Both routes share the same resolved target set and lifecycle: install before optimizer construction, train with frozen base weights, save adapter-only state, load onto a compatible base detector, merge, remove wrappers, and export. Convolution fusion is disabled while adapters remain unmerged. For the fallback convolutional LoRA, each group-local update $\Delta W_g = B_g A_g^{\top}$ΔWg​=Bg​Ag⊤​ is reshaped to its host-kernel partition; merging $W_0 \gets W_0 + s\Delta W$W0​←W0​+sΔW preserves the grouping structure and yields outputs equivalent to those of the wrapped module up to floating-point tolerance.

Four optional training strategies stabilize optimization without altering placement. Layer-wise learning-rate decay uses $\eta_{\ell} = \eta_0 \rho^{d_{\ell}}$ηℓ​=η0​ρdℓ​ with $\rho = 0.85$ρ=0.85, where normalized depth $d_{\ell}$dℓ​ reduces updates in early feature extractors; similar depth factors are coalesced into few optimizer groups. Alpha warmup ramps the LoRA scaling factor $\alpha/r$α/r from zero to its target with a cosine schedule, preventing a full-strength random adapter from destabilizing attention-heavy detectors at start (YOLO12 requires at least three warmup epochs). Orthogonal regularization adds $\lambda_{\mathrm{ortho}} (\|A^{\top}A - I\|_F + \|B^{\top}B - I\|_F)$λortho​(∥A⊤A−I∥F​+∥B⊤B−I∥F​) every $N$N batches with $\lambda_{\mathrm{ortho}} = 0.5$λortho​=0.5 by default, computed in chunks to bound memory. Dynamic dropout linearly increases adapter dropout from $0$0 to $0.15$0.15 after the first 30% of epochs, preserving early gradient signal while regularizing later updates. Each strategy is independently configurable.

Experiment

Using the PASCAL VOC dataset and a consistent fine-tuning protocol, the experiments assess parameter-efficient adapter placement across CNN, attention-augmented, text-fusion, and Transformer-based real-time detectors. PEFT reliability is strongly architecture-conditioned: attention-heavy models (YOLO12, RT-DETR-L) suffer severe collapses, while a structure-aware planner that respects operator semantics, excludes detection heads, and mitigates multi-modal gradient mismatches consistently outperforms naive placement and full fine-tuning. Ablations further show that semantic validity constraints act as stability drivers, training priors interact non-additively, and refusing unsafe configurations prevents wasted training attempts, although memory savings come with longer optimization time.

PEFT adapters applied to the YOLO11s detector consistently outperform full fine-tuning on VOC2007. LoRA and DoRA with RS-LoRA scaling achieve the highest gain of +0.071 mAP50:95, while disabling RS-LoRA on DoRA nearly eliminates improvement. The LoKr variant delivers a strong +0.0605 gain with no added inference cost and minimal parameter increase. LoRA and DoRA with rank 16 and RS-LoRA scaling both reach 0.7138 mAP50:95, improving over full fine-tuning by +0.071. Removing RS-LoRA scaling from DoRA drops the gain to only +0.005, showing it is critical for performance. LoKr improves mAP50:95 by +0.0605 while retaining the same inference GFLOPs as full fine-tuning and adding just 0.07M parameters.

The formal constraint interface categorically prevents unsafe structural modifications during detector adaptation. Experiments confirm that relaxing the semantic head restriction expands the candidate pool dramatically with only a marginal accuracy gain, while operator validity correctly excludes depthwise operations that offer no empirical benefit. The architecture policy is critical because even safe upstream adaptation propagates nontrivial drift into a frozen decoder, and training prior interactions show that reliability calibration must guard against non-additive configuration collapse. Removing the head-type restriction admits over 500× more candidate modules and yields only a modest accuracy improvement, suggesting a shift in optimization pressure rather than divergence. Depthwise convolutions excluded by operator validity contribute no viable high-rank targets, meaning the constraint costs nothing in practice. Safe upstream adaptation still causes measurable drift in a frozen RT-DETR decoder, justifying the policy that freezes sampling offsets and attention weights. A rank budget of 16 achieves the best accuracy-efficiency trade-off, with higher ranks producing diminishing returns and confirming the budget constraint's practical relevance. Naive adapter substitution without the deployment contract breaks ONNX/TensorRT export and checkpoint saving, while the contract layer preserves merge and export invariants. Joint training prior interactions are non-additive: enabling DoRA without RS-LoRA scaling causes a severe performance drop, motivating within-coverage reliability checks.

Switching from language-model defaults to structure-aware placement consistently improves detection accuracy. On YOLO11s and YOLO12s, using RS-LoRA instead of DoRA without rank-stabilized scaling raises mAP by approximately 0.066 and 0.120, respectively. For RT-DETR-L, the inherited LoRA sweep leads to catastrophic performance, and the planner safely refuses, falling back to full fine-tuning. YOLO12s benefits most, with RS-LoRA reaching 0.7307 mAP compared to 0.6112 for the default DoRA configuration. The planner prevents a catastrophic outcome on RT-DETR-L by refusing to return an adapter below the threshold and defaulting to full fine-tuning. Both columns share the same backbone and evaluation protocol, isolating structure-aware placement as the source of the observed gains.

PEFT adapters were stress-tested on a Mixture-of-Experts YOLO detector with all targeting controls disabled. HRA delivered the highest mAP50:95 at 0.7454, surpassing full fine-tuning by a substantial margin, while IA³ and LoKr also showed strong improvements. Two configurations failed before evaluation, illustrating why a planner must verify capability before deployment. HRA achieved the best mAP50:95 (0.7454), a gain of +0.0563 over the logged full fine-tuning score of 0.6891. IA³ and LoKr both exceeded 0.74 mAP50:95, whereas LoRA and RS-LoRA produced only marginal improvements below 0.70. BOFT and OFT failed without reporting evaluation metrics, while all six metric-bearing PEFT runs finished successfully. Wall-clock runtime varied widely among completed runs, from 2.14 hours (RS-LoRA) to 11.37 hours (HRA), with full fine-tuning at 5.54 hours.

Constraint predicate diagnostics on YOLO12s reveal that opening the attention-stream and typed-head rules admits many new modules and raises adapter gradient norms while keeping gradients finite, indicating altered optimization pressure rather than collapse. The depthwise rule adds no modules at rank 4 and thus cannot be credited with a training effect; counterfactual rules for DFL, MSDeform geometry, and output heads are not directly insertable due to incompatible semantics. Safe upstream adaptation on RT-DETR-L propagates measurable relative drift into a frozen decoder (around 0.5 ℓ₂ shift in decoder and final outputs). Removing the attention-stream constraint allows 32 modules into the adapter and raises the mean adapter-gradient norm by 1.581× without producing non-finite gradients, suggesting the rule modulates optimization pressure rather than preventing collapse. Opening the typed detection-head predicate admits 18 modules, increases the gradient norm by 1.091×, and yields a short-run mAP change of -0.0337, while the depthwise rule triggers no additional modules at rank 4 and therefore provides no training signal. Counterfactual predicates for DFL projection, MSDeform geometry, and score/bbox output layers cannot be tested via a standard leave-one-out approach because their fixed-bin, sampling-grid, or output-bias semantics require a qualitatively different insertion. Under safe placement on RT-DETR-L, comparing an active adapter to a zeroed adapter produces relative ℓ₂ drifts of 0.553±0.099 in decoder modules and 0.548±0.094 in final outputs, confirming that upstream adaptation propagates into the frozen decoder.

Experiments evaluate PEFT adapters applied to object detectors (YOLO and RT-DETR) under a formal constraint interface that enforces safe structural modifications. Key findings show that structure-aware placement and RS-LoRA scaling are critical for strong gains, while relaxing head-type or attention-stream constraints only modestly alters optimization pressure without causing divergence. A safety planner prevents catastrophic outcomes by refusing adapters that fall below a performance threshold, and stress-testing on a Mixture-of-Experts architecture reveals substantial method variability and outright failures, underscoring the need for pre-deployment capability verification. Overall, the studies validate that combining disciplined adapter insertion, budget-aware rank selection, and deployment-contract preservation yields reliable, efficient adaptation, whereas naive unconstrained approaches risk severe performance collapse or export incompatibility.