RT-DETR改进策略【Backbone/主干网络】| CVPR 2024 替换骨干网络为 RMT，增强空间信息的感知能力
一、本文介绍
本文记录的是
将RMT应用于RT-DETR骨干网络的改进方法研究
。
RMT
通过构建基于
曼哈顿距离
的空间衰减矩阵，引入
显式空间先验
，同时提出
新的注意力分解形式
，在不破坏空间衰减矩阵的前提下，以
线性复杂度
对全局信息进行
稀疏建模
。将
RMT
融
入RT-DETR
的骨干网络，能够
有效提升其对图像空间信息的感知能力
，在减少计算量的同时增强特征提取效果，进而提高
RT-DETR
在各项任务中的准确性与效率 。
在
RT-DETR
的基础上配置了原论文中
RMT_T
，
RMT_S
，
RMT_B
,
RMT_L
四种模型，以满足不同的需求。
二、RMT原理介绍
RMT:Retentive Networks Meet Vision Transformers
RMT模型
是一种具有显式空间先验的视觉骨干网络，旨在解决
Vision Transformer（ViT）
中
自注意力机制
存在的问题。其结构设计的出发点、原理和优势如下：
2.1 出发点
ViT
中的自注意力机制
缺乏显式空间先验
，且在
对全局信息建模时具有二次计算复杂度
，限制了
ViT
的应用。为缓解这些问题，作者从
自然语言处理
领域的
Retentive Network（RetNet）
中汲取灵感，提出
RMT模型
。
2.2 结构原理
-
Manhattan Self - Attention（MaSA）
：将
RetNet中的 单向一维时间衰减 扩展为 双向二维空间衰减 ，基于曼哈顿距离引入 显式空间先验 。-
通过从单向到双向衰减、从一维到二维衰减的转换，并结合
Softmax函数，构建了MaSA机制。
-
通过从单向到双向衰减、从一维到二维衰减的转换，并结合
- 为降低计算成本，提出一种分解方法，沿图像的两个轴分解自注意力和空间衰减矩阵， 使每个令牌的感受野形状与完整MaSA的感受野形状相同，从而保留显式空间先验 。
-
Local Context Enhancement（LCE）模块
：为增强
MaSA的 局部表达能力 ，引入LCE模块，使用DWConv进一步提升模型性能。 -
整体架构
：
RMT基于MaSA构建，分为四个阶段。前三个阶段使用分解后的MaSA，最后一个阶段使用原始MaSA。同时，模型中融入了CPE（Conditional Positional Encodings）， 为模型提供灵活的位置编码和更多位置信息。
2.3 优势
- 性能优越 ：在多个视觉任务上表现出色，如在ImageNet - 1K图像分类任务中，RMT - S在仅4.5GFLOPs的计算量下，Top1准确率达到84.1%；RMT - B在相似计算量下，比iFormer高出0.4%。在COCO检测任务和ADE20K语义分割任务中也取得了优异成绩，RMT - L在COCO检测任务中，box AP达到51.6，mask AP达到45.9；在ADE20K语义分割任务中，RMT - L的mIoU达到52.8。
- 推理速度快 ：与其他先进的视觉骨干网络相比，RMT在速度和准确性之间实现了最佳权衡（表9展示了RMT与其他模型的推理速度比较）。
论文： https://arxiv.org/pdf/2309.11523
源码： https://github.com/qhfan/RMT
三、RMT的实现代码
RMT
的实现代码如下：
import torch
import torch.nn as nn
from torch.nn.common_types import _size_2_t
import torch.utils.checkpoint as checkpoint
from timm.models.layers import DropPath, to_2tuple, trunc_normal_
import math
import torch
import torch.nn.functional as F
import torch.nn as nn
from timm.models.layers import DropPath, trunc_normal_
from timm.models.vision_transformer import VisionTransformer
from timm.models.registry import register_model
from timm.models.vision_transformer import _cfg
from typing import Tuple, Union
from functools import partial
__all__ = ['RMT_T', 'RMT_S', 'RMT_B', 'RMT_L']
class DWConv2d(nn.Module):
def __init__(self, dim, kernel_size, stride, padding):
super().__init__()
self.conv = nn.Conv2d(dim, dim, kernel_size, stride, padding, groups=dim)
def forward(self, x: torch.Tensor):
'''
x: (b h w c)
'''
x = x.permute(0, 3, 1, 2) #(b c h w)
x = self.conv(x) #(b c h w)
x = x.permute(0, 2, 3, 1) #(b h w c)
return x
class RelPos2d(nn.Module):
def __init__(self, embed_dim, num_heads, initial_value, heads_range):
'''
recurrent_chunk_size: (clh clw)
num_chunks: (nch ncw)
clh * clw == cl
nch * ncw == nc
default: clh==clw, clh != clw is not implemented
'''
super().__init__()
angle = 1.0 / (10000 ** torch.linspace(0, 1, embed_dim // num_heads // 2))
angle = angle.unsqueeze(-1).repeat(1, 2).flatten()
self.initial_value = initial_value
self.heads_range = heads_range
self.num_heads = num_heads
decay = torch.log(1 - 2 ** (-initial_value - heads_range * torch.arange(num_heads, dtype=torch.float) / num_heads))
self.register_buffer('angle', angle)
self.register_buffer('decay', decay)
def generate_2d_decay(self, H: int, W: int):
'''
generate 2d decay mask, the result is (HW)*(HW)
'''
index_h = torch.arange(H).to(self.decay)
index_w = torch.arange(W).to(self.decay)
grid = torch.meshgrid([index_h, index_w])
grid = torch.stack(grid, dim=-1).reshape(H*W, 2) #(H*W 2)
mask = grid[:, None, :] - grid[None, :, :] #(H*W H*W 2)
mask = (mask.abs()).sum(dim=-1)
mask = mask * self.decay[:, None, None] #(n H*W H*W)
return mask
def generate_1d_decay(self, l: int):
'''
generate 1d decay mask, the result is l*l
'''
index = torch.arange(l).to(self.decay)
mask = index[:, None] - index[None, :] #(l l)
mask = mask.abs() #(l l)
mask = mask * self.decay[:, None, None] #(n l l)
return mask
def forward(self, slen: Tuple[int], activate_recurrent=False, chunkwise_recurrent=False):
'''
slen: (h, w)
h * w == l
recurrent is not implemented
'''
if activate_recurrent:
retention_rel_pos = self.decay.exp()
elif chunkwise_recurrent:
mask_h = self.generate_1d_decay(slen[0])
mask_w = self.generate_1d_decay(slen[1])
retention_rel_pos = (mask_h, mask_w)
else:
mask = self.generate_2d_decay(slen[0], slen[1]) #(n l l)
retention_rel_pos = mask
return retention_rel_pos
class MaSAd(nn.Module):
def __init__(self, embed_dim, num_heads, value_factor=1):
super().__init__()
self.factor = value_factor
self.embed_dim = embed_dim
self.num_heads = num_heads
self.head_dim = self.embed_dim * self.factor // num_heads
self.key_dim = self.embed_dim // num_heads
self.scaling = self.key_dim ** -0.5
self.q_proj = nn.Linear(embed_dim, embed_dim, bias=True)
self.k_proj = nn.Linear(embed_dim, embed_dim, bias=True)
self.v_proj = nn.Linear(embed_dim, embed_dim * self.factor, bias=True)
self.lepe = DWConv2d(embed_dim, 5, 1, 2)
self.out_proj = nn.Linear(embed_dim*self.factor, embed_dim, bias=True)
self.reset_parameters()
def reset_parameters(self):
nn.init.xavier_normal_(self.q_proj.weight, gain=2 ** -2.5)
nn.init.xavier_normal_(self.k_proj.weight, gain=2 ** -2.5)
nn.init.xavier_normal_(self.v_proj.weight, gain=2 ** -2.5)
nn.init.xavier_normal_(self.out_proj.weight)
nn.init.constant_(self.out_proj.bias, 0.0)
def forward(self, x: torch.Tensor, rel_pos, chunkwise_recurrent=False, incremental_state=None):
'''
x: (b h w c)
mask_h: (n h h)
mask_w: (n w w)
'''
bsz, h, w, _ = x.size()
mask_h, mask_w = rel_pos
q = self.q_proj(x)
k = self.k_proj(x)
v = self.v_proj(x)
lepe = self.lepe(v)
k *= self.scaling
qr = q.view(bsz, h, w, self.num_heads, self.key_dim).permute(0, 3, 1, 2, 4) #(b n h w d1)
kr = k.view(bsz, h, w, self.num_heads, self.key_dim).permute(0, 3, 1, 2, 4) #(b n h w d1)
'''
qr: (b n h w d1)
kr: (b n h w d1)
v: (b h w n*d2)
'''
qr_w = qr.transpose(1, 2) #(b h n w d1)
kr_w = kr.transpose(1, 2) #(b h n w d1)
v = v.reshape(bsz, h, w, self.num_heads, -1).permute(0, 1, 3, 2, 4) #(b h n w d2)
qk_mat_w = qr_w @ kr_w.transpose(-1, -2) #(b h n w w)
qk_mat_w = qk_mat_w + mask_w #(b h n w w)
qk_mat_w = torch.softmax(qk_mat_w, -1) #(b h n w w)
v = torch.matmul(qk_mat_w, v) #(b h n w d2)
qr_h = qr.permute(0, 3, 1, 2, 4) #(b w n h d1)
kr_h = kr.permute(0, 3, 1, 2, 4) #(b w n h d1)
v = v.permute(0, 3, 2, 1, 4) #(b w n h d2)
qk_mat_h = qr_h @ kr_h.transpose(-1, -2) #(b w n h h)
qk_mat_h = qk_mat_h + mask_h #(b w n h h)
qk_mat_h = torch.softmax(qk_mat_h, -1) #(b w n h h)
output = torch.matmul(qk_mat_h, v) #(b w n h d2)
output = output.permute(0, 3, 1, 2, 4).flatten(-2, -1) #(b h w n*d2)
output = output + lepe
output = self.out_proj(output)
return output
class MaSA(nn.Module):
def __init__(self, embed_dim, num_heads, value_factor=1):
super().__init__()
self.factor = value_factor
self.embed_dim = embed_dim
self.num_heads = num_heads
self.head_dim = self.embed_dim * self.factor // num_heads
self.key_dim = self.embed_dim // num_heads
self.scaling = self.key_dim ** -0.5
self.q_proj = nn.Linear(embed_dim, embed_dim, bias=True)
self.k_proj = nn.Linear(embed_dim, embed_dim, bias=True)
self.v_proj = nn.Linear(embed_dim, embed_dim * self.factor, bias=True)
self.lepe = DWConv2d(embed_dim, 5, 1, 2)
self.out_proj = nn.Linear(embed_dim*self.factor, embed_dim, bias=True)
self.reset_parameters()
def reset_parameters(self):
nn.init.xavier_normal_(self.q_proj.weight, gain=2 ** -2.5)
nn.init.xavier_normal_(self.k_proj.weight, gain=2 ** -2.5)
nn.init.xavier_normal_(self.v_proj.weight, gain=2 ** -2.5)
nn.init.xavier_normal_(self.out_proj.weight)
nn.init.constant_(self.out_proj.bias, 0.0)
def forward(self, x: torch.Tensor, rel_pos, chunkwise_recurrent=False, incremental_state=None):
'''
x: (b h w c)
rel_pos: mask: (n l l)
'''
bsz, h, w, _ = x.size()
mask = rel_pos
assert h*w == mask.size(1)
q = self.q_proj(x)
k = self.k_proj(x)
v = self.v_proj(x)
lepe = self.lepe(v)
k *= self.scaling
qr = q.view(bsz, h, w, self.num_heads, -1).permute(0, 3, 1, 2, 4) #(b n h w d1)
kr = k.view(bsz, h, w, self.num_heads, -1).permute(0, 3, 1, 2, 4) #(b n h w d1)
qr = qr.flatten(2, 3) #(b n l d1)
kr = kr.flatten(2, 3) #(b n l d1)
vr = v.reshape(bsz, h, w, self.num_heads, -1).permute(0, 3, 1, 2, 4) #(b n h w d2)
vr = vr.flatten(2, 3) #(b n l d2)
qk_mat = qr @ kr.transpose(-1, -2) #(b n l l)
qk_mat = qk_mat + mask #(b n l l)
qk_mat = torch.softmax(qk_mat, -1) #(b n l l)
output = torch.matmul(qk_mat, vr) #(b n l d2)
output = output.transpose(1, 2).reshape(bsz, h, w, -1) #(b h w n*d2)
output = output + lepe
output = self.out_proj(output)
return output
class FeedForwardNetwork(nn.Module):
def __init__(
self,
embed_dim,
ffn_dim,
activation_fn=F.gelu,
dropout=0.0,
activation_dropout=0.0,
layernorm_eps=1e-6,
subln=False,
subconv=False
):
super().__init__()
self.embed_dim = embed_dim
self.activation_fn = activation_fn
self.activation_dropout_module = torch.nn.Dropout(activation_dropout)
self.dropout_module = torch.nn.Dropout(dropout)
self.fc1 = nn.Linear(self.embed_dim, ffn_dim)
self.fc2 = nn.Linear(ffn_dim, self.embed_dim)
self.ffn_layernorm = nn.LayerNorm(ffn_dim, eps=layernorm_eps) if subln else None
self.dwconv = DWConv2d(ffn_dim, 3, 1, 1) if subconv else None
def reset_parameters(self):
self.fc1.reset_parameters()
self.fc2.reset_parameters()
if self.ffn_layernorm is not None:
self.ffn_layernorm.reset_parameters()
def forward(self, x: torch.Tensor):
'''
x: (b h w c)
'''
x = self.fc1(x)
x = self.activation_fn(x)
x = self.activation_dropout_module(x)
if self.dwconv is not None:
residual = x
x = self.dwconv(x)
x = x + residual
if self.ffn_layernorm is not None:
x = self.ffn_layernorm(x)
x = self.fc2(x)
x = self.dropout_module(x)
return x
class RetBlock(nn.Module):
def __init__(self, retention: str, embed_dim: int, num_heads: int, ffn_dim: int, drop_path=0., layerscale=False, layer_init_values=1e-5):
super().__init__()
self.layerscale = layerscale
self.embed_dim = embed_dim
self.retention_layer_norm = nn.LayerNorm(self.embed_dim, eps=1e-6)
assert retention in ['chunk', 'whole']
if retention == 'chunk':
self.retention = MaSAd(embed_dim, num_heads)
else:
self.retention = MaSA(embed_dim, num_heads)
self.drop_path = DropPath(drop_path)
self.final_layer_norm = nn.LayerNorm(self.embed_dim, eps=1e-6)
self.ffn = FeedForwardNetwork(embed_dim, ffn_dim)
self.pos = DWConv2d(embed_dim, 3, 1, 1)
if layerscale:
self.gamma_1 = nn.Parameter(layer_init_values * torch.ones(1, 1, 1, embed_dim),requires_grad=True)
self.gamma_2 = nn.Parameter(layer_init_values * torch.ones(1, 1, 1, embed_dim),requires_grad=True)
def forward(
self,
x: torch.Tensor,
incremental_state=None,
chunkwise_recurrent=False,
retention_rel_pos=None
):
x = x + self.pos(x)
if self.layerscale:
x = x + self.drop_path(self.gamma_1 * self.retention(self.retention_layer_norm(x), retention_rel_pos, chunkwise_recurrent, incremental_state))
x = x + self.drop_path(self.gamma_2 * self.ffn(self.final_layer_norm(x)))
else:
x = x + self.drop_path(self.retention(self.retention_layer_norm(x), retention_rel_pos, chunkwise_recurrent, incremental_state))
x = x + self.drop_path(self.ffn(self.final_layer_norm(x)))
return x
class PatchMerging(nn.Module):
r""" Patch Merging Layer.
Args:
input_resolution (tuple[int]): Resolution of input feature.
dim (int): Number of input channels.
norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm
"""
def __init__(self, dim, out_dim, norm_layer=nn.LayerNorm):
super().__init__()
self.dim = dim
self.reduction = nn.Conv2d(dim, out_dim, 3, 2, 1)
self.norm = nn.BatchNorm2d(out_dim)
def forward(self, x):
'''
x: B H W C
'''
x = x.permute(0, 3, 1, 2).contiguous() #(b c h w)
x = self.reduction(x) #(b oc oh ow)
x = self.norm(x)
x = x.permute(0, 2, 3, 1) #(b oh ow oc)
return x
class BasicLayer(nn.Module):
""" A basic Swin Transformer layer for one stage.
Args:
dim (int): Number of input channels.
input_resolution (tuple[int]): Input resolution.
depth (int): Number of blocks.
num_heads (int): Number of attention heads.
window_size (int): Local window size.
mlp_ratio (float): Ratio of mlp hidden dim to embedding dim.
qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True
qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set.
drop (float, optional): Dropout rate. Default: 0.0
attn_drop (float, optional): Attention dropout rate. Default: 0.0
drop_path (float | tuple[float], optional): Stochastic depth rate. Default: 0.0
norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm
downsample (nn.Module | None, optional): Downsample layer at the end of the layer. Default: None
use_checkpoint (bool): Whether to use checkpointing to save memory. Default: False.
fused_window_process (bool, optional): If True, use one kernel to fused window shift & window partition for acceleration, similar for the reversed part. Default: False
"""
def __init__(self, embed_dim, out_dim, depth, num_heads,
init_value: float, heads_range: float,
ffn_dim=96., drop_path=0., norm_layer=nn.LayerNorm, chunkwise_recurrent=False,
downsample: PatchMerging=None, use_checkpoint=False,
layerscale=False, layer_init_values=1e-5):
super().__init__()
self.embed_dim = embed_dim
self.depth = depth
self.use_checkpoint = use_checkpoint
self.chunkwise_recurrent = chunkwise_recurrent
if chunkwise_recurrent:
flag = 'chunk'
else:
flag = 'whole'
self.Relpos = RelPos2d(embed_dim, num_heads, init_value, heads_range)
# build blocks
self.blocks = nn.ModuleList([
RetBlock(flag, embed_dim, num_heads, ffn_dim,
drop_path[i] if isinstance(drop_path, list) else drop_path, layerscale, layer_init_values)
for i in range(depth)])
# patch merging layer
if downsample is not None:
self.downsample = downsample(dim=embed_dim, out_dim=out_dim, norm_layer=norm_layer)
else:
self.downsample = None
def forward(self, x):
b, h, w, d = x.size()
rel_pos = self.Relpos((h, w), chunkwise_recurrent=self.chunkwise_recurrent)
for blk in self.blocks:
if self.use_checkpoint:
tmp_blk = partial(blk, incremental_state=None, chunkwise_recurrent=self.chunkwise_recurrent, retention_rel_pos=rel_pos)
x = checkpoint.checkpoint(tmp_blk, x)
else:
x = blk(x, incremental_state=None, chunkwise_recurrent=self.chunkwise_recurrent, retention_rel_pos=rel_pos)
if self.downsample is not None:
x = self.downsample(x)
return x
class LayerNorm2d(nn.Module):
def __init__(self, dim):
super().__init__()
self.norm = nn.LayerNorm(dim, eps=1e-6)
def forward(self, x: torch.Tensor):
'''
x: (b c h w)
'''
x = x.permute(0, 2, 3, 1).contiguous() #(b h w c)
x = self.norm(x) #(b h w c)
x = x.permute(0, 3, 1, 2).contiguous()
return x
class PatchEmbed(nn.Module):
r""" Image to Patch Embedding
Args:
img_size (int): Image size. Default: 224.
patch_size (int): Patch token size. Default: 4.
in_chans (int): Number of input image channels. Default: 3.
embed_dim (int): Number of linear projection output channels. Default: 96.
norm_layer (nn.Module, optional): Normalization layer. Default: None
"""
def __init__(self, in_chans=3, embed_dim=96, norm_layer=None):
super().__init__()
self.in_chans = in_chans
self.embed_dim = embed_dim
self.proj = nn.Sequential(
nn.Conv2d(in_chans, embed_dim//2, 3, 2, 1),
nn.BatchNorm2d(embed_dim//2),
nn.GELU(),
nn.Conv2d(embed_dim//2, embed_dim//2, 3, 1, 1),
nn.BatchNorm2d(embed_dim//2),
nn.GELU(),
nn.Conv2d(embed_dim//2, embed_dim, 3, 2, 1),
nn.BatchNorm2d(embed_dim),
nn.GELU(),
nn.Conv2d(embed_dim, embed_dim, 3, 1, 1),
nn.BatchNorm2d(embed_dim)
)
def forward(self, x):
B, C, H, W = x.shape
x = self.proj(x).permute(0, 2, 3, 1) #(b h w c)
return x
class VisRetNet(nn.Module):
def __init__(self, in_chans=3, num_classes=1000,
embed_dims=[96, 192, 384, 768], depths=[2, 2, 6, 2], num_heads=[3, 6, 12, 24],
init_values=[1, 1, 1, 1], heads_ranges=[3, 3, 3, 3], mlp_ratios=[3, 3, 3, 3], drop_path_rate=0.1, norm_layer=nn.LayerNorm,
patch_norm=True, use_checkpoints=[False, False, False, False], chunkwise_recurrents=[True, True, False, False],
layerscales=[False, False, False, False], layer_init_values=1e-6):
super().__init__()
self.num_classes = num_classes
self.num_layers = len(depths)
self.embed_dim = embed_dims[0]
self.patch_norm = patch_norm
self.num_features = embed_dims[-1]
self.mlp_ratios = mlp_ratios
# split image into non-overlapping patches
self.patch_embed = PatchEmbed(in_chans=in_chans, embed_dim=embed_dims[0],
norm_layer=norm_layer if self.patch_norm else None)
# stochastic depth
dpr = [x.item() for x in torch.linspace(0, drop_path_rate, sum(depths))] # stochastic depth decay rule
# build layers
self.layers = nn.ModuleList()
for i_layer in range(self.num_layers):
layer = BasicLayer(
embed_dim=embed_dims[i_layer],
out_dim=embed_dims[i_layer+1] if (i_layer < self.num_layers - 1) else None,
depth=depths[i_layer],
num_heads=num_heads[i_layer],
init_value=init_values[i_layer],
heads_range=heads_ranges[i_layer],
ffn_dim=int(mlp_ratios[i_layer]*embed_dims[i_layer]),
drop_path=dpr[sum(depths[:i_layer]):sum(depths[:i_layer + 1])],
norm_layer=norm_layer,
chunkwise_recurrent=chunkwise_recurrents[i_layer],
downsample=PatchMerging if (i_layer < self.num_layers - 1) else None,
use_checkpoint=use_checkpoints[i_layer],
layerscale=layerscales[i_layer],
layer_init_values=layer_init_values
)
self.layers.append(layer)
self.channel = [i.size(1) for i in self.forward(torch.randn(1, 3, 640, 640))]
self.apply(self._init_weights)
def _init_weights(self, m):
if isinstance(m, nn.Linear):
trunc_normal_(m.weight, std=.02)
if isinstance(m, nn.Linear) and m.bias is not None:
nn.init.constant_(m.bias, 0)
elif isinstance(m, nn.LayerNorm):
try:
nn.init.constant_(m.bias, 0)
nn.init.constant_(m.weight, 1.0)
except:
pass
@torch.jit.ignore
def no_weight_decay(self):
return {'absolute_pos_embed'}
@torch.jit.ignore
def no_weight_decay_keywords(self):
return {'relative_position_bias_table'}
def forward(self, x):
input_size = x.size(2)
scale = [4, 8, 16, 32]
features = [None, None, None, None]
x = self.patch_embed(x)
if input_size // x.size(2) in scale:
features[scale.index(input_size // x.size(2))] = x.permute(0, 3, 1, 2)
for layer in self.layers:
x = layer(x)
if input_size // x.size(2) in scale:
features[scale.index(input_size // x.size(2))] = x.permute(0, 3, 1, 2)
return features
def RMT_T():
model = VisRetNet(
embed_dims=[64, 128, 256, 512],
depths=[2, 2, 8, 2],
num_heads=[4, 4, 8, 16],
init_values=[2, 2, 2, 2],
heads_ranges=[4, 4, 6, 6],
mlp_ratios=[3, 3, 3, 3],
drop_path_rate=0.1,
chunkwise_recurrents=[True, True, False, False],
layerscales=[False, False, False, False]
)
model.default_cfg = _cfg()
return model
def RMT_S():
model = VisRetNet(
embed_dims=[64, 128, 256, 512],
depths=[3, 4, 18, 4],
num_heads=[4, 4, 8, 16],
init_values=[2, 2, 2, 2],
heads_ranges=[4, 4, 6, 6],
mlp_ratios=[4, 4, 3, 3],
drop_path_rate=0.15,
chunkwise_recurrents=[True, True, True, False],
layerscales=[False, False, False, False]
)
model.default_cfg = _cfg()
return model
def RMT_B():
model = VisRetNet(
embed_dims=[80, 160, 320, 512],
depths=[4, 8, 25, 8],
num_heads=[5, 5, 10, 16],
init_values=[2, 2, 2, 2],
heads_ranges=[5, 5, 6, 6],
mlp_ratios=[4, 4, 3, 3],
drop_path_rate=0.4,
chunkwise_recurrents=[True, True, True, False],
layerscales=[False, False, True, True],
layer_init_values=1e-6
)
model.default_cfg = _cfg()
return model
def RMT_L():
model = VisRetNet(
embed_dims=[112, 224, 448, 640],
depths=[4, 8, 25, 8],
num_heads=[7, 7, 14, 20],
init_values=[2, 2, 2, 2],
heads_ranges=[6, 6, 6, 6],
mlp_ratios=[4, 4, 3, 3],
drop_path_rate=0.5,
chunkwise_recurrents=[True, True, True, False],
layerscales=[False, False, True, True],
layer_init_values=1e-6
)
model.default_cfg = _cfg()
return model
if __name__ == '__main__':
model = RMT_T()
inputs = torch.randn((1, 3, 640, 640))
res = model(inputs)
for i in res:
print(i.size())
四、修改步骤
4.1 修改一
① 在
ultralytics/nn/
目录下新建
AddModules
文件夹用于存放模块代码
② 在
AddModules
文件夹下新建
RMT.py
，将
第三节
中的代码粘贴到此处
4.2 修改二
在
AddModules
文件夹下新建
__init__.py
（已有则不用新建），在文件内导入模块：
from .RMT import *
4.3 修改三
在
ultralytics/nn/modules/tasks.py
文件中，需要在两处位置添加各模块类名称。
① 首先：导入模块
② 其次：在
parse_model函数
的如下位置添加两行代码：
backbone = False
t=m
③ 接着，在此函数下添加如下代码：
elif m in {RMT_T, RMT_S, RMT_B, RMT_L, }:
m = m(*args)
c2 = m.channel
backbone = True
④ 然后，将下方红框内的代码全部替换：
if isinstance(c2, list):
is_backbone = True
m_ = m
m_.backbone = True
else:
m_ = nn.Sequential(*(m(*args) for _ in range(n))) if n > 1 else m(*args) # module
t = str(m)[8:-2].replace('__main__.', '') # module type
m.np = sum(x.numel() for x in m_.parameters()) # number params
m_.i, m_.f, m_.type = i + 4 if is_backbone else i, f, t # attach index, 'from' index, type
if verbose:
LOGGER.info(f'{i:>3}{str(f):>20}{n_:>3}{m.np:10.0f} {t:<45}{str(args):<30}') # print
save.extend(x % (i + 4 if is_backbone else i) for x in ([f] if isinstance(f, int) else f) if
x != -1) # append to savelist
layers.append(m_)
if i == 0:
ch = []
if isinstance(c2, list):
ch.extend(c2)
for _ in range(5 - len(ch)):
ch.insert(0, 0)
else:
ch.append(c2)
替换后如下：
⑤ 在此文件下找到
base_model
的
_predict_once
，并将其替换成如下代码。
def _predict_once(self, x, profile=False, visualize=False, embed=None):
y, dt, embeddings = [], [], [] # outputs
for m in self.model:
if m.f != -1: # if not from previous layer
x = y[m.f] if isinstance(m.f, int) else [x if j == -1 else y[j] for j in m.f] # from earlier layers
if profile:
self._profile_one_layer(m, x, dt)
if hasattr(m, 'backbone'):
x = m(x)
if len(x) != 5: # 0 - 5
x.insert(0, None)
for index, i in enumerate(x):
if index in self.save:
y.append(i)
else:
y.append(None)
x = x[-1] # 最后一个输出传给下一层
else:
x = m(x) # run
y.append(x if m.i in self.save else None) # save output
if visualize:
feature_visualization(x, m.type, m.i, save_dir=visualize)
if embed and m.i in embed:
embeddings.append(nn.functional.adaptive_avg_pool2d(x, (1, 1)).squeeze(-1).squeeze(-1)) # flatten
if m.i == max(embed):
return torch.unbind(torch.cat(embeddings, 1), dim=0)
return x
至此就修改完成了，可以配置模型开始训练了
五、yaml模型文件
5.1 模型改进⭐
在代码配置完成后，配置模型的YAML文件。
此处以
ultralytics/cfg/models/rt-detr/rtdetr-l.yaml
为例，在同目录下创建一个用于自己数据集训练的模型文件
rtdetr-l-RMT.yaml
。
将
rtdetr-l.yaml
中的内容复制到
rtdetr-l-RMT.yaml
文件下，修改
nc
数量等于自己数据中目标的数量。
📌 模型的修改方法是将
骨干网络
替换成
RMT_T
。
# Ultralytics YOLO 🚀, AGPL-3.0 license
# YOLO11 object detection model with P3-P5 outputs. For Usage examples see https://docs.ultralytics.com/tasks/detect
# Parameters
nc: 1 # number of classes
scales: # model compound scaling constants, i.e. 'model=yolo11n.yaml' will call yolo11.yaml with scale 'n'
# [depth, width, max_channels]
n: [0.50, 0.25, 1024] # summary: 319 layers, 2624080 parameters, 2624064 gradients, 6.6 GFLOPs
s: [0.50, 0.50, 1024] # summary: 319 layers, 9458752 parameters, 9458736 gradients, 21.7 GFLOPs
m: [0.50, 1.00, 512] # summary: 409 layers, 20114688 parameters, 20114672 gradients, 68.5 GFLOPs
l: [1.00, 1.00, 512] # summary: 631 layers, 25372160 parameters, 25372144 gradients, 87.6 GFLOPs
x: [1.00, 1.50, 512] # summary: 631 layers, 56966176 parameters, 56966160 gradients, 196.0 GFLOPs
# YOLO11n backbone
backbone:
# [from, repeats, module, args]
- [-1, 1, RMT_T, []] # 0-4 P1/2
- [-1, 1, SPPF, [1024, 5]] # 5
- [-1, 2, C2PSA, [1024]] # 6
# YOLO11n head
head:
- [-1, 1, nn.Upsample, [None, 2, "nearest"]]
- [[-1, 3], 1, Concat, [1]] # cat backbone P4
- [-1, 2, C3k2, [512, False]] # 9
- [-1, 1, nn.Upsample, [None, 2, "nearest"]]
- [[-1, 2], 1, Concat, [1]] # cat backbone P3
- [-1, 2, C3k2, [256, False]] # 12 (P3/8-small)
- [-1, 1, Conv, [256, 3, 2]]
- [[-1, 9], 1, Concat, [1]] # cat head P4
- [-1, 2, C3k2, [512, False]] # 15 (P4/16-medium)
- [-1, 1, Conv, [512, 3, 2]]
- [[-1, 6], 1, Concat, [1]] # cat head P5
- [-1, 2, C3k2, [1024, True]] # 18 (P5/32-large)
- [[12, 15, 18], 1, Detect, [nc]] # Detect(P3, P4, P5)
六、成功运行结果
分别打印网络模型可以看到
RMT_S
已经加入到模型中，并可以进行训练了。
rtdetr-l-RMT ：
rtdetr-l-RMT summary: 658 layers, 31,244,803 parameters, 31,244,803 gradients, 98.7 GFLOPs
from n params module arguments
0 -1 1 12678176 RMT_T []
1 -1 1 131584 ultralytics.nn.modules.conv.Conv [512, 256, 1, 1, None, 1, 1, False]
2 -1 1 789760 ultralytics.nn.modules.transformer.AIFI [256, 1024, 8]
3 -1 1 66048 ultralytics.nn.modules.conv.Conv [256, 256, 1, 1]
4 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
5 3 1 66048 ultralytics.nn.modules.conv.Conv [256, 256, 1, 1, None, 1, 1, False]
6 [-2, -1] 1 0 ultralytics.nn.modules.conv.Concat [1]
7 -1 3 2232320 ultralytics.nn.modules.block.RepC3 [512, 256, 3]
8 -1 1 66048 ultralytics.nn.modules.conv.Conv [256, 256, 1, 1]
9 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
10 2 1 33280 ultralytics.nn.modules.conv.Conv [128, 256, 1, 1, None, 1, 1, False]
11 [-2, -1] 1 0 ultralytics.nn.modules.conv.Concat [1]
12 -1 3 2232320 ultralytics.nn.modules.block.RepC3 [512, 256, 3]
13 -1 1 590336 ultralytics.nn.modules.conv.Conv [256, 256, 3, 2]
14 [-1, 12] 1 0 ultralytics.nn.modules.conv.Concat [1]
15 -1 3 2232320 ultralytics.nn.modules.block.RepC3 [512, 256, 3]
16 -1 1 590336 ultralytics.nn.modules.conv.Conv [256, 256, 3, 2]
17 [-1, 7] 1 0 ultralytics.nn.modules.conv.Concat [1]
18 -1 3 2232320 ultralytics.nn.modules.block.RepC3 [512, 256, 3]
19 [16, 19, 22] 1 7303907 ultralytics.nn.modules.head.RTDETRDecoder [1, [256, 256, 256]]
rtdetr-l-RMT summary: 658 layers, 31,244,803 parameters, 31,244,803 gradients, 98.7 GFLOPs