【YOLOv13多模态融合改进】| 引入轻量化特征提取模块，解决多模态中的双模型参数量、计算量增加问题（适用不同的轻量化模块）
一、本文介绍
本文记录的是
利用轻量化模块改进 YOLOv13 的多模态目标检测网络模型
。由于多模态模型在训练过程中使用的是两个模型，
整体的参数量计算量相比一般的单模态模型大
，所以
轻量化也是多模态模型改进过程中常见的一个改进方向
。本文介绍如何使用轻量化模块改进
YOLOv13
中的主要特征提取模块。效果如下，也可替换成其它轻量化模块。
| 模型 | 参数量（验证） | 计算量（验证） |
|---|---|---|
| YOLOv13n前期融合 | 2.5 M | 7.0 GFLOPs |
| 轻量后的前期融合 | 1.9 M（ -0.6 ） | 6.3 GFLOPs（ -0.7 ） |
| YOLOv13n中期融合 | 3.5 M | 9.7 GFLOPs |
| 轻量后的中期融合 | 2.5 M（ -1.0 ） | 8.3 GFLOPs（ -1.4 ） |
| YOLOv13n后期融合 | 4.6 M | 12.1 GFLOPs |
| 轻量后的后期融合 | 3.6 M（ -1.0 ） | 10.7 GFLOPs（ -1.4 ） |
二、轻量化模块介绍
本文以EfficientNet中的MBConv为例，介绍其原理，在后续的轻量化过程中，可按照同样的步骤替换成其它轻量化模块。
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
2.1 结构组成
- 逐点卷积（1×1卷积）升维 ：首先通过一个1×1的逐点卷积对输入特征图进行通道数扩展。目的是增加特征的维度，为后续的深度可分离卷积提供更多的特征信息，以便更好地提取特征。
- 深度可分离卷积 ：包括深度卷积（Depthwise Convolution）和逐点卷积（Pointwise Convolution）。深度卷积是逐通道进行的卷积运算，每个卷积核负责一个通道，它可以在不增加太多计算量的情况下，提取特征的空间信息。之后的逐点卷积则是在通道维度上对深度卷积产生的特征图进行加权运算，两者结合可有效降低模型的计算量与参数量。
- SE模块（Squeeze-and-Excitation） ：有助于模型在通道维度上对重要的特征信息产生更多的关注。
- 逐点卷积（1×1卷积）降维 ：最后再通过一个1×1的逐点卷积将特征图的通道数恢复到与输入相近的维度，实现特征的融合和压缩，减少模型的参数量和计算量。
- Shortcut连接 ：当输入MBConv结构的特征矩阵与输出的特征矩阵shape相同时存在shortcut连接，将输入直接与经过上述卷积操作后的输出相加，实现特征的复用，有助于解决梯度消失问题，使网络更容易训练。
2.2 工作原理
- 特征提取 ：先利用1×1卷积升维扩展通道，让网络有更多的维度去学习特征。然后深度可分离卷积中的深度卷积负责提取空间特征，逐点卷积负责融合通道特征。
- 注意力机制 ：SE模块通过对通道特征进行加权，使得网络能够自动关注到更重要的特征通道，抑制不重要的通道，从而提升模型的特征表达能力。
- 特征融合与输出 ：最后的1×1卷积降维将特征进行融合和压缩，得到最终的输出特征图。有shortcut连接时，将输入特征与输出特征相加，使网络能够更好地学习到输入与输出之间的映射关系。
2.3 优势
- 高效的计算性能 ：深度可分离卷积的使用大大减少了计算量，相比传统的卷积操作，能在降低计算成本的同时保持较好的特征提取能力，适用于移动设备等计算资源有限的场景。
- 强大的特征表达能力 ：通过倒置瓶颈结构，先升维再降维，以及SE模块的注意力机制，能够更有效地提取和利用特征，提高模型的准确性和泛化能力。
- 轻量化模型 ：减少了模型的参数量，降低了模型的存储需求和过拟合的风险，使模型更加轻量化，便于部署和应用。
论文： https://arxiv.org/pdf/1905.11946
源码： https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet
三、轻量化改进的实现代码
实现代码如下：
import math
import torch
import torch.nn as nn
from ultralytics.nn.modules.conv import DWConv
from ultralytics.utils.tal import dist2bbox, make_anchors
class SeBlock(nn.Module):
def __init__(self, in_channel, reduction=4):
super().__init__()
self.Squeeze = nn.AdaptiveAvgPool2d(1)
self.Excitation = nn.Sequential()
self.Excitation.add_module('FC1', nn.Conv2d(in_channel, in_channel // reduction, kernel_size=1)) # 1*1卷积与此效果相同
self.Excitation.add_module('ReLU', nn.ReLU())
self.Excitation.add_module('FC2', nn.Conv2d(in_channel // reduction, in_channel, kernel_size=1))
self.Excitation.add_module('Sigmoid', nn.Sigmoid())
def forward(self, x):
y = self.Squeeze(x)
ouput = self.Excitation(y)
return x*(ouput.expand_as(x))
class drop_connect:
def __init__(self, drop_connect_rate):
self.drop_connect_rate = drop_connect_rate
def forward(self, x, training):
if not training:
return x
keep_prob = 1.0 - self.drop_connect_rate
batch_size = x.shape[0]
random_tensor = keep_prob
random_tensor += torch.rand([batch_size, 1, 1, 1], dtype=x.dtype, device=x.device)
binary_mask = torch.floor(random_tensor) # 1
x = (x / keep_prob) * binary_mask
return x
class stem(nn.Module):
def __init__(self, c1, c2, act='ReLU6'):
super().__init__()
self.conv = nn.Conv2d(c1, c2, kernel_size=3, stride=2, padding=1, bias=False)
self.bn = nn.BatchNorm2d(num_features=c2)
if act == 'ReLU6':
self.act = nn.ReLU6(inplace=True)
def forward(self, x):
return self.act(self.bn(self.conv(x)))
class MBConvBlock(nn.Module):
def __init__(self, inp, final_oup, k=3, s=1, expand_ratio=1, drop_connect_rate=0.057, has_se=False):
super(MBConvBlock, self).__init__()
self._momentum = 0.01
self._epsilon = 1e-3
self.input_filters = inp
self.output_filters = final_oup
self.stride = s
self.expand_ratio = expand_ratio
self.has_se = has_se
self.id_skip = True # skip connection and drop connect
se_ratio = 0.25
# Expansion phase
oup = inp * expand_ratio # number of output channels
if expand_ratio != 1:
self._expand_conv = nn.Conv2d(in_channels=inp, out_channels=oup, kernel_size=1, bias=False)
self._bn0 = nn.BatchNorm2d(num_features=oup, momentum=self._momentum, eps=self._epsilon)
# Depthwise convolution phase
self._depthwise_conv = nn.Conv2d(
in_channels=oup, out_channels=oup, groups=oup, # groups makes it depthwise
kernel_size=k, padding=(k - 1) // 2, stride=s, bias=False)
self._bn1 = nn.BatchNorm2d(num_features=oup, momentum=self._momentum, eps=self._epsilon)
# Squeeze and Excitation layer, if desired
if self.has_se:
num_squeezed_channels = max(1, int(inp * se_ratio))
self.se = SeBlock(oup, 4)
# Output phase
self._project_conv = nn.Conv2d(in_channels=oup, out_channels=final_oup, kernel_size=1, bias=False)
self._bn2 = nn.BatchNorm2d(num_features=final_oup, momentum=self._momentum, eps=self._epsilon)
self._relu = nn.ReLU6(inplace=True)
self.drop_connect = drop_connect(drop_connect_rate)
def forward(self, x, drop_connect_rate=None):
"""
:param x: input tensor
:param drop_connect_rate: drop connect rate (float, between 0 and 1)
:return: output of block
"""
# Expansion and Depthwise Convolution
identity = x
if self.expand_ratio != 1:
x = self._relu(self._bn0(self._expand_conv(x)))
x = self._relu(self._bn1(self._depthwise_conv(x)))
# Squeeze and Excitation
if self.has_se:
x = self.se(x)
x = self._bn2(self._project_conv(x))
# Skip connection and drop connect
if self.id_skip and self.stride == 1 and self.input_filters == self.output_filters:
if drop_connect_rate:
x = self.drop_connect(x, training=self.training)
x += identity # skip connection
return x
def autopad(k, p=None, d=1): # kernel, padding, dilation
"""Pad to 'same' shape outputs."""
if d > 1:
k = d * (k - 1) + 1 if isinstance(k, int) else [d * (x - 1) + 1 for x in k] # actual kernel-size
if p is None:
p = k // 2 if isinstance(k, int) else [x // 2 for x in k] # auto-pad
return p
class Conv(nn.Module):
"""Standard convolution with args(ch_in, ch_out, kernel, stride, padding, groups, dilation, activation)."""
default_act = nn.SiLU() # default activation
def __init__(self, c1, c2, k=1, s=1, p=None, g=1, d=1, act=True):
"""Initialize Conv layer with given arguments including activation."""
super().__init__()
self.conv = nn.Conv2d(c1, c2, k, s, autopad(k, p, d), groups=g, dilation=d, bias=False)
self.bn = nn.BatchNorm2d(c2)
self.act = self.default_act if act is True else act if isinstance(act, nn.Module) else nn.Identity()
def forward(self, x):
"""Apply convolution, batch normalization and activation to input tensor."""
return self.act(self.bn(self.conv(x)))
def forward_fuse(self, x):
"""Perform transposed convolution of 2D data."""
return self.act(self.conv(x))
class Bottleneck(nn.Module):
"""Standard bottleneck."""
def __init__(self, c1, c2, shortcut=True, g=1, k=(3, 3), e=0.5):
"""Initializes a standard bottleneck module with optional shortcut connection and configurable parameters."""
super().__init__()
c_ = int(c2 * e) # hidden channels
self.cv1 = Conv(c1, c_, k[0], 1)
self.cv2 = Conv(c_, c2, k[1], 1, g=g)
self.add = shortcut and c1 == c2
def forward(self, x):
"""Applies the YOLO FPN to input data."""
return x + self.cv2(self.cv1(x)) if self.add else self.cv2(self.cv1(x))
class C2f(nn.Module):
"""Faster Implementation of CSP Bottleneck with 2 convolutions."""
def __init__(self, c1, c2, n=1, shortcut=False, g=1, e=0.5):
"""Initializes a CSP bottleneck with 2 convolutions and n Bottleneck blocks for faster processing."""
super().__init__()
self.c = int(c2 * e) # hidden channels
self.cv1 = Conv(c1, 2 * self.c, 1, 1)
self.cv2 = Conv((2 + n) * self.c, c2, 1) # optional act=FReLU(c2)
self.m = nn.ModuleList(Bottleneck(self.c, self.c, shortcut, g, k=((3, 3), (3, 3)), e=1.0) for _ in range(n))
def forward(self, x):
"""Forward pass through C2f layer."""
y = list(self.cv1(x).chunk(2, 1))
y.extend(m(y[-1]) for m in self.m)
return self.cv2(torch.cat(y, 1))
def forward_split(self, x):
"""Forward pass using split() instead of chunk()."""
y = list(self.cv1(x).split((self.c, self.c), 1))
y.extend(m(y[-1]) for m in self.m)
return self.cv2(torch.cat(y, 1))
class C3(nn.Module):
"""CSP Bottleneck with 3 convolutions."""
def __init__(self, c1, c2, n=1, shortcut=True, g=1, e=0.5):
"""Initialize the CSP Bottleneck with given channels, number, shortcut, groups, and expansion values."""
super().__init__()
c_ = int(c2 * e) # hidden channels
self.cv1 = Conv(c1, c_, 1, 1)
self.cv2 = Conv(c1, c_, 1, 1)
self.cv3 = Conv(2 * c_, c2, 1) # optional act=FReLU(c2)
self.m = nn.Sequential(*(Bottleneck(c_, c_, shortcut, g, k=((1, 1), (3, 3)), e=1.0) for _ in range(n)))
def forward(self, x):
"""Forward pass through the CSP bottleneck with 2 convolutions."""
return self.cv3(torch.cat((self.m(self.cv1(x)), self.cv2(x)), 1))
class C3k(C3):
"""C3k is a CSP bottleneck module with customizable kernel sizes for feature extraction in neural networks."""
def __init__(self, c1, c2, n=1, shortcut=True, g=1, e=0.5, k=3):
"""Initializes the C3k module with specified channels, number of layers, and configurations."""
super().__init__(c1, c2, n, shortcut, g, e)
c_ = int(c2 * e) # hidden channels
# self.m = nn.Sequential(*(RepBottleneck(c_, c_, shortcut, g, k=(k, k), e=1.0) for _ in range(n)))
self.m = nn.Sequential(*(Bottleneck(c_, c_, shortcut, g, k=(k, k), e=1.0) for _ in range(n)))
class C3k2_MBConv(C2f):
"""Faster Implementation of CSP Bottleneck with 2 convolutions."""
def __init__(self, c1, c2, n=1, c3k=False, e=0.5, g=1, shortcut=True):
"""Initializes the C3k2 module, a faster CSP Bottleneck with 2 convolutions and optional C3k blocks."""
super().__init__(c1, c2, n, shortcut, g, e)
self.m = nn.ModuleList(
C3k(self.c, self.c, 2, shortcut, g) if c3k else MBConvBlock(self.c, self.c) for _ in range(n)
)
class A2C2f_MBConv(nn.Module):
"""
Area-Attention C2f module for enhanced feature extraction with area-based attention mechanisms.
This module extends the C2f architecture by incorporating area-attention and ABlock layers for improved feature
processing. It supports both area-attention and standard convolution modes.
Attributes:
cv1 (Conv): Initial 1x1 convolution layer that reduces input channels to hidden channels.
cv2 (Conv): Final 1x1 convolution layer that processes concatenated features.
gamma (nn.Parameter | None): Learnable parameter for residual scaling when using area attention.
m (nn.ModuleList): List of either ABlock or C3k modules for feature processing.
Methods:
forward: Processes input through area-attention or standard convolution pathway.
Examples:
>>> m = A2C2f(512, 512, n=1, a2=True, area=1)
>>> x = torch.randn(1, 512, 32, 32)
>>> output = m(x)
>>> print(output.shape)
torch.Size([1, 512, 32, 32])
"""
def __init__(self, c1, c2, n=1, a2=True, area=1, residual=False, mlp_ratio=2.0, e=0.5, g=1, shortcut=True):
"""
Initialize Area-Attention C2f module.
Args:
c1 (int): Number of input channels.
c2 (int): Number of output channels.
n (int): Number of ABlock or C3k modules to stack.
a2 (bool): Whether to use area attention blocks. If False, uses C3k blocks instead.
area (int): Number of areas the feature map is divided.
residual (bool): Whether to use residual connections with learnable gamma parameter.
mlp_ratio (float): Expansion ratio for MLP hidden dimension.
e (float): Channel expansion ratio for hidden channels.
g (int): Number of groups for grouped convolutions.
shortcut (bool): Whether to use shortcut connections in C3k blocks.
"""
super().__init__()
c_ = int(c2 * e) # hidden channels
assert c_ % 32 == 0, "Dimension of ABlock be a multiple of 32."
self.cv1 = Conv(c1, c_, 1, 1)
self.cv2 = Conv((1 + n) * c_, c2, 1)
self.gamma = nn.Parameter(0.01 * torch.ones(c2), requires_grad=True) if a2 and residual else None
self.m = nn.ModuleList(
nn.Sequential(*(MBConvBlock(c_, c_) for _ in range(2)))
if a2
else C3k(c_, c_, 2, shortcut, g)
for _ in range(n)
)
def forward(self, x):
"""
Forward pass through A2C2f layer.
Args:
x (torch.Tensor): Input tensor.
Returns:
(torch.Tensor): Output tensor after processing.
"""
y = [self.cv1(x)]
y.extend(m(y[-1]) for m in self.m)
y = self.cv2(torch.cat(y, 1))
if self.gamma is not None:
return x + self.gamma.view(-1, len(self.gamma), 1, 1) * y
return y
四、改进点
对
YOLOv12
中的
A2C2f模块
进行改进，并将
MBConv
在加入到
A2C2f
模块中。
(第五节讲解添加步骤)
改进代码如下：
对
A2C2f
模块进行改进，加入
MBConv模块
。，替换原本的
ABlock
class A2C2f_MBConv(nn.Module):
"""
Area-Attention C2f module for enhanced feature extraction with area-based attention mechanisms.
This module extends the C2f architecture by incorporating area-attention and ABlock layers for improved feature
processing. It supports both area-attention and standard convolution modes.
Attributes:
cv1 (Conv): Initial 1x1 convolution layer that reduces input channels to hidden channels.
cv2 (Conv): Final 1x1 convolution layer that processes concatenated features.
gamma (nn.Parameter | None): Learnable parameter for residual scaling when using area attention.
m (nn.ModuleList): List of either ABlock or C3k modules for feature processing.
Methods:
forward: Processes input through area-attention or standard convolution pathway.
Examples:
>>> m = A2C2f(512, 512, n=1, a2=True, area=1)
>>> x = torch.randn(1, 512, 32, 32)
>>> output = m(x)
>>> print(output.shape)
torch.Size([1, 512, 32, 32])
"""
def __init__(self, c1, c2, n=1, a2=True, area=1, residual=False, mlp_ratio=2.0, e=0.5, g=1, shortcut=True):
"""
Initialize Area-Attention C2f module.
Args:
c1 (int): Number of input channels.
c2 (int): Number of output channels.
n (int): Number of ABlock or C3k modules to stack.
a2 (bool): Whether to use area attention blocks. If False, uses C3k blocks instead.
area (int): Number of areas the feature map is divided.
residual (bool): Whether to use residual connections with learnable gamma parameter.
mlp_ratio (float): Expansion ratio for MLP hidden dimension.
e (float): Channel expansion ratio for hidden channels.
g (int): Number of groups for grouped convolutions.
shortcut (bool): Whether to use shortcut connections in C3k blocks.
"""
super().__init__()
c_ = int(c2 * e) # hidden channels
assert c_ % 32 == 0, "Dimension of ABlock be a multiple of 32."
self.cv1 = Conv(c1, c_, 1, 1)
self.cv2 = Conv((1 + n) * c_, c2, 1)
self.gamma = nn.Parameter(0.01 * torch.ones(c2), requires_grad=True) if a2 and residual else None
self.m = nn.ModuleList(
nn.Sequential(*(MBConvBlock(c_, c_) for _ in range(2)))
if a2
else C3k(c_, c_, 2, shortcut, g)
for _ in range(n)
)
def forward(self, x):
"""
Forward pass through A2C2f layer.
Args:
x (torch.Tensor): Input tensor.
Returns:
(torch.Tensor): Output tensor after processing.
"""
y = [self.cv1(x)]
y.extend(m(y[-1]) for m in self.m)
y = self.cv2(torch.cat(y, 1))
if self.gamma is not None:
return x + self.gamma.view(-1, len(self.gamma), 1, 1) * y
return y
五、配置步骤
5.1 修改一
① 在
ultralytics/nn/
目录下新建
AddModules
文件夹用于存放模块代码（提供的项目包里已存在
AddModules
文件夹）
② 在
AddModules
文件夹下新建
MBConv.py
，将
第三节
中的代码粘贴到此处
5.2 修改二
在
AddModules
文件夹下新建
__init__.py
（提供的项目包里已存在
__init__.py
），在文件内导入模块：
from .MBConv import *
5.3 修改三
在
ultralytics/nn/modules/tasks.py
文件中，需要在两处位置添加各模块类名称。
首先：导入模块
其次：在
parse_model函数
中注册
A2C2f_MBConv
模块
六、yaml模型文件
！！！ 获取的项目包就已经把相关的多模态输入、训练等改动都已经配好了，只需要新建模型yaml文件，粘贴对应的模型，进行训练即可。 项目包获取及使用教程可参考链接： 《YOLO系列模型的多模态项目》配置使用教程
在什么地方新建，n,s,m,l,x,用哪个版本按自己的需求来即可，和普通的训练步骤一致。
除了模型结构方面的改动，在yaml文件中还传入了一个通道数
ch: 6
表示传入的是双模态，6通道 ，前三个是可见光，后三个是红外。
在default.yaml中也配置了这个参数。
下面便是在前期、中期、中后期、后期融合中添加
A2C2f_MBConv
6.1 前期融合
ch: 6
nc: 80 # number of classes
scales: # model compound scaling constants, i.e. 'model=yolov13n.yaml' will call yolov13.yaml with scale 'n'
# [depth, width, max_channels]
n: [0.50, 0.25, 1024] # Nano
s: [0.50, 0.50, 1024] # Small
l: [1.00, 1.00, 512] # Large
x: [1.00, 1.50, 512] # Extra Large
backbone:
# [from, repeats, module, args]
- [-1, 1, MF, [64]] # 0
- [-1, 1, Conv, [64, 3, 2]] # 1-P1/2
- [-1, 1, Conv, [128, 3, 2, 1, 2]] # 2-P2/4
- [-1, 2, DSC3k2, [256, False, 0.25]]
- [-1, 1, Conv, [256, 3, 2, 1, 4]] # 4-P3/8
- [-1, 2, DSC3k2, [512, False, 0.25]]
- [-1, 1, DSConv, [512, 3, 2]] # 6-P4/16
- [-1, 4, A2C2f_MBConv, [512, True, 4]]
- [-1, 1, DSConv, [1024, 3, 2]] # 8-P5/32
- [-1, 4, A2C2f_MBConv, [1024, True, 1]] # 9
head:
- [[5, 7, 9], 2, HyperACE, [512, 8, True, True, 0.5, 1, "both"]]
- [-1, 1, nn.Upsample, [None, 2, "nearest"]]
- [ 10, 1, DownsampleConv, []]
- [[7, 10], 1, FullPAD_Tunnel, []] #13
- [[5, 11], 1, FullPAD_Tunnel, []] #14
- [[9, 12], 1, FullPAD_Tunnel, []] #15
- [-1, 1, nn.Upsample, [None, 2, "nearest"]]
- [[-1, 13], 1, Concat, [1]] # cat backbone P4
- [-1, 2, DSC3k2, [512, True]] # 18
- [[-1, 10], 1, FullPAD_Tunnel, []] #19
- [18, 1, nn.Upsample, [None, 2, "nearest"]]
- [[-1, 14], 1, Concat, [1]] # cat backbone P3
- [-1, 2, DSC3k2, [256, True]] # 22
- [11, 1, Conv, [256, 1, 1]]
- [[22, 23], 1, FullPAD_Tunnel, []] #24
- [-1, 1, Conv, [256, 3, 2]]
- [[-1, 19], 1, Concat, [1]] # cat head P4
- [-1, 2, DSC3k2, [512, True]] # 27
- [[-1, 10], 1, FullPAD_Tunnel, []]
- [27, 1, Conv, [512, 3, 2]]
- [[-1, 15], 1, Concat, [1]] # cat head P5
- [-1, 2, DSC3k2, [1024,True]] # 31 (P5/32-large)
- [[-1, 12], 1, FullPAD_Tunnel, []]
- [[24, 28, 32], 1, Detect, [nc]] # Detect(P3, P4, P5)
6.2 中期融合
ch: 6
nc: 80 # number of classes
scales: # model compound scaling constants, i.e. 'model=yolov13n.yaml' will call yolov13.yaml with scale 'n'
# [depth, width, max_channels]
n: [0.50, 0.25, 1024] # Nano
s: [0.50, 0.50, 1024] # Small
l: [1.00, 1.00, 512] # Large
x: [1.00, 1.50, 512] # Extra Large
backbone:
# [from, repeats, module, args]
- [-1, 1, IN, []] # 0
- [-1, 1, Multiin, [1]] # 1
- [-2, 1, Multiin, [2]] # 2
- [1, 1, Conv, [64, 3, 2]] # 3-P1/2
- [-1, 1, Conv, [128, 3, 2, 1, 2]] # 4-P2/4
- [-1, 2, DSC3k2, [256, False, 0.25]]
- [-1, 1, Conv, [256, 3, 2, 1, 4]] # 6-P3/8
- [-1, 2, DSC3k2, [512, True]]
- [-1, 1, DSConv, [512, 3, 2]] # 8-P4/16
- [-1, 4, A2C2f_MBConv, [512, True, 4]]
- [-1, 1, DSConv, [1024, 3, 2]] # 10-P5/32
- [-1, 4, A2C2f_MBConv, [1024, True, 1]] # 11
- [2, 1, Conv, [64, 3, 2]] # 12-P1/2
- [-1, 1, Conv, [128, 3, 2, 1, 2]] # 13-P2/4
- [-1, 2, DSC3k2, [256, False, 0.25]]
- [-1, 1, Conv, [256, 3, 2, 1, 4]] # 15-P3/8
- [-1, 2, DSC3k2, [512, True]]
- [-1, 1, DSConv, [512, 3, 2]] # 17-P4/16
- [-1, 4, A2C2f_MBConv, [512, True, 4]]
- [-1, 1, DSConv, [1024, 3, 2]] # 19-P5/32
- [-1, 4, A2C2f_MBConv, [1024, True, 1]] # 20
- [[7, 16], 1, Add, [1]] # 21 cat backbone P3
- [[9, 18], 1, Add, [1]] # 22 cat backbone P4
- [[11, 20], 1, Add, [1]] # 23 cat backbone P5
head:
- [[21, 22, 23], 2, HyperACE, [512, 8, True, True, 0.5, 1, "both"]]
- [-1, 1, nn.Upsample, [None, 2, "nearest"]]
- [ 24, 1, DownsampleConv, []]
- [[22, 24], 1, FullPAD_Tunnel, []] # 27
- [[21, 25], 1, FullPAD_Tunnel, []] # 28
- [[23, 26], 1, FullPAD_Tunnel, []] # 29
- [-1, 1, nn.Upsample, [None, 2, "nearest"]]
- [[-1, 27], 1, Concat, [1]] # cat backbone P4
- [-1, 2, DSC3k2, [512, True]] # 32
- [[-1, 24], 1, FullPAD_Tunnel, []] # 33
- [32, 1, nn.Upsample, [None, 2, "nearest"]]
- [[-1, 28], 1, Concat, [1]] # cat backbone P3
- [-1, 2, DSC3k2, [256, True]] # 36
- [25, 1, Conv, [256, 1, 1]]
- [[36, 37], 1, FullPAD_Tunnel, []] # 38
- [-1, 1, Conv, [256, 3, 2]]
- [[-1, 33], 1, Concat, [1]] # cat head P4
- [-1, 2, DSC3k2, [512, True]] # 41
- [[-1, 24], 1, FullPAD_Tunnel, []]
- [41, 1, Conv, [512, 3, 2]]
- [[-1, 29], 1, Concat, [1]] # cat head P5
- [-1, 2, DSC3k2, [1024,True]] # 45 (P5/32-large)
- [[-1, 26], 1, FullPAD_Tunnel, []]
- [[38, 42, 46], 1, Detect, [nc]] # Detect(P3, P4, P5)
6.3 后期融合
ch: 6
nc: 80 # number of classes
scales: # model compound scaling constants, i.e. 'model=yolov13n.yaml' will call yolov13.yaml with scale 'n'
# [depth, width, max_channels]
n: [0.50, 0.25, 1024] # Nano
s: [0.50, 0.50, 1024] # Small
l: [1.00, 1.00, 512] # Large
x: [1.00, 1.50, 512] # Extra Large
backbone:
# [from, repeats, module, args]
- [-1, 1, IN, []] # 0
- [-1, 1, Multiin, [1]] # 1
- [-2, 1, Multiin, [2]] # 2
- [1, 1, Conv, [64, 3, 2]] # 3-P1/2
- [-1, 1, Conv, [128, 3, 2, 1, 2]] # 4-P2/4
- [-1, 2, DSC3k2, [256, False, 0.25]]
- [-1, 1, Conv, [256, 3, 2, 1, 4]] # 6-P3/8
- [-1, 2, DSC3k2, [512, True]]
- [-1, 1, DSConv, [512, 3, 2]] # 8-P4/16
- [-1, 4, A2C2f_MBConv, [512, True, 4]]
- [-1, 1, DSConv, [1024, 3, 2]] # 10-P5/32
- [-1, 4, A2C2f_MBConv, [1024, True, 1]] # 11
- [2, 1, Conv, [64, 3, 2]] # 12-P1/2
- [-1, 1, Conv, [128, 3, 2, 1, 2]] # 13-P2/4
- [-1, 2, DSC3k2, [256, False, 0.25]]
- [-1, 1, Conv, [256, 3, 2, 1, 4]] # 15-P3/8
- [-1, 2, DSC3k2, [512, True]]
- [-1, 1, DSConv, [512, 3, 2]] # 17-P4/16
- [-1, 4, A2C2f_MBConv, [512, True, 4]]
- [-1, 1, DSConv, [1024, 3, 2]] # 19-P5/32
- [-1, 4, A2C2f_MBConv, [1024, True, 1]] # 20
head:
- [[7, 9, 11], 2, HyperACE, [512, 8, True, True, 0.5, 1, "both"]]
- [-1, 1, nn.Upsample, [None, 2, "nearest"]]
- [ 21, 1, DownsampleConv, []]
- [[9, 21], 1, FullPAD_Tunnel, []] # 24
- [[7, 22], 1, FullPAD_Tunnel, []] # 25
- [[11, 23], 1, FullPAD_Tunnel, []] # 26
- [-1, 1, nn.Upsample, [None, 2, "nearest"]]
- [[-1, 24], 1, Concat, [1]] # cat backbone P4
- [-1, 2, DSC3k2, [512, True]] # 29
- [[-1, 21], 1, FullPAD_Tunnel, []] # 30
- [29, 1, nn.Upsample, [None, 2, "nearest"]]
- [[-1, 25], 1, Concat, [1]] # cat backbone P3
- [-1, 2, DSC3k2, [256, True]] # 33
- [22, 1, Conv, [256, 1, 1]]
- [[33, 34], 1, FullPAD_Tunnel, []] # 35
- [-1, 1, Conv, [256, 3, 2]]
- [[-1, 30], 1, Concat, [1]] # cat head P4
- [-1, 2, DSC3k2, [512, True]] # 38
- [[-1, 21], 1, FullPAD_Tunnel, []]
- [38, 1, Conv, [512, 3, 2]]
- [[-1, 26], 1, Concat, [1]] # cat head P5
- [-1, 2, DSC3k2, [1024,True]] # 42 (P5/32-large)
- [[-1, 23], 1, FullPAD_Tunnel, []]
- [[16, 18, 20], 2, HyperACE, [512, 8, True, True, 0.5, 1, "both"]]
- [-1, 1, nn.Upsample, [None, 2, "nearest"]]
- [ 44, 1, DownsampleConv, []]
- [[18, 44], 1, FullPAD_Tunnel, []] # 47
- [[16, 45], 1, FullPAD_Tunnel, []] # 48
- [[20, 46], 1, FullPAD_Tunnel, []] # 49
- [-1, 1, nn.Upsample, [None, 2, "nearest"]]
- [[-1, 47], 1, Concat, [1]] # cat backbone P4
- [-1, 2, DSC3k2, [512, True]] # 52
- [[-1, 44], 1, FullPAD_Tunnel, []] # 53
- [52, 1, nn.Upsample, [None, 2, "nearest"]]
- [[-1, 48], 1, Concat, [1]] # cat backbone P3
- [-1, 2, DSC3k2, [256, True]] # 56
- [45, 1, Conv, [256, 1, 1]]
- [[56, 57], 1, FullPAD_Tunnel, []] # 58
- [-1, 1, Conv, [256, 3, 2]]
- [[-1, 53], 1, Concat, [1]] # cat head P4
- [-1, 2, DSC3k2, [512, True]] # 61
- [[-1, 44], 1, FullPAD_Tunnel, []]
- [61, 1, Conv, [512, 3, 2]]
- [[-1, 49], 1, Concat, [1]] # cat head P5
- [-1, 2, DSC3k2, [1024,True]] # 65 (P5/32-large)
- [[-1, 46], 1, FullPAD_Tunnel, []]
- [[35, 58], 1, Add, [1]] # 67 cat backbone P3
- [[39, 62], 1, Add, [1]] # 68 cat backbone P4
- [[43, 66], 1, Add, [1]] # 69 cat backbone P5
- [[67, 68, 69], 1, Detect, [nc]] # Detect(P3, P4, P5)
七、成功运行结果
前期融合结果： 可以看到输入的通道数为6，表明可见光图像和红外图像均输入到了模型中进行融合训练。
YOLOv13-early-MBConv summary: 520 layers, 1,952,138 parameters, 1,952,122 gradients, 6.4 GFLOPs
from n params module arguments
0 -1 1 472 ultralytics.nn.AddModules.multimodal.MF [6, 16]
1 -1 1 2336 ultralytics.nn.modules.conv.Conv [16, 16, 3, 2]
2 -1 1 2368 ultralytics.nn.modules.conv.Conv [16, 32, 3, 2, 1, 2]
3 -1 1 5792 ultralytics.nn.modules.block.DSC3k2 [32, 64, 1, False, 0.25]
4 -1 1 9344 ultralytics.nn.modules.conv.Conv [64, 64, 3, 2, 1, 4]
5 -1 1 20800 ultralytics.nn.modules.block.DSC3k2 [64, 128, 1, False, 0.25]
6 -1 1 17792 ultralytics.nn.modules.conv.DSConv [128, 128, 3, 2]
7 -1 2 69632 ultralytics.nn.AddModules.MBConv.A2C2f_MBConv[128, 128, True, 4]
8 -1 1 34432 ultralytics.nn.modules.conv.DSConv [128, 256, 3, 2]
9 -1 2 270336 ultralytics.nn.AddModules.MBConv.A2C2f_MBConv[256, 256, True, 1]
10 [5, 7, 9] 1 273536 ultralytics.nn.modules.block.HyperACE [128, 128, 1, 4, True, True, 0.5, 1, 'both']
11 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
12 10 1 33280 ultralytics.nn.modules.block.DownsampleConv [128]
13 [7, 10] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
14 [5, 11] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
15 [9, 12] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
16 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
17 [-1, 13] 1 0 ultralytics.nn.modules.conv.Concat [1]
18 -1 1 115328 ultralytics.nn.modules.block.DSC3k2 [384, 128, 1, True]
19 [-1, 10] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
20 18 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
21 [-1, 14] 1 0 ultralytics.nn.modules.conv.Concat [1]
22 -1 1 35136 ultralytics.nn.modules.block.DSC3k2 [256, 64, 1, True]
23 11 1 8320 ultralytics.nn.modules.conv.Conv [128, 64, 1, 1]
24 [22, 23] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
25 -1 1 36992 ultralytics.nn.modules.conv.Conv [64, 64, 3, 2]
26 [-1, 19] 1 0 ultralytics.nn.modules.conv.Concat [1]
27 -1 1 90752 ultralytics.nn.modules.block.DSC3k2 [192, 128, 1, True]
28 [-1, 10] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
29 27 1 147712 ultralytics.nn.modules.conv.Conv [128, 128, 3, 2]
30 [-1, 15] 1 0 ultralytics.nn.modules.conv.Concat [1]
31 -1 1 345344 ultralytics.nn.modules.block.DSC3k2 [384, 256, 1, True]
32 [-1, 12] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
33 [24, 28, 32] 1 432427 ultralytics.nn.modules.head.Detect [9, [64, 128, 256]]
YOLOv13-early-MBConv summary: 520 layers, 1,952,138 parameters, 1,952,122 gradients, 6.4 GFLOPs
中期融合结果：
YOLOv13-mid-MBConv summary: 700 layers, 2,487,890 parameters, 2,487,874 gradients, 8.5 GFLOPs
from n params module arguments
0 -1 1 0 ultralytics.nn.AddModules.multimodal.IN []
1 -1 1 0 ultralytics.nn.AddModules.multimodal.Multiin [1]
2 -2 1 0 ultralytics.nn.AddModules.multimodal.Multiin [2]
3 1 1 464 ultralytics.nn.modules.conv.Conv [3, 16, 3, 2]
4 -1 1 2368 ultralytics.nn.modules.conv.Conv [16, 32, 3, 2, 1, 2]
5 -1 1 5792 ultralytics.nn.modules.block.DSC3k2 [32, 64, 1, False, 0.25]
6 -1 1 9344 ultralytics.nn.modules.conv.Conv [64, 64, 3, 2, 1, 4]
7 -1 1 74368 ultralytics.nn.modules.block.DSC3k2 [64, 128, 1, True]
8 -1 1 17792 ultralytics.nn.modules.conv.DSConv [128, 128, 3, 2]
9 -1 2 69632 ultralytics.nn.AddModules.MBConv.A2C2f_MBConv[128, 128, True, 4]
10 -1 1 34432 ultralytics.nn.modules.conv.DSConv [128, 256, 3, 2]
11 -1 2 270336 ultralytics.nn.AddModules.MBConv.A2C2f_MBConv[256, 256, True, 1]
12 2 1 464 ultralytics.nn.modules.conv.Conv [3, 16, 3, 2]
13 -1 1 2368 ultralytics.nn.modules.conv.Conv [16, 32, 3, 2, 1, 2]
14 -1 1 5792 ultralytics.nn.modules.block.DSC3k2 [32, 64, 1, False, 0.25]
15 -1 1 9344 ultralytics.nn.modules.conv.Conv [64, 64, 3, 2, 1, 4]
16 -1 1 74368 ultralytics.nn.modules.block.DSC3k2 [64, 128, 1, True]
17 -1 1 17792 ultralytics.nn.modules.conv.DSConv [128, 128, 3, 2]
18 -1 2 69632 ultralytics.nn.AddModules.MBConv.A2C2f_MBConv[128, 128, True, 4]
19 -1 1 34432 ultralytics.nn.modules.conv.DSConv [128, 256, 3, 2]
20 -1 2 270336 ultralytics.nn.AddModules.MBConv.A2C2f_MBConv[256, 256, True, 1]
21 [7, 16] 1 0 ultralytics.nn.AddModules.CFT.Add [128]
22 [9, 18] 1 0 ultralytics.nn.AddModules.CFT.Add [128]
23 [11, 20] 1 0 ultralytics.nn.AddModules.CFT.Add [256]
24 [21, 22, 23] 1 273536 ultralytics.nn.modules.block.HyperACE [128, 128, 1, 4, True, True, 0.5, 1, 'both']
25 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
26 24 1 33280 ultralytics.nn.modules.block.DownsampleConv [128]
27 [22, 24] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
28 [21, 25] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
29 [23, 26] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
30 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
31 [-1, 27] 1 0 ultralytics.nn.modules.conv.Concat [1]
32 -1 1 115328 ultralytics.nn.modules.block.DSC3k2 [384, 128, 1, True]
33 [-1, 24] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
34 32 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
35 [-1, 28] 1 0 ultralytics.nn.modules.conv.Concat [1]
36 -1 1 35136 ultralytics.nn.modules.block.DSC3k2 [256, 64, 1, True]
37 25 1 8320 ultralytics.nn.modules.conv.Conv [128, 64, 1, 1]
38 [36, 37] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
39 -1 1 36992 ultralytics.nn.modules.conv.Conv [64, 64, 3, 2]
40 [-1, 33] 1 0 ultralytics.nn.modules.conv.Concat [1]
41 -1 1 90752 ultralytics.nn.modules.block.DSC3k2 [192, 128, 1, True]
42 [-1, 24] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
43 41 1 147712 ultralytics.nn.modules.conv.Conv [128, 128, 3, 2]
44 [-1, 29] 1 0 ultralytics.nn.modules.conv.Concat [1]
45 -1 1 345344 ultralytics.nn.modules.block.DSC3k2 [384, 256, 1, True]
46 [-1, 26] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
47 [38, 42, 46] 1 432427 ultralytics.nn.modules.head.Detect [9, [64, 128, 256]]
YOLOv13-mid-MBConv summary: 700 layers, 2,487,890 parameters, 2,487,874 gradients, 8.5 GFLOPs
后期融合结果：
YOLOv13-late-MBConv summary: 983 layers, 3,574,297 parameters, 3,574,281 gradients, 10.9 GFLOPs
from n params module arguments
0 -1 1 0 ultralytics.nn.AddModules.multimodal.IN []
1 -1 1 0 ultralytics.nn.AddModules.multimodal.Multiin [1]
2 -2 1 0 ultralytics.nn.AddModules.multimodal.Multiin [2]
3 1 1 464 ultralytics.nn.modules.conv.Conv [3, 16, 3, 2]
4 -1 1 2368 ultralytics.nn.modules.conv.Conv [16, 32, 3, 2, 1, 2]
5 -1 1 5792 ultralytics.nn.modules.block.DSC3k2 [32, 64, 1, False, 0.25]
6 -1 1 9344 ultralytics.nn.modules.conv.Conv [64, 64, 3, 2, 1, 4]
7 -1 1 74368 ultralytics.nn.modules.block.DSC3k2 [64, 128, 1, True]
8 -1 1 17792 ultralytics.nn.modules.conv.DSConv [128, 128, 3, 2]
9 -1 2 69632 ultralytics.nn.AddModules.MBConv.A2C2f_MBConv[128, 128, True, 4]
10 -1 1 34432 ultralytics.nn.modules.conv.DSConv [128, 256, 3, 2]
11 -1 2 270336 ultralytics.nn.AddModules.MBConv.A2C2f_MBConv[256, 256, True, 1]
12 2 1 464 ultralytics.nn.modules.conv.Conv [3, 16, 3, 2]
13 -1 1 2368 ultralytics.nn.modules.conv.Conv [16, 32, 3, 2, 1, 2]
14 -1 1 5792 ultralytics.nn.modules.block.DSC3k2 [32, 64, 1, False, 0.25]
15 -1 1 9344 ultralytics.nn.modules.conv.Conv [64, 64, 3, 2, 1, 4]
16 -1 1 74368 ultralytics.nn.modules.block.DSC3k2 [64, 128, 1, True]
17 -1 1 17792 ultralytics.nn.modules.conv.DSConv [128, 128, 3, 2]
18 -1 2 69632 ultralytics.nn.AddModules.MBConv.A2C2f_MBConv[128, 128, True, 4]
19 -1 1 34432 ultralytics.nn.modules.conv.DSConv [128, 256, 3, 2]
20 -1 2 270336 ultralytics.nn.AddModules.MBConv.A2C2f_MBConv[256, 256, True, 1]
21 [7, 9, 11] 1 273536 ultralytics.nn.modules.block.HyperACE [128, 128, 1, 4, True, True, 0.5, 1, 'both']
22 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
23 21 1 33280 ultralytics.nn.modules.block.DownsampleConv [128]
24 [9, 21] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
25 [7, 22] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
26 [11, 23] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
27 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
28 [-1, 24] 1 0 ultralytics.nn.modules.conv.Concat [1]
29 -1 1 115328 ultralytics.nn.modules.block.DSC3k2 [384, 128, 1, True]
30 [-1, 21] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
31 29 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
32 [-1, 25] 1 0 ultralytics.nn.modules.conv.Concat [1]
33 -1 1 35136 ultralytics.nn.modules.block.DSC3k2 [256, 64, 1, True]
34 22 1 8320 ultralytics.nn.modules.conv.Conv [128, 64, 1, 1]
35 [33, 34] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
36 -1 1 36992 ultralytics.nn.modules.conv.Conv [64, 64, 3, 2]
37 [-1, 30] 1 0 ultralytics.nn.modules.conv.Concat [1]
38 -1 1 90752 ultralytics.nn.modules.block.DSC3k2 [192, 128, 1, True]
39 [-1, 21] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
40 38 1 147712 ultralytics.nn.modules.conv.Conv [128, 128, 3, 2]
41 [-1, 26] 1 0 ultralytics.nn.modules.conv.Concat [1]
42 -1 1 345344 ultralytics.nn.modules.block.DSC3k2 [384, 256, 1, True]
43 [-1, 23] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
44 [16, 18, 20] 1 273536 ultralytics.nn.modules.block.HyperACE [128, 128, 1, 4, True, True, 0.5, 1, 'both']
45 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
46 44 1 33280 ultralytics.nn.modules.block.DownsampleConv [128]
47 [18, 44] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
48 [16, 45] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
49 [20, 46] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
50 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
51 [-1, 47] 1 0 ultralytics.nn.modules.conv.Concat [1]
52 -1 1 115328 ultralytics.nn.modules.block.DSC3k2 [384, 128, 1, True]
53 [-1, 44] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
54 52 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
55 [-1, 48] 1 0 ultralytics.nn.modules.conv.Concat [1]
56 -1 1 35136 ultralytics.nn.modules.block.DSC3k2 [256, 64, 1, True]
57 45 1 8320 ultralytics.nn.modules.conv.Conv [128, 64, 1, 1]
58 [56, 57] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
59 -1 1 36992 ultralytics.nn.modules.conv.Conv [64, 64, 3, 2]
60 [-1, 53] 1 0 ultralytics.nn.modules.conv.Concat [1]
61 -1 1 90752 ultralytics.nn.modules.block.DSC3k2 [192, 128, 1, True]
62 [-1, 44] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
63 61 1 147712 ultralytics.nn.modules.conv.Conv [128, 128, 3, 2]
64 [-1, 49] 1 0 ultralytics.nn.modules.conv.Concat [1]
65 -1 1 345344 ultralytics.nn.modules.block.DSC3k2 [384, 256, 1, True]
66 [-1, 46] 1 1 ultralytics.nn.modules.block.FullPAD_Tunnel []
67 [35, 58] 1 0 ultralytics.nn.AddModules.CFT.Add [64]
68 [39, 62] 1 0 ultralytics.nn.AddModules.CFT.Add [128]
69 [43, 66] 1 0 ultralytics.nn.AddModules.CFT.Add [256]
70 [67, 68, 69] 1 432427 ultralytics.nn.modules.head.Detect [9, [64, 128, 256]]
YOLOv13-late-MBConv summary: 983 layers, 3,574,297 parameters, 3,574,281 gradients, 10.9 GFLOPs