学习资源站

【YOLOv11多模态融合改进】(可见光加红外)涉及前期,中期,中后期,后期融合方式的完整配置步骤以及二次改进方案-

【YOLOv11多模态融合改进】（可见光+红外）涉及前期、中期、中后期、后期融合方式的完整配置步骤以及二次改进方案

前言

主题： YOLOv11的多模态融合改进

方式： 前期融合、中期融合、中-后期融合、后期融合。

内容： 包含融合方式详解和完整的项目包和配置步骤以及二次改进建议，开箱即用，一键运行。

一、融合方式

1.1 前期融合方法及结构图

定义： 在网络输入阶段将多模态数据直接合并，形成统一的特征表示。

实现方式： 将 RGB（3 通道）与红外（3 通道）图像直接拼接为 6 通道输入，以保留原始模态的细节信息。

结构示意图：

1.2 中期融合方法及结构图

定义： 在网络中间层（骨干网络与颈部网络之间）对多模态特征进行融合。

实现方式： 每个模态通过独立的骨干网络提取特征，融合时采用Concat操作合并特征图，送入颈部网络。

结构示意图：

1.3 中-后期融合方法及结构图

定义： 在颈部网络的上采样之后对多模态特征进行融合。

实现方式： 每个模态通过独立的骨干网络和颈部的FPN网络提取特征，融合时采用Concat操作合并特征图，送入检测头。

结构示意图：

1.4 后期融合方法及结构图

定义： 在网络输出阶段（如检测头或分类器前）对多模态特征进行融合。

实现方式： 每个模态通过独立的骨干网络和颈部网络提取特征，融合时采用Concat操作合并特征图，送入检测头。

结构示意图：

二、完整配置步骤

！！！ 私信获取的项目包就已经把相关的多模态输入、训练等改动都已经配好了，只需要新建模型yaml文件，粘贴对应的模型，进行训练即可。 项目包获取及使用教程可参考链接： 《YOLO系列模型的多模态项目》配置使用教程

在什么地方新建，n,s,m,l,x,用哪个版本按自己的需求来即可，和普通的训练步骤一致。

除了模型结构方面的改动，在yaml文件中还传入了一个通道数 ch: 6 表示传入的是双模态，6通道 ，前三个是可见光，后三个是红外。
在default.yaml中也配置了这个参数。

2.1 前期融合 

# Ultralytics YOLO 🚀, AGPL-3.0 license
# YOLO11 object detection model with P3-P5 outputs. For Usage examples see https://docs.ultralytics.com/tasks/detect

# Parameters
ch: 6
nc: 80 # number of classes
scales: # model compound scaling constants, i.e. 'model=yolo11n.yaml' will call yolo11.yaml with scale 'n'
  # [depth, width, max_channels]
  n: [0.50, 0.25, 1024] # summary: 319 layers, 2624080 parameters, 2624064 gradients, 6.6 GFLOPs
  s: [0.50, 0.50, 1024] # summary: 319 layers, 9458752 parameters, 9458736 gradients, 21.7 GFLOPs
  m: [0.50, 1.00, 512] # summary: 409 layers, 20114688 parameters, 20114672 gradients, 68.5 GFLOPs
  l: [1.00, 1.00, 512] # summary: 631 layers, 25372160 parameters, 25372144 gradients, 87.6 GFLOPs
  x: [1.00, 1.50, 512] # summary: 631 layers, 56966176 parameters, 56966160 gradients, 196.0 GFLOPs

# YOLO11n backbone
backbone:
  # [from, repeats, module, args]
  - [-1, 1, MF, [64]]  # 0-P1/2
  - [-1, 1, Conv, [64, 3, 2]] # 1-P1/2
  - [-1, 1, Conv, [128, 3, 2]] # 2-P2/4
  - [-1, 2, C3k2, [256, False, 0.25]]
  - [-1, 1, Conv, [256, 3, 2]] # 4-P3/8
  - [-1, 2, C3k2, [512, False, 0.25]]
  - [-1, 1, Conv, [512, 3, 2]] # 6-P4/16
  - [-1, 2, C3k2, [512, True]]
  - [-1, 1, Conv, [1024, 3, 2]] # 8-P5/32
  - [-1, 2, C3k2, [1024, True]]
  - [-1, 1, SPPF, [1024, 5]] # 10
  - [-1, 2, C2PSA, [1024]] # 11

# YOLO11n head
head:
  - [-1, 1, nn.Upsample, [None, 2, "nearest"]]
  - [[-1, 7], 1, Concat, [1]] # cat backbone P4
  - [-1, 2, C3k2, [512, False]] # 14

  - [-1, 1, nn.Upsample, [None, 2, "nearest"]]
  - [[-1, 5], 1, Concat, [1]] # cat backbone P3
  - [-1, 2, C3k2, [256, False]] # 17 (P3/8-small)

  - [-1, 1, Conv, [256, 3, 2]]
  - [[-1, 14], 1, Concat, [1]] # cat head P4
  - [-1, 2, C3k2, [512, False]] # 20 (P4/16-medium)

  - [-1, 1, Conv, [512, 3, 2]]
  - [[-1, 11], 1, Concat, [1]] # cat head P5
  - [-1, 2, C3k2, [1024, True]] # 23 (P5/32-large)

  - [[17, 20, 23], 1, Detect, [nc]] # Detect(P3, P4, P5)

2.2 中期融合 

# Ultralytics YOLO 🚀, AGPL-3.0 license
# YOLO11 object detection model with P3-P5 outputs. For Usage examples see https://docs.ultralytics.com/tasks/detect

# Parameters
ch: 6
nc: 80 # number of classes
scales: # model compound scaling constants, i.e. 'model=yolo11n.yaml' will call yolo11.yaml with scale 'n'
  # [depth, width, max_channels]
  n: [0.50, 0.25, 1024] # summary: 319 layers, 2624080 parameters, 2624064 gradients, 6.6 GFLOPs
  s: [0.50, 0.50, 1024] # summary: 319 layers, 9458752 parameters, 9458736 gradients, 21.7 GFLOPs
  m: [0.50, 1.00, 512] # summary: 409 layers, 20114688 parameters, 20114672 gradients, 68.5 GFLOPs
  l: [1.00, 1.00, 512] # summary: 631 layers, 25372160 parameters, 25372144 gradients, 87.6 GFLOPs
  x: [1.00, 1.50, 512] # summary: 631 layers, 56966176 parameters, 56966160 gradients, 196.0 GFLOPs

# YOLO11n backbone
backbone:
  # [from, repeats, module, args]
  - [-1, 1, IN, []]  # 0
  - [-1, 1, Multiin, [1]]  # 1
  - [-2, 1, Multiin, [2]]  # 2

  - [1, 1, Conv, [64, 3, 2]] # 3-P1/2
  - [-1, 1, Conv, [128, 3, 2]] # 4-P2/4
  - [-1, 2, C3k2, [256, False, 0.25]]
  - [-1, 1, Conv, [256, 3, 2]] # 6-P3/8
  - [-1, 2, C3k2, [512, False, 0.25]]
  - [-1, 1, Conv, [512, 3, 2]] # 8-P4/16
  - [-1, 2, C3k2, [512, True]]
  - [-1, 1, Conv, [1024, 3, 2]] # 10-P5/32
  - [-1, 2, C3k2, [1024, True]]

  - [2, 1, Conv, [64, 3, 2]] # 12-P1/2
  - [-1, 1, Conv, [128, 3, 2]] # 13-P2/4
  - [-1, 2, C3k2, [256, False, 0.25]]
  - [-1, 1, Conv, [256, 3, 2]] # 15-P3/8
  - [-1, 2, C3k2, [512, False, 0.25]]
  - [-1, 1, Conv, [512, 3, 2]] # 17-P4/16
  - [-1, 2, C3k2, [512, True]]
  - [-1, 1, Conv, [1024, 3, 2]] # 19-P5/32
  - [-1, 2, C3k2, [1024, True]]

  - [[7, 16], 1, Concat, [1]]  # 21 cat backbone P3
  - [[9, 18], 1, Concat, [1]]  # 22 cat backbone P4
  - [[11, 20], 1, Concat, [1]]  # 23 cat backbone P5

  - [-1, 1, SPPF, [1024, 5]] # 24
  - [-1, 2, C2PSA, [1024]] # 25

# YOLO11n head
head:
  - [-1, 1, nn.Upsample, [None, 2, "nearest"]]
  - [[-1, 22], 1, Concat, [1]] # cat backbone P4
  - [-1, 2, C3k2, [512, False]] # 28

  - [-1, 1, nn.Upsample, [None, 2, "nearest"]]
  - [[-1, 21], 1, Concat, [1]] # cat backbone P3
  - [-1, 2, C3k2, [256, False]] # 31 (P3/8-small)

  - [-1, 1, Conv, [256, 3, 2]]
  - [[-1, 28], 1, Concat, [1]] # cat head P4
  - [-1, 2, C3k2, [512, False]] # 34 (P4/16-medium)

  - [-1, 1, Conv, [512, 3, 2]]
  - [[-1, 25], 1, Concat, [1]] # cat head P5
  - [-1, 2, C3k2, [1024, True]] # 37 (P5/32-large)

  - [[31, 34, 37], 1, Detect, [nc]] # Detect(P3, P4, P5)

2.3 中-后期融合 

# Ultralytics YOLO 🚀, AGPL-3.0 license
# YOLO11 object detection model with P3-P5 outputs. For Usage examples see https://docs.ultralytics.com/tasks/detect

# Parameters
ch: 6
nc: 80 # number of classes
scales: # model compound scaling constants, i.e. 'model=yolo11n.yaml' will call yolo11.yaml with scale 'n'
  # [depth, width, max_channels]
  n: [0.50, 0.25, 1024] # summary: 319 layers, 2624080 parameters, 2624064 gradients, 6.6 GFLOPs
  s: [0.50, 0.50, 1024] # summary: 319 layers, 9458752 parameters, 9458736 gradients, 21.7 GFLOPs
  m: [0.50, 1.00, 512] # summary: 409 layers, 20114688 parameters, 20114672 gradients, 68.5 GFLOPs
  l: [1.00, 1.00, 512] # summary: 631 layers, 25372160 parameters, 25372144 gradients, 87.6 GFLOPs
  x: [1.00, 1.50, 512] # summary: 631 layers, 56966176 parameters, 56966160 gradients, 196.0 GFLOPs

# YOLO11n backbone
backbone:
  # [from, repeats, module, args]
  - [-1, 1, IN, []]  # 0
  - [-1, 1, Multiin, [1]]  # 1
  - [-2, 1, Multiin, [2]]  # 2

  - [1, 1, Conv, [64, 3, 2]] # 3-P1/2
  - [-1, 1, Conv, [128, 3, 2]] # 4-P2/4
  - [-1, 2, C3k2, [256, False, 0.25]]
  - [-1, 1, Conv, [256, 3, 2]] # 6-P3/8
  - [-1, 2, C3k2, [512, False, 0.25]]
  - [-1, 1, Conv, [512, 3, 2]] # 8-P4/16
  - [-1, 2, C3k2, [512, True]]
  - [-1, 1, Conv, [1024, 3, 2]] # 10-P5/32
  - [-1, 2, C3k2, [1024, True]]
  - [-1, 1, SPPF, [1024, 5]] # 12
  - [-1, 2, C2PSA, [1024]] # 13

  - [2, 1, Conv, [64, 3, 2]] # 14-P1/2
  - [-1, 1, Conv, [128, 3, 2]] # 15-P2/4
  - [-1, 2, C3k2, [256, False, 0.25]]
  - [-1, 1, Conv, [256, 3, 2]] # 17-P3/8
  - [-1, 2, C3k2, [512, False, 0.25]]
  - [-1, 1, Conv, [512, 3, 2]] # 19-P4/16
  - [-1, 2, C3k2, [512, True]]
  - [-1, 1, Conv, [1024, 3, 2]] # 21-P5/32
  - [-1, 2, C3k2, [1024, True]]
  - [-1, 1, SPPF, [1024, 5]] # 23
  - [-1, 2, C2PSA, [1024]] # 24

# YOLO11n head
head:
  - [13, 1, nn.Upsample, [None, 2, "nearest"]]
  - [[-1, 9], 1, Concat, [1]] # cat backbone P4
  - [-1, 2, C3k2, [512, False]] # 27

  - [-1, 1, nn.Upsample, [None, 2, "nearest"]]
  - [[-1, 7], 1, Concat, [1]] # cat backbone P3
  - [-1, 2, C3k2, [256, False]] # 30 (P3/8-small)

  - [24, 1, nn.Upsample, [None, 2, "nearest"]]
  - [[-1, 20], 1, Concat, [1]] # cat backbone P4
  - [-1, 2, C3k2, [512, False]] # 33

  - [-1, 1, nn.Upsample, [None, 2, "nearest"]]
  - [[-1, 18], 1, Concat, [1]] # cat backbone P3
  - [-1, 2, C3k2, [256, False]] # 36 (P3/8-small)

  - [[13, 24], 1, Concat, [1]]  # 37 cat backbone P3
  - [[27, 33], 1, Concat, [1]]  # 38 cat backbone P4
  - [[30, 36], 1, Concat, [1]]  # 39 cat backbone P5

  - [-1, 1, Conv, [256, 3, 2]]
  - [[-1, 38], 1, Concat, [1]] # cat head P4
  - [-1, 2, C3k2, [512, False]] # 42 (P4/16-medium)

  - [-1, 1, Conv, [512, 3, 2]]
  - [[-1, 37], 1, Concat, [1]] # cat head P5
  - [-1, 2, C3k2, [1024, True]] # 45 (P5/32-large)

  - [[39, 42, 45], 1, Detect, [nc]] # Detect(P3, P4, P5)

2.4 后期融合 

# Ultralytics YOLO 🚀, AGPL-3.0 license
# YOLO11 object detection model with P3-P5 outputs. For Usage examples see https://docs.ultralytics.com/tasks/detect

# Parameters
ch: 6
nc: 80 # number of classes
scales: # model compound scaling constants, i.e. 'model=yolo11n.yaml' will call yolo11.yaml with scale 'n'
  # [depth, width, max_channels]
  n: [0.50, 0.25, 1024] # summary: 319 layers, 2624080 parameters, 2624064 gradients, 6.6 GFLOPs
  s: [0.50, 0.50, 1024] # summary: 319 layers, 9458752 parameters, 9458736 gradients, 21.7 GFLOPs
  m: [0.50, 1.00, 512] # summary: 409 layers, 20114688 parameters, 20114672 gradients, 68.5 GFLOPs
  l: [1.00, 1.00, 512] # summary: 631 layers, 25372160 parameters, 25372144 gradients, 87.6 GFLOPs
  x: [1.00, 1.50, 512] # summary: 631 layers, 56966176 parameters, 56966160 gradients, 196.0 GFLOPs

# YOLO11n backbone
backbone:
  # [from, repeats, module, args]
  - [-1, 1, IN, []]  # 0
  - [-1, 1, Multiin, [1]]  # 1
  - [-2, 1, Multiin, [2]]  # 2

  - [1, 1, Conv, [64, 3, 2]] # 3-P1/2
  - [-1, 1, Conv, [128, 3, 2]] # 4-P2/4
  - [-1, 2, C3k2, [256, False, 0.25]]
  - [-1, 1, Conv, [256, 3, 2]] # 6-P3/8
  - [-1, 2, C3k2, [512, False, 0.25]]
  - [-1, 1, Conv, [512, 3, 2]] # 8-P4/16
  - [-1, 2, C3k2, [512, True]]
  - [-1, 1, Conv, [1024, 3, 2]] # 10-P5/32
  - [-1, 2, C3k2, [1024, True]]
  - [-1, 1, SPPF, [1024, 5]] # 12
  - [-1, 2, C2PSA, [1024]] # 13

  - [2, 1, Conv, [64, 3, 2]] # 14-P1/2
  - [-1, 1, Conv, [128, 3, 2]] # 15-P2/4
  - [-1, 2, C3k2, [256, False, 0.25]]
  - [-1, 1, Conv, [256, 3, 2]] # 17-P3/8
  - [-1, 2, C3k2, [512, False, 0.25]]
  - [-1, 1, Conv, [512, 3, 2]] # 19-P4/16
  - [-1, 2, C3k2, [512, True]]
  - [-1, 1, Conv, [1024, 3, 2]] # 21-P5/32
  - [-1, 2, C3k2, [1024, True]]
  - [-1, 1, SPPF, [1024, 5]] # 23
  - [-1, 2, C2PSA, [1024]] # 24

# YOLO11n head
head:
  - [13, 1, nn.Upsample, [None, 2, "nearest"]]
  - [[-1, 9], 1, Concat, [1]] # cat backbone P4
  - [-1, 2, C3k2, [512, False]] # 27

  - [-1, 1, nn.Upsample, [None, 2, "nearest"]]
  - [[-1, 7], 1, Concat, [1]] # cat backbone P3
  - [-1, 2, C3k2, [256, False]] # 30 (P3/8-small)

  - [-1, 1, Conv, [256, 3, 2]]
  - [[-1, 27], 1, Concat, [1]] # cat head P4
  - [-1, 2, C3k2, [512, False]] # 33 (P4/16-medium)

  - [-1, 1, Conv, [512, 3, 2]]
  - [[-1, 13], 1, Concat, [1]] # cat head P5
  - [-1, 2, C3k2, [1024, True]] # 36 (P5/32-large)

  - [24, 1, nn.Upsample, [None, 2, "nearest"]]
  - [[-1, 20], 1, Concat, [1]] # cat backbone P4
  - [-1, 2, C3k2, [512, False]] # 39

  - [-1, 1, nn.Upsample, [None, 2, "nearest"]]
  - [[-1, 18], 1, Concat, [1]] # cat backbone P3
  - [-1, 2, C3k2, [256, False]] # 42 (P3/8-small)

  - [-1, 1, Conv, [256, 3, 2]]
  - [[-1, 39], 1, Concat, [1]] # cat head P4
  - [-1, 2, C3k2, [512, False]] # 45 (P4/16-medium)

  - [-1, 1, Conv, [512, 3, 2]]
  - [[-1, 24], 1, Concat, [1]] # cat head P5
  - [-1, 2, C3k2, [1024, True]] # 48 (P5/32-large)

  - [[30, 42], 1, Concat, [1]]  # cat head P5  49
  - [[33, 45], 1, Concat, [1]]  # cat head P5  50
  - [[36, 48], 1, Concat, [1]]  # cat head P5  51

  - [[49, 50, 51], 1, Detect, [nc]] # Detect(P3, P4, P5)

三、成功运行结果

前期融合结果： 可以看到输入的通道数为6，表明可见光图像和红外图像均输入到了模型中进行融合训练。

YOLO11-earlyfusion summary: 332 layers, 2,592,379 parameters, 2,592,363 gradients, 7.2 GFLOPs 

                   from  n    params  module                                       arguments
  0                  -1  1       472  ultralytics.nn.AddModules.multimodal.MF      [6, 16]
  1                  -1  1      2336  ultralytics.nn.modules.conv.Conv             [16, 16, 3, 2]
  2                  -1  1      4672  ultralytics.nn.modules.conv.Conv             [16, 32, 3, 2]
  3                  -1  1      6640  ultralytics.nn.modules.block.C3k2            [32, 64, 1, False, 0.25]
  4                  -1  1     36992  ultralytics.nn.modules.conv.Conv             [64, 64, 3, 2]
  5                  -1  1     26080  ultralytics.nn.modules.block.C3k2            [64, 128, 1, False, 0.25]
  6                  -1  1    147712  ultralytics.nn.modules.conv.Conv             [128, 128, 3, 2]
  7                  -1  1     87040  ultralytics.nn.modules.block.C3k2            [128, 128, 1, True]
  8                  -1  1    295424  ultralytics.nn.modules.conv.Conv             [128, 256, 3, 2]
  9                  -1  1    346112  ultralytics.nn.modules.block.C3k2            [256, 256, 1, True]
 10                  -1  1    164608  ultralytics.nn.modules.block.SPPF            [256, 256, 5]
 11                  -1  1    249728  ultralytics.nn.modules.block.C2PSA           [256, 256, 1]
 12                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 13             [-1, 7]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 14                  -1  1    111296  ultralytics.nn.modules.block.C3k2            [384, 128, 1, False]
 15                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 16             [-1, 5]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 17                  -1  1     32096  ultralytics.nn.modules.block.C3k2            [256, 64, 1, False]
 18                  -1  1     36992  ultralytics.nn.modules.conv.Conv             [64, 64, 3, 2]
 19            [-1, 14]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 20                  -1  1     86720  ultralytics.nn.modules.block.C3k2            [192, 128, 1, False]
 21                  -1  1    147712  ultralytics.nn.modules.conv.Conv             [128, 128, 3, 2]
 22            [-1, 11]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 23                  -1  1    378880  ultralytics.nn.modules.block.C3k2            [384, 256, 1, True]
 24        [17, 20, 23]  1    430867  ultralytics.nn.modules.head.Detect           [1, [64, 128, 256]]
YOLO11-earlyfusion summary: 332 layers, 2,592,379 parameters, 2,592,363 gradients, 7.2 GFLOPs

中期融合结果：

YOLO11-midfusion summary: 436 layers, 3,795,379 parameters, 3,795,363 gradients, 9.6 GFLOPs 

                   from  n    params  module                                       arguments
  0                  -1  1         0  ultralytics.nn.AddModules.multimodal.IN      []
  1                  -1  1         0  ultralytics.nn.AddModules.multimodal.Multiin [1]
  2                  -2  1         0  ultralytics.nn.AddModules.multimodal.Multiin [2]
  3                   1  1       464  ultralytics.nn.modules.conv.Conv             [3, 16, 3, 2]
  4                  -1  1      4672  ultralytics.nn.modules.conv.Conv             [16, 32, 3, 2]
  5                  -1  1      6640  ultralytics.nn.modules.block.C3k2            [32, 64, 1, False, 0.25]
  6                  -1  1     36992  ultralytics.nn.modules.conv.Conv             [64, 64, 3, 2]
  7                  -1  1     26080  ultralytics.nn.modules.block.C3k2            [64, 128, 1, False, 0.25]
  8                  -1  1    147712  ultralytics.nn.modules.conv.Conv             [128, 128, 3, 2]
  9                  -1  1     87040  ultralytics.nn.modules.block.C3k2            [128, 128, 1, True]
 10                  -1  1    295424  ultralytics.nn.modules.conv.Conv             [128, 256, 3, 2]
 11                  -1  1    346112  ultralytics.nn.modules.block.C3k2            [256, 256, 1, True]
 12                   2  1       464  ultralytics.nn.modules.conv.Conv             [3, 16, 3, 2]
 13                  -1  1      4672  ultralytics.nn.modules.conv.Conv             [16, 32, 3, 2]
 14                  -1  1      6640  ultralytics.nn.modules.block.C3k2            [32, 64, 1, False, 0.25]
 15                  -1  1     36992  ultralytics.nn.modules.conv.Conv             [64, 64, 3, 2]
 16                  -1  1     26080  ultralytics.nn.modules.block.C3k2            [64, 128, 1, False, 0.25]
 17                  -1  1    147712  ultralytics.nn.modules.conv.Conv             [128, 128, 3, 2]
 18                  -1  1     87040  ultralytics.nn.modules.block.C3k2            [128, 128, 1, True]
 19                  -1  1    295424  ultralytics.nn.modules.conv.Conv             [128, 256, 3, 2]
 20                  -1  1    346112  ultralytics.nn.modules.block.C3k2            [256, 256, 1, True]
 21             [7, 16]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 22             [9, 18]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 23            [11, 20]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 24                  -1  1    394240  ultralytics.nn.modules.block.SPPF            [512, 256, 5]
 25                  -1  1    249728  ultralytics.nn.modules.block.C2PSA           [256, 256, 1]
 26                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 27            [-1, 22]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 28                  -1  1    127680  ultralytics.nn.modules.block.C3k2            [512, 128, 1, False]
 29                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 30            [-1, 21]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 31                  -1  1     40288  ultralytics.nn.modules.block.C3k2            [384, 64, 1, False]
 32                  -1  1     36992  ultralytics.nn.modules.conv.Conv             [64, 64, 3, 2]
 33            [-1, 28]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 34                  -1  1     86720  ultralytics.nn.modules.block.C3k2            [192, 128, 1, False]
 35                  -1  1    147712  ultralytics.nn.modules.conv.Conv             [128, 128, 3, 2]
 36            [-1, 25]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 37                  -1  1    378880  ultralytics.nn.modules.block.C3k2            [384, 256, 1, True]
 38        [31, 34, 37]  1    430867  ultralytics.nn.modules.head.Detect           [1, [64, 128, 256]]
YOLO11-midfusion summary: 436 layers, 3,795,379 parameters, 3,795,363 gradients, 9.6 GFLOPs

中-后期融合结果：

YOLO11-mid-to-late-fusion summary: 508 layers, 4,332,051 parameters, 4,332,035 gradients, 11.5 GFLOPs 

                   from  n    params  module                                       arguments
  0                  -1  1         0  ultralytics.nn.AddModules.multimodal.IN      []
  1                  -1  1         0  ultralytics.nn.AddModules.multimodal.Multiin [1]
  2                  -2  1         0  ultralytics.nn.AddModules.multimodal.Multiin [2]
  3                   1  1       464  ultralytics.nn.modules.conv.Conv             [3, 16, 3, 2]
  4                  -1  1      4672  ultralytics.nn.modules.conv.Conv             [16, 32, 3, 2]
  5                  -1  1      6640  ultralytics.nn.modules.block.C3k2            [32, 64, 1, False, 0.25]
  6                  -1  1     36992  ultralytics.nn.modules.conv.Conv             [64, 64, 3, 2]
  7                  -1  1     26080  ultralytics.nn.modules.block.C3k2            [64, 128, 1, False, 0.25]
  8                  -1  1    147712  ultralytics.nn.modules.conv.Conv             [128, 128, 3, 2]
  9                  -1  1     87040  ultralytics.nn.modules.block.C3k2            [128, 128, 1, True]
 10                  -1  1    295424  ultralytics.nn.modules.conv.Conv             [128, 256, 3, 2]
 11                  -1  1    346112  ultralytics.nn.modules.block.C3k2            [256, 256, 1, True]
 12                  -1  1    164608  ultralytics.nn.modules.block.SPPF            [256, 256, 5]
 13                  -1  1    249728  ultralytics.nn.modules.block.C2PSA           [256, 256, 1]
 14                   2  1       464  ultralytics.nn.modules.conv.Conv             [3, 16, 3, 2]
 15                  -1  1      4672  ultralytics.nn.modules.conv.Conv             [16, 32, 3, 2]
 16                  -1  1      6640  ultralytics.nn.modules.block.C3k2            [32, 64, 1, False, 0.25]
 17                  -1  1     36992  ultralytics.nn.modules.conv.Conv             [64, 64, 3, 2]
 18                  -1  1     26080  ultralytics.nn.modules.block.C3k2            [64, 128, 1, False, 0.25]
 19                  -1  1    147712  ultralytics.nn.modules.conv.Conv             [128, 128, 3, 2]
 20                  -1  1     87040  ultralytics.nn.modules.block.C3k2            [128, 128, 1, True]
 21                  -1  1    295424  ultralytics.nn.modules.conv.Conv             [128, 256, 3, 2]
 22                  -1  1    346112  ultralytics.nn.modules.block.C3k2            [256, 256, 1, True]
 23                  -1  1    164608  ultralytics.nn.modules.block.SPPF            [256, 256, 5]
 24                  -1  1    249728  ultralytics.nn.modules.block.C2PSA           [256, 256, 1]
 25                  13  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 26             [-1, 9]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 27                  -1  1    111296  ultralytics.nn.modules.block.C3k2            [384, 128, 1, False]
 28                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 29             [-1, 7]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 30                  -1  1     32096  ultralytics.nn.modules.block.C3k2            [256, 64, 1, False]
 31                  24  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 32            [-1, 20]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 33                  -1  1    111296  ultralytics.nn.modules.block.C3k2            [384, 128, 1, False]
 34                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 35            [-1, 18]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 36                  -1  1     32096  ultralytics.nn.modules.block.C3k2            [256, 64, 1, False]
 37            [13, 24]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 38            [27, 33]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 39            [30, 36]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 40                  -1  1     73856  ultralytics.nn.modules.conv.Conv             [128, 64, 3, 2]
 41            [-1, 38]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 42                  -1  1    103104  ultralytics.nn.modules.block.C3k2            [320, 128, 1, False]
 43                  -1  1    147712  ultralytics.nn.modules.conv.Conv             [128, 128, 3, 2]
 44            [-1, 37]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 45                  -1  1    444416  ultralytics.nn.modules.block.C3k2            [640, 256, 1, True]
 46        [39, 42, 45]  1    545235  ultralytics.nn.modules.head.Detect           [1, [128, 128, 256]]
YOLO11-mid-to-late-fusion summary: 508 layers, 4,332,051 parameters, 4,332,035 gradients, 11.5 GFLOPs

后期融合结果：

YOLO11-latefusion summary: 564 layers, 5,138,131 parameters, 5,138,115 gradients, 12.5 GFLOPs 

                   from  n    params  module                                       arguments
  0                  -1  1         0  ultralytics.nn.AddModules.multimodal.IN      []
  1                  -1  1         0  ultralytics.nn.AddModules.multimodal.Multiin [1]
  2                  -2  1         0  ultralytics.nn.AddModules.multimodal.Multiin [2]
  3                   1  1       464  ultralytics.nn.modules.conv.Conv             [3, 16, 3, 2]
  4                  -1  1      4672  ultralytics.nn.modules.conv.Conv             [16, 32, 3, 2]
  5                  -1  1      6640  ultralytics.nn.modules.block.C3k2            [32, 64, 1, False, 0.25]
  6                  -1  1     36992  ultralytics.nn.modules.conv.Conv             [64, 64, 3, 2]
  7                  -1  1     26080  ultralytics.nn.modules.block.C3k2            [64, 128, 1, False, 0.25]
  8                  -1  1    147712  ultralytics.nn.modules.conv.Conv             [128, 128, 3, 2]
  9                  -1  1     87040  ultralytics.nn.modules.block.C3k2            [128, 128, 1, True]
 10                  -1  1    295424  ultralytics.nn.modules.conv.Conv             [128, 256, 3, 2]
 11                  -1  1    346112  ultralytics.nn.modules.block.C3k2            [256, 256, 1, True]
 12                  -1  1    164608  ultralytics.nn.modules.block.SPPF            [256, 256, 5]
 13                  -1  1    249728  ultralytics.nn.modules.block.C2PSA           [256, 256, 1]
 14                   2  1       464  ultralytics.nn.modules.conv.Conv             [3, 16, 3, 2]
 15                  -1  1      4672  ultralytics.nn.modules.conv.Conv             [16, 32, 3, 2]
 16                  -1  1      6640  ultralytics.nn.modules.block.C3k2            [32, 64, 1, False, 0.25]
 17                  -1  1     36992  ultralytics.nn.modules.conv.Conv             [64, 64, 3, 2]
 18                  -1  1     26080  ultralytics.nn.modules.block.C3k2            [64, 128, 1, False, 0.25]
 19                  -1  1    147712  ultralytics.nn.modules.conv.Conv             [128, 128, 3, 2]
 20                  -1  1     87040  ultralytics.nn.modules.block.C3k2            [128, 128, 1, True]
 21                  -1  1    295424  ultralytics.nn.modules.conv.Conv             [128, 256, 3, 2]
 22                  -1  1    346112  ultralytics.nn.modules.block.C3k2            [256, 256, 1, True]
 23                  -1  1    164608  ultralytics.nn.modules.block.SPPF            [256, 256, 5]
 24                  -1  1    249728  ultralytics.nn.modules.block.C2PSA           [256, 256, 1]
 25                  13  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 26             [-1, 9]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 27                  -1  1    111296  ultralytics.nn.modules.block.C3k2            [384, 128, 1, False]
 28                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 29             [-1, 7]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 30                  -1  1     32096  ultralytics.nn.modules.block.C3k2            [256, 64, 1, False]
 31                  -1  1     36992  ultralytics.nn.modules.conv.Conv             [64, 64, 3, 2]
 32            [-1, 27]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 33                  -1  1     86720  ultralytics.nn.modules.block.C3k2            [192, 128, 1, False]
 34                  -1  1    147712  ultralytics.nn.modules.conv.Conv             [128, 128, 3, 2]
 35            [-1, 13]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 36                  -1  1    378880  ultralytics.nn.modules.block.C3k2            [384, 256, 1, True]
 37                  24  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 38            [-1, 20]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 39                  -1  1    111296  ultralytics.nn.modules.block.C3k2            [384, 128, 1, False]
 40                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 41            [-1, 18]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 42                  -1  1     32096  ultralytics.nn.modules.block.C3k2            [256, 64, 1, False]
 43                  -1  1     36992  ultralytics.nn.modules.conv.Conv             [64, 64, 3, 2]
 44            [-1, 39]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 45                  -1  1     86720  ultralytics.nn.modules.block.C3k2            [192, 128, 1, False]
 46                  -1  1    147712  ultralytics.nn.modules.conv.Conv             [128, 128, 3, 2]
 47            [-1, 24]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 48                  -1  1    378880  ultralytics.nn.modules.block.C3k2            [384, 256, 1, True]
 49            [30, 42]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 50            [33, 45]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 51            [36, 48]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 52        [49, 50, 51]  1    819795  ultralytics.nn.modules.head.Detect           [1, [128, 256, 512]]
YOLO11-latefusion summary: 564 layers, 5,138,131 parameters, 5,138,115 gradients, 12.5 GFLOPs

四、二次改进方案

  1. 多模态模型的二次改进和普通模型的改进一致，主要涉及到C3k2、颈部结构、上采样、下采样等，可以增加或替换成其它模块，可以换成其它的颈部结构在进行融合。若有需要可查看主页的模块改进专栏。

  2. 两个骨干中均可以再次添加其它模块，需要注意的是融合的时候层数要对应上，即两层的特征图大小要一致。