一、本文介绍

本文记录的是利用 TFAM 模块改进 RT-DETR 的多模态融合部分 。

TFAM 模块（Temporal Fusion Attention Module，时序融合注意力模块） 通过在特征提取网络的深层 引入通道 - 空间双分支注意力机制 ，基于时序信息动态生成双模态特征融合权重。该模块 可自适应捕捉跨模态的重要特征，抑制双模态特征中的噪声干扰与无效信息，实现高层语义与低层空间特征的时序关联建模与互补融合 ，为变化检测提供跨模态的精准特征表示。

---

---

二、TFAM模块介绍

Exchanging Dual-Encoder–Decoder: A New Strategy for Change Detection With Semantic Guidance and Spatial Localization

2.1 设计出发点

当前基于深度学习的变化检测模型在进行双时相特征融合时，主要存在以下问题：

• 简单融合方法 ：直接进行元素加减或拼接，易受噪声干扰，难以实现有效融合。
• 卷积增强方法 ：虽通过多尺度卷积减少噪声，但忽略了双时相特征间的时序信息。
• 注意力增强方法 ：通常在通道维度拼接后使用注意力机制，但同样未充分利用时序信息。

为解决上述问题，TFAM（Temporal Fusion Attention Module）模块被提出，其核心目标是 利用时序信息实现双时相特征的有效融合 ，通过对比双时相特征在时间维度上的重要性，提升特征融合的准确性和鲁棒性。

2.2 结构原理

TFAM模块通过 通道注意力 和 空间注意力 两个分支，分别在通道和空间维度上计算双时相特征的权重，进而实现特征融合。其具体结构和工作流程如下：

2.2.1 通道注意力分支

• 特征聚合 ：对输入的双时相特征 $T_1$ 和 $T_2$ ，分别进行全局平均池化（Avgpool）和全局最大池化（Maxpool），聚合空间信息，得到 $S_c=\text{Concat}(Avg(T_1),\text{Max}(T_1),\text{Avg}(T_2),\text{Max}(T_2))$ 。
• 权重计算 ：通过两个1D卷积层（类似ECA模块）生成双时相的通道权重 $W_{c1}$ 和 $W_{c2}$ ，再经Softmax归一化，使权重和为1，公式为：
  $$W_{c1}^{\prime },W_{c2}^{\prime }=\frac{e^{W_{c1}}}{e^{W_{c1}}+e^{W_{c2}}},\frac{e^{W_{c2}}}{e^{W_{c1}}+e^{W_{c2}}}$$
  该过程通过对比通道权重，确定双时相特征在通道维度上的重要部分。

2.2.2 空间注意力分支

• 权重计算 ：采用与通道注意力类似的方法，对双时相特征在空间维度上进行池化和卷积操作，生成空间权重 $W_{s1}^{\prime }$ 和 $W_{s2}^{\prime }$ ，用于衡量空间位置的重要性。

2.2.3 特征融合

• 将通道权重和空间权重相加，得到双时相特征的综合权重，再与原始特征相乘并求和，公式为：
  $$\text{Output}=(W_{c1}^{\prime }+W_{s1}^{\prime })\cdot T_1+(W_{c2}^{\prime }+W_{s2}^{\prime })\cdot T_2$$
  最终输出融合后的特征，通过权重分配保留有用信息，丢弃冗余部分。

2.3 优势

1. 有效利用时序信息: 通过对比双时相特征在通道和空间维度的权重，TFAM能够 捕捉跨时相的重要特征 ，避免了传统方法中忽略时序关联的缺陷。

2. 增强特征融合的准确性

   • 通道与空间联合优化 ：同时在两个维度上进行注意力机制计算，使模型能够从“哪里变”（空间）和“什么类型变”（通道语义）两个层面精准定位变化区域。
   • 动态权重分配 ：Softmax归一化确保双时相特征的权重和为1，避免了简单相加或拼接可能引入的噪声放大问题，提升了融合特征的纯净度。

3. 轻量化与高效性

   • 模块通过1D卷积和池化操作实现，计算复杂度低，且可嵌入到主流网络架构中（如EDED backbone），适用于实时或资源受限的场景。
   • 实验表明，引入TFAM后，模型在LEVIR-CD数据集上的F1分数提升了约0.3%，验证了其有效性。

总结

TFAM模块通过 时序信息驱动的注意力机制 ，解决了双时相特征融合中时序关联不足的问题，实现了更精准的变化区域定位和特征表示。其结构轻量、泛化性强，为变化检测任务提供了一种高效的特征融合解决方案。

论文： https://ieeexplore.ieee.org/document/10296953

三、TFAM的实现代码

TFAM 的实现代码如下：

import torch
import torch.nn as nn
import math

def kernel_size(in_channel):
    """Compute kernel size for one dimension convolution in eca-net"""
    k = int((math.log2(in_channel) + 1) // 2)  # parameters from ECA-net
    if k % 2 == 0:
        return k + 1
    else:
        return k

class TFAM(nn.Module):
    """Fuse two feature into one feature."""

    def __init__(self, in_channel):
        super().__init__()

        self.avg_pool = nn.AdaptiveAvgPool2d(1)
        self.max_pool = nn.AdaptiveMaxPool2d(1)
        self.k = kernel_size(in_channel)
        self.channel_conv1 = nn.Conv1d(4, 1, kernel_size=self.k, padding=self.k // 2)
        self.channel_conv2 = nn.Conv1d(4, 1, kernel_size=self.k, padding=self.k // 2)
        self.spatial_conv1 = nn.Conv2d(4, 1, kernel_size=7, padding=3)
        self.spatial_conv2 = nn.Conv2d(4, 1, kernel_size=7, padding=3)
        self.softmax = nn.Softmax(0)

    def forward(self, x, log=None, module_name=None, img_name=None):
        t1 = x[0]  # 拆分输入元组为 t1
        t2 = x[1]  # 拆分输入元组为 t2
        # channel part
        t1_channel_avg_pool = self.avg_pool(t1)  # b,c,1,1
        t1_channel_max_pool = self.max_pool(t1)  # b,c,1,1
        t2_channel_avg_pool = self.avg_pool(t2)  # b,c,1,1
        t2_channel_max_pool = self.max_pool(t2)  # b,c,1,1

        channel_pool = torch.cat([t1_channel_avg_pool, t1_channel_max_pool,
                                  t2_channel_avg_pool, t2_channel_max_pool],
                                 dim=2).squeeze(-1).transpose(1, 2)  # b,4,c
        t1_channel_attention = self.channel_conv1(channel_pool)  # b,1,c
        t2_channel_attention = self.channel_conv2(channel_pool)  # b,1,c
        channel_stack = torch.stack([t1_channel_attention, t2_channel_attention],
                                    dim=0)  # 2,b,1,c
        channel_stack = self.softmax(channel_stack).transpose(-1, -2).unsqueeze(-1)  # 2,b,c,1,1

        # spatial part
        t1_spatial_avg_pool = torch.mean(t1, dim=1, keepdim=True)  # b,1,h,w
        t1_spatial_max_pool = torch.max(t1, dim=1, keepdim=True)[0]  # b,1,h,w
        t2_spatial_avg_pool = torch.mean(t2, dim=1, keepdim=True)  # b,1,h,w
        t2_spatial_max_pool = torch.max(t2, dim=1, keepdim=True)[0]  # b,1,h,w
        spatial_pool = torch.cat([t1_spatial_avg_pool, t1_spatial_max_pool,
                                  t2_spatial_avg_pool, t2_spatial_max_pool], dim=1)  # b,4,h,w
        t1_spatial_attention = self.spatial_conv1(spatial_pool)  # b,1,h,w
        t2_spatial_attention = self.spatial_conv2(spatial_pool)  # b,1,h,w
        spatial_stack = torch.stack([t1_spatial_attention, t2_spatial_attention], dim=0)  # 2,b,1,h,w
        spatial_stack = self.softmax(spatial_stack)  # 2,b,1,h,w

        # fusion part, add 1 means residual add
        stack_attention = channel_stack + spatial_stack + 1  # 2,b,c,h,w
        fuse = stack_attention[0] * t1 + stack_attention[1] * t2  # b,c,h,w

        return fuse

四、融合步骤

4.1 修改一

① 在 ultralytics/nn/ 目录下新建 AddModules 文件夹用于存放模块代码

② 在 AddModules 文件夹下新建 TFAM.py ，将 第三节 中的代码粘贴到此处

4.2 修改二

在 AddModules 文件夹下新建 __init__.py （已有则不用新建），在文件内导入模块： from .TFAM import *

4.3 修改三

在 ultralytics/nn/modules/tasks.py 文件中，需要在两处位置添加各模块类名称。

首先：导入模块

其次：在 parse_model函数 中注册 TFAM 模块

        elif m in {TFAM}:
            c2 = ch[f[0]]
            args = [c2]

在 DetectionModel 类下，添加如下代码

try:
   m.stride = torch.tensor([s / x.shape[-2] for x in _forward(torch.zeros(1, ch, s, s))])  # forward on CPU
except RuntimeError:
   try:
       self.model.to(torch.device('cuda'))
       m.stride = torch.tensor([s / x.shape[-2] for x in _forward(
           torch.zeros(1, ch, s, s).to(torch.device('cuda')))])  # forward on CUDA
   except RuntimeError as error:
       raise error

并注释这一行

# m.stride = torch.tensor([s / x.shape[-2] for x in _forward(torch.zeros(1, ch, s, s))])  # forward

---

五、yaml模型文件

5.1 中期融合⭐

📌 此模型的修方法是将原本的中期融合中的Concat融合部分换成TFAM，融合骨干部分的多模态信息。

# Ultralytics YOLO 🚀, AGPL-3.0 license
# RT-DETR-ResNet50 object detection model with P3-P5 outputs.

# Parameters
ch: 6
nc: 80 # number of classes
scales: # model compound scaling constants, i.e. 'model=yolov8n-cls.yaml' will call yolov8-cls.yaml with scale 'n'
  # [depth, width, max_channels]
  l: [1.00, 1.00, 1024]

backbone:
  # [from, repeats, module, args]
  - [-1, 1, IN, []]  # 0
  - [-1, 1, Multiin, [1]]  # 1
  - [-2, 1, Multiin, [2]]  # 2

  - [1, 1, ConvNormLayer, [32, 3, 2, 1, 'relu']] # 3-P1
  - [-1, 1, ConvNormLayer, [32, 3, 1, 1, 'relu']] # 4
  - [-1, 1, ConvNormLayer, [64, 3, 1, 1, 'relu']] # 5
  - [-1, 1, nn.MaxPool2d, [3, 2, 1]] # 6-P2

  - [-1, 2, Blocks, [64,  BasicBlock, 2, False]] # 7
  - [-1, 2, Blocks, [128, BasicBlock, 3, False]] # 8-P3
  - [-1, 2, Blocks, [256, BasicBlock, 4, False]] # 9-P4
  - [-1, 2, Blocks, [512, BasicBlock, 5, False]] # 10-P5

  - [2, 1, ConvNormLayer, [32, 3, 2, 1, 'relu']] # 11-P1
  - [-1, 1, ConvNormLayer, [32, 3, 1, 1, 'relu']] # 12
  - [-1, 1, ConvNormLayer, [64, 3, 1, 1, 'relu']] # 13
  - [-1, 1, nn.MaxPool2d, [3, 2, 1]] # 14-P2

  - [-1, 2, Blocks, [64,  BasicBlock, 2, False]] # 15
  - [-1, 2, Blocks, [128, BasicBlock, 3, False]] # 16-P3
  - [-1, 2, Blocks, [256, BasicBlock, 4, False]] # 17-P4
  - [-1, 2, Blocks, [512, BasicBlock, 5, False]] # 18-P5

  - [[8, 16], 1, TFAM, [128]]  # 19 cat backbone P3
  - [[9, 17], 1, TFAM, [256]]  # 20 cat backbone P4
  - [[10, 18], 1, TFAM, [512]]  # 21 cat backbone P5

head:
  - [-1, 1, Conv, [256, 1, 1, None, 1, 1, False]]  # 22 input_proj.2
  - [-1, 1, AIFI, [1024, 8]]
  - [-1, 1, Conv, [256, 1, 1]]  # 24, Y5, lateral_convs.0

  - [-1, 1, nn.Upsample, [None, 2, 'nearest']] # 25
  - [20, 1, Conv, [256, 1, 1, None, 1, 1, False]]  # 26 input_proj.1
  - [[-2, -1], 1, Concat, [1]]
  - [-1, 3, RepC3, [256, 0.5]]  # 28, fpn_blocks.0
  - [-1, 1, Conv, [256, 1, 1]]  # 29, Y4, lateral_convs.1

  - [-1, 1, nn.Upsample, [None, 2, 'nearest']] # 30
  - [19, 1, Conv, [256, 1, 1, None, 1, 1, False]]  # 31 input_proj.0
  - [[-2, -1], 1, Concat, [1]]  # 32 cat backbone P4
  - [-1, 3, RepC3, [256, 0.5]]  # X3 (33), fpn_blocks.1

  - [-1, 1, Conv, [256, 3, 2]]  # 34, downsample_convs.0
  - [[-1, 29], 1, Concat, [1]]  # 35 cat Y4
  - [-1, 3, RepC3, [256, 0.5]]  # F4 (36), pan_blocks.0

  - [-1, 1, Conv, [256, 3, 2]]  # 37, downsample_convs.1
  - [[-1, 24], 1, Concat, [1]]  # 38 cat Y5
  - [-1, 3, RepC3, [256, 0.5]]  # F5 (39), pan_blocks.1

  - [[33, 36, 39], 1, RTDETRDecoder, [nc, 256, 300, 4, 8, 3]]  # Detect(P3, P4, P5)

5.2 中-后期融合⭐

📌 此模型的修方法是将原本的中-后期融合中的Concat融合部分换成TFAM，融合FPN部分的多模态信息。

# Ultralytics YOLO 🚀, AGPL-3.0 license
# RT-DETR-ResNet50 object detection model with P3-P5 outputs.

# Parameters
ch: 6
nc: 80 # number of classes
scales: # model compound scaling constants, i.e. 'model=yolov8n-cls.yaml' will call yolov8-cls.yaml with scale 'n'
  # [depth, width, max_channels]
  l: [1.00, 1.00, 1024]

backbone:
  # [from, repeats, module, args]
  - [-1, 1, IN, []]  # 0
  - [-1, 1, Multiin, [1]]  # 1
  - [-2, 1, Multiin, [2]]  # 2

  - [1, 1, ConvNormLayer, [32, 3, 2, 1, 'relu']] # 3-P1
  - [-1, 1, ConvNormLayer, [32, 3, 1, 1, 'relu']] # 4
  - [-1, 1, ConvNormLayer, [64, 3, 1, 1, 'relu']] # 5
  - [-1, 1, nn.MaxPool2d, [3, 2, 1]] # 6-P2

  - [-1, 2, Blocks, [64,  BasicBlock, 2, False]] # 7
  - [-1, 2, Blocks, [128, BasicBlock, 3, False]] # 8-P3
  - [-1, 2, Blocks, [256, BasicBlock, 4, False]] # 9-P4
  - [-1, 2, Blocks, [512, BasicBlock, 5, False]] # 10-P5

  - [2, 1, ConvNormLayer, [32, 3, 2, 1, 'relu']] # 11-P1
  - [-1, 1, ConvNormLayer, [32, 3, 1, 1, 'relu']] # 12
  - [-1, 1, ConvNormLayer, [64, 3, 1, 1, 'relu']] # 13
  - [-1, 1, nn.MaxPool2d, [3, 2, 1]] # 14-P2

  - [-1, 2, Blocks, [64,  BasicBlock, 2, False]] # 15
  - [-1, 2, Blocks, [128, BasicBlock, 3, False]] # 16-P3
  - [-1, 2, Blocks, [256, BasicBlock, 4, False]] # 17-P4
  - [-1, 2, Blocks, [512, BasicBlock, 5, False]] # 18-P5

head:
  - [10, 1, Conv, [256, 1, 1, None, 1, 1, False]]  # 19 input_proj.2
  - [-1, 1, AIFI, [1024, 8]]
  - [-1, 1, Conv, [256, 1, 1]]  # 21, Y5, lateral_convs.0

  - [-1, 1, nn.Upsample, [None, 2, 'nearest']] # 22
  - [9, 1, Conv, [256, 1, 1, None, 1, 1, False]]  # 23 input_proj.1
  - [[-2, -1], 1, Concat, [1]]
  - [-1, 3, RepC3, [256, 0.5]]  # 25, fpn_blocks.0
  - [-1, 1, Conv, [256, 1, 1]]  # 26, Y4, lateral_convs.1

  - [-1, 1, nn.Upsample, [None, 2, 'nearest']] # 27
  - [8, 1, Conv, [256, 1, 1, None, 1, 1, False]]  # 28 input_proj.0
  - [[-2, -1], 1, Concat, [1]]  # 29 cat backbone P4
  - [-1, 3, RepC3, [256, 0.5]]  # X3 (30), fpn_blocks.1

  - [18, 1, Conv, [256, 1, 1, None, 1, 1, False]]  # 31 input_proj.2
  - [-1, 1, AIFI, [1024, 8]]
  - [-1, 1, Conv, [256, 1, 1]]  # 33, Y5, lateral_convs.0

  - [-1, 1, nn.Upsample, [None, 2, 'nearest']] # 34
  - [17, 1, Conv, [256, 1, 1, None, 1, 1, False]]  # 35 input_proj.1
  - [[-2, -1], 1, Concat, [1]]
  - [-1, 3, RepC3, [256, 0.5]]  # 37, fpn_blocks.0
  - [-1, 1, Conv, [256, 1, 1]]  # 38, Y4, lateral_convs.1

  - [-1, 1, nn.Upsample, [None, 2, 'nearest']] # 39
  - [16, 1, Conv, [256, 1, 1, None, 1, 1, False]]  # 40 input_proj.0
  - [[-2, -1], 1, Concat, [1]]  # 41 cat backbone P4
  - [-1, 3, RepC3, [256, 0.5]]  # X3 (42), fpn_blocks.1

  - [[21, 33], 1, TFAM, [256]]  # 43 cat backbone P3
  - [[26, 38], 1, TFAM, [256]]  # 44 cat backbone P4
  - [[30, 42], 1, TFAM, [256]]  # 45 cat backbone P5

  - [-1, 1, Conv, [256, 3, 2]]  # 46, downsample_convs.0
  - [[-1, 44], 1, Concat, [1]]  # 47 cat Y4
  - [-1, 3, RepC3, [256, 0.5]]  # F4 (48), pan_blocks.0

  - [-1, 1, Conv, [256, 3, 2]]  # 49, downsample_convs.1
  - [[-1, 43], 1, Concat, [1]]  # 50 cat Y5
  - [-1, 3, RepC3, [256, 0.5]]  # F5 (51), pan_blocks.1

  - [[45, 48, 51], 1, RTDETRDecoder, [nc, 256, 300, 4, 8, 3]]  # Detect(P3, P4, P5)

5.3 后期融合⭐

📌 此模型的修方法是将原本的后期融合中的Concat融合部分换成TFAM，融合颈部部分的多模态信息。

# Ultralytics YOLO 🚀, AGPL-3.0 license
# RT-DETR-ResNet50 object detection model with P3-P5 outputs.

# Parameters
ch: 6
nc: 80 # number of classes
scales: # model compound scaling constants, i.e. 'model=yolov8n-cls.yaml' will call yolov8-cls.yaml with scale 'n'
  # [depth, width, max_channels]
  l: [1.00, 1.00, 1024]

backbone:
  # [from, repeats, module, args]
  - [-1, 1, IN, []]  # 0
  - [-1, 1, Multiin, [1]]  # 1
  - [-2, 1, Multiin, [2]]  # 2

  - [1, 1, ConvNormLayer, [32, 3, 2, 1, 'relu']] # 3-P1
  - [-1, 1, ConvNormLayer, [32, 3, 1, 1, 'relu']] # 4
  - [-1, 1, ConvNormLayer, [64, 3, 1, 1, 'relu']] # 5
  - [-1, 1, nn.MaxPool2d, [3, 2, 1]] # 6-P2

  - [-1, 2, Blocks, [64,  BasicBlock, 2, False]] # 7
  - [-1, 2, Blocks, [128, BasicBlock, 3, False]] # 8-P3
  - [-1, 2, Blocks, [256, BasicBlock, 4, False]] # 9-P4
  - [-1, 2, Blocks, [512, BasicBlock, 5, False]] # 10-P5

  - [2, 1, ConvNormLayer, [32, 3, 2, 1, 'relu']] # 11-P1
  - [-1, 1, ConvNormLayer, [32, 3, 1, 1, 'relu']] # 12
  - [-1, 1, ConvNormLayer, [64, 3, 1, 1, 'relu']] # 13
  - [-1, 1, nn.MaxPool2d, [3, 2, 1]] # 14-P2

  - [-1, 2, Blocks, [64,  BasicBlock, 2, False]] # 15
  - [-1, 2, Blocks, [128, BasicBlock, 3, False]] # 16-P3
  - [-1, 2, Blocks, [256, BasicBlock, 4, False]] # 17-P4
  - [-1, 2, Blocks, [512, BasicBlock, 5, False]] # 18-P5

head:
  - [10, 1, Conv, [256, 1, 1, None, 1, 1, False]]  # 19 input_proj.2
  - [-1, 1, AIFI, [1024, 8]]
  - [-1, 1, Conv, [256, 1, 1]]  # 21, Y5, lateral_convs.0

  - [-1, 1, nn.Upsample, [None, 2, 'nearest']] # 22
  - [9, 1, Conv, [256, 1, 1, None, 1, 1, False]]  # 23 input_proj.1
  - [[-2, -1], 1, Concat, [1]]
  - [-1, 3, RepC3, [256, 0.5]]  # 25, fpn_blocks.0
  - [-1, 1, Conv, [256, 1, 1]]  # 26, Y4, lateral_convs.1

  - [-1, 1, nn.Upsample, [None, 2, 'nearest']] # 27
  - [8, 1, Conv, [256, 1, 1, None, 1, 1, False]]  # 28 input_proj.0
  - [[-2, -1], 1, Concat, [1]]  # 29 cat backbone P4
  - [-1, 3, RepC3, [256, 0.5]]  # X3 (30), fpn_blocks.1

  - [-1, 1, Conv, [256, 3, 2]]  # 31, downsample_convs.0
  - [[-1, 26], 1, Concat, [1]]  # 32 cat Y4
  - [-1, 3, RepC3, [256, 0.5]]  # F4 (33), pan_blocks.0

  - [-1, 1, Conv, [256, 3, 2]]  # 34, downsample_convs.1
  - [[-1, 21], 1, Concat, [1]]  # 35 cat Y5
  - [-1, 3, RepC3, [256, 0.5]]  # F5 (36), pan_blocks.1

  - [18, 1, Conv, [256, 1, 1, None, 1, 1, False]]  # 37 input_proj.2
  - [-1, 1, AIFI, [1024, 8]]
  - [-1, 1, Conv, [256, 1, 1]]  # 39, Y5, lateral_convs.0

  - [-1, 1, nn.Upsample, [None, 2, 'nearest']] # 40
  - [17, 1, Conv, [256, 1, 1, None, 1, 1, False]]  # 41 input_proj.1
  - [[-2, -1], 1, Concat, [1]]
  - [-1, 3, RepC3, [256, 0.5]]  # 43, fpn_blocks.0
  - [-1, 1, Conv, [256, 1, 1]]  # 44, Y4, lateral_convs.1

  - [-1, 1, nn.Upsample, [None, 2, 'nearest']] # 45
  - [16, 1, Conv, [256, 1, 1, None, 1, 1, False]]  # 46 input_proj.0
  - [[-2, -1], 1, Concat, [1]]  # 47 cat backbone P4
  - [-1, 3, RepC3, [256, 0.5]]  # X3 (48), fpn_blocks.1

  - [-1, 1, Conv, [256, 3, 2]]  # 49, downsample_convs.0
  - [[-1, 44], 1, Concat, [1]]  # 50 cat Y4
  - [-1, 3, RepC3, [256, 0.5]]  # F4 (51), pan_blocks.0

  - [-1, 1, Conv, [256, 3, 2]]  # 52, downsample_convs.1
  - [[-1, 39], 1, Concat, [1]]  # 53 cat Y5
  - [-1, 3, RepC3, [256, 0.5]]  # F5 (54), pan_blocks.1

  - [[30, 48], 1, TFAM, [256]]  # 55 cat backbone P3
  - [[33, 51], 1, TFAM, [256]]  # 56 cat backbone P4
  - [[36, 54], 1, TFAM, [256]]  # 57 cat backbone P5

  - [[55, 56, 57], 1, RTDETRDecoder, [nc, 256, 300, 4, 8, 3]]  # Detect(P3, P4, P5)

---

六、成功运行结果

打印网络模型可以看到不同的融合层已经加入到模型中，并可以进行训练了。

rtdetr-resnet18-mid-TFAM ：

rtdetr-resnet18-mid-TFAM summary: 502 layers, 31,304,176 parameters, 31,304,176 gradients, 92.3 GFLOPs

                   from  n    params  module                                       arguments
  0                  -1  1         0  ultralytics.nn.AddModules.multimodal.IN      []
  1                  -1  1         0  ultralytics.nn.AddModules.multimodal.Multiin [1]
  2                  -2  1         0  ultralytics.nn.AddModules.multimodal.Multiin [2]
  3                   1  1       960  ultralytics.nn.AddModules.ResNet.ConvNormLayer[3, 32, 3, 2, 1, 'relu']
  4                  -1  1      9312  ultralytics.nn.AddModules.ResNet.ConvNormLayer[32, 32, 3, 1, 1, 'relu']
  5                  -1  1     18624  ultralytics.nn.AddModules.ResNet.ConvNormLayer[32, 64, 3, 1, 1, 'relu']
  6                  -1  1         0  torch.nn.modules.pooling.MaxPool2d           [3, 2, 1]
  7                  -1  2    152512  ultralytics.nn.AddModules.ResNet.Blocks      [64, 64, 2, 'BasicBlock', 2, False]
  8                  -1  2    526208  ultralytics.nn.AddModules.ResNet.Blocks      [64, 128, 2, 'BasicBlock', 3, False]
  9                  -1  2   2100992  ultralytics.nn.AddModules.ResNet.Blocks      [128, 256, 2, 'BasicBlock', 4, False]
 10                  -1  2   8396288  ultralytics.nn.AddModules.ResNet.Blocks      [256, 512, 2, 'BasicBlock', 5, False]
 11                   2  1       960  ultralytics.nn.AddModules.ResNet.ConvNormLayer[3, 32, 3, 2, 1, 'relu']
 12                  -1  1      9312  ultralytics.nn.AddModules.ResNet.ConvNormLayer[32, 32, 3, 1, 1, 'relu']
 13                  -1  1     18624  ultralytics.nn.AddModules.ResNet.ConvNormLayer[32, 64, 3, 1, 1, 'relu']
 14                  -1  1         0  torch.nn.modules.pooling.MaxPool2d           [3, 2, 1]
 15                  -1  2    152512  ultralytics.nn.AddModules.ResNet.Blocks      [64, 64, 2, 'BasicBlock', 2, False]
 16                  -1  2    526208  ultralytics.nn.AddModules.ResNet.Blocks      [64, 128, 2, 'BasicBlock', 3, False]
 17                  -1  2   2100992  ultralytics.nn.AddModules.ResNet.Blocks      [128, 256, 2, 'BasicBlock', 4, False]
 18                  -1  2   8396288  ultralytics.nn.AddModules.ResNet.Blocks      [256, 512, 2, 'BasicBlock', 5, False]
 19             [8, 16]  1       436  ultralytics.nn.AddModules.TFAM.TFAM          [128]
 20             [9, 17]  1       436  ultralytics.nn.AddModules.TFAM.TFAM          [256]
 21            [10, 18]  1       436  ultralytics.nn.AddModules.TFAM.TFAM          [512]
 22                  -1  1    131584  ultralytics.nn.modules.conv.Conv             [512, 256, 1, 1, None, 1, 1, False]
 23                  -1  1    789760  ultralytics.nn.modules.transformer.AIFI      [256, 1024, 8]
 24                  -1  1     66048  ultralytics.nn.modules.conv.Conv             [256, 256, 1, 1]
 25                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 26                  20  1     66048  ultralytics.nn.modules.conv.Conv             [256, 256, 1, 1, None, 1, 1, False]
 27            [-2, -1]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 28                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 29                  -1  1     66048  ultralytics.nn.modules.conv.Conv             [256, 256, 1, 1]
 30                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 31                  19  1     33280  ultralytics.nn.modules.conv.Conv             [128, 256, 1, 1, None, 1, 1, False]
 32            [-2, -1]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 33                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 34                  -1  1    590336  ultralytics.nn.modules.conv.Conv             [256, 256, 3, 2]
 35            [-1, 29]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 36                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 37                  -1  1    590336  ultralytics.nn.modules.conv.Conv             [256, 256, 3, 2]
 38            [-1, 24]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 39                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 40        [33, 36, 39]  1   3927956  ultralytics.nn.modules.head.RTDETRDecoder    [9, [256, 256, 256], 256, 300, 4, 8, 3]
rtdetr-resnet18-mid-TFAM summary: 502 layers, 31,304,176 parameters, 31,304,176 gradients, 92.3 GFLOPs

rtdetr-resnet18-mid-to-late-TFAM ：

rtdetr-resnet18-mid-to-late-TFAM summary: 610 layers, 33,772,784 parameters, 33,772,784 gradients, 104.3 GFLOPs

                   from  n    params  module                                       arguments
  0                  -1  1         0  ultralytics.nn.AddModules.multimodal.IN      []
  1                  -1  1         0  ultralytics.nn.AddModules.multimodal.Multiin [1]
  2                  -2  1         0  ultralytics.nn.AddModules.multimodal.Multiin [2]
  3                   1  1       960  ultralytics.nn.AddModules.ResNet.ConvNormLayer[3, 32, 3, 2, 1, 'relu']
  4                  -1  1      9312  ultralytics.nn.AddModules.ResNet.ConvNormLayer[32, 32, 3, 1, 1, 'relu']
  5                  -1  1     18624  ultralytics.nn.AddModules.ResNet.ConvNormLayer[32, 64, 3, 1, 1, 'relu']
  6                  -1  1         0  torch.nn.modules.pooling.MaxPool2d           [3, 2, 1]
  7                  -1  2    152512  ultralytics.nn.AddModules.ResNet.Blocks      [64, 64, 2, 'BasicBlock', 2, False]
  8                  -1  2    526208  ultralytics.nn.AddModules.ResNet.Blocks      [64, 128, 2, 'BasicBlock', 3, False]
  9                  -1  2   2100992  ultralytics.nn.AddModules.ResNet.Blocks      [128, 256, 2, 'BasicBlock', 4, False]
 10                  -1  2   8396288  ultralytics.nn.AddModules.ResNet.Blocks      [256, 512, 2, 'BasicBlock', 5, False]
 11                   2  1       960  ultralytics.nn.AddModules.ResNet.ConvNormLayer[3, 32, 3, 2, 1, 'relu']
 12                  -1  1      9312  ultralytics.nn.AddModules.ResNet.ConvNormLayer[32, 32, 3, 1, 1, 'relu']
 13                  -1  1     18624  ultralytics.nn.AddModules.ResNet.ConvNormLayer[32, 64, 3, 1, 1, 'relu']
 14                  -1  1         0  torch.nn.modules.pooling.MaxPool2d           [3, 2, 1]
 15                  -1  2    152512  ultralytics.nn.AddModules.ResNet.Blocks      [64, 64, 2, 'BasicBlock', 2, False]
 16                  -1  2    526208  ultralytics.nn.AddModules.ResNet.Blocks      [64, 128, 2, 'BasicBlock', 3, False]
 17                  -1  2   2100992  ultralytics.nn.AddModules.ResNet.Blocks      [128, 256, 2, 'BasicBlock', 4, False]
 18                  -1  2   8396288  ultralytics.nn.AddModules.ResNet.Blocks      [256, 512, 2, 'BasicBlock', 5, False]
 19                  10  1    131584  ultralytics.nn.modules.conv.Conv             [512, 256, 1, 1, None, 1, 1, False]
 20                  -1  1    789760  ultralytics.nn.modules.transformer.AIFI      [256, 1024, 8]
 21                  -1  1     66048  ultralytics.nn.modules.conv.Conv             [256, 256, 1, 1]
 22                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 23                   9  1     66048  ultralytics.nn.modules.conv.Conv             [256, 256, 1, 1, None, 1, 1, False]
 24            [-2, -1]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 25                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 26                  -1  1     66048  ultralytics.nn.modules.conv.Conv             [256, 256, 1, 1]
 27                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 28                   8  1     33280  ultralytics.nn.modules.conv.Conv             [128, 256, 1, 1, None, 1, 1, False]
 29            [-2, -1]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 30                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 31                  18  1    131584  ultralytics.nn.modules.conv.Conv             [512, 256, 1, 1, None, 1, 1, False]
 32                  -1  1    789760  ultralytics.nn.modules.transformer.AIFI      [256, 1024, 8]
 33                  -1  1     66048  ultralytics.nn.modules.conv.Conv             [256, 256, 1, 1]
 34                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 35                  17  1     66048  ultralytics.nn.modules.conv.Conv             [256, 256, 1, 1, None, 1, 1, False]
 36            [-2, -1]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 37                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 38                  -1  1     66048  ultralytics.nn.modules.conv.Conv             [256, 256, 1, 1]
 39                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 40                  16  1     33280  ultralytics.nn.modules.conv.Conv             [128, 256, 1, 1, None, 1, 1, False]
 41            [-2, -1]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 42                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 43            [21, 33]  1       436  ultralytics.nn.AddModules.TFAM.TFAM          [256]
 44            [26, 38]  1       436  ultralytics.nn.AddModules.TFAM.TFAM          [256]
 45            [30, 42]  1       436  ultralytics.nn.AddModules.TFAM.TFAM          [256]
 46                  -1  1    590336  ultralytics.nn.modules.conv.Conv             [256, 256, 3, 2]
 47            [-1, 44]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 48                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 49                  -1  1    590336  ultralytics.nn.modules.conv.Conv             [256, 256, 3, 2]
 50            [-1, 43]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 51                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 52        [45, 48, 51]  1   3927956  ultralytics.nn.modules.head.RTDETRDecoder    [9, [256, 256, 256], 256, 300, 4, 8, 3]
rtdetr-resnet18-mid-to-late-TFAM summary: 610 layers, 33,772,784 parameters, 33,772,784 gradients, 104.3 GFLOPs

rtdetr-resnet18-late-TFAM ：

rtdetr-resnet18-late-TFAM summary: 694 layers, 36,269,296 parameters, 36,269,296 gradients, 109.3 GFLOPs

                   from  n    params  module                                       arguments
  0                  -1  1         0  ultralytics.nn.AddModules.multimodal.IN      []
  1                  -1  1         0  ultralytics.nn.AddModules.multimodal.Multiin [1]
  2                  -2  1         0  ultralytics.nn.AddModules.multimodal.Multiin [2]
  3                   1  1       960  ultralytics.nn.AddModules.ResNet.ConvNormLayer[3, 32, 3, 2, 1, 'relu']
  4                  -1  1      9312  ultralytics.nn.AddModules.ResNet.ConvNormLayer[32, 32, 3, 1, 1, 'relu']
  5                  -1  1     18624  ultralytics.nn.AddModules.ResNet.ConvNormLayer[32, 64, 3, 1, 1, 'relu']
  6                  -1  1         0  torch.nn.modules.pooling.MaxPool2d           [3, 2, 1]
  7                  -1  2    152512  ultralytics.nn.AddModules.ResNet.Blocks      [64, 64, 2, 'BasicBlock', 2, False]
  8                  -1  2    526208  ultralytics.nn.AddModules.ResNet.Blocks      [64, 128, 2, 'BasicBlock', 3, False]
  9                  -1  2   2100992  ultralytics.nn.AddModules.ResNet.Blocks      [128, 256, 2, 'BasicBlock', 4, False]
 10                  -1  2   8396288  ultralytics.nn.AddModules.ResNet.Blocks      [256, 512, 2, 'BasicBlock', 5, False]
 11                   2  1       960  ultralytics.nn.AddModules.ResNet.ConvNormLayer[3, 32, 3, 2, 1, 'relu']
 12                  -1  1      9312  ultralytics.nn.AddModules.ResNet.ConvNormLayer[32, 32, 3, 1, 1, 'relu']
 13                  -1  1     18624  ultralytics.nn.AddModules.ResNet.ConvNormLayer[32, 64, 3, 1, 1, 'relu']
 14                  -1  1         0  torch.nn.modules.pooling.MaxPool2d           [3, 2, 1]
 15                  -1  2    152512  ultralytics.nn.AddModules.ResNet.Blocks      [64, 64, 2, 'BasicBlock', 2, False]
 16                  -1  2    526208  ultralytics.nn.AddModules.ResNet.Blocks      [64, 128, 2, 'BasicBlock', 3, False]
 17                  -1  2   2100992  ultralytics.nn.AddModules.ResNet.Blocks      [128, 256, 2, 'BasicBlock', 4, False]
 18                  -1  2   8396288  ultralytics.nn.AddModules.ResNet.Blocks      [256, 512, 2, 'BasicBlock', 5, False]
 19                  10  1    131584  ultralytics.nn.modules.conv.Conv             [512, 256, 1, 1, None, 1, 1, False]
 20                  -1  1    789760  ultralytics.nn.modules.transformer.AIFI      [256, 1024, 8]
 21                  -1  1     66048  ultralytics.nn.modules.conv.Conv             [256, 256, 1, 1]
 22                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 23                   9  1     66048  ultralytics.nn.modules.conv.Conv             [256, 256, 1, 1, None, 1, 1, False]
 24            [-2, -1]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 25                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 26                  -1  1     66048  ultralytics.nn.modules.conv.Conv             [256, 256, 1, 1]
 27                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 28                   8  1     33280  ultralytics.nn.modules.conv.Conv             [128, 256, 1, 1, None, 1, 1, False]
 29            [-2, -1]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 30                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 31                  -1  1    590336  ultralytics.nn.modules.conv.Conv             [256, 256, 3, 2]
 32            [-1, 26]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 33                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 34                  -1  1    590336  ultralytics.nn.modules.conv.Conv             [256, 256, 3, 2]
 35            [-1, 21]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 36                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 37                  18  1    131584  ultralytics.nn.modules.conv.Conv             [512, 256, 1, 1, None, 1, 1, False]
 38                  -1  1    789760  ultralytics.nn.modules.transformer.AIFI      [256, 1024, 8]
 39                  -1  1     66048  ultralytics.nn.modules.conv.Conv             [256, 256, 1, 1]
 40                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 41                  17  1     66048  ultralytics.nn.modules.conv.Conv             [256, 256, 1, 1, None, 1, 1, False]
 42            [-2, -1]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 43                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 44                  -1  1     66048  ultralytics.nn.modules.conv.Conv             [256, 256, 1, 1]
 45                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 46                  16  1     33280  ultralytics.nn.modules.conv.Conv             [128, 256, 1, 1, None, 1, 1, False]
 47            [-2, -1]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 48                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 49                  -1  1    590336  ultralytics.nn.modules.conv.Conv             [256, 256, 3, 2]
 50            [-1, 44]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 51                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 52                  -1  1    590336  ultralytics.nn.modules.conv.Conv             [256, 256, 3, 2]
 53            [-1, 39]  1         0  ultralytics.nn.modules.conv.Concat           [1]
 54                  -1  3    657920  ultralytics.nn.modules.block.RepC3           [512, 256, 3, 0.5]
 55            [30, 48]  1       436  ultralytics.nn.AddModules.TFAM.TFAM          [256]
 56            [33, 51]  1       436  ultralytics.nn.AddModules.TFAM.TFAM          [256]
 57            [36, 54]  1       436  ultralytics.nn.AddModules.TFAM.TFAM          [256]
 58        [55, 56, 57]  1   3927956  ultralytics.nn.modules.head.RTDETRDecoder    [9, [256, 256, 256], 256, 300, 4, 8, 3]
rtdetr-resnet18-late-TFAM summary: 694 layers, 36,269,296 parameters, 36,269,296 gradients, 109.3 GFLOPs