学习资源站

YOLOv11改进-Neck篇-2024最新TPAMI机制FreqFusion二次创新BiFPN（全网独家创新）

一、本文介绍

本文给大家带来的改进机制是利用 2024-TPAMI 最新机制 FreqFusion二次创新BiFPN ， 《Frequency-aware Feature Fusion for Dense Image Prediction》这篇文章的主要贡献是提出了一种新的特征融合方法（FreqFusion），旨在解决密集图像预测任务中的类别内不一致性和边界位移问题。本文将其和 BiFPN 进行结合实现二次创新BiFPN机制， 相比于原始的YOLOv11本文的内容可以达到一定的轻量化，本文的内容在作者的多类别数据集上实现了涨点。

二、原理介绍

官方论文地址： 官方论文地址点击此处即可跳转

官方代码地址： 官方代码地址点击此处即可跳转

《Frequency-aware Feature Fusion for Dense Image Prediction》这篇文章的主要贡献是提出了一种新的特征融合方法，旨在解决密集图像预测任务中的类别内不一致性和边界位移问题。文章中的核心概念较多，以下是简要的总结和理解：

问题定义：
密集图像预测任务（例如语义分割、目标检测和实例分割）依赖于高精度的类别信息和空间边界。但传统的特征融合方法在类别内特征一致性和边界保留上表现不佳，容易导致类别内不一致（类别内部不同部分特征差异大）和边界模糊。

解决方案——FreqFusion：
文章提出了一种**频率感知特征融合（FreqFusion） ，它通过三个主要组件来提升融合效果：
1. 自适应低通滤波器（ALPF） 生成器 ：该模块通过生成空间可变的低通滤波器，平滑高层特征，减少类别内不一致。
2. 偏移生成器：通过重新采样，将类别一致性较高的特征替换掉不一致的特征，进一步增强边界的清晰度。
3. 自适应高通滤波器（AHPF）生成器：用于增强在下采样过程中丢失的高频信息，提升边界细节。

方法优势：
提升类别内一致性：通过ALPF组件减少了对象内部特征的波动，提升了类别内的相似度。
边界优化：通过偏移生成器和AHPF组件修正了对象边界，使得边界更加清晰。
广泛的适用性：该方法在多个任务上验证了其有效性，如语义分割、目标检测和实例分割。

实验结果：
在语义分割任务中，FreqFusion相比现有方法在多个数据集（如Cityscapes和ADE20K）上有显著的提升，例如在ADE20K上比现有最优方法提升了2.8 mIoU。
在目标检测任务中，使用Faster R-CNN的FreqFusion版本在MS COCO数据集 上提升了1.8 AP。
实例分割和全景分割任务中，也实现了显著的 性能 提升。

总结：
FreqFusion通过结合自适应低通和高通滤波器，解决了标准特征融合中的类别内不一致性和边界模糊问题，在多个 计算机视觉 任务上提升了预测性能。

三、核心代码

核心代码使用方式看章节四！ 

  1. # TPAMI 2024：Frequency-aware Feature Fusion for Dense Image Prediction
  3. import torch
  4. import torch.nn as nn
  5. import torch.nn.functional as F
  6. from mmcv.ops.carafe import normal_init, xavier_init, carafe
  7. import warnings
  8. import numpy as np
  10. __all__ = ['FreqFusion']
  12. def normal_init(module, mean=0, std=1, bias=0):
  13.     if hasattr(module, 'weight') and module.weight is not None:
  14.         nn.init.normal_(module.weight, mean, std)
  15.     if hasattr(module, 'bias') and module.bias is not None:
  16.         nn.init.constant_(module.bias, bias)
  19. def constant_init(module, val, bias=0):
  20.     if hasattr(module, 'weight') and module.weight is not None:
  21.         nn.init.constant_(module.weight, val)
  22.     if hasattr(module, 'bias') and module.bias is not None:
  23.         nn.init.constant_(module.bias, bias)
  25. def resize(input,
  26.            size=None,
  27.            scale_factor=None,
  28.            mode='nearest',
  29.            align_corners=None,
  30.            warning=True):
  31.     if warning:
  32.         if size is not None and align_corners:
  33.             input_h, input_w = tuple(int(x) for x in input.shape[2:])
  34.             output_h, output_w = tuple(int(x) for x in size)
  35.             if output_h > input_h or output_w > input_w:
  36.                 if ((output_h > 1 and output_w > 1 and input_h > 1
  37.                      and input_w > 1) and (output_h - 1) % (input_h - 1)
  38.                         and (output_w - 1) % (input_w - 1)):
  39.                     warnings.warn(
  40.                         f'When align_corners={align_corners}, '
  41.                         'the output would more aligned if '
  42.                         f'input size {(input_h, input_w)} is `x+1` and '
  43.                         f'out size {(output_h, output_w)} is `nx+1`')
  44.     return F.interpolate(input, size, scale_factor, mode, align_corners)
  46. def hamming2D(M, N):
  47.     """
  48.     生成二维Hamming窗
  50.     参数：
  51.     - M：窗口的行数
  52.     - N：窗口的列数
  54.     返回：
  55.     - 二维Hamming窗
  56.     """
  57.     # 生成水平和垂直方向上的Hamming窗
  58.     # hamming_x = np.blackman(M)
  59.     # hamming_x = np.kaiser(M)
  60.     hamming_x = np.hamming(M)
  61.     hamming_y = np.hamming(N)
  62.     # 通过外积生成二维Hamming窗
  63.     hamming_2d = np.outer(hamming_x, hamming_y)
  64.     return hamming_2d
  66. class FreqFusion(nn.Module):
  67.     def __init__(self,
  68.                 channels,
  69.                 scale_factor=1,
  70.                 lowpass_kernel=5,
  71.                 highpass_kernel=3,
  72.                 up_group=1,
  73.                 encoder_kernel=3,
  74.                 encoder_dilation=1,
  75.                 compressed_channels=64,
  76.                 align_corners=False,
  77.                 upsample_mode='nearest',
  78.                 feature_resample=False, # use offset generator or not
  79.                 feature_resample_group=4,
  80.                 comp_feat_upsample=True, # use ALPF & AHPF for init upsampling
  81.                 use_high_pass=True,
  82.                 use_low_pass=True,
  83.                 hr_residual=True,
  84.                 semi_conv=True,
  85.                 hamming_window=True, # for regularization, do not matter really
  86.                 feature_resample_norm=True,
  87.                 **kwargs):
  88.         super().__init__()
  89.         hr_channels, lr_channels = channels
  90.         self.scale_factor = scale_factor
  91.         self.lowpass_kernel = lowpass_kernel
  92.         self.highpass_kernel = highpass_kernel
  93.         self.up_group = up_group
  94.         self.encoder_kernel = encoder_kernel
  95.         self.encoder_dilation = encoder_dilation
  96.         self.compressed_channels = compressed_channels
  97.         self.hr_channel_compressor = nn.Conv2d(hr_channels, self.compressed_channels,1)
  98.         self.lr_channel_compressor = nn.Conv2d(lr_channels, self.compressed_channels,1)
  99.         self.content_encoder = nn.Conv2d( # ALPF generator
  100.             self.compressed_channels,
  101.             lowpass_kernel ** 2 * self.up_group * self.scale_factor * self.scale_factor,
  102.             self.encoder_kernel,
  103.             padding=int((self.encoder_kernel - 1) * self.encoder_dilation / 2),
  104.             dilation=self.encoder_dilation,
  105.             groups=1)
  107.         self.align_corners = align_corners
  108.         self.upsample_mode = upsample_mode
  109.         self.hr_residual = hr_residual
  110.         self.use_high_pass = use_high_pass
  111.         self.use_low_pass = use_low_pass
  112.         self.semi_conv = semi_conv
  113.         self.feature_resample = feature_resample
  114.         self.comp_feat_upsample = comp_feat_upsample
  115.         if self.feature_resample:
  116.             self.dysampler = LocalSimGuidedSampler(in_channels=compressed_channels, scale=2, style='lp', groups=feature_resample_group, use_direct_scale=True, kernel_size=encoder_kernel, norm=feature_resample_norm)
  117.         if self.use_high_pass:
  118.             self.content_encoder2 = nn.Conv2d( # AHPF generator
  119.                 self.compressed_channels,
  120.                 highpass_kernel ** 2 * self.up_group * self.scale_factor * self.scale_factor,
  121.                 self.encoder_kernel,
  122.                 padding=int((self.encoder_kernel - 1) * self.encoder_dilation / 2),
  123.                 dilation=self.encoder_dilation,
  124.                 groups=1)
  125.         self.hamming_window = hamming_window
  126.         lowpass_pad=0
  127.         highpass_pad=0
  128.         if self.hamming_window:
  129.             self.register_buffer('hamming_lowpass', torch.FloatTensor(hamming2D(lowpass_kernel + 2 * lowpass_pad, lowpass_kernel + 2 * lowpass_pad))[None, None,])
  130.             self.register_buffer('hamming_highpass', torch.FloatTensor(hamming2D(highpass_kernel + 2 * highpass_pad, highpass_kernel + 2 * highpass_pad))[None, None,])
  131.         else:
  132.             self.register_buffer('hamming_lowpass', torch.FloatTensor([1.0]))
  133.             self.register_buffer('hamming_highpass', torch.FloatTensor([1.0]))
  134.         self.init_weights()
  136.     def init_weights(self):
  137.         for m in self.modules():
  138.             # print(m)
  139.             if isinstance(m, nn.Conv2d):
  140.                 xavier_init(m, distribution='uniform')
  141.         normal_init(self.content_encoder, std=0.001)
  142.         if self.use_high_pass:
  143.             normal_init(self.content_encoder2, std=0.001)
  145.     def kernel_normalizer(self, mask, kernel, scale_factor=None, hamming=1):
  146.         if scale_factor is not None:
  147.             mask = F.pixel_shuffle(mask, self.scale_factor)
  148.         n, mask_c, h, w = mask.size()
  149.         mask_channel = int(mask_c / float(kernel**2))
  150.         # mask = mask.view(n, mask_channel, -1, h, w)
  151.         # mask = F.softmax(mask, dim=2, dtype=mask.dtype)
  152.         # mask = mask.view(n, mask_c, h, w).contiguous()
  154.         mask = mask.view(n, mask_channel, -1, h, w)
  155.         mask = F.softmax(mask, dim=2, dtype=mask.dtype)
  156.         mask = mask.view(n, mask_channel, kernel, kernel, h, w)
  157.         mask = mask.permute(0, 1, 4, 5, 2, 3).view(n, -1, kernel, kernel)
  158.         # mask = F.pad(mask, pad=[padding] * 4, mode=self.padding_mode) # kernel + 2 * padding
  159.         mask = mask * hamming
  160.         mask /= mask.sum(dim=(-1, -2), keepdims=True)
  161.         # print(hamming)
  162.         # print(mask.shape)
  163.         mask = mask.view(n, mask_channel, h, w, -1)
  164.         mask =  mask.permute(0, 1, 4, 2, 3).view(n, -1, h, w).contiguous()
  165.         return mask
  167.     def forward(self, x):
  168.         hr_feat, lr_feat = x
  169.         compressed_hr_feat = self.hr_channel_compressor(hr_feat)
  170.         compressed_lr_feat = self.lr_channel_compressor(lr_feat)
  171.         if self.semi_conv:
  172.             if self.comp_feat_upsample:
  173.                 if self.use_high_pass:
  174.                     mask_hr_hr_feat = self.content_encoder2(compressed_hr_feat)
  175.                     mask_hr_init = self.kernel_normalizer(mask_hr_hr_feat, self.highpass_kernel, hamming=self.hamming_highpass)
  176.                     compressed_hr_feat = compressed_hr_feat + compressed_hr_feat - carafe(compressed_hr_feat, mask_hr_init, self.highpass_kernel, self.up_group, 1)
  178.                     mask_lr_hr_feat = self.content_encoder(compressed_hr_feat)
  179.                     mask_lr_init = self.kernel_normalizer(mask_lr_hr_feat, self.lowpass_kernel, hamming=self.hamming_lowpass)
  181.                     mask_lr_lr_feat_lr = self.content_encoder(compressed_lr_feat)
  182.                     mask_lr_lr_feat = F.interpolate(
  183.                         carafe(mask_lr_lr_feat_lr, mask_lr_init, self.lowpass_kernel, self.up_group, 2), size=compressed_hr_feat.shape[-2:], mode='nearest')
  184.                     mask_lr = mask_lr_hr_feat + mask_lr_lr_feat
  186.                     mask_lr_init = self.kernel_normalizer(mask_lr, self.lowpass_kernel, hamming=self.hamming_lowpass)
  187.                     mask_hr_lr_feat = F.interpolate(
  188.                         carafe(self.content_encoder2(compressed_lr_feat), mask_lr_init, self.lowpass_kernel, self.up_group, 2), size=compressed_hr_feat.shape[-2:], mode='nearest')
  189.                     mask_hr = mask_hr_hr_feat + mask_hr_lr_feat
  190.                 else: raise NotImplementedError
  191.             else:
  192.                 mask_lr = self.content_encoder(compressed_hr_feat) + F.interpolate(self.content_encoder(compressed_lr_feat), size=compressed_hr_feat.shape[-2:], mode='nearest')
  193.                 if self.use_high_pass:
  194.                     mask_hr = self.content_encoder2(compressed_hr_feat) + F.interpolate(self.content_encoder2(compressed_lr_feat), size=compressed_hr_feat.shape[-2:], mode='nearest')
  195.         else:
  196.             compressed_x = F.interpolate(compressed_lr_feat, size=compressed_hr_feat.shape[-2:], mode='nearest') + compressed_hr_feat
  197.             mask_lr = self.content_encoder(compressed_x)
  198.             if self.use_high_pass:
  199.                 mask_hr = self.content_encoder2(compressed_x)
  201.         mask_lr = self.kernel_normalizer(mask_lr, self.lowpass_kernel, hamming=self.hamming_lowpass)
  202.         if self.semi_conv:
  203.                 lr_feat = carafe(lr_feat, mask_lr, self.lowpass_kernel, self.up_group, 2)
  204.         else:
  205.             lr_feat = resize(
  206.                 input=lr_feat,
  207.                 size=hr_feat.shape[2:],
  208.                 mode=self.upsample_mode,
  209.                 align_corners=None if self.upsample_mode == 'nearest' else self.align_corners)
  210.             lr_feat = carafe(lr_feat, mask_lr, self.lowpass_kernel, self.up_group, 1)
  212.         if self.use_high_pass:
  213.             mask_hr = self.kernel_normalizer(mask_hr, self.highpass_kernel, hamming=self.hamming_highpass)
  214.             hr_feat_hf = hr_feat - carafe(hr_feat, mask_hr, self.highpass_kernel, self.up_group, 1)
  215.             if self.hr_residual:
  216.                 # print('using hr_residual')
  217.                 hr_feat = hr_feat_hf + hr_feat
  218.             else:
  219.                 hr_feat = hr_feat_hf
  221.         if self.feature_resample:
  222.             # print(lr_feat.shape)
  223.             lr_feat = self.dysampler(hr_x=compressed_hr_feat,
  224.                                      lr_x=compressed_lr_feat, feat2sample=lr_feat)
  226.         return hr_feat + lr_feat
  230. class LocalSimGuidedSampler(nn.Module):
  231.     """
  232.     offset generator in FreqFusion
  233.     """
  234.     def __init__(self, in_channels, scale=2, style='lp', groups=4, use_direct_scale=True, kernel_size=1, local_window=3, sim_type='cos', norm=True, direction_feat='sim_concat'):
  235.         super().__init__()
  236.         assert scale==2
  237.         assert style=='lp'
  239.         self.scale = scale
  240.         self.style = style
  241.         self.groups = groups
  242.         self.local_window = local_window
  243.         self.sim_type = sim_type
  244.         self.direction_feat = direction_feat
  246.         if style == 'pl':
  247.             assert in_channels >= scale ** 2 and in_channels % scale ** 2 == 0
  248.         assert in_channels >= groups and in_channels % groups == 0
  250.         if style == 'pl':
  251.             in_channels = in_channels // scale ** 2
  252.             out_channels = 2 * groups
  253.         else:
  254.             out_channels = 2 * groups * scale ** 2
  255.         if self.direction_feat == 'sim':
  256.             self.offset = nn.Conv2d(local_window**2 - 1, out_channels, kernel_size=kernel_size, padding=kernel_size//2)
  257.         elif self.direction_feat == 'sim_concat':
  258.             self.offset = nn.Conv2d(in_channels + local_window**2 - 1, out_channels, kernel_size=kernel_size, padding=kernel_size//2)
  259.         else: raise NotImplementedError
  260.         normal_init(self.offset, std=0.001)
  261.         if use_direct_scale:
  262.             if self.direction_feat == 'sim':
  263.                 self.direct_scale = nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, padding=kernel_size//2)
  264.             elif self.direction_feat == 'sim_concat':
  265.                 self.direct_scale = nn.Conv2d(in_channels + local_window**2 - 1, out_channels, kernel_size=kernel_size, padding=kernel_size//2)
  266.             else: raise NotImplementedError
  267.             constant_init(self.direct_scale, val=0.)
  269.         out_channels = 2 * groups
  270.         if self.direction_feat == 'sim':
  271.             self.hr_offset = nn.Conv2d(local_window**2 - 1, out_channels, kernel_size=kernel_size, padding=kernel_size//2)
  272.         elif self.direction_feat == 'sim_concat':
  273.             self.hr_offset = nn.Conv2d(in_channels + local_window**2 - 1, out_channels, kernel_size=kernel_size, padding=kernel_size//2)
  274.         else: raise NotImplementedError
  275.         normal_init(self.hr_offset, std=0.001)
  277.         if use_direct_scale:
  278.             if self.direction_feat == 'sim':
  279.                 self.hr_direct_scale = nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, padding=kernel_size//2)
  280.             elif self.direction_feat == 'sim_concat':
  281.                 self.hr_direct_scale = nn.Conv2d(in_channels + local_window**2 - 1, out_channels, kernel_size=kernel_size, padding=kernel_size//2)
  282.             else: raise NotImplementedError
  283.             constant_init(self.hr_direct_scale, val=0.)
  285.         self.norm = norm
  286.         if self.norm:
  287.             self.norm_hr = nn.GroupNorm(in_channels // 8, in_channels)
  288.             self.norm_lr = nn.GroupNorm(in_channels // 8, in_channels)
  289.         else:
  290.             self.norm_hr = nn.Identity()
  291.             self.norm_lr = nn.Identity()
  292.         self.register_buffer('init_pos', self._init_pos())
  294.     def _init_pos(self):
  295.         h = torch.arange((-self.scale + 1) / 2, (self.scale - 1) / 2 + 1) / self.scale
  296.         return torch.stack(torch.meshgrid([h, h])).transpose(1, 2).repeat(1, self.groups, 1).reshape(1, -1, 1, 1)
  298.     def sample(self, x, offset, scale=None):
  299.         if scale is None: scale = self.scale
  300.         B, _, H, W = offset.shape
  301.         offset = offset.view(B, 2, -1, H, W)
  302.         coords_h = torch.arange(H) + 0.5
  303.         coords_w = torch.arange(W) + 0.5
  304.         coords = torch.stack(torch.meshgrid([coords_w, coords_h])
  305.                              ).transpose(1, 2).unsqueeze(1).unsqueeze(0).type(x.dtype).to(x.device)
  306.         normalizer = torch.tensor([W, H], dtype=x.dtype, device=x.device).view(1, 2, 1, 1, 1)
  307.         coords = 2 * (coords + offset) / normalizer - 1
  308.         coords = F.pixel_shuffle(coords.view(B, -1, H, W), scale).view(
  309.             B, 2, -1, scale * H, scale * W).permute(0, 2, 3, 4, 1).contiguous().flatten(0, 1)
  310.         return F.grid_sample(x.reshape(B * self.groups, -1, x.size(-2), x.size(-1)), coords, mode='bilinear',
  311.                              align_corners=False, padding_mode="border").view(B, -1, scale * H, scale * W)
  313.     def forward(self, hr_x, lr_x, feat2sample):
  314.         hr_x = self.norm_hr(hr_x)
  315.         lr_x = self.norm_lr(lr_x)
  316.         if self.direction_feat == 'sim':
  317.             hr_sim = compute_similarity(hr_x, self.local_window, dilation=2, sim='cos')
  318.             lr_sim = compute_similarity(lr_x, self.local_window, dilation=2, sim='cos')
  319.         elif self.direction_feat == 'sim_concat':
  320.             hr_sim = torch.cat([hr_x, compute_similarity(hr_x, self.local_window, dilation=2, sim='cos')], dim=1)
  321.             lr_sim = torch.cat([lr_x, compute_similarity(lr_x, self.local_window, dilation=2, sim='cos')], dim=1)
  322.             hr_x, lr_x = hr_sim, lr_sim
  323.         # offset = self.get_offset(hr_x, lr_x)
  324.         offset = self.get_offset_lp(hr_x, lr_x, hr_sim, lr_sim)
  325.         return self.sample(feat2sample, offset)
  327.     # def get_offset_lp(self, hr_x, lr_x):
  328.     def get_offset_lp(self, hr_x, lr_x, hr_sim, lr_sim):
  329.         if hasattr(self, 'direct_scale'):
  330.             # offset = (self.offset(lr_x) + F.pixel_unshuffle(self.hr_offset(hr_x), self.scale)) * (self.direct_scale(lr_x) + F.pixel_unshuffle(self.hr_direct_scale(hr_x), self.scale)).sigmoid() + self.init_pos
  331.             offset = (self.offset(lr_sim) + F.pixel_unshuffle(self.hr_offset(hr_sim), self.scale)) * (self.direct_scale(lr_x) + F.pixel_unshuffle(self.hr_direct_scale(hr_x), self.scale)).sigmoid() + self.init_pos
  332.             # offset = (self.offset(lr_sim) + F.pixel_unshuffle(self.hr_offset(hr_sim), self.scale)) * (self.direct_scale(lr_sim) + F.pixel_unshuffle(self.hr_direct_scale(hr_sim), self.scale)).sigmoid() + self.init_pos
  333.         else:
  334.             offset =  (self.offset(lr_x) + F.pixel_unshuffle(self.hr_offset(hr_x), self.scale)) * 0.25 + self.init_pos
  335.         return offset
  337.     def get_offset(self, hr_x, lr_x):
  338.         if self.style == 'pl':
  339.             raise NotImplementedError
  340.         return self.get_offset_lp(hr_x, lr_x)
  343. def compute_similarity(input_tensor, k=3, dilation=1, sim='cos'):
  344.     """
  345.     计算输入张量中每一点与周围KxK范围内的点的余弦相似度。
  347.     参数：
  348.     - input_tensor: 输入张量，形状为[B, C, H, W]
  349.     - k: 范围大小，表示周围KxK范围内的点
  351.     返回：
  352.     - 输出张量，形状为[B, KxK-1, H, W]
  353.     """
  354.     B, C, H, W = input_tensor.shape
  355.     # 使用零填充来处理边界情况
  356.     # padded_input = F.pad(input_tensor, (k // 2, k // 2, k // 2, k // 2), mode='constant', value=0)
  358.     # 展平输入张量中每个点及其周围KxK范围内的点
  359.     unfold_tensor = F.unfold(input_tensor, k, padding=(k // 2) * dilation, dilation=dilation) # B, CxKxK, HW
  360.     # print(unfold_tensor.shape)
  361.     unfold_tensor = unfold_tensor.reshape(B, C, k**2, H, W)
  363.     # 计算余弦相似度
  364.     if sim == 'cos':
  365.         similarity = F.cosine_similarity(unfold_tensor[:, :, k * k // 2:k * k // 2 + 1], unfold_tensor[:, :, :], dim=1)
  366.     elif sim == 'dot':
  367.         similarity = unfold_tensor[:, :, k * k // 2:k * k // 2 + 1] * unfold_tensor[:, :, :]
  368.         similarity = similarity.sum(dim=1)
  369.     else:
  370.         raise NotImplementedError
  372.     # 移除中心点的余弦相似度，得到[KxK-1]的结果
  373.     similarity = torch.cat((similarity[:, :k * k // 2], similarity[:, k * k // 2 + 1:]), dim=1)
  375.     # 将结果重塑回[B, KxK-1, H, W]的形状
  376.     similarity = similarity.view(B, k * k - 1, H, W)
  377.     return similarity

四、添加方法

4.1 修改一

第一还是建立文件，我们找到如下 ultralytics /nn文件夹下建立一个目录名字呢就是'Addmodules'文件夹( 用群内的文件的话已经有了无需新建) ！然后在其内部建立一个新的py文件将核心代码复制粘贴进去即可。

​

4.2 修改二

第二步我们在该目录下创建一个新的py文件名字为'__init__.py'( 用群内的文件的话已经有了无需新建) ，然后在其内部导入我们的检测头如下图所示。

​

4.3 修改三

第三步我门中到如下文件'ultralytics/nn/tasks.py'进行导入和注册我们的模块( 用群内的文件的话已经有了无需重新导入直接开始第四步即可) ！

​

4.4 修改四

按照我的添加在parse_model里添加即可。 

  1.         elif m in {FreqFusion}:
  2.             c2 = ch[f[0]]
  3.             args = [[ch[x] for x in f], *args]

​

4.5 修改五

第五步我门中到如下文件'ultralytics/nn/tasks.py'进行修改，按照红框的位置进行定位，用我给的代码进行替换红框中的代码. 

  1.             try:
  2.                 m.stride = torch.tensor([s / x.shape[-2] for x in _forward(torch.zeros(1, ch, s, s))])  # forward on CPU
  3.             except RuntimeError:
  4.                 try:
  5.                     self.model.to(torch.device('cuda'))
  6.                     m.stride = torch.tensor([s / x.shape[-2] for x in _forward(
  7.                         torch.zeros(1, ch, s, s).to(torch.device('cuda')))])  # forward on CUDA
  8.                 except RuntimeError as error:
  9.                     raise error

到此就修改完成了，大家可以复制下面的yaml文件运行。

五、正式训练

5.1 yaml文件

训练信息：YOLO11n-FreqFusion-BiFPN summary: 356 layers, 2,441,503 parameters, 2,441,487 gradients, 6.9 GFLOPs

注意：本文的机制需要关闭AMP训练否则会报错. 

  1. # Ultralytics YOLO 🚀, AGPL-3.0 license
  2. # YOLO11 object detection model with P3-P5 outputs. For Usage examples see https://docs.ultralytics.com/tasks/detect
  4. # Parameters
  5. nc: 80 # number of classes
  6. scales: # model compound scaling constants, i.e. 'model=yolo11n.yaml' will call yolo11.yaml with scale 'n'
  7.   # [depth, width, max_channels]
  8.   n: [0.50, 0.25, 1024] # summary: 319 layers, 2624080 parameters, 2624064 gradients, 6.6 GFLOPs
  9.   s: [0.50, 0.50, 1024] # summary: 319 layers, 9458752 parameters, 9458736 gradients, 21.7 GFLOPs
  10.   m: [0.50, 1.00, 512] # summary: 409 layers, 20114688 parameters, 20114672 gradients, 68.5 GFLOPs
  11.   l: [1.00, 1.00, 512] # summary: 631 layers, 25372160 parameters, 25372144 gradients, 87.6 GFLOPs
  12.   x: [1.00, 1.50, 512] # summary: 631 layers, 56966176 parameters, 56966160 gradients, 196.0 GFLOPs
  14. # YOLO11n backbone
  15. backbone:
  16.   # [from, repeats, module, args]
  17.   - [-1, 1, Conv, [64, 3, 2]] # 0-P1/2
  18.   - [-1, 1, Conv, [128, 3, 2]] # 1-P2/4
  19.   - [-1, 2, C3k2, [256, False, 0.25]]
  20.   - [-1, 1, Conv, [256, 3, 2]] # 3-P3/8
  21.   - [-1, 2, C3k2, [512, False, 0.25]]
  22.   - [-1, 1, Conv, [512, 3, 2]] # 5-P4/16
  23.   - [-1, 2, C3k2, [512, True]]
  24.   - [-1, 1, Conv, [1024, 3, 2]] # 7-P5/32
  25.   - [-1, 2, C3k2, [1024, True]]
  26.   - [-1, 1, SPPF, [1024, 5]] # 9
  27.   - [-1, 2, C2PSA, [1024]] # 10
  29. # YOLO11n head
  30. head:
  31.   - [4, 1, Conv, [256]]  # 11-P3/8
  32.   - [6, 1, Conv, [256]]  # 12-P4/16
  33.   - [10, 1, Conv, [256]]  # 13-P5/32
  35.   - [[12, -1], 1, FreqFusion, []] # 14
  36.   - [-1, 2, C3k2, [256, False]] # 15-P4/16
  38.   - [[11, -1], 1, FreqFusion, []] # 16
  39.   - [-1, 2, C3k2, [256, False]] # 17-P3/8
  41.   - [1, 1, Conv, [256, 3, 2]] # 18 P2->P3
  42.   - [[-1, 11, 17], 1, Bi_FPN, []] # 19
  43.   - [-1, 2, C3k2, [256, False]] # 20-P3/8
  45.   - [-1, 1, Conv, [256, 3, 2]] # 21 P3->P4
  46.   - [[-1, 12, 15], 1, Bi_FPN, []] # 22
  47.   - [-1, 2, C3k2, [512, False]] # 23-P4/16
  49.   - [-1, 1, Conv, [256, 3, 2]] # 24 P4->P5
  50.   - [[-1, 13], 1, Bi_FPN, []] # 25
  51.   - [-1, 2, C3k2, [1024, True]] # 26-P5/32
  53.   - [[20, 23, 26], 1, Detect, [nc]]  # Detect(P3, P4, P5)

5.2 训练代码

大家可以创建一个py文件将我给的代码复制粘贴进去，配置好自己的文件路径即可运行。 

  1. import warnings
  2. warnings.filterwarnings('ignore')
  3. from ultralytics import YOLO
  5. if __name__ == '__main__':
  6.     model = YOLO('yolov8-MLLA.yaml')
  7.     # 如何切换模型版本, 上面的ymal文件可以改为 yolov8s.yaml就是使用的v8s,
  8.     # 类似某个改进的yaml文件名称为yolov8-XXX.yaml那么如果想使用其它版本就把上面的名称改为yolov8l-XXX.yaml即可（改的是上面YOLO中间的名字不是配置文件的）！
  9.     # model.load('yolov8n.pt') # 是否加载预训练权重,科研不建议大家加载否则很难提升精度
  10.     model.train(data=r"C:\Users\Administrator\PycharmProjects\yolov5-master\yolov5-master\Construction Site Safety.v30-raw-images_latestversion.yolov8\data.yaml",
  11.                 # 如果大家任务是其它的'ultralytics/cfg/default.yaml'找到这里修改task可以改成detect, segment, classify, pose
  12.                 cache=False,
  13.                 imgsz=640,
  14.                 epochs=150,
  15.                 single_cls=False,  # 是否是单类别检测
  16.                 batch=16,
  17.                 close_mosaic=0,
  18.                 workers=0,
  19.                 device='0',
  20.                 optimizer='SGD', # using SGD
  21.                 # resume='runs/train/exp21/weights/last.pt', # 如过想续训就设置last.pt的地址
  22.                 amp=False,  # 如果出现训练损失为Nan可以关闭amp
  23.                 project='runs/train',
  24.                 name='exp',
  25.                 )

5.3 训练过程截图

五、本文总结

到此本文的正式分享内容就结束了，在这里给大家推荐我的YOLOv11改进有效涨点专栏，本专栏目前为新开的平均质量分98分，后期我会根据各种最新的前沿顶会进行论文复现，也会对一些老的改进机制进行补充，如果大家觉得本文帮助到你了，订阅本专栏，关注后续更多的更新~