学习资源站

YOLOv11改进-主干_Backbone篇-2024最新目标检测网络MobileNetV4改进YOLOv11（支持根据yolov11版本nsmlx进行自由放缩通道数）

一、本文介绍

本文给大家带来的改进机制是 MobileNetV4 ，其发布时间是2024.5月 。 MobileNetV4是一种高度优化的 神经网络 架构，专为 移动设备 设计。它最新的改动总结主要有两点 ， 采用了通用反向瓶颈（UIB） 和针对移动 加速器 优化的 Mobile MQA注意力模块 （一种全新的注意力机制）。这些创新有助于在不牺牲准确性的情况下，显著提高推理速度和计算效率。MobileNetV4作为一种移动端的网络，其实它的论文中主要是配合蒸馏技术进行改进， 大家可以搭配本专栏的蒸馏进行二次创新涨点， 本文的代码支持大家根据yolov11系列的nsmlx五个版本的通道放缩系数去主动放缩MobileNetV4轻量化更上一层，同时本文组合方式至少25种以上， 文章内容为个人所创抄袭必究。

欢迎大家订阅我的专栏一起学习YOLO！

二、原理介绍

官方论文地址： 官方论文地址点击此处即可跳转

官方代码地址： 官方代码地址点击此处即可跳转

MobileNetV4是MobileNet系列的最新版本，专为移动设备设计，引入了多种新颖且高效的架构组件。其中最关键的是通用反转瓶颈（UIB），它结合了以前模型如MobileNetV2的反转瓶颈和新元素，例如ConvNext块和视觉变换器（ViT）中的前馈网络。这种结构允许在不过度复杂化架构搜索过程的情况下，适应性地并有效地扩展模型到各种平台。

此外，MobileNetV4还包括一种名为Mobile MQA的新型注意力机制，该机制通过优化算术运算与内存访问的比率，显著提高了移动加速器上的推理速度，这是移动 性能 的关键因素。该架构通过精细的神经 网络架构 搜索（NAS）和新颖的蒸馏技术进一步优化，使得MobileNetV4能够在多种硬件平台上达到最优性能，包括移动CPU、DSP、GPU和特定的加速器，如Apple的Neural Engine和Google的Pixel EdgeTPU。

此外，MobileNetV4还引入了改进的NAS策略，通过粗粒度和细粒度搜索相结合的方法，显著提高搜索效率并改善模型质量。通过这种方法，MobileNetV4能够实现大多数情况下的Pareto最优性能，这意味着在不同设备上都能达到效率和准确性的最佳平衡。

最后，通过一种新的蒸馏技术，MobileNetV4进一步提高了准确性，其混合型大模型在ImageNet-1K数据集上达到了87%的顶级准确率，同时在Pixel 8 EdgeTPU上的运行时间仅为3.8毫秒。这些特性使MobileNetV4成为适用于移动环境中高效视觉任务的理想选择。

主要思想提取和总结：

1. 通用反转瓶颈（UIB）:

MobileNetV4引入了一种名为通用反转瓶颈（UIB）的新架构组件。UIB是一个灵活的架构单元，融合了反转瓶颈（IB）、ConvNext、前馈网络（FFN），以及新颖的额外深度（ExtraDW）变体。

2. Mobile MQA注意力机制:

为了优化移动加速器的性能，MobileNetV4设计了一个特殊的注意力模块，名为Mobile MQA。这一模块针对移动设备的计算和存储限制进行了优化，提供了高达39%的推理速度提升。

3. 优化的神经架构搜索（NAS）配方:

通过改进的NAS配方，MobileNetV4能够更高效地搜索和优化网络架构，这有助于发现适合特定 硬件 的最优模型配置。

4. 模型蒸馏技术:

引入了一种新的蒸馏技术，用以提高模型的准确性。通过这种技术，MNv4-Hybrid-Large模型在ImageNet-1K上达到了87%的准确率，并且在Pixel 8 EdgeTPU上的运行时间仅为3.8毫秒。

个人总结： MobileNetV4是一个专为移动设备设计的高效 深度学习 模型。它通过整合多种先进技术，如通用反转瓶颈（UIB）、针对移动设备优化的注意力机制（Mobile MQA），以及先进的架构搜索方法（NAS），实现了在不同硬件上的高效运行。这些技术的融合不仅大幅提升了模型的运行速度，还显著提高了准确率。特别是，它的一个变体模型在标准图像识别测试中取得了87%的准确率，运行速度极快。

三、核心代码

核心代码的使用方式看章节四！ 

  1. from typing import Optional
  2. import torch
  3. import torch.nn as nn
  4. import torch.nn.functional as F
  6. __all__ = ['MobileNetV4ConvLarge', 'MobileNetV4ConvSmall', 'MobileNetV4ConvMedium', 'MobileNetV4HybridMedium', 'MobileNetV4HybridLarge']
  8. MNV4ConvSmall_BLOCK_SPECS = {
  9.     "conv0": {
  10.         "block_name": "convbn",
  11.         "num_blocks": 1,
  12.         "block_specs": [
  13.             [3, 32, 3, 2]
  14.         ]
  15.     },
  16.     "layer1": {
  17.         "block_name": "convbn",
  18.         "num_blocks": 2,
  19.         "block_specs": [
  20.             [32, 32, 3, 2],
  21.             [32, 32, 1, 1]
  22.         ]
  23.     },
  24.     "layer2": {
  25.         "block_name": "convbn",
  26.         "num_blocks": 2,
  27.         "block_specs": [
  28.             [32, 96, 3, 2],
  29.             [96, 64, 1, 1]
  30.         ]
  31.     },
  32.     "layer3": {
  33.         "block_name": "uib",
  34.         "num_blocks": 6,
  35.         "block_specs": [
  36.             [64, 96, 5, 5, True, 2, 3],
  37.             [96, 96, 0, 3, True, 1, 2],
  38.             [96, 96, 0, 3, True, 1, 2],
  39.             [96, 96, 0, 3, True, 1, 2],
  40.             [96, 96, 0, 3, True, 1, 2],
  41.             [96, 96, 3, 0, True, 1, 4],
  42.         ]
  43.     },
  44.     "layer4": {
  45.         "block_name": "uib",
  46.         "num_blocks": 6,
  47.         "block_specs": [
  48.             [96,  128, 3, 3, True, 2, 6],
  49.             [128, 128, 5, 5, True, 1, 4],
  50.             [128, 128, 0, 5, True, 1, 4],
  51.             [128, 128, 0, 5, True, 1, 3],
  52.             [128, 128, 0, 3, True, 1, 4],
  53.             [128, 128, 0, 3, True, 1, 4],
  54.         ]
  55.     },
  56.     "layer5": {
  57.         "block_name": "convbn",
  58.         "num_blocks": 2,
  59.         "block_specs": [
  60.             [128, 960, 1, 1],
  61.             [960, 1280, 1, 1]
  62.         ]
  63.     }
  64. }
  66. MNV4ConvMedium_BLOCK_SPECS = {
  67.     "conv0": {
  68.         "block_name": "convbn",
  69.         "num_blocks": 1,
  70.         "block_specs": [
  71.             [3, 32, 3, 2]
  72.         ]
  73.     },
  74.     "layer1": {
  75.         "block_name": "fused_ib",
  76.         "num_blocks": 1,
  77.         "block_specs": [
  78.             [32, 48, 2, 4.0, True]
  79.         ]
  80.     },
  81.     "layer2": {
  82.         "block_name": "uib",
  83.         "num_blocks": 2,
  84.         "block_specs": [
  85.             [48, 80, 3, 5, True, 2, 4],
  86.             [80, 80, 3, 3, True, 1, 2]
  87.         ]
  88.     },
  89.     "layer3": {
  90.         "block_name": "uib",
  91.         "num_blocks": 8,
  92.         "block_specs": [
  93.             [80,  160, 3, 5, True, 2, 6],
  94.             [160, 160, 3, 3, True, 1, 4],
  95.             [160, 160, 3, 3, True, 1, 4],
  96.             [160, 160, 3, 5, True, 1, 4],
  97.             [160, 160, 3, 3, True, 1, 4],
  98.             [160, 160, 3, 0, True, 1, 4],
  99.             [160, 160, 0, 0, True, 1, 2],
  100.             [160, 160, 3, 0, True, 1, 4]
  101.         ]
  102.     },
  103.     "layer4": {
  104.         "block_name": "uib",
  105.         "num_blocks": 11,
  106.         "block_specs": [
  107.             [160, 256, 5, 5, True, 2, 6],
  108.             [256, 256, 5, 5, True, 1, 4],
  109.             [256, 256, 3, 5, True, 1, 4],
  110.             [256, 256, 3, 5, True, 1, 4],
  111.             [256, 256, 0, 0, True, 1, 4],
  112.             [256, 256, 3, 0, True, 1, 4],
  113.             [256, 256, 3, 5, True, 1, 2],
  114.             [256, 256, 5, 5, True, 1, 4],
  115.             [256, 256, 0, 0, True, 1, 4],
  116.             [256, 256, 0, 0, True, 1, 4],
  117.             [256, 256, 5, 0, True, 1, 2]
  118.         ]
  119.     },
  120.     "layer5": {
  121.         "block_name": "convbn",
  122.         "num_blocks": 2,
  123.         "block_specs": [
  124.             [256, 960, 1, 1],
  125.             [960, 1280, 1, 1]
  126.         ]
  127.     }
  128. }
  130. MNV4ConvLarge_BLOCK_SPECS = {
  131.     "conv0": {
  132.         "block_name": "convbn",
  133.         "num_blocks": 1,
  134.         "block_specs": [
  135.             [3, 24, 3, 2]
  136.         ]
  137.     },
  138.     "layer1": {
  139.         "block_name": "fused_ib",
  140.         "num_blocks": 1,
  141.         "block_specs": [
  142.             [24, 48, 2, 4.0, True]
  143.         ]
  144.     },
  145.     "layer2": {
  146.         "block_name": "uib",
  147.         "num_blocks": 2,
  148.         "block_specs": [
  149.             [48, 96, 3, 5, True, 2, 4],
  150.             [96, 96, 3, 3, True, 1, 4]
  151.         ]
  152.     },
  153.     "layer3": {
  154.         "block_name": "uib",
  155.         "num_blocks": 11,
  156.         "block_specs": [
  157.             [96,  192, 3, 5, True, 2, 4],
  158.             [192, 192, 3, 3, True, 1, 4],
  159.             [192, 192, 3, 3, True, 1, 4],
  160.             [192, 192, 3, 3, True, 1, 4],
  161.             [192, 192, 3, 5, True, 1, 4],
  162.             [192, 192, 5, 3, True, 1, 4],
  163.             [192, 192, 5, 3, True, 1, 4],
  164.             [192, 192, 5, 3, True, 1, 4],
  165.             [192, 192, 5, 3, True, 1, 4],
  166.             [192, 192, 5, 3, True, 1, 4],
  167.             [192, 192, 3, 0, True, 1, 4]
  168.         ]
  169.     },
  170.     "layer4": {
  171.         "block_name": "uib",
  172.         "num_blocks": 13,
  173.         "block_specs": [
  174.             [192, 512, 5, 5, True, 2, 4],
  175.             [512, 512, 5, 5, True, 1, 4],
  176.             [512, 512, 5, 5, True, 1, 4],
  177.             [512, 512, 5, 5, True, 1, 4],
  178.             [512, 512, 5, 0, True, 1, 4],
  179.             [512, 512, 5, 3, True, 1, 4],
  180.             [512, 512, 5, 0, True, 1, 4],
  181.             [512, 512, 5, 0, True, 1, 4],
  182.             [512, 512, 5, 3, True, 1, 4],
  183.             [512, 512, 5, 5, True, 1, 4],
  184.             [512, 512, 5, 0, True, 1, 4],
  185.             [512, 512, 5, 0, True, 1, 4],
  186.             [512, 512, 5, 0, True, 1, 4]
  187.         ]
  188.     },
  189.     "layer5": {
  190.         "block_name": "convbn",
  191.         "num_blocks": 2,
  192.         "block_specs": [
  193.             [512, 960, 1, 1],
  194.             [960, 1280, 1, 1]
  195.         ]
  196.     }
  197. }
  199. def mhsa(num_heads, key_dim, value_dim, px):
  200.     if px == 24:
  201.         kv_strides = 2
  202.     elif px == 12:
  203.         kv_strides = 1
  204.     query_h_strides = 1
  205.     query_w_strides = 1
  206.     use_layer_scale = True
  207.     use_multi_query = True
  208.     use_residual = True
  209.     return [
  210.         num_heads, key_dim, value_dim, query_h_strides, query_w_strides, kv_strides,
  211.         use_layer_scale, use_multi_query, use_residual
  212.     ]
  214. MNV4HybridConvMedium_BLOCK_SPECS = {
  215.     "conv0": {
  216.         "block_name": "convbn",
  217.         "num_blocks": 1,
  218.         "block_specs": [
  219.             [3, 32, 3, 2]
  220.         ]
  221.     },
  222.     "layer1": {
  223.         "block_name": "fused_ib",
  224.         "num_blocks": 1,
  225.         "block_specs": [
  226.             [32, 48, 2, 4.0, True]
  227.         ]
  228.     },
  229.     "layer2": {
  230.         "block_name": "uib",
  231.         "num_blocks": 2,
  232.         "block_specs": [
  233.             [48, 80, 3, 5, True, 2, 4],
  234.             [80, 80, 3, 3, True, 1, 2]
  235.         ]
  236.     },
  237.     "layer3": {
  238.         "block_name": "uib",
  239.         "num_blocks": 8,
  240.         "block_specs": [
  241.             [80,  160, 3, 5, True, 2, 6],
  242.             [160, 160, 0, 0, True, 1, 2],
  243.             [160, 160, 3, 3, True, 1, 4],
  244.             [160, 160, 3, 5, True, 1, 4, mhsa(4, 64, 64, 24)],
  245.             [160, 160, 3, 3, True, 1, 4, mhsa(4, 64, 64, 24)],
  246.             [160, 160, 3, 0, True, 1, 4, mhsa(4, 64, 64, 24)],
  247.             [160, 160, 3, 3, True, 1, 4, mhsa(4, 64, 64, 24)],
  248.             [160, 160, 3, 0, True, 1, 4]
  249.         ]
  250.     },
  251.     "layer4": {
  252.         "block_name": "uib",
  253.         "num_blocks": 12,
  254.         "block_specs": [
  255.             [160, 256, 5, 5, True, 2, 6],
  256.             [256, 256, 5, 5, True, 1, 4],
  257.             [256, 256, 3, 5, True, 1, 4],
  258.             [256, 256, 3, 5, True, 1, 4],
  259.             [256, 256, 0, 0, True, 1, 2],
  260.             [256, 256, 3, 5, True, 1, 2],
  261.             [256, 256, 0, 0, True, 1, 2],
  262.             [256, 256, 0, 0, True, 1, 4, mhsa(4, 64, 64, 12)],
  263.             [256, 256, 3, 0, True, 1, 4, mhsa(4, 64, 64, 12)],
  264.             [256, 256, 5, 5, True, 1, 4, mhsa(4, 64, 64, 12)],
  265.             [256, 256, 5, 0, True, 1, 4, mhsa(4, 64, 64, 12)],
  266.             [256, 256, 5, 0, True, 1, 4]
  267.         ]
  268.     },
  269.     "layer5": {
  270.         "block_name": "convbn",
  271.         "num_blocks": 2,
  272.         "block_specs": [
  273.             [256, 960, 1, 1],
  274.             [960, 1280, 1, 1]
  275.         ]
  276.     }
  277. }
  279. MNV4HybridConvLarge_BLOCK_SPECS = {
  280.     "conv0": {
  281.         "block_name": "convbn",
  282.         "num_blocks": 1,
  283.         "block_specs": [
  284.             [3, 24, 3, 2]
  285.         ]
  286.     },
  287.     "layer1": {
  288.         "block_name": "fused_ib",
  289.         "num_blocks": 1,
  290.         "block_specs": [
  291.             [24, 48, 2, 4.0, True]
  292.         ]
  293.     },
  294.     "layer2": {
  295.         "block_name": "uib",
  296.         "num_blocks": 2,
  297.         "block_specs": [
  298.             [48, 96, 3, 5, True, 2, 4],
  299.             [96, 96, 3, 3, True, 1, 4]
  300.         ]
  301.     },
  302.     "layer3": {
  303.         "block_name": "uib",
  304.         "num_blocks": 11,
  305.         "block_specs": [
  306.             [96,  192, 3, 5, True, 2, 4],
  307.             [192, 192, 3, 3, True, 1, 4],
  308.             [192, 192, 3, 3, True, 1, 4],
  309.             [192, 192, 3, 3, True, 1, 4],
  310.             [192, 192, 3, 5, True, 1, 4],
  311.             [192, 192, 5, 3, True, 1, 4],
  312.             [192, 192, 5, 3, True, 1, 4, mhsa(8, 48, 48, 24)],
  313.             [192, 192, 5, 3, True, 1, 4, mhsa(8, 48, 48, 24)],
  314.             [192, 192, 5, 3, True, 1, 4, mhsa(8, 48, 48, 24)],
  315.             [192, 192, 5, 3, True, 1, 4, mhsa(8, 48, 48, 24)],
  316.             [192, 192, 3, 0, True, 1, 4]
  317.         ]
  318.     },
  319.     "layer4": {
  320.         "block_name": "uib",
  321.         "num_blocks": 14,
  322.         "block_specs": [
  323.             [192, 512, 5, 5, True, 2, 4],
  324.             [512, 512, 5, 5, True, 1, 4],
  325.             [512, 512, 5, 5, True, 1, 4],
  326.             [512, 512, 5, 5, True, 1, 4],
  327.             [512, 512, 5, 0, True, 1, 4],
  328.             [512, 512, 5, 3, True, 1, 4],
  329.             [512, 512, 5, 0, True, 1, 4],
  330.             [512, 512, 5, 0, True, 1, 4],
  331.             [512, 512, 5, 3, True, 1, 4],
  332.             [512, 512, 5, 5, True, 1, 4, mhsa(8, 64, 64, 12)],
  333.             [512, 512, 5, 0, True, 1, 4, mhsa(8, 64, 64, 12)],
  334.             [512, 512, 5, 0, True, 1, 4, mhsa(8, 64, 64, 12)],
  335.             [512, 512, 5, 0, True, 1, 4, mhsa(8, 64, 64, 12)],
  336.             [512, 512, 5, 0, True, 1, 4]
  337.         ]
  338.     },
  339.     "layer5": {
  340.         "block_name": "convbn",
  341.         "num_blocks": 2,
  342.         "block_specs": [
  343.             [512, 960, 1, 1],
  344.             [960, 1280, 1, 1]
  345.         ]
  346.     }
  347. }
  349. MODEL_SPECS = {
  350.     "MobileNetV4ConvSmall": MNV4ConvSmall_BLOCK_SPECS,
  351.     "MobileNetV4ConvMedium": MNV4ConvMedium_BLOCK_SPECS,
  352.     "MobileNetV4ConvLarge": MNV4ConvLarge_BLOCK_SPECS,
  353.     "MobileNetV4HybridMedium": MNV4HybridConvMedium_BLOCK_SPECS,
  354.     "MobileNetV4HybridLarge": MNV4HybridConvLarge_BLOCK_SPECS
  355. }
  358. def make_divisible(
  359.         value: float,
  360.         divisor: int,
  361.         min_value: Optional[float] = None,
  362.         round_down_protect: bool = True,
  363. ) -> int:
  364.     """
  365.     This function is copied from here
  366.     "https://github.com/tensorflow/models/blob/master/official/vision/modeling/layers/nn_layers.py"
  368.     This is to ensure that all layers have channels that are divisible by 8.
  370.     Args:
  371.         value: A `float` of original value.
  372.         divisor: An `int` of the divisor that need to be checked upon.
  373.         min_value: A `float` of  minimum value threshold.
  374.         round_down_protect: A `bool` indicating whether round down more than 10%
  375.         will be allowed.
  377.     Returns:
  378.         The adjusted value in `int` that is divisible against divisor.
  379.     """
  380.     if min_value is None:
  381.         min_value = divisor
  382.     new_value = max(min_value, int(value + divisor / 2) // divisor * divisor)
  383.     # Make sure that round down does not go down by more than 10%.
  384.     if round_down_protect and new_value < 0.9 * value:
  385.         new_value += divisor
  386.     return int(new_value)
  389. def conv_2d(inp, oup, kernel_size=3, stride=1, groups=1, bias=False, norm=True, act=True):
  390.     conv = nn.Sequential()
  391.     padding = (kernel_size - 1) // 2
  392.     conv.add_module('conv', nn.Conv2d(inp, oup, kernel_size, stride, padding, bias=bias, groups=groups))
  393.     if norm:
  394.         conv.add_module('BatchNorm2d', nn.BatchNorm2d(oup))
  395.     if act:
  396.         conv.add_module('Activation', nn.ReLU6())
  397.     return conv
  400. class InvertedResidual(nn.Module):
  401.     def __init__(self, inp, oup, stride, expand_ratio, act=False, squeeze_excitation=False):
  402.         super(InvertedResidual, self).__init__()
  403.         self.stride = stride
  404.         assert stride in [1, 2]
  405.         hidden_dim = int(round(inp * expand_ratio))
  406.         self.block = nn.Sequential()
  407.         if expand_ratio != 1:
  408.             self.block.add_module('exp_1x1', conv_2d(inp, hidden_dim, kernel_size=3, stride=stride))
  409.         if squeeze_excitation:
  410.             self.block.add_module('conv_3x3',
  411.                                   conv_2d(hidden_dim, hidden_dim, kernel_size=3, stride=stride, groups=hidden_dim))
  412.         self.block.add_module('red_1x1', conv_2d(hidden_dim, oup, kernel_size=1, stride=1, act=act))
  413.         self.use_res_connect = self.stride == 1 and inp == oup
  415.     def forward(self, x):
  416.         if self.use_res_connect:
  417.             return x + self.block(x)
  418.         else:
  419.             return self.block(x)
  422. class UniversalInvertedBottleneckBlock(nn.Module):
  423.     def __init__(self,
  424.                  inp,
  425.                  oup,
  426.                  start_dw_kernel_size,
  427.                  middle_dw_kernel_size,
  428.                  middle_dw_downsample,
  429.                  stride,
  430.                  expand_ratio
  431.                  ):
  432.         """An inverted bottleneck block with optional depthwises.
  433.         Referenced from here https://github.com/tensorflow/models/blob/master/official/vision/modeling/layers/nn_blocks.py
  434.         """
  435.         super().__init__()
  436.         # Starting depthwise conv.
  437.         self.start_dw_kernel_size = start_dw_kernel_size
  438.         if self.start_dw_kernel_size:
  439.             stride_ = stride if not middle_dw_downsample else 1
  440.             self._start_dw_ = conv_2d(inp, inp, kernel_size=start_dw_kernel_size, stride=stride_, groups=inp, act=False)
  441.         # Expansion with 1x1 convs.
  442.         expand_filters = make_divisible(inp * expand_ratio, 8)
  443.         self._expand_conv = conv_2d(inp, expand_filters, kernel_size=1)
  444.         # Middle depthwise conv.
  445.         self.middle_dw_kernel_size = middle_dw_kernel_size
  446.         if self.middle_dw_kernel_size:
  447.             stride_ = stride if middle_dw_downsample else 1
  448.             self._middle_dw = conv_2d(expand_filters, expand_filters, kernel_size=middle_dw_kernel_size, stride=stride_,
  449.                                       groups=expand_filters)
  450.         # Projection with 1x1 convs.
  451.         self._proj_conv = conv_2d(expand_filters, oup, kernel_size=1, stride=1, act=False)
  453.         # Ending depthwise conv.
  454.         # this not used
  455.         # _end_dw_kernel_size = 0
  456.         # self._end_dw = conv_2d(oup, oup, kernel_size=_end_dw_kernel_size, stride=stride, groups=inp, act=False)
  458.     def forward(self, x):
  459.         if self.start_dw_kernel_size:
  460.             x = self._start_dw_(x)
  461.             # print("_start_dw_", x.shape)
  462.         x = self._expand_conv(x)
  463.         # print("_expand_conv", x.shape)
  464.         if self.middle_dw_kernel_size:
  465.             x = self._middle_dw(x)
  466.             # print("_middle_dw", x.shape)
  467.         x = self._proj_conv(x)
  468.         # print("_proj_conv", x.shape)
  469.         return x
  472. class MultiQueryAttentionLayerWithDownSampling(nn.Module):
  473.     def __init__(self, inp, num_heads, key_dim, value_dim, query_h_strides, query_w_strides, kv_strides,
  474.                  dw_kernel_size=3, dropout=0.0):
  475.         """Multi Query Attention with spatial downsampling.
  476.         Referenced from here https://github.com/tensorflow/models/blob/master/official/vision/modeling/layers/nn_blocks.py
  478.         3 parameters are introduced for the spatial downsampling:
  479.         1. kv_strides: downsampling factor on Key and Values only.
  480.         2. query_h_strides: vertical strides on Query only.
  481.         3. query_w_strides: horizontal strides on Query only.
  483.         This is an optimized version.
  484.         1. Projections in Attention is explict written out as 1x1 Conv2D.
  485.         2. Additional reshapes are introduced to bring a up to 3x speed up.
  486.         """
  487.         super().__init__()
  488.         self.num_heads = num_heads
  489.         self.key_dim = key_dim
  490.         self.value_dim = value_dim
  491.         self.query_h_strides = query_h_strides
  492.         self.query_w_strides = query_w_strides
  493.         self.kv_strides = kv_strides
  494.         self.dw_kernel_size = dw_kernel_size
  495.         self.dropout = dropout
  497.         self.head_dim = key_dim // num_heads
  499.         if self.query_h_strides > 1 or self.query_w_strides > 1:
  500.             self._query_downsampling_norm = nn.BatchNorm2d(inp)
  501.         self._query_proj = conv_2d(inp, num_heads * key_dim, 1, 1, norm=False, act=False)
  503.         if self.kv_strides > 1:
  504.             self._key_dw_conv = conv_2d(inp, inp, dw_kernel_size, kv_strides, groups=inp, norm=True, act=False)
  505.             self._value_dw_conv = conv_2d(inp, inp, dw_kernel_size, kv_strides, groups=inp, norm=True, act=False)
  506.         self._key_proj = conv_2d(inp, key_dim, 1, 1, norm=False, act=False)
  507.         self._value_proj = conv_2d(inp, key_dim, 1, 1, norm=False, act=False)
  509.         self._output_proj = conv_2d(num_heads * key_dim, inp, 1, 1, norm=False, act=False)
  510.         self.dropout = nn.Dropout(p=dropout)
  512.     def forward(self, x):
  513.         batch_size, seq_length, _, _ = x.size()
  514.         if self.query_h_strides > 1 or self.query_w_strides > 1:
  515.             q = F.avg_pool2d(self.query_h_stride, self.query_w_stride)
  516.             q = self._query_downsampling_norm(q)
  517.             q = self._query_proj(q)
  518.         else:
  519.             q = self._query_proj(x)
  520.         px = q.size(2)
  521.         q = q.view(batch_size, self.num_heads, -1, self.key_dim)  # [batch_size, num_heads, seq_length, key_dim]
  523.         if self.kv_strides > 1:
  524.             k = self._key_dw_conv(x)
  525.             k = self._key_proj(k)
  526.             v = self._value_dw_conv(x)
  527.             v = self._value_proj(v)
  528.         else:
  529.             k = self._key_proj(x)
  530.             v = self._value_proj(x)
  531.         k = k.view(batch_size, self.key_dim, -1)  # [batch_size, key_dim, seq_length]
  532.         v = v.view(batch_size, -1, self.key_dim)  # [batch_size, seq_length, key_dim]
  534.         # calculate attn score
  535.         attn_score = torch.matmul(q, k) / (self.head_dim ** 0.5)
  536.         attn_score = self.dropout(attn_score)
  537.         attn_score = F.softmax(attn_score, dim=-1)
  539.         context = torch.matmul(attn_score, v)
  540.         context = context.view(batch_size, self.num_heads * self.key_dim, px, px)
  541.         output = self._output_proj(context)
  542.         return output
  545. class MNV4LayerScale(nn.Module):
  546.     def __init__(self, init_value):
  547.         """LayerScale as introduced in CaiT: https://arxiv.org/abs/2103.17239
  548.         Referenced from here https://github.com/tensorflow/models/blob/master/official/vision/modeling/layers/nn_blocks.py
  550.         As used in MobileNetV4.
  552.         Attributes:
  553.             init_value (float): value to initialize the diagonal matrix of LayerScale.
  554.         """
  555.         super().__init__()
  556.         self.init_value = init_value
  558.     def forward(self, x):
  559.         gamma = self.init_value * torch.ones(x.size(-1), dtype=x.dtype, device=x.device)
  560.         return x * gamma
  563. class MultiHeadSelfAttentionBlock(nn.Module):
  564.     def __init__(
  565.             self,
  566.             inp,
  567.             num_heads,
  568.             key_dim,
  569.             value_dim,
  570.             query_h_strides,
  571.             query_w_strides,
  572.             kv_strides,
  573.             use_layer_scale,
  574.             use_multi_query,
  575.             use_residual=True
  576.     ):
  577.         super().__init__()
  578.         self.query_h_strides = query_h_strides
  579.         self.query_w_strides = query_w_strides
  580.         self.kv_strides = kv_strides
  581.         self.use_layer_scale = use_layer_scale
  582.         self.use_multi_query = use_multi_query
  583.         self.use_residual = use_residual
  585.         self._input_norm = nn.BatchNorm2d(inp)
  586.         if self.use_multi_query:
  587.             self.multi_query_attention = MultiQueryAttentionLayerWithDownSampling(
  588.                 inp, num_heads, key_dim, value_dim, query_h_strides, query_w_strides, kv_strides
  589.             )
  590.         else:
  591.             self.multi_head_attention = nn.MultiheadAttention(inp, num_heads, kdim=key_dim)
  593.         if self.use_layer_scale:
  594.             self.layer_scale_init_value = 1e-5
  595.             self.layer_scale = MNV4LayerScale(self.layer_scale_init_value)
  597.     def forward(self, x):
  598.         # Not using CPE, skipped
  599.         # input norm
  600.         shortcut = x
  601.         x = self._input_norm(x)
  602.         # multi query
  603.         if self.use_multi_query:
  604.             x = self.multi_query_attention(x)
  605.         else:
  606.             x = self.multi_head_attention(x, x)
  607.         # layer scale
  608.         if self.use_layer_scale:
  609.             x = self.layer_scale(x)
  610.         # use residual
  611.         if self.use_residual:
  612.             x = x + shortcut
  613.         return x
  616. def build_blocks(layer_spec, factor=0.25):
  617.     if not layer_spec.get('block_name'):
  618.         return nn.Sequential()
  619.     block_names = layer_spec['block_name']
  620.     layers = nn.Sequential()
  621.     if block_names == "convbn":
  622.         schema_ = ['inp', 'oup', 'kernel_size', 'stride']
  623.         for i in range(layer_spec['num_blocks']):
  624.             args = dict(zip(schema_, layer_spec['block_specs'][i]))
  625.             if args['inp'] != 3:
  626.                 args['inp'] = int(args['inp'] * factor)
  627.             args['oup'] = int(args['oup'] * factor)
  628.             layers.add_module(f"convbn_{i}", conv_2d(**args))
  629.     elif block_names == "uib":
  630.         schema_ = ['inp', 'oup', 'start_dw_kernel_size', 'middle_dw_kernel_size', 'middle_dw_downsample', 'stride',
  631.                    'expand_ratio', 'msha']
  632.         for i in range(layer_spec['num_blocks']):
  633.             args = dict(zip(schema_, layer_spec['block_specs'][i]))
  634.             args['inp'] = int(args['inp'] * factor)
  635.             args['oup'] = int(args['oup'] * factor)
  636.             msha = args.pop("msha") if "msha" in args else 0
  637.             layers.add_module(f"uib_{i}", UniversalInvertedBottleneckBlock(**args))
  638.             if msha:
  639.                 msha_schema_ = [
  640.                     "inp", "num_heads", "key_dim", "value_dim", "query_h_strides", "query_w_strides", "kv_strides",
  641.                     "use_layer_scale", "use_multi_query", "use_residual"
  642.                 ]
  643.                 args['inp'] = int(args['inp'] * factor)
  644.                 args = dict(zip(msha_schema_, [args['oup']] + (msha)))
  646.                 layers.add_module(f"msha_{i}", MultiHeadSelfAttentionBlock(**args))
  647.     elif block_names == "fused_ib":
  648.         schema_ = ['inp', 'oup', 'stride', 'expand_ratio', 'act']
  649.         for i in range(layer_spec['num_blocks']):
  650.             args = dict(zip(schema_, layer_spec['block_specs'][i]))
  651.             args['inp'] = int(args['inp'] * factor)
  652.             args['oup'] = int(args['oup'] * factor)
  653.             layers.add_module(f"fused_ib_{i}", InvertedResidual(**args))
  654.     else:
  655.         raise NotImplementedError
  656.     return layers
  659. class MobileNetV4(nn.Module):
  660.     def __init__(self, model, factor=0.25):
  661.         # MobileNetV4ConvSmall  MobileNetV4ConvMedium  MobileNetV4ConvLarge
  662.         # MobileNetV4HybridMedium  MobileNetV4HybridLarge
  663.         """Params to initiate MobilenNetV4
  664.         Args:
  665.             model : support 5 types of models as indicated in
  666.             "https://github.com/tensorflow/models/blob/master/official/vision/modeling/backbones/mobilenet.py"
  667.         """
  668.         super().__init__()
  669.         assert model in MODEL_SPECS.keys()
  670.         self.model = model
  671.         self.spec = MODEL_SPECS[self.model]
  673.         # conv0
  674.         self.conv0 = build_blocks(self.spec['conv0'], factor=factor)
  675.         # layer1
  676.         self.layer1 = build_blocks(self.spec['layer1'], factor=factor)
  677.         # layer2
  678.         self.layer2 = build_blocks(self.spec['layer2'], factor=factor)
  679.         # layer3
  680.         self.layer3 = build_blocks(self.spec['layer3'], factor=factor)
  681.         # layer4
  682.         self.layer4 = build_blocks(self.spec['layer4'], factor=factor)
  683.         self.width_list = [i.size(1) for i in self.forward(torch.randn(1, 3, 640, 640))]
  684.     def forward(self, x):
  685.         x0 = self.conv0(x)
  686.         x1 = self.layer1(x0)
  687.         x2 = self.layer2(x1)
  688.         x3 = self.layer3(x2)
  689.         x4 = self.layer4(x3)
  690.         return [x1, x2, x3, x4]
  693. def MobileNetV4ConvSmall(factor=0.5):
  694.     model = MobileNetV4('MobileNetV4ConvSmall', factor=factor)
  695.     return model
  697. def MobileNetV4ConvMedium(factor=0.5):
  698.     model = MobileNetV4('MobileNetV4ConvMedium', factor=factor)
  699.     return model
  701. def MobileNetV4ConvLarge(factor=0.5):
  702.     model = MobileNetV4('MobileNetV4ConvLarge', factor=factor)
  703.     return model
  705. def MobileNetV4HybridMedium(factor=0.5):
  706.     model = MobileNetV4('MobileNetV4HybridMedium', factor=factor)
  707.     return model
  709. def MobileNetV4HybridLarge(factor=0.5):
  710.     model = MobileNetV4('MobileNetV4HybridLarge', factor=factor)
  711.     return model
  714. if __name__ == "__main__":
  715.     # Generating Sample image
  716.     image_size = (1, 3, 640, 640)
  717.     image = torch.rand(*image_size)
  719.     # Model
  720.     model = MobileNetV4HybridMedium()
  722.     out = model(image)
  723.     for i in range(len(out)):
  724.         print(out[i].shape)

四、手把手教你添加MobileNetv4

4.1 修改一

第一步还是建立文件，我们找到如下ultralytics/nn文件夹下建立一个目录名字呢就是'Addmodules'文件夹( 用群内的文件的话已经有了无需新建) ！然后在其内部建立一个新的py文件将核心代码复制粘贴进去即可

4.2 修改二

第二步我们在该目录下创建一个新的py文件名字为'__init__.py'( 用群内的文件的话已经有了无需新建) ，然后在其内部导入我们的检测头如下图所示。

4.3 修改三

第三步我门中到如下文件'ultralytics/nn/tasks.py'进行导入和注册我们的模块( 用群内的文件的话已经有了无需重新导入直接开始第四步即可) ！

从今天开始以后的教程就都统一成这个样子了，因为我默认大家用了我群内的文件来进行修改！！

4.4 修改四

添加如下两行代码！！！

​

4.5 修改五

找到七百多行大概把具体看图片，按照图片来修改就行，添加红框内的部分，注意没有()只是函数名。 

  1.         elif m in {自行添加对应的模型即可，下面都是一样的}:
  2.             m = m(*args)
  3.             c2 = m.width_list  # 返回通道列表
  4.             backbone = True

4.6 修改六

下面的两个红框内都是需要改动的。

​ 

  1.         if isinstance(c2, list):
  2.             m_ = m
  3.             m_.backbone = True
  4.         else:
  5.             m_ = nn.Sequential(*(m(*args) for _ in range(n))) if n > 1 else m(*args)  # module
  6.             t = str(m)[8:-2].replace('__main__.', '')  # module type
  9.         m.np = sum(x.numel() for x in m_.parameters())  # number params
  10.         m_.i, m_.f, m_.type = i + 4 if backbone else i, f, t  # attach index, 'from' index, type

4.7 修改七

如下的也需要修改，全部按照我的来。

​

代码如下把原先的代码替换了即可。 

  1.         if verbose:
  2.             LOGGER.info(f'{i:>3}{str(f):>20}{n_:>3}{m.np:10.0f}  {t:<45}{str(args):<30}')  # print
  4.         save.extend(x % (i + 4 if backbone else i) for x in ([f] if isinstance(f, int) else f) if x != -1)  # append to savelist
  5.         layers.append(m_)
  6.         if i == 0:
  7.             ch = []
  8.         if isinstance(c2, list):
  9.             ch.extend(c2)
  10.             if len(c2) != 5:
  11.                 ch.insert(0, 0)
  12.         else:
  13.             ch.append(c2)

4.8 修改八

修改八和前面的都不太一样，需要修改前向传播中的一个部分， 已经离开了parse_model方法了。

可以在图片中开代码行数，没有离开task.py文件都是同一个文件。 同时这个部分有好几个前向传播都很相似，大家不要看错了， 是70多行左右的！！！，同时我后面提供了代码，大家直接复制粘贴即可，有时间我针对这里会出一个视频。

​​

代码如下-> 

  1.     def _predict_once(self, x, profile=False, visualize=False, embed=None):
  2.         """
  3.         Perform a forward pass through the network.
  5.         Args:
  6.             x (torch.Tensor): The input tensor to the model.
  7.             profile (bool):  Print the computation time of each layer if True, defaults to False.
  8.             visualize (bool): Save the feature maps of the model if True, defaults to False.
  9.             embed (list, optional): A list of feature vectors/embeddings to return.
  11.         Returns:
  12.             (torch.Tensor): The last output of the model.
  13.         """
  14.         y, dt, embeddings = [], [], []  # outputs
  15.         for m in self.model:
  16.             if m.f != -1:  # if not from previous layer
  17.                 x = y[m.f] if isinstance(m.f, int) else [x if j == -1 else y[j] for j in m.f]  # from earlier layers
  18.             if profile:
  19.                 self._profile_one_layer(m, x, dt)
  20.             if hasattr(m, 'backbone'):
  21.                 x = m(x)
  22.                 if len(x) != 5:  # 0 - 5
  23.                     x.insert(0, None)
  24.                 for index, i in enumerate(x):
  25.                     if index in self.save:
  26.                         y.append(i)
  27.                     else:
  28.                         y.append(None)
  29.                 x = x[-1]  # 最后一个输出传给下一层
  30.             else:
  31.                 x = m(x)  # run
  32.                 y.append(x if m.i in self.save else None)  # save output
  33.             if visualize:
  34.                 feature_visualization(x, m.type, m.i, save_dir=visualize)
  35.             if embed and m.i in embed:
  36.                 embeddings.append(nn.functional.adaptive_avg_pool2d(x, (1, 1)).squeeze(-1).squeeze(-1))  # flatten
  37.                 if m.i == max(embed):
  38.                     return torch.unbind(torch.cat(embeddings, 1), dim=0)
  39.         return x

到这里就完成了修改部分，但是这里面细节很多，大家千万要注意不要替换多余的代码，导致报错，也不要拉下任何一部，都会导致运行失败，而且报错很难排查！！！很难排查！！！

注意！！！ 额外的修改！

关注我的其实都知道，我大部分的修改都是一样的，这个网络需要额外的修改一步，就是s一个参数，将下面的s改为640！！！即可完美运行！！

打印计算量问题解决方案

我们找到如下文件'ultralytics/utils/torch_utils.py'按照如下的图片进行修改，否则容易打印不出来计算量。

​

注意事项！！！

如果大家在验证的时候报错形状不匹配的错误可以固定验证集的图片尺寸，方法如下 ->

找到下面这个文件ultralytics/models/yolo/detect/train.py然后其中有一个类是DetectionTrainer class中的build_dataset函数中的一个参数rect=mode == 'val'改为rect=False

五、MobileNetv4 的yaml文件

5.1 MobileNetv4 的yaml文件

此版本训练信息：YOLO11-MobileNetV4 summary: 386 layers, 2,173,383 parameters, 2,173,367 gradients, 5.3 GFLOPs

# 支持版本有__all__ = ['MobileNetV4ConvLarge', 'MobileNetV4ConvSmall', 'MobileNetV4ConvMedium', 'MobileNetV4HybridMedium', 'MobileNetV4HybridLarge'] 5
- [-1, 1, MobileNetV4ConvSmall, [0.75]]
# 上面的参数0.75支持大家自由根据v11的 0.25 0.50 0.75 1.00 1.5 进行放缩我只尝试过这几个如果你设置一些比较特殊的数字可能会报错.所以本文有至少 5 x 5 = 25种组合方式.

#本文是用MobileNetV4ConvSmall, [0.75]参数实验 

  1. # Ultralytics YOLO 🚀, AGPL-3.0 license
  2. # YOLO11 object detection model with P3-P5 outputs. For Usage examples see https://docs.ultralytics.com/tasks/detect
  4. # Parameters
  5. nc: 80 # number of classes
  6. scales: # model compound scaling constants, i.e. 'model=yolo11n.yaml' will call yolo11.yaml with scale 'n'
  7.   # [depth, width, max_channels]
  8.   n: [0.50, 0.25, 1024] # summary: 319 layers, 2624080 parameters, 2624064 gradients, 6.6 GFLOPs
  9.   s: [0.50, 0.50, 1024] # summary: 319 layers, 9458752 parameters, 9458736 gradients, 21.7 GFLOPs
  10.   m: [0.50, 1.00, 512] # summary: 409 layers, 20114688 parameters, 20114672 gradients, 68.5 GFLOPs
  11.   l: [1.00, 1.00, 512] # summary: 631 layers, 25372160 parameters, 25372144 gradients, 87.6 GFLOPs
  12.   x: [1.00, 1.50, 512] # summary: 631 layers, 56966176 parameters, 56966160 gradients, 196.0 GFLOPs
  14. # 支持版本有__all__ = ['MobileNetV4ConvLarge', 'MobileNetV4ConvSmall', 'MobileNetV4ConvMedium', 'MobileNetV4HybridMedium', 'MobileNetV4HybridLarge'] 5
  15. # YOLO11n backbone
  16. backbone:
  17.   # [from, repeats, module, args]
  18.   - [-1, 1, MobileNetV4ConvSmall, [0.75]] # 0-4 P1/2 这里是四层大家不要被yaml文件限制住了思维，不会画图进群看视频.
  19.   # 上面的参数0.75支持大家自由根据v11的 0.25 0.50 0.75 1.00 1.5 进行放缩我只尝试过这几个如果你设置一些比较特殊的数字可能会报错.
  20.   # 所以本文有至少 5 x 5 = 25种组合方式.
  21.   - [-1, 1, SPPF, [1024, 5]] # 5
  22.   - [-1, 2, C2PSA, [1024]] # 6
  24. # YOLO11n head
  25. head:
  26.   - [-1, 1, nn.Upsample, [None, 2, "nearest"]]
  27.   - [[-1, 3], 1, Concat, [1]] # cat backbone P4
  28.   - [-1, 2, C3k2, [512, False]] # 9
  30.   - [-1, 1, nn.Upsample, [None, 2, "nearest"]]
  31.   - [[-1, 2], 1, Concat, [1]] # cat backbone P3
  32.   - [-1, 2, C3k2, [256, False]] # 12 (P3/8-small)
  34.   - [-1, 1, Conv, [256, 3, 2]]
  35.   - [[-1, 9], 1, Concat, [1]] # cat head P4
  36.   - [-1, 2, C3k2, [512, False]] # 15 (P4/16-medium)
  38.   - [-1, 1, Conv, [512, 3, 2]]
  39.   - [[-1, 6], 1, Concat, [1]] # cat head P5
  40.   - [-1, 2, C3k2, [1024, True]] # 18 (P5/32-large)
  42.   - [[12, 15, 18], 1, Detect, [nc]] # Detect(P3, P4, P5)

5.2 训练文件的代码

可以复制我的运行文件进行运行。 

  1. import warnings
  2. warnings.filterwarnings('ignore')
  3. from ultralytics import YOLO
  5. if __name__ == '__main__':
  6.     model = YOLO('yolov8-MLLA.yaml')
  7.     # 如何切换模型版本, 上面的ymal文件可以改为 yolov8s.yaml就是使用的v8s,
  8.     # 类似某个改进的yaml文件名称为yolov8-XXX.yaml那么如果想使用其它版本就把上面的名称改为yolov8l-XXX.yaml即可（改的是上面YOLO中间的名字不是配置文件的）！
  9.     # model.load('yolov8n.pt') # 是否加载预训练权重,科研不建议大家加载否则很难提升精度
  10.     model.train(data=r"C:\Users\Administrator\PycharmProjects\yolov5-master\yolov5-master\Construction Site Safety.v30-raw-images_latestversion.yolov8\data.yaml",
  11.                 # 如果大家任务是其它的'ultralytics/cfg/default.yaml'找到这里修改task可以改成detect, segment, classify, pose
  12.                 cache=False,
  13.                 imgsz=640,
  14.                 epochs=150,
  15.                 single_cls=False,  # 是否是单类别检测
  16.                 batch=16,
  17.                 close_mosaic=0,
  18.                 workers=0,
  19.                 device='0',
  20.                 optimizer='SGD', # using SGD
  21.                 # resume='runs/train/exp21/weights/last.pt', # 如过想续训就设置last.pt的地址
  22.                 amp=True,  # 如果出现训练损失为Nan可以关闭amp
  23.                 project='runs/train',
  24.                 name='exp',
  25.                 )

六、成功运行记录

下面是成功运行的截图.

​

七、本文总结

到此本文的正式分享内容就结束了，在这里给大家推荐我的YOLOv11改进有效涨点专栏，本专栏目前为新开的平均质量分98分，后期我会根据各种最新的前沿顶会进行论文复现，也会对一些老的改进机制进行补充 ， 如果大家觉得本文帮助到你了，订阅本专栏，关注后续更多的更新~

​​