Once a while, I would like to build my own kernel with bbrplus congestion control builtin.
For normal bbr control, an add-on module is enough, but bbrplus is different, it requires the tcp stack to be modified, so a customized kernel is necessary.

Here are all the steps I walked through together with all the troubles I encountered and how I solved them.

Steps

  1. Find a kernel archive from https://www.kernel.org/, download it and extract it.
  2. Apply patches or modifications that suit your need. In my case, I overwrote 3 files: /include/net/inet_connection_sock, /net/ipv4/tcp_bbr.c and /net/ipv4/tcp_output.c.

  3. Install required component for building:

    apt-get install kernel-package libncursesw5-dev libncurses5-dev fakeroot wget bzip2 build-essential udev libelf-dev
  4. make menuconfig to config the kernel configuration.

  5. Build kernel .deb file with commandline

    INSTALL_MOD_STRIP=1 CONCURRENCY_LEVEL=2 fakeroot make-kpkg --initrd --revision=1.0.custom kernel_image kernel_headers

    where the version number 1.0 and the version name custom can be anything you like that comply with the version rule.
    for information about CONCURRENCY_LEVEL see [5]

  6. Install the kernel with dpkg -i linux-image-x.xx.deb and dpkg -i linux-headers-x.xx.deb

And here comes the troubleshoting time.

Troubles and Solutions

  • Q: kernel-package not found.
  • A: add stretch-backport branch to your apt-get repo.[1]
  • Q: How to compile bbr to the kernel?

  • A: There are several options that need to be enabled:

    • In Networking support > Networking options > Qos and/or fair queueing make sure Fair Queue Controlled Delay AQM and Fair Queue are selected.
    • In Networking support > Networking options > TCP: advanced congestion control make sure BBR is selected and compiled as builtin instead of the default <M>. If you compiled it as module, it won't be loaded on startup by default.
    • Set default TCP congestion method to BBR is optional and doesn't make much sense since BBR has to be used together with fq which is, by default, disabled.
  • Q: iptables nat stopped working after installing the new kernel.

  • A: Make sure you enabled the following components: [2]

    Linux Kernel Configuration
  -> Networking support
      -> Networking options
          -> Network packet filtering framework(netfilter)
              -> Core netfilter configuration
                  -> Netfilter connection tracking support
                  -> Netbios name service protocal support(new)
                  -> Netfilter Xtables support (required for ip_tables)
    Linux Kernel Configuration
  -> Networking support
      -> Networking options
          -> Network packet filtering framework(netfilter)
              -> IP: Netfilter Configuration
                  -> IPv4 connection tracking support (require for NAT)
                  -> IP tables support (required for filtering/masq/NAT)
              -> Full NAT
                  -> MASQUERADE target support
                  -> REDIRECT target support
    Networking —->
Networking options —->
[*] Network packet filtering (replaces ipchains)  —>
  Core Netfilter Configuration  —>
    <*> Netfilter Xtables support (required for ip_tables)
  IP: Netfilter Configuration —>
    <*> Connection tracking (required for masq/NAT)
    <*> IP tables support (required for filtering/masq/NAT)
    <*>   IP range match support
    <*>   Packet filtering
    <*>     REJECT target support
    <*>   Full NAT

    some of the options are version dependent, they may not available on all the versions.
    Check the final configuration file .config and make sure the following components were all set to M or y: [3]

    CONFIG_PACKET
CONFIG_NETFILTER
CONFIG_IP_NF_CONNTRACK
CONFIG_IP_NF_FTP
CONFIG_IP_NF_IRC
CONFIG_IP_NF_IPTABLES
CONFIG_IP_NF_FILTER
CONFIG_IP_NF_NAT
CONFIG_IP_NF_MATCH_STATE
CONFIG_IP_NF_TARGET_LOG
CONFIG_IP_NF_MATCH_LIMIT
CONFIG_IP_NF_TARGET_MASQUERADE
  • Q: My linux wasn't boot after installing the kernel and paniced with
Can not create /sys/kernel/uevent_helper Permission denied

  • A: uevent_helper is needed. Add: [4]

    CONFIG_UEVENT_HELPER=y
CONFIG_UEVENT_HELPER_PATH="/sys/kernel/uevent_helper"

    to your .config file and recompile the kernel.

  • Q: Anything else?
  • A: Uncheck Kernel hacking > Compile-time checks and compiler options to reduce the final .deb file size.
    Search for CONFIG_HW_RANDOM_TIMERIOMEM and change them from <M> to <*> to enable hardware random generator to improve entropy gathering.

And good luck compiling.

Refer:
[1] https://packages.debian.org/search?searchon=names&keywords=kernel-package
[2] https://blog.csdn.net/yandaqijian/article/details/44342027
[3] https://www.linuxtopia.org/Linux_Firewall_iptables/x651.html
[4] https://cateee.net/lkddb/web-lkddb/UEVENT_HELPER_PATH.html
[5] https://stackoverflow.com/questions/21001588/debian-make-kpkg-and-linux-kernel-development
[*] https://www.cyberciti.biz/faq/linux-install-ncurses-library-headers-on-debian-ubuntu-centos-fedora/