install.sh: improve Generic code path + Gentoo + tmpfs install (#304)

Signed-off-by: Adel KARA SLIMANE <adel.ks@zegrapher.com>

README.md

@@ -49,7 +49,7 @@ To apply your own patch files using the provided scripts, you will need to put t
 #### Anbox usage
 
 When enabling the anbox support option, the `binder` and `ashmem` modules are built-in. You don't have to load them. However you'll need to mount binderfs :
-```
+```shell
 sudo mkdir /dev/binderfs
 sudo mount -t binder binder /dev/binderfs
 ```
@@ -64,12 +64,12 @@ binder                         /dev/binderfs binder   nofail  0      0
 ```
 
 Then, if needed, start the anbox service :
-```
+```shell
 systemctl start anbox-container-manager.service
 ```
 
 You can also enable the service for it to be auto-started on boot :
-```
+```shell
 systemctl enable anbox-container-manager.service
 ```
 
@@ -80,7 +80,7 @@ If you prefer automatic setup you can install `anbox-support` from AUR which wil
 ### Install procedure
 
 #### Arch & derivatives
-```
+```shell
 git clone https://github.com/Frogging-Family/linux-tkg.git
 cd linux-tkg
 # Optional: edit the "customization.cfg" file
@@ -89,24 +89,24 @@ makepkg -si
 The script will use a slightly modified Arch config from the `linux-tkg-config` folder. The options selected at build-time are installed to `/usr/share/doc/$pkgbase/customization.cfg`, where `$pkgbase` is the package name.
 
 
-
 #### DEB (Debian, Ubuntu and derivatives) and RPM (Fedora, SUSE and derivatives) based distributions
-```
+The interactive `install.sh` script will create, depending on the selected distro, `.deb` or `.rpm` packages, move them in the the subfolder `DEBS` or `RPMS` then prompts to install them with the distro's package manager.
+```shell
 git clone https://github.com/Frogging-Family/linux-tkg.git
 cd linux-tkg
 # Optional: edit the "customization.cfg" file
 ./install.sh install
 ```
 Uninstalling custom kernels installed through the script has to be done
-manually. The script can can help out with some useful information:
+manually. `install.sh` can can help out with some useful information:
-```
+```shell
 cd path/to/linux-tkg
 ./install.sh uninstall-help
 ```
-The script will use your current kernel's `.config` file, which will be searched for either at ``/boot/config-`uname -r`.config`` or ``/proc/config.gz`` (otherwise the script won't work as-is). It's recommended to run the script booted on your distro-provided kernel.
+The script will use your current kernel's `.config` file, which will be searched for either at ``/boot/config-`uname -r`.config`` or ``/proc/config.gz`` otherwise it defaults the the "vanilla" upstream `.config` file. It's recommended to run the script booted on your distro-provided kernel.
 
 #### Void Linux
-```
+```shell
 git clone -b tkg https://github.com/Hyper-KVM/void-packages/
 cd void-packages
 ./xbps-src binary-bootstrap
@@ -115,22 +115,34 @@ cd void-packages
 ./xbps-src pkg -j$(nproc) linux-tkg
 ```
 If you have to restart the build for any reason, run `./xbps-src clean linux-tkg` first.
-
+#### Generic install
-#### Other linux distributions
+The interactive `install.sh` script can be used to perform a "Generic" install by choosing `Generic` when prompted. It git clones the kernel tree in the `linux-src-git` folder, patches the code and edits a `.config` file in it. The commands to do are the following:
-If your distro is neither DEB nor RPM based, `install.sh` script can clone the kernel tree in the `linux-src-git` folder, patch and edit a `.config` file based on your current kernel's. It's expected either at ``/boot/config-`uname -r`.config`` or ``/proc/config.gz`` (otherwise it won't work as-is).
-
-To do so, run:
-```
-# Optional: edit the "customization.cfg" file
-./install.sh config
-```
-
-When selecting `Generic` as distro, `./install.sh install` will compile the kernel then prompt before doing the following:
 ```shell
+git clone https://github.com/Frogging-Family/linux-tkg.git
+cd linux-tkg
+# Optional: edit the "customization.cfg" file
+./install.sh install
+```
+The script will compile the kernel then prompt before doing the following:
+```shell
+sudo cp -R . /usr/src/linux-tkg-${kernel_flavor}
+cd /usr/src/linux-tkg-${kernel_flavor}
 sudo make modules_install
-sudo make headers_install INSTALL_HDR_PATH=/usr # CAUTION: this will replace files in /usr/include
 sudo make install
-sudo dracut --force --hostonly --kver $_kernelname
+sudo dracut --force --hostonly --kver $_kernelname $_dracut_options
 sudo grub-mkconfig -o /boot/grub/grub.cfg
 ```
-**Note:** these changes will not be tracked by your package manager and uninstalling requires manual intervention. `./install.sh uninstall-help` can help with useful information if your install procedure follows the `Generic` approach.
+**Notes:**
+- If you only want the script to patch the sources in `linux-src-git`, you can use `./install.sh config`
+- `${kernel_flavor}` is a default naming scheme but can be customized with the variable `_kernel_localversion` in `customization.cfg`.
+- `_dracut_options` is a variable that can be changed in `customization.cfg`.
+- The script uses a base `.config` file. Its path can be provided through `_configfile` in `customization.cfg`. Otherwise ``/boot/config-`uname -r`.config``, ``/proc/config.gz`` or a "vanilla" upstream `.config` file are used, in that order, by default.
+- The installed files will not be tracked by your package manager and uninstalling requires manual intervention. `./install.sh uninstall-help` can help with useful information if your install procedure follows the `Generic` approach.
+#### Gentoo
+The interactive `install.sh` script supports Gentoo by following the same procedure as `Generic`. Then it offers to conveniently do an `emerge @module-rebuild` after symlinking the sources folder in `/usr/src/` to `/usr/src/linux`.
+```shell
+git clone https://github.com/Frogging-Family/linux-tkg.git
+cd linux-tkg
+# Optional: edit the "customization.cfg" file
+./install.sh install
+```

customization.cfg

@@ -1,6 +1,6 @@
 # linux-TkG config file
 
-# Linux distribution you are using, options are "Arch", "Void", "Ubuntu", "Debian", "Fedora", "Suse", "Generic".
+# Linux distribution you are using, options are "Arch", "Void", "Ubuntu", "Debian", "Fedora", "Suse", "Gentoo", "Generic".
 # It is automatically set to "Arch" when using PKGBUILD.
 # If left empty, the script will prompt
 _distro=""
@@ -58,6 +58,16 @@ _diffconfig=""
 # Set to the file name where the generated config fragment should be written to. Only used if _diffconfig is active.
 _diffconfig_name=""
 
+# [install.sh specific] Use tmpfs as a work directory, recommended when RAM >= 32GB to reduce HDD/SSD usage. For more information, see https://wiki.archlinux.org/title/Tmpfs
+_use_tmpfs="false"
+
+# [install.sh specific] tmpfs folder path, only used when _use_tmpfs="true".
+#                       Creates a linux-tkg work folder within that pathmake sure to have nothing important in "$_tmpfs_path/linux-tkg"
+_tmpfs_path="/tmp"
+
+# [install.sh: Generic and Gentoo specific] Dracut options when generating initramfs
+_dracut_options="--lz4"
+
 #### KERNEL OPTIONS ####
 
 # Name of the default config file to use for the kernel

install.sh

@@ -29,7 +29,7 @@ plain() {
 _distro_prompt() {
   echo "Which linux distribution are you running ?"
   echo "if it's not on the list, chose the closest one to it: Fedora/Suse for RPM, Ubuntu/Debian for DEB"
-  _prompt_from_array "Debian" "Fedora" "Suse" "Ubuntu" "Generic"
+  _prompt_from_array "Debian" "Fedora" "Suse" "Ubuntu" "Gentoo" "Generic"
   _distro="${_selected_value}"
 }
 
@@ -39,7 +39,7 @@ _install_dependencies() {
   fi
   if [ "$_distro" = "Ubuntu" ]; then
     msg2 "Installing dependencies"
-    sudo apt install git build-essential fakeroot libncurses5-dev libssl-dev ccache bison flex qtbase5-dev ${clang_deps} -y
+    sudo apt install git build-essential fakeroot libncurses5-dev libssl-dev ccache bison flex qtbase5-dev libelf-dev ${clang_deps} -y
   elif [ "$_distro" = "Debian" ]; then
     msg2 "Installing dependencies"
     sudo apt install git wget build-essential fakeroot libncurses5-dev libssl-dev ccache bison flex qtbase5-dev bc rsync kmod cpio libelf-dev ${clang_deps} -y
@@ -91,14 +91,14 @@ _linux_git_branch_checkout() {
     cd linux-src-git
 
     # Remove "origin" remote if present
-    if git remote -v | grep "origin" ; then
+    if git remote -v | grep -w "origin" ; then
       git remote rm origin
     fi
 
-    if ! git remote -v | grep "kernel.org" ; then
+    if ! git remote -v | grep -w "kernel.org" ; then
       git remote add kernel.org https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
     fi
-    if ! git remote -v | grep "googlesource.com" ; then
+    if ! git remote -v | grep -w "googlesource.com" ; then
       git remote add googlesource.com https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux-stable
     fi
 
@@ -106,14 +106,8 @@ _linux_git_branch_checkout() {
     msg2 "Reseting files to their original state"
 
     git reset --hard HEAD
-
-    if [ "${_distro}" = "Generic" ]; then
-      msg2 "Please enter your sudo password so the script can clean root owned files from the kernel sources"
-      sudo git clean -fdx
-    else
     git clean -f -d -x
   fi
-  fi
 
   if [[ "$_sub" = rc* ]]; then
     msg2 "Switching to master branch for RC Kernel"
@@ -141,7 +135,7 @@ _linux_git_branch_checkout() {
     fi
 
     msg2 "Switching to linux-${_basekernel}.y"
-    if ! git branch --list | grep "linux-${_basekernel}-${_git_mirror}" ; then
+    if ! git branch --list | grep -w "linux-${_basekernel}-${_git_mirror}" ; then
       msg2 "${_basekernel}.y branch doesn't locally exist, shallow cloning..."
       git remote set-branches --add kernel.org linux-${_basekernel}.y
       git remote set-branches --add googlesource.com linux-${_basekernel}.y
@@ -157,7 +151,6 @@ _linux_git_branch_checkout() {
     msg2 "Checking out latest release: ${_kernel_tag}"
     git checkout "${_kernel_tag}"
   fi
-
 }
 
 if [ "$1" != "install" ] && [ "$1" != "config" ] && [ "$1" != "uninstall-help" ]; then
@@ -191,18 +184,17 @@ if [ "$1" = "install" ] || [ "$1" = "config" ]; then
   # Run init script that is also run in PKGBUILD, it will define some env vars that we will use
   _tkg_initscript
 
-  if [[ "${_compiler}" = "llvm" && "${_distro}" = "Generic" ]]; then
+  if [[ "${_compiler}" = "llvm" && "${_distro}" =~ ^(Generic|Gentoo)$ ]]; then
-    read -p "Replace \"libunwind\" with \"llvm-libunwind\" ? y/[n]:" _libunwind_replace
+    read -p "Replace \"libunwind\" with \"llvm-libunwind\" ? Y/[n]:" _libunwind_replace
     if [[ "${_libunwind_replace}" =~ ^(y|yes|Yes|Y)$ ]]; then
       export LDFLAGS_MODULE="-unwindlib=libunwind"
       export HOSTLDFLAGS="-unwindlib=libunwind"
     fi
   fi
 
-  if [[ $1 = "install" && ! "$_distro" =~ ^(Ubuntu|Debian|Fedora|Suse|Generic)$ ]]; then
+  if [[ $1 = "install" && ! "$_distro" =~ ^(Ubuntu|Debian|Fedora|Suse|Gentoo|Generic)$ ]]; then
-    msg2 "Variable \"_distro\" in \"customization.cfg\" hasn't been set to \"Ubuntu\", \"Debian\",  \"Fedora\" or \"Suse\""
+    msg2 "Variable \"_distro\" in \"customization.cfg\" has been set to an unkown value. Prompting..."
-    msg2 "This script can only install custom kernels for RPM and DEB based distros, though only those keywords are permitted. Exiting..."
+    _distro_prompt
-    exit 0
   fi
 
   # Install the needed dependencies if the user wants to install the kernel
@@ -261,18 +253,25 @@ if [ "$1" = "install" ] || [ "$1" = "config" ]; then
     fi
   fi
 
-  # Follow Ubuntu install isntructions in https://wiki.ubuntu.com/KernelTeam/GitKernelBuild
+  # cd into the linux-src folder is important before calling _tkg_srcprep
-
+  cd "$_where/linux-src-git"
-  # cd in linux folder, copy Ubuntu's current config file, update with new params
-  cd "$_where"/linux-src-git
-
-
-  yes '' | make ${llvm_opt} oldconfig
-  msg2 "Done"
-
-  # apply linux-tkg patching script
   _tkg_srcprep
 
+  _build_dir="$_where"
+  if [ "$_use_tmpfs" = "true" ]; then
+    if [ -d "$_tmpfs_path/linux-tkg" ]; then
+      msg2 "Nuking linux-tkg tmpfs folder $_tmpfs_path/linux-tkg"
+      rm -rf "$_tmpfs_path/linux-tkg"
+    fi
+    mkdir "$_tmpfs_path/linux-tkg"
+    cp -r "$_where/linux-src-git" "$_tmpfs_path/linux-tkg/linux-src-git"
+
+    # cd into the linux-src folder is important before calling _tkg_srcprep
+    _build_dir="$_tmpfs_path/linux-tkg"
+    cd "$_tmpfs_path/linux-tkg/linux-src-git"
+  fi
+
+
   # Uppercase characters are not allowed in source package name for debian based distros
   if [ "$_distro" = "Debian" ] || [ "$_distro" = "Ubuntu" ] && [ "$_cpusched" = "MuQSS" ]; then
     _cpusched="muqss"
@@ -331,20 +330,24 @@ if [ "$1" = "install" ]; then
     _kernel_subver="${_sub}"
   fi
 
+  _timed_build() {
+    _runtime=$( time ( schedtool -B -n 1 -e ionice -n 1 "$@" 2>&1 ) 3>&1 1>&2 2>&3 ) || _runtime=$( time ( "$@" 2>&1 ) 3>&1 1>&2 2>&3 )
+  }
+
   if [ "$_distro" = "Ubuntu" ]  || [ "$_distro" = "Debian" ]; then
 
-    if make ${llvm_opt} -j ${_thread_num} deb-pkg LOCALVERSION=-${_kernel_flavor}; then
+    msg2 "Building kernel DEB packages"
+    _timed_build make ${llvm_opt} -j ${_thread_num} deb-pkg LOCALVERSION=-${_kernel_flavor}
     msg2 "Building successfully finished!"
 
-      cd "$_where"
-
     # Create DEBS folder if it doesn't exist
+    cd "$_where"
     mkdir -p DEBS
 
-      # Move rpm files to RPMS folder inside the linux-tkg folder
+    # Move deb files to DEBS folder inside the linux-tkg folder
-      mv "$_where"/*.deb "$_where"/DEBS/
+    mv "$_build_dir"/*.deb "$_where"/DEBS/
 
-      read -p "Do you want to install the new Kernel ? y/[n]: " _install
+    read -p "Do you want to install the new Kernel ? Y/[n]: " _install
     if [[ $_install =~ [yY] ]] || [ $_install = "yes" ] || [ $_install = "Yes" ]; then
       cd "$_where"
       if [[ "$_sub" = rc* ]]; then
@@ -359,7 +362,6 @@ if [ "$1" = "install" ]; then
       cd DEBS
       sudo dpkg -i $_headers_deb $_image_deb $_kernel_devel_deb
     fi
-    fi
 
   elif [[ "$_distro" = "Fedora" ||  "$_distro" = "Suse" ]]; then
 
@@ -373,19 +375,24 @@ if [ "$1" = "install" ]; then
       _extra_ver_str="_${_kernel_flavor}"
     fi
 
-    if RPMOPTS="--define '_topdir ${HOME}/.cache/linux-tkg-rpmbuild'" make ${llvm_opt} -j ${_thread_num} rpm-pkg EXTRAVERSION="${_extra_ver_str}"; then
+    _fedora_work_dir="${HOME}/.cache/linux-tkg-rpmbuild"
+    if [ "$_use_tmpfs" = "true" ]; then
+      _fedora_work_dir="$_tmpfs_path/linux-tkg/linux-tkg-rpmbuild"
+    fi
+
+    msg2 "Building kernel RPM packages"
+    RPMOPTS="--define '_topdir ${_fedora_work_dir}'" _timed_build make ${llvm_opt} -j ${_thread_num} rpm-pkg EXTRAVERSION="${_extra_ver_str}"
     msg2 "Building successfully finished!"
 
-      cd "$_where"
-
     # Create RPMS folder if it doesn't exist
+    cd "$_where"
     mkdir -p RPMS
 
     # Move rpm files to RPMS folder inside the linux-tkg folder
-      mv ${HOME}/.cache/linux-tkg-rpmbuild/RPMS/x86_64/*tkg* "$_where"/RPMS/
+    mv ${_fedora_work_dir}/RPMS/x86_64/*tkg* "$_where"/RPMS/
 
-      read -p "Do you want to install the new Kernel ? y/[n]: " _install
+    read -p "Do you want to install the new Kernel ? Y/[n]: " _install
-      if [ "$_install" = "y" ] || [ "$_install" = "Y" ] || [ "$_install" = "yes" ] || [ "$_install" = "Yes" ]; then
+    if [[ "$_install" =~ ^(Y|y|Yes|yes)$ ]]; then
 
       if [[ "$_sub" = rc* ]]; then
         _kernelname=$_basekernel.${_kernel_subver}_${_sub}_$_kernel_flavor
@@ -408,45 +415,115 @@ if [ "$1" = "install" ]; then
 
       msg2 "Install successful"
     fi
-    fi
 
-  elif [ "$_distro" = "Generic" ]; then
+  elif [[ "$_distro" =~ ^(Gentoo|Generic)$ ]]; then
 
     ./scripts/config --set-str LOCALVERSION "-${_kernel_flavor}"
 
-    if make ${llvm_opt} -j ${_thread_num}; then
-
     if [[ "$_sub" = rc* ]]; then
       _kernelname=$_basekernel.${_kernel_subver}-${_sub}-$_kernel_flavor
     else
       _kernelname=$_basekernel.${_kernel_subver}-$_kernel_flavor
     fi
 
-      msg2 "Building successful"
+    msg2 "Building kernel"
+    _timed_build make ${llvm_opt} -j ${_thread_num}
+    msg2 "Build successful"
+
+    if [ "$_STRIP" = "true" ]; then
+      echo "Stripping vmlinux..."
+      strip -v $STRIP_STATIC "vmlinux"
+    fi
+
+    _headers_folder_name="linux-$_kernelname"
+
+    msg2 "Removing unneeded architectures..."
+    for arch in arch/*/; do
+      [[ $arch = */x86/ ]] && continue
+      echo "Removing $(basename "$arch")"
+      rm -r "$arch"
+    done
+
+    msg2 "Removing broken symlinks..."
+    find -L . -type l -printf 'Removing %P\n' -delete
+
+    msg2 "Removing loose objects..."
+    find . -type f -name '*.o' -printf 'Removing %P\n' -delete
+
+    msg2 "Stripping build tools..."
+    while read -rd '' file; do
+      case "$(file -bi "$file")" in
+        application/x-sharedlib\;*)      # Libraries (.so)
+          strip -v $STRIP_SHARED "$file" ;;
+        application/x-archive\;*)        # Libraries (.a)
+          strip -v $STRIP_STATIC "$file" ;;
+        application/x-executable\;*)     # Binaries
+          strip -v $STRIP_BINARIES "$file" ;;
+        application/x-pie-executable\;*) # Relocatable binaries
+          strip -v $STRIP_SHARED "$file" ;;
+      esac
+    done < <(find . -type f -perm -u+x ! -name vmlinux -print0)
+
+    echo -e "\n\n"
+
     msg2 "The installation process will run the following commands:"
+    echo "    # copy the patched and compiled sources to /usr/src/$_headers_folder_name"
     echo "    sudo make modules_install"
-      echo "    sudo make headers_install INSTALL_HDR_PATH=/usr # CAUTION: this will replace files in /usr/include"
     echo "    sudo make install"
-      echo "    sudo dracut --hostonly --kver $_kernelname"
+    echo "    sudo dracut --force --hostonly ${_dracut_options} --kver $_kernelname"
     echo "    sudo grub-mkconfig -o /boot/grub/grub.cfg"
+
     msg2 "Note: Uninstalling requires manual intervention, use './install.sh uninstall-help' for more information."
-      read -p "Continue ? [Y/n]: " _continue
+    read -p "Continue ? Y/[n]: " _continue
 
     if ! [[ $_continue =~ ^(Y|y|Yes|yes)$ ]];then
       exit 0
     fi
 
+    msg2 "Copying files over to /usr/src/$_headers_folder_name"
+    if [ -d "/usr/src/$_headers_folder_name" ]; then
+      msg2 "Removing old folder in /usr/src/$_headers_folder_name"
+      sudo rm -rf "/usr/src/$_headers_folder_name"
+    fi
+    sudo cp -R . "/usr/src/$_headers_folder_name"
+    sudo rm -rf "/usr/src/$_headers_folder_name/.git"
+    cd "/usr/src/$_headers_folder_name"
+
     msg2 "Installing modules"
+    if [ "$_STRIP" = "true" ]; then
+      sudo make modules_install INSTALL_MOD_STRIP="1"
+    else
       sudo make modules_install
-      msg2 "Installing headers"
+    fi
-      sudo make headers_install INSTALL_HDR_PATH=/usr
+    msg2 "Removing modules from source folder in /usr/src/${_kernel_src_gentoo}"
+    sudo find . -type f -name '*.ko' -delete
+    sudo find . -type f -name '*.ko.cmd' -delete
+
     msg2 "Installing kernel"
     sudo make install
     msg2 "Creating initramfs"
-      sudo dracut --force --hostonly --kver $_kernelname
+    sudo dracut --force --hostonly ${_dracut_options} --kver $_kernelname
     msg2 "Updating GRUB"
     sudo grub-mkconfig -o /boot/grub/grub.cfg
+
+    if [ "$_distro" = "Gentoo" ]; then
+
+      msg2 "Selecting the kernel source code as default source folder"
+      sudo ln -sfn "/usr/src/$_headers_folder_name" "/usr/src/linux"
+
+      msg2 "Rebuild kernel modules with \"emerge @module-rebuild\" ?"
+      if [ "$_compiler" = "llvm" ];then
+        warning "Building modules with LLVM/Clang is mostly unsupported OOTB by \"emerge @module-rebuild\" except for Nvidia 465.31+"
+        warning "     Manually setting \"CC=clang\" for some modules may work if you haven't used LTO"
       fi
+
+      read -p "Y/[n]: " _continue
+      if [[ $_continue =~ ^(Y|y|Yes|yes)$ ]];then
+        sudo emerge @module-rebuild --keep-going
+      fi
+
+    fi
+
   fi
 fi
 
@@ -480,7 +557,7 @@ if [ "$1" = "uninstall-help" ]; then
     msg2 "      sudo zypper remove --no-clean-deps kernel-VERSION kernel-devel-VERSION kernel-headers-VERSION"
     msg2 "       where VERSION is displayed in the second to last column"
     msg2 "Note: kernel-headers packages are no longer created and installed, you can safely remove any remnants."
-  elif [ "$_distro" = "Generic" ]; then
+  elif [[ "$_distro" =~ ^(Generic|Gentoo)$ ]]; then
     msg2 "Folders in /lib/modules :"
     ls /lib/modules
     msg2 "Files in /boot :"

linux-tkg-config/prepare

@@ -1385,6 +1385,9 @@ exit_cleanup() {
 
   # Remove RPM temporary files left
   rm -rf ${HOME}/.cache/linux-tkg-rpmbuild
+  if [ "$_use_tmpfs" = "true" ]; then
+    rm -rf "$_tmpfs_path/linux-tkg"
+  fi
 
   # Community patches removal in case of failure
   for _p in ${_community_patches[@]}; do