diff --git a/README.md b/README.md new file mode 100644 index 0000000..012291c --- /dev/null +++ b/README.md @@ -0,0 +1,98 @@ +# UDIB + +UDIB is the Unattended Debian Installation Builder. +It provides a handy commandline utility for creating preseeded Debian installation ISOs. +Preseeded ISOs allow partially or fully automated Debian installations on bare metal (or anywhere else). + +## What's preseeding? + +A preseed file is a text file which provides the Debian installer with previously set (preseeded) answers during the installation process. +Preseeding a Debian ISO allows you to heavily reduce the amount of user interaction required during an installation, or even fully automate it. +The preseed file is written by you and then injected into the installation image. +When you start the installation, any answers you have provided to the debian installer as part of your preseed file are automatically applied during the installation. +If you want to know more, you can have a look at Debian's [official guide](https://www.debian.org/releases/stable/amd64/apb.en.html) or at the Debian wiki's [quick overview](https://wiki.debian.org/DebianInstaller/Preseed), both of which explain preseeding much better than I can. + +## How does UDIB work? + +UDIB's main purpose is the injection of preseed files into existing Debian installation ISOs. +In a nutshell, it does this by extracting the ISO, adding the preseed file to it, and repacking the ISO again. +You could do all of this manually of course by following the [basic](https://wiki.debian.org/DebianInstaller/Preseed/EditIso#Adding_a_Preseed_File_to_the_Initrd) and [advanced](https://wiki.debian.org/RepackBootableISO) guides for ISO repacking on the Debian wiki, but UDIB does all of this for you. + +# Dependencies + +UDIB has some hard- and software requirements, both regarding the [target machine](#target-machine) for which it can build ISOs, as well as on the [build machine](#build-machine) on which UDIB is run. + +## Target Machine + +Images built using UDIB can be used to install Debian on systems satisfying these requirements: + +- **CPU Architecture:** x86 (64-Bit) +- **Network:** an ethernet connection with internet access + +## Build Machine + +Using UDIB to create ISOs requires the following software: + +- GNU/Linux +- `python3` *(3.10.4 known to work)* + - [required python packages](./requirements.txt) can be installed in a virtual environment +- `xorriso` *(1.5.4 known to work)* + - **Debian (bullseye):** [xorriso](https://packages.debian.org/bullseye/xorriso) + - **Arch Linux:** [extra/libisoburn](extra/libisoburn) +- `gpg` *(2.2.32 known to work)* +- GNU `cpio` + - preinstalled on most distributions + +Internet access is (obviously) required if you want to fetch any files using UDIB. + +# Quick Start Guide + +This short guide explains how to build a Debian ISO with a customized and automated installation: + +1. make sure you have all the [required software](#build-machine) installed +2. clone this repo and `cd` into your local copy +3. (optional) create and activate a virtual environment: `python3 -m venv .venv && . .venv/bin/activate` +4. install the required python packages: `pip install --user -r requirements.txt` +5. get an example preseed file: `./udib.py -o my-preseed.cfg get preseed-file-basic` +6. customize your installation by editing `my-preseed.cfg` (the comments are pretty self-explanatory) +7. create a Debian ISO with your preseed file: `./udib.py -o my-image.iso inject my-preseed.cfg` +8. boot from your newly created ISO `my-image.iso` on your target machine (or in a VM) +9. in the Debian installer menu, navigate to *Advanced options > Automated install* +10. drink some coffee +11. return to your new Debian system + +Depending on how many answers you provided in the preseed file, the installation may require some manual interaction. +Preseed files are very powerful, and if you need more customization you can have a deeper look into [how they work](#whats-preseeding). +You can also use UDIB to get the full preseed example file: `./udib.py get preseed-file-full` and use that as a starting point. + +# Detailed usage guide + +You can view help at the commandline using `./udib.py --help` for general options and `./udib.py COMMAND --help` for help with a specific subcommand. + +The name and destination of files produced by `udib.py` can be specified with the `--output-file` option. +Alternatively, you can use the `--output-dir` option to specify the directory where produced files will be saved, without having to name them explicitly (default names will be used). + +## Retrieving example preseed files or vanilla ISOs + +As a starting point for creating your own preseeded ISO, you can retrieve one of Debian's example preseed files or an unmodified Debian ISO using UDIB: + +``` +udib.py [--output-file FILE | --output-dir DIR] get WHAT +``` + +`WHAT` must be a specific string and can be either one of: + +- `preseed-file-basic` to download Debian's basic example preseed file (sufficient in most cases) +- `preseed-file-full` to download Debian's full example preseed file (has a LOT of customization options) +- `iso` to download the latest, unmodified Debian stable x86-64 netinst ISO + +## Creating a preseeded ISO + +To inject an existing preseed file into an ISO, you can run the following command: + +``` +udib.py [--output-file FILE | --output-dir DIR] inject PRESEEDFILE [--image-file IMAGEFILE] +``` + +where `PRESEEDFILE` is the path to your preseed file. +If you don't specify an `--image-file`, UDIB will download the latest Debian x86-64 netinst ISO and inject your `PRESEEDFILE` into it. diff --git a/README.org b/README.org deleted file mode 100644 index 4d569ce..0000000 --- a/README.org +++ /dev/null @@ -1,60 +0,0 @@ -* UDIB - -=UDIB= is the /unattended debian installation (image) builder/. -As the name would suggest, it is a tool which builds a debian installation image which, if used to boot from, installs the *latest stable release* of Debian Linux to the machine without any user interaction required whatsoever. -Configuration of the installed system is done by editing preseed files. - -** Usage - -Obtaining a preseed example file: - -#+begin_src shell - ./udib.py get preseed-file -#+end_src - -** Prerequisites - -*** Target machine - -Any target machine on which you wish to install Debian using images built using =UDIB= must fulfil the following requirements: - -- *Architecture:* An x64-amd CPU -- *Network access:* an ethernet connection with internet access (/wifi does not work/) - -*** Build machine - -=UDIB= has the following dependencies, installed on and in the ~$PATH~ of the machine on which you build your image: - -- =python3= /(3.10.4 known to work)/ - -- =xorriso= /(1.5.4 known to work)/ - - part of [[https://packages.debian.org/bullseye/xorriso][xorriso]] on Debian bullseye - - part of [[https://www.archlinux.org/packages/extra/x86_64/libisoburn/][extra/libisoburn]] on Arch Linux - -- =gpg= /(2.2.32 known to work)/ - -- GNU =cpio= - - preinstalled on most distributions - -Internet access is required so that =UDIB= can download the Debian installation image. - -** How it works - -=UDIB= does the following: - -1. download the latest debian stable installation ISO -2. verify integrity and authenticity of the obtained ISO -3. extract the ISO -4. inject the preseed configuration into the ISO's initrd -5. repack the modified ISO - -*** Preseeding - -The Debian project provides a well-written and detailed [[https://www.debian.org/releases/stable/amd64/apb.en.html][official guide]] on how preseeding works, and how to implement it. The Debian wiki offers a [[https://wiki.debian.org/DebianInstaller/Preseed][quick overview]] of preseeding as well. - -Example preseeding configurations are provided as well. The [[https://www.debian.org/releases/stable/example-preseed.txt][basic example]] is what =UDIB= is based on, but much more detailed preseeding is also possible, as seen in the [[https://preseed.debian.net/debian-preseed/bullseye/amd64-main-full.txt][full example]]. - -*** ISO repacking - -The Debian wiki provides both a [[https://wiki.debian.org/DebianInstaller/Preseed/EditIso#Adding_a_Preseed_File_to_the_Initrd][basic guide]] and a [[https://wiki.debian.org/RepackBootableISO][detailed guide]] on repacking debian installation images with the preseed file (and other files) contained within. - diff --git a/__init__.py b/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/cli/__init__.py b/cli/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/udib/clibella.py b/cli/clibella.py similarity index 88% rename from udib/clibella.py rename to cli/clibella.py index a4c194d..0a56d30 100644 --- a/udib/clibella.py +++ b/cli/clibella.py @@ -1,6 +1,7 @@ """Library for consistent and visually appealing terminal output.""" import sys +from re import compile import colorama from colorama import Fore, Style @@ -335,3 +336,45 @@ class Printer: color_enabled=color_enabled, **kwargs ) + + + def prompt_yes_or_no(self, question: str, ask_until_valid: bool = False): + """Prompts the user for an answer to the specified yes/no question. + + If 'ask_until_valid' is True, keeps repeating the prompt until the user + provides recognizable input to answer the question. + + Parameters + ---------- + question : str + The question to ask the user while prompting. The string is suffixed + with " (Yes/No) ". + ask_until_valid : bool + If this is set to True and the user provides an ambiguous answer, keeps + prompting indefinitely until an unambiguous answer is read. + + Returns + ------- + True : bool + If the user has answered 'yes'. + False : bool + If the user has answered 'no'. + None : None + If 'ask_until_valid' is False and the user has provided an ambiguous + response to the prompt. + """ + + user_input_is_valid = False + + regex_yes = compile(r"^(y)$|^(Y)$|^(YES)$|^(Yes)$|^(yes)$") + regex_no = compile(r"^(n)$|^(N)$|^(NO)$|^(No)$|^(no)$") + + while(not user_input_is_valid): + user_input = self.input(f"{question} (Yes/No): ") + + if (regex_yes.match(user_input)): + return True + elif (regex_no.match(user_input)): + return False + elif (not ask_until_valid): + return None diff --git a/cli/parser.py b/cli/parser.py new file mode 100644 index 0000000..de57e58 --- /dev/null +++ b/cli/parser.py @@ -0,0 +1,81 @@ +"""Commandline argument parsing utilities.""" + +from argparse import ArgumentParser + + +def get_argument_parser(): + """Sets up an argparse ArgumentParser and returns it.""" + + mainparser = ArgumentParser( + description="Debian ISO preseeding tool.", + ) + + # add mutually exclusive optional arguments to top-level parser + mainparser_group = mainparser.add_mutually_exclusive_group() + mainparser_group.add_argument( + "-o", + "--output-file", + action='store', + type=str, + dest='path_to_output_file', + metavar='OUTPUTFILE', + help="File as which the retrieved/generated file will be saved", + ) + mainparser_group.add_argument( + "-O", + "--output-dir", + action='store', + type=str, + dest='path_to_output_dir', + metavar='OUTPUTDIR', + help="Directory into which the retrieved/generated file will be written", + ) + + # register subparsers for the 'get' and 'inject' subcommands + subparsers = mainparser.add_subparsers( + required=True, + title="Subcommands", + description="A choice of actions you want udib to take", + help="You must specify one of these", + dest="subparser_name", + ) + subparser_get = subparsers.add_parser( + "get", + description="Retrieve an unmodified Debian ISO or example preseed file", + ) + subparser_inject = subparsers.add_parser( + "inject", + description="Inject a preseed file into a Debian ISO", + ) + + # register arguments for the 'get' subcommand + subparser_get.add_argument( + "WHAT", + choices=['preseed-file-basic', 'preseed-file-full', 'iso'], + action='store', + type=str, + metavar='WHAT', + help="The type of file you want UDIB to retrieve. " + "Valid options are: 'preseed-file-basic', 'preseed-file-full' " + "or 'iso'.", + ) + + # register arguments for the 'inject' subcommand + subparser_inject.add_argument( + "PRESEEDFILE", + action='store', + type=str, + metavar='PRESEEDFILE', + help="Path to the preseed file you want to inject", + ) + subparser_inject.add_argument( + "-i", + "--image-file", + action='store', + type=str, + dest='path_to_image_file', + metavar='IMAGEFILE', + help="Path to the ISO you want to modify", + ) + + return mainparser diff --git a/core/__init__.py b/core/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/core/utils.py b/core/utils.py new file mode 100644 index 0000000..d1c7a63 --- /dev/null +++ b/core/utils.py @@ -0,0 +1,239 @@ +"""A collection of general utilities, not specific to any module.""" + +from pathlib import Path +from tempfile import TemporaryDirectory +from sys import exit +from os import remove, rename +from subprocess import run, STDOUT, PIPE + +from cli.clibella import Printer +from net.download import download_file +from net.scrape import get_debian_iso_urls +from gpg.verify import assert_detached_signature_is_valid +from gpg.exceptions import VerificationFailedError +from gpg.keystore import debian_signing_key_is_imported, import_debian_signing_key +from crypt import crypt, METHOD_SHA512 +from getpass import getpass + + +def hash_user_password(printer=None): + """Prompts for a password and prints the resulting hash. + + The resulting hash can be added to the preseed file to set a user password: + d-i passwd/root-password-crypted PASSWORDHASH + """ + + if printer is None: + p = Printer() + else: + if not isinstance(printer, Printer): + raise TypeError(f"Expected a {type(Printer)} object.") + + password = getpass("Enter a password: ") + password_confirmed = getpass("Enter the password again: ") + if not password_confirmed == password: + p.failure("Passwords did not match") + return + + p.info("Password hash:") + p.info(crypt(password, crypt.METHOD_SHA512)) + +def find_all_files_under(parent_dir): + """Recursively finds all files anywhere under the specified directory. + + Returns a list of absolute Path objects. Symlinks are ignored. + + Parameters + ---------- + parent_dir : str or pathlike object + The directory under which to recursively find files. + + Raises + ------ + NotADirectoryError + Raised if the specified parent directory is not a directory. + + Examples + -------- + config_files = find_all_files_under("~/.config") + + """ + + if "~" in str(parent_dir): + parent_dir = Path(parent_dir).expanduser() + parent_dir = Path(parent_dir).resolve() + + if not parent_dir.is_dir(): + raise NotADirectoryError(f"No such directory: '{parent_dir}'.") + + files = [] + + for subpath in parent_dir.iterdir(): + if subpath.is_file(): + files.append(subpath.resolve()) + elif not subpath.is_symlink() and subpath.is_dir(): + files += find_all_files_under(subpath) + + return files + + +def trim_text_file(path_to_input_file, substring): + """Removes all lines not containing the substring from the input file. + + The input file is overwritten. + If the substring is the empty string, no action is taken. + If the substring contains a newline, no action is taken. + If the substring does not match any line of the file, the resulting file + will be left empty. + + Parameters + ---------- + path_to_input_file : str or pathlike object + The file to trim. + substring : str + The string to filter for. + """ + + if not isinstance(substring, str): + raise TypeError("Expected a string.") + if "\n" in substring or len(substring) == 0: + return + + if "~" in str(path_to_input_file): + path_to_input_file = Path(path_to_input_file).expanduser() + path_to_input_file = Path(path_to_input_file).resolve() + + if not path_to_input_file.is_file(): + raise FileNotFoundError(f"No such file: '{path_to_input_file}'.") + + # remove unwanted lines from hash file + input_file = open(path_to_input_file, "r") + input_file_lines = input_file.readlines() + input_file_lines_to_keep = [] + for line in input_file_lines: + if substring in line: + input_file_lines_to_keep.append(line) + input_file.close() + remove(path_to_input_file) + with open(path_to_input_file, "w") as input_file: + input_file.writelines(input_file_lines_to_keep) + + +def file_is_empty(path_to_input_file): + """Checks whether the input file is empty or not.""" + + if "~" in str(path_to_input_file): + path_to_input_file = Path(path_to_input_file).expanduser() + path_to_input_file = Path(path_to_input_file).resolve() + + if not path_to_input_file.is_file(): + raise FileNotFoundError(f"No such file: '{path_to_input_file}'.") + + with open(path_to_input_file, 'r') as input_file: + # try to read a byte + file_has_content = len(input_file.read(1)) > 0 + + return not file_has_content + + +def download_and_verify_debian_iso(path_to_output_file, printer=None): + """Downloads the latest Debian ISO as the specified output file. + + The file's integrity is validated using a SHA512 checksum. + The PGP signature of the SHA512SUMS file is checked using gpg. + + Attributes + ---------- + path_to_output_file : str or pathlike object + Path to the file as which the downloaded image will be saved. + printer : clibella.Printer + A CLI printer to be used for output. + """ + + if "~" in str(path_to_output_file): + path_to_output_file = Path(path_to_output_file).expanduser() + path_to_output_file = Path(path_to_output_file).resolve() + + if path_to_output_file.is_file(): + raise FileExistsError( + f"Output file '{path_to_output_file}' already exists." + ) + if not path_to_output_file.parent.is_dir(): + raise NotADirectoryError( + f"No such directory: '{path_to_output_file.parent}'." + ) + + if printer is None: + printer = Printer() + + # create a temporary directory + with TemporaryDirectory() as temp_dir: + # scrape for URLs and filenames + files = get_debian_iso_urls() + + # set file paths + path_to_hash_file = Path(temp_dir)/files["hash_file"]["name"] + path_to_signature_file = Path(temp_dir)/files["signature_file"]["name"] + path_to_image_file = Path(temp_dir)/files["image_file"]["name"] + + # download hash file and signature, and verify with gpg + download_file( + path_to_hash_file, + files["hash_file"]["url"], + show_progress=False, + printer=printer, + ) + download_file( + path_to_signature_file, + files["signature_file"]["url"], + show_progress=False, + printer=printer, + ) + + # verify the hash file using gpg + printer.info("Verifying hash file using gpg...") + if not debian_signing_key_is_imported(): + printer.info("Importing Debian public GPG CD signing key...") + import_debian_signing_key() + else: + printer.info("Found Debian public GPG CD signing key.") + try: + assert_detached_signature_is_valid( + path_to_hash_file, + path_to_signature_file, + ) + except VerificationFailedError: + printer.error("PGP signature of the hash file was invalid!") + exit(1) + printer.ok("HASH file PGP authenticity check passed.") + + # remove all lines from hash file not containing the image file name + trim_text_file(path_to_hash_file, files["image_file"]["name"]) + if file_is_empty(path_to_hash_file): + raise RuntimeError("Failed to locate SHA512 hash sum for image.") + + # download image file + download_file( + path_to_image_file, + files["image_file"]["url"], + show_progress=True, + printer=printer, + ) + + # validate SHA512 checksum + printer.info("Validating ISO file integrity...") + hash_check_result = run( + [ + "sha512sum", "--check", path_to_hash_file + ], + text=True, + stdout=PIPE, + stderr=STDOUT, + cwd=path_to_image_file.parent, + ) + if hash_check_result.returncode != 0: + raise RuntimeError("SHA512 checksum verification of the ISO failed.") + printer.ok("ISO file integrity check passed.") + + # move downloaded file to specified destination + rename(path_to_image_file, path_to_output_file) diff --git a/default-preseed.cfg b/default-preseed.cfg deleted file mode 100644 index 45184c1..0000000 --- a/default-preseed.cfg +++ /dev/null @@ -1,478 +0,0 @@ -#_preseed_V1 - -#### TODOs: -# configure partman answers -# install sudo and add user to sudo group -# configure sshd - -#### Contents of the preconfiguration file (for bullseye) -### Localization -# Preseeding only locale sets language, country and locale. -#d-i debian-installer/locale string en_US - -# The values can also be preseeded individually for greater flexibility. -d-i debian-installer/language string en -d-i debian-installer/country string DE -d-i debian-installer/locale string en_GB.UTF-8 -# Optionally specify additional locales to be generated. -d-i localechooser/supported-locales multiselect en_US.UTF-8, en_DK.UTF-8, de_DE.UTF-8 - -# Keyboard selection. -d-i keyboard-configuration/xkb-keymap select de -# d-i keyboard-configuration/toggle select No toggling - -### Network configuration -# Disable network configuration entirely. This is useful for cdrom -# installations on non-networked devices where the network questions, -# warning and long timeouts are a nuisance. -#d-i netcfg/enable boolean false - -# netcfg will choose an interface that has link if possible. This makes it -# skip displaying a list if there is more than one interface. -d-i netcfg/choose_interface select auto - -# To pick a particular interface instead: -#d-i netcfg/choose_interface select eth1 - -# To set a different link detection timeout (default is 3 seconds). -# Values are interpreted as seconds. -#d-i netcfg/link_wait_timeout string 10 - -# If you have a slow dhcp server and the installer times out waiting for -# it, this might be useful. -#d-i netcfg/dhcp_timeout string 60 -#d-i netcfg/dhcpv6_timeout string 60 - -# Automatic network configuration is the default. -# If you prefer to configure the network manually, uncomment this line and -# the static network configuration below. -#d-i netcfg/disable_autoconfig boolean true - -# If you want the preconfiguration file to work on systems both with and -# without a dhcp server, uncomment these lines and the static network -# configuration below. -#d-i netcfg/dhcp_failed note -#d-i netcfg/dhcp_options select Configure network manually - -# Static network configuration. -# -# IPv4 example -#d-i netcfg/get_ipaddress string 192.168.1.42 -#d-i netcfg/get_netmask string 255.255.255.0 -#d-i netcfg/get_gateway string 192.168.1.1 -#d-i netcfg/get_nameservers string 192.168.1.1 -#d-i netcfg/confirm_static boolean true -# -# IPv6 example -#d-i netcfg/get_ipaddress string fc00::2 -#d-i netcfg/get_netmask string ffff:ffff:ffff:ffff:: -#d-i netcfg/get_gateway string fc00::1 -#d-i netcfg/get_nameservers string fc00::1 -#d-i netcfg/confirm_static boolean true - -# Any hostname and domain names assigned from dhcp take precedence over -# values set here. However, setting the values still prevents the questions -# from being shown, even if values come from dhcp. -d-i netcfg/get_hostname string my-debian-machine -d-i netcfg/get_domain string example.com - -# If you want to force a hostname, regardless of what either the DHCP -# server returns or what the reverse DNS entry for the IP is, uncomment -# and adjust the following line. -#d-i netcfg/hostname string somehost - -# Disable that annoying WEP key dialog. -d-i netcfg/wireless_wep string -# The wacky dhcp hostname that some ISPs use as a password of sorts. -#d-i netcfg/dhcp_hostname string radish - -# If non-free firmware is needed for the network or other hardware, you can -# configure the installer to always try to load it, without prompting. Or -# change to false to disable asking. -#d-i hw-detect/load_firmware boolean true - -### Network console -# Use the following settings if you wish to make use of the network-console -# component for remote installation over SSH. This only makes sense if you -# intend to perform the remainder of the installation manually. -#d-i anna/choose_modules string network-console -#d-i network-console/authorized_keys_url string http://10.0.0.1/openssh-key -#d-i network-console/password password r00tme -#d-i network-console/password-again password r00tme - -### Mirror settings -# Mirror protocol: -# If you select ftp, the mirror/country string does not need to be set. -# Default value for the mirror protocol: http. -#d-i mirror/protocol string ftp -d-i mirror/country string manual -d-i mirror/http/hostname string debian.tu-bs.de -d-i mirror/http/directory string /debian -d-i mirror/http/proxy string - -# Suite to install. -#d-i mirror/suite string testing -# Suite to use for loading installer components (optional). -#d-i mirror/udeb/suite string testing - -### Account setup -# Skip creation of a root account (normal user account will be able to -# use sudo). -#d-i passwd/root-login boolean false -# Alternatively, to skip creation of a normal user account. -#d-i passwd/make-user boolean false - -# Root password, either in clear text -#d-i passwd/root-password password r00tme -#d-i passwd/root-password-again password r00tme -# or encrypted using a crypt(3) hash. -# generate using: python3 -c 'import crypt, getpass; print(crypt.crypt(getpass.getpass(), crypt.METHOD_SHA512))' -d-i passwd/root-password-crypted password $6$ejIby666HDLKIcgq$SthCbLDIgPlQZb/DNB0XOdgXTYGJIx5cu4qqSAH/KiyFEsDPKWTzyUipqu3TcsMziBo5WxiavPO.a0j8qW2ZY/ - -# To create a normal user account. -d-i passwd/user-fullname string John Doe -d-i passwd/username string administrator -# Normal user's password, either in clear text -#d-i passwd/user-password password insecure -#d-i passwd/user-password-again password insecure -# or encrypted using a crypt(3) hash. -d-i passwd/user-password-crypted password $6$ejIby666HDLKIcgq$SthCbLDIgPlQZb/DNB0XOdgXTYGJIx5cu4qqSAH/KiyFEsDPKWTzyUipqu3TcsMziBo5WxiavPO.a0j8qW2ZY/ -# Create the first user with the specified UID instead of the default. -#d-i passwd/user-uid string 1010 - -# The user account will be added to some standard initial groups. To -# override that, use this. -#d-i passwd/user-default-groups string audio cdrom video - -### Clock and time zone setup -# Controls whether or not the hardware clock is set to UTC. -d-i clock-setup/utc boolean true - -# You may set this to any valid setting for $TZ; see the contents of -# /usr/share/zoneinfo/ for valid values. -d-i time/zone string Europe/Amsterdam - -# Controls whether to use NTP to set the clock during the install -d-i clock-setup/ntp boolean true -# NTP server to use. The default is almost always fine here. -#d-i clock-setup/ntp-server string ntp.example.com - -### Partitioning -## Partitioning example -# If the system has free space you can choose to only partition that space. -# This is only honoured if partman-auto/method (below) is not set. -#d-i partman-auto/init_automatically_partition select biggest_free - -# Alternatively, you may specify a disk to partition. If the system has only -# one disk the installer will default to using that, but otherwise the device -# name must be given in traditional, non-devfs format (so e.g. /dev/sda -# and not e.g. /dev/discs/disc0/disc). -# For example, to use the first SCSI/SATA hard disk: -#d-i partman-auto/disk string /dev/sda -# In addition, you'll need to specify the method to use. -# The presently available methods are: -# - regular: use the usual partition types for your architecture -# - lvm: use LVM to partition the disk -# - crypto: use LVM within an encrypted partition -d-i partman-auto/method string lvm - -# You can define the amount of space that will be used for the LVM volume -# group. It can either be a size with its unit (eg. 20 GB), a percentage of -# free space or the 'max' keyword. -d-i partman-auto-lvm/guided_size string max - -# If one of the disks that are going to be automatically partitioned -# contains an old LVM configuration, the user will normally receive a -# warning. This can be preseeded away... -d-i partman-lvm/device_remove_lvm boolean true -# The same applies to pre-existing software RAID array: -d-i partman-md/device_remove_md boolean true -# And the same goes for the confirmation to write the lvm partitions. -d-i partman-lvm/confirm boolean true -d-i partman-lvm/confirm_nooverwrite boolean true - -# You can choose one of the three predefined partitioning recipes: -# - atomic: all files in one partition -# - home: separate /home partition -# - multi: separate /home, /var, and /tmp partitions -d-i partman-auto/choose_recipe select atomic - -# Or provide a recipe of your own... -# If you have a way to get a recipe file into the d-i environment, you can -# just point at it. -#d-i partman-auto/expert_recipe_file string /hd-media/recipe - -# If not, you can put an entire recipe into the preconfiguration file in one -# (logical) line. This example creates a small /boot partition, suitable -# swap, and uses the rest of the space for the root partition: -#d-i partman-auto/expert_recipe string \ -# boot-root :: \ -# 40 50 100 ext3 \ -# $primary{ } $bootable{ } \ -# method{ format } format{ } \ -# use_filesystem{ } filesystem{ ext3 } \ -# mountpoint{ /boot } \ -# . \ -# 500 10000 1000000000 ext3 \ -# method{ format } format{ } \ -# use_filesystem{ } filesystem{ ext3 } \ -# mountpoint{ / } \ -# . \ -# 64 512 300% linux-swap \ -# method{ swap } format{ } \ -# . - -# The full recipe format is documented in the file partman-auto-recipe.txt -# included in the 'debian-installer' package or available from D-I source -# repository. This also documents how to specify settings such as file -# system labels, volume group names and which physical devices to include -# in a volume group. - -## Partitioning for EFI -# If your system needs an EFI partition you could add something like -# this to the recipe above, as the first element in the recipe: -# 538 538 1075 free \ -# $iflabel{ gpt } \ -# $reusemethod{ } \ -# method{ efi } \ -# format{ } \ -# . \ -# -# The fragment above is for the amd64 architecture; the details may be -# different on other architectures. The 'partman-auto' package in the -# D-I source repository may have an example you can follow. - -# This makes partman automatically partition without confirmation, provided -# that you told it what to do using one of the methods above. -d-i partman-partitioning/confirm_write_new_label boolean true -d-i partman/choose_partition select finish -d-i partman/confirm boolean true -d-i partman/confirm_nooverwrite boolean true - -# Force UEFI booting ('BIOS compatibility' will be lost). Default: false. -#d-i partman-efi/non_efi_system boolean true -# Ensure the partition table is GPT - this is required for EFI -#d-i partman-partitioning/choose_label select gpt -#d-i partman-partitioning/default_label string gpt - -# When disk encryption is enabled, skip wiping the partitions beforehand. -#d-i partman-auto-crypto/erase_disks boolean false - -## Partitioning using RAID -# The method should be set to "raid". -#d-i partman-auto/method string raid -# Specify the disks to be partitioned. They will all get the same layout, -# so this will only work if the disks are the same size. -#d-i partman-auto/disk string /dev/sda /dev/sdb - -# Next you need to specify the physical partitions that will be used. -#d-i partman-auto/expert_recipe string \ -# multiraid :: \ -# 1000 5000 4000 raid \ -# $primary{ } method{ raid } \ -# . \ -# 64 512 300% raid \ -# method{ raid } \ -# . \ -# 500 10000 1000000000 raid \ -# method{ raid } \ -# . - -# Last you need to specify how the previously defined partitions will be -# used in the RAID setup. Remember to use the correct partition numbers -# for logical partitions. RAID levels 0, 1, 5, 6 and 10 are supported; -# devices are separated using "#". -# Parameters are: -# \ -# - -#d-i partman-auto-raid/recipe string \ -# 1 2 0 ext3 / \ -# /dev/sda1#/dev/sdb1 \ -# . \ -# 1 2 0 swap - \ -# /dev/sda5#/dev/sdb5 \ -# . \ -# 0 2 0 ext3 /home \ -# /dev/sda6#/dev/sdb6 \ -# . - -# For additional information see the file partman-auto-raid-recipe.txt -# included in the 'debian-installer' package or available from D-I source -# repository. - -# This makes partman automatically partition without confirmation. -d-i partman-md/confirm boolean true -d-i partman-partitioning/confirm_write_new_label boolean true -d-i partman/choose_partition select finish -d-i partman/confirm boolean true -d-i partman/confirm_nooverwrite boolean true - -## Controlling how partitions are mounted -# The default is to mount by UUID, but you can also choose "traditional" to -# use traditional device names, or "label" to try filesystem labels before -# falling back to UUIDs. -#d-i partman/mount_style select uuid - -### Base system installation -# Configure APT to not install recommended packages by default. Use of this -# option can result in an incomplete system and should only be used by very -# experienced users. -#d-i base-installer/install-recommends boolean false - -# The kernel image (meta) package to be installed; "none" can be used if no -# kernel is to be installed. -#d-i base-installer/kernel/image string linux-image-686 - -### Apt setup -# Choose, if you want to scan additional installation media -# (default: false). -d-i apt-setup/cdrom/set-first boolean false -# You can choose to install non-free and contrib software. -#d-i apt-setup/non-free boolean true -#d-i apt-setup/contrib boolean true -# Uncomment the following line, if you don't want to have the sources.list -# entry for a DVD/BD installation image active in the installed system -# (entries for netinst or CD images will be disabled anyway, regardless of -# this setting). -#d-i apt-setup/disable-cdrom-entries boolean true -# Uncomment this if you don't want to use a network mirror. -#d-i apt-setup/use_mirror boolean false -# Select which update services to use; define the mirrors to be used. -# Values shown below are the normal defaults. -#d-i apt-setup/services-select multiselect security, updates -#d-i apt-setup/security_host string security.debian.org - -# Additional repositories, local[0-9] available -#d-i apt-setup/local0/repository string \ -# http://local.server/debian stable main -#d-i apt-setup/local0/comment string local server -# Enable deb-src lines -#d-i apt-setup/local0/source boolean true -# URL to the public key of the local repository; you must provide a key or -# apt will complain about the unauthenticated repository and so the -# sources.list line will be left commented out. -#d-i apt-setup/local0/key string http://local.server/key -# If the provided key file ends in ".asc" the key file needs to be an -# ASCII-armoured PGP key, if it ends in ".gpg" it needs to use the -# "GPG key public keyring" format, the "keybox database" format is -# currently not supported. - -# By default the installer requires that repositories be authenticated -# using a known gpg key. This setting can be used to disable that -# authentication. Warning: Insecure, not recommended. -#d-i debian-installer/allow_unauthenticated boolean true - -# Uncomment this to add multiarch configuration for i386 -#d-i apt-setup/multiarch string i386 - - -### Package selection -tasksel tasksel/first multiselect standard, ssh-server - -# Or choose to not get the tasksel dialog displayed at all (and don't install -# any packages): -#d-i pkgsel/run_tasksel boolean false - -# Individual additional packages to install -#d-i pkgsel/include string openssh-server build-essential -# Whether to upgrade packages after debootstrap. -# Allowed values: none, safe-upgrade, full-upgrade -d-i pkgsel/upgrade select full-upgrade - -# You can choose, if your system will report back on what software you have -# installed, and what software you use. The default is not to report back, -# but sending reports helps the project determine what software is most -# popular and should be included on the first CD/DVD. -popularity-contest popularity-contest/participate boolean false - -### Boot loader installation -# Grub is the boot loader (for x86). - -# This is fairly safe to set, it makes grub install automatically to the UEFI -# partition/boot record if no other operating system is detected on the machine. -d-i grub-installer/only_debian boolean true - -# This one makes grub-installer install to the UEFI partition/boot record, if -# it also finds some other OS, which is less safe as it might not be able to -# boot that other OS. -d-i grub-installer/with_other_os boolean true - -# Due notably to potential USB sticks, the location of the primary drive can -# not be determined safely in general, so this needs to be specified: -d-i grub-installer/bootdev string /dev/sda -# To install to the primary device (assuming it is not a USB stick): -#d-i grub-installer/bootdev string default - -# Alternatively, if you want to install to a location other than the UEFI -# parition/boot record, uncomment and edit these lines: -#d-i grub-installer/only_debian boolean false -#d-i grub-installer/with_other_os boolean false -#d-i grub-installer/bootdev string (hd0,1) -# To install grub to multiple disks: -#d-i grub-installer/bootdev string (hd0,1) (hd1,1) (hd2,1) - -# Optional password for grub, either in clear text -#d-i grub-installer/password password r00tme -#d-i grub-installer/password-again password r00tme -# or encrypted using an MD5 hash, see grub-md5-crypt(8). -#d-i grub-installer/password-crypted password [MD5 hash] - -# Use the following option to add additional boot parameters for the -# installed system (if supported by the bootloader installer). -# Note: options passed to the installer will be added automatically. -#d-i debian-installer/add-kernel-opts string nousb - -### Finishing up the installation -# During installations from serial console, the regular virtual consoles -# (VT1-VT6) are normally disabled in /etc/inittab. Uncomment the next -# line to prevent this. -#d-i finish-install/keep-consoles boolean true - -# Avoid that last message about the install being complete. -d-i finish-install/reboot_in_progress note - -# This will prevent the installer from ejecting the CD during the reboot, -# which is useful in some situations. -#d-i cdrom-detect/eject boolean false - -# This is how to make the installer shutdown when finished, but not -# reboot into the installed system. -#d-i debian-installer/exit/halt boolean true -# This will power off the machine instead of just halting it. -#d-i debian-installer/exit/poweroff boolean true - -### Preseeding other packages -# Depending on what software you choose to install, or if things go wrong -# during the installation process, it's possible that other questions may -# be asked. You can preseed those too, of course. To get a list of every -# possible question that could be asked during an install, do an -# installation, and then run these commands: -# debconf-get-selections --installer > file -# debconf-get-selections >> file - - -#### Advanced options -### Running custom commands during the installation -# d-i preseeding is inherently not secure. Nothing in the installer checks -# for attempts at buffer overflows or other exploits of the values of a -# preconfiguration file like this one. Only use preconfiguration files from -# trusted locations! To drive that home, and because it's generally useful, -# here's a way to run any shell command you'd like inside the installer, -# automatically. - -# This first command is run as early as possible, just after -# preseeding is read. -#d-i preseed/early_command string anna-install some-udeb -# This command is run immediately before the partitioner starts. It may be -# useful to apply dynamic partitioner preseeding that depends on the state -# of the disks (which may not be visible when preseed/early_command runs). -#d-i partman/early_command \ -# string debconf-set partman-auto/disk "$(list-devices disk | head -n1)" -# This command is run just before the install finishes, but when there is -# still a usable /target directory. You can chroot to /target and use it -# directly, or use the apt-install and in-target commands to easily install -# packages and run commands in the target system. -#d-i preseed/late_command string apt-install zsh; in-target chsh -s /bin/zsh - - diff --git a/gpg/__init__.py b/gpg/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/gpg/exceptions.py b/gpg/exceptions.py new file mode 100644 index 0000000..7490aa4 --- /dev/null +++ b/gpg/exceptions.py @@ -0,0 +1,37 @@ +"""Exceptions which may be raised during execution of gpg wrapper functions.""" + + +class GpgProgrammingException(Exception): + """A general exception indicating a programming error.""" + + pass + + +class GpgRuntimeError(RuntimeError): + """A general exception indicating a runtime error.""" + + pass + + +class UnexpectedOutputException(GpgProgrammingException): + """Raised when a gpg subprocess produces unexpected output.""" + + pass + + +class MissingLocalKeyError(GpgRuntimeError): + """Raised when an expected key is missing from the local keystore.""" + + pass + + +class InvalidSignatureError(GpgRuntimeError): + """Raised when an invalid gpg signature is encountered.""" + + pass + + +class VerificationFailedError(GpgRuntimeError): + """Raised when a gpg verfication encounters a bad signature.""" + + pass diff --git a/gpg/keystore.py b/gpg/keystore.py new file mode 100644 index 0000000..1c82489 --- /dev/null +++ b/gpg/keystore.py @@ -0,0 +1,112 @@ +"""Utilities for searching and importing GPG keys locally.""" + +from subprocess import run, STDOUT, PIPE +from re import compile + + +_DEBIAN_KEY_SERVER_HOSTNAME = "keyring.debian.org" +_DEBIAN_CD_SIGNING_KEY_ID = "DA87E80D6294BE9B" + + +def import_debian_signing_key(): + """Imports the public debian CD signing key using gpg. + + The key is imported from keyring.debian.org into the invoking user's + GPG public key store using a shell command. + """ + + # execute a gpg key import as a shell command, redirecting stderr to stdout + process_result = run( + [ + "gpg", "--keyserver", _DEBIAN_KEY_SERVER_HOSTNAME, + "--recv-key", _DEBIAN_CD_SIGNING_KEY_ID + ], + stdout=PIPE, + stderr=STDOUT, + text=True, + ) + + # check shell return code + if process_result.returncode != 0: + if process_result.stdout: + raise RuntimeError( + f"Failed to import key using gpg:\n{process_result.stdout}" + ) + else: + raise RuntimeError("Failed to import key using gpg.") + + # check shell output: + # the first line of stdout should look like this + expected_first_line = str( + f"gpg: key {_DEBIAN_CD_SIGNING_KEY_ID}: public key " + f"\"Debian CD signing key \"" + f" imported" + ) + + if not process_result.stdout.split("\n")[0] == expected_first_line: + raise RuntimeError( + f"Unexpected output while importing PGP public key:\n" + f"{process_result.stdout}" + ) + + +def debian_signing_key_is_imported(): + """Checks whether the debian PGP signing key exists in the local key store. + + The invoking user's GPG key store is checked using a shell command. + + Returns + ------- + True : bool + If the public PGP debian cd signing key exists in the invoking user's + GPG key store. + False : bool + If the public PGP debian cd signing key does not exist in the invoking + user's GPG key store. + """ + + # execute a local gpg key lookup as a shell command, redirecting stderr to + # stdout + # NOTE: this command returns 0 even if the key is not present + process_result = run( + ["gpg", "--locate-keys", _DEBIAN_CD_SIGNING_KEY_ID], + stdout=PIPE, + stderr=STDOUT, + text=True, + ) + + # check shell return code + if process_result.returncode != 0: + raise RuntimeError("Failed to search local keys using gpg.") + + # no shell output means that the key does not exist locally + if not process_result.stdout: + return False + + # verify existing key shell output using regex: + # it should contain six lines in the following format + expected_output_lines_regexes = [ + compile(r"^pub .*$"), + compile(r"^ *[0-9A-F]{40}$"), + compile(r"^uid .*$"), + compile(r"^sub .*$"), + compile(r"^$"), + compile(r"^$"), + ] + + actual_output_lines = process_result.stdout.split("\n") + if not len(actual_output_lines) == len(expected_output_lines_regexes): + raise RuntimeError( + f"Unexpected line count in shell output while performing local" + f"GPG key lookup:\n" + f"{process_result.stdout}" + ) + for i in range(4): + if not expected_output_lines_regexes[i].match(actual_output_lines[i]): + raise RuntimeError( + f"Unexpected shell output format while performing local" + f"GPG key lookup:\n" + f"{process_result.stdout}" + ) + + return True diff --git a/gpg/verify.py b/gpg/verify.py new file mode 100644 index 0000000..5a3460c --- /dev/null +++ b/gpg/verify.py @@ -0,0 +1,128 @@ +"""Utilities for verifying files using gpg.""" + +from re import compile +from pathlib import Path +from subprocess import run, PIPE, STDOUT + +import gpg.exceptions as ex + + +def assert_detached_signature_is_valid( + path_to_input_file, + path_to_signature_file +): + """Verifies the input file using the specified detached gpg signature. + + The invoking user's local gpg key store is used for verification, the + public key used to create the signature must be present in the invoking + user's local gpg key store. + + Parameters + ---------- + path_to_input_file : str or pathlike object + Path to the file which should be verified. + path_to_signature_file : str or pathlike object + Path to the file containing a detached gpg signature of the input file. + + Raises + ------ + FileNotFoundError + If either of the input files do not exist. + gpg.exceptions.MissingLocalKeyError + If a key referenced by the signature file could not be found in the + invoking user's local key store. + gpg.exceptions.VerificationFailedError + If the gpg verification detects a bad signature. + """ + + if '~' in str(path_to_input_file): + path_to_input_file = Path(path_to_input_file).expanduser() + path_to_input_file = Path(path_to_input_file).resolve() + + if '~' in str(path_to_signature_file): + path_to_signature_file = Path(path_to_signature_file).expanduser() + path_to_signature_file = Path(path_to_signature_file).resolve() + + if not path_to_input_file.is_file(): + raise FileNotFoundError( + f"No such file: '{path_to_input_file}'." + ) + if not path_to_signature_file.is_file(): + raise FileNotFoundError( + f"No such file: '{path_to_signature_file}'." + ) + + # execute a gpg verification as a shell command, redirecting stderr to + # stdout + process_result = run( + [ + "gpg", "--verify", path_to_signature_file, path_to_input_file, + ], + stdout=PIPE, + stderr=STDOUT, + text=True, + ) + output_lines = process_result.stdout.split("\n") + if len(output_lines) < 3: + raise ex.UnexpectedOutputException( + f"Unexpected output during gpg verification:\n" + f"{process_result.stdout}" + ) + + if process_result.returncode == 2: + # a missing local key causes return code 2 + # and the following output on the third line: + missing_key_regex = compile( + r"^gpg: Can't check signature: No public key$" + ) + # an invalid detached signature file causes return code 2 + # and the following output on the first line: + invalid_signature_regex = compile( + r"^gpg: no valid OpenPGP data found.$" + ) + + if missing_key_regex.match(output_lines[2]): + raise ex.MissingLocalKeyError( + "Failed to verify gpg signature: no matching local key." + ) + elif invalid_signature_regex.match(output_lines[0]): + raise ex.InvalidSignatureError( + "Invalid signature file." + ) + else: + raise ex.UnexpectedOutputException( + f"Unexpected output during gpg verification:\n" + f"{process_result.stdout}" + ) + elif process_result.returncode == 1: + # failed verification causes return code 1 + # and the following output on line 3: + verification_failed_regex = compile( + r"^gpg: BAD signature from .*$" + ) + if not verification_failed_regex.match(output_lines[2]): + raise ex.UnexpectedOutputException( + f"Unexpected output during gpg verification:\n" + f"{process_result.stdout}" + ) + else: + raise ex.VerificationFailedError( + "gpg signature verification failed: BAD SIGNATURE!" + ) + elif process_result.returncode == 0: + # successful verification causes return code 0 + # and the following output on line 3: + verification_successful_regex = compile( + r"^gpg: Good signature from .*$" + ) + if not verification_successful_regex.match(output_lines[2]): + raise ex.UnexpectedOutputException( + f"Unexpected output during gpg verification:\n" + f"{process_result.stdout}" + ) + else: + raise ex.UnexpectedOutputException( + f"Unexpected return code during gpg verification:\n" + f"{process_result.returncode}" + f"{process_result.stdout}" + ) diff --git a/iso/__init__.py b/iso/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/udib/modiso.py b/iso/injection.py similarity index 62% rename from udib/modiso.py rename to iso/injection.py index 42a86d3..a6067ca 100644 --- a/udib/modiso.py +++ b/iso/injection.py @@ -1,4 +1,4 @@ -"""Library for modification of disk image files. +"""Utilities for modification of disk image files. Image file modification includes extracting ISO archives, adding files to initrd-archives contained within the ISO, recalculating md5sum-files @@ -8,48 +8,15 @@ local filesystem. """ from pathlib import Path +from tempfile import TemporaryDirectory import gzip import hashlib import re import shutil import subprocess - -def _find_all_files_under(parent_dir): - """Recursively finds all files anywhere under the specified directory. - - Returns a list of absolute Path objects. Symlinks are ignored. - - Parameters - ---------- - parent_dir : str or pathlike object - The directory under which to recursively find files. - - Raises - ------ - NotADirectoryError - Raised if the specified parent directory is not a directory. - - Examples - -------- - config_files = _find_all_files_under("~/.config") - - """ - - parent_dir = Path(parent_dir).resolve() - - if not parent_dir.is_dir(): - raise NotADirectoryError(f"No such directory: '{parent_dir}'.") - - files = [] - - for subpath in parent_dir.iterdir(): - if subpath.is_file(): - files.append(subpath.resolve()) - elif not subpath.is_symlink() and subpath.is_dir(): - files += _find_all_files_under(subpath) - - return files +from cli.clibella import Printer +from core.utils import find_all_files_under def extract_iso(path_to_output_dir, path_to_input_file): @@ -78,8 +45,13 @@ def extract_iso(path_to_output_dir, path_to_input_file): """ - path_to_output_dir = Path(path_to_output_dir) - path_to_input_file = Path(path_to_input_file) + if "~" in str(path_to_output_dir): + path_to_output_dir = Path(path_to_output_dir).expanduser() + path_to_output_dir = Path(path_to_output_dir).resolve() + + if "~" in str(path_to_input_file): + path_to_input_file = Path(path_to_input_file).expanduser() + path_to_input_file = Path(path_to_input_file).resolve() # check if paths are valid if not path_to_output_dir.is_dir(): @@ -93,19 +65,23 @@ def extract_iso(path_to_output_dir, path_to_input_file): [ "xorriso", "-osirrox", "on", - "-indev", path_to_input_file.resolve(), + "-indev", path_to_input_file, "-extract", "/", - path_to_output_dir.resolve() + path_to_output_dir ], capture_output=True, - check=True) + check=True + ) except subprocess.CalledProcessError: raise RuntimeError( - f"An error occurred while extracting '{path_to_input_file}'.") + f"An error occurred while extracting '{path_to_input_file}'." + ) -def append_file_contents_to_initrd_archive(path_to_initrd_archive, - path_to_input_file): +def append_file_contents_to_initrd_archive( + path_to_initrd_archive, + path_to_input_file +): """Appends the input file to the specified initrd archive. The initrd archive is extracted, the input file is appended, and @@ -137,8 +113,13 @@ def append_file_contents_to_initrd_archive(path_to_initrd_archive, """ - path_to_initrd_archive = Path(path_to_initrd_archive) - path_to_input_file = Path(path_to_input_file) + if "~" in str(path_to_initrd_archive): + path_to_initrd_archive = Path(path_to_initrd_archive).expanduser() + path_to_initrd_archive = Path(path_to_initrd_archive).resolve() + + if "~" in str(path_to_input_file): + path_to_input_file = Path(path_to_input_file).expanduser() + path_to_input_file = Path(path_to_input_file).resolve() # check if initrd file exists and has the correct name if not path_to_initrd_archive.is_file(): @@ -218,12 +199,15 @@ def regenerate_iso_md5sums_file(path_to_extracted_iso_root): """ + if "~" in str(path_to_extracted_iso_root): + path_to_extracted_iso_root = Path(path_to_extracted_iso_root).expanduser() path_to_extracted_iso_root = Path(path_to_extracted_iso_root).resolve() # check if input path exists if not path_to_extracted_iso_root.is_dir(): - raise NotADirectoryError(f"No such directory: " - f"'{path_to_extracted_iso_root}'.") + raise NotADirectoryError( + f"No such directory: '{path_to_extracted_iso_root}'." + ) path_to_md5sum_file = path_to_extracted_iso_root/"md5sum.txt" @@ -240,7 +224,7 @@ def regenerate_iso_md5sums_file(path_to_extracted_iso_root): # Note the two spaces between hash and filepath! # find all files - subpaths = _find_all_files_under(path_to_extracted_iso_root) + subpaths = find_all_files_under(path_to_extracted_iso_root) with open(path_to_md5sum_file, "w") as md5sum_file: for subpath in subpaths: @@ -252,7 +236,8 @@ def regenerate_iso_md5sums_file(path_to_extracted_iso_root): md5hash.hexdigest() + " " + str(subpath.relative_to(path_to_extracted_iso_root)) - + "\n") + + "\n" + ) # revert write permissions from md5sum.txt and its parent dir path_to_md5sum_file.chmod(0o444) @@ -291,7 +276,12 @@ def extract_mbr_from_iso(path_to_output_file, path_to_source_iso): """ - path_to_output_file = Path(path_to_output_file) + if "~" in str(path_to_output_file): + path_to_output_file = Path(path_to_output_file).expanduser() + path_to_output_file = Path(path_to_output_file).resolve() + + if "~" in str(path_to_source_iso): + path_to_source_iso = Path(path_to_source_iso).expanduser() path_to_source_iso = Path(path_to_source_iso) # make sure output file does not exist already @@ -363,9 +353,17 @@ def repack_iso(path_to_output_iso, """ - path_to_output_iso = Path(path_to_output_iso) - path_to_mbr_data_file = Path(path_to_mbr_data_file) - path_to_input_files_root_dir = Path(path_to_input_files_root_dir) + if "~" in str(path_to_output_iso): + path_to_output_iso = Path(path_to_output_iso).expanduser() + path_to_output_iso = Path(path_to_output_iso).resolve() + + if "~" in str(path_to_mbr_data_file): + path_to_mbr_data_file = Path(path_to_mbr_data_file).expanduser() + path_to_mbr_data_file = Path(path_to_mbr_data_file).resolve() + + if "~" in str(path_to_input_files_root_dir): + path_to_input_files_root_dir = Path(path_to_input_files_root_dir).expanduser() + path_to_input_files_root_dir = Path(path_to_input_files_root_dir).resolve() # make sure output file does not exist yet if path_to_output_iso.exists(): @@ -391,21 +389,136 @@ def repack_iso(path_to_output_iso, # repack the ISO using xorriso try: subprocess.run( - ["xorriso", "-as", "mkisofs", - "-r", "-V", created_iso_filesystem_name, - "-o", path_to_output_iso.resolve(), - "-J", "-J", "-joliet-long", "-cache-inodes", - "-isohybrid-mbr", path_to_mbr_data_file.resolve(), - "-b", "isolinux/isolinux.bin", - "-c", "isolinux/boot.cat", - "-boot-load-size", "4", "-boot-info-table", "-no-emul-boot", - "-eltorito-alt-boot", - "-e", "boot/grub/efi.img", "-no-emul-boot", - "-isohybrid-gpt-basdat", "-isohybrid-apm-hfsplus", - path_to_input_files_root_dir.resolve()], + [ + "xorriso", "-as", "mkisofs", + "-r", "-V", created_iso_filesystem_name, + "-o", path_to_output_iso, + "-J", "-J", "-joliet-long", "-cache-inodes", + "-isohybrid-mbr", path_to_mbr_data_file, + "-b", "isolinux/isolinux.bin", + "-c", "isolinux/boot.cat", + "-boot-load-size", "4", "-boot-info-table", "-no-emul-boot", + "-eltorito-alt-boot", + "-e", "boot/grub/efi.img", "-no-emul-boot", + "-isohybrid-gpt-basdat", "-isohybrid-apm-hfsplus", + path_to_input_files_root_dir, + ], capture_output=True, - check=True) + check=True, + ) except subprocess.CalledProcessError: raise RuntimeError(f"Failed while repacking ISO from source files: " f"'{path_to_input_files_root_dir}'.") + + +def inject_preseed_file_into_iso( + path_to_output_iso_file, + path_to_input_iso_file, + path_to_input_preseed_file, + iso_filesystem_name="Debian", + printer=None, +): + """Injects the specified preseed file into the specified ISO file. + + Extracts the input ISO into a temporary directory, then extracts the input + ISO's MBR into a temporary file, then appends the preseed file to the + extracted ISO's initrd, then regenerates the extracted ISO's internal MD5 + hash list and finally repacks the extracted ISO into the output ISO. + + The input ISO file itself is left unchanged. + The output ISO file is newly created. + + Parameters + ---------- + path_to_output_iso_file : str or pathlike object + Path to which the resulting ISO file will be saved. + path_to_input_iso_file : str or pathlike object + Path to the origin ISO file. + path_to_input_preseed_file : str or pathlike object + Path to the input preseed file. + printer : clibella.Printer + A printer for CLI output. + """ + + # verify and resolve paths + if "~" in str(path_to_input_iso_file): + path_to_input_iso_file = Path(path_to_input_iso_file).expanduser() + path_to_input_iso_file = Path(path_to_input_iso_file).resolve() + if not path_to_input_iso_file.is_file(): + raise FileNotFoundError(f"No such file: '{path_to_input_iso_file}'.") + + if "~" in str(path_to_output_iso_file): + path_to_output_iso_file = Path(path_to_output_iso_file).expanduser() + path_to_output_iso_file = Path(path_to_output_iso_file).resolve() + if path_to_output_iso_file.is_file(): + raise FileExistsError(f"Output file exists: '{path_to_input_iso_file}'.") + if not path_to_output_iso_file.parent.is_dir(): + raise NotADirectoryError(f"No such directory: '{path_to_output_iso_file.parent}'.") + + if "~" in str(path_to_input_preseed_file): + path_to_input_preseed_file = Path(path_to_input_preseed_file).expanduser() + path_to_input_preseed_file = Path(path_to_input_preseed_file).resolve() + if not path_to_input_preseed_file.is_file(): + raise FileNotFoundError(f"No such file: '{path_to_input_preseed_file}'.") + + if printer is None: + p = Printer() + else: + if not isinstance(printer, Printer): + raise TypeError(f"Expected a {type(Printer)} object.") + p = printer + + # extract image file to a temporary directory + temp_extracted_iso_dir = TemporaryDirectory() + path_to_extracted_iso_dir = Path(temp_extracted_iso_dir.name) + p.info(f"Extracting contents of {path_to_input_iso_file.name}...") + extract_iso( + path_to_extracted_iso_dir, + path_to_input_iso_file + ) + p.ok("ISO extraction complete.") + + # extract ISO MBR into a temporary directory + p.info(f"Extracting MBR from {path_to_input_iso_file.name}...") + temp_mbr_dir = TemporaryDirectory() + path_to_mbr_dir = Path(temp_mbr_dir.name) + path_to_mbr_file = path_to_mbr_dir/"mbr.bin" + extract_mbr_from_iso( + path_to_mbr_file, + path_to_input_iso_file, + ) + p.ok("MBR extraction complete.") + + # create a correctly named copy of the input preseed file and append it + # to the extracted ISO's initrd + p.info(f"Appending {path_to_input_preseed_file.name} to initrd...") + temp_preseed_dir = TemporaryDirectory() + path_to_preseed_dir = Path(temp_preseed_dir.name) + path_to_preseed_file = path_to_preseed_dir/"preseed.cfg" + shutil.copy(path_to_input_preseed_file, path_to_preseed_file) + append_file_contents_to_initrd_archive( + path_to_extracted_iso_dir/"install.amd"/"initrd.gz", + path_to_preseed_file + ) + p.ok("Preseed file appended successfully.") + + # regenerate extracted ISO's md5sum.txt file + p.info("Regenerating MD5 checksums...") + regenerate_iso_md5sums_file(path_to_extracted_iso_dir) + p.ok("MD5 calculations complete.") + + # repack exctracted ISO into a single file + p.info("Repacking ISO...") + repack_iso( + path_to_output_iso_file, + path_to_mbr_file, + path_to_extracted_iso_dir, + iso_filesystem_name + ) + p.success(f"ISO file was created successfully at '{path_to_output_iso_file}'.") + + # clear out temporary directories + temp_preseed_dir.cleanup() + temp_mbr_dir.cleanup() + temp_extracted_iso_dir.cleanup() diff --git a/net/__init__.py b/net/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/net/download.py b/net/download.py new file mode 100644 index 0000000..95e419b --- /dev/null +++ b/net/download.py @@ -0,0 +1,79 @@ +"""Library for downloading files from the web with CLI output.""" + +from pathlib import Path + +import requests +from tqdm import tqdm + +from cli.clibella import Printer + + +def download_file( + path_to_output_file, + url_to_file, + show_progress=False, + printer=None, +): + """Downloads the file at the input URL to the specified path. + + The file is downloaded via HTTP/HTTPS and saved to the specified path. + Optionally, displays a nice status bar. + + Parameters + ---------- + path_to_output_file : str or pathlike object + Path to a file as which the downloaded file is saved. + url_to_file : str + URL to the file to be downloaded. + show_progress : bool + When True, a progress bar is displayed on StdOut indicating the + progress of the download. + printer : clibella.Printer + A clibella.Printer used to print CLI output. + """ + + if '~' in str(path_to_output_file): + path_to_output_file = Path(path_to_output_file).expanduser() + path_to_output_file = Path(path_to_output_file).resolve() + + if not path_to_output_file.parent.is_dir(): + raise FileNotFoundError( + f"No such directory: '{path_to_output_file.parent}'." + ) + if path_to_output_file.exists(): + raise FileExistsError( + f"File already exists: '{path_to_output_file}'" + ) + + if printer is None: + p = Printer() + else: + p = printer + + output_file_name = path_to_output_file.name + with open(path_to_output_file, "wb") as output_file: + p.info(f"Downloading '{output_file_name}'...") + file_response = requests.get(url_to_file, stream=True) + total_length = file_response.headers.get('content-length') + + if total_length is None: # no content length header + output_file.write(file_response.content) + else: + if (show_progress): + total_length = int(total_length) + progress_bar = tqdm( + total=total_length, + unit="B", + unit_scale=True, + unit_divisor=1024 + ) + + for data in file_response.iter_content(chunk_size=4096): + output_file.write(data) + if (show_progress): + progress_bar.update(len(data)) + + if (show_progress): + progress_bar.close() + + p.ok(f"Received '{output_file_name}'.") diff --git a/net/scrape.py b/net/scrape.py new file mode 100644 index 0000000..fae1aaf --- /dev/null +++ b/net/scrape.py @@ -0,0 +1,108 @@ +"""Methods for scraping the debian website for specific file URLs.""" + +from re import compile + +import requests +from bs4 import BeautifulSoup + + +def get_debian_preseed_file_urls(): + """Returns a dict containing the URLs for the debian example preseed files. + + The dict has the following structure: + { + "basic": { + "url": "https://...", + "name": "...", + }, + "full": { + "url": "https://...", + "name": "...", + }, + } + where "basic" points to the basic preseed file and its filename, and "full" + points to the full preseed file and its filename. + """ + + preseed_file_urls = { + "basic": { + "url": "https://www.debian.org/releases/stable/example-preseed.txt", + "name": "example-preseed.txt", + }, + "full": { + "url": "https://preseed.debian.net/debian-preseed/bullseye/amd64-main-full.txt", + "name": "amd64-main-full.txt", + }, + } + + return preseed_file_urls + + +def get_debian_iso_urls(): + """Retrieves a dict containing the URLs for a debian installation image. + + The dict has the following structure: + { + "image_file": { + "url": "https://...", + "name": "debian-xx.x.x-amd64-netinst.iso", + }, + "hash_file": { + "url": "https://...", + "name": "SHA512SUMS", + }, + "signature_file": { + "url": "https://...", + "name": "SHA512SUMS.sign", + }, + } + where "image_file" is points to the latest debian stable x86-64bit + net-installation ISO image, "hash_file" points to a SHA512SUMS file + containing the SHA512 checksum for the ISO file, and "signature_file" + points to a file containing a PGP signature for verification of the + SHA512SUMS file. + Each top-level dict entry contains a "name" key representing a file name, + and a "url" key specifying a URL to that file. + + The function scrapes the official debian.org website to retrieve the URLs. + """ + + # request the debian releases page + releases_url = "https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/" + releases_page = requests.get(releases_url) + if not releases_page.status_code == 200: + raise RuntimeError("Unexpected status code during request.") + + hash_file_name = "SHA512SUMS" + hash_file_url = releases_url + hash_file_name + signature_file_name = "SHA512SUMS.sign" + signature_file_url = releases_url + signature_file_name + + # find the exact URL to the latest stable x64 netinst ISO file + soup = BeautifulSoup(releases_page.content, "html.parser") + image_file_links = soup.find_all( + name="a", + string=compile(r"debian-[0-9.]*-amd64-netinst.iso") + ) + if len(image_file_links) != 1: + raise RuntimeError( + "Failed to find an exact match while looking for " + "a link to the latest debian image file." + ) + image_file_name = image_file_links[0]['href'] + image_file_url = releases_url + image_file_name + + return { + "image_file": { + "url": image_file_url, + "name": image_file_name, + }, + "hash_file": { + "url": hash_file_url, + "name": hash_file_name, + }, + "signature_file": { + "url": signature_file_url, + "name": signature_file_name, + }, + } diff --git a/requirements.txt b/requirements.txt index dbf02cc..887d986 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,10 +1,10 @@ beautifulsoup4==4.11.1 -certifi==2021.10.8 -charset-normalizer==2.0.12 -colorama==0.4.4 +bs4==0.0.1 +certifi==2022.6.15 +charset-normalizer==2.1.0 +colorama==0.4.5 idna==3.3 -python-gnupg==0.4.8 -requests==2.27.1 +requests==2.28.1 soupsieve==2.3.2.post1 tqdm==4.64.0 -urllib3==1.26.9 +urllib3==1.26.11 diff --git a/udib.py b/udib.py new file mode 100755 index 0000000..e8d2045 --- /dev/null +++ b/udib.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python3 +"""Main entry point for the interactive udib CLI tool.""" + +from pathlib import Path +from sys import exit +from tempfile import TemporaryDirectory + +from cli.clibella import Printer +from cli.parser import get_argument_parser +from core.utils import download_and_verify_debian_iso +from iso.injection import inject_preseed_file_into_iso +from net.download import download_file +from net.scrape import get_debian_preseed_file_urls, get_debian_iso_urls + + +def main(): + + # create a CLI printer + p = Printer() + + # create an argument parser and read arguments + parser = get_argument_parser() + args = parser.parse_args() + + # parse and verify output file if sepcified + if args.path_to_output_file: + path_to_output_file = Path(args.path_to_output_file) + if "~" in str(path_to_output_file): + path_to_output_file = path_to_output_file.expanduser() + path_to_output_file = path_to_output_file.resolve() + + if path_to_output_file.exists(): + p.error(f"Output file already exists: '{path_to_output_file}'.") + exit(1) + else: + path_to_output_file = None + + # parse and verify output dir if sepcified + if args.path_to_output_dir: + path_to_output_dir = Path(args.path_to_output_dir) + if "~" in str(path_to_output_dir): + path_to_output_dir = path_to_output_dir.expanduser() + path_to_output_dir = path_to_output_dir.resolve() + + if not path_to_output_dir.is_dir(): + p.error(f"No such directory: '{path_to_output_dir}'.") + exit(1) + else: + path_to_output_dir = None + + if args.subparser_name == "get": + if args.WHAT == "preseed-file-basic": + # download the basic example preseedfile + + if not path_to_output_file: + output_file_name = get_debian_preseed_file_urls()["basic"]["name"] + if path_to_output_dir: + path_to_output_file = path_to_output_dir / output_file_name + else: + path_to_output_file = Path.cwd() / output_file_name + + p.info("Retrieving basic preseed example file...") + download_file( + path_to_output_file, + get_debian_preseed_file_urls()["basic"]["url"], + show_progress=False, + printer=p, + ) + p.success( + f"Basic preseed example file was saved to '{path_to_output_file}'." + ) + exit(0) + + elif args.WHAT == "preseed-file-full": + # download the full example preseedfile + + if not path_to_output_file: + output_file_name = get_debian_preseed_file_urls()["full"]["name"] + if path_to_output_dir: + path_to_output_file = path_to_output_dir / output_file_name + else: + path_to_output_file = Path.cwd() / output_file_name + + p.info("Retrieving full preseed example file...") + download_file( + path_to_output_file, + get_debian_preseed_file_urls()["full"]["url"], + show_progress=False, + printer=p, + ) + p.success( + f"Full preseed example file was saved to '{path_to_output_file}'." + ) + exit(0) + + elif args.WHAT == "iso": + # download and verify installation image + p.info("Downloading latest Debian stable x86-64 netinst ISO...") + + if not path_to_output_file: + output_file_name = get_debian_iso_urls()["image_file"]["name"] + if path_to_output_dir: + path_to_output_file = path_to_output_dir / output_file_name + else: + path_to_output_file = Path.cwd() / output_file_name + + download_and_verify_debian_iso(path_to_output_file, printer=p) + p.success(f"Debian ISO saved to '{path_to_output_file}'.") + exit(0) + + elif args.subparser_name == "inject": + image_file_name = Path(get_debian_iso_urls()["image_file"]["name"]) + if not path_to_output_file: + output_file_name = image_file_name.stem + "-preseeded" + image_file_name.suffix + if path_to_output_dir: + path_to_output_file = path_to_output_dir / output_file_name + else: + path_to_output_file = Path.cwd() / output_file_name + + # verify preseed file path + path_to_preseed_file = Path(args.PRESEEDFILE) + if "~" in str(path_to_preseed_file): + path_to_preseed_file = Path(path_to_preseed_file).expanduser() + path_to_preseed_file = Path(path_to_preseed_file).resolve() + if not path_to_preseed_file.is_file(): + p.error(f"No such file: '{path_to_preseed_file}'.") + exit(1) + + # verify image file path if set by user or download fresh iso if unset + temp_iso_dir = None + if args.path_to_image_file: + path_to_image_file = Path(args.path_to_image_file) + if "~" in str(path_to_image_file): + path_to_image_file = Path(path_to_image_file).expanduser() + path_to_image_file = Path(path_to_image_file).resolve() + if not path_to_image_file.is_file(): + p.error(f"No such file: '{path_to_image_file}'.") + exit(1) + else: + # download a Debian ISO to a temporary directory + p.info("Downloading the latest Debian x86-64 netinst image...") + temp_iso_dir = TemporaryDirectory() + path_to_iso_dir = Path(temp_iso_dir.name) + path_to_image_file = path_to_iso_dir/image_file_name + download_and_verify_debian_iso(path_to_image_file, printer=p) + + # inject the preseed file + inject_preseed_file_into_iso( + path_to_output_file, + path_to_image_file, + path_to_preseed_file, + printer=p, + ) + + # clear out temporary directory if one was created earlier + if temp_iso_dir: + temp_iso_dir.cleanup() + + exit(0) + +if __name__ == '__main__': + main() diff --git a/udib/__init__.py b/udib/__init__.py deleted file mode 100644 index e5a0d9b..0000000 --- a/udib/__init__.py +++ /dev/null @@ -1 +0,0 @@ -#!/usr/bin/env python3 diff --git a/udib/gpgverify.py b/udib/gpgverify.py deleted file mode 100644 index f161332..0000000 --- a/udib/gpgverify.py +++ /dev/null @@ -1,130 +0,0 @@ -"""Library for verification of GPG signatures.""" - -from pathlib import Path - -import gnupg - -import userinput -import clibella - - -p = clibella.Printer() - - -def gpg_signature_is_valid( - path_to_signature_file, - path_to_data_file, - fallback_keyserver_name -): - """Validates a PGP signature for a data file. - - The detached signature must be provided as plaintext (UTF8) in the - specified signature file. - If the discovered signing key is unknown to gpg on this system for the - invoking user, and if 'fallback_keyserver_name' is not None, an attempt is - made to import the key from the specified keyserver using its ID after - prompting the user for permission to import the key. - - Parameters - ---------- - path_to_signature_file : str or pathlike object - Path to a detached PGP signature stored in a standalone file. - Example value: "/path/to/SHA256SUMS.sig" - path_to_data_file : str or pathlike object - Path to a signed data file, for which the signature is to be verified. - Example value: "/path/to/SHA256SUMS" - fallback_keyserver_name : str - FQDN of a keyserver from which to import unknown public keys. - Example value: "keyring.debian.org" - - Returns - ------- - True : bool - If the signature is valid. - False : bool - If the signature is invalid or could not be validated. - """ - - path_to_signature_file = Path(path_to_signature_file) - path_to_data_file = Path(path_to_data_file) - - gpg = gnupg.GPG() - gpg.encoding = "utf-8" - - p.info("Validating signature...") - with open(path_to_signature_file, "rb") as signature_file: - verification = gpg.verify_file( - signature_file, - path_to_data_file, - close_file=False - ) - - # check if a key and fingerprint were found in the signature file - if verification.key_id is None or verification.fingerprint is None: - raise ValueError( - f"Not a valid PGP signature file: '{path_to_signature_file}'." - ) - else: - p.info(f"Signature mentions a key with ID " - f"{verification.key_id} and fingerprint " - f"{verification.fingerprint}." - ) - - if verification.valid: - p.ok(f"GPG signature is valid with trustlevel " - f"'{verification.trust_level}'." - ) - return True - - # this case commonly occurrs when the GPG key has not been imported - if verification.status == "no public key": - p.warning("Could not find the public GPG key locally!") - - # prompt user until answer is unambiguous - key_will_be_imported = None - while key_will_be_imported is None: - key_will_be_imported = userinput.prompt_yes_or_no( - f"[PROMPT] Do you want to import the GPG key from " - f"'{fallback_keyserver_name}'?" - ) - - if key_will_be_imported is None: - p.error("Unrecognized input. Please try again.") - - if not key_will_be_imported: - p.warning("Aborting without importing key.") - return False - - # import missing key - p.info("Importing key...") - import_result = gpg.recv_keys( - fallback_keyserver_name, verification.key_id - ) - - if import_result.count < 1: - p.error("Failed to import key.") - return False - - # display some of gpg's output - gpg_output = import_result.stderr.split('\n') - for line in gpg_output: - if line.startswith("gpg: "): - p.info(f"{line}") - - # validate signature again - p.info("Validating signature...") - with open(path_to_signature_file, "rb") as signature_file: - verification = gpg.verify_file( - signature_file, - path_to_data_file, - close_file=False - ) - - if verification.valid: - p.ok(f"GPG signature is valid with trustlevel " - f"'{verification.trust_level}'." - ) - return True - else: - p.error("GPG signature is not valid!!!") - return False diff --git a/udib/udib.py b/udib/udib.py deleted file mode 100755 index 0208160..0000000 --- a/udib/udib.py +++ /dev/null @@ -1,333 +0,0 @@ -#!/usr/bin/env python3 - -"""Main entry point for the interactive udib CLI tool.""" - -from pathlib import Path -import argparse -import os -import shutil -import subprocess -import sys -import tempfile - -import clibella -import modiso -import userinput -import webdownload - - -p = clibella.Printer() - - -def _assert_system_dependencies_installed(): - """Asserts that all system dependencies are installed. - - System dependencies are those programs whose names are listed in - the global 'system_programs_required' variable. - - Raises - ------ - RuntimeError - Raised when a required program is not accessible in the system - $PATH. - - """ - - # contains the names of all unix program dependencies which must be - # installed on the local system and available in the local system's $PATH - system_programs_required = [ - "cpio", - "gpg", - "xorriso", - ] - - for program_name in system_programs_required: - try: - subprocess.run( - ["command", "-v", program_name], - shell=True, - check=True) - except subprocess.CalledProcessError: - raise RuntimeError( - f"Program not installed or not in $PATH: " - f"'{program_name}'.") - - -def _get_argument_parser(): - """Instantiates and configures an ArgumentParser and returns it. - - Examples - -------- - parser = _get_argument_parser() - args = parser.parse_args() - - """ - - # FIXME capture ISO filesystem name - - parser = argparse.ArgumentParser() - - parser.add_argument( - "-o", - "--output-file", - help="Path to which the resulting file is written.", - type=str, - dest="path_to_output_file", - action="store") - - subparsers = parser.add_subparsers(required=True) - - parser_get = subparsers.add_parser("get") - parser_get.add_argument( - "WHAT", - type=str, - action="store", - help="What to retrieve (the latest installation image or the latest " \ - "preseed file).", - choices=["image", "preseed-file"]) - - parser_insert = subparsers.add_parser("insert") - parser_insert.add_argument( - "FILES", - nargs="+", - type=str, - help="A list of files to insert into the root of an installation " \ - "image.", - action="store") - parser_insert.add_argument( - "-i", - "--existing-image", - help="Insert into an existing debian image file, instead of " \ - "downloading a new one.", - type=str, - dest="path_to_existing_image", - action="store") - - return parser - - -def _chmod_recursively(input_path, mode): - """Recursively changes file permissions on the specified path. - - The file and directory permissions for the specified path itself, - as well as any files and directories anywhere below it are set to - the specified mode. - Symlinks are ignored. - - Parameters - ---------- - input_path : str or pathlike object - Path of which to change the permissions recursively. - mode : int - Permission mode as accepted by os.chmod() - - Raises - ------ - ValueError - Raised if the specified path points to a nonexisting - filesystem node, or if the specified permission mode is - invalid. - TypeError - Raised if mode is not an integer. - - Examples - -------- - _chmod_recursively("/tmp/mydir", 0o755) - - """ - - input_path = Path(input_path) - - if not input_path.exists(): - raise ValueError(f"Path does not exist: '{input_path}'.") - - if not isinstance(mode, int): - raise TypeError("Expected type 'int' for parameter 'mode'.") - if mode not in range(0, 0o1000): - raise ValueError("Invalid mode.") - - if not input_path.is_symlink(): - input_path.chmod(mode) - for child in input_path.iterdir(): - if not child.is_symlink(): - child.chmod(mode) - if child.is_dir(): - _chmod_recursively(child, mode) - - -def main(): - - # FIXME capture ISO filesystem name - iso_filesystem_name = "Debian UDIB" - - # FIXME adjust to new parser namespace - - parser = _get_argument_parser() - args = parser.parse_args() - - # check if all system dependencies are installed - try: - _assert_system_dependencies_installed() - except RuntimeError as e: - p.error(e) - sys.exit(1) - - # determine where to output files - # default to ouputting files to the current directory - # using their original name if not specified by '-o' argument - path_to_output_dir = Path.cwd() - path_to_output_file = None - - # check if path_to_output_file is valid and adjust path_to_output_dir if - # '-o' was set - if args.path_to_output_file: - path_to_output_file = Path(args.path_to_output_file).resolve() - - if path_to_output_file.exists() and not path_to_output_file.is_file(): - p.error(f"Specified output location is not a file: " - f"'{path_to_output_file}'.") - sys.exit(1) - - path_to_output_dir = path_to_output_file.parent - - if args.get_image: - # download latest image and exit - with tempfile.TemporaryDirectory() as tmp_dir: - # save image to a temporary directory - path_to_image_file = webdownload.debian_obtain_image( - tmp_dir) - - if path_to_output_file: - # rename file if '-o' flag was specified - path_to_image_file = path_to_image_file.rename( - path_to_output_file.name) - else: - # else determine the intended final path - path_to_output_file = path_to_output_dir/path_to_image_file.name - - # prompt to confirm file removal if output file already exists - if path_to_output_file.is_file(): - prompt = f"File '{path_to_output_file}' already exists. " \ - f"Overwrite it?" - if userinput.prompt_yes_or_no(prompt, ask_until_valid=True): - os.remove(path_to_output_file) - else: - p.failure("Did not obtain a new image file.") - sys.exit(1) - - # move file from temporary directory to intended path - path_to_image_file.rename(path_to_output_file) - - sys.exit(0) - - elif args.get_preseed_file: - # download latest preseed file and exit - preseed_file_name = "example-preseed.txt" - preseed_file_url = "https://www.debian.org/"\ - "releases/stable/" + preseed_file_name - - if not path_to_output_file: - path_to_output_file = path_to_output_dir/preseed_file_name - - # prompt to confirm file removal if output file already exists - if path_to_output_file.is_file(): - prompt = f"File '{path_to_output_file}' already exists. " \ - f"Overwrite it?" - if userinput.prompt_yes_or_no(prompt, ask_until_valid=True): - os.remove(path_to_output_file) - else: - p.failure("Did not obtain a new preseed file.") - sys.exit(1) - - webdownload.download_file(path_to_output_file, preseed_file_url) - - sys.exit(0) - - else: - # modify image file using specified preseed file - path_to_preseed_file = Path(args.existing_preseed_file) - if not path_to_preseed_file.is_file(): - p.error(f"No such file: '{path_to_preseed_file}'.") - sys.exit(1) - - if args.path_to_existing_image: - # user has specified an existing image file - path_to_image_file = Path(args.path_to_existing_image) - # sanity check - if path_to_image_file.suffix not in [".iso", ".img"]: - p.warning(f"Specified image file does not appear to be an ISO " - f"or IMG file: '{path_to_image_file}'!") - if not userinput.prompt_yes_or_no("Proceed anyways?"): - p.failure("Did not create a new image file.") - sys.exit(1) - else: - # download a fresh image file to a temporary directory - # remember to delete it later! - path_to_image_file = webdownload.debian_obtain_image( - tempfile.mkdtemp()) - - # extract image file to a temporary directory - path_to_extracted_iso_dir = Path(tempfile.mkdtemp()) - p.info("Extracting ISO contents...") - modiso.extract_iso( - path_to_extracted_iso_dir, - path_to_image_file) - - # extract image MBR to a temporary directory - p.info("Extracting master boot record...") - path_to_mbr = Path(tempfile.mkdtemp())/"mbr.bin" - modiso.extract_mbr_from_iso( - path_to_mbr, - path_to_image_file) - - # append preseed file to extracted initrd - p.info("Appending preseed file...") - # make a temporary copy called 'preseed.cfg' - with tempfile.TemporaryDirectory() as tmpdir: - path_to_renamed_preseed_file = Path(tmpdir).resolve()/"preseed.cfg" - shutil.copy(path_to_preseed_file, path_to_renamed_preseed_file) - modiso.append_file_contents_to_initrd_archive( - path_to_extracted_iso_dir/"install.amd"/"initrd.gz", - path_to_renamed_preseed_file) - - # regenerate md5sum.txt - p.info("Regenerating MD5 checksum...") - modiso.regenerate_iso_md5sums_file(path_to_extracted_iso_dir) - - if not path_to_output_file: - # use original filename with "-udib" appended before the extension - path_to_output_file = (path_to_output_dir - / (path_to_image_file.with_suffix("").name - + "-udib" - + path_to_image_file.suffix)) - - # check if outputfile already exists - if path_to_output_file.is_file(): - prompt = f"File '{path_to_output_file}' already exists. " \ - f"Overwrite it?" - if userinput.prompt_yes_or_no(prompt, ask_until_valid=True): - os.remove(path_to_output_file) - else: - p.failure("Did not create a new ISO file.") - sys.exit(1) - - # repack ISO file - p.info("Repacking ISO file...") - modiso.repack_iso( - path_to_output_file, - path_to_mbr, - path_to_extracted_iso_dir, - iso_filesystem_name) - - # remove temporary directories - p.info("Cleaning up...") - _chmod_recursively(path_to_extracted_iso_dir, 0o755) - shutil.rmtree(path_to_extracted_iso_dir) - _chmod_recursively(path_to_mbr.parent, 0o755) - shutil.rmtree(path_to_mbr.parent) - - p.success(f"Wrote the modified ISO to '{path_to_output_file}'.") - - -if __name__ == "__main__": - main() diff --git a/udib/userinput.py b/udib/userinput.py deleted file mode 100644 index f5d1963..0000000 --- a/udib/userinput.py +++ /dev/null @@ -1,50 +0,0 @@ -"""Library for consistent interactive user input capturing.""" - -import re - -import clibella - - -p = clibella.Printer() - - -def prompt_yes_or_no(question, ask_until_valid=False): - """Prompts the user for an answer to the specified yes/no question. - - If 'ask_until_valid' is True, keeps repeating the prompt until the user - provides recognizable input to answer the question. - - Parameters - ---------- - question : str - The question to ask the user while prompting. The string is suffixed - with " (Yes/No) ". - ask_until_valid : bool - If this is set to True and the user provides an ambiguous answer, keeps - prompting indefinitely until an unambiguous answer is read. - - Returns - ------- - True : bool - If the user has answered 'yes'. - False : bool - If the user has answered 'no'. - None : None - If 'ask_until_valid' is False and the user has provided an ambiguous - response to the prompt. - """ - - user_input_is_valid = False - - regex_yes = re.compile(r"^(y)$|^(Y)$|^(YES)$|^(Yes)$|^(yes)$") - regex_no = re.compile(r"^(n)$|^(N)$|^(NO)$|^(No)$|^(no)$") - - while(not user_input_is_valid): - user_input = p.input(f"{question} (Yes/No): ") - - if (regex_yes.match(user_input)): - return True - elif (regex_no.match(user_input)): - return False - elif (not ask_until_valid): - return None diff --git a/udib/webdownload.py b/udib/webdownload.py deleted file mode 100644 index 9704c25..0000000 --- a/udib/webdownload.py +++ /dev/null @@ -1,197 +0,0 @@ -"""Library for downloading files from the web. - -Specifically, debian installation images and any accompanying files -required for verification of authenticity and integrity of the -obtained images. - -""" - -import os -import re -import subprocess -from pathlib import Path - -import requests -from bs4 import BeautifulSoup -from tqdm import tqdm - -import clibella -import gpgverify - - -p = clibella.Printer() - - -def download_file(path_to_output_file, url_to_file, show_progress=False): - """Downloads the file at the input URL to the specified path. - - The file is downloaded via HTTP/HTTPS and saved to the specified path. - Optionally, displays a nice status bar. - - Parameters - ---------- - path_to_output_file : str or pathlike object - Path to a file as which the downloaded file is saved. - url_to_file : str - URL to the file to be downloaded. - show_progress : bool - When True, a progress bar is displayed on StdOut indicating the - progress of the download. - """ - - path_to_output_file = Path(path_to_output_file).resolve() - if not path_to_output_file.parent.is_dir(): - raise FileNotFoundError( - f"No such directory: '{path_to_output_file.parent}'." - ) - if path_to_output_file.exists(): - raise FileExistsError( - f"File already exists: '{path_to_output_file}'" - ) - - output_file_name = path_to_output_file.name - with open(path_to_output_file, "wb") as output_file: - p.info(f"Downloading '{output_file_name}'...") - file_response = requests.get(url_to_file, stream=True) - total_length = file_response.headers.get('content-length') - - if total_length is None: # no content length header - output_file.write(file_response.content) - else: - if (show_progress): - total_length = int(total_length) - progress_bar = tqdm( - total=total_length, - unit="B", - unit_scale=True, - unit_divisor=1024 - ) - - for data in file_response.iter_content(chunk_size=4096): - output_file.write(data) - if (show_progress): - progress_bar.update(len(data)) - - if (show_progress): - progress_bar.close() - - p.ok(f"Received '{output_file_name}'.") - - -def debian_obtain_image(path_to_output_dir): - """Downloads the latest debian installation image and its hashes. - - The image file, the SHA512SUMS file it is listed in, as well as the GPG - signature for the SHA512SUMS file are downloaded from the debian.org HTTPS - mirrors and written into the specified output directory. - The obtained image is for the FOSS-only, stable x64 build. - - Once the image file is downloaded, the GPG signature of the hash is - validated. Then, the hash of the image file is verified. If either check - fails, an exception is raised. - - If the verification suceeds, the hash file and GPG signature file are - removed again. - - Parameters - ---------- - path_to_output_dir : str or pathlike object - Path to the directory to which all downloaded files will be saved. - - Returns - ------- - path_to_image_file : str - Full path to the obtained and validated image file. - """ - - # FIXME download files to a temporary directory - p.info("Obtaining and verifying the latest Debian stable image...") - - path_to_output_dir = Path(path_to_output_dir) - - if not path_to_output_dir.is_dir(): - raise ValueError(f"No such directory: '{path_to_output_dir}'") - - releases_url = "https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/" - releases_page = requests.get(releases_url) - soup = BeautifulSoup(releases_page.content, "html.parser") - - hash_file_name = "SHA512SUMS" - hash_file_url = releases_url + hash_file_name - signature_file_name = "SHA512SUMS.sign" - signature_file_url = releases_url + signature_file_name - - # find the URL to the latest stable x64 image - image_file_links = soup.find_all( - name="a", - string=re.compile(r"debian-[0-9.]*-amd64-netinst.iso")) - if len(image_file_links) != 1: - raise RuntimeError("Failed to find an exact match while looking for " - "a link to the latest debian image file." - ) - image_file_name = image_file_links[0]['href'] - image_file_url = releases_url + image_file_name - - # download the SHA512SUMS file - download_file(path_to_output_dir/hash_file_name, hash_file_url) - # download the GPG signature file - download_file(path_to_output_dir/signature_file_name, signature_file_url) - - # verify GPG signature of hash file - if not gpgverify.gpg_signature_is_valid( - path_to_output_dir/signature_file_name, - path_to_output_dir/hash_file_name, - "keyring.debian.org" - ): - raise RuntimeError("GPG signature verification failed!") - - # download the image file - download_file(path_to_output_dir/image_file_name, image_file_url, True) - - # remove unwanted lines from hash file - hash_file = open(path_to_output_dir/hash_file_name, "r") - hash_file_lines = hash_file.readlines() - hash_file_lines_to_keep = [] - for line in hash_file_lines: - if image_file_name in line: - hash_file_lines_to_keep.append(line) - hash_file.close() - if len(hash_file_lines_to_keep) != 1: - raise RuntimeError("Unexpected error while truncating hash file.") - os.remove(path_to_output_dir/hash_file_name) - with open(path_to_output_dir/hash_file_name, "w") as hash_file: - hash_file.writelines(hash_file_lines_to_keep) - - # validate SHA512 checksum - p.info("Validating file integrity...") - hash_check_result = subprocess.run( - ["sha512sum", "--check", path_to_output_dir/hash_file_name], - capture_output=True, - cwd=path_to_output_dir - ) - - stdout_lines = hash_check_result.stdout.decode("utf-8").split('\n') - stderr_lines = hash_check_result.stderr.decode("utf-8").split('\n') - - if len(stdout_lines) > 0: - for line in stdout_lines: - if len(line) > 0: - p.info(f"{line}") - - if hash_check_result.returncode != 0: - if len(stderr_lines) > 0: - for line in stderr_lines: - if len(line) > 0: - p.error(f"{line}") - raise RuntimeError("SHA512 validation failed.") - - p.ok("File integrity checks passed.") - - # clean up obsolete files - p.info("Cleaning up files...") - os.remove(path_to_output_dir/hash_file_name) - os.remove(path_to_output_dir/signature_file_name) - - p.success("Debian image obtained.") - - return path_to_output_dir/image_file_name