lapatatedouce/custom-debian-iso
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Reimplementation (#1)

Browse Source 
* refactor into separate modules

* update README
This commit is contained in:
ulinja
2022-08-01 12:23:41 +02:00
committed by GitHub
parent 5665759667
commit b69fd8cfa9
25 changed files with 1274 additions and 1323 deletions
Download Patch File Download Diff File Expand all files Collapse all files

98
README.md Normal file
View File

@@ -0,0 +1,98 @@
# UDIB
UDIB is the Unattended Debian Installation Builder.
It provides a handy commandline utility for creating preseeded Debian installation ISOs.
Preseeded ISOs allow partially or fully automated Debian installations on bare metal (or anywhere else).
## What's preseeding?
A preseed file is a text file which provides the Debian installer with previously set (preseeded) answers during the installation process.
Preseeding a Debian ISO allows you to heavily reduce the amount of user interaction required during an installation, or even fully automate it.
The preseed file is written by you and then injected into the installation image.
When you start the installation, any answers you have provided to the debian installer as part of your preseed file are automatically applied during the installation.
If you want to know more, you can have a look at Debian's [official guide](https://www.debian.org/releases/stable/amd64/apb.en.html) or at the Debian wiki's [quick overview](https://wiki.debian.org/DebianInstaller/Preseed), both of which explain preseeding much better than I can.
## How does UDIB work?
UDIB's main purpose is the injection of preseed files into existing Debian installation ISOs.
In a nutshell, it does this by extracting the ISO, adding the preseed file to it, and repacking the ISO again.
You could do all of this manually of course by following the [basic](https://wiki.debian.org/DebianInstaller/Preseed/EditIso#Adding_a_Preseed_File_to_the_Initrd) and [advanced](https://wiki.debian.org/RepackBootableISO) guides for ISO repacking on the Debian wiki, but UDIB does all of this for you.
# Dependencies
UDIB has some hard- and software requirements, both regarding the [target machine](#target-machine) for which it can build ISOs, as well as on the [build machine](#build-machine) on which UDIB is run.
## Target Machine
Images built using UDIB can be used to install Debian on systems satisfying these requirements:
- **CPU Architecture:** x86 (64-Bit)
- **Network:** an ethernet connection with internet access
## Build Machine
Using UDIB to create ISOs requires the following software:
- GNU/Linux
- `python3` *(3.10.4 known to work)*
- [required python packages](./requirements.txt) can be installed in a virtual environment
- `xorriso` *(1.5.4 known to work)*
- **Debian (bullseye):** [xorriso](https://packages.debian.org/bullseye/xorriso)
- **Arch Linux:** [extra/libisoburn](extra/libisoburn)
- `gpg` *(2.2.32 known to work)*
- GNU `cpio`
- preinstalled on most distributions
Internet access is (obviously) required if you want to fetch any files using UDIB.
# Quick Start Guide
This short guide explains how to build a Debian ISO with a customized and automated installation:
1. make sure you have all the [required software](#build-machine) installed
2. clone this repo and `cd` into your local copy
3. (optional) create and activate a virtual environment: `python3 -m venv .venv && . .venv/bin/activate`
4. install the required python packages: `pip install --user -r requirements.txt`
5. get an example preseed file: `./udib.py -o my-preseed.cfg get preseed-file-basic`
6. customize your installation by editing `my-preseed.cfg` (the comments are pretty self-explanatory)
7. create a Debian ISO with your preseed file: `./udib.py -o my-image.iso inject my-preseed.cfg`
8. boot from your newly created ISO `my-image.iso` on your target machine (or in a VM)
9. in the Debian installer menu, navigate to *Advanced options > Automated install*
10. drink some coffee
11. return to your new Debian system
Depending on how many answers you provided in the preseed file, the installation may require some manual interaction.
Preseed files are very powerful, and if you need more customization you can have a deeper look into [how they work](#whats-preseeding).
You can also use UDIB to get the full preseed example file: `./udib.py get preseed-file-full` and use that as a starting point.
# Detailed usage guide
You can view help at the commandline using `./udib.py --help` for general options and `./udib.py COMMAND --help` for help with a specific subcommand.
The name and destination of files produced by `udib.py` can be specified with the `--output-file` option.
Alternatively, you can use the `--output-dir` option to specify the directory where produced files will be saved, without having to name them explicitly (default names will be used).
## Retrieving example preseed files or vanilla ISOs
As a starting point for creating your own preseeded ISO, you can retrieve one of Debian's example preseed files or an unmodified Debian ISO using UDIB:
```
udib.py [--output-file FILE | --output-dir DIR] get WHAT
```
`WHAT` must be a specific string and can be either one of:
- `preseed-file-basic` to download Debian's basic example preseed file (sufficient in most cases)
- `preseed-file-full` to download Debian's full example preseed file (has a LOT of customization options)
- `iso` to download the latest, unmodified Debian stable x86-64 netinst ISO
## Creating a preseeded ISO
To inject an existing preseed file into an ISO, you can run the following command:
```
udib.py [--output-file FILE | --output-dir DIR] inject PRESEEDFILE [--image-file IMAGEFILE]
```
where `PRESEEDFILE` is the path to your preseed file.
If you don't specify an `--image-file`, UDIB will download the latest Debian x86-64 netinst ISO and inject your `PRESEEDFILE` into it.

60
README.org
View File

@@ -1,60 +0,0 @@
* UDIB
=UDIB= is the /unattended debian installation (image) builder/.
As the name would suggest, it is a tool which builds a debian installation image which, if used to boot from, installs the *latest stable release* of Debian Linux to the machine without any user interaction required whatsoever.
Configuration of the installed system is done by editing preseed files.
** Usage
Obtaining a preseed example file:
#+begin_src shell
./udib.py get preseed-file
#+end_src
** Prerequisites
*** Target machine
Any target machine on which you wish to install Debian using images built using =UDIB= must fulfil the following requirements:
- *Architecture:* An x64-amd CPU
- *Network access:* an ethernet connection with internet access (/wifi does not work/)
*** Build machine
=UDIB= has the following dependencies, installed on and in the ~$PATH~ of the machine on which you build your image:
- =python3= /(3.10.4 known to work)/
- =xorriso= /(1.5.4 known to work)/
- part of [[https://packages.debian.org/bullseye/xorriso][xorriso]] on Debian bullseye
- part of [[https://www.archlinux.org/packages/extra/x86_64/libisoburn/][extra/libisoburn]] on Arch Linux
- =gpg= /(2.2.32 known to work)/
- GNU =cpio=
- preinstalled on most distributions
Internet access is required so that =UDIB= can download the Debian installation image.
** How it works
=UDIB= does the following:
1. download the latest debian stable installation ISO
2. verify integrity and authenticity of the obtained ISO
3. extract the ISO
4. inject the preseed configuration into the ISO's initrd
5. repack the modified ISO
*** Preseeding
The Debian project provides a well-written and detailed [[https://www.debian.org/releases/stable/amd64/apb.en.html][official guide]] on how preseeding works, and how to implement it. The Debian wiki offers a [[https://wiki.debian.org/DebianInstaller/Preseed][quick overview]] of preseeding as well.
Example preseeding configurations are provided as well. The [[https://www.debian.org/releases/stable/example-preseed.txt][basic example]] is what =UDIB= is based on, but much more detailed preseeding is also possible, as seen in the [[https://preseed.debian.net/debian-preseed/bullseye/amd64-main-full.txt][full example]].
*** ISO repacking
The Debian wiki provides both a [[https://wiki.debian.org/DebianInstaller/Preseed/EditIso#Adding_a_Preseed_File_to_the_Initrd][basic guide]] and a [[https://wiki.debian.org/RepackBootableISO][detailed guide]] on repacking debian installation images with the preseed file (and other files) contained within.

0
__init__.py Normal file
View File

0
cli/__init__.py Normal file
View File

43
udib/clibella.py → cli/clibella.py
View File

@@ -1,6 +1,7 @@
"""Library for consistent and visually appealing terminal output."""
import sys
from re import compile
import colorama
from colorama import Fore, Style
@@ -335,3 +336,45 @@ class Printer:
color_enabled=color_enabled,
**kwargs
)
def prompt_yes_or_no(self, question: str, ask_until_valid: bool = False):
"""Prompts the user for an answer to the specified yes/no question.
If 'ask_until_valid' is True, keeps repeating the prompt until the user
provides recognizable input to answer the question.
Parameters
----------
question : str
The question to ask the user while prompting. The string is suffixed
with " (Yes/No) ".
ask_until_valid : bool
If this is set to True and the user provides an ambiguous answer, keeps
prompting indefinitely until an unambiguous answer is read.
Returns
-------
True : bool
If the user has answered 'yes'.
False : bool
If the user has answered 'no'.
None : None
If 'ask_until_valid' is False and the user has provided an ambiguous
response to the prompt.
"""
user_input_is_valid = False
regex_yes = compile(r"^(y)$|^(Y)$|^(YES)$|^(Yes)$|^(yes)$")
regex_no = compile(r"^(n)$|^(N)$|^(NO)$|^(No)$|^(no)$")
while(not user_input_is_valid):
user_input = self.input(f"{question} (Yes/No): ")
if (regex_yes.match(user_input)):
return True
elif (regex_no.match(user_input)):
return False
elif (not ask_until_valid):
return None

81
cli/parser.py Normal file
View File

@@ -0,0 +1,81 @@
"""Commandline argument parsing utilities."""
from argparse import ArgumentParser
def get_argument_parser():
"""Sets up an argparse ArgumentParser and returns it."""
mainparser = ArgumentParser(
description="Debian ISO preseeding tool.",
)
# add mutually exclusive optional arguments to top-level parser
mainparser_group = mainparser.add_mutually_exclusive_group()
mainparser_group.add_argument(
"-o",
"--output-file",
action='store',
type=str,
dest='path_to_output_file',
metavar='OUTPUTFILE',
help="File as which the retrieved/generated file will be saved",
)
mainparser_group.add_argument(
"-O",
"--output-dir",
action='store',
type=str,
dest='path_to_output_dir',
metavar='OUTPUTDIR',
help="Directory into which the retrieved/generated file will be written",
)
# register subparsers for the 'get' and 'inject' subcommands
subparsers = mainparser.add_subparsers(
required=True,
title="Subcommands",
description="A choice of actions you want udib to take",
help="You must specify one of these",
dest="subparser_name",
)
subparser_get = subparsers.add_parser(
"get",
description="Retrieve an unmodified Debian ISO or example preseed file",
)
subparser_inject = subparsers.add_parser(
"inject",
description="Inject a preseed file into a Debian ISO",
)
# register arguments for the 'get' subcommand
subparser_get.add_argument(
"WHAT",
choices=['preseed-file-basic', 'preseed-file-full', 'iso'],
action='store',
type=str,
metavar='WHAT',
help="The type of file you want UDIB to retrieve. "
"Valid options are: 'preseed-file-basic', 'preseed-file-full' "
"or 'iso'.",
)
# register arguments for the 'inject' subcommand
subparser_inject.add_argument(
"PRESEEDFILE",
action='store',
type=str,
metavar='PRESEEDFILE',
help="Path to the preseed file you want to inject",
)
subparser_inject.add_argument(
"-i",
"--image-file",
action='store',
type=str,
dest='path_to_image_file',
metavar='IMAGEFILE',
help="Path to the ISO you want to modify",
)
return mainparser

0
core/__init__.py Normal file
View File

239
core/utils.py Normal file
View File

@@ -0,0 +1,239 @@
"""A collection of general utilities, not specific to any module."""
from pathlib import Path
from tempfile import TemporaryDirectory
from sys import exit
from os import remove, rename
from subprocess import run, STDOUT, PIPE
from cli.clibella import Printer
from net.download import download_file
from net.scrape import get_debian_iso_urls
from gpg.verify import assert_detached_signature_is_valid
from gpg.exceptions import VerificationFailedError
from gpg.keystore import debian_signing_key_is_imported, import_debian_signing_key
from crypt import crypt, METHOD_SHA512
from getpass import getpass
def hash_user_password(printer=None):
"""Prompts for a password and prints the resulting hash.
The resulting hash can be added to the preseed file to set a user password:
d-i passwd/root-password-crypted PASSWORDHASH
"""
if printer is None:
p = Printer()
else:
if not isinstance(printer, Printer):
raise TypeError(f"Expected a {type(Printer)} object.")
password = getpass("Enter a password: ")
password_confirmed = getpass("Enter the password again: ")
if not password_confirmed == password:
p.failure("Passwords did not match")
return
p.info("Password hash:")
p.info(crypt(password, crypt.METHOD_SHA512))
def find_all_files_under(parent_dir):
"""Recursively finds all files anywhere under the specified directory.
Returns a list of absolute Path objects. Symlinks are ignored.
Parameters
----------
parent_dir : str or pathlike object
The directory under which to recursively find files.
Raises
------
NotADirectoryError
Raised if the specified parent directory is not a directory.
Examples
--------
config_files = find_all_files_under("~/.config")
"""
if "~" in str(parent_dir):
parent_dir = Path(parent_dir).expanduser()
parent_dir = Path(parent_dir).resolve()
if not parent_dir.is_dir():
raise NotADirectoryError(f"No such directory: '{parent_dir}'.")
files = []
for subpath in parent_dir.iterdir():
if subpath.is_file():
files.append(subpath.resolve())
elif not subpath.is_symlink() and subpath.is_dir():
files += find_all_files_under(subpath)
return files
def trim_text_file(path_to_input_file, substring):
"""Removes all lines not containing the substring from the input file.
The input file is overwritten.
If the substring is the empty string, no action is taken.
If the substring contains a newline, no action is taken.
If the substring does not match any line of the file, the resulting file
will be left empty.
Parameters
----------
path_to_input_file : str or pathlike object
The file to trim.
substring : str
The string to filter for.
"""
if not isinstance(substring, str):
raise TypeError("Expected a string.")
if "\n" in substring or len(substring) == 0:
return
if "~" in str(path_to_input_file):
path_to_input_file = Path(path_to_input_file).expanduser()
path_to_input_file = Path(path_to_input_file).resolve()
if not path_to_input_file.is_file():
raise FileNotFoundError(f"No such file: '{path_to_input_file}'.")
# remove unwanted lines from hash file
input_file = open(path_to_input_file, "r")
input_file_lines = input_file.readlines()
input_file_lines_to_keep = []
for line in input_file_lines:
if substring in line:
input_file_lines_to_keep.append(line)
input_file.close()
remove(path_to_input_file)
with open(path_to_input_file, "w") as input_file:
input_file.writelines(input_file_lines_to_keep)
def file_is_empty(path_to_input_file):
"""Checks whether the input file is empty or not."""
if "~" in str(path_to_input_file):
path_to_input_file = Path(path_to_input_file).expanduser()
path_to_input_file = Path(path_to_input_file).resolve()
if not path_to_input_file.is_file():
raise FileNotFoundError(f"No such file: '{path_to_input_file}'.")
with open(path_to_input_file, 'r') as input_file:
# try to read a byte
file_has_content = len(input_file.read(1)) > 0
return not file_has_content
def download_and_verify_debian_iso(path_to_output_file, printer=None):
"""Downloads the latest Debian ISO as the specified output file.
The file's integrity is validated using a SHA512 checksum.
The PGP signature of the SHA512SUMS file is checked using gpg.
Attributes
----------
path_to_output_file : str or pathlike object
Path to the file as which the downloaded image will be saved.
printer : clibella.Printer
A CLI printer to be used for output.
"""
if "~" in str(path_to_output_file):
path_to_output_file = Path(path_to_output_file).expanduser()
path_to_output_file = Path(path_to_output_file).resolve()
if path_to_output_file.is_file():
raise FileExistsError(
f"Output file '{path_to_output_file}' already exists."
)
if not path_to_output_file.parent.is_dir():
raise NotADirectoryError(
f"No such directory: '{path_to_output_file.parent}'."
)
if printer is None:
printer = Printer()
# create a temporary directory
with TemporaryDirectory() as temp_dir:
# scrape for URLs and filenames
files = get_debian_iso_urls()
# set file paths
path_to_hash_file = Path(temp_dir)/files["hash_file"]["name"]
path_to_signature_file = Path(temp_dir)/files["signature_file"]["name"]
path_to_image_file = Path(temp_dir)/files["image_file"]["name"]
# download hash file and signature, and verify with gpg
download_file(
path_to_hash_file,
files["hash_file"]["url"],
show_progress=False,
printer=printer,
)
download_file(
path_to_signature_file,
files["signature_file"]["url"],
show_progress=False,
printer=printer,
)
# verify the hash file using gpg
printer.info("Verifying hash file using gpg...")
if not debian_signing_key_is_imported():
printer.info("Importing Debian public GPG CD signing key...")
import_debian_signing_key()
else:
printer.info("Found Debian public GPG CD signing key.")
try:
assert_detached_signature_is_valid(
path_to_hash_file,
path_to_signature_file,
)
except VerificationFailedError:
printer.error("PGP signature of the hash file was invalid!")
exit(1)
printer.ok("HASH file PGP authenticity check passed.")
# remove all lines from hash file not containing the image file name
trim_text_file(path_to_hash_file, files["image_file"]["name"])
if file_is_empty(path_to_hash_file):
raise RuntimeError("Failed to locate SHA512 hash sum for image.")
# download image file
download_file(
path_to_image_file,
files["image_file"]["url"],
show_progress=True,
printer=printer,
)
# validate SHA512 checksum
printer.info("Validating ISO file integrity...")
hash_check_result = run(
[
"sha512sum", "--check", path_to_hash_file
],
text=True,
stdout=PIPE,
stderr=STDOUT,
cwd=path_to_image_file.parent,
)
if hash_check_result.returncode != 0:
raise RuntimeError("SHA512 checksum verification of the ISO failed.")
printer.ok("ISO file integrity check passed.")
# move downloaded file to specified destination
rename(path_to_image_file, path_to_output_file)

478
default-preseed.cfg
View File

@@ -1,478 +0,0 @@
#_preseed_V1
#### TODOs:
# configure partman answers
# install sudo and add user to sudo group
# configure sshd
#### Contents of the preconfiguration file (for bullseye)
### Localization
# Preseeding only locale sets language, country and locale.
#d-i debian-installer/locale string en_US
# The values can also be preseeded individually for greater flexibility.
d-i debian-installer/language string en
d-i debian-installer/country string DE
d-i debian-installer/locale string en_GB.UTF-8
# Optionally specify additional locales to be generated.
d-i localechooser/supported-locales multiselect en_US.UTF-8, en_DK.UTF-8, de_DE.UTF-8
# Keyboard selection.
d-i keyboard-configuration/xkb-keymap select de
# d-i keyboard-configuration/toggle select No toggling
### Network configuration
# Disable network configuration entirely. This is useful for cdrom
# installations on non-networked devices where the network questions,
# warning and long timeouts are a nuisance.
#d-i netcfg/enable boolean false
# netcfg will choose an interface that has link if possible. This makes it
# skip displaying a list if there is more than one interface.
d-i netcfg/choose_interface select auto
# To pick a particular interface instead:
#d-i netcfg/choose_interface select eth1
# To set a different link detection timeout (default is 3 seconds).
# Values are interpreted as seconds.
#d-i netcfg/link_wait_timeout string 10
# If you have a slow dhcp server and the installer times out waiting for
# it, this might be useful.
#d-i netcfg/dhcp_timeout string 60
#d-i netcfg/dhcpv6_timeout string 60
# Automatic network configuration is the default.
# If you prefer to configure the network manually, uncomment this line and
# the static network configuration below.
#d-i netcfg/disable_autoconfig boolean true
# If you want the preconfiguration file to work on systems both with and
# without a dhcp server, uncomment these lines and the static network
# configuration below.
#d-i netcfg/dhcp_failed note
#d-i netcfg/dhcp_options select Configure network manually
# Static network configuration.
#
# IPv4 example
#d-i netcfg/get_ipaddress string 192.168.1.42
#d-i netcfg/get_netmask string 255.255.255.0
#d-i netcfg/get_gateway string 192.168.1.1
#d-i netcfg/get_nameservers string 192.168.1.1
#d-i netcfg/confirm_static boolean true
#
# IPv6 example
#d-i netcfg/get_ipaddress string fc00::2
#d-i netcfg/get_netmask string ffff:ffff:ffff:ffff::
#d-i netcfg/get_gateway string fc00::1
#d-i netcfg/get_nameservers string fc00::1
#d-i netcfg/confirm_static boolean true
# Any hostname and domain names assigned from dhcp take precedence over
# values set here. However, setting the values still prevents the questions
# from being shown, even if values come from dhcp.
d-i netcfg/get_hostname string my-debian-machine
d-i netcfg/get_domain string example.com
# If you want to force a hostname, regardless of what either the DHCP
# server returns or what the reverse DNS entry for the IP is, uncomment
# and adjust the following line.
#d-i netcfg/hostname string somehost
# Disable that annoying WEP key dialog.
d-i netcfg/wireless_wep string
# The wacky dhcp hostname that some ISPs use as a password of sorts.
#d-i netcfg/dhcp_hostname string radish
# If non-free firmware is needed for the network or other hardware, you can
# configure the installer to always try to load it, without prompting. Or
# change to false to disable asking.
#d-i hw-detect/load_firmware boolean true
### Network console
# Use the following settings if you wish to make use of the network-console
# component for remote installation over SSH. This only makes sense if you
# intend to perform the remainder of the installation manually.
#d-i anna/choose_modules string network-console
#d-i network-console/authorized_keys_url string http://10.0.0.1/openssh-key
#d-i network-console/password password r00tme
#d-i network-console/password-again password r00tme
### Mirror settings
# Mirror protocol:
# If you select ftp, the mirror/country string does not need to be set.
# Default value for the mirror protocol: http.
#d-i mirror/protocol string ftp
d-i mirror/country string manual
d-i mirror/http/hostname string debian.tu-bs.de
d-i mirror/http/directory string /debian
d-i mirror/http/proxy string
# Suite to install.
#d-i mirror/suite string testing
# Suite to use for loading installer components (optional).
#d-i mirror/udeb/suite string testing
### Account setup
# Skip creation of a root account (normal user account will be able to
# use sudo).
#d-i passwd/root-login boolean false
# Alternatively, to skip creation of a normal user account.
#d-i passwd/make-user boolean false
# Root password, either in clear text
#d-i passwd/root-password password r00tme
#d-i passwd/root-password-again password r00tme
# or encrypted using a crypt(3) hash.
# generate using: python3 -c 'import crypt, getpass; print(crypt.crypt(getpass.getpass(), crypt.METHOD_SHA512))'
d-i passwd/root-password-crypted password $6$ejIby666HDLKIcgq$SthCbLDIgPlQZb/DNB0XOdgXTYGJIx5cu4qqSAH/KiyFEsDPKWTzyUipqu3TcsMziBo5WxiavPO.a0j8qW2ZY/
# To create a normal user account.
d-i passwd/user-fullname string John Doe
d-i passwd/username string administrator
# Normal user's password, either in clear text
#d-i passwd/user-password password insecure
#d-i passwd/user-password-again password insecure
# or encrypted using a crypt(3) hash.
d-i passwd/user-password-crypted password $6$ejIby666HDLKIcgq$SthCbLDIgPlQZb/DNB0XOdgXTYGJIx5cu4qqSAH/KiyFEsDPKWTzyUipqu3TcsMziBo5WxiavPO.a0j8qW2ZY/
# Create the first user with the specified UID instead of the default.
#d-i passwd/user-uid string 1010
# The user account will be added to some standard initial groups. To
# override that, use this.
#d-i passwd/user-default-groups string audio cdrom video
### Clock and time zone setup
# Controls whether or not the hardware clock is set to UTC.
d-i clock-setup/utc boolean true
# You may set this to any valid setting for $TZ; see the contents of
# /usr/share/zoneinfo/ for valid values.
d-i time/zone string Europe/Amsterdam
# Controls whether to use NTP to set the clock during the install
d-i clock-setup/ntp boolean true
# NTP server to use. The default is almost always fine here.
#d-i clock-setup/ntp-server string ntp.example.com
### Partitioning
## Partitioning example
# If the system has free space you can choose to only partition that space.
# This is only honoured if partman-auto/method (below) is not set.
#d-i partman-auto/init_automatically_partition select biggest_free
# Alternatively, you may specify a disk to partition. If the system has only
# one disk the installer will default to using that, but otherwise the device
# name must be given in traditional, non-devfs format (so e.g. /dev/sda
# and not e.g. /dev/discs/disc0/disc).
# For example, to use the first SCSI/SATA hard disk:
#d-i partman-auto/disk string /dev/sda
# In addition, you'll need to specify the method to use.
# The presently available methods are:
# - regular: use the usual partition types for your architecture
# - lvm: use LVM to partition the disk
# - crypto: use LVM within an encrypted partition
d-i partman-auto/method string lvm
# You can define the amount of space that will be used for the LVM volume
# group. It can either be a size with its unit (eg. 20 GB), a percentage of
# free space or the 'max' keyword.
d-i partman-auto-lvm/guided_size string max
# If one of the disks that are going to be automatically partitioned
# contains an old LVM configuration, the user will normally receive a
# warning. This can be preseeded away...
d-i partman-lvm/device_remove_lvm boolean true
# The same applies to pre-existing software RAID array:
d-i partman-md/device_remove_md boolean true
# And the same goes for the confirmation to write the lvm partitions.
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
# You can choose one of the three predefined partitioning recipes:
# - atomic: all files in one partition
# - home: separate /home partition
# - multi: separate /home, /var, and /tmp partitions
d-i partman-auto/choose_recipe select atomic
# Or provide a recipe of your own...
# If you have a way to get a recipe file into the d-i environment, you can
# just point at it.
#d-i partman-auto/expert_recipe_file string /hd-media/recipe
# If not, you can put an entire recipe into the preconfiguration file in one
# (logical) line. This example creates a small /boot partition, suitable
# swap, and uses the rest of the space for the root partition:
#d-i partman-auto/expert_recipe string \
# boot-root :: \
# 40 50 100 ext3 \
# $primary{ } $bootable{ } \
# method{ format } format{ } \
# use_filesystem{ } filesystem{ ext3 } \
# mountpoint{ /boot } \
# . \
# 500 10000 1000000000 ext3 \
# method{ format } format{ } \
# use_filesystem{ } filesystem{ ext3 } \
# mountpoint{ / } \
# . \
# 64 512 300% linux-swap \
# method{ swap } format{ } \
# .
# The full recipe format is documented in the file partman-auto-recipe.txt
# included in the 'debian-installer' package or available from D-I source
# repository. This also documents how to specify settings such as file
# system labels, volume group names and which physical devices to include
# in a volume group.
## Partitioning for EFI
# If your system needs an EFI partition you could add something like
# this to the recipe above, as the first element in the recipe:
# 538 538 1075 free \
# $iflabel{ gpt } \
# $reusemethod{ } \
# method{ efi } \
# format{ } \
# . \
#
# The fragment above is for the amd64 architecture; the details may be
# different on other architectures. The 'partman-auto' package in the
# D-I source repository may have an example you can follow.
# This makes partman automatically partition without confirmation, provided
# that you told it what to do using one of the methods above.
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
# Force UEFI booting ('BIOS compatibility' will be lost). Default: false.
#d-i partman-efi/non_efi_system boolean true
# Ensure the partition table is GPT - this is required for EFI
#d-i partman-partitioning/choose_label select gpt
#d-i partman-partitioning/default_label string gpt
# When disk encryption is enabled, skip wiping the partitions beforehand.
#d-i partman-auto-crypto/erase_disks boolean false
## Partitioning using RAID
# The method should be set to "raid".
#d-i partman-auto/method string raid
# Specify the disks to be partitioned. They will all get the same layout,
# so this will only work if the disks are the same size.
#d-i partman-auto/disk string /dev/sda /dev/sdb
# Next you need to specify the physical partitions that will be used.
#d-i partman-auto/expert_recipe string \
# multiraid :: \
# 1000 5000 4000 raid \
# $primary{ } method{ raid } \
# . \
# 64 512 300% raid \
# method{ raid } \
# . \
# 500 10000 1000000000 raid \
# method{ raid } \
# .
# Last you need to specify how the previously defined partitions will be
# used in the RAID setup. Remember to use the correct partition numbers
# for logical partitions. RAID levels 0, 1, 5, 6 and 10 are supported;
# devices are separated using "#".
# Parameters are:
# <raidtype> <devcount> <sparecount> <fstype> <mountpoint> \
# <devices> <sparedevices>
#d-i partman-auto-raid/recipe string \
# 1 2 0 ext3 / \
# /dev/sda1#/dev/sdb1 \
# . \
# 1 2 0 swap - \
# /dev/sda5#/dev/sdb5 \
# . \
# 0 2 0 ext3 /home \
# /dev/sda6#/dev/sdb6 \
# .
# For additional information see the file partman-auto-raid-recipe.txt
# included in the 'debian-installer' package or available from D-I source
# repository.
# This makes partman automatically partition without confirmation.
d-i partman-md/confirm boolean true
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
## Controlling how partitions are mounted
# The default is to mount by UUID, but you can also choose "traditional" to
# use traditional device names, or "label" to try filesystem labels before
# falling back to UUIDs.
#d-i partman/mount_style select uuid
### Base system installation
# Configure APT to not install recommended packages by default. Use of this
# option can result in an incomplete system and should only be used by very
# experienced users.
#d-i base-installer/install-recommends boolean false
# The kernel image (meta) package to be installed; "none" can be used if no
# kernel is to be installed.
#d-i base-installer/kernel/image string linux-image-686
### Apt setup
# Choose, if you want to scan additional installation media
# (default: false).
d-i apt-setup/cdrom/set-first boolean false
# You can choose to install non-free and contrib software.
#d-i apt-setup/non-free boolean true
#d-i apt-setup/contrib boolean true
# Uncomment the following line, if you don't want to have the sources.list
# entry for a DVD/BD installation image active in the installed system
# (entries for netinst or CD images will be disabled anyway, regardless of
# this setting).
#d-i apt-setup/disable-cdrom-entries boolean true
# Uncomment this if you don't want to use a network mirror.
#d-i apt-setup/use_mirror boolean false
# Select which update services to use; define the mirrors to be used.
# Values shown below are the normal defaults.
#d-i apt-setup/services-select multiselect security, updates
#d-i apt-setup/security_host string security.debian.org
# Additional repositories, local[0-9] available
#d-i apt-setup/local0/repository string \
# http://local.server/debian stable main
#d-i apt-setup/local0/comment string local server
# Enable deb-src lines
#d-i apt-setup/local0/source boolean true
# URL to the public key of the local repository; you must provide a key or
# apt will complain about the unauthenticated repository and so the
# sources.list line will be left commented out.
#d-i apt-setup/local0/key string http://local.server/key
# If the provided key file ends in ".asc" the key file needs to be an
# ASCII-armoured PGP key, if it ends in ".gpg" it needs to use the
# "GPG key public keyring" format, the "keybox database" format is
# currently not supported.
# By default the installer requires that repositories be authenticated
# using a known gpg key. This setting can be used to disable that
# authentication. Warning: Insecure, not recommended.
#d-i debian-installer/allow_unauthenticated boolean true
# Uncomment this to add multiarch configuration for i386
#d-i apt-setup/multiarch string i386
### Package selection
tasksel tasksel/first multiselect standard, ssh-server
# Or choose to not get the tasksel dialog displayed at all (and don't install
# any packages):
#d-i pkgsel/run_tasksel boolean false
# Individual additional packages to install
#d-i pkgsel/include string openssh-server build-essential
# Whether to upgrade packages after debootstrap.
# Allowed values: none, safe-upgrade, full-upgrade
d-i pkgsel/upgrade select full-upgrade
# You can choose, if your system will report back on what software you have
# installed, and what software you use. The default is not to report back,
# but sending reports helps the project determine what software is most
# popular and should be included on the first CD/DVD.
popularity-contest popularity-contest/participate boolean false
### Boot loader installation
# Grub is the boot loader (for x86).
# This is fairly safe to set, it makes grub install automatically to the UEFI
# partition/boot record if no other operating system is detected on the machine.
d-i grub-installer/only_debian boolean true
# This one makes grub-installer install to the UEFI partition/boot record, if
# it also finds some other OS, which is less safe as it might not be able to
# boot that other OS.
d-i grub-installer/with_other_os boolean true
# Due notably to potential USB sticks, the location of the primary drive can
# not be determined safely in general, so this needs to be specified:
d-i grub-installer/bootdev string /dev/sda
# To install to the primary device (assuming it is not a USB stick):
#d-i grub-installer/bootdev string default
# Alternatively, if you want to install to a location other than the UEFI
# parition/boot record, uncomment and edit these lines:
#d-i grub-installer/only_debian boolean false
#d-i grub-installer/with_other_os boolean false
#d-i grub-installer/bootdev string (hd0,1)
# To install grub to multiple disks:
#d-i grub-installer/bootdev string (hd0,1) (hd1,1) (hd2,1)
# Optional password for grub, either in clear text
#d-i grub-installer/password password r00tme
#d-i grub-installer/password-again password r00tme
# or encrypted using an MD5 hash, see grub-md5-crypt(8).
#d-i grub-installer/password-crypted password [MD5 hash]
# Use the following option to add additional boot parameters for the
# installed system (if supported by the bootloader installer).
# Note: options passed to the installer will be added automatically.
#d-i debian-installer/add-kernel-opts string nousb
### Finishing up the installation
# During installations from serial console, the regular virtual consoles
# (VT1-VT6) are normally disabled in /etc/inittab. Uncomment the next
# line to prevent this.
#d-i finish-install/keep-consoles boolean true
# Avoid that last message about the install being complete.
d-i finish-install/reboot_in_progress note
# This will prevent the installer from ejecting the CD during the reboot,
# which is useful in some situations.
#d-i cdrom-detect/eject boolean false
# This is how to make the installer shutdown when finished, but not
# reboot into the installed system.
#d-i debian-installer/exit/halt boolean true
# This will power off the machine instead of just halting it.
#d-i debian-installer/exit/poweroff boolean true
### Preseeding other packages
# Depending on what software you choose to install, or if things go wrong
# during the installation process, it's possible that other questions may
# be asked. You can preseed those too, of course. To get a list of every
# possible question that could be asked during an install, do an
# installation, and then run these commands:
# debconf-get-selections --installer > file
# debconf-get-selections >> file
#### Advanced options
### Running custom commands during the installation
# d-i preseeding is inherently not secure. Nothing in the installer checks
# for attempts at buffer overflows or other exploits of the values of a
# preconfiguration file like this one. Only use preconfiguration files from
# trusted locations! To drive that home, and because it's generally useful,
# here's a way to run any shell command you'd like inside the installer,
# automatically.
# This first command is run as early as possible, just after
# preseeding is read.
#d-i preseed/early_command string anna-install some-udeb
# This command is run immediately before the partitioner starts. It may be
# useful to apply dynamic partitioner preseeding that depends on the state
# of the disks (which may not be visible when preseed/early_command runs).
#d-i partman/early_command \
# string debconf-set partman-auto/disk "$(list-devices disk | head -n1)"
# This command is run just before the install finishes, but when there is
# still a usable /target directory. You can chroot to /target and use it
# directly, or use the apt-install and in-target commands to easily install
# packages and run commands in the target system.
#d-i preseed/late_command string apt-install zsh; in-target chsh -s /bin/zsh

0
gpg/__init__.py Normal file
View File

37
gpg/exceptions.py Normal file
View File

@@ -0,0 +1,37 @@
"""Exceptions which may be raised during execution of gpg wrapper functions."""
class GpgProgrammingException(Exception):
"""A general exception indicating a programming error."""
pass
class GpgRuntimeError(RuntimeError):
"""A general exception indicating a runtime error."""
pass
class UnexpectedOutputException(GpgProgrammingException):
"""Raised when a gpg subprocess produces unexpected output."""
pass
class MissingLocalKeyError(GpgRuntimeError):
"""Raised when an expected key is missing from the local keystore."""
pass
class InvalidSignatureError(GpgRuntimeError):
"""Raised when an invalid gpg signature is encountered."""
pass
class VerificationFailedError(GpgRuntimeError):
"""Raised when a gpg verfication encounters a bad signature."""
pass

112
gpg/keystore.py Normal file
View File

@@ -0,0 +1,112 @@
"""Utilities for searching and importing GPG keys locally."""
from subprocess import run, STDOUT, PIPE
from re import compile
_DEBIAN_KEY_SERVER_HOSTNAME = "keyring.debian.org"
_DEBIAN_CD_SIGNING_KEY_ID = "DA87E80D6294BE9B"
def import_debian_signing_key():
"""Imports the public debian CD signing key using gpg.
The key is imported from keyring.debian.org into the invoking user's
GPG public key store using a shell command.
"""
# execute a gpg key import as a shell command, redirecting stderr to stdout
process_result = run(
[
"gpg", "--keyserver", _DEBIAN_KEY_SERVER_HOSTNAME,
"--recv-key", _DEBIAN_CD_SIGNING_KEY_ID
],
stdout=PIPE,
stderr=STDOUT,
text=True,
)
# check shell return code
if process_result.returncode != 0:
if process_result.stdout:
raise RuntimeError(
f"Failed to import key using gpg:\n{process_result.stdout}"
)
else:
raise RuntimeError("Failed to import key using gpg.")
# check shell output:
# the first line of stdout should look like this
expected_first_line = str(
f"gpg: key {_DEBIAN_CD_SIGNING_KEY_ID}: public key "
f"\"Debian CD signing key <debian-cd@lists.debian.org>\""
f" imported"
)
if not process_result.stdout.split("\n")[0] == expected_first_line:
raise RuntimeError(
f"Unexpected output while importing PGP public key:\n"
f"{process_result.stdout}"
)
def debian_signing_key_is_imported():
"""Checks whether the debian PGP signing key exists in the local key store.
The invoking user's GPG key store is checked using a shell command.
Returns
-------
True : bool
If the public PGP debian cd signing key exists in the invoking user's
GPG key store.
False : bool
If the public PGP debian cd signing key does not exist in the invoking
user's GPG key store.
"""
# execute a local gpg key lookup as a shell command, redirecting stderr to
# stdout
# NOTE: this command returns 0 even if the key is not present
process_result = run(
["gpg", "--locate-keys", _DEBIAN_CD_SIGNING_KEY_ID],
stdout=PIPE,
stderr=STDOUT,
text=True,
)
# check shell return code
if process_result.returncode != 0:
raise RuntimeError("Failed to search local keys using gpg.")
# no shell output means that the key does not exist locally
if not process_result.stdout:
return False
# verify existing key shell output using regex:
# it should contain six lines in the following format
expected_output_lines_regexes = [
compile(r"^pub .*$"),
compile(r"^ *[0-9A-F]{40}$"),
compile(r"^uid .*$"),
compile(r"^sub .*$"),
compile(r"^$"),
compile(r"^$"),
]
actual_output_lines = process_result.stdout.split("\n")
if not len(actual_output_lines) == len(expected_output_lines_regexes):
raise RuntimeError(
f"Unexpected line count in shell output while performing local"
f"GPG key lookup:\n"
f"{process_result.stdout}"
)
for i in range(4):
if not expected_output_lines_regexes[i].match(actual_output_lines[i]):
raise RuntimeError(
f"Unexpected shell output format while performing local"
f"GPG key lookup:\n"
f"{process_result.stdout}"
)
return True

128
gpg/verify.py Normal file
View File

@@ -0,0 +1,128 @@
"""Utilities for verifying files using gpg."""
from re import compile
from pathlib import Path
from subprocess import run, PIPE, STDOUT
import gpg.exceptions as ex
def assert_detached_signature_is_valid(
path_to_input_file,
path_to_signature_file
):
"""Verifies the input file using the specified detached gpg signature.
The invoking user's local gpg key store is used for verification, the
public key used to create the signature must be present in the invoking
user's local gpg key store.
Parameters
----------
path_to_input_file : str or pathlike object
Path to the file which should be verified.
path_to_signature_file : str or pathlike object
Path to the file containing a detached gpg signature of the input file.
Raises
------
FileNotFoundError
If either of the input files do not exist.
gpg.exceptions.MissingLocalKeyError
If a key referenced by the signature file could not be found in the
invoking user's local key store.
gpg.exceptions.VerificationFailedError
If the gpg verification detects a bad signature.
"""
if '~' in str(path_to_input_file):
path_to_input_file = Path(path_to_input_file).expanduser()
path_to_input_file = Path(path_to_input_file).resolve()
if '~' in str(path_to_signature_file):
path_to_signature_file = Path(path_to_signature_file).expanduser()
path_to_signature_file = Path(path_to_signature_file).resolve()
if not path_to_input_file.is_file():
raise FileNotFoundError(
f"No such file: '{path_to_input_file}'."
)
if not path_to_signature_file.is_file():
raise FileNotFoundError(
f"No such file: '{path_to_signature_file}'."
)
# execute a gpg verification as a shell command, redirecting stderr to
# stdout
process_result = run(
[
"gpg", "--verify", path_to_signature_file, path_to_input_file,
],
stdout=PIPE,
stderr=STDOUT,
text=True,
)
output_lines = process_result.stdout.split("\n")
if len(output_lines) < 3:
raise ex.UnexpectedOutputException(
f"Unexpected output during gpg verification:\n"
f"{process_result.stdout}"
)
if process_result.returncode == 2:
# a missing local key causes return code 2
# and the following output on the third line:
missing_key_regex = compile(
r"^gpg: Can't check signature: No public key$"
)
# an invalid detached signature file causes return code 2
# and the following output on the first line:
invalid_signature_regex = compile(
r"^gpg: no valid OpenPGP data found.$"
)
if missing_key_regex.match(output_lines[2]):
raise ex.MissingLocalKeyError(
"Failed to verify gpg signature: no matching local key."
)
elif invalid_signature_regex.match(output_lines[0]):
raise ex.InvalidSignatureError(
"Invalid signature file."
)
else:
raise ex.UnexpectedOutputException(
f"Unexpected output during gpg verification:\n"
f"{process_result.stdout}"
)
elif process_result.returncode == 1:
# failed verification causes return code 1
# and the following output on line 3:
verification_failed_regex = compile(
r"^gpg: BAD signature from .*$"
)
if not verification_failed_regex.match(output_lines[2]):
raise ex.UnexpectedOutputException(
f"Unexpected output during gpg verification:\n"
f"{process_result.stdout}"
)
else:
raise ex.VerificationFailedError(
"gpg signature verification failed: BAD SIGNATURE!"
)
elif process_result.returncode == 0:
# successful verification causes return code 0
# and the following output on line 3:
verification_successful_regex = compile(
r"^gpg: Good signature from .*$"
)
if not verification_successful_regex.match(output_lines[2]):
raise ex.UnexpectedOutputException(
f"Unexpected output during gpg verification:\n"
f"{process_result.stdout}"
)
else:
raise ex.UnexpectedOutputException(
f"Unexpected return code during gpg verification:\n"
f"{process_result.returncode}"
f"{process_result.stdout}"
)

0
iso/__init__.py Normal file
View File

249
udib/modiso.py → iso/injection.py
View File

@@ -1,4 +1,4 @@
"""Library for modification of disk image files.
"""Utilities for modification of disk image files.
Image file modification includes extracting ISO archives, adding files to
initrd-archives contained within the ISO, recalculating md5sum-files
@@ -8,48 +8,15 @@ local filesystem.
"""
from pathlib import Path
from tempfile import TemporaryDirectory
import gzip
import hashlib
import re
import shutil
import subprocess
def _find_all_files_under(parent_dir):
"""Recursively finds all files anywhere under the specified directory.
Returns a list of absolute Path objects. Symlinks are ignored.
Parameters
----------
parent_dir : str or pathlike object
The directory under which to recursively find files.
Raises
------
NotADirectoryError
Raised if the specified parent directory is not a directory.
Examples
--------
config_files = _find_all_files_under("~/.config")
"""
parent_dir = Path(parent_dir).resolve()
if not parent_dir.is_dir():
raise NotADirectoryError(f"No such directory: '{parent_dir}'.")
files = []
for subpath in parent_dir.iterdir():
if subpath.is_file():
files.append(subpath.resolve())
elif not subpath.is_symlink() and subpath.is_dir():
files += _find_all_files_under(subpath)
return files
from cli.clibella import Printer
from core.utils import find_all_files_under
def extract_iso(path_to_output_dir, path_to_input_file):
@@ -78,8 +45,13 @@ def extract_iso(path_to_output_dir, path_to_input_file):
"""
path_to_output_dir = Path(path_to_output_dir)
path_to_input_file = Path(path_to_input_file)
if "~" in str(path_to_output_dir):
path_to_output_dir = Path(path_to_output_dir).expanduser()
path_to_output_dir = Path(path_to_output_dir).resolve()
if "~" in str(path_to_input_file):
path_to_input_file = Path(path_to_input_file).expanduser()
path_to_input_file = Path(path_to_input_file).resolve()
# check if paths are valid
if not path_to_output_dir.is_dir():
@@ -93,19 +65,23 @@ def extract_iso(path_to_output_dir, path_to_input_file):
[
"xorriso",
"-osirrox", "on",
"-indev", path_to_input_file.resolve(),
"-indev", path_to_input_file,
"-extract", "/",
path_to_output_dir.resolve()
path_to_output_dir
],
capture_output=True,
check=True)
check=True
)
except subprocess.CalledProcessError:
raise RuntimeError(
f"An error occurred while extracting '{path_to_input_file}'.")
f"An error occurred while extracting '{path_to_input_file}'."
)
def append_file_contents_to_initrd_archive(path_to_initrd_archive,
path_to_input_file):
def append_file_contents_to_initrd_archive(
path_to_initrd_archive,
path_to_input_file
):
"""Appends the input file to the specified initrd archive.
The initrd archive is extracted, the input file is appended, and
@@ -137,8 +113,13 @@ def append_file_contents_to_initrd_archive(path_to_initrd_archive,
"""
path_to_initrd_archive = Path(path_to_initrd_archive)
path_to_input_file = Path(path_to_input_file)
if "~" in str(path_to_initrd_archive):
path_to_initrd_archive = Path(path_to_initrd_archive).expanduser()
path_to_initrd_archive = Path(path_to_initrd_archive).resolve()
if "~" in str(path_to_input_file):
path_to_input_file = Path(path_to_input_file).expanduser()
path_to_input_file = Path(path_to_input_file).resolve()
# check if initrd file exists and has the correct name
if not path_to_initrd_archive.is_file():
@@ -218,12 +199,15 @@ def regenerate_iso_md5sums_file(path_to_extracted_iso_root):
"""
if "~" in str(path_to_extracted_iso_root):
path_to_extracted_iso_root = Path(path_to_extracted_iso_root).expanduser()
path_to_extracted_iso_root = Path(path_to_extracted_iso_root).resolve()
# check if input path exists
if not path_to_extracted_iso_root.is_dir():
raise NotADirectoryError(f"No such directory: "
f"'{path_to_extracted_iso_root}'.")
raise NotADirectoryError(
f"No such directory: '{path_to_extracted_iso_root}'."
)
path_to_md5sum_file = path_to_extracted_iso_root/"md5sum.txt"
@@ -240,7 +224,7 @@ def regenerate_iso_md5sums_file(path_to_extracted_iso_root):
# Note the two spaces between hash and filepath!
# find all files
subpaths = _find_all_files_under(path_to_extracted_iso_root)
subpaths = find_all_files_under(path_to_extracted_iso_root)
with open(path_to_md5sum_file, "w") as md5sum_file:
for subpath in subpaths:
@@ -252,7 +236,8 @@ def regenerate_iso_md5sums_file(path_to_extracted_iso_root):
md5hash.hexdigest()
+ " "
+ str(subpath.relative_to(path_to_extracted_iso_root))
+ "\n")
+ "\n"
)
# revert write permissions from md5sum.txt and its parent dir
path_to_md5sum_file.chmod(0o444)
@@ -291,7 +276,12 @@ def extract_mbr_from_iso(path_to_output_file, path_to_source_iso):
"""
path_to_output_file = Path(path_to_output_file)
if "~" in str(path_to_output_file):
path_to_output_file = Path(path_to_output_file).expanduser()
path_to_output_file = Path(path_to_output_file).resolve()
if "~" in str(path_to_source_iso):
path_to_source_iso = Path(path_to_source_iso).expanduser()
path_to_source_iso = Path(path_to_source_iso)
# make sure output file does not exist already
@@ -363,9 +353,17 @@ def repack_iso(path_to_output_iso,
"""
path_to_output_iso = Path(path_to_output_iso)
path_to_mbr_data_file = Path(path_to_mbr_data_file)
path_to_input_files_root_dir = Path(path_to_input_files_root_dir)
if "~" in str(path_to_output_iso):
path_to_output_iso = Path(path_to_output_iso).expanduser()
path_to_output_iso = Path(path_to_output_iso).resolve()
if "~" in str(path_to_mbr_data_file):
path_to_mbr_data_file = Path(path_to_mbr_data_file).expanduser()
path_to_mbr_data_file = Path(path_to_mbr_data_file).resolve()
if "~" in str(path_to_input_files_root_dir):
path_to_input_files_root_dir = Path(path_to_input_files_root_dir).expanduser()
path_to_input_files_root_dir = Path(path_to_input_files_root_dir).resolve()
# make sure output file does not exist yet
if path_to_output_iso.exists():
@@ -391,21 +389,136 @@ def repack_iso(path_to_output_iso,
# repack the ISO using xorriso
try:
subprocess.run(
["xorriso", "-as", "mkisofs",
"-r", "-V", created_iso_filesystem_name,
"-o", path_to_output_iso.resolve(),
"-J", "-J", "-joliet-long", "-cache-inodes",
"-isohybrid-mbr", path_to_mbr_data_file.resolve(),
"-b", "isolinux/isolinux.bin",
"-c", "isolinux/boot.cat",
"-boot-load-size", "4", "-boot-info-table", "-no-emul-boot",
"-eltorito-alt-boot",
"-e", "boot/grub/efi.img", "-no-emul-boot",
"-isohybrid-gpt-basdat", "-isohybrid-apm-hfsplus",
path_to_input_files_root_dir.resolve()],
[
"xorriso", "-as", "mkisofs",
"-r", "-V", created_iso_filesystem_name,
"-o", path_to_output_iso,
"-J", "-J", "-joliet-long", "-cache-inodes",
"-isohybrid-mbr", path_to_mbr_data_file,
"-b", "isolinux/isolinux.bin",
"-c", "isolinux/boot.cat",
"-boot-load-size", "4", "-boot-info-table", "-no-emul-boot",
"-eltorito-alt-boot",
"-e", "boot/grub/efi.img", "-no-emul-boot",
"-isohybrid-gpt-basdat", "-isohybrid-apm-hfsplus",
path_to_input_files_root_dir,
],
capture_output=True,
check=True)
check=True,
)
except subprocess.CalledProcessError:
raise RuntimeError(f"Failed while repacking ISO from source files: "
f"'{path_to_input_files_root_dir}'.")
def inject_preseed_file_into_iso(
path_to_output_iso_file,
path_to_input_iso_file,
path_to_input_preseed_file,
iso_filesystem_name="Debian",
printer=None,
):
"""Injects the specified preseed file into the specified ISO file.
Extracts the input ISO into a temporary directory, then extracts the input
ISO's MBR into a temporary file, then appends the preseed file to the
extracted ISO's initrd, then regenerates the extracted ISO's internal MD5
hash list and finally repacks the extracted ISO into the output ISO.
The input ISO file itself is left unchanged.
The output ISO file is newly created.
Parameters
----------
path_to_output_iso_file : str or pathlike object
Path to which the resulting ISO file will be saved.
path_to_input_iso_file : str or pathlike object
Path to the origin ISO file.
path_to_input_preseed_file : str or pathlike object
Path to the input preseed file.
printer : clibella.Printer
A printer for CLI output.
"""
# verify and resolve paths
if "~" in str(path_to_input_iso_file):
path_to_input_iso_file = Path(path_to_input_iso_file).expanduser()
path_to_input_iso_file = Path(path_to_input_iso_file).resolve()
if not path_to_input_iso_file.is_file():
raise FileNotFoundError(f"No such file: '{path_to_input_iso_file}'.")
if "~" in str(path_to_output_iso_file):
path_to_output_iso_file = Path(path_to_output_iso_file).expanduser()
path_to_output_iso_file = Path(path_to_output_iso_file).resolve()
if path_to_output_iso_file.is_file():
raise FileExistsError(f"Output file exists: '{path_to_input_iso_file}'.")
if not path_to_output_iso_file.parent.is_dir():
raise NotADirectoryError(f"No such directory: '{path_to_output_iso_file.parent}'.")
if "~" in str(path_to_input_preseed_file):
path_to_input_preseed_file = Path(path_to_input_preseed_file).expanduser()
path_to_input_preseed_file = Path(path_to_input_preseed_file).resolve()
if not path_to_input_preseed_file.is_file():
raise FileNotFoundError(f"No such file: '{path_to_input_preseed_file}'.")
if printer is None:
p = Printer()
else:
if not isinstance(printer, Printer):
raise TypeError(f"Expected a {type(Printer)} object.")
p = printer
# extract image file to a temporary directory
temp_extracted_iso_dir = TemporaryDirectory()
path_to_extracted_iso_dir = Path(temp_extracted_iso_dir.name)
p.info(f"Extracting contents of {path_to_input_iso_file.name}...")
extract_iso(
path_to_extracted_iso_dir,
path_to_input_iso_file
)
p.ok("ISO extraction complete.")
# extract ISO MBR into a temporary directory
p.info(f"Extracting MBR from {path_to_input_iso_file.name}...")
temp_mbr_dir = TemporaryDirectory()
path_to_mbr_dir = Path(temp_mbr_dir.name)
path_to_mbr_file = path_to_mbr_dir/"mbr.bin"
extract_mbr_from_iso(
path_to_mbr_file,
path_to_input_iso_file,
)
p.ok("MBR extraction complete.")
# create a correctly named copy of the input preseed file and append it
# to the extracted ISO's initrd
p.info(f"Appending {path_to_input_preseed_file.name} to initrd...")
temp_preseed_dir = TemporaryDirectory()
path_to_preseed_dir = Path(temp_preseed_dir.name)
path_to_preseed_file = path_to_preseed_dir/"preseed.cfg"
shutil.copy(path_to_input_preseed_file, path_to_preseed_file)
append_file_contents_to_initrd_archive(
path_to_extracted_iso_dir/"install.amd"/"initrd.gz",
path_to_preseed_file
)
p.ok("Preseed file appended successfully.")
# regenerate extracted ISO's md5sum.txt file
p.info("Regenerating MD5 checksums...")
regenerate_iso_md5sums_file(path_to_extracted_iso_dir)
p.ok("MD5 calculations complete.")
# repack exctracted ISO into a single file
p.info("Repacking ISO...")
repack_iso(
path_to_output_iso_file,
path_to_mbr_file,
path_to_extracted_iso_dir,
iso_filesystem_name
)
p.success(f"ISO file was created successfully at '{path_to_output_iso_file}'.")
# clear out temporary directories
temp_preseed_dir.cleanup()
temp_mbr_dir.cleanup()
temp_extracted_iso_dir.cleanup()

0
net/__init__.py Normal file
View File

79
net/download.py Normal file
View File

@@ -0,0 +1,79 @@
"""Library for downloading files from the web with CLI output."""
from pathlib import Path
import requests
from tqdm import tqdm
from cli.clibella import Printer
def download_file(
path_to_output_file,
url_to_file,
show_progress=False,
printer=None,
):
"""Downloads the file at the input URL to the specified path.
The file is downloaded via HTTP/HTTPS and saved to the specified path.
Optionally, displays a nice status bar.
Parameters
----------
path_to_output_file : str or pathlike object
Path to a file as which the downloaded file is saved.
url_to_file : str
URL to the file to be downloaded.
show_progress : bool
When True, a progress bar is displayed on StdOut indicating the
progress of the download.
printer : clibella.Printer
A clibella.Printer used to print CLI output.
"""
if '~' in str(path_to_output_file):
path_to_output_file = Path(path_to_output_file).expanduser()
path_to_output_file = Path(path_to_output_file).resolve()
if not path_to_output_file.parent.is_dir():
raise FileNotFoundError(
f"No such directory: '{path_to_output_file.parent}'."
)
if path_to_output_file.exists():
raise FileExistsError(
f"File already exists: '{path_to_output_file}'"
)
if printer is None:
p = Printer()
else:
p = printer
output_file_name = path_to_output_file.name
with open(path_to_output_file, "wb") as output_file:
p.info(f"Downloading '{output_file_name}'...")
file_response = requests.get(url_to_file, stream=True)
total_length = file_response.headers.get('content-length')
if total_length is None: # no content length header
output_file.write(file_response.content)
else:
if (show_progress):
total_length = int(total_length)
progress_bar = tqdm(
total=total_length,
unit="B",
unit_scale=True,
unit_divisor=1024
)
for data in file_response.iter_content(chunk_size=4096):
output_file.write(data)
if (show_progress):
progress_bar.update(len(data))
if (show_progress):
progress_bar.close()
p.ok(f"Received '{output_file_name}'.")

108
net/scrape.py Normal file
View File

@@ -0,0 +1,108 @@
"""Methods for scraping the debian website for specific file URLs."""
from re import compile
import requests
from bs4 import BeautifulSoup
def get_debian_preseed_file_urls():
"""Returns a dict containing the URLs for the debian example preseed files.
The dict has the following structure:
{
"basic": {
"url": "https://...",
"name": "...",
},
"full": {
"url": "https://...",
"name": "...",
},
}
where "basic" points to the basic preseed file and its filename, and "full"
points to the full preseed file and its filename.
"""
preseed_file_urls = {
"basic": {
"url": "https://www.debian.org/releases/stable/example-preseed.txt",
"name": "example-preseed.txt",
},
"full": {
"url": "https://preseed.debian.net/debian-preseed/bullseye/amd64-main-full.txt",
"name": "amd64-main-full.txt",
},
}
return preseed_file_urls
def get_debian_iso_urls():
"""Retrieves a dict containing the URLs for a debian installation image.
The dict has the following structure:
{
"image_file": {
"url": "https://...",
"name": "debian-xx.x.x-amd64-netinst.iso",
},
"hash_file": {
"url": "https://...",
"name": "SHA512SUMS",
},
"signature_file": {
"url": "https://...",
"name": "SHA512SUMS.sign",
},
}
where "image_file" is points to the latest debian stable x86-64bit
net-installation ISO image, "hash_file" points to a SHA512SUMS file
containing the SHA512 checksum for the ISO file, and "signature_file"
points to a file containing a PGP signature for verification of the
SHA512SUMS file.
Each top-level dict entry contains a "name" key representing a file name,
and a "url" key specifying a URL to that file.
The function scrapes the official debian.org website to retrieve the URLs.
"""
# request the debian releases page
releases_url = "https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/"
releases_page = requests.get(releases_url)
if not releases_page.status_code == 200:
raise RuntimeError("Unexpected status code during request.")
hash_file_name = "SHA512SUMS"
hash_file_url = releases_url + hash_file_name
signature_file_name = "SHA512SUMS.sign"
signature_file_url = releases_url + signature_file_name
# find the exact URL to the latest stable x64 netinst ISO file
soup = BeautifulSoup(releases_page.content, "html.parser")
image_file_links = soup.find_all(
name="a",
string=compile(r"debian-[0-9.]*-amd64-netinst.iso")
)
if len(image_file_links) != 1:
raise RuntimeError(
"Failed to find an exact match while looking for "
"a link to the latest debian image file."
)
image_file_name = image_file_links[0]['href']
image_file_url = releases_url + image_file_name
return {
"image_file": {
"url": image_file_url,
"name": image_file_name,
},
"hash_file": {
"url": hash_file_url,
"name": hash_file_name,
},
"signature_file": {
"url": signature_file_url,
"name": signature_file_name,
},
}

12
requirements.txt
View File

@@ -1,10 +1,10 @@
beautifulsoup4==4.11.1
certifi==2021.10.8
charset-normalizer==2.0.12
colorama==0.4.4
bs4==0.0.1
certifi==2022.6.15
charset-normalizer==2.1.0
colorama==0.4.5
idna==3.3
python-gnupg==0.4.8
requests==2.27.1
requests==2.28.1
soupsieve==2.3.2.post1
tqdm==4.64.0
urllib3==1.26.9
urllib3==1.26.11

162
udib.py Executable file
View File

@@ -0,0 +1,162 @@
#!/usr/bin/env python3
"""Main entry point for the interactive udib CLI tool."""
from pathlib import Path
from sys import exit
from tempfile import TemporaryDirectory
from cli.clibella import Printer
from cli.parser import get_argument_parser
from core.utils import download_and_verify_debian_iso
from iso.injection import inject_preseed_file_into_iso
from net.download import download_file
from net.scrape import get_debian_preseed_file_urls, get_debian_iso_urls
def main():
# create a CLI printer
p = Printer()
# create an argument parser and read arguments
parser = get_argument_parser()
args = parser.parse_args()
# parse and verify output file if sepcified
if args.path_to_output_file:
path_to_output_file = Path(args.path_to_output_file)
if "~" in str(path_to_output_file):
path_to_output_file = path_to_output_file.expanduser()
path_to_output_file = path_to_output_file.resolve()
if path_to_output_file.exists():
p.error(f"Output file already exists: '{path_to_output_file}'.")
exit(1)
else:
path_to_output_file = None
# parse and verify output dir if sepcified
if args.path_to_output_dir:
path_to_output_dir = Path(args.path_to_output_dir)
if "~" in str(path_to_output_dir):
path_to_output_dir = path_to_output_dir.expanduser()
path_to_output_dir = path_to_output_dir.resolve()
if not path_to_output_dir.is_dir():
p.error(f"No such directory: '{path_to_output_dir}'.")
exit(1)
else:
path_to_output_dir = None
if args.subparser_name == "get":
if args.WHAT == "preseed-file-basic":
# download the basic example preseedfile
if not path_to_output_file:
output_file_name = get_debian_preseed_file_urls()["basic"]["name"]
if path_to_output_dir:
path_to_output_file = path_to_output_dir / output_file_name
else:
path_to_output_file = Path.cwd() / output_file_name
p.info("Retrieving basic preseed example file...")
download_file(
path_to_output_file,
get_debian_preseed_file_urls()["basic"]["url"],
show_progress=False,
printer=p,
)
p.success(
f"Basic preseed example file was saved to '{path_to_output_file}'."
)
exit(0)
elif args.WHAT == "preseed-file-full":
# download the full example preseedfile
if not path_to_output_file:
output_file_name = get_debian_preseed_file_urls()["full"]["name"]
if path_to_output_dir:
path_to_output_file = path_to_output_dir / output_file_name
else:
path_to_output_file = Path.cwd() / output_file_name
p.info("Retrieving full preseed example file...")
download_file(
path_to_output_file,
get_debian_preseed_file_urls()["full"]["url"],
show_progress=False,
printer=p,
)
p.success(
f"Full preseed example file was saved to '{path_to_output_file}'."
)
exit(0)
elif args.WHAT == "iso":
# download and verify installation image
p.info("Downloading latest Debian stable x86-64 netinst ISO...")
if not path_to_output_file:
output_file_name = get_debian_iso_urls()["image_file"]["name"]
if path_to_output_dir:
path_to_output_file = path_to_output_dir / output_file_name
else:
path_to_output_file = Path.cwd() / output_file_name
download_and_verify_debian_iso(path_to_output_file, printer=p)
p.success(f"Debian ISO saved to '{path_to_output_file}'.")
exit(0)
elif args.subparser_name == "inject":
image_file_name = Path(get_debian_iso_urls()["image_file"]["name"])
if not path_to_output_file:
output_file_name = image_file_name.stem + "-preseeded" + image_file_name.suffix
if path_to_output_dir:
path_to_output_file = path_to_output_dir / output_file_name
else:
path_to_output_file = Path.cwd() / output_file_name
# verify preseed file path
path_to_preseed_file = Path(args.PRESEEDFILE)
if "~" in str(path_to_preseed_file):
path_to_preseed_file = Path(path_to_preseed_file).expanduser()
path_to_preseed_file = Path(path_to_preseed_file).resolve()
if not path_to_preseed_file.is_file():
p.error(f"No such file: '{path_to_preseed_file}'.")
exit(1)
# verify image file path if set by user or download fresh iso if unset
temp_iso_dir = None
if args.path_to_image_file:
path_to_image_file = Path(args.path_to_image_file)
if "~" in str(path_to_image_file):
path_to_image_file = Path(path_to_image_file).expanduser()
path_to_image_file = Path(path_to_image_file).resolve()
if not path_to_image_file.is_file():
p.error(f"No such file: '{path_to_image_file}'.")
exit(1)
else:
# download a Debian ISO to a temporary directory
p.info("Downloading the latest Debian x86-64 netinst image...")
temp_iso_dir = TemporaryDirectory()
path_to_iso_dir = Path(temp_iso_dir.name)
path_to_image_file = path_to_iso_dir/image_file_name
download_and_verify_debian_iso(path_to_image_file, printer=p)
# inject the preseed file
inject_preseed_file_into_iso(
path_to_output_file,
path_to_image_file,
path_to_preseed_file,
printer=p,
)
# clear out temporary directory if one was created earlier
if temp_iso_dir:
temp_iso_dir.cleanup()
exit(0)
if __name__ == '__main__':
main()

1
udib/__init__.py
View File

@@ -1 +0,0 @@
#!/usr/bin/env python3

130
udib/gpgverify.py
View File

@@ -1,130 +0,0 @@
"""Library for verification of GPG signatures."""
from pathlib import Path
import gnupg
import userinput
import clibella
p = clibella.Printer()
def gpg_signature_is_valid(
path_to_signature_file,
path_to_data_file,
fallback_keyserver_name
):
"""Validates a PGP signature for a data file.
The detached signature must be provided as plaintext (UTF8) in the
specified signature file.
If the discovered signing key is unknown to gpg on this system for the
invoking user, and if 'fallback_keyserver_name' is not None, an attempt is
made to import the key from the specified keyserver using its ID after
prompting the user for permission to import the key.
Parameters
----------
path_to_signature_file : str or pathlike object
Path to a detached PGP signature stored in a standalone file.
Example value: "/path/to/SHA256SUMS.sig"
path_to_data_file : str or pathlike object
Path to a signed data file, for which the signature is to be verified.
Example value: "/path/to/SHA256SUMS"
fallback_keyserver_name : str
FQDN of a keyserver from which to import unknown public keys.
Example value: "keyring.debian.org"
Returns
-------
True : bool
If the signature is valid.
False : bool
If the signature is invalid or could not be validated.
"""
path_to_signature_file = Path(path_to_signature_file)
path_to_data_file = Path(path_to_data_file)
gpg = gnupg.GPG()
gpg.encoding = "utf-8"
p.info("Validating signature...")
with open(path_to_signature_file, "rb") as signature_file:
verification = gpg.verify_file(
signature_file,
path_to_data_file,
close_file=False
)
# check if a key and fingerprint were found in the signature file
if verification.key_id is None or verification.fingerprint is None:
raise ValueError(
f"Not a valid PGP signature file: '{path_to_signature_file}'."
)
else:
p.info(f"Signature mentions a key with ID "
f"{verification.key_id} and fingerprint "
f"{verification.fingerprint}."
)
if verification.valid:
p.ok(f"GPG signature is valid with trustlevel "
f"'{verification.trust_level}'."
)
return True
# this case commonly occurrs when the GPG key has not been imported
if verification.status == "no public key":
p.warning("Could not find the public GPG key locally!")
# prompt user until answer is unambiguous
key_will_be_imported = None
while key_will_be_imported is None:
key_will_be_imported = userinput.prompt_yes_or_no(
f"[PROMPT] Do you want to import the GPG key from "
f"'{fallback_keyserver_name}'?"
)
if key_will_be_imported is None:
p.error("Unrecognized input. Please try again.")
if not key_will_be_imported:
p.warning("Aborting without importing key.")
return False
# import missing key
p.info("Importing key...")
import_result = gpg.recv_keys(
fallback_keyserver_name, verification.key_id
)
if import_result.count < 1:
p.error("Failed to import key.")
return False
# display some of gpg's output
gpg_output = import_result.stderr.split('\n')
for line in gpg_output:
if line.startswith("gpg: "):
p.info(f"{line}")
# validate signature again
p.info("Validating signature...")
with open(path_to_signature_file, "rb") as signature_file:
verification = gpg.verify_file(
signature_file,
path_to_data_file,
close_file=False
)
if verification.valid:
p.ok(f"GPG signature is valid with trustlevel "
f"'{verification.trust_level}'."
)
return True
else:
p.error("GPG signature is not valid!!!")
return False

333
udib/udib.py
View File

@@ -1,333 +0,0 @@
#!/usr/bin/env python3
"""Main entry point for the interactive udib CLI tool."""
from pathlib import Path
import argparse
import os
import shutil
import subprocess
import sys
import tempfile
import clibella
import modiso
import userinput
import webdownload
p = clibella.Printer()
def _assert_system_dependencies_installed():
"""Asserts that all system dependencies are installed.
System dependencies are those programs whose names are listed in
the global 'system_programs_required' variable.
Raises
------
RuntimeError
Raised when a required program is not accessible in the system
$PATH.
"""
# contains the names of all unix program dependencies which must be
# installed on the local system and available in the local system's $PATH
system_programs_required = [
"cpio",
"gpg",
"xorriso",
]
for program_name in system_programs_required:
try:
subprocess.run(
["command", "-v", program_name],
shell=True,
check=True)
except subprocess.CalledProcessError:
raise RuntimeError(
f"Program not installed or not in $PATH: "
f"'{program_name}'.")
def _get_argument_parser():
"""Instantiates and configures an ArgumentParser and returns it.
Examples
--------
parser = _get_argument_parser()
args = parser.parse_args()
"""
# FIXME capture ISO filesystem name
parser = argparse.ArgumentParser()
parser.add_argument(
"-o",
"--output-file",
help="Path to which the resulting file is written.",
type=str,
dest="path_to_output_file",
action="store")
subparsers = parser.add_subparsers(required=True)
parser_get = subparsers.add_parser("get")
parser_get.add_argument(
"WHAT",
type=str,
action="store",
help="What to retrieve (the latest installation image or the latest " \
"preseed file).",
choices=["image", "preseed-file"])
parser_insert = subparsers.add_parser("insert")
parser_insert.add_argument(
"FILES",
nargs="+",
type=str,
help="A list of files to insert into the root of an installation " \
"image.",
action="store")
parser_insert.add_argument(
"-i",
"--existing-image",
help="Insert into an existing debian image file, instead of " \
"downloading a new one.",
type=str,
dest="path_to_existing_image",
action="store")
return parser
def _chmod_recursively(input_path, mode):
"""Recursively changes file permissions on the specified path.
The file and directory permissions for the specified path itself,
as well as any files and directories anywhere below it are set to
the specified mode.
Symlinks are ignored.
Parameters
----------
input_path : str or pathlike object
Path of which to change the permissions recursively.
mode : int
Permission mode as accepted by os.chmod()
Raises
------
ValueError
Raised if the specified path points to a nonexisting
filesystem node, or if the specified permission mode is
invalid.
TypeError
Raised if mode is not an integer.
Examples
--------
_chmod_recursively("/tmp/mydir", 0o755)
"""
input_path = Path(input_path)
if not input_path.exists():
raise ValueError(f"Path does not exist: '{input_path}'.")
if not isinstance(mode, int):
raise TypeError("Expected type 'int' for parameter 'mode'.")
if mode not in range(0, 0o1000):
raise ValueError("Invalid mode.")
if not input_path.is_symlink():
input_path.chmod(mode)
for child in input_path.iterdir():
if not child.is_symlink():
child.chmod(mode)
if child.is_dir():
_chmod_recursively(child, mode)
def main():
# FIXME capture ISO filesystem name
iso_filesystem_name = "Debian UDIB"
# FIXME adjust to new parser namespace
parser = _get_argument_parser()
args = parser.parse_args()
# check if all system dependencies are installed
try:
_assert_system_dependencies_installed()
except RuntimeError as e:
p.error(e)
sys.exit(1)
# determine where to output files
# default to ouputting files to the current directory
# using their original name if not specified by '-o' argument
path_to_output_dir = Path.cwd()
path_to_output_file = None
# check if path_to_output_file is valid and adjust path_to_output_dir if
# '-o' was set
if args.path_to_output_file:
path_to_output_file = Path(args.path_to_output_file).resolve()
if path_to_output_file.exists() and not path_to_output_file.is_file():
p.error(f"Specified output location is not a file: "
f"'{path_to_output_file}'.")
sys.exit(1)
path_to_output_dir = path_to_output_file.parent
if args.get_image:
# download latest image and exit
with tempfile.TemporaryDirectory() as tmp_dir:
# save image to a temporary directory
path_to_image_file = webdownload.debian_obtain_image(
tmp_dir)
if path_to_output_file:
# rename file if '-o' flag was specified
path_to_image_file = path_to_image_file.rename(
path_to_output_file.name)
else:
# else determine the intended final path
path_to_output_file = path_to_output_dir/path_to_image_file.name
# prompt to confirm file removal if output file already exists
if path_to_output_file.is_file():
prompt = f"File '{path_to_output_file}' already exists. " \
f"Overwrite it?"
if userinput.prompt_yes_or_no(prompt, ask_until_valid=True):
os.remove(path_to_output_file)
else:
p.failure("Did not obtain a new image file.")
sys.exit(1)
# move file from temporary directory to intended path
path_to_image_file.rename(path_to_output_file)
sys.exit(0)
elif args.get_preseed_file:
# download latest preseed file and exit
preseed_file_name = "example-preseed.txt"
preseed_file_url = "https://www.debian.org/"\
"releases/stable/" + preseed_file_name
if not path_to_output_file:
path_to_output_file = path_to_output_dir/preseed_file_name
# prompt to confirm file removal if output file already exists
if path_to_output_file.is_file():
prompt = f"File '{path_to_output_file}' already exists. " \
f"Overwrite it?"
if userinput.prompt_yes_or_no(prompt, ask_until_valid=True):
os.remove(path_to_output_file)
else:
p.failure("Did not obtain a new preseed file.")
sys.exit(1)
webdownload.download_file(path_to_output_file, preseed_file_url)
sys.exit(0)
else:
# modify image file using specified preseed file
path_to_preseed_file = Path(args.existing_preseed_file)
if not path_to_preseed_file.is_file():
p.error(f"No such file: '{path_to_preseed_file}'.")
sys.exit(1)
if args.path_to_existing_image:
# user has specified an existing image file
path_to_image_file = Path(args.path_to_existing_image)
# sanity check
if path_to_image_file.suffix not in [".iso", ".img"]:
p.warning(f"Specified image file does not appear to be an ISO "
f"or IMG file: '{path_to_image_file}'!")
if not userinput.prompt_yes_or_no("Proceed anyways?"):
p.failure("Did not create a new image file.")
sys.exit(1)
else:
# download a fresh image file to a temporary directory
# remember to delete it later!
path_to_image_file = webdownload.debian_obtain_image(
tempfile.mkdtemp())
# extract image file to a temporary directory
path_to_extracted_iso_dir = Path(tempfile.mkdtemp())
p.info("Extracting ISO contents...")
modiso.extract_iso(
path_to_extracted_iso_dir,
path_to_image_file)
# extract image MBR to a temporary directory
p.info("Extracting master boot record...")
path_to_mbr = Path(tempfile.mkdtemp())/"mbr.bin"
modiso.extract_mbr_from_iso(
path_to_mbr,
path_to_image_file)
# append preseed file to extracted initrd
p.info("Appending preseed file...")
# make a temporary copy called 'preseed.cfg'
with tempfile.TemporaryDirectory() as tmpdir:
path_to_renamed_preseed_file = Path(tmpdir).resolve()/"preseed.cfg"
shutil.copy(path_to_preseed_file, path_to_renamed_preseed_file)
modiso.append_file_contents_to_initrd_archive(
path_to_extracted_iso_dir/"install.amd"/"initrd.gz",
path_to_renamed_preseed_file)
# regenerate md5sum.txt
p.info("Regenerating MD5 checksum...")
modiso.regenerate_iso_md5sums_file(path_to_extracted_iso_dir)
if not path_to_output_file:
# use original filename with "-udib" appended before the extension
path_to_output_file = (path_to_output_dir
/ (path_to_image_file.with_suffix("").name
+ "-udib"
+ path_to_image_file.suffix))
# check if outputfile already exists
if path_to_output_file.is_file():
prompt = f"File '{path_to_output_file}' already exists. " \
f"Overwrite it?"
if userinput.prompt_yes_or_no(prompt, ask_until_valid=True):
os.remove(path_to_output_file)
else:
p.failure("Did not create a new ISO file.")
sys.exit(1)
# repack ISO file
p.info("Repacking ISO file...")
modiso.repack_iso(
path_to_output_file,
path_to_mbr,
path_to_extracted_iso_dir,
iso_filesystem_name)
# remove temporary directories
p.info("Cleaning up...")
_chmod_recursively(path_to_extracted_iso_dir, 0o755)
shutil.rmtree(path_to_extracted_iso_dir)
_chmod_recursively(path_to_mbr.parent, 0o755)
shutil.rmtree(path_to_mbr.parent)
p.success(f"Wrote the modified ISO to '{path_to_output_file}'.")
if __name__ == "__main__":
main()

50
udib/userinput.py
View File

@@ -1,50 +0,0 @@
"""Library for consistent interactive user input capturing."""
import re
import clibella
p = clibella.Printer()
def prompt_yes_or_no(question, ask_until_valid=False):
"""Prompts the user for an answer to the specified yes/no question.
If 'ask_until_valid' is True, keeps repeating the prompt until the user
provides recognizable input to answer the question.
Parameters
----------
question : str
The question to ask the user while prompting. The string is suffixed
with " (Yes/No) ".
ask_until_valid : bool
If this is set to True and the user provides an ambiguous answer, keeps
prompting indefinitely until an unambiguous answer is read.
Returns
-------
True : bool
If the user has answered 'yes'.
False : bool
If the user has answered 'no'.
None : None
If 'ask_until_valid' is False and the user has provided an ambiguous
response to the prompt.
"""
user_input_is_valid = False
regex_yes = re.compile(r"^(y)$|^(Y)$|^(YES)$|^(Yes)$|^(yes)$")
regex_no = re.compile(r"^(n)$|^(N)$|^(NO)$|^(No)$|^(no)$")
while(not user_input_is_valid):
user_input = p.input(f"{question} (Yes/No): ")
if (regex_yes.match(user_input)):
return True
elif (regex_no.match(user_input)):
return False
elif (not ask_until_valid):
return None

197
udib/webdownload.py
View File

@@ -1,197 +0,0 @@
"""Library for downloading files from the web.
Specifically, debian installation images and any accompanying files
required for verification of authenticity and integrity of the
obtained images.
"""
import os
import re
import subprocess
from pathlib import Path
import requests
from bs4 import BeautifulSoup
from tqdm import tqdm
import clibella
import gpgverify
p = clibella.Printer()
def download_file(path_to_output_file, url_to_file, show_progress=False):
"""Downloads the file at the input URL to the specified path.
The file is downloaded via HTTP/HTTPS and saved to the specified path.
Optionally, displays a nice status bar.
Parameters
----------
path_to_output_file : str or pathlike object
Path to a file as which the downloaded file is saved.
url_to_file : str
URL to the file to be downloaded.
show_progress : bool
When True, a progress bar is displayed on StdOut indicating the
progress of the download.
"""
path_to_output_file = Path(path_to_output_file).resolve()
if not path_to_output_file.parent.is_dir():
raise FileNotFoundError(
f"No such directory: '{path_to_output_file.parent}'."
)
if path_to_output_file.exists():
raise FileExistsError(
f"File already exists: '{path_to_output_file}'"
)
output_file_name = path_to_output_file.name
with open(path_to_output_file, "wb") as output_file:
p.info(f"Downloading '{output_file_name}'...")
file_response = requests.get(url_to_file, stream=True)
total_length = file_response.headers.get('content-length')
if total_length is None: # no content length header
output_file.write(file_response.content)
else:
if (show_progress):
total_length = int(total_length)
progress_bar = tqdm(
total=total_length,
unit="B",
unit_scale=True,
unit_divisor=1024
)
for data in file_response.iter_content(chunk_size=4096):
output_file.write(data)
if (show_progress):
progress_bar.update(len(data))
if (show_progress):
progress_bar.close()
p.ok(f"Received '{output_file_name}'.")
def debian_obtain_image(path_to_output_dir):
"""Downloads the latest debian installation image and its hashes.
The image file, the SHA512SUMS file it is listed in, as well as the GPG
signature for the SHA512SUMS file are downloaded from the debian.org HTTPS
mirrors and written into the specified output directory.
The obtained image is for the FOSS-only, stable x64 build.
Once the image file is downloaded, the GPG signature of the hash is
validated. Then, the hash of the image file is verified. If either check
fails, an exception is raised.
If the verification suceeds, the hash file and GPG signature file are
removed again.
Parameters
----------
path_to_output_dir : str or pathlike object
Path to the directory to which all downloaded files will be saved.
Returns
-------
path_to_image_file : str
Full path to the obtained and validated image file.
"""
# FIXME download files to a temporary directory
p.info("Obtaining and verifying the latest Debian stable image...")
path_to_output_dir = Path(path_to_output_dir)
if not path_to_output_dir.is_dir():
raise ValueError(f"No such directory: '{path_to_output_dir}'")
releases_url = "https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/"
releases_page = requests.get(releases_url)
soup = BeautifulSoup(releases_page.content, "html.parser")
hash_file_name = "SHA512SUMS"
hash_file_url = releases_url + hash_file_name
signature_file_name = "SHA512SUMS.sign"
signature_file_url = releases_url + signature_file_name
# find the URL to the latest stable x64 image
image_file_links = soup.find_all(
name="a",
string=re.compile(r"debian-[0-9.]*-amd64-netinst.iso"))
if len(image_file_links) != 1:
raise RuntimeError("Failed to find an exact match while looking for "
"a link to the latest debian image file."
)
image_file_name = image_file_links[0]['href']
image_file_url = releases_url + image_file_name
# download the SHA512SUMS file
download_file(path_to_output_dir/hash_file_name, hash_file_url)
# download the GPG signature file
download_file(path_to_output_dir/signature_file_name, signature_file_url)
# verify GPG signature of hash file
if not gpgverify.gpg_signature_is_valid(
path_to_output_dir/signature_file_name,
path_to_output_dir/hash_file_name,
"keyring.debian.org"
):
raise RuntimeError("GPG signature verification failed!")
# download the image file
download_file(path_to_output_dir/image_file_name, image_file_url, True)
# remove unwanted lines from hash file
hash_file = open(path_to_output_dir/hash_file_name, "r")
hash_file_lines = hash_file.readlines()
hash_file_lines_to_keep = []
for line in hash_file_lines:
if image_file_name in line:
hash_file_lines_to_keep.append(line)
hash_file.close()
if len(hash_file_lines_to_keep) != 1:
raise RuntimeError("Unexpected error while truncating hash file.")
os.remove(path_to_output_dir/hash_file_name)
with open(path_to_output_dir/hash_file_name, "w") as hash_file:
hash_file.writelines(hash_file_lines_to_keep)
# validate SHA512 checksum
p.info("Validating file integrity...")
hash_check_result = subprocess.run(
["sha512sum", "--check", path_to_output_dir/hash_file_name],
capture_output=True,
cwd=path_to_output_dir
)
stdout_lines = hash_check_result.stdout.decode("utf-8").split('\n')
stderr_lines = hash_check_result.stderr.decode("utf-8").split('\n')
if len(stdout_lines) > 0:
for line in stdout_lines:
if len(line) > 0:
p.info(f"{line}")
if hash_check_result.returncode != 0:
if len(stderr_lines) > 0:
for line in stderr_lines:
if len(line) > 0:
p.error(f"{line}")
raise RuntimeError("SHA512 validation failed.")
p.ok("File integrity checks passed.")
# clean up obsolete files
p.info("Cleaning up files...")
os.remove(path_to_output_dir/hash_file_name)
os.remove(path_to_output_dir/signature_file_name)
p.success("Debian image obtained.")
return path_to_output_dir/image_file_name