From ca436981e197dfa5c96295fe76abe76173e37320 Mon Sep 17 00:00:00 2001 From: Daniel Akulenok Date: Wed, 8 Apr 2026 14:19:47 +0200 Subject: [PATCH] init --- README.md | 333 ++++++++++++++++++++++++++++++ aptly-mirror.conf | 105 ++++++++++ aptly-mirror.sh | 505 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 943 insertions(+) create mode 100644 README.md create mode 100644 aptly-mirror.conf create mode 100755 aptly-mirror.sh diff --git a/README.md b/README.md new file mode 100644 index 0000000..4f0f248 --- /dev/null +++ b/README.md @@ -0,0 +1,333 @@ +# aptly-mirror.sh + +A comprehensive bash script to create and maintain aptly mirrors for Debian and Ubuntu repositories. Automate the mirroring of package repositories, create time-based snapshots, and publish merged snapshot sets to serve to clients. + +## Features + +- **Multi-distribution support**: Mirror Debian 11/12/13 and Ubuntu 20.04/22.04/24.04/26.04 +- **Flexible mirroring**: Mirror main, updates, and security streams separately +- **Snapshot management**: Create dated snapshots of mirrors and merge them +- **Automatic publishing**: Publish snapshots as APT repositories for client consumption +- **Retention policy**: Keep only N recent snapshot sets, automatically cleanup old ones +- **GPG verification**: Verify repository signatures during download and signing +- **Bandwidth control**: Optional download speed limiting +- **Syslog integration**: All operations logged to syslog for monitoring + +## Getting Started + +### Prerequisites + +- `aptly` (>= 1.4.0): Install via your system package manager or from [aptly.info](https://www.aptly.info/) +- `gpg`: For GPG key management (usually pre-installed) +- Bash 4+ +- Sufficient disk space for mirror storage (50GB+ recommended depending on architecture count) + +### Installation + +1. **Clone or download the script**: + ```bash + git clone aptly-mirror + cd aptly-mirror + ``` + +2. **Create configuration**: + ```bash + cp aptly-mirror.conf.example aptly-mirror.conf + ``` + +3. **Edit configuration** for your environment: + ```bash + nano aptly-mirror.conf + ``` + + Key settings: + - `ARCHITECTURES`: CPU architectures to mirror (e.g., `"amd64"` or `"amd64 arm64"`) + - `APTLY_ROOT`: Where aptly stores its database and mirrors (default: `~/.aptly`) + - `PUBLISH_ROOT`: Where published repos are served (default: `~/.aptly/public`) + - `KEEP_SNAPSHOTS`: Number of old snapshot sets to retain (default: `2`) + +4. **Make the script executable**: + ```bash + chmod +x aptly-mirror.sh + ``` + +5. **Import GPG keys** (recommended for production): + ```bash + ./aptly-mirror.sh import-keys + ``` + +6. **Create initial mirrors**: + ```bash + ./aptly-mirror.sh create + ``` + This may take 10-30 minutes depending on your network and disk speed. + +7. **Create initial snapshots and publish**: + ```bash + ./aptly-mirror.sh publish + ``` + +### Verify Setup + +List all mirrors, snapshots, and published repositories: +```bash +./aptly-mirror.sh list +``` + +Check syslog for operation logs: +```bash +journalctl -t aptly-mirror -f +``` + +## Manual + +### Commands + +#### `create` — Initialize all mirrors + +Create mirrors for all configured Debian and Ubuntu releases. Mirrors are created empty and must be populated with the first `update` command. + +```bash +./aptly-mirror.sh create +``` + +**Output**: +- Creates mirrors for each release: + - `debian-{codename}-main` + - `debian-{codename}-updates` + - `debian-{codename}-security` + - Similar pattern for Ubuntu + +**Time**: 1-2 minutes + +#### `update` — Fetch latest packages and republish + +Updates all mirrors to the latest packages, creates timestamped snapshots, merges them, and publishes a fresh snapshot set. This is the main command to run regularly. + +```bash +./aptly-mirror.sh update +``` + +**Process**: +1. Updates all existing mirrors (downloads new packages) +2. Creates dated snapshots (e.g., `debian-bullseye-main-20260408123045`) +3. Merges main + updates + security snapshots for each release +4. Publishes or switches the published repository to the merged snapshot + +**Time**: 5-30 minutes (depends on network, disk speed, and download limit) + +**Output example**: +``` +Repositories are published under /var/aptly/public/ + +Client sources.list entries: + deb http:///debian bullseye main contrib non-free + deb http:///debian bookworm main contrib non-free non-free-firmware + deb http:///ubuntu focal main restricted universe multiverse +``` + +#### `publish` — Snapshot and publish current state + +Creates fresh snapshots of current mirrors without updating them. Useful when you want to take a snapshot without fetching new packages. + +```bash +./aptly-mirror.sh publish +``` + +**Use cases**: +- Create a point-in-time snapshot after verifying upstream stability +- Re-publish without re-fetching packages + +#### `cleanup` — Remove old snapshots and optimize database + +Removes old snapshot sets beyond the `KEEP_SNAPSHOTS` retention limit and runs aptly's database cleanup. + +```bash +./aptly-mirror.sh cleanup +``` + +**Safety**: +- Only removes snapshots that are not currently published +- Never removes the active published snapshot +- Safe to run after each update + +#### `list` — Show all mirrors, snapshots, and published repos + +Display the current state of all mirrors, snapshots, and published repositories. + +```bash +./aptly-mirror.sh list +``` + +#### `import-keys` — Import GPG signing keys + +Imports GPG keys for verifying Debian and Ubuntu repository signatures. Requires network access to GPG keyserver. + +```bash +./aptly-mirror.sh import-keys +``` + +**Notes**: +- Only needs to run once during setup +- Requires `~/.gnupg/` to exist +- Keyserver can be customized via `GPG_KEYSERVER` in config + +### Options + +#### `-c ` — Specify config file + +```bash +./aptly-mirror.sh -c /etc/aptly-mirror.conf update +``` + +Config file search order (if `-c` not specified): +1. `$APTLY_MIRROR_CONF` environment variable +2. `./aptly-mirror.conf` (same directory as script) +3. `/etc/aptly-mirror.conf` + +#### `-h`, `--help` — Show usage + +```bash +./aptly-mirror.sh --help +``` + +### Configuration + +All settings are in `aptly-mirror.conf`. See the included example for detailed comments. + +**Key settings**: + +| Setting | Default | Description | +|---------|---------|-------------| +| `ARCHITECTURES` | `amd64` | Architectures to mirror (space-separated) | +| `KEEP_SNAPSHOTS` | `2` | Number of old snapshot sets to retain | +| `DOWNLOAD_LIMIT` | `0` | Speed limit in KiB/s (0 = unlimited) | +| `IGNORE_SIGNATURES` | `0` | Skip GPG verification (not recommended) | +| `SKIP_SIGNING` | `0` | Skip GPG signing when publishing | +| `APTLY_ROOT` | `~/.aptly` | Aptly database and mirrors location | +| `PUBLISH_ROOT` | `~/.aptly/public` | Published repositories root | + +### Logging + +All output is logged via syslog with tag `aptly-mirror`. View logs with: + +```bash +# Follow in real-time +journalctl -t aptly-mirror -f + +# View recent entries +journalctl -t aptly-mirror -n 50 + +# Search for errors +journalctl -t aptly-mirror | grep ERROR +``` + +Older systemd-free systems can check `/var/log/syslog`: +```bash +grep aptly-mirror /var/log/syslog +``` + +## Sample Crontab + +Run weekly updates every Sunday at 2 AM: + +```bash +# Edit crontab +crontab -e + +# Add this line: +0 2 * * 0 /home/dak/Code/aptly/aptly-mirror.sh update +``` + +**Full recommended setup** with logging and error notifications: + +```bash +# Weekly update and cleanup +0 2 * * 0 /home/dak/Code/aptly/aptly-mirror.sh update && /home/dak/Code/aptly/aptly-mirror.sh cleanup + +# Check syslog for errors (runs at 3 AM, 1 hour after update) +0 3 * * 0 journalctl -t aptly-mirror -n 100 | grep -i error && echo "aptly-mirror errors detected" | mail -s "aptly-mirror Alert" admin@example.com +``` + +**Multiple updates per week**: + +```bash +# Every 12 hours (twice daily) +0 2,14 * * * /home/dak/Code/aptly/aptly-mirror.sh update + +# Cleanup on Sundays +0 4 * * 0 /home/dak/Code/aptly/aptly-mirror.sh cleanup +``` + +**For systemd systems** (alternative to cron): + +Create `/etc/systemd/system/aptly-mirror.timer`: +```ini +[Unit] +Description=Weekly aptly mirror update +After=network-online.target +Wants=network-online.target + +[Timer] +OnCalendar=Sun *-*-* 02:00:00 +Persistent=true + +[Install] +WantedBy=timers.target +``` + +Create `/etc/systemd/system/aptly-mirror.service`: +```ini +[Unit] +Description=Update aptly mirrors +After=network-online.target + +[Service] +Type=oneshot +ExecStart=/home/dak/Code/aptly/aptly-mirror.sh update +ExecStartPost=/home/dak/Code/aptly/aptly-mirror.sh cleanup +User=aptly +StandardOutput=journal +StandardError=journal +``` + +Enable and start: +```bash +sudo systemctl daemon-reload +sudo systemctl enable aptly-mirror.timer +sudo systemctl start aptly-mirror.timer +``` + +## Troubleshooting + +### "Mirror already exists" +This is normal on repeated runs. The script skips existing mirrors. + +### GPG signature verification failures +Either: +- Import keys: `./aptly-mirror.sh import-keys` +- Or skip verification (not recommended): set `IGNORE_SIGNATURES=1` in config + +### Disk space exhausted +Monitor with: +```bash +du -sh ~/.aptly/ +``` + +Reduce `KEEP_SNAPSHOTS` in config, or clean up old mirrors manually via `aptly mirror drop`. + +### Network timeout during update +Set a `DOWNLOAD_LIMIT` to avoid overwhelming the network, or run during off-peak hours. + +### Can't write to log +Ensure log directory exists and is writable. Check that `LOG_DIR` in config is accessible. + +## AI Attribution + +AIA EAI Hin R Claude Code v1.0 + +This work was entirely AI-generated. AI was prompted for its contributions, or AI assistance was enabled. AI-generated content was reviewed and approved. The following model(s) or application(s) were used: Claude Code. + +## License + +MIT diff --git a/aptly-mirror.conf b/aptly-mirror.conf new file mode 100644 index 0000000..fa92fc5 --- /dev/null +++ b/aptly-mirror.conf @@ -0,0 +1,105 @@ +# --------------------------------------------------------------------------- +# aptly-mirror.conf — Configuration for aptly-mirror.sh +# --------------------------------------------------------------------------- +# +# This file is sourced by aptly-mirror.sh. It uses plain bash syntax. +# To override the default location, set APTLY_MIRROR_CONF or pass -c . +# + +# --------------------------------------------------------------------------- +# General settings +# --------------------------------------------------------------------------- + +# Architectures to mirror (space-separated for multiple, e.g. "amd64 arm64") +ARCHITECTURES="amd64" + +# GPG keyring for repository signature verification (empty = default trustedkeys.gpg) +GPG_KEYRING="" + +# Set to 1 to skip GPG signature verification (not recommended for production) +IGNORE_SIGNATURES=0 + +# Number of old snapshot sets to retain during cleanup (0 = keep only current) +KEEP_SNAPSHOTS=2 + +# Download speed limit in KiB/s (0 = unlimited) +DOWNLOAD_LIMIT=0 + +# Set to 1 to skip GPG signing when publishing (useful if no GPG key configured) +SKIP_SIGNING=0 + +# --------------------------------------------------------------------------- +# Storage paths +# --------------------------------------------------------------------------- + +# Root directory for aptly data (mirrors, snapshots, db) +# Leave empty to use aptly's default (~/.aptly) +APTLY_ROOT="" + +# Directory where published repositories are served from +# Leave empty to use aptly's default (~/.aptly/public) +PUBLISH_ROOT="" + + +# --------------------------------------------------------------------------- +# Debian mirrors +# --------------------------------------------------------------------------- + +DEBIAN_MIRROR="http://deb.debian.org/debian" +DEBIAN_SECURITY_MIRROR="http://security.debian.org/debian-security" + +# Debian releases: codename -> version number +declare -A DEBIAN_RELEASES=( + [bullseye]="11" + [bookworm]="12" + [trixie]="13" +) + +# Components for Debian 11 and earlier (before non-free-firmware split) +DEBIAN_COMPONENTS_LEGACY="main contrib non-free" +# Components for Debian 12+ (non-free-firmware is a separate component) +DEBIAN_COMPONENTS_MODERN="main contrib non-free non-free-firmware" + +# Security mirror components use an "updates/" prefix (applies to all Debian releases) +DEBIAN_SECURITY_COMPONENTS_LEGACY="updates/main updates/contrib updates/non-free" +DEBIAN_SECURITY_COMPONENTS_MODERN="updates/main updates/contrib updates/non-free updates/non-free-firmware" + +# --------------------------------------------------------------------------- +# Ubuntu mirrors +# --------------------------------------------------------------------------- + +UBUNTU_MIRROR="http://archive.ubuntu.com/ubuntu" +UBUNTU_SECURITY_MIRROR="http://security.ubuntu.com/ubuntu" + +# Ubuntu releases: codename -> version number +declare -A UBUNTU_RELEASES=( + [focal]="20.04" + [jammy]="22.04" + [noble]="24.04" + [resolute]="26.04" +) + +UBUNTU_COMPONENTS="main restricted universe multiverse" + +# --------------------------------------------------------------------------- +# GPG keys +# --------------------------------------------------------------------------- + +# Debian archive signing key fingerprints +DEBIAN_GPG_KEYS=( + "1F89983E0081FDE018F3CC9673A4F27B8DD47936" # Debian Archive (11/bullseye) + "A7236886F3CCCAAD148A27F80E98404D386FA1D9" # Debian Archive (12/bookworm) + "B7C5D7829D669F88788D4C1BEFCC6169C4AE0E3C" # Debian Security Archive + "4D64FEC119C2029067D6E791F8D2585B8783D481" # Debian Stable Release Key + "A48449044AAD5C5DEEDE9B19B7B0C1CF28EA3E34" # Debian Archive (13/trixie) + "89C87ACEA5DD6B8E6A7068808E9F831205B4BA95" # Debian Security Archive (13/trixie) +) + +# Ubuntu archive signing key fingerprints +UBUNTU_GPG_KEYS=( + "3B4FE6ACC0B21F32" # Ubuntu Archive + "871920D1991BC93C" # Ubuntu Archive (older) +) + +# GPG keyserver to fetch keys from +GPG_KEYSERVER="hkps://keyserver.ubuntu.com" diff --git a/aptly-mirror.sh b/aptly-mirror.sh new file mode 100755 index 0000000..25b28b3 --- /dev/null +++ b/aptly-mirror.sh @@ -0,0 +1,505 @@ +#!/usr/bin/env bash +# +# aptly-mirror.sh — Create and maintain aptly mirrors for Debian and Ubuntu repositories. +# +# Mirrors: +# Debian: 11 (bullseye), 12 (bookworm), 13 (trixie) +# Ubuntu: 20.04 (focal), 22.04 (jammy), 24.04 (noble), 26.04 (roaring) +# +# Usage: +# ./aptly-mirror.sh create — Create all mirrors (first-time setup) +# ./aptly-mirror.sh update — Update all mirrors, snapshot, merge, and publish +# ./aptly-mirror.sh publish — Snapshot current state and publish/switch +# ./aptly-mirror.sh cleanup — Drop old snapshots and run db cleanup +# ./aptly-mirror.sh list — List all mirrors, snapshots, and published repos +# +set -euo pipefail + +# --------------------------------------------------------------------------- +# Load configuration +# --------------------------------------------------------------------------- + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# Config file search order: +# 1. -c command-line flag +# 2. APTLY_MIRROR_CONF environment variable +# 3. ./aptly-mirror.conf (next to the script) +# 4. /etc/aptly-mirror.conf + +find_config() { + if [[ -n "${APTLY_MIRROR_CONF:-}" && -r "$APTLY_MIRROR_CONF" ]]; then + echo "$APTLY_MIRROR_CONF" + elif [[ -r "${SCRIPT_DIR}/aptly-mirror.conf" ]]; then + echo "${SCRIPT_DIR}/aptly-mirror.conf" + elif [[ -r "/etc/aptly-mirror.conf" ]]; then + echo "/etc/aptly-mirror.conf" + else + echo "" + fi +} + +CONFIG_FILE="" + +# Parse flags before subcommand +while [[ "${1:-}" == -* ]]; do + case "$1" in + -c) CONFIG_FILE="$2"; shift 2 ;; + -h|--help) CONFIG_FILE="__help__"; break ;; + *) break ;; + esac +done + +if [[ -z "$CONFIG_FILE" ]]; then + CONFIG_FILE=$(find_config) +fi + +if [[ "$CONFIG_FILE" == "__help__" ]]; then + # Defer to main() which will show usage + : +elif [[ -z "$CONFIG_FILE" ]]; then + echo "Error: No config file found." >&2 + echo "Searched: \$APTLY_MIRROR_CONF, ${SCRIPT_DIR}/aptly-mirror.conf, /etc/aptly-mirror.conf" >&2 + echo "Create one from aptly-mirror.conf.example or pass -c ." >&2 + exit 1 +else + # shellcheck source=aptly-mirror.conf + source "$CONFIG_FILE" +fi + +DATE=$(date +%Y%m%d%H%M%S) + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +log() { + local msg="$*" + echo "$msg" + logger -t aptly-mirror "$msg" +} + +die() { + log "FATAL: $*" + exit 1 +} + +run_aptly() { + log " -> aptly $*" + if ! aptly "$@" 2>&1; then + log " !! aptly $1 failed (exit $?)" + return 1 + fi +} + +mirror_exists() { + aptly mirror show "$1" &>/dev/null +} + +snapshot_exists() { + aptly snapshot show "$1" &>/dev/null +} + +published_exists() { + # $1 = distribution, $2 = prefix (default ".") + local dist="$1" prefix="${2:-.}" + aptly publish list -raw 2>/dev/null | grep -qE "^${prefix} ${dist}$" +} + +common_mirror_flags() { + local flags="-architectures=${ARCHITECTURES}" + if [[ -n "$GPG_KEYRING" ]]; then + flags+=" -keyring=${GPG_KEYRING}" + fi + if (( IGNORE_SIGNATURES )); then + flags+=" -ignore-signatures" + fi + echo "$flags" +} + +common_update_flags() { + local flags="" + if (( DOWNLOAD_LIMIT > 0 )); then + flags+=" -download-limit=${DOWNLOAD_LIMIT}" + fi + if (( IGNORE_SIGNATURES )); then + flags+=" -ignore-signatures" + fi + echo "$flags" +} + +publish_flags() { + local flags="" + if (( SKIP_SIGNING )); then + flags+=" -skip-signing" + fi + echo "$flags" +} + +# --------------------------------------------------------------------------- +# get_debian_components +# --------------------------------------------------------------------------- +get_debian_components() { + local codename="$1" + case "$codename" in + bullseye) echo "$DEBIAN_COMPONENTS_LEGACY" ;; + *) echo "$DEBIAN_COMPONENTS_MODERN" ;; + esac +} + +get_debian_security_components() { + local codename="$1" + case "$codename" in + bullseye) echo "$DEBIAN_SECURITY_COMPONENTS_LEGACY" ;; + *) echo "$DEBIAN_SECURITY_COMPONENTS_MODERN" ;; + esac +} + +# --------------------------------------------------------------------------- +# CREATE — set up all mirrors +# --------------------------------------------------------------------------- + +create_debian_mirrors() { + local codename version components security_components + for codename in "${!DEBIAN_RELEASES[@]}"; do + version="${DEBIAN_RELEASES[$codename]}" + components=$(get_debian_components "$codename") + security_components=$(get_debian_security_components "$codename") + + log "Creating Debian ${version} (${codename}) mirrors..." + + local name="debian-${codename}-main" + if ! mirror_exists "$name"; then + run_aptly mirror create $(common_mirror_flags) \ + "$name" "$DEBIAN_MIRROR" "$codename" $components + else + log " Mirror $name already exists, skipping." + fi + + name="debian-${codename}-updates" + if ! mirror_exists "$name"; then + run_aptly mirror create $(common_mirror_flags) \ + "$name" "$DEBIAN_MIRROR" "${codename}-updates" $components + else + log " Mirror $name already exists, skipping." + fi + + name="debian-${codename}-security" + if ! mirror_exists "$name"; then + run_aptly mirror create $(common_mirror_flags) \ + "$name" "$DEBIAN_SECURITY_MIRROR" "${codename}-security" $security_components + else + log " Mirror $name already exists, skipping." + fi + done +} + +create_ubuntu_mirrors() { + local codename version + for codename in "${!UBUNTU_RELEASES[@]}"; do + version="${UBUNTU_RELEASES[$codename]}" + + log "Creating Ubuntu ${version} (${codename}) mirrors..." + + local name="ubuntu-${codename}-main" + if ! mirror_exists "$name"; then + run_aptly mirror create $(common_mirror_flags) \ + "$name" "$UBUNTU_MIRROR" "$codename" $UBUNTU_COMPONENTS + else + log " Mirror $name already exists, skipping." + fi + + name="ubuntu-${codename}-updates" + if ! mirror_exists "$name"; then + run_aptly mirror create $(common_mirror_flags) \ + "$name" "$UBUNTU_MIRROR" "${codename}-updates" $UBUNTU_COMPONENTS + else + log " Mirror $name already exists, skipping." + fi + + name="ubuntu-${codename}-security" + if ! mirror_exists "$name"; then + run_aptly mirror create $(common_mirror_flags) \ + "$name" "$UBUNTU_SECURITY_MIRROR" "${codename}-security" $UBUNTU_COMPONENTS + else + log " Mirror $name already exists, skipping." + fi + done +} + +do_create() { + log "=== Creating all mirrors ===" + create_debian_mirrors + create_ubuntu_mirrors + log "=== Mirror creation complete ===" +} + +# --------------------------------------------------------------------------- +# UPDATE — fetch latest package metadata and files +# --------------------------------------------------------------------------- + +update_all_mirrors() { + local mirror_name + log "=== Updating all mirrors ===" + + while IFS= read -r mirror_name; do + [[ -z "$mirror_name" ]] && continue + log "Updating mirror: ${mirror_name}" + run_aptly mirror update $(common_update_flags) "$mirror_name" || \ + log " !! Warning: failed to update ${mirror_name}, continuing..." + done < <(aptly mirror list -raw 2>/dev/null) + + log "=== Mirror updates complete ===" +} + +# --------------------------------------------------------------------------- +# SNAPSHOT & PUBLISH — create dated snapshots, merge, and publish/switch +# --------------------------------------------------------------------------- + +snapshot_and_publish_debian() { + local codename version snap_main snap_updates snap_security snap_merged + local pub_flags + pub_flags=$(publish_flags) + + for codename in "${!DEBIAN_RELEASES[@]}"; do + version="${DEBIAN_RELEASES[$codename]}" + + log "Snapshotting Debian ${version} (${codename})..." + + snap_main="debian-${codename}-main-${DATE}" + snap_updates="debian-${codename}-updates-${DATE}" + snap_security="debian-${codename}-security-${DATE}" + snap_merged="debian-${codename}-merged-${DATE}" + + run_aptly snapshot create "$snap_main" from mirror "debian-${codename}-main" + run_aptly snapshot create "$snap_updates" from mirror "debian-${codename}-updates" + run_aptly snapshot create "$snap_security" from mirror "debian-${codename}-security" + + log "Merging snapshots for Debian ${codename}..." + run_aptly snapshot merge -latest \ + "$snap_merged" "$snap_main" "$snap_updates" "$snap_security" + + local prefix="debian" + if published_exists "$codename" "$prefix"; then + log "Switching published Debian ${codename} to ${snap_merged}..." + run_aptly publish switch $pub_flags "$codename" "$prefix" "$snap_merged" + else + log "Publishing Debian ${codename} for the first time..." + run_aptly publish snapshot $pub_flags \ + -distribution="$codename" -architectures="$ARCHITECTURES" \ + "$snap_merged" "$prefix" + fi + done +} + +snapshot_and_publish_ubuntu() { + local codename version snap_main snap_updates snap_security snap_merged + local pub_flags + pub_flags=$(publish_flags) + + for codename in "${!UBUNTU_RELEASES[@]}"; do + version="${UBUNTU_RELEASES[$codename]}" + + log "Snapshotting Ubuntu ${version} (${codename})..." + + snap_main="ubuntu-${codename}-main-${DATE}" + snap_updates="ubuntu-${codename}-updates-${DATE}" + snap_security="ubuntu-${codename}-security-${DATE}" + snap_merged="ubuntu-${codename}-merged-${DATE}" + + run_aptly snapshot create "$snap_main" from mirror "ubuntu-${codename}-main" + run_aptly snapshot create "$snap_updates" from mirror "ubuntu-${codename}-updates" + run_aptly snapshot create "$snap_security" from mirror "ubuntu-${codename}-security" + + log "Merging snapshots for Ubuntu ${codename}..." + run_aptly snapshot merge -latest \ + "$snap_merged" "$snap_main" "$snap_updates" "$snap_security" + + local prefix="ubuntu" + if published_exists "$codename" "$prefix"; then + log "Switching published Ubuntu ${codename} to ${snap_merged}..." + run_aptly publish switch $pub_flags "$codename" "$prefix" "$snap_merged" + else + log "Publishing Ubuntu ${codename} for the first time..." + run_aptly publish snapshot $pub_flags \ + -distribution="$codename" -architectures="$ARCHITECTURES" \ + "$snap_merged" "$prefix" + fi + done +} + +do_publish() { + log "=== Snapshot and publish ===" + snapshot_and_publish_debian + snapshot_and_publish_ubuntu + log "=== Publish complete ===" + log "" + log "Repositories are published under ${PUBLISH_ROOT:-~/.aptly/public}/" + log "" + log "Client sources.list entries:" + for codename in "${!DEBIAN_RELEASES[@]}"; do + log " deb http:///debian ${codename} $(get_debian_components "$codename")" + done + for codename in "${!UBUNTU_RELEASES[@]}"; do + log " deb http:///ubuntu ${codename} ${UBUNTU_COMPONENTS}" + done +} + +# --------------------------------------------------------------------------- +# UPDATE (full) — update mirrors then snapshot & publish +# --------------------------------------------------------------------------- + +do_update() { + update_all_mirrors + do_publish +} + +# --------------------------------------------------------------------------- +# CLEANUP — drop old snapshots, run db cleanup +# --------------------------------------------------------------------------- + +do_cleanup() { + log "=== Cleanup ===" + + # Collect all snapshot names grouped by prefix (everything before the timestamp) + declare -A snap_groups + while IFS= read -r snap; do + [[ -z "$snap" ]] && continue + # Snapshot names follow: -YYYYMMDDHHMMSS + # Strip the 14-digit timestamp suffix to get the group key + local prefix="${snap%-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]}" + if [[ "$prefix" == "$snap" ]]; then + # No timestamp suffix — skip (manually created snapshot) + continue + fi + snap_groups["$prefix"]+="${snap}"$'\n' + done < <(aptly snapshot list -raw -sort=name 2>/dev/null) + + local dropped=0 + for prefix in "${!snap_groups[@]}"; do + # Sort snapshots by name (timestamp order) and keep the newest KEEP_SNAPSHOTS + local snaps + snaps=$(echo -n "${snap_groups[$prefix]}" | sort) + local count + count=$(echo "$snaps" | wc -l) + local to_drop=$(( count - KEEP_SNAPSHOTS )) + + if (( to_drop > 0 )); then + echo "$snaps" | head -n "$to_drop" | while IFS= read -r old_snap; do + [[ -z "$old_snap" ]] && continue + log "Dropping snapshot: ${old_snap}" + run_aptly snapshot drop "$old_snap" || \ + log " !! Could not drop ${old_snap} (may be published or referenced)" + (( dropped++ )) || true + done + fi + done + + log "Running database cleanup..." + run_aptly db cleanup + + log "=== Cleanup complete ===" +} + +# --------------------------------------------------------------------------- +# LIST — show current state +# --------------------------------------------------------------------------- + +do_list() { + echo "===============================" + echo " Mirrors" + echo "===============================" + aptly mirror list 2>/dev/null || true + echo "" + echo "===============================" + echo " Snapshots" + echo "===============================" + aptly snapshot list -sort=time 2>/dev/null || true + echo "" + echo "===============================" + echo " Published Repositories" + echo "===============================" + aptly publish list 2>/dev/null || true +} + +# --------------------------------------------------------------------------- +# GPG key import helper +# --------------------------------------------------------------------------- + +do_import_keys() { + log "=== Importing repository signing keys ===" + + local keyserver="${GPG_KEYSERVER:-hkps://keyserver.ubuntu.com}" + + log "Importing Debian GPG keys..." + for key in "${DEBIAN_GPG_KEYS[@]}"; do + gpg --no-default-keyring --keyring trustedkeys.gpg \ + --keyserver "$keyserver" --recv-keys "$key" 2>&1 || \ + log " !! Warning: could not import key ${key}" + done + + log "Importing Ubuntu GPG keys..." + for key in "${UBUNTU_GPG_KEYS[@]}"; do + gpg --no-default-keyring --keyring trustedkeys.gpg \ + --keyserver "$keyserver" --recv-keys "$key" 2>&1 || \ + log " !! Warning: could not import key ${key}" + done + + log "=== Key import complete ===" +} + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + +usage() { + cat <] + +Commands: + create Create all mirrors (first-time setup) + update Update mirrors, snapshot, merge, and publish + publish Snapshot current mirror state and publish/switch + cleanup Drop old snapshots and run aptly db cleanup + list List all mirrors, snapshots, and published repos + import-keys Import GPG keys for Debian and Ubuntu repositories + +Options: + -c Path to config file (default: auto-detected) + +Configuration: + The config file is searched in this order: + 1. -c flag + 2. \$APTLY_MIRROR_CONF environment variable + 3. ${SCRIPT_DIR}/aptly-mirror.conf + 4. /etc/aptly-mirror.conf +USAGE +} + +main() { + if (( $# < 1 )); then + usage + exit 1 + fi + + local cmd="$1"; shift + case "$cmd" in + create) do_create ;; + update) do_update ;; + publish) do_publish ;; + cleanup) do_cleanup ;; + list) do_list ;; + import-keys) do_import_keys ;; + -h|--help|help) + usage + exit 0 + ;; + *) + echo "Unknown command: $cmd" >&2 + usage + exit 1 + ;; + esac +} + +main "$@"