init
This commit is contained in:
Daniel Akulenok
333
README.md
Normal file
333
README.md
Normal file
@@ -0,0 +1,333 @@
|
|||||||
|
# aptly-mirror.sh
|
||||||
|
|
||||||
|
A comprehensive bash script to create and maintain aptly mirrors for Debian and Ubuntu repositories. Automate the mirroring of package repositories, create time-based snapshots, and publish merged snapshot sets to serve to clients.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- **Multi-distribution support**: Mirror Debian 11/12/13 and Ubuntu 20.04/22.04/24.04/26.04
|
||||||
|
- **Flexible mirroring**: Mirror main, updates, and security streams separately
|
||||||
|
- **Snapshot management**: Create dated snapshots of mirrors and merge them
|
||||||
|
- **Automatic publishing**: Publish snapshots as APT repositories for client consumption
|
||||||
|
- **Retention policy**: Keep only N recent snapshot sets, automatically cleanup old ones
|
||||||
|
- **GPG verification**: Verify repository signatures during download and signing
|
||||||
|
- **Bandwidth control**: Optional download speed limiting
|
||||||
|
- **Syslog integration**: All operations logged to syslog for monitoring
|
||||||
|
|
||||||
|
## Getting Started
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
|
||||||
|
- `aptly` (>= 1.4.0): Install via your system package manager or from [aptly.info](https://www.aptly.info/)
|
||||||
|
- `gpg`: For GPG key management (usually pre-installed)
|
||||||
|
- Bash 4+
|
||||||
|
- Sufficient disk space for mirror storage (50GB+ recommended depending on architecture count)
|
||||||
|
|
||||||
|
### Installation
|
||||||
|
|
||||||
|
1. **Clone or download the script**:
|
||||||
|
```bash
|
||||||
|
git clone <repository> aptly-mirror
|
||||||
|
cd aptly-mirror
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Create configuration**:
|
||||||
|
```bash
|
||||||
|
cp aptly-mirror.conf.example aptly-mirror.conf
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Edit configuration** for your environment:
|
||||||
|
```bash
|
||||||
|
nano aptly-mirror.conf
|
||||||
|
```
|
||||||
|
|
||||||
|
Key settings:
|
||||||
|
- `ARCHITECTURES`: CPU architectures to mirror (e.g., `"amd64"` or `"amd64 arm64"`)
|
||||||
|
- `APTLY_ROOT`: Where aptly stores its database and mirrors (default: `~/.aptly`)
|
||||||
|
- `PUBLISH_ROOT`: Where published repos are served (default: `~/.aptly/public`)
|
||||||
|
- `KEEP_SNAPSHOTS`: Number of old snapshot sets to retain (default: `2`)
|
||||||
|
|
||||||
|
4. **Make the script executable**:
|
||||||
|
```bash
|
||||||
|
chmod +x aptly-mirror.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
5. **Import GPG keys** (recommended for production):
|
||||||
|
```bash
|
||||||
|
./aptly-mirror.sh import-keys
|
||||||
|
```
|
||||||
|
|
||||||
|
6. **Create initial mirrors**:
|
||||||
|
```bash
|
||||||
|
./aptly-mirror.sh create
|
||||||
|
```
|
||||||
|
This may take 10-30 minutes depending on your network and disk speed.
|
||||||
|
|
||||||
|
7. **Create initial snapshots and publish**:
|
||||||
|
```bash
|
||||||
|
./aptly-mirror.sh publish
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verify Setup
|
||||||
|
|
||||||
|
List all mirrors, snapshots, and published repositories:
|
||||||
|
```bash
|
||||||
|
./aptly-mirror.sh list
|
||||||
|
```
|
||||||
|
|
||||||
|
Check syslog for operation logs:
|
||||||
|
```bash
|
||||||
|
journalctl -t aptly-mirror -f
|
||||||
|
```
|
||||||
|
|
||||||
|
## Manual
|
||||||
|
|
||||||
|
### Commands
|
||||||
|
|
||||||
|
#### `create` — Initialize all mirrors
|
||||||
|
|
||||||
|
Create mirrors for all configured Debian and Ubuntu releases. Mirrors are created empty and must be populated with the first `update` command.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./aptly-mirror.sh create
|
||||||
|
```
|
||||||
|
|
||||||
|
**Output**:
|
||||||
|
- Creates mirrors for each release:
|
||||||
|
- `debian-{codename}-main`
|
||||||
|
- `debian-{codename}-updates`
|
||||||
|
- `debian-{codename}-security`
|
||||||
|
- Similar pattern for Ubuntu
|
||||||
|
|
||||||
|
**Time**: 1-2 minutes
|
||||||
|
|
||||||
|
#### `update` — Fetch latest packages and republish
|
||||||
|
|
||||||
|
Updates all mirrors to the latest packages, creates timestamped snapshots, merges them, and publishes a fresh snapshot set. This is the main command to run regularly.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./aptly-mirror.sh update
|
||||||
|
```
|
||||||
|
|
||||||
|
**Process**:
|
||||||
|
1. Updates all existing mirrors (downloads new packages)
|
||||||
|
2. Creates dated snapshots (e.g., `debian-bullseye-main-20260408123045`)
|
||||||
|
3. Merges main + updates + security snapshots for each release
|
||||||
|
4. Publishes or switches the published repository to the merged snapshot
|
||||||
|
|
||||||
|
**Time**: 5-30 minutes (depends on network, disk speed, and download limit)
|
||||||
|
|
||||||
|
**Output example**:
|
||||||
|
```
|
||||||
|
Repositories are published under /var/aptly/public/
|
||||||
|
|
||||||
|
Client sources.list entries:
|
||||||
|
deb http://<server>/debian bullseye main contrib non-free
|
||||||
|
deb http://<server>/debian bookworm main contrib non-free non-free-firmware
|
||||||
|
deb http://<server>/ubuntu focal main restricted universe multiverse
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `publish` — Snapshot and publish current state
|
||||||
|
|
||||||
|
Creates fresh snapshots of current mirrors without updating them. Useful when you want to take a snapshot without fetching new packages.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./aptly-mirror.sh publish
|
||||||
|
```
|
||||||
|
|
||||||
|
**Use cases**:
|
||||||
|
- Create a point-in-time snapshot after verifying upstream stability
|
||||||
|
- Re-publish without re-fetching packages
|
||||||
|
|
||||||
|
#### `cleanup` — Remove old snapshots and optimize database
|
||||||
|
|
||||||
|
Removes old snapshot sets beyond the `KEEP_SNAPSHOTS` retention limit and runs aptly's database cleanup.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./aptly-mirror.sh cleanup
|
||||||
|
```
|
||||||
|
|
||||||
|
**Safety**:
|
||||||
|
- Only removes snapshots that are not currently published
|
||||||
|
- Never removes the active published snapshot
|
||||||
|
- Safe to run after each update
|
||||||
|
|
||||||
|
#### `list` — Show all mirrors, snapshots, and published repos
|
||||||
|
|
||||||
|
Display the current state of all mirrors, snapshots, and published repositories.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./aptly-mirror.sh list
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `import-keys` — Import GPG signing keys
|
||||||
|
|
||||||
|
Imports GPG keys for verifying Debian and Ubuntu repository signatures. Requires network access to GPG keyserver.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./aptly-mirror.sh import-keys
|
||||||
|
```
|
||||||
|
|
||||||
|
**Notes**:
|
||||||
|
- Only needs to run once during setup
|
||||||
|
- Requires `~/.gnupg/` to exist
|
||||||
|
- Keyserver can be customized via `GPG_KEYSERVER` in config
|
||||||
|
|
||||||
|
### Options
|
||||||
|
|
||||||
|
#### `-c <path>` — Specify config file
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./aptly-mirror.sh -c /etc/aptly-mirror.conf update
|
||||||
|
```
|
||||||
|
|
||||||
|
Config file search order (if `-c` not specified):
|
||||||
|
1. `$APTLY_MIRROR_CONF` environment variable
|
||||||
|
2. `./aptly-mirror.conf` (same directory as script)
|
||||||
|
3. `/etc/aptly-mirror.conf`
|
||||||
|
|
||||||
|
#### `-h`, `--help` — Show usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./aptly-mirror.sh --help
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration
|
||||||
|
|
||||||
|
All settings are in `aptly-mirror.conf`. See the included example for detailed comments.
|
||||||
|
|
||||||
|
**Key settings**:
|
||||||
|
|
||||||
|
| Setting | Default | Description |
|
||||||
|
|---------|---------|-------------|
|
||||||
|
| `ARCHITECTURES` | `amd64` | Architectures to mirror (space-separated) |
|
||||||
|
| `KEEP_SNAPSHOTS` | `2` | Number of old snapshot sets to retain |
|
||||||
|
| `DOWNLOAD_LIMIT` | `0` | Speed limit in KiB/s (0 = unlimited) |
|
||||||
|
| `IGNORE_SIGNATURES` | `0` | Skip GPG verification (not recommended) |
|
||||||
|
| `SKIP_SIGNING` | `0` | Skip GPG signing when publishing |
|
||||||
|
| `APTLY_ROOT` | `~/.aptly` | Aptly database and mirrors location |
|
||||||
|
| `PUBLISH_ROOT` | `~/.aptly/public` | Published repositories root |
|
||||||
|
|
||||||
|
### Logging
|
||||||
|
|
||||||
|
All output is logged via syslog with tag `aptly-mirror`. View logs with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Follow in real-time
|
||||||
|
journalctl -t aptly-mirror -f
|
||||||
|
|
||||||
|
# View recent entries
|
||||||
|
journalctl -t aptly-mirror -n 50
|
||||||
|
|
||||||
|
# Search for errors
|
||||||
|
journalctl -t aptly-mirror | grep ERROR
|
||||||
|
```
|
||||||
|
|
||||||
|
Older systemd-free systems can check `/var/log/syslog`:
|
||||||
|
```bash
|
||||||
|
grep aptly-mirror /var/log/syslog
|
||||||
|
```
|
||||||
|
|
||||||
|
## Sample Crontab
|
||||||
|
|
||||||
|
Run weekly updates every Sunday at 2 AM:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Edit crontab
|
||||||
|
crontab -e
|
||||||
|
|
||||||
|
# Add this line:
|
||||||
|
0 2 * * 0 /home/dak/Code/aptly/aptly-mirror.sh update
|
||||||
|
```
|
||||||
|
|
||||||
|
**Full recommended setup** with logging and error notifications:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Weekly update and cleanup
|
||||||
|
0 2 * * 0 /home/dak/Code/aptly/aptly-mirror.sh update && /home/dak/Code/aptly/aptly-mirror.sh cleanup
|
||||||
|
|
||||||
|
# Check syslog for errors (runs at 3 AM, 1 hour after update)
|
||||||
|
0 3 * * 0 journalctl -t aptly-mirror -n 100 | grep -i error && echo "aptly-mirror errors detected" | mail -s "aptly-mirror Alert" admin@example.com
|
||||||
|
```
|
||||||
|
|
||||||
|
**Multiple updates per week**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Every 12 hours (twice daily)
|
||||||
|
0 2,14 * * * /home/dak/Code/aptly/aptly-mirror.sh update
|
||||||
|
|
||||||
|
# Cleanup on Sundays
|
||||||
|
0 4 * * 0 /home/dak/Code/aptly/aptly-mirror.sh cleanup
|
||||||
|
```
|
||||||
|
|
||||||
|
**For systemd systems** (alternative to cron):
|
||||||
|
|
||||||
|
Create `/etc/systemd/system/aptly-mirror.timer`:
|
||||||
|
```ini
|
||||||
|
[Unit]
|
||||||
|
Description=Weekly aptly mirror update
|
||||||
|
After=network-online.target
|
||||||
|
Wants=network-online.target
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnCalendar=Sun *-*-* 02:00:00
|
||||||
|
Persistent=true
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
|
```
|
||||||
|
|
||||||
|
Create `/etc/systemd/system/aptly-mirror.service`:
|
||||||
|
```ini
|
||||||
|
[Unit]
|
||||||
|
Description=Update aptly mirrors
|
||||||
|
After=network-online.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
ExecStart=/home/dak/Code/aptly/aptly-mirror.sh update
|
||||||
|
ExecStartPost=/home/dak/Code/aptly/aptly-mirror.sh cleanup
|
||||||
|
User=aptly
|
||||||
|
StandardOutput=journal
|
||||||
|
StandardError=journal
|
||||||
|
```
|
||||||
|
|
||||||
|
Enable and start:
|
||||||
|
```bash
|
||||||
|
sudo systemctl daemon-reload
|
||||||
|
sudo systemctl enable aptly-mirror.timer
|
||||||
|
sudo systemctl start aptly-mirror.timer
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### "Mirror already exists"
|
||||||
|
This is normal on repeated runs. The script skips existing mirrors.
|
||||||
|
|
||||||
|
### GPG signature verification failures
|
||||||
|
Either:
|
||||||
|
- Import keys: `./aptly-mirror.sh import-keys`
|
||||||
|
- Or skip verification (not recommended): set `IGNORE_SIGNATURES=1` in config
|
||||||
|
|
||||||
|
### Disk space exhausted
|
||||||
|
Monitor with:
|
||||||
|
```bash
|
||||||
|
du -sh ~/.aptly/
|
||||||
|
```
|
||||||
|
|
||||||
|
Reduce `KEEP_SNAPSHOTS` in config, or clean up old mirrors manually via `aptly mirror drop`.
|
||||||
|
|
||||||
|
### Network timeout during update
|
||||||
|
Set a `DOWNLOAD_LIMIT` to avoid overwhelming the network, or run during off-peak hours.
|
||||||
|
|
||||||
|
### Can't write to log
|
||||||
|
Ensure log directory exists and is writable. Check that `LOG_DIR` in config is accessible.
|
||||||
|
|
||||||
|
## AI Attribution
|
||||||
|
|
||||||
|
AIA EAI Hin R Claude Code v1.0
|
||||||
|
|
||||||
|
This work was entirely AI-generated. AI was prompted for its contributions, or AI assistance was enabled. AI-generated content was reviewed and approved. The following model(s) or application(s) were used: Claude Code.
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT
|
||||||
105
aptly-mirror.conf
Normal file
105
aptly-mirror.conf
Normal file
@@ -0,0 +1,105 @@
|
|||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# aptly-mirror.conf — Configuration for aptly-mirror.sh
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
#
|
||||||
|
# This file is sourced by aptly-mirror.sh. It uses plain bash syntax.
|
||||||
|
# To override the default location, set APTLY_MIRROR_CONF or pass -c <path>.
|
||||||
|
#
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# General settings
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# Architectures to mirror (space-separated for multiple, e.g. "amd64 arm64")
|
||||||
|
ARCHITECTURES="amd64"
|
||||||
|
|
||||||
|
# GPG keyring for repository signature verification (empty = default trustedkeys.gpg)
|
||||||
|
GPG_KEYRING=""
|
||||||
|
|
||||||
|
# Set to 1 to skip GPG signature verification (not recommended for production)
|
||||||
|
IGNORE_SIGNATURES=0
|
||||||
|
|
||||||
|
# Number of old snapshot sets to retain during cleanup (0 = keep only current)
|
||||||
|
KEEP_SNAPSHOTS=2
|
||||||
|
|
||||||
|
# Download speed limit in KiB/s (0 = unlimited)
|
||||||
|
DOWNLOAD_LIMIT=0
|
||||||
|
|
||||||
|
# Set to 1 to skip GPG signing when publishing (useful if no GPG key configured)
|
||||||
|
SKIP_SIGNING=0
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Storage paths
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# Root directory for aptly data (mirrors, snapshots, db)
|
||||||
|
# Leave empty to use aptly's default (~/.aptly)
|
||||||
|
APTLY_ROOT=""
|
||||||
|
|
||||||
|
# Directory where published repositories are served from
|
||||||
|
# Leave empty to use aptly's default (~/.aptly/public)
|
||||||
|
PUBLISH_ROOT=""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Debian mirrors
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
DEBIAN_MIRROR="http://deb.debian.org/debian"
|
||||||
|
DEBIAN_SECURITY_MIRROR="http://security.debian.org/debian-security"
|
||||||
|
|
||||||
|
# Debian releases: codename -> version number
|
||||||
|
declare -A DEBIAN_RELEASES=(
|
||||||
|
[bullseye]="11"
|
||||||
|
[bookworm]="12"
|
||||||
|
[trixie]="13"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Components for Debian 11 and earlier (before non-free-firmware split)
|
||||||
|
DEBIAN_COMPONENTS_LEGACY="main contrib non-free"
|
||||||
|
# Components for Debian 12+ (non-free-firmware is a separate component)
|
||||||
|
DEBIAN_COMPONENTS_MODERN="main contrib non-free non-free-firmware"
|
||||||
|
|
||||||
|
# Security mirror components use an "updates/" prefix (applies to all Debian releases)
|
||||||
|
DEBIAN_SECURITY_COMPONENTS_LEGACY="updates/main updates/contrib updates/non-free"
|
||||||
|
DEBIAN_SECURITY_COMPONENTS_MODERN="updates/main updates/contrib updates/non-free updates/non-free-firmware"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Ubuntu mirrors
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
UBUNTU_MIRROR="http://archive.ubuntu.com/ubuntu"
|
||||||
|
UBUNTU_SECURITY_MIRROR="http://security.ubuntu.com/ubuntu"
|
||||||
|
|
||||||
|
# Ubuntu releases: codename -> version number
|
||||||
|
declare -A UBUNTU_RELEASES=(
|
||||||
|
[focal]="20.04"
|
||||||
|
[jammy]="22.04"
|
||||||
|
[noble]="24.04"
|
||||||
|
[resolute]="26.04"
|
||||||
|
)
|
||||||
|
|
||||||
|
UBUNTU_COMPONENTS="main restricted universe multiverse"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# GPG keys
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# Debian archive signing key fingerprints
|
||||||
|
DEBIAN_GPG_KEYS=(
|
||||||
|
"1F89983E0081FDE018F3CC9673A4F27B8DD47936" # Debian Archive (11/bullseye)
|
||||||
|
"A7236886F3CCCAAD148A27F80E98404D386FA1D9" # Debian Archive (12/bookworm)
|
||||||
|
"B7C5D7829D669F88788D4C1BEFCC6169C4AE0E3C" # Debian Security Archive
|
||||||
|
"4D64FEC119C2029067D6E791F8D2585B8783D481" # Debian Stable Release Key
|
||||||
|
"A48449044AAD5C5DEEDE9B19B7B0C1CF28EA3E34" # Debian Archive (13/trixie)
|
||||||
|
"89C87ACEA5DD6B8E6A7068808E9F831205B4BA95" # Debian Security Archive (13/trixie)
|
||||||
|
)
|
||||||
|
|
||||||
|
# Ubuntu archive signing key fingerprints
|
||||||
|
UBUNTU_GPG_KEYS=(
|
||||||
|
"3B4FE6ACC0B21F32" # Ubuntu Archive
|
||||||
|
"871920D1991BC93C" # Ubuntu Archive (older)
|
||||||
|
)
|
||||||
|
|
||||||
|
# GPG keyserver to fetch keys from
|
||||||
|
GPG_KEYSERVER="hkps://keyserver.ubuntu.com"
|
||||||
505
aptly-mirror.sh
Executable file
505
aptly-mirror.sh
Executable file
@@ -0,0 +1,505 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
#
|
||||||
|
# aptly-mirror.sh — Create and maintain aptly mirrors for Debian and Ubuntu repositories.
|
||||||
|
#
|
||||||
|
# Mirrors:
|
||||||
|
# Debian: 11 (bullseye), 12 (bookworm), 13 (trixie)
|
||||||
|
# Ubuntu: 20.04 (focal), 22.04 (jammy), 24.04 (noble), 26.04 (roaring)
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./aptly-mirror.sh create — Create all mirrors (first-time setup)
|
||||||
|
# ./aptly-mirror.sh update — Update all mirrors, snapshot, merge, and publish
|
||||||
|
# ./aptly-mirror.sh publish — Snapshot current state and publish/switch
|
||||||
|
# ./aptly-mirror.sh cleanup — Drop old snapshots and run db cleanup
|
||||||
|
# ./aptly-mirror.sh list — List all mirrors, snapshots, and published repos
|
||||||
|
#
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Load configuration
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
# Config file search order:
|
||||||
|
# 1. -c <path> command-line flag
|
||||||
|
# 2. APTLY_MIRROR_CONF environment variable
|
||||||
|
# 3. ./aptly-mirror.conf (next to the script)
|
||||||
|
# 4. /etc/aptly-mirror.conf
|
||||||
|
|
||||||
|
find_config() {
|
||||||
|
if [[ -n "${APTLY_MIRROR_CONF:-}" && -r "$APTLY_MIRROR_CONF" ]]; then
|
||||||
|
echo "$APTLY_MIRROR_CONF"
|
||||||
|
elif [[ -r "${SCRIPT_DIR}/aptly-mirror.conf" ]]; then
|
||||||
|
echo "${SCRIPT_DIR}/aptly-mirror.conf"
|
||||||
|
elif [[ -r "/etc/aptly-mirror.conf" ]]; then
|
||||||
|
echo "/etc/aptly-mirror.conf"
|
||||||
|
else
|
||||||
|
echo ""
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
CONFIG_FILE=""
|
||||||
|
|
||||||
|
# Parse flags before subcommand
|
||||||
|
while [[ "${1:-}" == -* ]]; do
|
||||||
|
case "$1" in
|
||||||
|
-c) CONFIG_FILE="$2"; shift 2 ;;
|
||||||
|
-h|--help) CONFIG_FILE="__help__"; break ;;
|
||||||
|
*) break ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
if [[ -z "$CONFIG_FILE" ]]; then
|
||||||
|
CONFIG_FILE=$(find_config)
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$CONFIG_FILE" == "__help__" ]]; then
|
||||||
|
# Defer to main() which will show usage
|
||||||
|
:
|
||||||
|
elif [[ -z "$CONFIG_FILE" ]]; then
|
||||||
|
echo "Error: No config file found." >&2
|
||||||
|
echo "Searched: \$APTLY_MIRROR_CONF, ${SCRIPT_DIR}/aptly-mirror.conf, /etc/aptly-mirror.conf" >&2
|
||||||
|
echo "Create one from aptly-mirror.conf.example or pass -c <path>." >&2
|
||||||
|
exit 1
|
||||||
|
else
|
||||||
|
# shellcheck source=aptly-mirror.conf
|
||||||
|
source "$CONFIG_FILE"
|
||||||
|
fi
|
||||||
|
|
||||||
|
DATE=$(date +%Y%m%d%H%M%S)
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
log() {
|
||||||
|
local msg="$*"
|
||||||
|
echo "$msg"
|
||||||
|
logger -t aptly-mirror "$msg"
|
||||||
|
}
|
||||||
|
|
||||||
|
die() {
|
||||||
|
log "FATAL: $*"
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
run_aptly() {
|
||||||
|
log " -> aptly $*"
|
||||||
|
if ! aptly "$@" 2>&1; then
|
||||||
|
log " !! aptly $1 failed (exit $?)"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
mirror_exists() {
|
||||||
|
aptly mirror show "$1" &>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
snapshot_exists() {
|
||||||
|
aptly snapshot show "$1" &>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
published_exists() {
|
||||||
|
# $1 = distribution, $2 = prefix (default ".")
|
||||||
|
local dist="$1" prefix="${2:-.}"
|
||||||
|
aptly publish list -raw 2>/dev/null | grep -qE "^${prefix} ${dist}$"
|
||||||
|
}
|
||||||
|
|
||||||
|
common_mirror_flags() {
|
||||||
|
local flags="-architectures=${ARCHITECTURES}"
|
||||||
|
if [[ -n "$GPG_KEYRING" ]]; then
|
||||||
|
flags+=" -keyring=${GPG_KEYRING}"
|
||||||
|
fi
|
||||||
|
if (( IGNORE_SIGNATURES )); then
|
||||||
|
flags+=" -ignore-signatures"
|
||||||
|
fi
|
||||||
|
echo "$flags"
|
||||||
|
}
|
||||||
|
|
||||||
|
common_update_flags() {
|
||||||
|
local flags=""
|
||||||
|
if (( DOWNLOAD_LIMIT > 0 )); then
|
||||||
|
flags+=" -download-limit=${DOWNLOAD_LIMIT}"
|
||||||
|
fi
|
||||||
|
if (( IGNORE_SIGNATURES )); then
|
||||||
|
flags+=" -ignore-signatures"
|
||||||
|
fi
|
||||||
|
echo "$flags"
|
||||||
|
}
|
||||||
|
|
||||||
|
publish_flags() {
|
||||||
|
local flags=""
|
||||||
|
if (( SKIP_SIGNING )); then
|
||||||
|
flags+=" -skip-signing"
|
||||||
|
fi
|
||||||
|
echo "$flags"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# get_debian_components <codename>
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
get_debian_components() {
|
||||||
|
local codename="$1"
|
||||||
|
case "$codename" in
|
||||||
|
bullseye) echo "$DEBIAN_COMPONENTS_LEGACY" ;;
|
||||||
|
*) echo "$DEBIAN_COMPONENTS_MODERN" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
get_debian_security_components() {
|
||||||
|
local codename="$1"
|
||||||
|
case "$codename" in
|
||||||
|
bullseye) echo "$DEBIAN_SECURITY_COMPONENTS_LEGACY" ;;
|
||||||
|
*) echo "$DEBIAN_SECURITY_COMPONENTS_MODERN" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# CREATE — set up all mirrors
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
create_debian_mirrors() {
|
||||||
|
local codename version components security_components
|
||||||
|
for codename in "${!DEBIAN_RELEASES[@]}"; do
|
||||||
|
version="${DEBIAN_RELEASES[$codename]}"
|
||||||
|
components=$(get_debian_components "$codename")
|
||||||
|
security_components=$(get_debian_security_components "$codename")
|
||||||
|
|
||||||
|
log "Creating Debian ${version} (${codename}) mirrors..."
|
||||||
|
|
||||||
|
local name="debian-${codename}-main"
|
||||||
|
if ! mirror_exists "$name"; then
|
||||||
|
run_aptly mirror create $(common_mirror_flags) \
|
||||||
|
"$name" "$DEBIAN_MIRROR" "$codename" $components
|
||||||
|
else
|
||||||
|
log " Mirror $name already exists, skipping."
|
||||||
|
fi
|
||||||
|
|
||||||
|
name="debian-${codename}-updates"
|
||||||
|
if ! mirror_exists "$name"; then
|
||||||
|
run_aptly mirror create $(common_mirror_flags) \
|
||||||
|
"$name" "$DEBIAN_MIRROR" "${codename}-updates" $components
|
||||||
|
else
|
||||||
|
log " Mirror $name already exists, skipping."
|
||||||
|
fi
|
||||||
|
|
||||||
|
name="debian-${codename}-security"
|
||||||
|
if ! mirror_exists "$name"; then
|
||||||
|
run_aptly mirror create $(common_mirror_flags) \
|
||||||
|
"$name" "$DEBIAN_SECURITY_MIRROR" "${codename}-security" $security_components
|
||||||
|
else
|
||||||
|
log " Mirror $name already exists, skipping."
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
create_ubuntu_mirrors() {
|
||||||
|
local codename version
|
||||||
|
for codename in "${!UBUNTU_RELEASES[@]}"; do
|
||||||
|
version="${UBUNTU_RELEASES[$codename]}"
|
||||||
|
|
||||||
|
log "Creating Ubuntu ${version} (${codename}) mirrors..."
|
||||||
|
|
||||||
|
local name="ubuntu-${codename}-main"
|
||||||
|
if ! mirror_exists "$name"; then
|
||||||
|
run_aptly mirror create $(common_mirror_flags) \
|
||||||
|
"$name" "$UBUNTU_MIRROR" "$codename" $UBUNTU_COMPONENTS
|
||||||
|
else
|
||||||
|
log " Mirror $name already exists, skipping."
|
||||||
|
fi
|
||||||
|
|
||||||
|
name="ubuntu-${codename}-updates"
|
||||||
|
if ! mirror_exists "$name"; then
|
||||||
|
run_aptly mirror create $(common_mirror_flags) \
|
||||||
|
"$name" "$UBUNTU_MIRROR" "${codename}-updates" $UBUNTU_COMPONENTS
|
||||||
|
else
|
||||||
|
log " Mirror $name already exists, skipping."
|
||||||
|
fi
|
||||||
|
|
||||||
|
name="ubuntu-${codename}-security"
|
||||||
|
if ! mirror_exists "$name"; then
|
||||||
|
run_aptly mirror create $(common_mirror_flags) \
|
||||||
|
"$name" "$UBUNTU_SECURITY_MIRROR" "${codename}-security" $UBUNTU_COMPONENTS
|
||||||
|
else
|
||||||
|
log " Mirror $name already exists, skipping."
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
do_create() {
|
||||||
|
log "=== Creating all mirrors ==="
|
||||||
|
create_debian_mirrors
|
||||||
|
create_ubuntu_mirrors
|
||||||
|
log "=== Mirror creation complete ==="
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# UPDATE — fetch latest package metadata and files
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
update_all_mirrors() {
|
||||||
|
local mirror_name
|
||||||
|
log "=== Updating all mirrors ==="
|
||||||
|
|
||||||
|
while IFS= read -r mirror_name; do
|
||||||
|
[[ -z "$mirror_name" ]] && continue
|
||||||
|
log "Updating mirror: ${mirror_name}"
|
||||||
|
run_aptly mirror update $(common_update_flags) "$mirror_name" || \
|
||||||
|
log " !! Warning: failed to update ${mirror_name}, continuing..."
|
||||||
|
done < <(aptly mirror list -raw 2>/dev/null)
|
||||||
|
|
||||||
|
log "=== Mirror updates complete ==="
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# SNAPSHOT & PUBLISH — create dated snapshots, merge, and publish/switch
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
snapshot_and_publish_debian() {
|
||||||
|
local codename version snap_main snap_updates snap_security snap_merged
|
||||||
|
local pub_flags
|
||||||
|
pub_flags=$(publish_flags)
|
||||||
|
|
||||||
|
for codename in "${!DEBIAN_RELEASES[@]}"; do
|
||||||
|
version="${DEBIAN_RELEASES[$codename]}"
|
||||||
|
|
||||||
|
log "Snapshotting Debian ${version} (${codename})..."
|
||||||
|
|
||||||
|
snap_main="debian-${codename}-main-${DATE}"
|
||||||
|
snap_updates="debian-${codename}-updates-${DATE}"
|
||||||
|
snap_security="debian-${codename}-security-${DATE}"
|
||||||
|
snap_merged="debian-${codename}-merged-${DATE}"
|
||||||
|
|
||||||
|
run_aptly snapshot create "$snap_main" from mirror "debian-${codename}-main"
|
||||||
|
run_aptly snapshot create "$snap_updates" from mirror "debian-${codename}-updates"
|
||||||
|
run_aptly snapshot create "$snap_security" from mirror "debian-${codename}-security"
|
||||||
|
|
||||||
|
log "Merging snapshots for Debian ${codename}..."
|
||||||
|
run_aptly snapshot merge -latest \
|
||||||
|
"$snap_merged" "$snap_main" "$snap_updates" "$snap_security"
|
||||||
|
|
||||||
|
local prefix="debian"
|
||||||
|
if published_exists "$codename" "$prefix"; then
|
||||||
|
log "Switching published Debian ${codename} to ${snap_merged}..."
|
||||||
|
run_aptly publish switch $pub_flags "$codename" "$prefix" "$snap_merged"
|
||||||
|
else
|
||||||
|
log "Publishing Debian ${codename} for the first time..."
|
||||||
|
run_aptly publish snapshot $pub_flags \
|
||||||
|
-distribution="$codename" -architectures="$ARCHITECTURES" \
|
||||||
|
"$snap_merged" "$prefix"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
snapshot_and_publish_ubuntu() {
|
||||||
|
local codename version snap_main snap_updates snap_security snap_merged
|
||||||
|
local pub_flags
|
||||||
|
pub_flags=$(publish_flags)
|
||||||
|
|
||||||
|
for codename in "${!UBUNTU_RELEASES[@]}"; do
|
||||||
|
version="${UBUNTU_RELEASES[$codename]}"
|
||||||
|
|
||||||
|
log "Snapshotting Ubuntu ${version} (${codename})..."
|
||||||
|
|
||||||
|
snap_main="ubuntu-${codename}-main-${DATE}"
|
||||||
|
snap_updates="ubuntu-${codename}-updates-${DATE}"
|
||||||
|
snap_security="ubuntu-${codename}-security-${DATE}"
|
||||||
|
snap_merged="ubuntu-${codename}-merged-${DATE}"
|
||||||
|
|
||||||
|
run_aptly snapshot create "$snap_main" from mirror "ubuntu-${codename}-main"
|
||||||
|
run_aptly snapshot create "$snap_updates" from mirror "ubuntu-${codename}-updates"
|
||||||
|
run_aptly snapshot create "$snap_security" from mirror "ubuntu-${codename}-security"
|
||||||
|
|
||||||
|
log "Merging snapshots for Ubuntu ${codename}..."
|
||||||
|
run_aptly snapshot merge -latest \
|
||||||
|
"$snap_merged" "$snap_main" "$snap_updates" "$snap_security"
|
||||||
|
|
||||||
|
local prefix="ubuntu"
|
||||||
|
if published_exists "$codename" "$prefix"; then
|
||||||
|
log "Switching published Ubuntu ${codename} to ${snap_merged}..."
|
||||||
|
run_aptly publish switch $pub_flags "$codename" "$prefix" "$snap_merged"
|
||||||
|
else
|
||||||
|
log "Publishing Ubuntu ${codename} for the first time..."
|
||||||
|
run_aptly publish snapshot $pub_flags \
|
||||||
|
-distribution="$codename" -architectures="$ARCHITECTURES" \
|
||||||
|
"$snap_merged" "$prefix"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
do_publish() {
|
||||||
|
log "=== Snapshot and publish ==="
|
||||||
|
snapshot_and_publish_debian
|
||||||
|
snapshot_and_publish_ubuntu
|
||||||
|
log "=== Publish complete ==="
|
||||||
|
log ""
|
||||||
|
log "Repositories are published under ${PUBLISH_ROOT:-~/.aptly/public}/"
|
||||||
|
log ""
|
||||||
|
log "Client sources.list entries:"
|
||||||
|
for codename in "${!DEBIAN_RELEASES[@]}"; do
|
||||||
|
log " deb http://<server>/debian ${codename} $(get_debian_components "$codename")"
|
||||||
|
done
|
||||||
|
for codename in "${!UBUNTU_RELEASES[@]}"; do
|
||||||
|
log " deb http://<server>/ubuntu ${codename} ${UBUNTU_COMPONENTS}"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# UPDATE (full) — update mirrors then snapshot & publish
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
do_update() {
|
||||||
|
update_all_mirrors
|
||||||
|
do_publish
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# CLEANUP — drop old snapshots, run db cleanup
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
do_cleanup() {
|
||||||
|
log "=== Cleanup ==="
|
||||||
|
|
||||||
|
# Collect all snapshot names grouped by prefix (everything before the timestamp)
|
||||||
|
declare -A snap_groups
|
||||||
|
while IFS= read -r snap; do
|
||||||
|
[[ -z "$snap" ]] && continue
|
||||||
|
# Snapshot names follow: <prefix>-YYYYMMDDHHMMSS
|
||||||
|
# Strip the 14-digit timestamp suffix to get the group key
|
||||||
|
local prefix="${snap%-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]}"
|
||||||
|
if [[ "$prefix" == "$snap" ]]; then
|
||||||
|
# No timestamp suffix — skip (manually created snapshot)
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
snap_groups["$prefix"]+="${snap}"$'\n'
|
||||||
|
done < <(aptly snapshot list -raw -sort=name 2>/dev/null)
|
||||||
|
|
||||||
|
local dropped=0
|
||||||
|
for prefix in "${!snap_groups[@]}"; do
|
||||||
|
# Sort snapshots by name (timestamp order) and keep the newest KEEP_SNAPSHOTS
|
||||||
|
local snaps
|
||||||
|
snaps=$(echo -n "${snap_groups[$prefix]}" | sort)
|
||||||
|
local count
|
||||||
|
count=$(echo "$snaps" | wc -l)
|
||||||
|
local to_drop=$(( count - KEEP_SNAPSHOTS ))
|
||||||
|
|
||||||
|
if (( to_drop > 0 )); then
|
||||||
|
echo "$snaps" | head -n "$to_drop" | while IFS= read -r old_snap; do
|
||||||
|
[[ -z "$old_snap" ]] && continue
|
||||||
|
log "Dropping snapshot: ${old_snap}"
|
||||||
|
run_aptly snapshot drop "$old_snap" || \
|
||||||
|
log " !! Could not drop ${old_snap} (may be published or referenced)"
|
||||||
|
(( dropped++ )) || true
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
log "Running database cleanup..."
|
||||||
|
run_aptly db cleanup
|
||||||
|
|
||||||
|
log "=== Cleanup complete ==="
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# LIST — show current state
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
do_list() {
|
||||||
|
echo "==============================="
|
||||||
|
echo " Mirrors"
|
||||||
|
echo "==============================="
|
||||||
|
aptly mirror list 2>/dev/null || true
|
||||||
|
echo ""
|
||||||
|
echo "==============================="
|
||||||
|
echo " Snapshots"
|
||||||
|
echo "==============================="
|
||||||
|
aptly snapshot list -sort=time 2>/dev/null || true
|
||||||
|
echo ""
|
||||||
|
echo "==============================="
|
||||||
|
echo " Published Repositories"
|
||||||
|
echo "==============================="
|
||||||
|
aptly publish list 2>/dev/null || true
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# GPG key import helper
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
do_import_keys() {
|
||||||
|
log "=== Importing repository signing keys ==="
|
||||||
|
|
||||||
|
local keyserver="${GPG_KEYSERVER:-hkps://keyserver.ubuntu.com}"
|
||||||
|
|
||||||
|
log "Importing Debian GPG keys..."
|
||||||
|
for key in "${DEBIAN_GPG_KEYS[@]}"; do
|
||||||
|
gpg --no-default-keyring --keyring trustedkeys.gpg \
|
||||||
|
--keyserver "$keyserver" --recv-keys "$key" 2>&1 || \
|
||||||
|
log " !! Warning: could not import key ${key}"
|
||||||
|
done
|
||||||
|
|
||||||
|
log "Importing Ubuntu GPG keys..."
|
||||||
|
for key in "${UBUNTU_GPG_KEYS[@]}"; do
|
||||||
|
gpg --no-default-keyring --keyring trustedkeys.gpg \
|
||||||
|
--keyserver "$keyserver" --recv-keys "$key" 2>&1 || \
|
||||||
|
log " !! Warning: could not import key ${key}"
|
||||||
|
done
|
||||||
|
|
||||||
|
log "=== Key import complete ==="
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
cat <<USAGE
|
||||||
|
Usage: aptly-mirror.sh [-c <config>] <command>
|
||||||
|
|
||||||
|
Commands:
|
||||||
|
create Create all mirrors (first-time setup)
|
||||||
|
update Update mirrors, snapshot, merge, and publish
|
||||||
|
publish Snapshot current mirror state and publish/switch
|
||||||
|
cleanup Drop old snapshots and run aptly db cleanup
|
||||||
|
list List all mirrors, snapshots, and published repos
|
||||||
|
import-keys Import GPG keys for Debian and Ubuntu repositories
|
||||||
|
|
||||||
|
Options:
|
||||||
|
-c <path> Path to config file (default: auto-detected)
|
||||||
|
|
||||||
|
Configuration:
|
||||||
|
The config file is searched in this order:
|
||||||
|
1. -c <path> flag
|
||||||
|
2. \$APTLY_MIRROR_CONF environment variable
|
||||||
|
3. ${SCRIPT_DIR}/aptly-mirror.conf
|
||||||
|
4. /etc/aptly-mirror.conf
|
||||||
|
USAGE
|
||||||
|
}
|
||||||
|
|
||||||
|
main() {
|
||||||
|
if (( $# < 1 )); then
|
||||||
|
usage
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
local cmd="$1"; shift
|
||||||
|
case "$cmd" in
|
||||||
|
create) do_create ;;
|
||||||
|
update) do_update ;;
|
||||||
|
publish) do_publish ;;
|
||||||
|
cleanup) do_cleanup ;;
|
||||||
|
list) do_list ;;
|
||||||
|
import-keys) do_import_keys ;;
|
||||||
|
-h|--help|help)
|
||||||
|
usage
|
||||||
|
exit 0
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Unknown command: $cmd" >&2
|
||||||
|
usage
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
Reference in New Issue
Block a user