aptly/README.md

# aptly-mirror.sh

A comprehensive bash script to create and maintain aptly mirrors for Debian and Ubuntu repositories. Automate the mirroring of package repositories, create time-based snapshots, and publish merged snapshot sets to serve to clients.

## Features

- **Multi-distribution support**: Mirror Debian 11/12/13 and Ubuntu 20.04/22.04/24.04/26.04
- **Flexible mirroring**: Mirror main, updates, and security streams separately
- **Snapshot management**: Create dated snapshots of mirrors and merge them
- **Automatic publishing**: Publish snapshots as APT repositories for client consumption
- **Retention policy**: Keep only N recent snapshot sets, automatically cleanup old ones
- **GPG verification**: Verify repository signatures during download and signing
- **Bandwidth control**: Optional download speed limiting
- **Syslog integration**: All operations logged to syslog for monitoring

## Getting Started

### Prerequisites

- `aptly` (>= 1.4.0): Install via your system package manager or from [aptly.info](https://www.aptly.info/)
- `gpg`: For GPG key management (usually pre-installed)
- Bash 4+
- Sufficient disk space for mirror storage (50GB+ recommended depending on architecture count)

### Installation

1. **Clone or download the script**:
   ```bash
   git clone <repository> aptly-mirror
   cd aptly-mirror
   ```

2. **Create configuration**:
   ```bash
   cp aptly-mirror.conf.example aptly-mirror.conf
   ```

3. **Edit configuration** for your environment:
   ```bash
   nano aptly-mirror.conf
   ```

   Key settings:
   - `ARCHITECTURES`: CPU architectures to mirror (e.g., `"amd64"` or `"amd64 arm64"`)
   - `APTLY_ROOT`: Where aptly stores its database and mirrors (default: `~/.aptly`)
   - `PUBLISH_ROOT`: Where published repos are served (default: `~/.aptly/public`)
   - `KEEP_SNAPSHOTS`: Number of old snapshot sets to retain (default: `2`)

4. **Make the script executable**:
   ```bash
   chmod +x aptly-mirror.sh
   ```

5. **Import GPG keys** (recommended for production):
   ```bash
   ./aptly-mirror.sh import-keys
   ```

6. **Create initial mirrors**:
   ```bash
   ./aptly-mirror.sh create
   ```
   This may take 10-30 minutes depending on your network and disk speed.

7. **Create initial snapshots and publish**:
   ```bash
   ./aptly-mirror.sh publish
   ```

### Verify Setup

List all mirrors, snapshots, and published repositories:
```bash
./aptly-mirror.sh list
```

Check syslog for operation logs:
```bash
journalctl -t aptly-mirror -f
```

## Manual

### Commands

#### `create` — Initialize all mirrors

Create mirrors for all configured Debian and Ubuntu releases. Mirrors are created empty and must be populated with the first `update` command.

```bash
./aptly-mirror.sh create
```

**Output**:
- Creates mirrors for each release:
  - `debian-{codename}-main`
  - `debian-{codename}-updates`
  - `debian-{codename}-security`
  - Similar pattern for Ubuntu

**Time**: 1-2 minutes

#### `update` — Fetch latest packages and republish

Updates all mirrors to the latest packages, creates timestamped snapshots, merges them, and publishes a fresh snapshot set. This is the main command to run regularly.

```bash
./aptly-mirror.sh update
```

**Process**:
1. Updates all existing mirrors (downloads new packages)
2. Creates dated snapshots (e.g., `debian-bullseye-main-20260408123045`)
3. Merges main + updates + security snapshots for each release
4. Publishes or switches the published repository to the merged snapshot

**Time**: 5-30 minutes (depends on network, disk speed, and download limit)

**Output example**:
```
Repositories are published under /var/aptly/public/

Client sources.list entries:
  deb http://<server>/debian bullseye main contrib non-free
  deb http://<server>/debian bookworm main contrib non-free non-free-firmware
  deb http://<server>/ubuntu focal main restricted universe multiverse
```

#### `publish` — Snapshot and publish current state

Creates fresh snapshots of current mirrors without updating them. Useful when you want to take a snapshot without fetching new packages.

```bash
./aptly-mirror.sh publish
```

**Use cases**:
- Create a point-in-time snapshot after verifying upstream stability
- Re-publish without re-fetching packages

#### `cleanup` — Remove old snapshots and optimize database

Removes old snapshot sets beyond the `KEEP_SNAPSHOTS` retention limit and runs aptly's database cleanup.

```bash
./aptly-mirror.sh cleanup
```

**Safety**:
- Only removes snapshots that are not currently published
- Never removes the active published snapshot
- Safe to run after each update

#### `list` — Show all mirrors, snapshots, and published repos

Display the current state of all mirrors, snapshots, and published repositories.

```bash
./aptly-mirror.sh list
```

#### `import-keys` — Import GPG signing keys

Imports GPG keys for verifying Debian and Ubuntu repository signatures. Requires network access to GPG keyserver.

```bash
./aptly-mirror.sh import-keys
```

**Notes**:
- Only needs to run once during setup
- Requires `~/.gnupg/` to exist
- Keyserver can be customized via `GPG_KEYSERVER` in config

### Options

#### `-c <path>` — Specify config file

```bash
./aptly-mirror.sh -c /etc/aptly-mirror.conf update
```

Config file search order (if `-c` not specified):
1. `$APTLY_MIRROR_CONF` environment variable
2. `./aptly-mirror.conf` (same directory as script)
3. `/etc/aptly-mirror.conf`

#### `-h`, `--help` — Show usage

```bash
./aptly-mirror.sh --help
```

### Configuration

All settings are in `aptly-mirror.conf`. See the included example for detailed comments.

**Key settings**:

| Setting | Default | Description |
|---------|---------|-------------|
| `ARCHITECTURES` | `amd64` | Architectures to mirror (space-separated) |
| `KEEP_SNAPSHOTS` | `2` | Number of old snapshot sets to retain |
| `DOWNLOAD_LIMIT` | `0` | Speed limit in KiB/s (0 = unlimited) |
| `IGNORE_SIGNATURES` | `0` | Skip GPG verification (not recommended) |
| `SKIP_SIGNING` | `0` | Skip GPG signing when publishing |
| `APTLY_ROOT` | `~/.aptly` | Aptly database and mirrors location |
| `PUBLISH_ROOT` | `~/.aptly/public` | Published repositories root |

### Logging

All output is logged via syslog with tag `aptly-mirror`. View logs with:

```bash
# Follow in real-time
journalctl -t aptly-mirror -f

# View recent entries
journalctl -t aptly-mirror -n 50

# Search for errors
journalctl -t aptly-mirror | grep ERROR
```

Older systemd-free systems can check `/var/log/syslog`:
```bash
grep aptly-mirror /var/log/syslog
```

## Sample Crontab

Run weekly updates every Sunday at 2 AM:

```bash
# Edit crontab
crontab -e

# Add this line:
0 2 * * 0 /home/dak/Code/aptly/aptly-mirror.sh update
```

**Full recommended setup** with logging and error notifications:

```bash
# Weekly update and cleanup
0 2 * * 0 /home/dak/Code/aptly/aptly-mirror.sh update && /home/dak/Code/aptly/aptly-mirror.sh cleanup

# Check syslog for errors (runs at 3 AM, 1 hour after update)
0 3 * * 0 journalctl -t aptly-mirror -n 100 | grep -i error && echo "aptly-mirror errors detected" | mail -s "aptly-mirror Alert" admin@example.com
```

**Multiple updates per week**:

```bash
# Every 12 hours (twice daily)
0 2,14 * * * /home/dak/Code/aptly/aptly-mirror.sh update

# Cleanup on Sundays
0 4 * * 0 /home/dak/Code/aptly/aptly-mirror.sh cleanup
```

**For systemd systems** (alternative to cron):

Create `/etc/systemd/system/aptly-mirror.timer`:
```ini
[Unit]
Description=Weekly aptly mirror update
After=network-online.target
Wants=network-online.target

[Timer]
OnCalendar=Sun *-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target
```

Create `/etc/systemd/system/aptly-mirror.service`:
```ini
[Unit]
Description=Update aptly mirrors
After=network-online.target

[Service]
Type=oneshot
ExecStart=/home/dak/Code/aptly/aptly-mirror.sh update
ExecStartPost=/home/dak/Code/aptly/aptly-mirror.sh cleanup
User=aptly
StandardOutput=journal
StandardError=journal
```

Enable and start:
```bash
sudo systemctl daemon-reload
sudo systemctl enable aptly-mirror.timer
sudo systemctl start aptly-mirror.timer
```

## Troubleshooting

### "Mirror already exists"
This is normal on repeated runs. The script skips existing mirrors.

### GPG signature verification failures
Either:
- Import keys: `./aptly-mirror.sh import-keys`
- Or skip verification (not recommended): set `IGNORE_SIGNATURES=1` in config

### Disk space exhausted
Monitor with:
```bash
du -sh ~/.aptly/
```

Reduce `KEEP_SNAPSHOTS` in config, or clean up old mirrors manually via `aptly mirror drop`.

### Network timeout during update
Set a `DOWNLOAD_LIMIT` to avoid overwhelming the network, or run during off-peak hours.

### Can't write to log
Ensure log directory exists and is writable. Check that `LOG_DIR` in config is accessible.

## AI Attribution

AIA EAI Hin R Claude Code v1.0

This work was entirely AI-generated. AI was prompted for its contributions, or AI assistance was enabled. AI-generated content was reviewed and approved. The following model(s) or application(s) were used: Claude Code.

## License

MIT