Mirroring with snapshots
Goal: build mirror of Debian wheezy, with security updates incorporated, done using snapshots to make package installation consistent on all the hosts.
Preparation
First, install aptly. Please verify that you have aptly
dependencies installed: packages bzip2, gnupg and gpgv.
Second, choose some user that would
be running all aptly commands (it shouldn’t be root user).
We would be signing published repositories with our GPG key, so if you don’t have GPG key yet, it’s time to create one:
$ gpg --gen-key
gpg (GnuPG) 1.4.12; Copyright (C) 2012 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
Your selection?
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)
Requested keysize is 2048 bits
Please specify how long the key should be valid.
0 = key does not expire
= key expires in n days
w = key expires in n weeks
m = key expires in n months
y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
"Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"
Real name: Andrey Smirnov
Email address: me@smira.ru
Comment: Signing repos
You selected this USER-ID:
"Andrey Smirnov (Signing repos) <me@smira.ru>"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
pub 2048R/6430156A 2014-10-09
Key fingerprint = 52EC DF0E EE8B 5DDF CC8F 7038 B9C8 F8B4 6430 156A
uid Andrey Smirnov (Signing repos) <me@smira.ru>
sub 2048R/E0C10001 2014-10-09
Default options should be fine, you need to backup your key and prepare revocation certificate, for example like described here.
Now we’re ready to start!
Creating Mirrors
Let’s assume that we need to mirror wheezy (current stable) Debian distribution for architecture amd64,
Only main component is required and this repository is targeted for servers, not desktops.
Choose one of the Debian servers to download packages from the
official list. Let it be ftp.ru.debian.org in this tutorial.
Create first mirror for main component:
$ aptly mirror create -architectures=amd64 -filter='Priority (required) | Priority (important) | Priority (standard)' wheezy-main http://ftp.ru.debian.org/debian/ wheezy main Looks like your keyring with trusted keys is empty. You might consider importing some keys. If you're running Debian or Ubuntu, it's a good idea to import current archive keys by running: gpg --no-default-keyring --keyring /usr/share/keyrings/debian-archive-keyring.gpg --export | gpg --no-default-keyring --keyring trustedkeys.gpg --import (for Ubuntu, use /usr/share/keyrings/ubuntu-archive-keyring.gpg) Downloading http://ftp.ru.debian.org/debian/dists/wheezy/InRelease... Downloading http://ftp.ru.debian.org/debian/dists/wheezy/Release... Downloading http://ftp.ru.debian.org/debian/dists/wheezy/Release.gpg... gpgv: Signature made Sat 12 Jul 2014 10:59:56 AM UTC using RSA key ID 46925553 gpgv: Can't check signature: public key not found gpgv: Signature made Sat 12 Jul 2014 11:04:06 AM UTC using RSA key ID 65FFB764 gpgv: Can't check signature: public key not found Looks like some keys are missing in your trusted keyring, you may consider importing them from keyserver: gpg --no-default-keyring --keyring trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 46925553 65FFB764 Sometimes keys are stored in repository root in file named Release.key, to import such key: wget -O - https://some.repo/repository/Release.key | gpg --no-default-keyring --keyring trustedkeys.gpg --import ERROR: unable to fetch mirror: verification of detached signature failed: exit status 2
aptly is complaining about missing keys in our trusted keyring, as it’s not possible to verify authencity of files being downloaded. Let’s follow the advice and import default Debian keyring:
$ gpg --no-default-keyring --keyring /usr/share/keyrings/debian-archive-keyring.gpg --export | gpg --no-default-keyring --keyring trustedkeys.gpg --import gpg: key 6430156A: public key "Andrey Smirnov (Signing repos) <me@smira.ru>" imported gpg: key 2A194991: public key "Andrey Smirnov <me@smira.ru>" imported gpg: key B98321F9: public key "Squeeze Stable Release Key <debian-release@lists.debian.org>" imported gpg: key 473041FA: public key "Debian Archive Automatic Signing Key (6.0/squeeze) <ftpmaster@debian.org>" imported gpg: key 65FFB764: public key "Wheezy Stable Release Key <debian-release@lists.debian.org>" imported gpg: key 46925553: public key "Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>" imported gpg: Total number processed: 6 gpg: imported: 6 (RSA: 6) gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
Now, let’s retry the mirror creation procedure:
$ aptly mirror create -architectures=amd64 -filter='Priority (required) | Priority (important) | Priority (standard) | nginx | postgresql | redis-server | memcached | ruby | golang' -filter-with-deps wheezy-main http://ftp.ru.debian.org/debian/ wheezy main Downloading http://ftp.ru.debian.org/debian/dists/wheezy/InRelease... Downloading http://ftp.ru.debian.org/debian/dists/wheezy/Release... Downloading http://ftp.ru.debian.org/debian/dists/wheezy/Release.gpg... gpgv: Signature made Sat 12 Jul 2014 10:59:56 AM UTC using RSA key ID 46925553 gpgv: Good signature from "Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>" gpgv: Signature made Sat 12 Jul 2014 11:04:06 AM UTC using RSA key ID 65FFB764 gpgv: Good signature from "Wheezy Stable Release Key <debian-release@lists.debian.org>" Mirror [wheezy-main]: http://ftp.ru.debian.org/debian/ wheezy successfully added. You can run 'aptly mirror update wheezy-main' to download repository contents.
Now the signature has been verified: files in the mirror are signed with Debian key (list of active keys).
The flag -filter= allows us to cut down number of packages to download. First part, Priority (required) | Priority (important) | Priority (standard) is essential “base” Debian system, then several (virtual) packages are added that I
need: nginx, postgresql, etc. Flag -filter-with-deps instructs aptly to include dependencies of
matching packages as well. If filter is not specified, all packages would be included in the mirror and that would
require more space and download size would be bigger.
Create also wheezy-updates and wheezy-security mirrors to get important updates:
$ aptly mirror create -architectures=amd64 -filter='Priority (required) | Priority (important) | Priority (standard) | nginx | postgresql | redis-server | memcached | ruby | golang' -filter-with-deps wheezy-updates http://ftp.ru.debian.org/debian/ wheezy-updates main ... $ aptly mirror create -architectures=amd64 -filter='Priority (required) | Priority (important) | Priority (standard) | nginx | postgresql | redis-server | memcached | ruby | golang' -filter-with-deps wheezy-security http://security.debian.org/ wheezy/updates main ...
Updating mirrors
Update wheezy-main mirror for the first time:
$ aptly mirror update wheezy-main Downloading http://ftp.ru.debian.org/debian/dists/wheezy/InRelease... Downloading http://ftp.ru.debian.org/debian/dists/wheezy/Release... Downloading http://ftp.ru.debian.org/debian/dists/wheezy/Release.gpg... gpgv: Signature made Sat 12 Jul 2014 10:59:56 AM UTC using RSA key ID 46925553 gpgv: Good signature from "Debian Archive Automatic Signing Key (7.0/wheezy) <ftpmaster@debian.org>" gpgv: Signature made Sat 12 Jul 2014 11:04:06 AM UTC using RSA key ID 65FFB764 gpgv: Good signature from "Wheezy Stable Release Key <debian-release@lists.debian.org>" Downloading & parsing package files... Downloading http://ftp.ru.debian.org/debian/dists/wheezy/main/binary-amd64/Packages.bz2... Applying filter... Packages filtered: 35933 -> 304. Building download queue... Download queue: 304 items (147.18 MiB) Downloading http://ftp.ru.debian.org/debian/pool/main/g/gnutls26/libgnutls26_2.12.20-8+deb7u2_amd64.deb... Downloading http://ftp.ru.debian.org/debian/pool/main/e/eglibc/locales_2.13-38+deb7u2_all.deb... Downloading http://ftp.ru.debian.org/debian/pool/main/p/perl/perl-modules_5.14.2-21+deb7u1_all.deb... Downloading http://ftp.ru.debian.org/debian/pool/main/v/vim/vim-tiny_7.3.547-7_amd64.deb... Downloading http://ftp.ru.debian.org/debian/pool/main/libt/libtext-iconv-perl/libtext-iconv-perl_1.7-5_amd64.deb... .... Mirror `wheezy-main` has been successfully updated.
Packages files have been downloaded to ~/.aptly/pool/, current mirror contents stored in the package database.
You can update mirror at any moment as required.
And, for two other mirrors:
$ aptly mirror update wheezy-security ... $ aptly mirror update wheezy-updates ...
Taking snapshots
It’s time take snapshots of the mirrors to preserve exact current mirror state. I will label snapshots
after mirror name, applying version for the main wheezy mirror and current date for frequently updated
security and updates mirrors:
$ aptly snapshot create wheezy-main-7.6 from mirror wheezy-main Snapshot wheezy-main-7.6 successfully created. You can run 'aptly publish snapshot wheezy-main-7.6' to publish snapshot as Debian repository. $ aptly snapshot create wheezy-updates-20141009 from mirror wheezy-updates Snapshot wheezy-updates-20141009 successfully created. You can run 'aptly publish snapshot wheezy-updates-20141009' to publish snapshot as Debian repository. $ aptly snapshot create wheezy-security-20141009 from mirror wheezy-security Snapshot wheezy-security-20141009 successfully created. You can run 'aptly publish snapshot wheezy-security-20141009' to publish snapshot as Debian repository.
Snapshots are created instantly and they don’t occupy any extra space: only list of packages is stored inside the snapshot.
Merging snapshots
Now we can merge snapshots into one, producing “wheezy with security updates applied”:
$ aptly snapshot merge -latest wheezy-final-20141009 wheezy-main-7.6 wheezy-updates-20141009 wheezy-security-20141009 Snapshot wheezy-final-20141009 successfully created. You can run 'aptly publish snapshot wheezy-final-20141009' to publish snapshot as Debian repository.
Flag -latest chooses merge strategy: package with latest version “wins”.
Let’s verify that merge was correct:
$ aptly package show -with-references 'Name (nginx)' Package: nginx Version: 1.2.1-2.2+wheezy2 Installed-Size: 87 Priority: optional Section: httpd Maintainer: Kartik Mistry <kartik@debian.org> Architecture: all Description: small, powerful, scalable web/proxy server MD5sum: fa5f86939aace1957b1fc846573cd2b1 SHA1: 05e521a9ec7854ed706aabcb0c0ea83fdee980d2 SHA256: 846905a046608aded2d29e05409ec3c0024717e6446cd0bdb544396a9652644d Tag: implemented-in::c, interface::daemon, network::server, network::service,protocol::http, role::program, use::proxying Description-md5: 19a4ea43e33eae4a46abf8a78966deb5 Source: Filename: nginx_1.2.1-2.2+wheezy2_all.deb Size: 61254 Depends: nginx-full | nginx-light Homepage: http://nginx.net References to package: mirror [wheezy-main]: http://ftp.ru.debian.org/debian/ wheezy snapshot [wheezy-main-7.6]: Snapshot from mirror [wheezy-main]: http://ftp.ru.debian.org/debian/ wheezy Package: nginx Version: 1.2.1-2.2+wheezy3 Installed-Size: 87 Priority: optional Section: web Maintainer: Kartik Mistry <kartik@debian.org> Architecture: all Description: small, powerful, scalable web/proxy server MD5sum: 25ae5234388762babbfbe3632dbdcc57 SHA1: 34d7ff8f24cc47960688ae84d6d98f92275ad453 SHA256: 516d33cf93f20ca070a203bafacc6f7ceb04bd3ae221d5a9a59f90e2ab828245 Depends: nginx-full | nginx-light Description-md5: 19a4ea43e33eae4a46abf8a78966deb5 Homepage: http://nginx.net Source: Filename: nginx_1.2.1-2.2+wheezy3_all.deb Size: 61374 References to package: mirror [wheezy-security]: http://security.debian.org/ wheezy/updates snapshot [wheezy-final-20141009]: Merged from sources: 'wheezy-main-7.6', 'wheezy-updates-20141009', 'wheezy-security-20141009' snapshot [wheezy-security-20141009]: Snapshot from mirror [wheezy-security]: http://security.debian.org/ wheezy/updates
New version of nginx package which came from wheezy-security mirror is now part of our final snapshot
wheezy-final-20141009.
PUBLISHING REPOSITORY
Final step: publishing our snapshot as Debian repository, ready to be consumed by apt-get:
$ aptly publish snapshot wheezy-final-20141009 ERROR: unable to publish: unable to guess distribution name, please specify explicitly
aptly complains that it can’t guess distribution name, as the source mirrors we had (main, security and updates) all have different distribution names. So we need to help aptly a bit by supplying distribution name:
$ aptly publish snapshot -distribution=wheezy wheezy-final-20141009 Loading packages... Generating metadata files and linking package files... Finalizing metadata files... Signing file 'Release' with gpg, please enter your passphrase when prompted: You need a passphrase to unlock the secret key for user: "Andrey Smirnov (Signing repos) <me@smira.ru>" 2048-bit RSA key, ID 6430156A, created 2014-10-09 Clearsigning file 'Release' with gpg, please enter your passphrase when prompted: You need a passphrase to unlock the secret key for user: "Andrey Smirnov (Signing repos) <me@smira.ru>" 2048-bit RSA key, ID 6430156A, created 2014-10-09 Snapshot wheezy-final-20141009 has been successfully published. Please setup your webserver to serve directory '/home/smira/.aptly/public' with autoindexing. Now you can add following line to apt sources: deb http://your-server/ wheezy main Don't forget to add your GPG key to apt with apt-key. You can also use `aptly serve` to publish your repositories over HTTP quickly.
The repository is published to ~/.aptly/public/ directory. You can start any HTTP server
to serve this directory as static files, or use aptly built-in webserver for testing:
$ aptly serve
Serving published repositories, recommended apt sources list:
# ./wheezy [amd64] publishes {main: [wheezy-final-20141009]: Merged from sources: 'wheezy-main-7.6', 'wheezy-updates-20141009', 'wheezy-security-20141009'}
deb http://debian:8080/ wheezy main
Starting web server at: :8080 (press Ctrl+C to quit)...
I can also visalize the objects I’ve created and dependencies:
$ aptly graph Loading mirrors... Loading local repos... Loading snapshots... Loading published repos... Generating graph... Rendered to PNG file: /tmp/aptly-graph128726742.png
This would result in picture like this one:
Using repository
First I need to import public part of the key that was used to sign the repository into trusted apt keyring on target machine.
Export the key on machine with aptly:
$ gpg --export --armor > my_key.pub
Copy my_key.pub to target machine and import it into apt keyring:
$ sudo apt-key add my_key.pub OK
It’s time to edit /etc/apt/sources.list. I comment out default sources and add address of my
first machine running aptly serve currently:
#deb http://ftp.ru.debian.org/debian/ wheezy main
#deb-src http://ftp.ru.debian.org/debian/ wheezy main
deb http://10.0.2.14:8080/ wheezy main
Repository is ready to be used:
$ sudo apt-get update Get:1 http://10.0.2.14 wheezy Release.gpg [490 B] Hit http://10.0.2.14 wheezy Release Hit http://10.0.2.14 wheezy/main amd64 Packages Ign http://10.0.2.14 wheezy/main Translation-en_US Ign http://10.0.2.14 wheezy/main Translation-en Fetched 490 B in 0s (33.1 kB/s) Reading package lists... Done $ sudo apt-get upgrade Reading package lists... Done Building dependency tree Reading state information... Done The following packages have been kept back: libc-bin libc6 openssh-client The following packages will be upgraded: apt apt-utils base-files bash dpkg gnupg gpgv krb5-locales libapt-inst1.5 libapt-pkg4.12 libgnutls26 libgssapi-krb5-2 libk5crypto3 libkrb5-3 libkrb5support0 libssl1.0.0 libxfont1 locales multiarch-support tzdata 20 upgraded, 0 newly installed, 0 to remove and 3 not upgraded. Need to get 19.6 MB of archives. After this operation, 74.8 kB of additional disk space will be used. Do you want to continue [Y/n]? y Get:1 http://10.0.2.14/ wheezy/main base-files amd64 7.1wheezy6 [78.7 kB] Get:2 http://10.0.2.14/ wheezy/main bash amd64 4.2+dfsg-0.1+deb7u3 [1,501 kB] ...
As this repository has been published from snapshot, it would never change until it is update to new snapshot. Good thing is that I can setup all my machine to use this repo and get identical set of packages installed. Bad thing is that I need to maintain and update my repo as updates are coming, but if I have many machines, the advantage of predictable upgrades outweighs the maintenance costs.
Upgrading repository
Several days after, when I update my mirrors I discover new security updates:
$ aptly mirror update wheezy-security ....
So it’s time to create new snapshot:
$ aptly snapshot create wheezy-security-20141019 from mirror wheezy-security Snapshot wheezy-security-20141019 successfully created. You can run 'aptly publish snapshot wheezy-security-20141019' to publish snapshot as Debian repository.
And do merge once again:
$ aptly snapshot merge -latest wheezy-final-20141019 wheezy-main-7.6 wheezy-updates-20141009 wheezy-security-20141019 Snapshot wheezy-final-20141019 successfully created. You can run 'aptly publish snapshot wheezy-final-20141019' to publish snapshot as Debian repository.
Now I can update my published repository by switching it to new snapshot:
$ aptly publish switch wheezy wheezy-final-20141019
Loading packages...
Generating metadata files and linking package files...
Finalizing metadata files...
...
Cleaning up prefix "." components main...
Publish for snapshot ./wheezy [amd64] publishes {main: [wheezy-final-20141019]: Merged from sources: 'wheezy-main-7.6', 'wheezy-updates-20141009', 'wheezy-security-20141019'} has been successfully switched to new snapshot.
apt-get would notice new packages on next update, and I can upgrade my target machine.
If run aptly graph again, the picture would be more interesting:
Maintenance
You can drop old snapshots as they’re not used anymore. This would allow to cleanup some files
with aptly db cleanup:
$ aptly db cleanup Loading mirrors, local repos and snapshots... Loading list of all packages... Deleting unreferenced packages (23)... Building list of files referenced by packages... Building list of files in package pool... Deleting unreferenced files (10)... Compacting database...
Package files are only deleted if no objects (snapshots, mirrors, local repositories) reference them.
Next tutorial is about pulling new version of packages from backports or 3rd party repos.