Tue, 20 Dec 2011

Host a Debian Package Repository on S3

The following steps can be used to create a debian package repository easily and host it on Amazon Web Services S3.

First, install reprepro, which will create the repository file structure from .deb packages. Also install s3cmd to sync a local copy of the repository to s3.

$ sudo apt-get install reprepro s3cmd

Create a directory for the repository and a conf sub-directory.

$ mkdir -p /path/to/my-repo/conf

Create the config file, conf/distributions, describing the repository. Setting Codename, Components, and Architectures are sufficient to get started. If your packages are specific to a Debian distribution (or other Debian-based distro like Ubuntu), you can set Codename to the code name of the distro, e.g. squeeze. It should not be set to stable, testing, or unstable; these can be set in the Suite option. See reprepro(1) for more details.

Codename: example
Components: main
Architectures: i386 amd64

Add a package to the repo using reprepro.

$ reprepro -b /path/to/my-repo includedeb example /path/to/package.deb 

Here are the contents of the repo after adding one package:

my-repo/
my-repo/pool
my-repo/pool/main
my-repo/pool/main/m
my-repo/pool/main/m/mypackage
my-repo/pool/main/m/mypackage/mypackage_1.0_all.deb
my-repo/dists
my-repo/dists/example
my-repo/dists/example/main
my-repo/dists/example/main/binary-i386
my-repo/dists/example/main/binary-i386/Packages.gz
my-repo/dists/example/main/binary-i386/Release
my-repo/dists/example/main/binary-i386/Packages
my-repo/dists/example/main/binary-amd64
my-repo/dists/example/main/binary-amd64/Packages.gz
my-repo/dists/example/main/binary-amd64/Release
my-repo/dists/example/main/binary-amd64/Packages
my-repo/dists/example/Release
my-repo/conf
my-repo/conf/distributions
my-repo/db
my-repo/db/packages.db
my-repo/db/release.caches.db
my-repo/db/checksums.db
my-repo/db/version
my-repo/db/references.db
my-repo/db/contents.cache.db

Configure s3cmd with your AWS credentials:

$ s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key [XXXXXXXXXXXXXXXXXXXX]:
Secret Key [XX+XXXXXX+XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]:

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [Yes]:

You can leave the encryption password blank; you don't want to encrypt the files in the repository.

Create a bucket in S3 and sync your repository to the bucket. The bucket name must be globally unique. You will get an error if you use the name of an existing S3 bucket.

$ s3cmd mb s3://my-repo/
$ s3cmd --verbose --acl-public --delete-removed  sync /path/to/my-repo/ s3://my-repo/

Note the trailing slash after /path/to/my-repo. Without it, the my-repo folder itself will be created in your bucket.

Add the repository to your sources.list:

deb http://my-repo.s3.amazonaws.com example main

Now you can install packages from the repository:

$ sudo apt-get update
$ apt-cache policy mypackage
mypackage:
  Installed: (none)
  Candidate: 1.0
  Version table:
     1.0 0
        500 http://my-repo.s3.amazonaws.com/ example/main amd64 Packages

tech | Permanent Link

The state is that great fiction by which everyone tries to live at the expense of everyone else. - Frederic Bastiat