# svn-buildpackage - maintaining Debian packages with Subversion

## Introduction

### Purpose

This short document is only intended to give a short help in converting packages to Subversion management. It is primarily intended for developers not really familiar with Subversion or CVS management and/or converting from maintaining their packages using common tools (`dpkg-dev`, `devscripts`) only to version control system Subversion.

### Why a version control system?

But the first question may be: why use a version control system at all? Look at how the source is handled by the Debian package. First, we have the pure upstream source, which is often maintained by another person. The upstream author has his own development line and releases the source in snapshots (often called releases or program versions).

The Debian maintainer adds an own set of modifications, leading to an own version of the upstream package. The difference set between this two version finally ends in Debian's `.diff.gz` files, and this patchset is often appliable to future upstream versions in order to get the "Debian versions".

So the obvious way to deal with source upgrades/changes is using local copies, patch, different patchutils and scripts to automate all this, e.g. `uupdate`. However, it often becomes nasty and uncomfortable and there is no way to undo changes that you may do by mistakes.

At this point, the Subversion system can be used to simplify that work. It does the same things that you normaly would do by-hand but keeps it in an own archive (a repository). It stores the development lines of Upstream and Debian source, keeping them in different directories (different branches). The branches are wired internally (the VCS "knows" the history of the file and tracks the differences between the Upstream and Debian versions). When a new upstream version is installed, the differences between the old and new upstream versions and the Debian version are merged together.

You can create snapshots of your Debian version ("tag" it) and switch back to a previous state, or see the changes done in the files. You can store when commiting the file to the repository or place custom tags on the files ("properties") serving various purposes.

### Features

`svn-buildpackage` and other scripts around it has been created to do the following things:

- Keep Debian package under revision control, which means storing different versions of files in a Subversion repository.

- Allow easy walking back trough time using `svn` command.

- Easy retrieval of past versions.

- Keep track of upstream source versions and modified Debian versions.

- Easy installation of new upstream versions, merging the Debian changes into it when needed (similar to the `uupdate` program).

- Automated package building in clean environment, notifying about uncommited changes.

- Create version tags when requested to do the final build and update changelog when needed.

- allow co-work of multiple Debian developers on the same project.

- Auto-configure the repository layout, making it easy to use by people without knowing much about Subversion usage (mostly you need only the `add`, `rm` and `mv` commands of `svn`).

- Allow to store only the Debian specific changes in the repository and merge them into the upstream source in the build area (which nicely completes build systems like `dpatch` or `dbs`).

- If wished, keep the upstream tarballs inside of the repository.

### Contents overview

There are currently three scripts provided by the `svn-buildpackage` package:

- `svn-inject`: script used to insert an existing Debian package into a Subversion repository, creating the repository layout as needed.

- `svn-buildpackage`: exports the contents of the directory associated with the starting directory from the Subversion repository to the clean environment and build the package there.

- `svn-upgrade`: similar to `uupdate`, upgrades the trunk to a new upstream version, preserving and merging Debian specific changes.

### Popular repository layouts

There are different ways to store the packages in the repositories (or in multiple repositories at your choice). `svn-buildpackage` normally expects a directory structure similar to the one well described in the Subversion Book, which looks like:

#### Directory hierarchy example.

    packageA/
       trunk/
       branches/
       branches/upstream
       tags/

    projectB/
       trunk/
       branches/
       branches/developerFoo
       tags/

`packageA` above may be a typical upstream-based source package and a `projectB` may be a Debian native package with a separate branch created by developer Foo for his own experiments. See Subversion Book/Branches for more details about using Subversion branches.

Also note that Tags work quite differently to those in CVS. Subversion does not maintain magic tags associated with some files. Instead, it tracks the file state and moves, so Tagging something means creating a copy (inside of the Repository, harddisk-space efficient) of a certain version of the file set. So the Debian branch of the released package source is contained in `trunk/` and is tagged by copying (mirroring) the trunk tree to `tags/DEBIAN-REVISION`. The same happens for the upstream releases. In addition, the most recent upstream version is mirrored to `branches/upstream/current`. After few package upgrade cycles, the directory tree may look like:

#### Example branch directory hierarchy

    # svn ls -R file:///home/user/svn-repo/dev/translucency
    branches/
    branches/upstream/
    branches/upstream/0.5.9/
    branches/upstream/0.5.9/AUTHORS
    branches/upstream/0.5.9/COPYING
    ...
    branches/upstream/0.6.0/
    branches/upstream/0.6.0/AUTHORS
    branches/upstream/0.6.0/COPYING
    ...
    branches/upstream/current/
    branches/upstream/current/AUTHORS
    branches/upstream/current/COPYING
    ... same stuff as in 0.6.0 ...
    tags/
    tags/0.5.9-1/
    ...
    tags/0.5.9-1/debian/
    tags/0.5.9-1/debian/README.Debian
    ...
    tags/0.6.0-1/
    tags/0.6.0-1/AUTHORS
    ...
    tags/0.6.0-1/debian/
    tags/0.6.0-1/debian/README.Debian
    tags/0.6.0-1/debian/changelog
    ...
    trunk/
    trunk/AUTHORS
    trunk/COPYING
    ... trunk where 0.6.0-2 is beeing prepared ...

`svn-buildpackage` also supports the second repository layout suggested in the Subversion Book (function/package) but `svn-inject` prefers the one documented above. Both `svn-buildpackage` and `svn-upgrade` should be able to auto-detect the repository layout and the location of package files.

In theory, you do not have to follow that examples and place the `trunk`, `branches` and `tags` directory on the locations you like more. But `svn-buildpackage` and other scripts won't locate the files automaticaly so you will need to edit the `.svn/deb-layout` file in your working directory and set paths. See the old abstract about how auto-detection works and the config example.

Finally, the working directory structure on your development system may look like:

#### Example configuration

    dev/ # base directory, may be under version control or not
    dev/foo # trunk directories of various packages
    dev/bar # contents correspond to trunk, see above
    dev/tarballs # where "orig" tarballs are stored, may be under VC or not
    dev/build-area # where the packages are exported temporarily and built

## Getting started

Besides the packages that are installed by dependencies when you install `svn-buildpackage`, you may need `ssh` and the obligatory tool chain: `dpkg-dev`, `build-essential` and all the packages they pull into the system.

### Quick guide

Here is a quick guide for those who wish to build an existing package using an existing, publically available SVN repository. To create own repositories, skip this section and look for more details below.

- svn co <svn://server/path/to/trunk> package

- mkdir tarballs

- cp dir-where-you-keep-the-source/package_version.orig.tar.gz tarballs/

NOTE: you need the upstream source tarballs, stored under a usual `dpkg-source`-compatible filename in `tarballs/`

- cd `package`

- svn-buildpackage -us -uc -rfakeroot

### Basic svn usage

You need only few commands to start using `svn` with `svn-buildpackage` scripts. If you wish to learn more about it, read parts of the Subversion Book. The most used commands are:

- `add` -- put new files unto the revision control.

- `rm` -- remove the files from the repository.

- `mv` -- move files around, leting revision control system know about it.

- `commit` -- commit your changes to the repository.

- `resolved` -- tell `svn` that you have resolved a conflict.

- `diff` -- creates a "`diff -u`" between two versions, specified by file revision number or by date. See the `diff --help` output.

- `cat -r `revision`` -- useful to browse in some previous revision of the file.

If you are familiar with CVS you will probably know almost all you need.

### Creating Subversion repository

The main Subversion repository is easily created with:

#### Repository creation example.

    $ svnadmin create repo-directory

For our example, we choose the name `svn-deb-repo` and put it in `/home/`user``.

If you plan to keep many packages in the one repository including upstream tarballs, consider to put it on a hard disk with much free space and good performance (especially short disk access times) since the repository will grow and the filesystem may become fragmented over time.

### Using by multiple developers

Multiple developers with local access to the repository may share it using a common group. To do so, create a new group and add all developers to it. Run "chgrp -R sharedGroup repdir ; chmod -R g+s repdir" for the shared group and the repository directory. Now, on local access to this repository everybody will create files with the appropriate group setting. However, the developers will need to set a liberal umask before using `svn` (like "0022").

If somebody resists to do so, there is still a brute-force solution: fix the permissions with a post-commit script. However, this is an "unsound" solution and may lead to ALL KINDS OF PROBLEMS. MAKE SURE THAT YOU ARE AWARE OF THE POSSIBLE CONSEQUENCES BEFORE YOU OPEN THE PANDORA BOX. See Debian BTS for details. When you damage your repository, don't blame me and remember that there is "`svnadmin recover`".

#### post-commit hook example

    #!/bin/sh

    # POST-COMMIT HOOK
    # The following corrects the permissions of the repository files

    REPOS="$1"
    REV="$2"

    chgrp -R sharedGroup $REPOS
    # replace sharedGroup with your group
    chmod -R g+r $REPOS
    chmod -R g+w $REPOS

### SVN over SSH

To run Subversion over SSH, you basically need a shell on the target system and a subversion repository located there which is created following the description above. The repository must be configured for access by the system users of the remote system.

Assuming that your user name on the client system is the same as on the server side, there is not much to configure. Just change the protocol specification from `file://` to `svn+ssh://`remoteusername`@`server-hostname`` in all examples showed in this manual.

Note that during `svn-buildpackage` tools actions a lot of SSH calls can be made and so the user is asked for his login data. The regular method to deal with that is using an SSH key authentication method together with `ssh-agent` and `ssh-add` to cache the passphrase in memory. Another approach, which also brings a significant speed boost, is using a cached SSH connection. This can be done with a new feature of OpenSSH (see GCC SSH connection caching howto) or a third-party tool like `fsh`.

If you wish to use `fsh` over `ssh` you could specify a custom transport method in the subversion configuration. To do so, edit the file `~/.subversion/config` and add the section `[tunnels]` to it, following by your custom transport definition. Example:

#### Example of a custom ssh tunnel command.

    # personal subversion config with custom ssh tunnel command
    [tunnels]
    # SSH account on svn.d.o
    # compression is enabled in the ssh config
    deb = fsh -l <user>
    # SSH account for NQ intranet, set fix username
    nq = ssh -C -l zomb

You can use the new defined tunnels in a similar ways as described above but replace `svn+ssh` with `svn+`tunnelname``, so the final URL looks like:

    svn+deb://svn.debian.org/svn/myproject/ourpackage/trunk

### Anonymous access

You can allow outsiders to have anonymous (read-only) access using the `svnserve` program, as described in the Subversion documentation.

Another method is using HTTP/WebDAV with Apache2. More about a such setup can be found in the Subversion Book and the SubversionApache2SSL Howto. svn.debian.org is an example site granting anonymous access to some selected projects hosted there.

## Importing Debian packages

### Importing from existing source package files

The `svn-inject` utility is intended to import already packaged source packages into a new subdirectory of the repository, creating the repository layout as needed. Normally, it takes two arguments: the `.dsc` file of your package and the base URL of the Subversion repository.

#### svn-inject example

    svn-inject translucency_*dsc file:///tmp/z
    cp /tmp/translucency_0.6.0.orig.tar.gz /tmp/tarballs || true
    mkdir -p translucency/branches/upstream
    tar  -z -x -f /tmp/translucency_0.6.0.orig.tar.gz
    mv * current
    svn -q import -m"Installing original source version" translucency file:///tmp/z/translucency
    svn -m Tagging upstream source version copy file:///tmp/z/translucency/branches/
    upstream/current file:///tmp/z/translucency/branches/upstream/0.6.0 -q
    svn -m Forking to Trunk copy file:///tmp/z/translucency/branches/upstream/current file:///tmp/z/translucency/trunk -q
    dpkg-source -x /tmp/translucency_0.6.0-1.dsc
    dpkg-source: extracting translucency in translucency-0.6.0
    svn_load_dirs file:///tmp/z/translucency/trunk . *
    ...
    Running /usr/bin/svn propset svn:executable  initscript
    Running /usr/bin/svn propset svn:executable  debian/rules
    Running /usr/bin/svn propset svn:executable  mounttest.sh
    Running /usr/bin/svn propset svn:executable  mount.translucency
    Running /usr/bin/svn propget svn:eol-style base.h
    Running /usr/bin/svn propget svn:eol-style Makefile
    Running /usr/bin/svn propget svn:eol-style translucency.8
    Running /usr/bin/svn commit -m Load translucency-0.6.0 into translucency/trunk.

    Running /usr/bin/svn update
    Cleaning up /tmp/svn_load_dirs_jD7OenzVjI
    Storing trunk copy in /tmp/translucency.
    svn co file:///tmp/z/translucency/trunk /tmp/translucency -q
    svn propset svn:executable 1 debian/rules -q
    svn -m"Fixing debian/rules permissions" commit debian -q
    Done! Removing tempdir.
    Your working directory is /tmp/translucency - have fun!

If you omit the URL, `svn-inject` will try to use the URL of the current directory as base URL. I would not rely on this, however.

### On-Build-Time merging

A special feature of `svn-buildpackage` is so called mergeWithUpstream mode. Many projects do not want to keep the whole upstream source under revision control, eg. because of the large amount of required disc space and process time. Sometimes it makes sense to keep only the `debian/` directory any maybe few other files under revision control.

The task of exporting the source from repository and adding it to the upstream source before building becomes annoying after a little time. But the `svn-buildpackage` tools automate most of this work for you by switching to mergeWithUpstream mode if a special flag has been detected: the mergeWithUpstream (Subversion) property of the `debian` directory. `svn-buildpackage` will merge the trunk with upstream source on build time and `svn-upgrade` will only update the changed files in this case.

To enable this feature during the initial import of the source package, simply add the `-o` switch to the `svn-inject` call and it will prepare the source for with mergeWithUpstream mode. This reduces the set of files to those modified for Debian and set the mergeWithUpstream property.

But what, if you decide to switch to mergeWithUpstream-mode after the package has been injected? To do this, checkout the whole repository, remove the files not changed in the Debian package from both upstream source and Debian branch (`svn rm`) and set the mergeWithUpstream property on `debian` in the `trunk` directory with:

    svn propset mergeWithUpstream 1 debian

If you actually decide to stop using the mergeWithUpstream mode, unset the mergeWithUpstream property as follows:

    svn propdel mergeWithUpstream debian/

If you don't want to store the upstream sources of all your packages in the repository, you can pass the `--no-branches` switch to `svn-inject`, which will prevent `svn-inject` from creating a `branches` subdirectory.

#### `dpkg-source` format 3.0 support

`svn-buildpackage` can build packages using source format 3.0, including using `../tarballs/foo_1.2.3-1.orig.tar.bz2` in mergeWithUpstream mode. Ensure that `debian/source/format` exists and has been added to the local Subversion working copy.

If `debian/source/format` exists and contains 3.0 *, `svn-buildpackage` checks for a `.orig.tar.bz2` in the `tarballs` directory and uses that for the subsequent build.

##### checking source formats with `dpkg-source`

`dpkg-source` `--print-format` needs a little help to understand common layouts used with subversion. The command needs two directories - the current working directory is used to find the original tarball and the specified directory is used to locate `./debian/changelog`. Change to the directory containing the tarball before running `dpkg-source` `--print-format`:

    $ svn mkdir debian/source
    $ echo "3.0 (quilt)" > debian/source/format
    $ svn add debian/source/format
    $ pushd ../tarballs/
    $ dpkg-source --print-format ../trunk/
    3.0 (quilt)
    $ popd

#### Preparing patches in `dpkg-source` 3.0 packages

When using `3.0 (quilt)` and `mergeWithUpstream`, use `svn-do` to unpack the upstream source and then simply edit or patch the files necessary for the first patch. Then get `dpkg-source` to prepare the patches for you:

##### Letting `dpkg-source` prepare patches for `3.0 (quilt)`

$ dpkg-buildpackage -uc -us -S

Now use a serviceable name for the patch and change that name in the `series` file, exit the `svn-do` subshell for the results to be copied back into `trunk`. Repeat for subsequent patches.

### Using `svn-buildpackage` with native packages

A native package is designed to work with Debian rather than general GNU/Linux distributions, many depend directly on specific Debian behaviour or other Debian native tools. svn-buildpackage is one such native package.

With a native package, the contents of the `debian/` directory are included in the source tarball (which uses a plain `.tar.gz` suffix not the `.orig.tar.gz` suffix) and no `.diff.gz` is generated.

The principal marker for a native package is the Debian version string. An upstream package uses:

     foo (0.1.2-3)

A native package uses:

     foo (0.1.2)

#### Compiled native packages

Some native packages are compiled from source code and a lot of these packages use autotools like `./configure`, `autoreconf` and `make dist`. Such packages can use autotools support to generate a typical GNU or autotools style tarball with support for EXTRA_DIST and similar rules in `Makefile.am`. This allows the maintainers to not need to keep generated files (`configure`, `aclocal.m4`, `ltmain.sh` and `libtool`) in the subversion repository. With this support, the package can directly control which files are included into the source for the native package.

To build such packages with `svn-buildpackage`, mergeWithUpstream can be used even if the Debian version string indicates a native package. The upstream tarball is the one created by the `make dist` target and the maintainer can choose how to make that tarball available to `svn-buildpackage`:

##### Using a native tarball and mergeWithUpstream

Makefile.am

    all-local: Makefile
    	ln -sf $(PACKAGE)-$(VERSION).tar.gz $(PACKAGE)_$(VERSION).orig.tar.gz

(Yes, this is an artefact of using mergeWithUpstream. The `.orig.tar.gz` is not actually used - the Debian version string overrides the merge property.)

SVN properties:

    $ svn proplist ./debian/
    Properties on 'debian':
      mergeWithUpstream

##### Output of using mergeWithUpstream inside a native package

The tarball generated by `make dist` is located using the origDir property, in this example, set to the top level package directory where `make dist` would normally create it and where the symlink is also created, as above.

`svn-buildpackage` output:

(using langupdate as the example package)

    $ svn-buildpackage -us -uc
    	origDir: ./
    Complete layout information:
    	buildArea=/opt/working/emdebian/host/trunk/langupdate/build-area
    	origDir=./
    	tagsDir=/opt/working/emdebian/host/trunk/langupdate/tags
    	tagsUrl=svn+ssh://codehelp@buildd.emdebian.org/var/emdebian/svn/current/host/trunk/langupdate/tags
    	trunkDir=/opt/working/emdebian/host/trunk/langupdate/trunk
    	trunkUrl=svn+ssh://codehelp@buildd.emdebian.org/var/emdebian/svn/current/host/trunk/langupdate/trunk
    mergeWithUpstream mode detected, looking for .//langupdate_0.1.1.orig.tar.gz
    I: mergeWithUpstream property set, looking for upstream source tarball...
     tar --no-same-owner --no-same-permissions --extract --gzip --file /opt/working/emdebian/host/trunk/langupdate/trunk/langupdate-0.1.1.tar.gz --directory <1 more argument>
     mv /opt/working/emdebian/host/trunk/langupdate/build-area/tmp-0.15173904069616/langupdate-0.1.1 /opt/working/emdebian/host/trunk/langupdate/build-area/langupdate-0.1.1
    svn --force export /opt/working/emdebian/host/trunk/langupdate/trunk /opt/working/emdebian/host/trunk/langupdate/build-area/langupdate-0.1.1
    Export complete.

Note how the tarball (created by `make dist` and located by mergeWithUpstream due to the symlink) is unpacked and then the exported SVN is applied on top. This allows maintainers to test local changes using `--svn-ignore`.

#### Native packages not using autotools

The useNativeDist property.

When `make dist` is not available, there are still situations where generated files may need to be included into the source tarball of the native package. `svn-buildpackage` supports the useNativeDist SVN property (applied to the `./debian/` directory) which allows for customised `make native-dist` target to be run immediately after the svn export.

##### The useNativeDist property.

`svn-buildpackage` uses this feature to include the POT file to aid translators. The custom `make` rule needs to be defined in the top level `Makefile`:

    # adds the POT file to the source tarball
    native-dist: Makefile
    	po4a-build --pot-only

The native-dist target needs to be idempotent and able to run straight from the SVN export. Any changes made by running the target will be directly reflected in the source tarball. Ensure that no files are removed that would prevent the package being built as a normal Debian package, using only the resulting source package. (e.g. `cd` into the `build-area`, unpacking the `.dsc` with `dpkg-source -x` and rebuild the package, then remove the test directory.)

The net result is that the resulting source tarball includes the `po/svn-buildpackage.pot` generated by the make native-dist target without needing to add the generated POT file to SVN (and requiring repeated commits each time the POT file is timestamped).

## Common tasks

### Checkout

`svn-inject` will do the initial checkout for you. If you need another working copy, run

#### `svn-inject` initial checkout

    svn co protocol://repository-base-url/yourpackage

### Building the package

Change to your `trunk` directory and run:

#### `svn-buildpackage`

    $ svn-buildpackage -us -uc -rfakeroot

You may recognise the options above -- they are passed directly to the build command (`dpkg-buildpackage` by default). Normally, the build is done in another directory (exporting the source with `cp-la-like` method). If you want the resulting packages to be placed in the directory above, use the `--svn-move` option. To run Lintian after the build, use `--svn-lintian` option. More options are described in the manpage.

To build in a chroot using `pdebuild`, use:

#### pdebuild example

    svn-buildpackage --svn-builder pdebuild

For frequently used commands, use an alias in your `~/.bashrc` file:

#### Useful aliases

    alias svn-bp='svn-buildpackage -rfakeroot -D --svn-noautodch --svn-builder debuild'
    alias svn-bpi='svn-buildpackage -rfakeroot -D --svn-noautodch --svn-builder debuild --svn-ignore-new'
    alias svn-bpir='svn-buildpackage -us -uc -rfakeroot -D --svn-noautodch --svn-builder debuild --svn-ignore-new --svn-rm-prev-dir'
    alias svn-bpr='svn-buildpackage -us -uc -rfakeroot -D --svn-noautodch --svn-builder debuild --svn-rm-prev-dir'
    alias svn-pbp='svn-buildpackage --svn-noautodch --svn-builder pdebuild'

### Working with source

Every time when you add or modify something, `svn-buildpackage` won't let you proceed unless suspicious files are in the clean state (unless you use the `--svn-ignore` switch). You use the commands described in "basic svn usage" to register the new files (or move or delete the old ones) and commit the changes to the repository.

### Handling new upstream versions

Upgrading with new upstream version normally happens in two steps:

- the `current` tree in the upstream branch is upgraded with the source from the new upstream package (the old version is kept in repository in `branches/upstream/`oldVersion``).

- the version in `trunk/` becomes upgraded by merging the changes between the upstream versions into the `trunk/` directory.

The script `svn-upgrade` (formerly `svn-uupdate`) does both things for you and also creates a new changelog entry. The first step is done internally by using a third party script (`svn_load_dirs`, see Subversion book for documentation), the second step is done with the `merge` command of `svn`. Just run `svn-upgrade` from you local working directory (which corresponds the `trunk/` checkout product).

After running `svn-upgrade` some files may be in conflicting state. This is naturally happens if you have modified some files in the upstream package and now upstream did something similar on the same positions so `svn merge` was confused.

When `svn-upgrade` complains about files in conflicting state, fix them manually. When done, use the `svn resolved` command to mark them as clean and `svn commit` to update the repository.

### Finalizing the Revision

When you are ready to upload a new revision of your package, everything builds fine, the changelog is cleaned up and the package is tested, you can do the final build and tag the end version. To do so, add `--svn-tag` switch and after the package is built, it will be tagged (by creating a copy of the `trunk/` directory as said above).

## Command reference

&svn-buildpackage; &svn-inject; &svn-upgrade; &svn-do; &uclean;

## Further documentation

### Various links

- Subversion Homepage: http://subversion.tigris.org

- The Subversion Book: http://svnbook.red-bean.com/

- Subversion vs. CVS and others: http://better-scm.berlios.de/

### Copyright

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

A copy of the GNU General Public License is available as /usr/share/common-licenses/GPL in the Debian GNU/Linux distribution or on the World Wide Web at http://www.gnu.org/copyleft/gpl.html. You can also obtain it by writing to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.