path: root/debian/pkg-zsh-workflow.org

* Branches

  This is a quick overview of the branches that are being used for
  debian's zsh package maintenance. This part does not go into much
  detail on how the workflow works. For details about that, see the
  `Workflow' part below.


** upstream

   The upstream sources from <git://zsh.git.sf.net/gitroot/zsh/zsh>.


** debian

   The debian changes for the debian `zsh' package.


** others

   These branches are rather optional and not required for basic
   maintenance for of the `zsh' package.


*** master

    This is the old repository's main branch. Kept for historical
    reasons.


*** debian-beta

    This could basically do *exactly* the same thing as `debian' does,
    but just merge more often. Such a branch *could* then be used to
    keep the packaging information for `zsh-beta'.


* Workflow

  This is a diagram, that outlines the proposed workflow. Basically,
  the `debian' branch walks alongside the `upstream' branch and
  upstream changes get merged into `debian' at defined points (for the
  non-beta zsh package, that should be upstream commit tags).


#+BEGIN_EXAMPLE
            * debian/4.3.11-4
            |
            |   * debian/4.3.11-5
            |   |
            |   |       * debian/4.3.12-1
            |   |       |
            |   |       |   * debian/4.3.12-2
            |   |       |   |
            |   |       |   |           * debian/4.3.13-1
            |   |       |   |           |
            |   |       |   |           | * debian/4.3.13-2
            |   |       |   |           | |
            |   |       |   |           | |   * debian/4.3.13-3
            |   |       |   |           | |   |
            |   |       |   |           | |   |
          a-b-c-d---e-f-g-h-i-------j-k-l-m-n-o-p   `debian'
         /         /               /
      A-B-C-D-E-F-G-H-I-J-K-L-M-N-O-P-Q             `upstream'
        |         |               |
        |         |               |
        |         |               * zsh-4.3.13
        |         * zsh-4.3.12
        |
        * zsh-4.3.11
#+END_EXAMPLE


** Working on the package


*** Changes

    Every change to the source (non-debian-dir) should be done by
    adding quilt patches. Permanent patches (patches that would be
    kept in 4.3.12-1) should be named specially ("deb_*" comes to
    mind).

    To keep order for cases in which a vast number of patches are
    required, any quilt patch needs to be numbered according to the
    order in which "quilt series" would list them. The format looks
    like this:

       - NNNN_<name>.diff
       - deb_NNNN_<name>.diff

    Where `N' is a digit from 0 to 9. Counting starts at `0000'.

    Debian-specific (i.e. deb_NNNN_<name>.diff) patches should be
    applied after patches targeted towards upstream since the patches
    targeted for upstream should not be against debian-specific code.

    Every patch should have a short annotation above the actual
    unified diff content, outlining the status of the patch. It is
    especially important to know whether a change is already applied
    upstream because if so, the patch in question must be dropped
    before merging a new upstream release into the `debian' branch.


**** Dealing with quilt's .pc directory

     When quilt works, it keeps track of what it does in a directory
     by the name `.pc'. This directory will show up in the output of
     "git status" etc. It should *never* *ever* by committed to the
     git repository at any point.

     We cannot add the directory to `.gitignore' because we would
     change the zsh source directly instead of via `debian/patches'.

     To deal with the directory, there is another git-facility, that
     we can use for fun and profit.

     % echo .pc/ >> .git/info/exclude

     Now git will ignore quilt's directory for this
     repository. Unfortunately, this has to be done for each
     checkout. Luckily, this only has an impact for people who want to
     work on `pkg-zsh'. Everyone else can savely ignore the directory.


**** Example of adding a fix via a quilt patch

     Let's say, there is an issue `#12345' which can be fixed by
     patching `Functions/Misc/colors'. Here is how you could add that
     patch (assuming clean git working directory and the topmost patch
     is 0002-foo.diff):

#+BEGIN_EXAMPLE
     ## First, add all existing non-debian patches, so the new one is
     ## added on top.

     % quilt push 0002-foo.diff

     ## Add the new patch (say, the topmost patch is 0002-foo.diff).
     % quilt new 0003-fix-colors.diff

     ## Tell quilt which files are going to be changed.
     % quilt add Functions/Misc/colors

     ## Import the fix (manually by editor or by patching).
     % vi Functions/Misc/colors

     ## Refresh the patch
     % quilt refresh

     ## Pop all patches again to clean up the upstream source.
     % quilt pop -a

     ## Commit the new patch and the changes `series' file to git.
     % git add debian/patches/0003-fix-colors.diff
     % git add debian/patches/series
     % git commit -a -m'Fixing foo in colors function (Closes: #12345)'
#+END_EXAMPLE

     That's all.


**** Cherry-picking patches from upstream into quilt

     When there is an existing patch (e.g. from upstream's git
     repository), the above can be largely automated if the patch
     applies cleanly to the current state of the debian branch.

     The `./debian/patch2quilt' helper script takes care of that
     task. It's called like this:

#+BEGIN_EXAMPLE
     % ./debian/patch2quilt ../existing.diff 0023-new-quilt.diff
#+END_EXAMPLE

     Here "../existing.diff" is the file containing the existing patch
     and "0023-new-quilt.diff" is the name of the to-be-added quilt
     series patch (make sure its nameing is in line with the
     established conventions).

     The exact operation of the script is described at the top of the
     script file. There are a few things to keep in mind:

       - At the end of successful operation you are dropped into an
         editor which gives you the opportunity to add annotations at
         the top of the patch file (like if the patch in question is
         included upstream already).

       - Never *ever* run the script when you got uncommitted changes
         in the worktree, which you don't plan on losing. The worktree
         will be cleaned and reset first thing in the script.

       - As an extension of the previous point, don't put the existing
         patch you're planing to import into the git working tree. It
         would be wiped away, too.

       - Patches from upstream will likely include changes to the
         ChangeLog file. Those changes will probably not apply
         cleanly, which will break the scripts execution. Just open
         the existing patch and delete all hunks that do changes in
         ChangeLog.

     When the script finishes (after you exit your editor), it will
     suggest how to commit the newly intoduced patch. Season to taste.


**** Keeping the local repository clean

     Before making changes of any kind, it should be made sure that
     the local repository you are working on is in a clean state. To
     clean up the local repository do this:

#+BEGIN_EXAMPLE
     % git clean -xdf
     % git reset --hard
#+END_EXAMPLE

     That will make sure that any non-tracked files are removed and
     that any changes in tracked files are reverted. The latter will
     also make sure that no parts of a quilt patch-queue are still
     applied.


*** Releases

    When a change justifies the release of a new package version, the
    debian/changelog file should be updated and the resulting commit
    should be tagged debian/<zsh-version>-n+1.


*** Updating debian/changelog

    This file should *not* be updated manually. The changes should be
    inserted by running the `git-dch' tool from the package
    `git-buildpackage' before a new release is about to be made.

    Changelog entries should be prefixed by a "[hashsum] " string,
    where `hashsum' is a string that represents the first eight
    characters of commit the changelog entry was generated from.

    Also, if multiple authors are involved in a changelog entry-set,
    each author should only appear once in the series with all her/his
    changes listed below her/him in chronological order.

    Here is a command line which will result in the desired changelog
    format:

#+BEGIN_EXAMPLE
    % git-dch --debian-branch=debian --id-length=8 --multimaint-merge
#+END_EXAMPLE

    If you absolutely *must* make changelog entries by other means, you
    should make sure that you prefix any resulting commits with
    "[dch-ignore] ", so those commits can be weeded out easily.

    There is a helper script "debian/do-dch" which takes care of all
    formatting options as well as the "[dch-ignored] " weeding. The
    script should be used unless there is a good reason not to.


** Transitioning to a new upstream version

   When upstream releases a new version, we should follow these steps:


*** Removing non deb_* quilt patches

    All non deb_* patches should be removed from `debian/patches'
    directory, unless they fix an issue that was *not* addressed
    upstream and is therefore missing from upstream's code base.

    If such a change should prove to be required to be kept with the
    package permanently (e.g. because upstream refuses to apply the
    patch), the patch should eventually be renamed to match the
    "deb_*" nameing convention.


*** Merging `upstream' into `debian'

    After the `debian/patches' directory was cleaned up in the
    previous step, merging `upstream' into `debian' should generally
    lead to a working package again.

    If old patches were still around, that could lead to conflicts
    when those would be applied during the build process.


*** Update the autotools files for the new release

    This should be done to make sure, the build systems is always kept
    up to date. For convenience, there is a script that automates this
    process:

#+BEGIN_EXAMPLE
    % ./debian/at2quilt
#+END_EXAMPLE

    After that, the involved quilt patches are updated, but not yet
    committed to the git repository to allow for review.


**** Updating autotools files whenever input files are changing

     Sometimes, it will be necessary to backport changes to autotools
     input files (such as `configure.ac'). As with any other change,
     such a task should be done by adding a patch to the packages
     quilt queue.

     Obviously, if an input file changes, the autotools related
     patches for the package need to be updated. Again, the `at2quilt'
     script from the `debian/' subdirectory helps. The only difference
     to the way it was used before is that you need to tell the script
     which patches in the patch-queue are relevant for autotools. Say
     there is a patch `0017-autotools-fix-gnu-hurd.diff' which alters
     `configure.ac', here is what would need to be done:

#+BEGIN_EXAMPLE
     % ./debian/at2quilt debian/patches/0017-autotools-fix-gnu-hurd.diff
#+END_EXAMPLE

     If there is more than one patch `at2quilt' need to consider, list
     all of them in the order in which `quilt' would apply them (if in
     doubt, ask "quilt series").

*** Insert initial changelog for the new upstream release

    `git-dch' seems to be in trouble with non-linear histories. Therefore
    we introduced a small helper script that will help `git-dch' to a
    linear history again.

    Basically, you after merging the upstream release tag into the debian
    branch, you'll be left with an history that looks something like
    this:

#+BEGIN_EXAMPLE
    *    at2quilt: Updating autotools patches
    M    Merge commit 'zsh-4.3.13' into debian
    |`*  unposted: released 4.3.13
    | *  ...
    | *  ... lots of other upstream commits ...
    | *  ...
    * |  Removing upstream patches due to new release
    * |  Last debian/4.3.12-* commit
    * |  ...
    * |  ... lot's of other debian/4.3.12-* commits
    * |  ...
    M´   Merge commit 'zsh-4.3.12' into debian
    |`*  unposted: released 4.3.12
    ...  older history
#+END_EXAMPLE

    And what you really want added to debian/changelog is the "atquilt:
    Updating autotools patches" and the "Removing upstream patches due to
    new release" commits. You need to figure out the sha1 sums of the
    commits and then call this:

#+BEGIN_EXAMPLE
    % ./debian/urcl -p=zsh -v=4.3.13-1 b495ba1e f575f568
#+END_EXAMPLE

    ...where "4.3.13-1" is the version of the upcoming debian package and
    "b495ba1e" and "f575f568" are the sha1 sums of the wanted commits.

    At the end the script will drop you into an editor pointed at the
    changelog file so you can sanity-check the generated output.

    At this point it would also make sense to add a line like this:

#+BEGIN_EXAMPLE
  * New upstream release
#+END_EXAMPLE

    or something like this if the release fixes outstanding bugs:

#+BEGIN_EXAMPLE
  * New upstream release (Closes: #1234567890)
#+END_EXAMPLE

    When creating a commit with these changelog changes, make sure you
    prefix the commit message with "[dch-ignore] " so it doesn't come up
    in later git-dch runs.


*** Update debian/gbp.conf

    The debian/gbp.conf file contains a reference pointing to the upstream
    branch. Therefore the upstream-branch configuration inside debian/gbp.conf
    needs to be adjusted for new upstream releases.


*** Fix outstanding bug

    If *any* outstanding bugs are known, they should be fixed before
    releasing a new package. Obviously, if any of the known bugs are
    very hard to fix and the issue is not serious in nature, releasing
    the package with the issue may be more important.

    Again, all changes to non `debian/*' files should be done via
    quilt patches.


*** Verify that the package builds

#+BEGIN_EXAMPLE
    % git reset --hard
    % git clean -xdf
    % QUILT_PATCHES=debian/patches
    % export QUILT_PATCHES
    % quilt push -a
    % ./configure
    % make all test
#+END_EXAMPLE


*** Update changelog again for the release

    The `do-dch' helper script should be used to do this. It wraps
    git-dch with appropriate options and weeds out any commits that are
    prefixed with "[dch-ignore] ". All options to the script are turned
    over to git-dch and at least `--since=...' should be used.

    At this particular point the sha1 of the previous initial changelog
    update commit would be a good idea. Also "-R" to tell git-dch to
    prepare the changelog for an actual commit. So:

#+BEGIN_EXAMPLE
    % ./debian/do-dch --since=1234deadbeef -R
#+END_EXAMPLE

    You'll be dropped into an editor again to double check the generated
    changelog.


*** Tag debian/<new-zsh-version>-1

    After fixes for all serious and trivially fixable issues have been
    added and it has been verified that the package builds and `do-dch'
    has updated `debian/changelog' and the resulting commit should be
    tagged as `debian/<new-zsh-version>-1'.


** Generating packages

*** gitpkg

    `gitpkg' is a simple tool to help generating packages from debian
    packages which are maintained in the git version control system. It
    works quite well in this workflow. In fact, it works out of the box:

#+BEGIN_EXAMPLE
    % gitpkg debian/4.3.12-5 zsh-4.3.12
#+END_EXAMPLE

    The first parameter (debian/4.3.12-5) is the debian tag which
    points at the debian package version you want to build. The second
    parameter is the tag of the upstream version of the corresponding
    upstream release (zsh-4.3.12).

    Per default, `gitpkg' generates it's output in
    `../deb-packages'. This location is configurable.

    Below directories for individual packages are created and in those,
    data for individual package versions are created. For the above
    example, this would look like this:

#+BEGIN_EXAMPLE
    ../deb-packages/zsh/
    ../deb-packages/zsh/zsh-4.3.12/
    ../deb-packages/zsh/zsh_4.3.12.orig.tar.gz
    ../deb-packages/zsh/zsh_4.3.12-5.debian.tar.gz
    ../deb-packages/zsh/zsh_4.3.12-5.dsc
#+END_EXAMPLE

    You may now change to `../deb-packages/zsh/zsh-4.3.12/' and build
    binary package using `dpkg-buildpackage', `debuild' or the like.

    `gitpkg' is available as Debian package or from:

    <http://git.debian.org/?p=users/ron/gitpkg.git>

*** git-buildpackage

    Alternatively, `git-buildpackage' also provides ways of building
    packages from our packaging codebase. And since we are using the
    `git-dch' tool from this utility suite anyway, the tool should be
    available already.

    `git-buildpackage' allows building the package from within the
    package repository:

#+BEGIN_EXAMPLE
    % git-buildpackage --debian-branch=debian
#+END_EXAMPLE

    Make sure that the local repository is cleaned up after doing this
    before working on the package again, to avoid accidentially
    committing anything. See "Cleaning up the local repository" above
    for details.

    `git-buildpackage' is available as Debian package or from:

    <https://honk.sigxcpu.org/piki/projects/git-buildpackage/>


** Git repository setup

   Getting the basic pkg-zsh git repository is quite easy. If you want
   a read only clone, use this:

#+BEGIN_EXAMPLE
   % git clone git://git.debian.org/collab-maint/zsh.git pkg-zsh
#+END_EXAMPLE

   If you are reading this, though, you probably want write access. To
   get a thusly cloned repository, first get an alioth login and
   upload an ssh-public key. As soon as the key made it to all
   involved machines, use this:

#+BEGIN_EXAMPLE
   % git clone ssh://<user>@git.debian.org/git/collab-maint/zsh.git pkg-zsh
#+END_EXAMPLE

   Where `<user>' is your alioth login. (Note, that this may be
   something with a `-guest' suffix, in case you're not a debian
   developer.)

*** Branches

    Like described earlier, pkg-zsh development involves two branches;
    `debian' and `upstream'. The former is checked out by default for
    freshly cloned repositories. To get a local version of the
    `upstream' branch, use:

#+BEGIN_EXAMPLE
    % git checkout -b upstream origin/upstream
#+END_EXAMPLE

    This is useful to update the remote upstream branch with ongoing
    development from the zsh project.

*** Remotes

    There is one remote repository with direct interest for pkg-zsh,
    and that is the zsh project's git repository. Currently, this is
    only a mirror of the project's cvs repository. But it is updated
    every ten minutes by one of zsh's developers. (Also note, that
    there has been a brief discussion about whether git may become the
    official VCS for git after a bigger future release.)

    In order to have zsh's ongoing development available from within
    your pkg-zsh repository, do this:

#+BEGIN_EXAMPLE
    % git remote add zsh.git git://zsh.git.sf.net/gitroot/zsh/zsh -t master
    % git fetch zsh.git
#+END_EXAMPLE

*** Merging and pushing upstream changes

    To get updates back into origin/upstream, do this:

#+BEGIN_EXAMPLE
    ## Get the latest updates.
    % git fetch zsh.git
    ## Switch to the local `upstream' branch for integration.
    % git checkout upstream
    ## Merge upstream's changes (*).
    % git merge zsh.git/master
    ## Push the code into pkg-zsh's central repository.
    % git push origin
    ## Make sure the central repository also has all tags.
    % git push --tags origin

    (*) This step should *always* result in a fast-forward merge. If
        it does not, something went terribly wrong. Investigate and
        fix the situation *before* pushing to origin.
#+END_EXAMPLE