git, nagios and hooks, corrupted git repo

Question

Background

We're using nagios to monitor our infrastructure. We don't have the nagios configs under version control at the moment, and there are two of us that manage nagios configuration. As such, I'm working to get our nagios config into a central git repo, using some hooks to do syntax checking and then if the configs look good, make them "active". I'm using this guy's post as a starting point.

The general workflow I'm trying to implement is:

1. Edit local git repo of nagios config. Add edited files, commit locally.
2. git push origin master to the remote repo.
3. Push is intercepted by the pre-receive hook, which takes the files, moves them to a temporary directory on the server, and runs them through the nagios syntax checker.
4. If the syntax checker passes, accept the push, then use the post-commit hook to git pull the new code into the live nagios configuration directory and then restart nagios.
5. If the syntax checker fails, reject the push, showing the nagios syntax error to the user.

I'm running into an odd behavior, though, when I reject a git push due to syntax errors in the nagios config. What I expect to happen is that if I reject the hook, the attempted push should leave the repository just how it was, untouched. That doesn't appear to be the case, though. Below are the details of what I'm seeing:

Problem

I edit the nagios config locally, intentionally including a syntax error, add, then commit locally:

host:nagios erik$ vi nagios.cfg
host:nagios erik$ git add nagios.cfg
host:nagios erik$ git commit -m "syntax error"
[master da71aed] syntax error
 1 files changed, 1 insertions(+), 0 deletions(-)

Now I push those changes to the master repo. This will be rejected due to the syntax error:

host:nagios erik$ git push origin master
Counting objects: 5, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 12.74 KiB, done.
Total 3 (delta 1), reused 2 (delta 1)
remote: Previous HEAD position was 3ddc880... removed syntax error
remote: HEAD is now at da71aed... syntax error
remote: Nagios Config Check Exit Status: 254
remote: Your configs did not parse correctly, there was an error. Output follows.
remote:
remote: Nagios Core 3.2.3
remote: Copyright (c) 2009-2010 Nagios Core Development Team and Community Contributors
remote: Copyright (c) 1999-2009 Ethan Galstad
remote: Last Modified: 10-03-2010
remote: License: GPL
remote:
remote: Website: http://www.nagios.org
remote: Reading configuration data...
remote: Error in configuration file '/tmp/nagiosworkdir/nagios.cfg' - Line 23 (NULL value)
remote:    Error processing main config file!
remote:
remote:
remote:
remote: ***> One or more problems was encountered while processing the config files...
remote:
remote:      Check your configuration file(s) to ensure that they contain valid
remote:      directives and data defintions.  If you are upgrading from a previous
remote:      version of Nagios, you should be aware that some variables/definitions
remote:      may have been removed or modified in this version.  Make sure to read
remote:      the HTML documentation regarding the config files, as well as the
remote:      'Whats New' section to find out what has changed.
remote:
To [email protected]:nagios
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to '[email protected]:nagios'

This shouldn't have touched the remote repo, but it did. If I change to another local temp directory and try to clone the repo, I get:

host:temp erik$ git clone [email protected]:nagios
Cloning into nagios...
remote: Counting objects: 30, done.
remote: Compressing objects: 100% (29/29), done.
remote: Total 30 (delta 12), reused 0 (delta 0)
Receiving objects: 100% (30/30), 29.81 KiB, done.
Resolving deltas: 100% (12/12), done.
error: Trying to write ref HEAD with nonexistant object da71aedfde2e0469288acd9e45bb8b57a6e5a7b3
fatal: Cannot update the ref 'HEAD'.

Now I go back to the original work directory, fix the syntax error, add, commit, and push:

host:nagios erik$ vi nagios.cfg
host:nagios erik$ git add nagios.cfg
host:nagios erik$ git commit -m "removing syntax error, push should succeed this time"
[master f147ded] removing syntax error, push should succeed this time
 1 files changed, 0 insertions(+), 2 deletions(-)
host:nagios erik$ git push origin master
Counting objects: 6, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 487 bytes, done.
Total 4 (delta 2), reused 0 (delta 0)
remote: Previous HEAD position was 4c80d45... syntax error
remote: HEAD is now at f147ded... removing syntax error, push should succeed this time
remote: Nagios Config Check Exit Status: 0
remote: Your configs look good and parsed correctly.
To [email protected]:nagios
   3ddc880..f147ded  master -> master

At this point, the repository is fine, and I'm able to change to a temporary directory and clone the repo again:

host:temp erik$ git clone [email protected]:nagios
Cloning into nagios...
remote: Counting objects: 34, done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 34 (delta 14), reused 0 (delta 0)
Receiving objects: 100% (34/34), 30.22 KiB, done.
Resolving deltas: 100% (14/14), done.

Here is the pre-receive hook I'm using.

I'm using git v1.7.5.4 on the client, and v1.7.2.3 on the server.

So, to the question: why is the repository being left in an inconsistent state when I reject the push? Is something awry with my git hook or perhaps my understanding of git is lacking?

Answer 1

You're doing:

export GIT_WORK_TREE=/tmp/nagiosworkdir
/usr/bin/git checkout -f $NEW_SHA1

in your hook. Although it's not touching your usual working-copy it is updating references in the git-dir (specifically the HEAD reference), as shown in your error:

...
remote: HEAD is now at da71aed... syntax error
...

Your hook is doing exit 1 to reject the update, but it's not (re)resetting the HEAD reference after failure.

I think you need to update the failure branch in your hook like so:

...
if [ "$NAGIOS_CHECK_STATUS" -ne 0 ]
   then
   echo "Your configs did not parse correctly, there was an error. Output follows."
   cat $GIT_WORK_TREE/check.out
   /usr/bin/git reset --hard $OLD_SHA1    # <-- Add This
   exit 1
else
   ...

Answer 2

The git checkout command in your hook is creating/updating the HEAD ref in your repository.

If your repository is a bare repository, it can live without a HEAD ref (new clones will default to checking out its master branch, if it has one); just delete the HEAD ref before exiting (maybe in a trap so that you do not have to arrange to do it before each exit individually). Anywhere “early” in your script:

trap 'git update-ref -m "removing HEAD after temporary checkout to alternate workdir" -d HEAD "$NEW_SHA1"' 0

If your repository is not bare, or you want to maintain a HEAD ref (so that clones will, by default, check out some other branch), then you will have to save the HEAD ref and restore it before exiting.

First, in the server’s repository, reset the HEAD ref to point to the branch that you want to be checked out by default in new clones:

git symbolic-ref -m 'setting default branch for new clones' HEAD refs/heads/master

Then, in your hook script (anywhere before your checkout):

# Restore HEAD symref when exiting
saved_HEAD=$(git symbolic-ref HEAD)
trap 'git symbolic-ref -m "restoring HEAD after temporary checkout to alternate workdir" HEAD "$saved_HEAD"' 0

---

By the way, pre-receive hooks should make sure that they fully read stdin and process all the lines they are fed. Exiting before consuming all the input can sometimes trigger a SIGPIPE in the git-receive-pack process; this probably does not come up in your case if you only push one ref at a time (since you read at least one line), but it is something to keep in mind. Probably it is easier to do this hook as an update hook where you only need to be concerned with one ref at a time and can reject each ref’s push individually (maybe you only care about keeping the tip of master “clean”; while you check and report on the tips of other branches, but never reject them so that they can be used for collaboration on incomplete work).