How to unpack packed-refs?

Question

I cloned a project from github with git clone --mirror. That left me with a repository with a packed-refs file, a .pack and an .idx file. For developement purposes I want to look at the loose objects, so I unpacked the objects with git unpack-objects < <pack file> which worked fine (I unpacked the pack file into a new repo if you're wondering). Only thing is that refs/heads/ is still empty, all the refs are still only in packed-refs but I need them in refs/heads/. I wasn't able to find a command that would extract or unpack those references and I can somehow not believe that I would have to do this by hand (or via pipes).

So actually I have two questions:

1. Is there an easy way to "restore" refs from packed-refs?
2. If not, why isn't there? If there's a command for unpacking objects, what is the reasoning behind not providing the same for refs (don't forget that there's even a command git pack-refs...)

Thanks for any tips and ideas.

Answer 1

The short answer is "no" - there is no "easy way" to unpack the refs the way you're asking.

The slightly longer answer is, each ref is just a 41-byte text file (40 byte SHA1 in hex + newline) in a specific path, so the "hard" version just requires something like this in your ~/.gitconfig:

[alias]
unpack-refs = "!bash -c 'IFS=$''\\n''; for f in $(git show-ref --heads); do /bin/echo ''Writing  '' $(echo $f | cut -c42-); echo $(echo $f | cut -c1-40) > \"${GIT_DIR:-.git}/$(echo $f | cut -c42-)\"; done'"

Took a little trickiness to figure out how to get it to work properly, but there you go! Now you have 'git unpack-refs' and it does what you expect, and as a bonus it even works with $GIT_DIR if that's set (otherwise it assumes you're in the root of the git tree). If you haven't read up on git aliases, https://git.wiki.kernel.org/index.php/Aliases is a great reference and even includes an example 'git alias' extension you can use to extend your own aliases.

Answer 2

The reason the packed refs exist is to speed up access in a repo with zillions of refs - it's easier to look at a single file with many lines than to hit the file system once for every single ref. Anything in git which needs to know about refs goes through code which can read both the refs directory and the packed refs file. Unpacking it would defeat its purpose. If you want to access refs, use the plumbing commands (e.g. show-ref, for-each-ref, update-ref...). I can't really think of any kind of access which would be faster and easier with the directory structure than with the plumbing commands (especially with for-each-ref available).

And yes, packed objects are (like packed refs) created for improved performance, but there's a huge difference. A packed refs file is just a bunch of independent lines. You can, essentially for free, add to or remove from it. There's no need to unpack it in order to modify it. Packed objects, on the other hand, are delta-compressed, so the objects inside depend on each other. They greatly reduce disk usage, and objects can be read from them at reasonable cost, but attempting to modify the set of objects in the pack is much more expensive than modifying loose objects, so it's only done periodically by git repack (called by git gc), though I don't believe git repack actually unpacks the objects - it just reads them from the packfile, packs them with the loose ones, and makes a new pack.

However, when a pack is transferred from a remote, it's unpacked on the local side. I see a call to an unpack method in the git receive-pack source, and the pack-objects manpage says:

The git unpack-objects command can read the packed archive and expand the objects contained in the pack into "one-file one-object" format; this is typically done by the smart-pull commands when a pack is created on-the-fly for efficient network transport by their peers.

Answer 3

Yet another answer to your question 1:

I assume you already used a loop like this to use git unpack-objects:

 mkdir CLONE
 mv .git/objects/pack/* CLONE/
 for pack in CLONE/*.pack; do
    git unpack-objects < $pack
 done
 rm -rf CLONE/

which unpacks all objects, but leaves the packed branch heads in the file .git/packed-refs

Thanks to Clee's answer, which I reused here, I found that the following commands will be needed to unpack the branch heads, and clean up:

(
   IFS=$'\n'; # set the input field separator to new line
   for f in $(git show-ref --heads); do
      ref_hash="$(echo $f | cut -c1-40)"
      ref_label="$(echo $f | cut -c42-)"
      echo " unpack: $ref_hash $ref_label"
      echo "$ref_hash" > ".git/$ref_label";
      sed -i "s~^${ref_hash} ${ref_label}\$~~" .git/packed-refs
      sed -i '/^$/d'                           .git/packed-refs
   done
   rm .git/info/refs
   rm .git/objects/info/packs
)

Note the difference to Clee's answer:

A) the packed refs are removed from the .git/packed-refs file, and

B) the files .git/info/refs and .git/objects/info/packs are deleted.

I admit that it might not be a good idea to delete files just like that in the .git folder, however this is what I needed to do in order to do a clean unpack.

Question 2 is still not answered though.

Answer 4

There is a workaround that worked for me and might work for some of you:

Basically, Delete all the tags locally (removes it from packed-refs) and then fetch it again.

git tag | xargs git tag -d && git fetch --tags