31

I tried to open my repository today and it came up with no commit history. Everything I tried (git status, git log, git checkout...) threw an error about a corrupt object.

I researched this problem online and found the article by Linus Torvalds, but got lost at the point where he found the broken link ID: none of my file IDs, tree or blob, match the culprit ID thrown by the error message.

I then returned to the article on recovering "git objects damaged by hard disk failure" and (after moving the culprit object out of the way) worked my way through until 

$ cat packed-refs

at which point my computer said: cat: packed-refs: No such file or directory I skipped that step and did the

$ git fsck --full

and got the appropriate output, but then I was supposed to copy the culprit (or what I was referring to as the culprit, the sha1 ID thrown by the error) from a backup repository back into the main repository, then copy the missing objects from the backup repository into the main repository, as far as I can tell; and I don't want to do anything too drastic or I might force something I can't unforce later.

So my question(s) is (are), was I supposed to have made a backup (ooh, newbie alert), or was that what happened when I unpacked the .pack file? And is the "culprit" I'm copying back actually a clean file, i.e. not corrupted?

(I think it only fair to tell you that I was initially confused by a simple dash in Torvalds' file between the "git" and "fsck." So I'm REALLY new at this.)

BUG-LIST

Original bug:

$ git status
fatal: object 016660b7605cfc2da85f631bbe809f7cb7962608 is corrupted

Bug after moving corrupt object:

$ git status
fatal: bad object HEAD
$ git fsck --full
error: HEAD: invalid sha1 pointer 016660b7605cfc2da85f631bbe809f7cb7962608
error: refs/heads/RPG does not point to a valid object!
dangling tree 2c1033501b82e301d47dbf53ba0a199003af25a8
dangling blob 531aca5783033131441ac7e132789cfcad82d06d
dangling blob 74a47ff40a8c5149a8701c2f4b29bba408fa36f5
dangling blob b8df4d9751c0518c3560e650b21a182ea6d7bd5e
dangling blob fc2d15aead4bd0c197604a9f9822d265bb986d8b

$ git ls-tree 2c1033501b82e301d47dbf53ba0a199003af25a8
040000 tree 4a8b0b3747450085b1cd920c22ec82c18d9311bd    folder1
040000 tree 33298295f646e8b378299191ce20b4594f5eb625    folder2
040000 tree dec82bad6283fc7fcc869c20fdea9f8588a2f1b2    folder3
040000 tree 4544967c6b04190f4c95b516ba8a86cab266a872    folder4

$ git ls-tree dec82bad6283fc7fcc869c20fdea9f8588a2f1b2
100644 blob 67bda6df733f6cd76fc0fc4c8a6132d8015591d8    fileA
100644 blob 4cb7272c9e268bfbd83a04e568d7edd87f78589c    fileB
100644 blob ce9e0f2cc4d3b656fa30340afbdfed47fe35f3ef    fileC

$ git ls-tree 4544967c6b04190f4c95b516ba8a86cab266a872
100644 blob d64fe3add8328d81b1f31c9dbd528956ab391fb6    fileD
100644 blob d1ebd7df7082abc5190d87caa821bf3edb7b68e8    fileE
100644 blob bb6cd264e47a3e5bc7beadf35ea13bac86024b02    ...
100644 blob 995d622b9012f4ef69921091d1e1a73f32aa94e6
100644 blob 9141dbd2b1c7931a6461195934b6599f5dfb485a 
100644 blob ab128da1d82907cd0568448dc089a7996d5f79d3
100644 blob 57b11a7eb408a79739d2bb60a0dc35c591340d18
100644 blob 118105291c1c6ca4a01744889ffafbb018bc7ed3
100644 blob 86b1dfda56d0603f16910228327751f869d16bdc
100644 blob 077fe0cddde0d0be9d0974f928f66815caca7b76
100644 blob c0b32fd0450f21994bdc53ea83d3cf0bccd74004
100644 blob 37b87a4d11453468c4ae04572db5d322cd2d1d80
100644 blob 79d39f8d4e57fa3a71664598a63b6dfd88149638
100644 blob ee07bbe3e8cb5d6bb79fb0cd52cfbc9bd830498d    files

$ git ls-tree 33298295f646e8b378299191ce20b4594f5eb625
100644 blob f9d6f45cd028aec97f761f00c5f4f2f6b50fb925    MoreFiles
100644 blob 0cb9eed1d0dd9214d54a03af1bda21f37b8c0d02
100644 blob 198e4f97ece735cce47b7e99b54f1b5fa99fabf5
100644 blob fc004212fa8e483e5a8ab35b508027c7a9a1cbfa
100644 blob 0c7d74c7a9a8337b4a9f20802b63d71d42287f89

$ git ls-tree 4a8b0b3747450085b1cd920c22ec82c18d9311bd
100644 blob 0320f5b23dd7cce677fac60b9ad03f418cff5c88    oneLASTfile

After moving the corrupted object back:

$ git log --raw --all
fatal: object 016660b7605cfc2da85f631bbe809f7cb7962608 is corrupted

$ cat packed-refs
cat: packed-refs: No such file or directory

$ git fsck --full
fatal: object 016660b7605cfc2da85f631bbe809f7cb7962608 is corrupted

After moving the file back out:

$ git fsck --full`  
error: HEAD: invalid sha1 pointer 016660b7605cfc2da85f631bbe809f7cb7962608
error: refs/heads/RPG does not point to a valid object!
dangling tree 2c1033501b82e301d47dbf53ba0a199003af25a8
dangling blob 531aca5783033131441ac7e132789cfcad82d06d
dangling blob 74a47ff40a8c5149a8701c2f4b29bba408fa36f5
dangling blob b8df4d9751c0518c3560e650b21a182ea6d7bd5e
dangling blob fc2d15aead4bd0c197604a9f9822d265bb986d8b

After unpacking the .pack file:

$ git log
fatal: bad object HEAD

$ cat packed-refs
cat: packed-refs: No such file or directory

$ git fsck --full
error: HEAD: invalid sha1 pointer 016660b7605cfc2da85f631bbe809f7cb7962608
error: refs/heads/RPG does not point to a valid object!
dangling tree 2c1033501b82e301d47dbf53ba0a199003af25a8
dangling blob 531aca5783033131441ac7e132789cfcad82d06d
dangling blob 74a47ff40a8c5149a8701c2f4b29bba408fa36f5
dangling blob b8df4d9751c0518c3560e650b21a182ea6d7bd5e
dangling blob fc2d15aead4bd0c197604a9f9822d265bb986d8b
Improve this question
5
  • Could you edit your question and add links to the articles you mention, so we can see what you've done? I'm not sure what you mean by "unpacking the .pack file" otherwise, though that's certainly not a backup - the pack file is just a bunch of objects delta-compressed into one file. A good way to back up a git repository is to clone it!
    – Cascabel
    Commented Aug 31, 2010 at 23:15
  • If this SO question stackoverflow.com/questions/801577/… is the second article you mention, it doesn't sound like it's the same error you mentioned - unless you really did get all those errors. Could you post the exact error(s) you're getting too?
    – Cascabel
    Commented Aug 31, 2010 at 23:20
  • Also, what was the last thing you did in your repo before this? Just on the off chance this isn't a hard drive failure...
    – Cascabel
    Commented Aug 31, 2010 at 23:21
  • I did cut some steps, mainly due to identical errors or previous functions. I also ommitted some steps which had output similar to the articles mentioned, and others which were simply me wrestling with the computer (ugly). The last thing I did was a commit to the branch I was on (not master) for some files in folder3. Immediately afterwards, I turned off my computer. If it helps, my computer sometimes doesn't completely turn off sometimes, and this incomplete shutdown occurred at that time also. If necessary I can post the entire session.
    – Humble Penguin
    Commented Sep 1, 2010 at 3:06
  • By the way, if you put @Jefromi in your comment, SO will let me know about it so I can get back to you sooner. (It does this automatically if you comment on my answer; no need for it there.)
    – Cascabel
    Commented Sep 1, 2010 at 14:38

3 Answers 3

Reset to default
19

Okay, so. We can see from the second error message that the corrupt object which you moved was a commit. (HEAD was pointing to it!) Unfortunately, this means that it's hard to manually repair it. (By "hard" I mean likely impossible unless you can remember exactly what the commit message was and what time you made the commit.) Fortunately, this does mean that it's easy to resurrect a new commit with the same file contents - you'll just have to write a new message for it.

Before you start, have a look at the contents of .git/HEAD - if it's a branch name, remember that for later.

First, we need to figure out what the parent of this commit should've been. You can use git reflog to look at the reflog of HEAD, and find the SHA1 of where HEAD was just before you made commit 016660b. It should look something like this:

016660b HEAD@{n}: commit: <subject of commit>
1234abc HEAD@{n-1}: ...

You can copy the SHA1 of the previous position of HEAD, and check out that commit:

git checkout 1234abc

Then you can read in the tree that your corrupted commit had:

git read-tree 2c1033501b82e301d47dbf53ba0a199003af25a8

And then commit!

git commit

Now, there's some question here about what should've happened to your branches. If HEAD was pointing to a branch (say master) which in turn pointed to the corrupted commit, we definitely want to fix that up:

git branch -d master       # remove the original master branch
git checkout -b master     # recreate it here

If there are other branches which contained the corrupted commit, you'll have to do some restoration on them too - let me know if you need help with that.

Improve this answer
13
  • With corrupt file displaced: $ git reflog fatal: bad object HEAD With corrupt file in place: $ git reflog fatal: object 016660b7605cfc2da85f631bbe809f7cb7962608 is corrupted I read the manual and tried: $ git reflog show HEAD@{1} Which gave me activity from 1 to 12, and then restarted at 1 and continued until the initial pull I had done on my repositories. Is this restart typical of the command?
    – Humble Penguin
    Commented Sep 2, 2010 at 3:36
  • Continuing on... $ git checkout bcbd203 #which was HEAD@{1} fatal: object 016660b7605cfc2da85f631bbe809f7cb7962608 is corrupted Should I simply move the file out and attempt the checkout again? Sorry about the absence of user-friendly formatting in these comments. I don't currently know a better way to do this.
    – Humble Penguin
    Commented Sep 2, 2010 at 3:40
  • 1
    @Humble: Oops, sorry about that. I haven't actually had this happen to me, so I was writing untested things. I didn't know reflog actually tried to read the objects! Anyway, as long as you've figured out what the previous commit was, that part's all good. (You could also have looked directly at .git/logs/HEAD)
    – Cascabel
    Commented Sep 2, 2010 at 3:52
  • @Humble: For the checkout... yeah, pretty much whatever you have to do to get it to succeed. Move the corrupt object out, and if that doesn't work... try checkout -f, then try reset --hard <SHA1>... once it works you should be able to continue on.
    – Cascabel
    Commented Sep 2, 2010 at 3:53
  • 4
    Aha! Git is just doing its best not to let you throw away any information - unfortunately we know we have to. You're going to have to get rid of the old branch one way or another. If git branch -f master; git checkout master doesn't work, just go in the .git directory and yank it out yourself - remove refs/heads/master and logs/refs/heads/master, then create it normally.
    – Cascabel
    Commented Sep 2, 2010 at 21:41
16

FWIW, here's an answer that is more practical than a lot of the hopelessly optimistic ones on other questions about corrupted git repos - most of which make the baseless assumption that the poor OP can 'just re-clone from the remote origin!' Hmm. But. Wait a second. What if I am the origin?

The horror

The story begins when an attempt to run a simple git gc --aggressive revealed that, unbeknownst to me, at some point my - completely local - git repo had somehow gotten completely hosed: It was unable to log anything beyond a few months ago, thereby losing the majority of its history, and screamed gutturally whenever asked to git fsck --full | grep -v dangling. Several objects were identified as lost by git fsck.

git-repair: minimising terrifying manual labour since 2014

After panicking and finding lots of questions with over-voted poor answers just saying 'just re-clone from the remote origin!' - which, let me belabour the point, I don't have because I am the origin - I found git-repair, did a simple sudo aptitude install git-repair, and let it do all the tedious automatable fixes that would've probably taken me hours (please: run on a copy of your corrupted repo [duh], without --force!)

That helped reduce the amount of horror reported by git fsck --full | grep -v dangling. But things beyond the middle of August were still unreachable.

In particular, this all seemed to centre around one commit that couldn't be retrieved. How could I recover it? Searching Stack Overflow wasn't much help!

You have a backup, right?

Here's where I was lucky enough to have a backup from the end of November. You take backups, right? In my case, it was a manual zip of the repo (my daily backup routine is some horrible incremental tar thing that I've never actually tested... a-cough)... but phew, it was good enough. It hadn't suffered whatever indignity had afflicted my live repo.

But the missing object didn't seem to be simply stored in this backup's .git/objects/XY/RESTOFHASHBLAHBLAHBLAH. That's probably because it was a commit, not a file. I don't know! git is wizardry to me, forever beyond my capacity to comprehend. I just needed a fix, fast. Isn't that why we're all here?

Recover objects from a backup (which you have, right?)

With backup now in hand, I had a hilariously dumb idea, said 'there's no way this can work!', and immediately found that simply naively running cp -fr /path/to/repo_backup/.git/objects/* /path/to/repo_git-repaired/.git/objects to file-merge the.git/objects directory of the backup into its counterpart in my corrupted,-somewhat-repaired repo... worked to recover all history - right back to good ol' initial commit lol. Proving the fix: git fsck --full was now happy too (notwithstanding all the dangling bits).

I then took backups of the live/corrupted, partly repaired, and seemingly recovered repos onto a separate drive, in case I need any of them again.

You should double-check that methodology and command before you run it, or perhaps find a much better way that I'm too lazy to think of. I don't know. But for me, it saved my repo. And on the subject of second-guessing what I say...

Infinite disclaimers in infinite combinations

Now, obviously all of this comes with caveats: you must try everything on a copy of your corrupted repo, read all the documentation, consider being a bit more careful than I was (that forced cp command), and not hold me responsible or liable for absolutely anything that goes wrong*... but it gives you something better to try than 'just re-clone from the remote origin!', right?

* If, however, everything goes right, a large donation may be in order. ;)

If anyone needs me, I'll be glaring suspiciously in the general direction of my disk drive, hopefully coming up with a backup routine that won't take double-digit hours to restore from (if at all), and maybe even getting some sleep.

Improve this answer
4
  • hi cant use git-repair command, i tried 'git git-repair' 'git-repair' on mac, nothing working, please help
    – Halid
    Commented Jul 22, 2017 at 11:21
  • 1
    As mentioned in my post, of course git-repair won't work if you don't have it installed. I don't know about macOS but presume you can get it through some package manager like Homebrew. Or if not, it's quite likely that someone somewhere has compiled it and offered it as a download on their own site. You'll just need to look for it.
    – underscore_d
    Commented Jul 23, 2017 at 9:14
  • i have cloned, build an installed it on my mac, but dont know the command to use it
    – Halid
    Commented Jul 23, 2017 at 11:07
  • 2
    So read the documentation? I'm not sure what you want me to do about that.
    – underscore_d
    Commented Jul 23, 2017 at 14:50
6

I had the same problem. However, my problem was resolved with changing permissions of the .git/objects folders and subfolders (recursively) on server. Something like:

chmod -R 770 .git/objects

I think that isn't your problem, but in my case it was resolved.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.