3

I have a self-hosted gitlab instance and suddenly one of my projects is sending emails telling me that the repo check is failing.

I run gitlab-15.0.0-ee via docker. I checked the log file repocheck.log and this is the content:

E, [2022-06-07T19:08:10.599782 #435] ERROR -- : xxx/xxx: Could not fsck repository: dangling blob ed44e089d0eaf14fc152871bdca12aa6de01a5f1
dangling blob 4317819507d8e4e3ddcce6f3c01b8890e56cc157
dangling commit 2b1f190885dae6bbef0986cd50bc61c790331312
dangling blob 0f7c7218e71e1bd8b964184e149c214734d41944
dangling blob 6f16132326f43f1fd3a9a9cd717830643a162c8a
dangling commit 3b87a380a8095fe83fed257fd89e178323b89bea
dangling commit 14ca539d07483fc309f94cea685fc893a3efaf51
dangling commit 3cf64b574d2ecbc4ffe06559f9495db22bb5f2c9
dangling blob 535b94cb0a9048bbe45eb35ba53a02fe795be823
dangling commit 6778eef6143982b0d96f2934e515b36254e9a31f
dangling commit 0294e6bca9f2a5cacd7bbea61c62e5b5d92eb98f
dangling blob 4144f7437938ea8b0fe5ce6c8a498995b4f96f7c
dangling blob b64cd7abaa200ac4fea060b0dfb8542e260cb301
dangling blob 90545fea55b5e5c1f6245281ebac02b1dd8bfd04
error: Could not read 0957f8065b8fecfa80005eafa673a2a8b67ddbed
failed to parse commit 0957f8065b8fecfa80005eafa673a2a8b67ddbed from object database for commit-graph
error: Could not read 0a70090f6519febb2edbd9c416b5dff46d92d1a1
failed to parse commit 0a70090f6519febb2edbd9c416b5dff46d92d1a1 from object database for commit-graph
error: Could not read 126be254518dfd5b29a553fcfe7843787cbeed08
....

and the file has around 200 more commits that are not present in the commit-graph.

On my local repository most of those commits are still present and they are indeed not reachable, so a git gc on the server removed those commits that are still present on my local repository. I don't seem to be missing merge requests so it doesn't make sense to me why gitlab thinks that this is a problem.

Sadly the official documentation is very vague about what do do other than tell to to locate the repocheck.log file and or to click on Clear all repository checks if the error persists. That doesn't help at all, because even after clicking on Clear all repository checks, I'm still getting the same error. So what is gitlab actually testing and why is it failing?

Improve this question
2
  • Probably fixes itself through gitlab's automatic housekeeping. docs.gitlab.com/ee/administration/housekeeping.html <your_instance>/admin/application_settings/repository > Repository Maintenance.
    – immeëmosol
    Commented Jun 13, 2022 at 8:24
  • 1
    It didn't. Yesterday I got the same error again :(
    – Pablo
    Commented Jun 13, 2022 at 23:36

2 Answers 2

Reset to default
2

  1. Find all repositories that failed on link “See the affected projects in the GitLab admin panel�� that you’ve received from GitLab http://yourgitlab_link/admin/projects?last_repository_check_failed=1

  2. Check /var/log/gitlab/gitlab-rails/repocheck.log file that is generated when “Trigger repository check” is triggered in project admin area. If file is empty then trigger repository check manually again. This step is not so important even though the file is empty or not, because you will get similar output with command fsck described little lower.

  3. Open specific project Menu->Admin->Projects->your project and search for “Gitaly relative path” @hashed/48/b3/48b361d46638bfa4eee090c158a750a69c7beec3a62e703e2801125551b1b157.git

  4. Go to docker console (connect to container) or the console of the Gitlab Omnibus Server and run fsck command on that specific repository command. Be sure to run the command as the correct user git at omnibus installations, otherwise files are created with the owner root, to which gitlab cannot write to:

    /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/48/b3/48b361d46638bfa4eee090c158a750a69c7beec3a62e703e2801125551b1b157.git fsck and check if there are any errors like

    error: Could not read 098b53ffbe581e25b… failed to parse commit 098b53ffbe581e25b… from object database for commit-graph

    now run garbage collector gc with

    /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/48/b3/48b361d46638bfa4eee090c158a750a69c7beec3a62e703e2801125551b1b157.git gc

    and check again repository with fsck

    /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/48/b3/48b361d46638bfa4eee090c158a750a69c7beec3a62e703e2801125551b1b157.git fsck and errors and fails should be cleared

  5. run again “Trigger repository check” in check if repository check has passed successfully. If it was successful then this repository shouldn’t be in repository list that have failed to pass the repository check (See the affected projects in the GitLab admin panel) or http://yourgitlab_link/admin/projects?last_repository_check_failed=1

Source: https://forum.gitlab.com/t/gitlab-projects-failed-their-last-repository-check/19147/3

Improve this answer
1
  • Thanks, I'll check that later
    – Pablo
    Commented Oct 24, 2022 at 8:51
0

See GitLab's documentation:
Administer GitLab > ... > Repository Checks > Run a check using the command line

(select the version of the documentation that best matches your installation)

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .