Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trim down metrics cardinality #3003

Merged
merged 1 commit into from
Oct 20, 2023

Conversation

nikola-jokic
Copy link
Contributor

Runner ID increases metrics cardinality, since all runners are ephemeral.

To address this problem:

  • Exclude runner_id and runner_name from all metrics labels
  • Exclude job_workflow_ref and the job_name from the histogram

In the future releases, it should be considered adding configurable labels. However, removing the labels mentioned above makes sense since they increase cardinality substantially. If there is a need to return them back, we can do it in the same time when configurable metrics are introduced.

Fixes #2873

@nikola-jokic nikola-jokic added the gha-runner-scale-set Related to the gha-runner-scale-set mode label Oct 19, 2023
@nikola-jokic nikola-jokic merged commit b511953 into master Oct 20, 2023
15 checks passed
@nikola-jokic nikola-jokic deleted the nikola-jokic/trim-metrics-cardinality branch October 20, 2023 10:20
@MPV
Copy link

MPV commented Dec 7, 2023

Did this PR remove these labels from both:

  • gha_job_execution_duration_seconds
  • gha_job_startup_duration_seconds

We are seeing lowered cardinality on: 🙏🏻

  • gha_job_startup_duration_seconds

But still high cardinality on: 😅

  • gha_job_execution_duration_seconds (especially the _bucket)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
gha-runner-scale-set Related to the gha-runner-scale-set mode
3 participants