Cloud Monitoring provides visibility into the performance, uptime, and overall health of cloud-powered applications. Google Cloud Observability collects and ingests metrics, events, and metadata from Dataproc Metastore services to generate insights using dashboards and charts.
See Monitoring Pricing to understand your costs.
See Monitoring Quotas and limits for information on metric data retention.
Access service metrics in Monitoring
Dataproc Metastore service resource metrics are automatically enabled on Dataproc Metastore services. Use Monitoring to view these metrics.
You can access Monitoring from the Google Cloud console or by using the Monitoring API.
Console
In the Google Cloud console, go to the Cloud Monitoring > Metrics explorer page:
From the "Find resource type and metric" drop-down list, select the resource
Dataproc Metastore Service
.Click again in the input box, and then select a metric from the drop-down list. Hovering over the metric name displays information about the metric.
You can select filters, group by metric labels, perform aggregations, and select chart viewing options.
REST
You can use the Monitoring timeSeries.list
API to capture and list metrics defined by a filter
expression.
Use the Try this API template on the API page to send
an API request and display the response.
Dataproc Metastore service metrics in Monitoring
Metrics from Dataproc Metastore. Launch stages of these metrics: BETA GA
The "metric type" strings in this table must be prefixed
with metastore.googleapis.com/
. That prefix has been
omitted from the entries in the table.
When querying a label, use the metric.labels.
prefix; for
example, metric.labels.LABEL="VALUE"
.
Metric type Launch stage Display name |
|
---|---|
Kind, Type, Unit Monitored resources |
Description Labels |
federation/server/request_count
BETA
Federation service request count |
|
DELTA , INT64 , {request}
metastore.googleapis.com/Federation |
The number of requests sent to the Federated Dataproc Metastore Service.
method_name :
The request method name.
status :
End status of the request, one of [success, client_error, server_error].
version :
The Hive Metastore version of the service.
|
service/grpc/finished_requests_count
BETA
gRPC Requests Finished Count |
|
DELTA , INT64 , 1
metastore.googleapis.com/Service |
The number of requests finished by the GRPC Dataproc Metastore Service.
tier :
Service tier or instance size of the service, such as ENTERPRISE.
version :
The Hive Metastore version of the service.
status :
The status of the request, such as `BAD_REQUEST`.
method_name :
The request method name.
|
service/grpc/received_bytes_count
BETA
gRPC Bytes Received |
|
DELTA , INT64 , By
metastore.googleapis.com/Service |
Number of bytes received by the Dataproc Metastore Service.
tier :
Service tier or instance size of the service.
version :
The Hive Metastore version of the service.
|
service/grpc/request_latencies
BETA
gRPC Request Latencies |
|
DELTA , DISTRIBUTION , ms
metastore.googleapis.com/Service |
Distribution of gRPC request latencies for a Dataproc Metastore Service.
tier :
Service tier or instance size of the service.
version :
The Hive Metastore version of the service.
status :
The status of the request, such as `BAD_REQUEST`.
method_name :
The request method name.
|
service/grpc/sent_bytes_count
BETA
gRPC Bytes Sent |
|
DELTA , INT64 , By
metastore.googleapis.com/Service |
Number of bytes sent by the Dataproc Metastore Service.
tier :
Service tier or instance size of the service.
version :
The Hive Metastore version of the service.
|
service/grpc/started_requests_count
BETA
gRPC Requests Started Count |
|
DELTA , INT64 , 1
metastore.googleapis.com/Service |
The number of requests received and started by the GRPC Dataproc Metastore Service.
tier :
Service tier or instance size of the service.
version :
The Hive Metastore version of the service.
method_name :
The request method name.
|
service/health
GA
Service health |
|
GAUGE , INT64 , 1
metastore.googleapis.com/Service |
The health of the Dataproc Metastore Service.
tier :
Service tier or instance size of the service, such as ENTERPRISE.
version :
The Hive Metastore version of the service.
release_tag :
Dataproc Metastore release tag, populated by the service backend and may change over service lifetime.
|
service/multi_region/health
GA
Multiregional service health |
|
GAUGE , INT64 , 1
metastore.googleapis.com/Service |
The health of the multiregional Dataproc Metastore Service.
version :
The Hive Metastore version of the service.
release_tag :
Dataproc Metastore release tag, populated by the service backend and may change over service lifetime.
service_region :
The constituent region that Dataproc Metastore service is hosted in.
scaling_factor :
Scaling factor of the service, one of [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0].
|
service/multi_region/request_count
GA
Multiregional service request count |
|
DELTA , INT64 , {request}
metastore.googleapis.com/Service |
The number of requests sent to the multiregional Dataproc Metastore Service.
method_name :
The request method name.
version :
The Hive Metastore version of the service.
service_region :
The constituent region that Dataproc Metastore service is hosted in.
scaling_factor :
Scaling factor of the service, one of [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0].
|
service/multi_region/request_latencies
GA
Multiregional service request latencies |
|
DELTA , DISTRIBUTION , ms
metastore.googleapis.com/Service |
The latencies of the requests sent to the multiregional Dataproc Metastore Service.
method_name :
The request method name.
version :
The Hive Metastore version of the service.
service_region :
The constituent region that Dataproc Metastore service is hosted in.
scaling_factor :
Scaling factor of the service, one of [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0].
|
service/query_metadata/request_count
BETA
Admin Interface request count |
|
DELTA , INT64 , {request}
metastore.googleapis.com/Service |
The number of Admin Interface requests.
database_type :
The type of database, one of [SQL, Spanner].
code :
The gRPC status code.
|
service/query_metadata/request_latencies
BETA
Admin Interface request latency |
|
DELTA , DISTRIBUTION , ms
metastore.googleapis.com/Service |
The end to end request latency for Admin Interface requests.
database_type :
The type of database, one of [SQL, Spanner].
code :
The gRPC status code.
|
service/request_count
GA
Request count |
|
DELTA , INT64 , {request}
metastore.googleapis.com/Service |
The number of requests sent to the Dataproc Metastore Service.
method_name :
The request method name.
version :
The Hive Metastore version of the service.
|
service/scheduled_backups/backup_count
BETA
Scheduled Backups Count |
|
DELTA , INT64 , 1
metastore.googleapis.com/Service |
The number of scheduled backups currently stored for the service.
state :
State of the backup, one of [SUCCEEDED, FAILED, IN_PROGRESS].
database_type :
The type of database, one of [SQL, Spanner].
version :
The Hive Metastore version of the service.
|
service/scheduled_backups/last_backup_duration
BETA
Scheduled Backups - Last Backup Duration |
|
GAUGE , INT64 , s
metastore.googleapis.com/Service |
The duration of the last successful scheduled backup in seconds. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
state :
State of the backup, one of [SUCCEEDED, FAILED, IN_PROGRESS].
database_type :
The type of database, one of [SQL, Spanner].
version :
The Hive Metastore version of the service.
|
service/scheduled_backups/last_backup_size
BETA
Scheduled Backups Request - Last Backup Size |
|
GAUGE , INT64 , By
metastore.googleapis.com/Service |
The size of the last successful scheduled backup in bytes. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
state :
State of the backup, one of [SUCCEEDED, FAILED, IN_PROGRESS].
database_type :
The type of database, one of [SQL, Spanner].
version :
The Hive Metastore version of the service.
|
Table generated at 2024-06-27 18:22:22 UTC.
Build a custom Monitoring dashboard
You can build a custom Monitoring dashboard that displays charts of selected Dataproc Metastore service metrics.
Select + CREATE DASHBOARD from the Monitoring Dashboards Overview page.
Provide a name for the dashboard.
In the upper-right menu, click Add Chart to open the Add Chart window.
For the resource type, select
Dataproc Metastore Service
.Select one or more metrics and metric and chart properties, and then click Save to save the chart.
Add additional charts to your dashboard if you'd like.
To save the dashboard, click Save . Its title appears in the Monitoring Dashboards Overview page.
You can view, update, and delete dashboard charts from the dashboard display page.
Use Monitoring alerts
You can create a Monitoring alert that notifies you when a Dataproc Metastore service or job metric crosses a specified threshold.
Create an alert
In the Google Cloud console, open Monitoring Alerting.
To open the Create new alerting policy form, click + CREATE POLICY
Define an alert by adding alert conditions, policy triggers, notification channels, and documentation.
Select ADD CONDITION to open the alert condition form with the Metric tab selected.
Fill in the fields to define an alert condition, and then click ADD.
After adding the alert condition, complete the alert policy by setting notification channels, policy triggers, documentation, and the alert policy name.
View alerts
When an alert is triggered by a metric threshold condition, Monitoring creates an incident and a corresponding event. You can view incidents from the Monitoring Alerting page in the Google Cloud console. If you defined a notification mechanism in the alert policy, such as an email or SMS notification, Monitoring also sends a notification of the incident.