VictoriaLogs: Open Source Log Management System - Preview

© 2023 VictoriaMetrics
VictoriaLogs: preview
Aliaksandr Valialkin, CTO VictoriaMetrics

Existing open source log management systems
● ELK stack - https://www.elastic.co/observability/log-monitoring
● Grafana Loki - https://grafana.com/oss/loki/

ELK stack
● Strong points:
○ Fast full-text search via ElasticSearch

ELK stack
● Strong points:
○ Widespread usage of Logstash and Fluentbit for logs’ ingestion

ELK stack
● Strong points:
○ Advanced log analysis via Kibana

ELK stack
● Strong points:
● Weak points:
○ Slow data ingestion

ELK stack
● Strong points:
● Weak points:
○ High CPU and RAM usage during data ingestion

ELK stack
● Strong points:
● Weak points:
○ Bad on-disk compression for stored logs

ELK stack
● Strong points:
● Weak points:
○ Missing log stream concept

ELK stack
● Strong points:
● Weak points:
○ Missing ability for querying advanced stats from access logs

ELK stack
● Strong points:
● Weak points:
○ Missing ability for querying advanced stats from access logs
○ Missing CLI integration (e.g. $ elk “some query” | grep … | sort … | tail)

Grafana Loki
● Strong points:
○ Lower CPU and RAM usage during data ingestion

Grafana Loki
● Strong points:
○ Good on-disk compression for stored logs

Grafana Loki
● Strong points:
○ Log stream concept - https://grafana.com/docs/loki/latest/fundamentals/overview/

Grafana Loki
● Strong points:
○ Ability to query advanced stats from access logs

Grafana Loki
● Strong points:
○ CLI integration - https://grafana.com/docs/loki/latest/tools/logcli/

Grafana Loki
● Strong points:
● Weak points:
○ LogQL is non-trivial to use for typical queries - https://grafana.com/docs/loki/latest/logql/

Grafana Loki
● Strong points:
● Weak points:
○ Full-text search queries may be slow

Grafana Loki
● Strong points:
● Weak points:
○ Grafana UI for logs isn’t so good compared to Kibana

Grafana Loki
● Strong points:
● Weak points:
○ Grafana UI for logs isn’t so good compared to Kibana
○ Missing ability to set individual labels per each log line (ip, user_id, trace_id, etc.) during data
ingestion - https://grafana.com/docs/loki/latest/fundamentals/labels/ . This frequently leads to
high cardinality issues.

What is VictoriaLogs?
● Open source log management system from VictoriaMetrics

● Easy to setup and operate

● Scales vertically and horizontally

● Optimized for low resource usage (CPU, RAM, disk space)

● Accepts data from Logstash and Fluentbit in Elasticsearch format

● Accepts data from Promtail in Loki format

● Supports stream concept from Loki

● Provides easy to use yet powerful query language - LogsQL

● Provides easy to use yet powerful query language - LogsQL
● It is in active development right now

LogsQL examples: search by time
● _time:-5m.. - search for logs for the last 5 minutes

● _time:-1h..-5m - search for logs on the time range [now()-1h … now()-5m]

● _time:2023-03-01..2023-03-31 - search for logs at March 2023

● _time:2023-03 - the same as above

● _time:2023-03-20T22 - search for logs at March 20, 2023, at 22 UTC

● _time:2023-03-20T22 - search for logs at March 20, 2023, at 22 UTC
● It is recommended specifying _time ﬁlter in order to narrow down the search
scope and speed up the query

LogsQL examples: full-text search
● error - case-insensitive search for log messages with “error” word. Messages with
“Error”, “ERROR”, “eRRoR”, etc. will be also found

● fail* - case-insensitive search for log messages with words starting with “fail”,
such as “fail”, “failure”, “FAILED”, etc.

● “Error” - case-sensitive search for log messages with the “Error” word

● “failed to open” - search for logs containing “failed to open” phrase

● exact(“foo bar”) - search for logs with the exact “foo bar” message

● exact(“foo bar”) - search for logs with the exact “foo bar” message
● re(“https?://[^s]+”) - search for logs matching the given regexp, e.g. logs with
http or https urls

LogsQL examples: combining search queries
● error OR warning - search for log messages with either “error” or “warning”
words

words
● err* AND fail* - search for logs containing both words, which start from err and
fail. For example, “ERROR: the ﬁle /foo/bar failed to open”

words
● error AND NOT “/foo/bar” - search for logs containing “error” word, but without
“/foo/bar” string

words
● error AND NOT “/foo/bar” - search for logs containing “error” word, but without
“/foo/bar” string
● _time:-1h.. AND (error OR warning) AND NOT debug - search for logs for the
last hour with either “error” or “warning” words, but without “debug” word

LogsQL examples: searching arbitrary labels
● By default the search is performed in the log message

● Every log entry can contain additional labels. For example, level, ip, user_id,
trace_id, etc.

● Every log entry can contain additional labels. For example, level, ip, user_id,
trace_id, etc.
● LogsQL allows searching in any label via label_name:query syntax

● level:(error or warning) - case-insensitive search for logs with level label
containing “error” or “warning”

● level:(error or warning) - case-insensitive search for logs with level label
containing “error” or “warning”
● trace_id:”012345-6789ab-cdef” AND error - search for logs with the given
trace_id label, which contain “error” word

What is a log stream?
● A log stream is logs generated by a single instance of some application:
○ Some Linux process (Unix daemon)
○ Docker container
○ Kubernetes container running in a pod

● A log stream is logs generated by a single instance of some application:
○ Some Linux process (Unix daemon)
○ Docker container
○ Kubernetes container running in a pod
● Logs belonging to a single stream are traditionally written to a single ﬁle and
investigated with cat, grep, sort, cut, uniq, tail, etc. commands.

● A stream in distributed system can be uniquely identiﬁed by the instance location
such as its TCP address (aka instance label in Prometheus ecosystem).

● Multiple instances of a single application (aka shards or replicas) can be identiﬁed
by the application name (aka job label in Prometheus ecosystem).

● Multiple instances of a single application (aka shards or replicas) can be identiﬁed
by the application name (aka job label in Prometheus ecosystem).
● Additional labels can be attached to log streams, so they could be used during log
analysis. For example, environment, datacenter, zone, namespace, etc.

● ELK misses the concept of log streams, so it may be non-trivial to perform
stream-based log analysis there.

● Grafana Loki supports the log stream concept from the beginning.

● Grafana Loki supports the log stream concept from the beginning.
● VictoriaLogs provides support for log streams.

LogsQL examples: querying log streams
● VictoriaLogs allows querying streams via _stream label with Prometheus label
ﬁlters

ﬁlters
● _stream:{job=”nginx”} - search for logs from nginx streams

ﬁlters
● _stream:{env=~”qa|staging”,zone!=”us-east”} - search for log streams from qa or
staging environments at all the zones except us-east

ﬁlters
● _stream:{env=~”qa|staging”,zone!=”us-east”} - search for log streams from qa or
staging environments at all the zones except us-east
● _time:-1h.. AND _stream:{job=”nginx”} AND level:error - search for logs for the
last hour from nginx streams with the level label containing the “error” word

Stream labels vs log labels
● Stream labels remain static, e.g. they do not change across logs belonging to the
same stream.

same stream.
● The recommended stream labels are instance and job. These labels simplify
correlation between Prometheus metrics and logs.

same stream.
● Stream labels allow grouping logs by individual streams during querying, which
can simplify log analysis.

same stream.
● Stream labels allow grouping logs by individual streams during querying, which
can simplify log analysis.
● Stream labels can be used for narrowing down the amounts of logs to search and
optimizing the query speed.

● Log labels can change inside the same stream. For example, level, trace_id, ip,
user_id, response_duration, etc.

● Log labels are known as log ﬁelds from structured logging.

● Log labels are known as log ﬁelds from structured logging.
● Searching via log labels simplify narrowing down the search results.

LogsQL: stats over access logs

● It is quite common to collect access logs (e.g. nginx or apache logs)

● It is quite common to analyze access logs with grep, cut, sort, uniq, tail, etc.
commands

commands
● Examples:
○ To get the top 10 paths with the biggest number of 404 HTTP errors
○ To calculate per-domain p99 response duration and the number of requests
○ To get the number of unique IPs, which requested the given url

commands
● Examples:
● ELK and Grafana Loki do not provide functionality to efﬁciently perform these
tasks :(

commands
● Examples:
● ELK and Grafana Loki do not provide functionality to efﬁciently perform these
tasks :(
● VictoraLogs comes to rescue!

LogsQL: stats over access logs: example
Top 10 paths from nginx streams at prod for the last hour with the biggest
number of requests, which led to 404 error.
Additionally, calculate the number of unique ip addresses seen per path.

_time:-1h.. AND _stream:{job=”nginx”,env=”prod”}
Select logs from nginx at prod for the last hour

_time:-1h.. AND _stream:{job=”nginx”,env=”prod”} |
extract ‘<ip> <*> “<*> <path> <*>” <status>’
Extract ip, path and http status code from nginx log message

extract ‘<ip> <*> “<*> <path> <*>” <status>’ |
filter status:404
Filter logs with http status code = 404

filter status:404 |
stats by (path) (
count() as requests,
uniq(ip) as uniq_ips,
)
Count the number of requests and unique ip addresses per each path

filter status:404 |
stats by (path) (
) |
sort by requests desc
Sort by requests in descending order

filter status:404 |
stats by (path) (
) |
sort by requests desc |
limit 10
Leave the ﬁrst 10 entries

VictoriaLogs: CLI integration
VictoriaLogs comes with vlogs-cli tool, which can be combined with the traditional CLI
commands during log investigation.

VictoriaLogs: CLI integration
VictoriaLogs comes with vlogs-cli tool, which can be combined with the traditional CLI
commands during log investigation.
Example: obtain logs from nginx streams for the last hour and then feed the results to
standard CLI tools - jq, grep and tail - for further processing:
vlogs-cli -q ‘_time:-1h.. AND _stream:{job=”nginx”}’ | jq ._msg | grep 404 | tail

VictoriaLogs: recap
● Open source solution for log management
● Easy yet powerful query language - LogsQL
● Scales both vertically and horizontally
● Supports data ingestion from Logstash, Fluentd and Promtail

When VictoriaLogs will be ready to use?
Soon - it is in active development now

Is VictoraLogs open source?
Yes!

How about enterprise features?
Sure! GDPR, security, auth, rate limiting and anomaly
detection will be available in VictoriaLogs enterprise!

How does VictoriaLogs compare to
ClickHouse for logs?
VictoriaLogs uses core optimizations similar to ClickHouse
VictoriaLogs is easier to setup and operate than ClickHouse

Will VictoriaLogs provide datasources for
Grafana and Kibana?
Yes, eventually

How about cloud version of VictoriaLogs?
Yes, eventually

Will VictoriaLogs support JSON and
structured logs?
Yes, from day one!

Will data partitioning be supported?
Yes, VictoriaLogs partitions data by weeks
Partitions are self-contained and can be removed / moved /
archived independently

VictoriaLogs: questions?
Aliaksandr Valialkin, CTO VictoriaMetrics

VictoriaLogs: Open Source Log Management System - Preview

Related slideshows

More Related Content

VictoriaLogs: Open Source Log Management System - Preview