0

I have a Graylog server (newest version) collecting data from nginx. It´s been running for a while. I retrieve the collected data by querying my Elasticsearch cluster (v7 newest version) which consists of 4 nodes ATM. This all work(s/ed) fine. Now I have the problem that Elasticsearch won´t return the newest data. I checked that it exists. I do not see any differences to previous data stored. I did have a downtime (aka crash) because my disks where running full. All statuses show that the system is running fine again.

I checked everything I could find but ran out of ideas. Where do I need to look to solve this? I appreciate any ideas!

Update

What I also noticed is that I increased the Index Replicas from 0 (default) to 1 for the default index set but there seems to be no replicas. Shouldn´t those be created automatically after changing the settings?

enter image description here

Improve this question
2
  • Can you take a look at the Nodes status page and see if the nodes aren't storing data in the Disk Journal?
    – Swisstone
    Commented May 12, 2022 at 20:53
  • Thank you for your comment @Swisstone If you are referring to the Graylog Disk Journal then yes there is data written to it.
    – OpenHaus
    Commented May 12, 2022 at 21:45

1 Answer 1

Reset to default
1
+100

I am not sure exactly how to guide you, but have you looked at the log agent?

From my experience, I can not think of a simple reason that ES would not return data that are digested, except when querying with wrong filters such as wrong datetime (timezone changes).

After a crash on the ElasticSearch side of things, some times filebeat requires a restart. You should check its logs too, it might help you pin point the problem.

On the replication option, now.

Τhe option index.number_of_replicas is a dynamic option and should be updated live. If this option is passed through the index template it gets applied on index rotation ( aka, a new index is spawned ). Otherwise you might need to update it on the index itself

Improve this answer
4
  • Thank you @cr3a7ure ! I rotated the active write index manually which created the replicas. I do not think filebeat is used in our setup. Did I read correctly that query logging is only available with a paid subscriptions? The logs do not display any helpful information.
    – OpenHaus
    Commented May 23, 2022 at 21:14
  • I brought up filebeat as a typical example agent. I have no hands on experience with Graylog. I can see that they use an agent called Forwarder. Other than that, I would ask in their forums too, about subscriptions and pricing/features.
    – cr3a7ure
    Commented May 24, 2022 at 7:56
  • I solved it. At the end it wasn´t Elastic at all. You were right there was a timezone offset which let to the missing results.
    – OpenHaus
    Commented May 24, 2022 at 10:37
  • Happy to help. I had stepped on it by myself once and learned the hard way!
    – cr3a7ure
    Commented May 24, 2022 at 11:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .