1

I have cardano-node and db-sync running just fine, latest versions. Every now and then though, db-sync stops and message in console says "Killed". This was not the case before the Vasil HF.

Why does db-sync stop and exit with message "Killed" ?

I start it back up, and it starts np - and runs fine for awhile, but sooner or later, after days...for some reason it will stop and the last thing in the console window is one line with the word: Killed.

Improve this question

3 Answers 3

Reset to default
5

db-sync has a rather high memory usage. We (I am one of the developers) are working on reducing it, but for now, the only solution is to provide more memory.

If you are running node on the same machine as db-sync (this is the recommended setup) the machine will likely need 32G of RAM.

Improve this answer
4
  • 1
    I am not sure for the current version, but the previous version (which worked with cardano-node 1.34.1) did not run for many days with only 32 GB of RAM. I had to increase to 48 GB RAM to avoid it being killed for out of memory.
    – George - APEX Stake Pool
    Commented Oct 30, 2022 at 7:48
  • ok, thank you George & Erik. I got a machine that has 32 GB of RAM as that was the spec given in the docs at the time I was setting it up. I will examine using dmesg -T to see if that msg is indicating RAM exhaustion is indeed the culprit. If so, I'll just be glad I have determined the cause and buy more RAM. I will say though: pre-Vasil HF I was running on the 32 GB of RAM and never experienced the same over months of running. I will leave this open and once examined, report back here.
    – RelevantData
    Commented Oct 30, 2022 at 17:08
  • This has little or nothing to do with Vasil. Its caused by the growing size of the blockchain itself and the number of stake addresses slowly increasing.
    – Erik de Castro Lopo
    Commented Oct 30, 2022 at 19:19
  • Even with 32G you will still need swap space.
    – Erik de Castro Lopo
    Commented Dec 1, 2022 at 21:35
4

Most likely it is killed because the server's RAM is exhausted. How much RAM does the server have? I assume it is running on mainnet. You can double check with dmesg -T to see if it was really killed for Out of Memory (oom). You should see something like this (but for cardano-db-sync, not for cardano-node):

[Wed Sep 21 15:18:41 2022] Out of memory: Kill process 77436 (cardano-node) score 637 or sacrifice child
[Wed Sep 21 15:18:41 2022] Killed process 77436 (cardano-node) total-vm:1075768476kB, anon-rss:13651644kB, file-rss:0kB, shmem-rss:0kB
Improve this answer
-1

RAM. That's it. I have 32 GB on board, but although it was enough until recently the node & db-sync combined on one machine are using about 26 GB - which on Ubuntu, unless I do some tweaking, makes 32 GB not quite enough.

Thanks George & Erik. here's a couple screen caps to show:

enter image description here

enter image description here

Luckily the system tells me there's space enough to expand to 64 GB, so I will.

enter image description here

Improve this answer
1
  • 1
    The best solution is to add RAM, but adding say 8G of swap space might be a good shorter term solution.
    – Erik de Castro Lopo
    Commented Oct 31, 2022 at 7:07

Not the answer you're looking for? Browse other questions tagged or ask your own question.