1

I've updated pm2 from 2.10.1 to 3.5.1. The god daemon process gets high memory usage comparing with the old version. Also about 15 minutes after start the pm2 suddenly kills with the following error. 

 ==== JS stack trace =========================================

    0: ExitFrame [pc: 0x24b91f2dbe1d]
Security context: 0x065f6f41e6e9 <JSObject>
    1: /* anonymous */(aka /* anonymous */) [0x162edca8f91] [/usr/lib/node_modules/pm2/node_modules/js-git/mixins/fs-db.js:~300] [pc=0x24b91f6da008](this=0x0308a98826f1 <undefined>,data=0x13d779ef8e21 <Object map = 0x2d75412e0a29>)
    2: arguments adaptor frame: 3->1
    3: parseIndex(aka parseIndex) [0x3c7e7b5b4151] [/usr/lib/node_modules/pm2/node_modules/...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8f9d10 node::Abort() [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
 2: 0x8f9d5c  [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
 3: 0xaffd0e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
 4: 0xafff44 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
 5: 0xef4152  [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
 6: 0xef4258 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
 7: 0xf00332 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
 8: 0xf00c64 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
 9: 0xf0215a v8::internal::Heap::CollectAllAvailableGarbage(v8::internal::GarbageCollectionReason) [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
10: 0xf0393e v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
11: 0xecc9c6 v8::internal::Factory::AllocateRawArray(int, v8::internal::PretenureFlag) [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
12: 0xecd24a v8::internal::Factory::NewFixedArrayWithFiller(v8::internal::Heap::RootListIndex, int, v8::internal::Object*, v8::internal::PretenureFlag) [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
13: 0xecd2f0 v8::internal::Handle<v8::internal::FixedArray> v8::internal::Factory::NewFixedArrayWithMap<v8::internal::FixedArray>(v8::internal::Heap::RootListIndex, int, v8::internal::PretenureFlag) [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
14: 0x100c7d9 v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::New(v8::internal::Isolate*, int, v8::internal::PretenureFlag, v8::internal::MinimumCapacity) [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
15: 0x10316d2 v8::internal::BaseNameDictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::EnsureCapacity(v8::internal::Handle<v8::internal::NameDictionary>, int) [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
16: 0x10318a0 v8::internal::BaseNameDictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::AddNoUpdateNextEnumerationIndex(v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, int*) [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
17: 0x1031b4b v8::internal::BaseNameDictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add(v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, int*) [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
18: 0x1195188 v8::internal::Runtime_AddDictionaryProperty(int, v8::internal::Object**, v8::internal::Isolate*) [PM2 v3.5.1: God Daemon (/home/invoker/.pm2)]
19: 0x24b91f2dbe1d 
2019-07-30T12:35:09: PM2 log: ===============================================================================
2019-07-30T12:35:09: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
2019-07-30T12:35:09: PM2 log: Time                 : Tue Jul 30 2019 12:35:09 GMT+0430 (Iran Daylight Time)
2019-07-30T12:35:09: PM2 log: PM2 version          : 3.5.1
2019-07-30T12:35:09: PM2 log: Node.js version      : 10.16.0
2019-07-30T12:35:09: PM2 log: Current arch         : x64
2019-07-30T12:35:09: PM2 log: PM2 home             : /home/invoker/.pm2
2019-07-30T12:35:09: PM2 log: PM2 PID file         : /home/invoker/.pm2/pm2.pid
2019-07-30T12:35:09: PM2 log: RPC socket file      : /home/invoker/.pm2/rpc.sock
2019-07-30T12:35:09: PM2 log: BUS socket file      : /home/invoker/.pm2/pub.sock
2019-07-30T12:35:09: PM2 log: Application log path : /home/invoker/.pm2/logs
2019-07-30T12:35:09: PM2 log: Process dump file    : /home/invoker/.pm2/dump.pm2
2019-07-30T12:35:09: PM2 log: Concurrent actions   : 2
2019-07-30T12:35:09: PM2 log: SIGTERM timeout      : 1600
2019-07-30T12:35:09: PM2 log: ===============================================================================

I'm starting pm2 with the following command: 

pm2 start process.config.js --node-args="--max_new_space_size=4096 --max_executable_size=4096 --max_old_space_size=4096"

And in the process.config.js has about 100 processes with this format:

 {
  "services": [
    {
      "name": "a",
      "language": "nodejs",
      "replicas": 1,
      "script": "./a/index.js",
      "log_type": "json",
      "exec_mode": "fork",
      "max_memory_restart": "500M",
      "env_production": {
        "NODE_ENV": "PRODUCTION",
        "SERVICE": "a"
      }
    },
    {
      "name": "b",
      "language": "nodejs",
      "replicas": 1,
      "script": "./b/index.js",
      "log_type": "json",
      "exec_mode": "fork",
      "max_memory_restart": "500M",
      "env_production": {
        "NODE_ENV": "PRODUCTION",
        "SERVICE": "b"
      }
    }
  ]
}

I have no issue with the old version of pm2 and nothing get stopped even after days, but in pm2 3.5.1 it runs for only 10-15 minutes.

Improve this question

1 Answer 1

Reset to default
0

PM2 uses vizion and js-git dependency doesn't work properly with big git repositories.

You can find that module tries to load .git/objects/pack in js-git sources:

  function loadIndex() {
    var indexFile = pathJoin(packDir, "pack-" + packHash + ".idx" );
    fs.readFile(indexFile, function (err, buffer) {
      if (!buffer) return callback(err);
      try {
        cachedIndexes[packHash] = parseIndex(buffer);
      }
      catch (err) { return callback(err); }
      onIndex();
    });
  }

If you have huge git repo, it's easy to get OOM here. You can disable vizion (it's true by default), if you don't need such functionality.

http://pm2.keymetrics.io/docs/usage/application-declaration/

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .