When I run the following command, I expect the exit code to be 0 since my combined
container runs a test that successfully exits with an exit code of 0.
docker-compose up --build --exit-code-from combined
Unfortunately, I consistently receive an exit code of 137 even when the tests in my combined
container run successfully and I exit that container with an exit code of 0 (more details on how that happens are specified below).
Below is my docker-compose version:
docker-compose version 1.25.0, build 0a186604
According to this post, the exit code of 137 can be due to two main issues.
- The container received a
docker stop
and the app is not gracefully handling SIGTERM - The container has run out of memory (OOM).
I know the 137 exit code is not because my container has run out of memory. When I run docker inspect <container-id>
, I can see that "OOMKilled" is false as shown in the snippet below. I also have 6GB of memory allocated to the Docker Engine which is plenty for my application.
[
{
"Id": "db4a48c8e4bab69edff479b59d7697362762a8083db2b2088c58945fcb005625",
"Created": "2019-12-12T01:43:16.9813461Z",
"Path": "/scripts/init.sh",
"Args": [],
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false, <---- shows container did not run out of memory
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2019-12-12T01:44:01.346592Z",
"FinishedAt": "2019-12-12T01:44:11.5407553Z"
},
My container doesn't exit from a docker stop
so I don't think the first reason is relevant to my situation either.
How my Docker containers are set up
I have two Docker containers:
- b-db - contains my database
- b-combined - contains my web application and a series of tests, which run once the container is up and running.
I'm using a docker-compose.yml file to start both containers.
version: '3'
services:
db:
build:
context: .
dockerfile: ./docker/db/Dockerfile
container_name: b-db
restart: unless-stopped
volumes:
- dbdata:/data/db
ports:
- "27017:27017"
networks:
- app-network
combined:
build:
context: .
dockerfile: ./docker/combined/Dockerfile
container_name: b-combined
restart: unless-stopped
env_file: .env
ports:
- "5000:5000"
- "8080:8080"
networks:
- app-network
depends_on:
- db
networks:
app-network:
driver: bridge
volumes:
dbdata:
node_modules:
Below is the Dockerfile for the combined
service in docker-compose.yml
.
FROM cypress/included:3.4.1
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 5000
RUN npm install -g history-server nodemon
RUN npm run build-test
EXPOSE 8080
COPY ./docker/combined/init.sh /scripts/init.sh
RUN ["chmod", "+x", "/scripts/init.sh"]
ENTRYPOINT [ "/scripts/init.sh" ]
Below is what is in my init.sh
file.
#!/bin/bash
# Start front end server
history-server dist -p 8080 &
front_pid=$!
# Start back end server that interacts with DB
nodemon -L server &
back_pid=$!
# Run tests
NODE_ENV=test $(npm bin)/cypress run --config video=false --browser chrome
# Error code of the test
test_exit_code=$?
echo "TEST ENDED WITH EXIT CODE OF: $test_exit_code"
# End front and backend server
kill -9 $front_pid
kill -9 $back_pid
# Exit with the error code of the test
echo "EXITING SCRIPT WITH EXIT CODE OF: $test_exit_code"
exit "$test_exit_code"
Below is the Dockerfile for my db
service. All its doing is copying some local data into the Docker container and then initialising the database with this data.
FROM mongo:3.6.14-xenial
COPY ./dump/ /tmp/dump/
COPY mongo_restore.sh /docker-entrypoint-initdb.d/
RUN chmod 777 /docker-entrypoint-initdb.d/mongo_restore.sh
Below is what is in mongo_restore.sh
.
#!/bin/bash
# Creates db using copied data
mongorestore /tmp/dump
Below are the last few lines of output when I run docker-compose up --build --exit-code-from combined; echo $?
.
...
b-combined | user disconnected
b-combined | Mongoose disconnected
b-combined | Mongoose disconnected through Heroku app shutdown
b-combined | TEST ENDED WITH EXIT CODE OF: 0 ===========================
b-combined | EXITING SCRIPT WITH EXIT CODE OF: 0 =====================================
Aborting on container exit...
Stopping b-combined ... done
137
What is confusing as you can see above, is that the test and script ended with exit code of 0 since all my tests passed successfully but the container still exited with an exit code of 137.
What is even more confusing is that when I comment out the following line (which runs my Cypress integration tests) from my init.sh
file, the container exits with a 0 exit code as shown below.
NODE_ENV=test $(npm bin)/cypress run --config video=false --browser chrome
Below is the output I receive when I comment out / remove the above line from init.sh
, which is a command that runs my Cypress integration tests.
...
b-combined | TEST ENDED WITH EXIT CODE OF: 0 ===========================
b-combined | EXITING SCRIPT WITH EXIT CODE OF: 0 =====================================
Aborting on container exit...
Stopping b-combined ... done
0
How do I get docker-compose to return me a zero exit code when my tests run successfully and a non-zero exit code when they fail?
EDIT:
After running the following docker-compose command in debug mode, I noticed that b-db seems to have some trouble shutting down and potentially is receiving a SIGKILL signal from Docker because of that.
docker-compose --log-level DEBUG up --build --exit-code-from combined; echo $?
Is this indeed the case according to the following output?
...
b-combined exited with code 0
Aborting on container exit...
http://localhost:None "GET /v1.25/containers/196f3e622847b4c4c82d8d761f9f19155561be961eecfe874bbb04def5b7c9e5/json HTTP/1.1" 200 None
http://localhost:None "GET /v1.25/containers/json?limit=-1&all=1&size=0&trunc_cmd=0&filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Db-property%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D HTTP/1.1" 200 3819
http://localhost:None "GET /v1.25/containers/196f3e622847b4c4c82d8d761f9f19155561be961eecfe874bbb04def5b7c9e5/json HTTP/1.1" 200 None
http://localhost:None "GET /v1.25/containers/0626d6bf49e5236440c82de4e969f31f4f86280d6f8f555f05b157fa53bae9b8/json HTTP/1.1" 200 None
http://localhost:None "GET /v1.25/containers/196f3e622847b4c4c82d8d761f9f19155561be961eecfe874bbb04def5b7c9e5/json HTTP/1.1" 200 None
http://localhost:None "GET /v1.25/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Db-property%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D HTTP/1.1" 200 4039
http://localhost:None "POST /v1.25/containers/196f3e622847b4c4c82d8d761f9f19155561be961eecfe874bbb04def5b7c9e5/attach?logs=0&stdout=1&stderr=1&stream=1 HTTP/1.1" 101 0
http://localhost:None "GET /v1.25/containers/196f3e622847b4c4c82d8d761f9f19155561be961eecfe874bbb04def5b7c9e5/json HTTP/1.1" 200 None
http://localhost:None "GET /v1.25/containers/196f3e622847b4c4c82d8d761f9f19155561be961eecfe874bbb04def5b7c9e5/json HTTP/1.1" 200 None
http://localhost:None "GET /v1.25/containers/0626d6bf49e5236440c82de4e969f31f4f86280d6f8f555f05b157fa53bae9b8/json HTTP/1.1" 200 None
Stopping b-combined ...
Stopping b-db ...
Pending: {<Container: b-db (0626d6)>, <Container: b-combined (196f3e)>}
Starting producer thread for <Container: b-combined (196f3e)>
http://localhost:None "GET /v1.25/containers/196f3e622847b4c4c82d8d761f9f19155561be961eecfe874bbb04def5b7c9e5/json HTTP/1.1" 200 None
http://localhost:None "GET /v1.25/containers/196f3e622847b4c4c82d8d761f9f19155561be961eecfe874bbb04def5b7c9e5/json HTTP/1.1" 200 None
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
http://localhost:None "GET /v1.25/containers/196f3e622847b4c4c82d8d761f9f19155561be961eecfe874bbb04def5b7c9e5/json HTTP/1.1" 200 None
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
Pending: {<Container: b-db (0626d6)>}
http://localhost:None "POST /v1.25/containers/196f3e622847b4c4c82d8d761f9f19155561be961eecfe874bbb04def5b7c9e5/wait HTTP/1.1" 200 32
http://localhost:None "POST /v1.25/containers/196f3e622847b4c4c82d8d761f9f19155561be961eecfe874bbb04def5b7c9e5/stop?t=10 HTTP/1.1" 204 0
http://localhost:None "GET /v1.25/containers/196f3e622847b4c4c82d8d761f9f19155561be961eecfe874bbb04def5b7c9e5/json HTTP/1.1" 200 None
http://localhost:None "POST /v1.25/containers/196f3e622847b4c4c82d8d761f9f19155561bStopping b-combined ... done
Finished processing: <Container: b-combined (196f3e)>
Pending: {<Container: b-db (0626d6)>}
Starting producer thread for <Container: b-db (0626d6)>
http://localhost:None "GET /v1.25/containers/196f3e622847b4c4c82d8d761f9f19155561be961eecfe874bbb04def5b7c9e5/json HTTP/1.1" 200 None
http://localhost:None "GET /v1.25/containers/0626d6bf49e5236440c82de4e969f31f4f86280d6f8f555f05b157fa53bae9b8/json HTTP/1.1" 200 None
Pending: set()
Pending: set()
Pending: set()
Pending: set()
Pending: set()
Pending: set()
http://localhost:None "GET /v1.25/containers/0626d6bf49e5236440c82de4e969f31f4f86280d6f8f555f05b157fa53bae9b8/json HTTP/1.1" 200 None
http://localhost:None "POST /v1.25/containers/0626d6bf49e5236440c82de4e969f31f4f86280d6f8f555f05b157fa53bae9b8/stop?t=10 HTTP/1.1" 204 0
http://localhost:None "POST /v1.25/containers/0626d6bf49e5236440c82de4e969f31f4f86280d6f8f555f05b157fa53bae9b8/wait HTTP/1.1" 200 30
Stopping b-db ... done
Pending: set()
http://localhost:None "GET /v1.25/containers/0626d6bf49e5236440c82de4e969f31f4f86280d6f8f555f05b157fa53bae9b8/json HTTP/1.1" 200 None
http://localhost:None "GET /v1.25/containers/196f3e622847b4c4c82d8d761f9f19155561be961eecfe874bbb04def5b7c9e5/json HTTP/1.1" 200 None
137