I'm in the beginning of developing a workflow yml script. The main idea is to have a manually triggered "remote server" build file, to set up all the necessary applications in a "Host Server" that will later handle docker containers and other workflows will be handles by merge triggers.
Problem I'm facing is with the first step of the server configuration. Manually, I do:
apt update
apt apt upgrade -y
sudo reboot
Wait for the server to reboot (come back online) so it loads the new kernel and continue with the installation.
I'm trying to automate the setup in GitHub Actions Workflow. Up till now, this is how my file looks like:
name: Host Server Build V1.0.0
on:
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
env:
CONFIG_SERVER_SSH_KEY: '${{ secrets.CONFIG_SERVER_SSH_KEY }}'
CONFIG_SERVER_BACKEND_IP: '${{ secrets.CONFIG_SERVER_BACKEND_IP }}'
steps:
- name: Ubuntu Update / Upgrade / Reboot (back-end)
shell: bash
run: |
BACKEND_HOST_SERVER_SH_SCRIPT=' \
sudo DEBIAN_FRONTEND=noninteractive apt update && sudo DEBIAN_FRONTEND=noninteractive apt upgrade -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold"; \
sleep 5; \
sudo reboot || true; \
';
echo "${{ env.CONFIG_SERVER_SSH_KEY }}" > id_rsa_server_private_key_temp.pem;
chmod 600 id_rsa_server_private_key_temp.pem;
ssh -v -t -t -i id_rsa_server_private_key_temp.pem -o ConnectTimeout=300 -o StrictHostKeyChecking=no ubuntu@${{ env.CONFIG_SERVER_BACKEND_IP }} "$BACKEND_HOST_SERVER_SH_SCRIPT"
# rm id_rsa_server_private_key_temp.pem;
echo "Status check: Reboot initiated.";
sleep 60;
# Verify if the server is back online
# until ping -c 1 ${{ env.CONFIG_SERVER_BACKEND_IP }} & > /dev/null; do
# echo "Server not yet available, retrying in 15 seconds...";
# sleep 15;
# done;
# echo "Status check: Server is back online.";
echo 'Waiting for server to come back online...';
until ssh -o StrictHostKeyChecking=no -i id_rsa_server_private_key_temp.pem ubuntu@${{ env.CONFIG_SERVER_BACKEND_IP }} 'echo "Server is back online"'; do
echo "Server not yet available, retrying in 30 seconds...";
sleep 30;
done
echo 'Status check: Server is back online, proceeding with the workflow.';
rm id_rsa_server_private_key_temp.pem;
- name: Docker - Install
shell: bash
run: |
BACKEND_HOST_SERVER_SH_SCRIPT=' \
sudo apt install -y apt-transport-https ca-certificates curl software-properties-common; \
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg; \
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null; \
sudo apt install -y docker-ce docker-ce-cli containerd.io; \
';
echo "${{ secrets.CONFIG_SERVER_SSH_KEY }}" > id_rsa_server_private_key_temp.pem;
chmod 600 id_rsa_server_private_key_temp.pem;
ssh -v -t -t -i id_rsa_server_private_key_temp.pem -o StrictHostKeyChecking=no ubuntu@${{ env.CONFIG_SERVER_BACKEND_IP }} "$BACKEND_HOST_SERVER_SH_SCRIPT"
rm id_rsa_server_private_key_temp.pem;
echo "Status check: Docker and dependencies installed successfully.";
I had to modify apt upgrade -y part because of the prompts it would be stuck on.
However, seems like the runner is encountering an error after reboot and can't continue to the next steps. The error looks something like this:
NEEDRESTART-VER: 3.5
NEEDRESTART-KCUR: 6.***.***-1***17-aws
NEEDRESTART-KEXP: 6.***.***-1***18-aws
NEEDRESTART-KSTA: 3
NEEDRESTART-SVC: acpid.service
NEEDRESTART-SVC: chrony.service
NEEDRESTART-SVC: cron.service
NEEDRESTART-SVC: dbus.service
NEEDRESTART-SVC: [email protected]
NEEDRESTART-SVC: networkd-dispatcher.service
NEEDRESTART-SVC: packagekit.service
NEEDRESTART-SVC: polkit.service
NEEDRESTART-SVC: rsyslog.service
NEEDRESTART-SVC: serial-getty@ttyS***.service
NEEDRESTART-SVC: snapd.service
NEEDRESTART-SVC: systemd-logind.service
NEEDRESTART-SVC: unattended-upgrades.service
NEEDRESTART-SVC: user@1***.service
debug1: channel ***: free: client-session, nchannels 1
Connection to *** closed by remote host.
Connection to *** closed.
Transferred: sent ***9***, received 1***584 bytes, in 83.*** seconds
Bytes per second: sent 34.9, received 1***9.3
debug1: Exit status -1
Anyone has an idea of how this can be done? Or if this type of approach is correct? Maybe splitting into 2 jobs? Initially, I wanted to put the "wait" part into a separate step, like this:
- name: Server - Wait for it to be Back Online (back-end)
shell: bash
run: |
echo "${{ secrets.CONFIG_SERVER_SSH_KEY }}" > id_rsa_server_private_key_temp.pem
chmod 600 id_rsa_server_private_key_temp.pem
echo 'Waiting for server to come back online...';
until ssh -o StrictHostKeyChecking=no -i id_rsa_server_private_key_temp.pem ubuntu@${{ env.CONFIG_SERVER_BACKEND_IP }} 'echo "Server is back online"'; do
echo "Server not yet available, retrying in 30 seconds...";
sleep 30;
done
echo 'Status check: Server is back online, proceeding with the workflow.';
rm id_rsa_server_private_key_temp.pem;
Server context:
- AWS / EC2
- Ubuntu
- Fresh install
The solution with the help of Daniel B:
The solution with the help of Daniel B: How my YML file step ended up looking like.
steps:
- name: Ubuntu Update / Upgrade / Reboot (back-end)
shell: bash
run: |
BACKEND_HOST_SERVER_SH_SCRIPT=' \
sudo DEBIAN_FRONTEND=noninteractive apt update && sudo DEBIAN_FRONTEND=noninteractive apt upgrade -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold"; \
';
echo "${{ env.CONFIG_SERVER_SSH_KEY }}" > id_rsa_server_private_key_temp.pem;
chmod 600 id_rsa_server_private_key_temp.pem;
ssh -v -t -t -i id_rsa_server_private_key_temp.pem -o ConnectTimeout=300 -o StrictHostKeyChecking=no ubuntu@${{ env.CONFIG_SERVER_BACKEND_IP }} "$BACKEND_HOST_SERVER_SH_SCRIPT"
ssh -v -t -t -i id_rsa_server_private_key_temp.pem -o ConnectTimeout=300 -o StrictHostKeyChecking=no ubuntu@${{ env.CONFIG_SERVER_BACKEND_IP }} "sudo reboot" || true;
echo "Status check: Reboot initiated.";
sleep 60;
echo 'Status check: Waiting for server to come back online...';
until ssh -o StrictHostKeyChecking=no -i id_rsa_server_private_key_temp.pem ubuntu@${{ env.CONFIG_SERVER_BACKEND_IP }} 'echo "Server is back online"'; do
echo "Status check: Server not yet available, retrying in 30 seconds...";
sleep 30;
done
echo 'Status check: Server is back online, proceeding with the workflow.';
rm id_rsa_server_private_key_temp.pem;
