Cron jobs not running on Kubernetes + Alpine linux

Question

I am trying to run a cron job on Alpine Linux docker image which is then pushed in Kubernetes container. The Cron job is that I want the container to restart itself after every 15 mins. So my command is pkill processname.

Little bit about Alpine Cron jobs: For Alpine Linux cron is followed periodically that is it has already created folders /15min. /hourly. /daily. /weekly. /monthly. So I have to just copy my script in one of the folders to run it.

I created the script and as requested on Alpine FAQ's pages I didn't give any name extension to the file and copied it with execute permissions in the folder. When I try and run command run-parts --test /etc/periodic/15min it shows me that it has one script that needs to be executed. Although that script never executes. When I try and run that docker image on my local machine, not exactly with that script but a different one where I am just creating a file in the folder, it works well.

I have also tried and run crond -f -l 8 on Kube cluster and ti still does not work.

I would really appreciate help on this part what I am missing or where is it going wrong.

Answer 1

Wellcome to SuperUser.

First of all, maybe you're facing it the wrong way. If the goal is to restart the complete container every 15 minutes, maybe you should check the links bellow.

Question about restarting pods with multiple possible solutions

Pod lifecycle documentation

Kubernetes concepts - Explanation about how kubernetes works and the names people uses for things there.

Just for completity, and in case you actually want to run a periodic task, check specific Kubernetes docs, they state how to create cronjobs and debug them. I'd also suggest to read about Kubernetes limitations in documentation, as there may be something actively preventing to run your cronjobs

GCP Documentation about cronjobs

Kubernetes documentation about Cronjobs

I'm giving you a resume about how to create a cronjob task, but you should carefully read the above if you're not familiar with kubernetes and follow the links inside, they've environments ready to play and learn the basics of kubernetes.

It is more elaborated inside, but it basically gives you an example command and the way to deploy it. I've modified it to be once every 15 minutes and run a pkill. You've to create a yaml file with the following contents (check syntax for more complex cron behaviours):

# cronjob.yaml
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: hello
spec:
  schedule: "*/15 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            args:
            - /bin/sh
            - -c
            - pkill yourprocess
          restartPolicy: OnFailure

and then deploy it using

kubectl apply -f [FILENAME]

Use the following command to check if your cronjob was deployed:

kubectl describe cronjob [CRON_JOB]

Post describing the above processes with more screenshots and examples

After this point this is a very generic answer. You can post comments and I'll try to help you out, but I currently have no access to an Alpine Linux installation, so I may not be helpful. This is the Alpine Linux documentation page addressing your issue

Tracing your error: The simple way

There's a very simple way to narrow down whats going out, although it is not very precise. There are two chances here: There's a problem with CRON itself, or you have a problem with your script (permissions, for instance). The process is simple:

• Create a task that creates an empty file inside a folder where you're ABSOLUTLY SURE everyone can read and write.
• If the new file appears when the task is suposed to run, it means that your cron is executing properly and that you have a problem with your script. Go to "Script problems" section. If not, you've a problem with cron. Go to Cron problems section

Tracing your error: Logs

The best way to solve problems in any system is looking at the logs. Software like cron normally stores logs on syslog (normally located at /var/log/syslog), unless configured otherwise. This post speaks about cron logging folders.

If you find out where your cron instance is logging and there's nothing there, it means that your cron service is not running at all. There's a Alpine Linux specific solution for this problem in Cron problems. If you can see errors, you should look at them and try to narrow down your problem. If not, you should make sure that your script is actually not running.

Cron problems

Ok, now you know cron is not working properly. There are various possible scenarios:

• Cron is not running at all

Maybe the cron service is not started. Check if the service is running. If it is not, try to start it manually. Check if it has started properly and then add it to startup process. The way to do so varies depending on the OS. In Alpine Linux, this should do the trick:

rc-service crond start && rc-update add crond

• The cron folder you're using is not being used by cron

You should check the main cron config file in order to verify that the folder you're putting scripts on is actually added.

• Typos

Maybe the config line has some error with spaces on arguments or things like that, although in this case this seem not to be the scenario, as you used test commands on your folder.

Script problems

If your script runs but it is not producing the desired output, it is a good idea to perform the following steps:

• Separate your script into a file. Give it run permissions and ensure that the first line properly states the shell to use, in Alpine's case, this line should be #!/bin/sh
• Redirect the standard and error output to a file to analize what's going on.

General suggestions

• As stated in Script problems section, it is a good practise to separate your oneliners into script files and point their output to files, in order to be able to debug problems of all kinds.
• Take a close look at directory and file permissions (those directly accesed by your script and those that cron should read / user in order to execute your scripts), it is a common cause of failure, specially in containers.

EDIT: Maybe I understood your situation in the wrong way and you're in an scenario like this one