How I should run my Golang process in background?

Question

This question is not strictly programming related, but for sure important for programmers.

I wrote a simple smtp server, when I run it from console all is fine, except it is blocking the command line.

I know I can run it via

nohup ... &

or via screen / tmux etc

But the question is, how should I implement my program it runs in the background and it will be a pleasure for a system administrator to set it up and manage the process ?

Some guys with far more experience than me, at golang-nuts, wrote, they don't use fork etc, and use some "wrapper" in form from monit etc.

The target platform is Debian based, all other stuff on the box are init.d based.

Any good resources for that topic or sources of a well written example project ?

Answer 1

As Nick mentioned Supervisord is a great option that has also worked well in my experience.

Nick mentioned problems with forking- forking itself works fine AFAICT. The issue is not forking but dropping privileges. Due to the way the Go runtime starts the thread pool that goroutines are multiplexed over (when GOMAXPROX > 1), the setuid systemcall is not a reliable way to drop permissions.

Instead, you should run your program as a non-privileged user and use the setcap utility to grant it the needed permissions.

For example, to allow binding to a low port number (like 80) run will need to run setcap once on the executable: sudo setcap 'cap_net_bind_service=+ep' /opt/yourGoBinary

You may need to install setcap: sudo aptitude install libcap2-bin

Answer 2

There are already good answers but I will add some additional information.

You do not need to install additional software such as supervisord on Debian to take care of backgrounding the process.

Debian comes with a tool called start-stop-daemon which is a standard way for starting daemons in init.d scripts. It can also also put the process in background for you if the program does not do it on its own. Have a look at the --background option.

Use /etc/init.d/skeleton as the basis of your init script, but change the do_start() function as follows:

start-stop-daemon --start --quiet --pidfile $PIDFILE --make-pidfile \
    --background --exec $DAEMON --test > /dev/null \
            || return 1
start-stop-daemon --start --quiet --pidfile $PIDFILE --make-pidfile \
    --background --exec $DAEMON -- $DAEMON_ARGS \
            || return 2

I also added the --make-pidfile option which creates the PID file for you.

In case you need to switch to a different user in a secure way, there is also --chuid option.

On Ubuntu and RHEL/CentOS/SL 6.X the simplest way is to write an upstart job configuration file. Just put exec /usr/sbin/yourprogram in the /etc/init/yourprogram.conf configuration file. With upstart there is no need to force the program in background. Do not add expect fork or expect daemon which you need with traditional daemons. With upstart it is better if the process does not fork.

Answer 3

I wrote a blog post about this a while back. The idea of daemonizing seems wrong to me as it leaves you with a lot of other things to worry about (e.g. what happens when it fails? how do you manage a process restart? how do you handle logging, working directory, cores, system restart, etc...)

Turns out, if you don't try to do all that, things get much easier.

Answer 4

Supervisord works really well for this in my experience.

You write your app to run on the command line, print stuff etc and supervisord takes care of all the daemonising, restarting if it goes wrong, rate limiting etc, etc

I believe that forking go programs into the background in the traditional unix way is difficult because the runtime starts some threads before it runs your main() routine

Answer 5

But the question is, how should I implement my programm it runs in the background and it will be a pleasure for a system administrator to set it up and manage the process ?

A few thoughts here.

Provide packages and a repository

Installing a software is one thing, maintaining and running it is a totally different story. Sure, I can download a zip, unpack it, put the files into the proper directories (keep in mind that packages are usually scattered throughout the file system), create a system user to run the daemon, set the according permissions. But that's tedious, error prone (which will lead to a high influx of tickets like "Bah! Does not run! Fix it!") and hardly practical if the software is going to be installed on many systems.

So we need a package to lower the hurd of adoption. Providing a repository for those packages usually is no rocket science and makes installation and/or updates much easier. There is a difference between "Download->Distribute->Install/Update" and a single command/server like

$ awesomePm update coolApplication

Provide at least packages for RedHat and Debian based systems. Personally, I'd go for CentOS (which would make your package compatible to almost any RHEL derivate) and basic Debian. The latter should make it trivial to provide a package for Ubuntu, too. Since I don't use Debian or derivates any more, I am not too sure wether they are really compatible, there have been startup issues when I last built a .deb.

Provide proper documentation. Document what is installed, where, and why. Provide links to the according documentation. manpage references to dependencies are sufficient. This way, you enable even the most inexperienced admin to configure your package.

Use the most defensive, sane defaults.

Special note with regard to golang: Most package build tools strip the binaries contained in the package by default. Go does not support that, so take care here.

Be complete

There is nothing more annoying than an incomplete package.

Use syslog if at all possible and adhere to it's conventions. This way, your logs will be put into places where the system admin expects them and they are taken care of if outdated automatically,preventing your application to cause disks running full. If the system admin wants special treatment of your application's logs, he will configure it this way.

Do not rotate the logs via your application. It is the users choice on what to do with them (which might be relevant for their SLAs). Even if you make the way logs are rotated configurable, the admin has to learn how to configure it – which introduces unnecessary redundancy.

If you have to write to a log file, adhere to the target system's logging policy and provide a log rotate configuration file. You don't want you application be the cause of downtime just because the machine ran out of disk space, do you?

Use system tools instead of reinventing the wheel. If your application needs to do some maintenance, do not bother to use a scheduler inside your application. Write a specialized tool for the maintenance (monolithic applications are so '00s) and utilize cron. Specifically add according files to one of the /etc/cron* directories.

Provide proper init scripts! This way, the admin can use well known tools like systemctl to mange startup and shutdown of your application. It is pretty annoying when you have to su to a user or use sudo -u to call a shell script on startup. Even when this script is called @onboot, the deviation from standards is annoying. Just because a startup method works it does not mean that it should be used.

Bonus points for adding an SE-Linux profile!

It should go without saying, but I have seen misconfigured packages way to often, so: Test your packages! Start with a minimal install of the target OS, install your package and make sure it runs as expected. Check every configuration you provide.

If you plan to have a package put into the official Debian repositories, you should plan some time: the reason why Debian is as stable is that the requirements on packages are pretty tight, and even when you fulfill all requirements, you have to go all the way from testing via unstable to stable.

Be precise

Do not use existing users just because it is convenient. If you create a web application, do not reuse the "apache" or "www" user. Create a user dedicated to your package and add this user to the according groups.

Adhere to the principle of least necessary permissions. There is hardly a reason to have a binary world executable, much less world writable (which would be an extreme security hole). What you often see here on SO if an application does not run is the suggestion to set the permissions to [0]777, which allows every user to make modifications whatsoever on the file. Actually, there is almost no reason to make a binary writable to any user: root, who does the updates anyway, can always write anything. So the permissions should be 0550 for binaries. The principle also applies to data directories and such. Invest some time and effort here. You don't want your app to be the vector for an successful attack, do you? Even potential security risks tend to fire back on you and your reputation. What I tend to do is to set all data files to 0600 for files that need to be written by the system user of the application, 0400 for read-only files and 0500 for binaries. Then, I do a granular analysis what the group permissions should be. For example: A group might change the individual templates for a web application – but most likely not the directory structure of the resource directory subtree.

If you put effort into security, you will increase trust. Be aware that packages are often checked for their security impact prior to the decision wether they are adopted.

Adhere to the FHS(!!)! And even then: just because you can do anything under /opt/yourapplication, it is not always a good idea to do so. Rather install into /usr and /var respectively (assuming your application is not necessary during boot time).

If you have dependencies, define them. Do not merely assume that a package is present.

If you have a dependency for a local SMTP server, do not declare a dependency on postfix. Maybe the admin prefers sendmail (for whatever reason that may be). So define a dependency on mail-transport-agent (Debian) or mta (RH, iirc) instead.

Conclusion

This is what I expect of a good software – integrating nicely with existing software and to make it easy to install, maintain, configure and run it without having to learn redundant configuration. If I see a SELinux profile for a package, that really gives the vendor a bonus – unless the profile is extremely sloppy, it shows that the vendor takes security very serious.