Pipe your script to slack

Editor's Notes

1. Today’s my talk is ‘Pipe your script to Slack’.
2. (15 - 30 sec) Before talking about what it means, let me introduce myself and my company ‘Course Hero’ briefly. I’m Chuck Kato. Engineering Manager at Course Hero. Course Hero is a mid size EdTech start-up company. We are providing cloud sources materials and online tutoring platform. We started using Slack from Aug. 2014 in the engineering team and then it became our core communication tool quickly.
3. (30 sec - 1 min) Today, I’m introducing our open source custom slack integration ‘slacktee’. ‘slacktee’ is a bash script which works like the 'tee' command. As you may know, the 'tee' command reads standard input and writes it to standard output and one or more files. Instead of writing to files, 'slacktee' posts the input to a Slack channel. So, basically, you can pipe any scripts/commands to Slack without any additional development! [Show very quick demo] > echo “Hello everyone!” | slacktee.sh > ls -l | slacktee.sh Easy, isn’t it?
4. (30 sec) At Course Hero, we execute a bunch of scripts and commands every day to accomplish backend tasks. Since Slack is our core communication tool, we wanted to see the result of tasks on Slack. However, even though implementing a custom Slack integration is easy, most of the backend tasks were small or one time, so we couldn't justify the development cost. For these two reasons, we created 'slacktee'. If you have the same dilemma, ‘slacktee’ should work for you too.
5. (5 - 10 sec) Actually, 'slacktee' changed the way we monitor backend tasks. I'd like to show you 3 real life use cases today.
6. (30 sec) First use case is an error notification. We are maintaining multiple replication slave databases, but sometimes they are backed up due to long-running queries from our analytics team. To detect the backup, we wrote a tiny script to monitor the replication status. Here is the example of the output.
7. (30 sec) To send a backup notification to engineering team, we used to use an email. However, as you can see, it’s difficult to notice and often we overlooked it. Also, personally, I don’t check inbox frequently, so I cannot notice it in a timely manner. But, ‘slacktee’ solved this problem.
8. (30 sec) With ‘slacktee’, we can send the notification on Slack. Here is the command and example of the notification. To make the notification prominent, we added a few options, but, basically, we just piped ‘slacktee’ instead of ‘mail’ command. As you can see, it stands out and it’s difficult to overlook. Also, the notification is sent to mobile too, if we set up custom notification.
9. (30 sec) Let’s move to 2nd use case. One day, we executed a script which fixed missing data in the database. But, the script was slow and took almost a day to finish. So, we executed the script in the background using the ‘screen’ command, but we needed to check its progress periodically and monitor for errors.
10. (30 sec) Before ‘slacktee’, we had to login to the server and attached the screen each time when we checked the progress. To access server, we had to establish VPN session with 2-way authentication. This monitoring process was really tedious and not mobile friendly. Also, it was impossible to notice errors immediately, because there was no way to see the errors until we access to the server. Actually, ‘slacktee’ solved these problems.
11. (30 sec) With ‘no buffering’ option, we can send the input to Slack line by line. So, we can see the progress on Slack in real time without logging into the server. Since we are redirecting the standard error to the standard input here, we can also notice errors immediately. This was amazing improvement for us.
12. (30 sec) This is the last use case I’m explaining today. To fix permission issues, we recently executed a ‘find’ command on our NFS server and listed the files which have the wrong owner. Since many mllions of files are stored in our NFS, the output of the command was huge, and we needed to download it to our local PC to check it.
13. (30 sec) Before ‘slacktee’, we needed to ‘scp’ the file from the server. But, frankly speaking, using ‘scp’ is not a pleasant experience. We cannot use path auto completion, so we have to remember the file path of the target file correctly. Also, if the file has been created by different user, we have to change the permission on the server side first. ‘scp’ is a necessary tool, but I’m not a big fan of it. And, after we downloaded it, we needed to share it with other members. So we had to email or uploaded it to other place such as Slack. Yeah, Slack.
14. (30 sec) Actually, these problems could be solved with ‘slacktee’. By using slacktee’s file upload mode, we can upload the result to Slack directly. Now, we don’t have to use ‘scp’ to download the result file from the server. Just click the download link on Slack. Also, if you’d like to share it with your colleagues, simply share the link or post itself with them! BTW, slacktee is using Incoming Webhook to post the message to Slack, but it doesn’t support file upload. So, in the file upload mode’, slacktee uses the user’s token for uploading.
15. (30 sec) Slack integration and ‘slacktee’ have really helped us. After creating ‘slacktee’, more information is available on Slack and it makes our daily job much easier. Also, non-technical people can get insight from the results. Sometimes, they point out issues, sometimes, they get an inspiration about new ideas from the information. This was a positive surprise for engineers. I believe these things will happen in your team too. So, if you have a script, pipe it to Slack now.
16. (5 sec) Thank you. Any questions?