Using Mon.sh to Monitor Recurring Jobs
Related Guides
Overview
This guide outlines some suggestions for how to set up mon.sh to monitor scripts that run on a recurring basis.
Before getting started be sure to check out the quickstart guide for how to get setup. At the end of the setup you should at least have the mon.sh client installed, and the client configured or an API key handy.
A recurring job for the sake of this guide is just a script that you run on a recurring basis. A unique invocation of a script is referred to as a "Job" in mon.sh, and is given a globally unique invocation id displayed as the "job id". This is not to be confused with the logical "job" you wish your scripts to do. Your job can be a script that you or someone on your team runs manually each day, or a script that is run automatically via a schedule (such as a CRON configured script).
About workspaces in mon.sh
Because notifications are set on a specific workspace, it's recommended that you should use a unique workspace for each job. A workspace is a unique string (no spaces) that you specify during the mon invocation e.g. "... | mon -ws MyWorkspace". Workspaces do not need to be created before creating them, if they don't exist a new one will be created. Event configurations and workspace settings can be configured via the web on the workspaces page. If you don't specify a workspace the "default" workspace is used. Note that because of the way notifications work, if you setup your recurring job notifications on a workspace you run other scripts on, any of those script executions against a workspace would look like the job you want to monitor! So keep your scripts separate by workspace.
Client usage
Usage of mon is as simple as piping your script output to the installed mon command. Refer to the quickstart guide for full setup instructions but installation is as simple as pip3 install monsh. With the installed client you can redirect your output to mon.sh by piping the output like so your-command... |& mon -ws MyWorkspace. The |& redirects both stdout and stderr. Note that by default the mon client treats carriage returns as new lines, this is important for syncing many progress bars.
If your use case is not supported by this format we'd love to hear about it at support@mon.sh. We're looking to expand how we support getting data into mon.sh to make it as easy as possible to use.
Notes on CRON
Piping commands in CRON does not work directly, you'll need to batch the commands with bash -c like * * * * * bash -c 'ls -t /mytest | tail -n +2 | xargs rm --'. You can refer to https://stackoverflow.com/questions/56311070/bash-command-with-pipe-not-working-in-crontab.
Setup error reporting
If you want to be notified when a script has a failure, you'll need to setup regex matches against common error patterns.
Note that at the moment mon.sh does not monitor script exit status, though wrapped command syntax in the near future will support a completion state of failed based on exit status for notifications.
Setting up a regex pattern is as simple as going to the workspace configuration and configuring a "On Regex Match" / "Line Match" event.
Note that line matches may be throttled, allowing a maximum of one match / notification per minute. Note that currently regular expressions are case sensitive unless otherwise noted. An example regular expression you could use would be (?i)exception|fault|error, which matches all of those strings in a case insensitive manner. When the line is matched you'll receive an email (or other notification as selected) like below containing the matched line and a link to the dashboard to see the full logs.
Setting up a recurring job notice
At the moment mon.sh supports simple detection when a job misses a given schedule. This is supported by specifying an interval. Everytime the job misses that interval you will be sent a notification email like below.
Setting up a job interval is as simple as going to the workspace config page and adding a interval.
The schedule parameter must be set to a valid ISO8601 duration string. If the string is not valid you will not be notified. The time interval is based on the start time of each job.
If you run the script manually each day, be sure to set the interval wider than the expected interval. For example, if you intend to run the script during working hours 9am-5pm you may wish to set an interval of 32 hours to ensure that if the script is run at 9am one day and 5pm the next this would be allowed. If the script is run automatically every day you can set the schedule of the missed job event notification to match.
Optionally setup success monitoring
You can also receive notifications when a job has succeeded. I would recommend you use a different email alias for missed job executions and failures (e.g. your_email+mon@gmail.com, which works on many email providers) as it can be easy to ignore or bin these automated emails together if you're constantly getting them.
Setting up success is as simple as adding a new rule "On State Change" / "Completed" to the workspace. When a job runs successfully you'll receive an email notification containing the first few and last few log lines as well as a link to the full logs online. You can also use "On State Change" / "Running" to get notified when a job starts instead.
