Using Mon.sh to Monitor Script Runs
Related Guides
Overview
This guide outlines some suggestions for how to set up mon.sh to monitor scripts that you run in a one off manner. Mon.sh has been designed to help monitor long running scripts so that you don't have to periodically check on them, turning dealing with scripts from a "pull" oriented way of check in on them to a "push" way.
Before getting started be sure to check out the quickstart guide for how to get setup. At the end of the setup you should at least have the mon.sh client installed, and the client configured or an API key handy.
About the terminology
Jobs
A unique invocation of a script is referred to as a "Job" in mon.sh, and is given a globally unique invocation id displayed as the "job id". This is not to be confused with the logical "job" you wish your scripts to do. You can view all jobs on the jobs page or recent jobs on the dashboard. The latest job that was run is available to you via the latest job link in the menu or via this link.
Workspaces
A workspace is a way to organize jobs. A job always runs in the context of a workspace. If you don't specify the workspace it runs under the "default" workspace. You can specify the workpace when running the mon command with the "-ws" switch e.g. "... | mon -ws MyWorkspace".
Events
Activities that happen on a workspace are called events. You can configure events to have notifications. All events also show up in the event stream. We are constantly expanding the supported events but today you can do things like match regular expressions, get notified when a script misses a schedule, or get notified when scripts start or complete in a workspace.
Client usage
Usage of mon is as simple as piping your script output to the installed mon command. Refer to the quickstart guide for full setup instructions but installation is as simple as pip3 install monsh. With the installed client you can redirect your output to mon.sh by piping the output like so your-command... |& mon -ws MyWorkspace. The |& redirects both stdout and stderr. On windows based systems we recommend using git bash or similar posix style shells to execute your script. | will work in Powershell, however the |& does not have the same meaning, and without stderr streaming you may miss important logs or progress bars.
If your use case is not supported by this format we'd love to hear about it at support@mon.sh. We're looking to expand how we support getting data into mon.sh to make it as easy as possible to use.
Monitoring running scripts
Running scripts can be monitored from the link that is spit out at the beginning of the script execution, or by navigating to the dashboard and finding the running script. All log output should be synced to the logs on the job details page. Additionally, progress is synced from any progress bars found in the logs. Estimated completion time of running scripts is extrapolated from the current synced progress. Note that currently this may not work correctly if your script spits out multiple progress bars.
Setup error reporting
If you want to be notified when a script has a failure, you'll need to setup regex matches against common error patterns.
Setting up a regex pattern is as simple as going to the workspace configuration and configuring a "On Regex Match" / "Line Match" event.
Note that line matches may be throttled, allowing a maximum of one match / notification per minute. Note that currently regular expressions are case sensitive unless otherwise noted. An example regular expression you could use would be (?i)exception|fault|error, which matches all of those strings in a case insensitive manner. When the line is matched you'll receive an email (or other notification as selected) like below containing the matched line and a link to the dashboard to see the full logs.
Setup completion monitoring
You can also receive notifications when your script has finished running. Setting up success is as simple as adding a new rule "On State Change" / "Completed" to the workspace. When a job runs successfully you'll receive an email notification containing the first few and last few log lines as well as a link to the full logs online.
