Setup notifications to Slack
Send yourself notifications about CsvPath runs via Slack webhooks
Last updated
Send yourself notifications about CsvPath runs via Slack webhooks
Last updated
CsvPath can send alerts to Slack as run events happen. It looks basically like this:
To recap, there are five event types. Each event goes to listeners. CsvPath has its own listener that creates new or updated manifests. The types are:
Named-file staging: a file event is fired at the time you add a file to the file inputs directory using the file manager.
Named-paths loading: a paths event is fired when you load a named-paths group into the inputs directory using the paths manager.
Run start: when a run starts a run notification indicates at the highest-level the inputs and start time. It tells you that an activity is happening in the archive.
Results available: results events are fired when at the beginning and end of a named-paths group run. It is summary-level information about the group's progress.
Result available: a result event is fired when an instance of a csvpath in a named-paths group starts or completes. This is the most detailed event.
All of these events are received by all listeners configured in config/config.ini to listen for events of a type. For example, the Marquez OpenLineage listeners are configured like this:
There is a Slack listener that can receive any of those same four events. (Neither Marquez or Slack support the very general run event). To configure Slack you simple add any or all of these lines to your config.ini file:
You will also need to add or update a [slack] section to say what webhook you want the events to go to. That looks like this:
Obviously you need to add your webhook URL.
And finally you need to tell CsvPath that you want the slack group of event listeners to receive events. Do that by adding slack to the groups key in the [listeners] section of config.ini.
As the comments in your config file say, you can also configure the result Slack events on a csvpath-by-csvpath basis. Only the result events can be configured by a csvpath. That is because a result event is tied to a single csvpath; whereas, the other events apply to named-paths groups of csvpaths or to input files.
Within your csvpath you need an external comment. An external comment is one that is above or below the csvpath, not within the match part of the csvpath. In the external comment you may use one or both of the custom metadata fields that the Slack integration knows to look for:
on-valid-slack:
on-invalid-slack:
The value of the field is a webhook URL. When you add these fields remember to only give the URL starting with the subdomain and domain. So rather than:
You just use:
The reason to use the shorter form is because a full URL has a protocol signifier that includes a colon. Since CsvPath metadata fields are defined as names followed by a colon, https://... looks to CsvPath like a metadata field named https.
If neither of these metadata fields is present, your event will go to the default URL in config/config.ini. As you would guess, if your csvpath is valid — per the valid field in the metadata collected during the run — the on-valid-slack webhook is called. If not valid the on-invalid-slack webhook gets the call.
And in case you don't remember, you set the valid value using the fail() function.
A csvpath is considered valid by default. Under certain circumstances it may have indications that something is wrong (e.g. the expected files not generated and stopped early indicators) which generally you see in the metadata and/or as errors in errors.json. But unless you explicitly say a file is invalid, it is valid.
That said, bear in mind that built-in validations, when tripped, can mark a file as invalid. For example, if you try to add("five", none()) you will raise an error and depending on your , your file may be marked invalid without you having to do anything. In that case, if you had on-invalid-slack configured with a webhook, you would get an alert.
