Validating DataFrames
DataFrames are flexible and CsvPath Validation Language is onboard with that
As you know just from landing on this page, CsvPath Framework handles DataFrames validation. You have three complementary validation options for DataFrames:
ODCS validation
CsvPath Validation Langage (CVL)
JSONPath
Each has different use cases and strengths. (The other three validation tools, XSD, JSON Schema, and XPath, are not as likely to be useful with DataFrames).
Why Use CsvPath Framework With DataFrames?
There are many good options for validating your data frames, including CsvPath Framework. Why might you want to use the Framework?
The option to combine validation approaches — use the Framework's native validation, JSONPath, or ODCS
Modularity — layer rules and validation languages in separate files in a named-paths group that can be applied as a unit
The integration with OpenLineage and OpenTelemetry — DataFrames passing through pipelines especially benefit from the visibility and traceability
Consistent validation strategies across types and locations — your DataFrames can be handled exactly the same as your API endpoints and file feeds
Tons of metadata capture, process transparency, and workflow options, if you need it
Simplicity and flexibility with significantly less risk of drift and spaghetti code across the whole data estate
That's a lot. But let's start by just seeing a simple example
Show Me How
We'll use the same basic example as on the JSONPath validation page. It helps illustrate how CVL and JSONPath work together.
This is pretty basic stuff. If you strip out the comments and fluff, it comes down to more or less just this:
Seven lines + a bit of CVL
How Does the CVL Work?
Let's look at that CVL statement more closely. This is a good, small example of CsvPath Validation Langage's flexibility and debugging tools.
When we ingest our JSONL into a DataFrame we capture the animals header's JSON value, but Pandas drops the name animals. That's fine for us, we can just use #0 to address that header by index, instead of name. But why not put that header back so we can make a more explicit reference to the header? Easy to do with rename().
Next, on line 4, we can use row_table() to get a nice printout showing what each line looks like. That can be useful in a report, but here we're just using it to get comfort with how the DataFrame translated our data. Our first line reports:
That immediately tells us two things.
We got the data and headers we would expect
JSONPath could be helpful to extract information from
#animals!
On line 6 we grab the information that determines if we validate or not. The @fish variable takes the value found by the jsonpath() function. On line 7 we print a message if we find a fish, but just as a debugging check.
The real validation happens on line 8. Here we accept any line where a fish existed. The validation strategy we're taking in this example is that the data is valid if there is a fish in the dataset. Otherwise, if there are too many or not enough fish the data is invalid. But we could also do something like:
This would check that path.is_valid is True. Or you could taken one of many other validation strategies. There's a lot of flexibility to validate however works best for your use case.
How Simple Can The CVL Be?
Ultimately our CVL really boils down to just:
The ~ validation-mode:print, raise ~ is good practice, just to be explicit. But since the default is that built-in validations are always printed and exceptions raised noisily, it isn't needed, if those are the options you want. (In practice, outside of development and debugging no-raise is probably a good idea).
And just for fun, we can boil the ocean down to:
Or, if we only care if there is a sea in the domain header on at least one line we could do:
And that's how easy it is to validate a DataFrame. Give it a shot.
Last updated