Data Contracts and Validation
Write schemas, data contracts, and validation rules in the language of your choice
There is no one-size-fits-all validation language. CsvPath Framework strives to support all common edge data formats at the ingress point.
Today, the following validation languages and formats are supported:
JSON Schema: JSON, including lists of JSON documents - start here.
ODCS (Open Data Contract Standard): tabular data - start here.
XSD (XML Schema Definition): XML - start here.
CsvPath Validation Language (CVL): tabular data, JSONL/NDJSON, live objects - start here.
Pandas DataFrames: tabular data in data frames (by CVL, JSONPath)
XPath and JSONPath: supporting Schematron-like rules-based validation over JSONL and XML
Parquet: output of Parquet files based on
parquet()schema function structures
EDI, fixed-width, natural language entity recognition, and SodaCL remain on the backlog pending a groundswell of requests.
Further SQL support is naturally higher on the list. However, at this time CsvPath Framework does not plan to widen it's role beyond acting as an edge data governance data firewall defending the ingress point. Adding data quality monitoring across the core database data estate is an interesting possibility, but other tools do that well. ETL/ELT functionality is firmly off the menu. The Framework is laser-focused on filling today's biggest blindspot in protecting and observing the data estate: data engineering at the edge.
Last updated