For the complete documentation index, see llms.txt. This page is also available as Markdown.

Release Notes

High-level descriptions of CsvPath Framework and FlightPath Data releases

CsvPath Framework 0.0.630 - 14 October 2026

  • OpenLineage v2 event producer

The v2 of CsvPath's OpenLineage support is a configurable option. It is the default for new CsvPath Framework project config.ini files going forward. Compared to v1:

◦ Namespacing and names configuration to satisfy DataOps scale or consumer requirements

◦ Greatly increased metadata coverage in standard and custom facets

◦ Tighter and more up to date spec conformance

◦ A framework for simpler versions, in case we ever go for a v3

◦ More interoperability testing, including DataHub and DataDog, and of course Marquez

  • ODCS validation support

ODCS data contracts join XSD, JSON Schema, and CsvPath Validation Language as the fourth 1st-class validation standard. Supported ODCS versions: v3.0.x, v3.1.0, v3.2.0.

To validate tabular data against an ODCS contract just load the contract in the same way you would load a csvpath statement.

◦ The ODCS is translated and stored in your group.csvpath file

◦ The rules in the contract are what validates your data.

◦ Complete context and business info is available in the contract file and every run's meta.json

◦ An ODCS file is copied to a run's results dir and can be transferred anywhere, like any result file

There are of course differences between CSV/JSONL/XLSX files or live data objects vs. SQL databases. For example, relationships don't work the same. But overall, the fit is remarkably good and very effective.

  • XPath and JSONPath against whole documents

XPath and JSONPath can now be written against the whole XML or JSON document, not just individual headers. This allows a Schematron-like validation by path rules approach.

  • New function: collect_original()

collect_original() copies the registered source document verbatim to the run results. The copied file is named original.<ext> where <ext> is .xml, .json, .csv, etc.

CsvPath Framework 0.0.622 - 18 September 2026

JSON Schema and XSD are now first-class validation languages within CsvPath Framework. Both ad hoc and programmatic runs are supported, using CsvPath and CsvPaths instances respectively.

JSON Schema and XSD are peers to CsvPath Validation Language. Each of the three has primacy in its own space: JSON documents, XML documents, and tabular data in CSV, JSONL, and Excel. When run through the full data preboarding lifecycle using a CsvPaths instance, all the languages generate the same expected files.

  • Ad hoc dynamic Validation of JSON

The same method of validation supported for ad hoc validation of data frames using CsvPath Validation Language is now expanded to cover JSONL and JSON. JSON can also be validated using JSON Schema as the validation language.

Ad-hoc validation runs are output-lite programmatic validations that are quick to create, but are not tied into the full Framework lifecycle. Bolting data contract validation into a workflow job, API, Jypter notebook is quick and effective, and leverages the same Framework tool chain as more formal uses of the Framework do.

  • Full Collect-Store-Validate-Publish lifecycle for dynamic API-driven and message JSON

This feature ties the new validation languages and the expanded dynamic object validation capabilities into the larger governance framework using two new run methods:

  • collect_dynamic — peer to collect_paths and collect_by_line

  • fast_forward_dynamic — peer to fast_forward_paths and fast_forward_by_line

In each case, the use pattern is to assign live Python objects to the run when it is triggered. Developers treat data as JSON, JSONL, or microbatched sequences of JSON objects.

Registration of live data into named-files is optional. If data is registered and staged it happens automatically; otherwise, the data remains uncaptured. Data contracts for dynamic data are created and applied as named-paths groups using CsvPath Validation Language and/or JSON Schema and/or XSD in exactly the same way as for data files. Run results flow into run directories within the archive, as usual. Templates for named-file registration and runs are supported.

Using fast_forward_dynamic allows you to run without capturing primary results data, as well as optionally not registering the raw data into the Framework's staging area. This, along with the Framework's other many config combinations and validation strategies, allows you to tailor rules and contracts to the scale, velocity, and compliance requirements of real-time systems.

Summer 2026 FlightPath - v1.1.89

Transfers and Multiple SFTP servers

  • Transfers can now be defined on a named-paths group (the old csvpath-by-csvpath method continues to be available) using a form

    • Csvpath statements can have any number of transfers triggered by csvpath end-state: valid, invalid, errors, all

    • All files can be transferred, including named printouts, jinja() output, and parquet files

    • Source data (from named-files) can be transferred, even though it is not a run result

    • Transfers can be sent to any location in S3, SFTP, Azure Blob, or Google Cloud Storage

  • Any number of SFTP servers can be added to named-files and named-paths definitions for import registrations and results transfers, respectively, using a form

FlightPath Server v2

  • HTTP methods, error codes, and endpoints follow REST conventions

  • Asynchronous end points for file registration and named-paths group runs

  • Improvements to admin endpoints and API version support settable in server config

Improvements

  • Viewing and editing XLSX worksheets is supported with a right-click to select. Edited worksheets must be saved as individual CSV files.

  • The Find Data Dialog displays registration or run date-times with the paths of the files found

  • Generate multiple printout files by printer name using print-mode:separate

  • When registering files in a directory, files can be selected by a regex

  • Ability to pass UUIDs into file registrations to enable retrieving the exact-match named-file reference when registering asynchronously

  • Pass arbitrary metadata into run to be stored with run results

  • tmp qualifier to limit the data stored in vars.json

New Functions, Tokens, and Fields

  • New functions:

    • runtime() — accesses runtime variables already available to print(), error(), and jinja()

    • metadata() — accesses user-defined csvpath metadata fields

    • env() — accesses environment variables available to the project, env.json or OS

    • uuid() — schema type for line() schemas

    • select() — selects a column of data from a SQL database

  • Date-time tokens in registration and run templates:

    • :day

    • :month and :month_name

    • :year

    • :hour_24 and :hour

    • :minute

    • :second

  • Additional fields available in runtime metadata, meta.json, print(), etc.:

    • day, day_of_week, month, month_of_year, hour, hour_of_day, minute, second

    • run_dir and the run's reference

    • named-paths and named-file

Other Changes:

  • Bug fixes and minor enhancements

  • More than 1000 new unit and integration tests

  • Legacy SFTPPlus integration removed in favor of no-code integration using webhooks, transfers, and FlightPath Server. See https://www.flightpathdata.com/data/videos.html for an SFTPPlus how-to.

CsvPath Framework 0.0.614 - 17 June 2026

  • Generate multiple printout files by printer name using print-mode:separate

  • Transfers improvements encapsulated in the named-paths definition.json descriptor (csvpath statement transfer-mode: directives continue to exist as a simpler option):

    • Multiple transfers per csvpath statement

    • Transfers within csvpaths grouped by run state: all runs, valid, invalid, and runs with errors

    • Ability to transfer original source file, as well as any result file

    • Transfers can be sent to any backend

    • Multiple transfers-only SFTP servers can be configured, allowing you to use SFTP destinations that are not configured as a CsvPath Framework project backend

    • All files in the results directory can be transferred, including the new arbitrarily named printouts files that can be created by print(), error(), and jinja() using print-mode:separate

  • Registration of all files in a directory can be filtered by a regex

  • SFTP improvements:

    • Multiple SFTP source locations can be configured on a named file. This allows registering from SFTP servers that are not acting as the project's backend.

    • Multiple SFTP destination locations can be configured on a named-paths group. This allows SFTP transfers to go to SFTP servers that are not acting as the project's backend.

  • Pass UUID into file registrations to enable retrieving the exact-match named-file reference when registering asynchronously

  • Pass arbitrary metadata into run. Metadata is a dict[str,str] and stored in: <run_dir>/_extra_data/extra.json . This capability allows run lookup by inspecting metadata, and also creates a space available for integration listeners' use.

  • New functions:

    • runtime() — provides access to the runtime variables already available to print(), error(), and jinja()

    • metadata() — provides access to the user defined metadata fields in a csvpath statement's leading comment, as well as modes configurations

    • env() — provides access to the environment variables available to the project; either OS env variables or the variables in the project's env.json.

    • uuid() — a schema type for line() schemas that also generates UUIDs to populate variables

    • select() — selects a column of data from a SQL database into a stack variable

  • New .tmp qualifier to limit the data stored permanently in vars.json at the end of a run

  • Date-time tokens in templates:

    • :day

    • :month

    • :month_name

    • :year

    • :hour_24

    • :hour

    • :minute

    • :second

  • Additional fields in runtime metadata (print(), error(), jinja() and captured to meta.json):

    • The datetime fields: day, day_of_week, month, month_of_year, hour, hour_of_day, minute, second

    • run_dir

    • The run's reference

    • named-paths

    • named-file

Spring 2026 FlightPath - v1.1.88

AI-Assisted Authoring

FlightPath’s sidebar AI assistant helps developers and BizOps team members with four core authoring tasks:

  • Generate validations from requirements: describe a data contract in plain language and get a working CsvPath script as a starting point

  • Explain validation scripts: Get a complete plain-English rule-by-rule walkthrough of what an existing script does

  • Create test data: Generate sample files that exercise your validation logic before real data arrives

  • Refactor validation scripts: Clean up, reorganize, or modernize existing CsvPath code without rewriting from scratch

Autonomous workflows

Three new operational capabilities reduce manual intervention in day-to-day data operations:

  • Arrival activations: Automatically trigger runs when data files arrive — no scheduling, no polling, no manual handoff

  • Async job control: Full visibility into running jobs: status, metrics, and results accessible at any point, with clean access to results when a run completes

  • No-code webhooks: Fire automatically based on run outcome criteria, keeping downstream consumers, monitoring tools, and notification systems informed without anyone in the middle. The original webhooks integration can still be used csvpath-statement-by-csvpath-statement, using both capabilities at the same time is fine.

Config variable interpolation

Variable swapping support is significantly expanded. Projects can use OS-level or project-specific environment variables interchangeably. Config values written in ALL CAPS are automatically resolved against the environment. Config values can contain {...} replacement tokens that pull from OS or project environment variables. Resolved values can themselves can point to further environment variables, keeping credentials separated from runtime configuration.

Registration path templates

Templates can now be set as a default directly in the named-file definition. Previously, the same template had to be passed explicitly on every registration.

Post-Run transfers

Transfers now support all files generated during a run, not just the primary output. Transfers work across all configured storage backends.

Parquet output generation

Parquet output is based on a data schema defined using the parquet() function. parquet() is like line(), FlightPath’s structural schema function. Multiple parquet() functions can be active simultaneously, each generating its own Parquet file. This makes it possible to decompose a flat CSV into separate relational entities in a single pass.

Other improvements

  • 10 new analytical and transformation functions

  • Grid View for run metadata

  • Default Markdown documentation Files generated in named-paths groups and named-files

CsvPath Framework 0.0.599 - 31 March 2026

  • Config variables are parsed for braces-bracketed substitution text. A OS or env.json variable in the form "This is {who} variable" will be rendered as "This is my variable" if there is another variable who with the value "my".

  • transfer-mode configuration also follows the braces-bracketed interpolation on the transfer-to path, enabling destinations like sftp://{SFTP_SERVER}:{SFTP_PORT}/my/destination.

  • The new sql_in() function is a SQL column analog to the existing in() function. sql_in() enables live or cached presence tests against a column from a table in Postgres, MySQL, or Sqlite.

CsvPath Framework 0.0.598 - 11 March 2026

Major changes

  • Support CsvPath-to-Parquet schemas and Parquet file output

  • Addition of a definition.json and README.md to named-files

  • Added a configuration to automatically trigger runs on file registration

  • Added a configuration to automatically use a default named-file path template for registrations

  • Update to both transfer-mode and the SFTP integration to allow sending any/all files generated in a run

  • Fix for a problem where delimiter and quotechar choices were not being applied to output files

  • Substantially improved validation error reporting, including:

    • Deduplication of similar errors

    • Improved descriptions

    • Providing more complete error information

    • Improving ID chains to better identify error sources within csvpaths

  • Fix for a problem where CsvPaths run methods would act on only the first file returned by a reference, rather than all results found by the reference.

  • Eight function improvements including the following new functions:

    • parquet() - a specialization of line() that uses its schema to create a parquet file of all matching data

    • sort() a stack var

    • slice() a stack var

    • percent_matching(header, value|stack)

    • matches() - the same capability to indicate matching lines as the onmatch qualifier, but in the form of a function suitable for use cases where a function test is needed

Winter 2025-2026 FlightPath - v1.1.87

  • Comprises: CsvPath 0.0.591, FlightPath Server 0.1.27, FlightPath Data 0.1.28

  • FlightPath changes:

    • Support for creating and editing JSONL

    • Support for editing CSV and JSON in grid view

    • New JSON text editing view

    • Edit JSONL as JSON

    • CSV and JSONL grid view editing, including copy-selected-to-new

    • Support for custom functions in FlightPath Data and FlightPath Server

    • Added use-delimiter and use-quotechar for FlightPath Data and FlightPath Server

    • Added a config dialog to compare and sync config between projects in Data and Server

    • Improved the env vars config dialog to act as a compare and sync, not just an upload, between projects in Data and Server

    • Changed to a fixed width font in the printouts tab for better table printouts

    • Many minor fixes and improvements

CsvPath Framework 0.0.591 - 25 Jan 2026

  • Added jsonpath() function using jsonpath-ng

  • Dozens of minor fixes and improvements

CsvPath Framework 0.0.581 - 11 Jan 2026

Major changes

When/do changes:

  • When/do operator right-hand sides now contribute to matching

  • When/do right-hand side now respects nocontrib

Custom functions changes:

  • External functions can be loaded from a file anywhere visible to the Python process

  • Functions loaded from different function.imports files within the same Python process do not collide

  • Multiple functions.imports will be loaded

  • FunctionFactory supports reloadeding imports files

Three new functions:

  • Added line_before() to compare current line header value to last line header value

  • Added remove() to drop one or more headers

  • Set headers manually using rename(@stack) or rename(“a”, “b”, “c”…)

Three qualifier changes:

  • Support for name qualifiers in error() that are visibile in error events. E.g. error.myname(“this is a msg”) outputs: 2025-12-08 00h42m40s-886915:temps.txt:1:Ⓜ️myname[1]: this is a msg

  • Added skipnone qualifier primarily for push(). With notnone on push() None is flagged as an error; whereas, in some cases we just want to not push Nones.

  • Support for the distinct qualifier in string(), integer(), decimal(), date(), datetime()

Other functions changes:

  • New markdown docs generated from/by the functions. E.g. https://github.com/csvpath/csvpath/blob/main/docs/func_gen/advance.md. These docs are barebones, but always 100% up to date. The original function docs .md pages will be maintained but should not be considered the most current and/or correct.

  • blank(name|header) and wildcard() support Any in actuals, including None

  • put() support added for dict, list, date, date time, and None

Added project and project_context args to the CsvPath __init__ in order to populate logger name fields.

CsvPath Framework 0.0.576 - 7 dec 2025

Major changes

  • JSONL now available with all backends

  • External custom functions are found before internals, allowing overriding

  • 13 new CsvPath Validation Language functions. See the CLI functions look up for usage.

    • headers_stack(): returns the names of the current headers

    • clear(var|var-name): removes a variable

    • index_of(stack-var|stack-name, value): returns the int position of the value

    • track_any(): tracks typed values, optionally collecting values in a stack or summing them

    • roll(date|datetime, number, unit-name): returns a date/datetime that is N-units in the future or past

    • day(date): returns the date of the month as an int

    • month(date): returns the month of the year as an int

    • year(date): returns the year as an int

    • format_date(date|datetime, str): formats a date/datetime as a string

    • fingerprint(Optional[Header], ...): returns the hash of the line or the indicated header values

    • format(str-value, str-format): formats a value according to a Python string template

    • interpolate(str-value, str-format): formats a value within a longer string using Python formatting

    • xpath(xml-str, xpath-str): extracts a value from an XML document using an XPath expression

Fixes and Updates

  • 15 function updates:

    • Number of times reset_headers() has been called and on what lines tracked in variables

    • Wildcard max length in string() using none()

    • track(): added 3rd arg add or collect, essentially the same function as for track_any()

    • push() and get() now able to take both stack vars and stack var names.

    • get(x) creates and returns a dict var if the variable named x is not found

    • Added an optional default arg to get()

    • concat() able to take a stack var of values to concatinate

    • Apply regex() to all headers using the headers() function like: regex( headers(), /…/ )

    • size() now takes the notnone qualifier and either a stack var or name of a stack var

    • eq() accepts None-valued vars, as well as none()

  • Other minor user-facing changes

    • Removed function name limitations disallowing numbers, '_', and '.' after an inital alpha char.

    • Made an alias to header_names_mismatch for clarity.

    • Added ability for transfer mode to append, rather than overwrite, using a target path ending in +

Last updated