Paths To Production
There are many paths. What's the simplest thing that could possibly work?
Last updated
There are many paths. What's the simplest thing that could possibly work?
Last updated
As you saw in and , there are two ways to use the CsvPath library.
Single csvpath pointing to a single file, simple results
Multi-csvpath
Both can work in a production setting. The latter is more flexible and offers more capabilities with little additional complexity. We'll focus on the multi-csvpath option.
The path to production starts with just getting the simplest possible thing working—a proof of concept. We can quickly sketch that out.
As our starting point, here is a high-level view of everything you need to know about the library. (Setting aside how to write csvpaths using the built-in functions, for now).
Here's what we think the simplest-possible thing might be, at least for some companies. (Apologies to non-AWS users). Could you do something simpler? Probably. Is reality more complicated. For sure!
Your Lambda, where CsvPath runs, is pretty simple. (Remembering this is just a sketch to give you ideas). In mostly accurate pseudo-code, it might look like:
Adding a simple email giving validation results using SES is an easy add-on. The Web is littered with SES email examples.
And that's pretty much all of it. Obviously, some assembly required. As they say, an exercise for the reader.
Our goal is to create an automated CSV validation capability. We're not actually picking the right technologies that work for your specific situation, obviously! Writing the minimal implementation code for this is an exercise for you and . A POC like what we are sketching should be pretty quick.
Imagine your CSV files are arriving by SFTP. You need a landing zone for your CSV files and csvpaths. Let's say you use AWS Transfer Family fronting S3. When a file arrives it triggers a Lambda that runs your csvpath validation. This scenario is .
And we can repurpose the multi-csvpath file from to stand in for your inbound file validation rules.
Likewise, pushing metadata or lines from the Results object into a database would be a straightforward database insert—at least if we're sticking with the .
