Tabalyst.csv
CSV
Profile an export column by column: types, gaps, ambiguous dates, duplicate rows.
Tabalyst for data engineers
Know what a source really holds before you move it.
Exports arrive with mixed types, ambiguous dates, nested records and several sheets. Tabalyst reads the whole file on your machine and shows how it is built, so the mapping, the cleaning and the checks of your ETL, migration or integration start from facts.
tabalyst inspect sales.xlsx
Inspect: sales.xlsx-inspect.json
Selection: $.Orders (much larger than $.Regions)
tabalyst inspect orders.json
Inspect: orders.json-inspect.json
Selection: $.customers[] (the only eligible collection)
Real output of tabalyst 0.6.0, paths shortened.
Your angle
What goes wrong between a source and a pipeline
- 01 / HETEROGENEITY
Mixed types and spelling variants
A date column can mix ISO and day-first values. Tabalyst flags ambiguous dates without guessing, and groups the values that are written in several ways but compare as equal once normalized.
- 02 / STRUCTURE
Files that are not tables
A JSON file holds several arrays, a workbook several sheets and a title above the header. Tabalyst Inspect finds the collection or the table, and stops with the candidates when the choice is not clear.
- 03 / EXPORTS
Separators, encodings, bad lines
Say the delimiter and the encoding once. A malformed CSV record stops the analysis unless you choose the tolerant policy. In JSONL, invalid lines are excluded, counted and listed.
Workflow
Look first, then decide
The commands are the same for every format. Tabalyst never modifies a source, and it writes one report and one profile per file: files are not merged.
- 1 · Install
pip install tabalyst# Also updates it: Tabalyst changes often
- 2 · Choose the table
tabalyst inspect sales.xlsx# Lists the sheets and named tables, proposes one, writes sales.xlsx-inspect.json
- 3 · Report a source
tabalyst report sales.xlsx# The HTML report and the JSON profile of the table that Inspect selected
- 4 · Describe every field
tabalyst scan data.csv -d scans/# One scan document per source: types, frequencies and detectors of every field
import tabalyst # Where does the data sit? Writes the Inspect file and returns itinspection = tabalyst.inspect("sales.xlsx") # One report per source, whatever the formatbatch = tabalyst.generate_reports(["data/*.json", "*.xlsx"], output_dir="reports") # The full description of every field, without a reportscans = tabalyst.generate_scans(["data/*.json"], output_dir="scans")
Large files. Reports, scans and samples read a file as a stream, with memory bounded by configurable limits. Files of 16 MiB or more are analyzed by several worker processes. An Excel sheet is read into memory instead, and a sheet above 1 GiB of XML is refused. Tabalyst publishes no benchmark: test your own files.
Reusable outputs. The scan document holds the complete result. tabalyst report --scan data.scan.json builds a report from it without reading the source again, and Tabalyst refuses a scan whose source changed since it was written.
Proof
Four sources, four real reports
Each report below was written by Tabalyst Report from a synthetic source of a different kind. Open it to see what the analysis keeps from a CSV export, a workbook, a nested JSON export and a log with bad lines.
- CSVinsurance-customers.csv 3,000 rows · 34 columns
- EXCELsales.xlsx 120 rows · 9 columns · 13 missing cells
- JSONorders.json 60 records · 17 fields · 48 missing cells
- JSONLweb-events.jsonl 300 records · 14 fields · 3 excluded lines
In the JSONL log, three lines are not valid records. They are left out, counted and named in the profile, and the other 300 records are analyzed:
{
"code": "excluded_records",
"severity": "warning",
"message": "Records excluded by the tolerant error policy, not analyzed",
"count": 3,
"column_ids": [],
"row_numbers": [121, 201, 251]
}One of the issues of the profile, with the same fields and values.
Scope, not hype
What to know before you rely on it
- One Excel table or JSON collection per report by default. Blocks of a sheet separated by blank rows are not split into tables, and relations between collections are not analyzed.
- A syntax error anywhere in a
.jsonfile stops the analysis: Tabalyst does not repair a damaged document. - Wildcards such as
*.csvmatch the files of one folder, not its subfolders. - Tabalyst profiles a source. It does not clean it, convert it or check your business rules, and its profile format is still experimental.
The known limitations ↗ list the rest.
Next step
Where to go next
Start with the format of your source, then read how Inspect and Scan make their choices.
.xlsx, .xlsm
Excel
Report one sheet or named table of a workbook, header found even under a title.
.json
JSON
Report a collection of records, with nested fields named by their path.
.jsonl, .ndjson
JSONL
Report one record per line, with invalid lines excluded, counted and listed.
DOCS
Tabalyst Inspect
How a JSON, JSONL or Excel source is read.
DOCS
Tabalyst Scan
The complete description of every field, in one document.
Your next source
Read the source before you write the pipeline.
Install Tabalyst, point it at an export and open the report.