Tabalyst ProductsAvailable

Realistic data you are free to share.

Tabalyst Generate creates synthetic CSV datasets for tests, demos and teaching: believable values, related files and, when you ask for them, defects that are injected on purpose and logged cell by cell. It runs on your machine, in the same package as Tabalyst Report.

One file, with errorsComputer offers with defects
tabalyst generate --example computer --rows 20000 -o pcs.csv --anomaly-profile dirty_realistic
Created: pcs.csv
Created: pcs.changes.csv
20,000 rows in 4.6s (4,391 rows/s, 3 processes), 5,672 cells changed: generating rows 4.0s, injecting errors 0.6s
Related filesInsurance, three branches and a total
tabalyst generate --example insurance --rows 3000 -d insurance-output/
Created: insurance-output/insurance-main.csv
Created: insurance-output/insurance-ontario.csv
Created: insurance-output/insurance-quebec.csv
Created: insurance-output/insurance-combined.csv
Done in 0.9s

Real output of tabalyst 0.8.0, paths shortened. Times were measured on one machine with four processors.

What it does

Real data is hard to share, and a demo needs data that looks real: believable names, addresses, amounts and dates, and the kind of problems that real exports have. A synthetic dataset gives you that without exposing anyone.

  • 01 / THEMED

    Datasets for a business domain

    Packaged examples for CRM contacts, insurance customers and computer offers, each with the structure and the rules of its domain.

  • 02 / REPRODUCIBLE

    The same seed, the same files

    A package fixes its seed and its reference date; --seed and --as-of-date override them. Generation is reproducible with the same package, options and environment.

  • 03 / CONTROLLED PROBLEMS

    Problems you can count

    An anomaly profile writes an altered CSV beside the clean one, a log of every changed cell and a JSON summary of what was requested, applied and skipped, so a profile of the data can be checked against what was injected. Name several profiles to get one variant for each level of errors, from the same rows.

Three packaged examples

  • CRM: one contacts CSV with companies, opportunities and lifecycle stages. Read the CRM contacts page.
  • Insurance: three branch files and one combined file. --rows sets the combined total, shared across the branches. Read the insurance customers page.
  • Computer: offers of computers, written in English, in French or in both from the same rows, with three column profiles (40, 70 or 111 columns), two markets, generation profiles such as gaming or enterprise, and measurements as plain numbers or with units. Read the computer offers page, which details the 40, 70 and 111 columns.
  • Your own package: a directory with a dataset.yaml, a catalog.yaml and the schemas and tables they reference. --validate-only checks it without writing anything. The package format is experimental during beta.

Existing files are never overwritten without --force.

Create a dataset with known problems

An anomaly profile such as dirty_realistic or dirty_stress keeps the clean CSV and adds the altered one, a *.changes.csv log and a summary.

tabalyst generate --example insurance --rows 3000 --anomaly-profile dirty_realistic -d insurance-dirty/

For Computer, -o writes one CSV that holds the errors, with its .changes.csv log beside it. With -d, name several profiles to get one variant for each level of errors from a single generation of the rows, so every variant describes the same offers:

tabalyst generate --example computer --rows 30000 -d out/ --anomaly-profile dirty_realistic,dirty_stress

Computer rows are generated by several processes from 5,000 rows (--workers sets the number), and the command shows its progress, then ends with a timing report. The same seed gives the same files whatever the number of processes. On a machine with four processors, 1,000,000 offers with a profile of errors took about three minutes.

The synthetic insurance file of the Tabalyst Report demo was made this way: 100,000 customers and 34 columns with the dirty_stress profile, which gives each kind of defect a fixed number of rows. Tabalyst Report then analyzes it as it is, with its real gaps and duplicates.

PRODUCTAvailable

Tabalyst Report

Open the demo on the synthetic insurance file.

Next step

Where to go next

Install Tabalyst 0.8.0 or later with pip install --upgrade tabalyst (0.7.0 is enough for the examples and packages, 0.7.3 adds the single Computer file with errors, several profiles and the progress), then read the walkthrough or the command reference. To analyze a file, use Tabalyst Report.

PRODUCTAvailable

Tabalyst Report

Interactive HTML reports and JSON profiles from your files.

TOOLSAvailable

Tools

Tabalyst Inspect, Tabalyst Scan and Tabalyst Sample.