Skip to content
DataMaker Docs

Python script tips

A short list of things that surprise people the first time they hit them.

Per-field Python is sandboxed and capped at 2 seconds

Per-field Python generators run inside a sandbox with a 2-second per-row timeout. They are not full scenarios.

  • No pip install; standard library only.
  • No subprocess. Read reference data via dm.workspace_file(...).
  • No long-running loops. If you need state across rows, use dm.counter().

If you’re reaching for any of those, lift the work into a scenario.

row only contains previously generated fields

In a per-field Python generator, row[name] only sees fields that come before the current one in the template. Drag your Python field below any siblings it reads. For a value that depends on later fields, use a derived field, which runs in a second pass.

Use the rng argument, not random.random()

Per-field generators get a seeded rng, so use it. Module-level random.random() isn’t seeded by DataMaker, so your output won’t be reproducible when the template has a seed set.

def value(rng, row, dm):
    return rng.choice(["a", "b", "c"])    # ✓ seeded
    # NOT: return random.choice([...])     # ✗ unseeded

The rest are scenario tips. Inside a scenario you have the full datamaker SDK plus plain Python.

Generate large sets in batches

dm.generate_from_template_id(id, quantity=1_000_000) returns a list of a million dicts, hundreds of MB in RAM. Generate in batches instead: produce a few thousand rows, export them, let them go out of scope, and repeat.

BATCH = 5_000
for _ in range(total // BATCH):
    rows = dm.generate_from_template_id("tmpl_customer", quantity=BATCH)
    dm.export_to_database(export_data)   # export, then let `rows` be collected

print() lands in the live log. It uses Python’s default buffering. Wrap in print(..., flush=True) if you want the log to update line-by-line during a long step. There’s no dm.log; for structured logs, print JSON yourself.

Idempotency is your problem

Scenarios don’t checkpoint: a retried run starts from the top, so a script that POSTs to a non-idempotent endpoint can double-create. Guard against it: look resources up before creating them, and key external writes on something stable so re-runs are detectable.

existing = {t["name"] for t in dm.get_templates()}
if "Customer" not in existing:
    dm.create_template(template_data, project_id, team_id)

Read run parameters from the environment

There’s no dm.params. Configuration arrives as environment variables. Read and cast them with the standard library:

import os
size  = int(os.environ.get("SIZE", "100"))
debug = os.environ.get("DEBUG", "false").lower() == "true"

Don’t print secrets

DataMaker doesn’t print your auth headers, but a raw print(response.headers) will land a token in the log. Print a status code or a redacted summary instead, never the header.

# requirements: is parsed only at the top

# requirements: arrow~=1.3
import arrow

The # requirements: comment is read once at script load. Don’t put it in the middle of the file or inside a function. Already preinstalled: requests, httpx, pandas, numpy, pydantic (plus the standard library and datamaker-py).