Tags: NVIDIA-NeMo/DataDesigner
Tags
fix: unify Parquet schemas when reading (#858) * fix: normalize structured JSON numbers (#857) Signed-off-by: Steve Han <sthan@nvidia.com> * fix: handle boolean JSON subschemas (#857) Signed-off-by: Steve Han <sthan@nvidia.com> * fix: unify parquet schemas when reading (#857) Signed-off-by: Steve Han <sthan@nvidia.com> --------- Signed-off-by: Steve Han <sthan@nvidia.com>
fix: bump vulnerable dependencies (#855) Raise the minimum and locked versions for aiohttp, MCP, setuptools, and Soup Sieve to releases recommended by the security scan. Keep the documented Bleach exception because nbconvert still requires it and no patched release is available. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>
fix: bump vulnerable dependencies (#855) Raise the minimum and locked versions for aiohttp, MCP, setuptools, and Soup Sieve to releases recommended by the security scan. Keep the documented Bleach exception because nbconvert still requires it and no patched release is available. Signed-off-by: Nabin Mulepati <nmulepati@nvidia.com>
chore: regenerate only changed colab notebooks (#838) * chore: regenerate only changed colab notebooks Pass changed notebook sources through the existing generator filter to avoid unrelated cell ID churn. Remove stale generated notebooks when sources are deleted or renamed. Closes #413 Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: detect new colab notebooks Signed-off-by: Andre Manoel <amanoel@nvidia.com> * fix: preserve notebook filename boundaries Signed-off-by: Andre Manoel <amanoel@nvidia.com> --------- Signed-off-by: Andre Manoel <amanoel@nvidia.com> Co-authored-by: Nabin Mulepati <nmulepati@nvidia.com>
fix: drop invalid expression rows (#757) * fix: drop invalid expression rows Handle per-row expression render and cast failures by dropping only affected rows, preserving row identity in sync and async full-column paths, and failing when no valid rows remain. Fixes #749 * fix: fail async all-dropped expressions * fix: extend expression row drops to root from_scratch tasks Address PR #757 review feedback from @andreatgretel: - Root expression columns are scheduled as ``from_scratch`` rather than ``batch``, so the prior ``_task_supports_row_drops`` check (which gated on ``task_type == "batch"``) silently dropped the whole row group on a user template error instead of raising. Accept both task types and route expression ``from_scratch`` tasks through a shared row-drop helper that preserves indices and honors the fatal-when-all-dropped contract. - ``DataFrame.to_dict(orient='records')`` returns ``[]`` for any 0-column DataFrame regardless of row count, so a root expression dispatched against a fresh row group buffer would otherwise skip rendering entirely. Synthesize empty per-row dicts in ``ExpressionColumnGenerator`` so the loop fires once per row. - Add async coverage for the root-expression fatal path and for the skip-then-drop interaction (async equivalent of the existing sync skip-aware regression).
fix: drop invalid expression rows (#757) * fix: drop invalid expression rows Handle per-row expression render and cast failures by dropping only affected rows, preserving row identity in sync and async full-column paths, and failing when no valid rows remain. Fixes #749 * fix: fail async all-dropped expressions * fix: extend expression row drops to root from_scratch tasks Address PR #757 review feedback from @andreatgretel: - Root expression columns are scheduled as ``from_scratch`` rather than ``batch``, so the prior ``_task_supports_row_drops`` check (which gated on ``task_type == "batch"``) silently dropped the whole row group on a user template error instead of raising. Accept both task types and route expression ``from_scratch`` tasks through a shared row-drop helper that preserves indices and honors the fatal-when-all-dropped contract. - ``DataFrame.to_dict(orient='records')`` returns ``[]`` for any 0-column DataFrame regardless of row count, so a root expression dispatched against a fresh row group buffer would otherwise skip rendering entirely. Synthesize empty per-row dicts in ``ExpressionColumnGenerator`` so the loop fires once per row. - Add async coverage for the root-expression fatal path and for the skip-then-drop interaction (async equivalent of the existing sync skip-aware regression).
fix: preserve ordered seed dataset position on resume (#710) * fix ordered seed resume offsets Preserve planned row-group start offsets during resume so ordered seed datasets continue from the next seed row instead of replaying already-consumed rows. Fixes #709 * simplify async resume offset planning test * address review comments on seed resume offset fix - Simplify _run_batch context-var setup so current_row_group is set consistently in fresh and resumed sync runs (matches the (x/X) log prefix the async path already emits) and add a docstring spelling out which ContextVars the function owns. - Document RowGroupResumePlan and build_row_group_resume_plan, and make the plan dataclass frozen+slots since it is a one-shot value. - Comment the modulo cycling logic in _index_range_at_offset. - Add a scheduler test verifying fresh async runs auto-derive the per-row-group offsets from row-group sizes (no caller-supplied offsets) so ordered generators stay parallel-safe across row groups. - Add a wraparound regression test that resumes past a full seed cycle, exercising the relative_offset == 0 branch the original #709 regression test missed. * address review suggestions on seed resume offset fix - update current_row_group ContextVar comment to reflect that both the async scheduler and the sync engine's _run_batch set it - move pre_batch_snapshot capture (and ran_pre_batch flag) inside the try/finally in _run_batch so a failure between ContextVar.set and the snapshot call still resets the tokens - add a direct unit test for the relative_offset == 0 wraparound branch in _index_range_at_offset to lock in the fresh-cycle restart behavior * add ORDERED + PartitionBlock resume regression test Companion to the existing IndexRange resume test. Locks in correct behavior when the seed selection comes from PartitionBlock — its to_index_range produces a contiguous range today, but nothing else asserts that contract. The test crosses a cycle boundary inside the partition (4 records over a 2-row partition) so it exercises both the offset-into-partition branch and the relative_offset == 0 wraparound branch end-to-end. * fix async resume progress accounting
fix: preserve ordered seed dataset position on resume (#710) * fix ordered seed resume offsets Preserve planned row-group start offsets during resume so ordered seed datasets continue from the next seed row instead of replaying already-consumed rows. Fixes #709 * simplify async resume offset planning test * address review comments on seed resume offset fix - Simplify _run_batch context-var setup so current_row_group is set consistently in fresh and resumed sync runs (matches the (x/X) log prefix the async path already emits) and add a docstring spelling out which ContextVars the function owns. - Document RowGroupResumePlan and build_row_group_resume_plan, and make the plan dataclass frozen+slots since it is a one-shot value. - Comment the modulo cycling logic in _index_range_at_offset. - Add a scheduler test verifying fresh async runs auto-derive the per-row-group offsets from row-group sizes (no caller-supplied offsets) so ordered generators stay parallel-safe across row groups. - Add a wraparound regression test that resumes past a full seed cycle, exercising the relative_offset == 0 branch the original #709 regression test missed. * address review suggestions on seed resume offset fix - update current_row_group ContextVar comment to reflect that both the async scheduler and the sync engine's _run_batch set it - move pre_batch_snapshot capture (and ran_pre_batch flag) inside the try/finally in _run_batch so a failure between ContextVar.set and the snapshot call still resets the tokens - add a direct unit test for the relative_offset == 0 wraparound branch in _index_range_at_offset to lock in the fresh-cycle restart behavior * add ORDERED + PartitionBlock resume regression test Companion to the existing IndexRange resume test. Locks in correct behavior when the seed selection comes from PartitionBlock — its to_index_range produces a contiguous range today, but nothing else asserts that contract. The test crosses a cycle boundary inside the partition (4 records over a 2-row partition) so it exercises both the offset-into-partition branch and the relative_offset == 0 wraparound branch end-to-end. * fix async resume progress accounting
PreviousNext
