Skip to main content

Dynamic Mapping & Collect

These APIs provide the means for a simple kind of dynamic orchestration — where the work to be orchestrated is determined not at job definition time but at runtime, dependent on data that’s observed as part of job execution.

class dagster.DynamicOut

Variant of Out for an output that will dynamically alter the graph at runtime.

When using in a composition function such as @graph, dynamic outputs must be used with either

  • map - clone downstream ops for each separate DynamicOut
  • collect - gather across all DynamicOut in to a list

Uses the same constructor as Out

@op(
    config_schema={
        "path": Field(str, default_value=file_relative_path(__file__, "sample"))
    },
    out=DynamicOut(str),
)
def files_in_directory(context):
    path = context.op_config["path"]
    dirname, _, filenames = next(os.walk(path))
    for file in filenames:
        yield DynamicOutput(os.path.join(dirname, file), mapping_key=_clean(file))

@job
def process_directory():
    files = files_in_directory()

    # use map to invoke an op on each dynamic output
    file_results = files.map(process_file)

    # use collect to gather the results in to a list
    summarize_directory(file_results.collect())
class dagster.DynamicOutput

Variant of Output used to support dynamic mapping & collect. Each DynamicOutput produced by an op represents one item in a set that can be processed individually with map or gathered with collect.

Each DynamicOutput must have a unique mapping_key to distinguish it with it’s set.

Parameters:

  • value (Any) – The value returned by the compute function.
  • mapping_key (str) – The key that uniquely identifies this dynamic value relative to its peers. This key will be used to identify the downstream ops when mapped, ie mapped_op[example_mapping_key]
  • output_name (Optional[str]) – Name of the corresponding DynamicOut defined on the op. (default: “result”)
  • metadata (Optional[Dict[str, Union[str, float, int, MetadataValue]]]) – Arbitrary metadata about the failure. Keys are displayed string labels, and values are one of the following: string, float, int, JSON-serializable dict, JSON-serializable list, and one of the data classes returned by a MetadataValue static method.
property mapping_key

The mapping_key that was set for this DynamicOutput at instantiation.

property output_name

Name of the DynamicOut defined on the op that this DynamicOut is associated with.

property value

The value that is returned by the compute function for this DynamicOut.