actk.bin package

Submodules

actk.bin.all module

This script will run all tasks in a prefect Flow.

When you add steps to you step workflow be sure to add them to the step list and configure their IO in the run function.

class actk.bin.all.All[source]

Bases: object

Set all of your available steps here. This is only used for data logging operations, not computation purposes.

checkout()[source]

Checkout all steps.

clean()[source]

Clean all steps.

pull()[source]

Pull all steps.

push()[source]

Push all steps.

run(dataset: str, include_raw: bool = False, batch_size: Optional[int] = None, distributed: bool = False, n_workers: int = 10, worker_cpu: int = 8, worker_mem: str = '120GB', overwrite: bool = False, debug: bool = False, **kwargs)[source]

Run a flow with your steps.

Parameters

  • dataset (str) – The dataset to use for the pipeline.

  • include_raw (bool) – A boolean option to determine if the raw data should be included in the Quilt package. Default: False (Do not include the raw data)

  • batch_size (Optional[int]) – An optional batch size to provide to each step for processing their items. Default: None (auto batch size depending on CPU / threads available)

  • distributed (bool) – A boolean option to determine if the jobs should be distributed to a SLURM cluster when possible. Default: False (Do not distribute)

  • n_workers (int) – Number of workers to request (when distributed is enabled). Default: 10

  • worker_cpu (int) – Number of cores to provide per worker (when distributed is enabled). Default: 8

  • worker_mem (str) – Amount of memory to provide per worker (when distributed is enabled). Default: 120GB

  • overwrite (bool) – If this pipeline has already partially or completely run, should it overwrite the previous files or not. Default: False (Do not overwrite or regenerate files)

  • debug (bool) – A debug flag for the developer to use to manipulate how much data runs, how it is processed, etc. Additionally, if debug is True, any mapped operation will run on threads instead of processes. Default: False (Do not debug)

actk.bin.cli module

This script will convert all the steps into CLI callables.

You should not edit this script.

actk.bin.cli.cli()[source]

Module contents

Bin scripts package for actk.