pydoit - Task Runner - Python CLI Tool
Keywords: python, pydoit, doit, task runner, build tool, task management, python workflow automation, task management python, open source, make, data pipeline, python tool, python automation, ad-hoc command, python rake, python grunt, DAG, incremental build, incremental task, python tool

# doit comes from the idea of bringing the power of build-tools to execute any kind of task

    pip install doit

People often compare doit to tools like make, grunt, rake, scons, snakemake.

They appreciate doit strong features, flexibility, simplicity of authoring and ease of use.

## A game developer uses `doit` to

- Automate all project related tasks (code generation, cross-compilation, resource generation)
- Simplify cumbersome command line calls
- Optimize processing time by skipping tasks already done

## A bioinformatics developer uses `doit` to

- Create a reproduceable computational pipeline
- Manage a complex workflow (set of depending tasks)
- Optimize processing time by skipping tasks already done

## Nikola, a static site generator, uses `doit` to

- Provide the command line interface
- Speed up by parallel task execution
- Optimize processing time by skipping tasks already done

View all use-cases

## Sample Code

Define functions returning python dict with task's meta-data.

Snippet from tutorial.

    def task_imports():
        """find imports from a python module"""
        for name, module in PKG_MODULES.by_name.items():
            yield {
                'name': name,
                'file_dep': [module.path],
                'actions': [(get_imports, (PKG_MODULES, module.path))],
            }

    def task_dot():
        """generate a graphviz's dot graph from module imports"""
        return {
            'targets': ['requests.dot'],
            'actions': [module_to_dot],
            'getargs': {'imports': ('imports', 'modules')},
            'clean': True,
        }

    def task_draw():
        """generate image from a dot file"""
        return {
            'file_dep': ['requests.dot'],
            'targets': ['requests.png'],
            'actions': ['dot -Tpng %(dependencies)s -o %(targets)s'],
            'clean': True,
        }

    $ doit list
    dot       generate a graphviz's dot graph from module imports
    draw      generate image from a dot file
    imports   find imports from a python module
    $ doit
    .  imports:requests.models
    .  imports:requests.__init__
    .  imports:requests.help
    (...)
    .  dot
    .  draw

## Task Runner

doit allows you to easily define ad-hoc tasks, helping you to organize all your project related tasks in an unified easy-to-use & discoverable way.

### python powered

doit uses plain python to define tasks.

Task's meta-data are better described in a declarative way, but often you want to create this meta-data programmatically.

### easy authoring

NO API: Tasks are described by a python dict (can also be easily customized)

Tasks can execute external process (shell commands) or python code

### debugger & self documented

Since plain python is used to define your tasks the python debugger pdb is available

doit command allows you to list and obtain help/documentation for tasks

## Build tool & Pipelines

Simple task runners simply do not scale-up. doit as other build-tools can be much more efficient at repeateadly running tasks.

### cache task results
aka incremental-builds

doit creates a DAG and ensures that only required tasks will be executed and in the correct order.

doit checks if the task is up-to-date and skips its execution if the task would produce the same result of a previous execution.

### up-to-date check

dependencies can be dynamically calculated by other tasks

the up-to-date check to cache task results is not restricted to looking for file modification on dependencies. Nor requires target files.

### Pipelines

Traditional build-tools were created mainly to deal with compile/link process of source code. doit was designed to solve a broader range of workflows.

results from a task can be used by another task without resorting to the creation of intermediate files

## Advanced Features

### Extensible

Custom output
Command line output can be completely customized through reporters

Plugins
allow you to create/modify sub-commands, storage backend, task loader, and output reporter

Framework
API is exposed so you can create new applications/tools leveraging doit functionality

### Batteries included

parallel execution
built-in support for parallel (threaded or multi-process) task execution

watch/auto execution
built-in support watching for file changes and automatically re-execute tasks based on file changes by external process [linux/mac only]

tab-completion
built-in support tab-completion for commands/task (supports bash and zsh)

DAG Visualisation
create task's dependency-graph image using graphviz

IPython
IPython integration provide `%doit` magic function that loads tasks defined directly in IPython's global namespace

strace
Integration with strace helps you understand effects of third-part commands

## Testimonials

- Congratulations! Your tool follows the KISS principle very closely. I always wondered why build tools had to be that complicated.

  ##### Elena

  source

- Let me start by saying I'm really lovin doit, at first the interface seemed verbose but quickly changed my mind when I started using it and realized the flexibility. Many thanks for the great software!

  ##### Michael Gliwinski

  source

- I love all the traditional unix power tools, like cron, `make`, perl, ..., I also like new comprehensive configuration management tools like `CFEngine` and `Puppet`. But I find doit to be so versatile and so productive.

  ##### Charlie Guo

  source

- I needed a sort of `make` tool to glue things together and after trying out all kinds, doit ... has actually turned out to be beautiful. Its easy to add and manage tasks, even complex ones-- gluing things together with decorators and 'library' functions I've written to do certain similar things.

  ##### Matthew

  source

- I went back and forth on different Pythonic build tools for awhile. `Scons` is pretty great if you're doing 'standard' sorts of builds, but I found it a little heavy for my tastes and really hard to customize to my tool flow (in `FPGA` land, there are all kinds of nonstandard vendor tools that all need to play together). I've been using doit more and more over the past few months, and I'm continually impressed by the tool (aside from the goofy name). It works amazingly well for automating tricky/exotic build processes. Check it out!

  ##### SkOink

  source

- Some time ago, I grew frustrated with `Make` and `Ant` and started porting my build files to every build tool I found (`SCons`, `Waf`, etc.). Each time, as soon as I stepped out of already available rules, I ran into some difficult to overcome stumbling blocks. Then I discovered this little gem of simplicity: doit. It's Python-based. It doesn't try to be smart, it does not try to be cool, it just works. If you are looking for a flexible little build tool for different languages and tasks, give it a chance. (...)

  ##### lelele

  source

## Status

`doit` is under active development. Version 0.36.0 released on 2022-04.

doit runs on Python 3.8 through 3.10 (including PyPy). For python 2 support please use doit version `0.29`.

This blog post explains how everything started in 2008.

doit core features are quite stable. If there is no recent development, it does NOT mean the project is not being maintained... The project has 100% unit-test code coverage.

Development is done based on real world use cases. It is well designed and has a small code base, so adding new features is not hard. Contributions are welcome.

## Project Details

LICENSE
  This is an open-source project (MIT license) written in python.

DOWNLOAD
  from PyPi

SUPPORT
  See support page

DEVELOP
  Project management (bug tracker, feature requests and source code ) on github

WEBSITE
  This web site is hosted on http://pages.github.com
  Powered by Universal template and Sphinx.