Dask: Parallel Computation with Blocked algorithms and Task Scheduling
Matthew Rocklin · Proceedings of the Python in Science Conferences · 2015
Dask enables parallel and out-of-core computation.We couple blocked algorithms with dynamic and memory aware task scheduling to achieve a parallel and out-of-core NumPy clone.We show how this extends the effective scale of modern hardware to larger datasets and discuss how these ideas can be more broadly applied to other parallel collections.
Cited by 801 works. doi:10.25080/majora-7b98e3ed-013 Open-access PDF
Authors
References
- The NumPy Array: A Structure for Efficient Numerical Computation
- Dryad
- Data Structures for Statistical Computing in Python
- Python for Scientific Computing
- Spark: cluster computing with working sets
- Theano: A CPU and GPU Math Compiler in Python
- A class of parallel tiled linear algebra algorithms for multicore architectures
- Numerical linear algebra on emerging architectures: The PLASMA and MAGMA projects
- DAGuE: A generic distributed DAG engine for High Performance Computing
- Elemental