dpark changelog

Changelog History

• v0.5.0 Changes

  July 27, 2018

  API change

  • ✂ Remove module-level api like dpark.textFile.
  • 👌 Support Streaming shuffle and Disk shuffle (Experimental, compatible).

  🛠 Fixes

  • 🐛 Bug when parsing mfs chunk info.

  👌 Improvement

  • 👍 Better broadcast impl using shared memory for tasks on the same slave to reduce memory cost.
  • 👍 Better offer-matching logic for MesosScheduler which remember bad slaves.
  • 💅 Refactor: style and layout.

  🆕 New Feature

  • Multi segment dump to save memory.
  • Gather statics for stage.
  • 👌 Support run tests/test_rdd on mesos.
  • ➕ Add colorful progress bar for dpark.
  • 👌 Support mesos role.
  • 👌 Support multi named mesos master in conf.
  • Loghub for admin.

• v0.4.2 Changes

  March 19, 2018

  • 👌 Support Python3 & PyPy
  • 👌 Support MooseFS 3.x & refactor on file-system interface

• v0.4.1 Changes

  March 09, 2017

  • ✨ Enhancement for the containerizer in DPark
  • 👉 Use broadcast when parallelize big dataset
  • 🛠 Fix missing line bug for bzip2 files
  • ➕ Add TopByKey in RDD
  • Other minor bugs

• v0.4.0 Changes

  December 06, 2016

  • 🛠 Bugfix: deserialize error of old-style class.
  • 🔨 Refactor beansdb RDD
  • 💻 Web UI support for dpark
  • 👉 Use pymesos >= 0.2.0
  • Eager serialize values of ParallelCollection

• v0.3.5

  October 27, 2016

• v0.3.3

  September 20, 2016

• v0.3.2

  June 07, 2016

• v0.3.1

  May 25, 2016

• v0.2.9

  March 15, 2016

• v0.2.8

  March 04, 2016