Design a pluggable interface to place replicas of blocks in HDFS
Details
Description
The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models.
Attachments
Attachments
- BlockPlacementPluggable.txt
- 16 kB
- Dhruba Borthakur
- BlockPlacementPluggable2.txt
- 61 kB
- Dhruba Borthakur
- BlockPlacementPluggable3.txt
- 70 kB
- Dhruba Borthakur
- BlockPlacementPluggable4.txt
- 79 kB
- Dhruba Borthakur
- BlockPlacementPluggable4.txt
- 76 kB
- Dhruba Borthakur
- BlockPlacementPluggable5.txt
- 79 kB
- Dhruba Borthakur
- BlockPlacementPluggable6.txt
- 77 kB
- Dhruba Borthakur
- BlockPlacementPluggable7.txt
- 77 kB
- Dhruba Borthakur
- blockplacementpolicy2-branch-1.patch
- 85 kB
- Sumadhur Reddy Bolli
- blockplacementpolicy2-branch-1-win.patch
- 84 kB
- Sumadhur Reddy Bolli
- blockplacementpolicy3-branch-1.patch
- 85 kB
- Sumadhur Reddy Bolli
- blockplacementpolicy3-branch-1-win.patch
- 84 kB
- Sumadhur Reddy Bolli
- blockplacementpolicy-branch-1.patch
- 28 kB
- Sumadhur Reddy Bolli
- blockplacementpolicy-branch-1-win.patch
- 27 kB
- Sumadhur Reddy Bolli
- rat094.txt
- 0.4 kB
- Michael Stack
Issue Links
- is related to
-
Improvement - An improvement or enhancement to an existing feature or task. HDFS-347 DFS read performance suboptimal when client co-located on nodes with data
- Major - Major loss of function.
- Closed
- relates to
-
Bug - A problem which impairs or prevents the functions of the product. HDFS-629 Remove ReplicationTargetChooser.java along with fixing import warnings.
- Major - Major loss of function.
- Closed
-
Improvement - An improvement or enhancement to an existing feature or task. HDFS-1094 Intelligent block placement policy to decrease probability of block loss
- Major - Major loss of function.
- Open
-
Improvement - An improvement or enhancement to an existing feature or task. HDFS-1451 Extend the pluggable block placement policy interface to handle/allow file-level intelligence
- Major - Major loss of function.
- Open
-
Improvement - An improvement or enhancement to an existing feature or task. HDFS-3564 Design enhancements to the pluggable blockplacementpolicy
- Major - Major loss of function.
- Resolved
-
Improvement - An improvement or enhancement to an existing feature or task. HDFS-3649 Port HDFS-385 to branch-1-win
- Major - Major loss of function.
- Resolved