Page MenuHomec4science
Files F103062788

index.xml
No OneTemporary
Actions

File Metadata

Created
Wed, Feb 26, 23:36

index.xml
View Options

<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright 2002-2004 The Apache Software Foundation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<document xmlns="http://maven.apache.org/XDOC/2.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/XDOC/2.0 http://maven.apache.org/xsd/xdoc-2.0.xsd">
<head>
<title>DistCp</title>
</head>
<body>
<section name="Overview">
<p>
DistCp (distributed copy) is a tool used for large inter/intra-cluster
copying. It uses Map/Reduce to effect its distribution, error
handling and recovery, and reporting. It expands a list of files and
directories into input to map tasks, each of which will copy a partition
of the files specified in the source list.
</p>
<p>
The erstwhile implementation of DistCp has its share of quirks and
drawbacks, both in its usage, as well as its extensibility and
performance. The purpose of the DistCp refactor was to fix these shortcomings,
enabling it to be used and extended programmatically. New paradigms have
been introduced to improve runtime and setup performance, while simultaneously
retaining the legacy behaviour as default.
</p>
<p>
This document aims to describe the design of the new DistCp, its spanking
new features, their optimal use, and any deviance from the legacy
implementation.
</p>
</section>
</body>
</document>

Event Timeline

c4science · Help