Homec4science
Diffusion elastic-yarn (master) Browse

R3704/2.3.0/hadoop-2.3.0/share/hadoop/hadoop-project-dist/hadoop-common/Compatibility.html
6c2cef95801emaster

Last Change
Raw File

Compatibility.html
View Options

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<!-- Generated by Apache Maven Doxia at 2014-02-11 -->
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Apache Hadoop 2.3.0 - Apache Hadoop Compatibility</title>
<style type="text/css" media="all">
@import url("./css/maven-base.css");
@import url("./css/maven-theme.css");
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
<meta name="Date-Revision-yyyymmdd" content="20140211" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body class="composite">
<div id="banner">
<a href="http://hadoop.apache.org/" id="bannerLeft">
<img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" />
</a>
<a href="http://www.apache.org/" id="bannerRight">
<img src="http://www.apache.org/images/asf_logo_wide.png" alt="" />
</a>
<div class="clear">
<hr/>
</div>
</div>
<div id="breadcrumbs">
<div class="xleft">
<a href="http://www.apache.org/" class="externalLink">Apache</a>
&gt;
<a href="http://hadoop.apache.org/" class="externalLink">Hadoop</a>
&gt;
<a href="../">Apache Hadoop Project Dist POM</a>
&gt;
Apache Hadoop 2.3.0
</div>
<div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a>
|
<a href="https://svn.apache.org/repos/asf/hadoop/" class="externalLink">SVN</a>
|
<a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a>
&nbsp;| Last Published: 2014-02-11
&nbsp;| Version: 2.3.0
</div>
<div class="clear">
<hr/>
</div>
</div>
<div id="leftColumn">
<div id="navcolumn">
<h5>General</h5>
<ul>
<li class="none">
<a href="../../index.html">Overview</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/CommandsManual.html">Hadoop Commands Reference</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/FileSystemShell.html">File System Shell</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/Compatibility.html">Hadoop Compatibility</a>
</li>
</ul>
<h5>Common</h5>
<ul>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/Superusers.html">Superusers</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a>
</li>
</ul>
<h5>HDFS</h5>
<ul>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">HDFS User Guide</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html">High Availability With QJM</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html">High Availability With NFS</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">HDFS Snapshots</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">HDFS Architecture</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/Hftp.html">HFTP</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">C API libhdfs</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS REST API</a>
</li>
<li class="none">
<a href="../../hadoop-hdfs-httpfs/index.html">HttpFS Gateway</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">HDFS NFS Gateway</a>
</li>
</ul>
<h5>MapReduce</h5>
<ul>
<li class="none">
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibilty between Hadoop 1.x and Hadoop 2.x</a>
</li>
<li class="none">
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a>
</li>
<li class="none">
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a>
</li>
<li class="none">
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a>
</li>
</ul>
<h5>YARN</h5>
<ul>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/YARN.html">YARN Architecture</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">YARN Commands</a>
</li>
<li class="none">
<a href="../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a>
</li>
</ul>
<h5>YARN REST APIs</h5>
<ul>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/MapredAppMasterRest.html">MR Application Master</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/HistoryServerRest.html">History Server</a>
</li>
</ul>
<h5>Auth</h5>
<ul>
<li class="none">
<a href="../../hadoop-auth/index.html">Overview</a>
</li>
<li class="none">
<a href="../../hadoop-auth/Examples.html">Examples</a>
</li>
<li class="none">
<a href="../../hadoop-auth/Configuration.html">Configuration</a>
</li>
<li class="none">
<a href="../../hadoop-auth/BuildingIt.html">Building</a>
</li>
</ul>
<h5>Reference</h5>
<ul>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/releasenotes.html">Release Notes</a>
</li>
<li class="none">
<a href="../../api/index.html">API docs</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/CHANGES.txt">Common CHANGES.txt</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/CHANGES.txt">HDFS CHANGES.txt</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-mapreduce/CHANGES.txt">MapReduce CHANGES.txt</a>
</li>
</ul>
<h5>Configuration</h5>
<ul>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a>
</li>
<li class="none">
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a>
</li>
</ul>
<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
<img alt="Built by Maven" src="./images/logos/maven-feather.png"/>
</a>
</div>
</div>
<div id="bodyColumn">
<div id="contentBox">
<!-- Licensed under the Apache License, Version 2.0 (the "License"); --><!-- you may not use this file except in compliance with the License. --><!-- You may obtain a copy of the License at --><!-- --><!-- http://www.apache.org/licenses/LICENSE-2.0 --><!-- --><!-- Unless required by applicable law or agreed to in writing, software --><!-- distributed under the License is distributed on an "AS IS" BASIS, --><!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. --><!-- See the License for the specific language governing permissions and --><!-- limitations under the License. See accompanying LICENSE file. --><div class="section">
<h2>Apache Hadoop Compatibility<a name="Apache_Hadoop_Compatibility"></a></h2>
<ul>
<li><a href="#Apache_Hadoop_Compatibility">Apache Hadoop Compatibility</a>
<ul>
<li><a href="#Purpose">Purpose</a></li>
<li><a href="#Compatibility_types">Compatibility types</a>
<ul>
<li><a href="#Java_API">Java API</a>
<ul>
<li><a href="#Use_Cases">Use Cases</a></li>
<li><a href="#Policy">Policy</a></li></ul></li>
<li><a href="#Semantic_compatibility">Semantic compatibility</a>
<ul>
<li><a href="#Policy">Policy</a></li></ul></li>
<li><a href="#Wire_compatibility">Wire compatibility</a>
<ul>
<li><a href="#Use_Cases">Use Cases</a></li>
<li><a href="#Policy">Policy</a></li></ul></li>
<li><a href="#Java_Binary_compatibility_for_end-user_applications_i.e._Apache_Hadoop_ABI">Java Binary compatibility for end-user applications i.e. Apache Hadoop ABI</a>
<ul>
<li><a href="#Use_cases">Use cases</a></li>
<li><a href="#Policy">Policy</a></li></ul></li>
<li><a href="#REST_APIs">REST APIs</a>
<ul>
<li><a href="#Policy">Policy</a></li></ul></li>
<li><a href="#MetricsJMX">Metrics/JMX</a>
<ul>
<li><a href="#Policy">Policy </a></li></ul></li>
<li><a href="#File_formats__Metadata">File formats &amp; Metadata</a>
<ul>
<li><a href="#User-level_file_formats">User-level file formats</a>
<ul>
<li><a href="#Policy">Policy</a></li></ul></li>
<li><a href="#System-internal_file_formats">System-internal file formats</a>
<ul>
<li><a href="#MapReduce">MapReduce</a></li>
<li><a href="#Policy">Policy</a></li>
<li><a href="#HDFS_Metadata">HDFS Metadata</a></li>
<li><a href="#Policy">Policy</a></li></ul></li></ul></li>
<li><a href="#Command_Line_Interface_CLI">Command Line Interface (CLI)</a>
<ul>
<li><a href="#Policy">Policy </a></li></ul></li>
<li><a href="#Web_UI">Web UI</a>
<ul>
<li><a href="#Policy">Policy</a></li></ul></li>
<li><a href="#Hadoop_Configuration_Files">Hadoop Configuration Files</a>
<ul>
<li><a href="#Policy">Policy </a></li></ul></li>
<li><a href="#Directory_Structure">Directory Structure </a>
<ul>
<li><a href="#Policy">Policy</a></li></ul></li>
<li><a href="#Java_Classpath">Java Classpath</a>
<ul>
<li><a href="#Policy">Policy</a></li></ul></li>
<li><a href="#Environment_variables">Environment variables</a>
<ul>
<li><a href="#Policy">Policy</a></li></ul></li>
<li><a href="#Build_artifacts">Build artifacts</a>
<ul>
<li><a href="#Policy">Policy</a></li></ul></li>
<li><a href="#HardwareSoftware_Requirements">Hardware/Software Requirements</a>
<ul>
<li><a href="#Policies">Policies</a></li></ul></li></ul></li>
<li><a href="#References">References</a></li></ul></li></ul>
<div class="section">
<h3>Purpose<a name="Purpose"></a></h3>
<p>This document captures the compatibility goals of the Apache Hadoop project. The different types of compatibility between Hadoop releases that affects Hadoop developers, downstream projects, and end-users are enumerated. For each type of compatibility we:</p>
<ul>
<li>describe the impact on downstream projects or end-users</li>
<li>where applicable, call out the policy adopted by the Hadoop developers when incompatible changes are permitted.</li></ul></div>
<div class="section">
<h3>Compatibility types<a name="Compatibility_types"></a></h3>
<div class="section">
<h4>Java API<a name="Java_API"></a></h4>
<p>Hadoop interfaces and classes are annotated to describe the intended audience and stability in order to maintain compatibility with previous releases. See <a href="./InterfaceClassification.html">Hadoop Interface Classification</a> for details.</p>
<ul>
<li>InterfaceAudience: captures the intended audience, possible values are Public (for end users and external projects), LimitedPrivate (for other Hadoop components, and closely related projects like YARN, MapReduce, HBase etc.), and Private (for intra component use).</li>
<li>InterfaceStability: describes what types of interface changes are permitted. Possible values are Stable, Evolving, Unstable, and Deprecated.</li></ul>
<div class="section">
<h5>Use Cases<a name="Use_Cases"></a></h5>
<ul>
<li>Public-Stable API compatibility is required to ensure end-user programs and downstream projects continue to work without modification.</li>
<li>LimitedPrivate-Stable API compatibility is required to allow upgrade of individual components across minor releases.</li>
<li>Private-Stable API compatibility is required for rolling upgrades.</li></ul></div>
<div class="section">
<h5>Policy<a name="Policy"></a></h5>
<ul>
<li>Public-Stable APIs must be deprecated for at least one major release prior to their removal in a major release.</li>
<li>LimitedPrivate-Stable APIs can change across major releases, but not within a major release.</li>
<li>Private-Stable APIs can change across major releases, but not within a major release.</li>
<li>Note: APIs generated from the proto files need to be compatible for rolling-upgrades. See the section on wire-compatibility for more details. The compatibility policies for APIs and wire-communication need to go hand-in-hand to address this.</li></ul></div></div>
<div class="section">
<h4>Semantic compatibility<a name="Semantic_compatibility"></a></h4>
<p>Apache Hadoop strives to ensure that the behavior of APIs remains consistent over versions, though changes for correctness may result in changes in behavior. Tests and javadocs specify the API's behavior. The community is in the process of specifying some APIs more rigorously, and enhancing test suites to verify compliance with the specification, effectively creating a formal specification for the subset of behaviors that can be easily tested.</p>
<div class="section">
<h5>Policy<a name="Policy"></a></h5>
<p>The behavior of API may be changed to fix incorrect behavior, such a change to be accompanied by updating existing buggy tests or adding tests in cases there were none prior to the change.</p></div></div>
<div class="section">
<h4>Wire compatibility<a name="Wire_compatibility"></a></h4>
<p>Wire compatibility concerns data being transmitted over the wire between Hadoop processes. Hadoop uses Protocol Buffers for most RPC communication. Preserving compatibility requires prohibiting modification as described below. Non-RPC communication should be considered as well, for example using HTTP to transfer an HDFS image as part of snapshotting or transferring MapTask output. The potential communications can be categorized as follows:</p>
<ul>
<li>Client-Server: communication between Hadoop clients and servers (e.g., the HDFS client to NameNode protocol, or the YARN client to ResourceManager protocol).</li>
<li>Client-Server (Admin): It is worth distinguishing a subset of the Client-Server protocols used solely by administrative commands (e.g., the HAAdmin protocol) as these protocols only impact administrators who can tolerate changes that end users (which use general Client-Server protocols) can not.</li>
<li>Server-Server: communication between servers (e.g., the protocol between the DataNode and NameNode, or NodeManager and ResourceManager)</li></ul>
<div class="section">
<h5>Use Cases<a name="Use_Cases"></a></h5>
<ul>
<li>Client-Server compatibility is required to allow users to continue using the old clients even after upgrading the server (cluster) to a later version (or vice versa). For example, a Hadoop 2.1.0 client talking to a Hadoop 2.3.0 cluster.</li>
<li>Client-Server compatibility is also required to allow upgrading individual components without upgrading others. For example, upgrade HDFS from version 2.1.0 to 2.2.0 without upgrading MapReduce.</li>
<li>Server-Server compatibility is required to allow mixed versions within an active cluster so the cluster may be upgraded without downtime in a rolling fashion.</li></ul></div>
<div class="section">
<h5>Policy<a name="Policy"></a></h5>
<ul>
<li>Both Client-Server and Server-Server compatibility is preserved within a major release. (Different policies for different categories are yet to be considered.)</li>
<li>Compatibility can be broken only at a major release, though breaking compatibility even at major releases has grave consequences and should be discussed in the Hadoop community.</li>
<li>Hadoop protocols are defined in .proto (ProtocolBuffers) files. Client-Server protocols and Server-protocol .proto files are marked as stable. When a .proto file is marked as stable it means that changes should be made in a compatible fashion as described below:
<ul>
<li>The following changes are compatible and are allowed at any time:
<ul>
<li>Add an optional field, with the expectation that the code deals with the field missing due to communication with an older version of the code.</li>
<li>Add a new rpc/method to the service</li>
<li>Add a new optional request to a Message</li>
<li>Rename a field</li>
<li>Rename a .proto file</li>
<li>Change .proto annotations that effect code generation (e.g. name of java package)</li></ul></li>
<li>The following changes are incompatible but can be considered only at a major release
<ul>
<li>Change the rpc/method name</li>
<li>Change the rpc/method parameter type or return type</li>
<li>Remove an rpc/method</li>
<li>Change the service name</li>
<li>Change the name of a Message</li>
<li>Modify a field type in an incompatible way (as defined recursively)</li>
<li>Change an optional field to required</li>
<li>Add or delete a required field</li>
<li>Delete an optional field as long as the optional field has reasonable defaults to allow deletions</li></ul></li>
<li>The following changes are incompatible and hence never allowed
<ul>
<li>Change a field id</li>
<li>Reuse an old field that was previously deleted.</li>
<li>Field numbers are cheap and changing and reusing is not a good idea.</li></ul></li></ul></li></ul></div></div>
<div class="section">
<h4>Java Binary compatibility for end-user applications i.e. Apache Hadoop ABI<a name="Java_Binary_compatibility_for_end-user_applications_i.e._Apache_Hadoop_ABI"></a></h4>
<p>As Apache Hadoop revisions are upgraded end-users reasonably expect that their applications should continue to work without any modifications. This is fulfilled as a result of support API compatibility, Semantic compatibility and Wire compatibility.</p>
<p>However, Apache Hadoop is a very complex, distributed system and services a very wide variety of use-cases. In particular, Apache Hadoop MapReduce is a very, very wide API; in the sense that end-users may make wide-ranging assumptions such as layout of the local disk when their map/reduce tasks are executing, environment variables for their tasks etc. In such cases, it becomes very hard to fully specify, and support, absolute compatibility.</p>
<div class="section">
<h5>Use cases<a name="Use_cases"></a></h5>
<ul>
<li>Existing MapReduce applications, including jars of existing packaged end-user applications and projects such as Apache Pig, Apache Hive, Cascading etc. should work unmodified when pointed to an upgraded Apache Hadoop cluster within a major release. </li>
<li>Existing YARN applications, including jars of existing packaged end-user applications and projects such as Apache Tez etc. should work unmodified when pointed to an upgraded Apache Hadoop cluster within a major release. </li>
<li>Existing applications which transfer data in/out of HDFS, including jars of existing packaged end-user applications and frameworks such as Apache Flume, should work unmodified when pointed to an upgraded Apache Hadoop cluster within a major release. </li></ul></div>
<div class="section">
<h5>Policy<a name="Policy"></a></h5>
<ul>
<li>Existing MapReduce, YARN &amp; HDFS applications and frameworks should work unmodified within a major release i.e. Apache Hadoop ABI is supported.</li>
<li>A very minor fraction of applications maybe affected by changes to disk layouts etc., the developer community will strive to minimize these changes and will not make them within a minor version. In more egregious cases, we will consider strongly reverting these breaking changes and invalidating offending releases if necessary.</li>
<li>In particular for MapReduce applications, the developer community will try our best to support provide binary compatibility across major releases e.g. applications using org.apache.hadoop.mapred.</li>
<li>APIs are supported compatibly across hadoop-1.x and hadoop-2.x. See <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html"> Compatibility for MapReduce applications between hadoop-1.x and hadoop-2.x</a> for more details.</li></ul></div></div>
<div class="section">
<h4>REST APIs<a name="REST_APIs"></a></h4>
<p>REST API compatibility corresponds to both the request (URLs) and responses to each request (content, which may contain other URLs). Hadoop REST APIs are specifically meant for stable use by clients across releases, even major releases. The following are the exposed REST APIs:</p>
<ul>
<li><a href="../hadoop-hdfs/WebHDFS.html">WebHDFS</a> - Stable</li>
<li><a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">ResourceManager</a></li>
<li><a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">NodeManager</a></li>
<li><a href="../../hadoop-yarn/hadoop-yarn-site/MapredAppMasterRest.html">MR Application Master</a></li>
<li><a href="../../hadoop-yarn/hadoop-yarn-site/HistoryServerRest.html">History Server</a></li></ul>
<div class="section">
<h5>Policy<a name="Policy"></a></h5>
<p>The APIs annotated stable in the text above preserve compatibility across at least one major release, and maybe deprecated by a newer version of the REST API in a major release.</p></div></div>
<div class="section">
<h4>Metrics/JMX<a name="MetricsJMX"></a></h4>
<p>While the Metrics API compatibility is governed by Java API compatibility, the actual metrics exposed by Hadoop need to be compatible for users to be able to automate using them (scripts etc.). Adding additional metrics is compatible. Modifying (eg changing the unit or measurement) or removing existing metrics breaks compatibility. Similarly, changes to JMX MBean object names also break compatibility.</p>
<div class="section">
<h5>Policy <a name="Policy"></a></h5>
<p>Metrics should preserve compatibility within the major release.</p></div></div>
<div class="section">
<h4>File formats &amp; Metadata<a name="File_formats__Metadata"></a></h4>
<p>User and system level data (including metadata) is stored in files of different formats. Changes to the metadata or the file formats used to store data/metadata can lead to incompatibilities between versions.</p>
<div class="section">
<h5>User-level file formats<a name="User-level_file_formats"></a></h5>
<p>Changes to formats that end-users use to store their data can prevent them for accessing the data in later releases, and hence it is highly important to keep those file-formats compatible. One can always add a &quot;new&quot; format improving upon an existing format. Examples of these formats include har, war, SequenceFileFormat etc.</p>
<div class="section">
<h6>Policy<a name="Policy"></a></h6>
<ul>
<li>Non-forward-compatible user-file format changes are restricted to major releases. When user-file formats change, new releases are expected to read existing formats, but may write data in formats incompatible with prior releases. Also, the community shall prefer to create a new format that programs must opt in to instead of making incompatible changes to existing formats.</li></ul></div></div>
<div class="section">
<h5>System-internal file formats<a name="System-internal_file_formats"></a></h5>
<p>Hadoop internal data is also stored in files and again changing these formats can lead to incompatibilities. While such changes are not as devastating as the user-level file formats, a policy on when the compatibility can be broken is important.</p>
<div class="section">
<h6>MapReduce<a name="MapReduce"></a></h6>
<p>MapReduce uses formats like I-File to store MapReduce-specific data.</p></div>
<div class="section">
<h6>Policy<a name="Policy"></a></h6>
<p>MapReduce-internal formats like IFile maintain compatibility within a major release. Changes to these formats can cause in-flight jobs to fail and hence we should ensure newer clients can fetch shuffle-data from old servers in a compatible manner.</p></div>
<div class="section">
<h6>HDFS Metadata<a name="HDFS_Metadata"></a></h6>
<p>HDFS persists metadata (the image and edit logs) in a particular format. Incompatible changes to either the format or the metadata prevent subsequent releases from reading older metadata. Such incompatible changes might require an HDFS &quot;upgrade&quot; to convert the metadata to make it accessible. Some changes can require more than one such &quot;upgrades&quot;.</p>
<p>Depending on the degree of incompatibility in the changes, the following potential scenarios can arise:</p>
<ul>
<li>Automatic: The image upgrades automatically, no need for an explicit &quot;upgrade&quot;.</li>
<li>Direct: The image is upgradable, but might require one explicit release &quot;upgrade&quot;.</li>
<li>Indirect: The image is upgradable, but might require upgrading to intermediate release(s) first.</li>
<li>Not upgradeable: The image is not upgradeable.</li></ul></div>
<div class="section">
<h6>Policy<a name="Policy"></a></h6>
<ul>
<li>A release upgrade must allow a cluster to roll-back to the older version and its older disk format. The rollback needs to restore the original data, but not required to restore the updated data.</li>
<li>HDFS metadata changes must be upgradeable via any of the upgrade paths - automatic, direct or indirect.</li>
<li>More detailed policies based on the kind of upgrade are yet to be considered.</li></ul></div></div></div>
<div class="section">
<h4>Command Line Interface (CLI)<a name="Command_Line_Interface_CLI"></a></h4>
<p>The Hadoop command line programs may be use either directly via the system shell or via shell scripts. Changing the path of a command, removing or renaming command line options, the order of arguments, or the command return code and output break compatibility and may adversely affect users.</p>
<div class="section">
<h5>Policy <a name="Policy"></a></h5>
<p>CLI commands are to be deprecated (warning when used) for one major release before they are removed or incompatibly modified in a subsequent major release.</p></div></div>
<div class="section">
<h4>Web UI<a name="Web_UI"></a></h4>
<p>Web UI, particularly the content and layout of web pages, changes could potentially interfere with attempts to screen scrape the web pages for information.</p>
<div class="section">
<h5>Policy<a name="Policy"></a></h5>
<p>Web pages are not meant to be scraped and hence incompatible changes to them are allowed at any time. Users are expected to use REST APIs to get any information.</p></div></div>
<div class="section">
<h4>Hadoop Configuration Files<a name="Hadoop_Configuration_Files"></a></h4>
<p>Users use (1) Hadoop-defined properties to configure and provide hints to Hadoop and (2) custom properties to pass information to jobs. Hence, compatibility of config properties is two-fold:</p>
<ul>
<li>Modifying key-names, units of values, and default values of Hadoop-defined properties.</li>
<li>Custom configuration property keys should not conflict with the namespace of Hadoop-defined properties. Typically, users should avoid using prefixes used by Hadoop: hadoop, io, ipc, fs, net, file, ftp, s3, kfs, ha, file, dfs, mapred, mapreduce, yarn.</li></ul>
<div class="section">
<h5>Policy <a name="Policy"></a></h5>
<ul>
<li>Hadoop-defined properties are to be deprecated at least for one major release before being removed. Modifying units for existing properties is not allowed.</li>
<li>The default values of Hadoop-defined properties can be changed across minor/major releases, but will remain the same across point releases within a minor release.</li>
<li>Currently, there is NO explicit policy regarding when new prefixes can be added/removed, and the list of prefixes to be avoided for custom configuration properties. However, as noted above, users should avoid using prefixes used by Hadoop: hadoop, io, ipc, fs, net, file, ftp, s3, kfs, ha, file, dfs, mapred, mapreduce, yarn.</li></ul></div></div>
<div class="section">
<h4>Directory Structure <a name="Directory_Structure"></a></h4>
<p>Source code, artifacts (source and tests), user logs, configuration files, output and job history are all stored on disk either local file system or HDFS. Changing the directory structure of these user-accessible files break compatibility, even in cases where the original path is preserved via symbolic links (if, for example, the path is accessed by a servlet that is configured to not follow symbolic links).</p>
<div class="section">
<h5>Policy<a name="Policy"></a></h5>
<ul>
<li>The layout of source code and build artifacts can change anytime, particularly so across major versions. Within a major version, the developers will attempt (no guarantees) to preserve the directory structure; however, individual files can be added/moved/deleted. The best way to ensure patches stay in sync with the code is to get them committed to the Apache source tree.</li>
<li>The directory structure of configuration files, user logs, and job history will be preserved across minor and point releases within a major release.</li></ul></div></div>
<div class="section">
<h4>Java Classpath<a name="Java_Classpath"></a></h4>
<p>User applications built against Hadoop might add all Hadoop jars (including Hadoop's library dependencies) to the application's classpath. Adding new dependencies or updating the version of existing dependencies may interfere with those in applications' classpaths.</p>
<div class="section">
<h5>Policy<a name="Policy"></a></h5>
<p>Currently, there is NO policy on when Hadoop's dependencies can change.</p></div></div>
<div class="section">
<h4>Environment variables<a name="Environment_variables"></a></h4>
<p>Users and related projects often utilize the exported environment variables (eg HADOOP_CONF_DIR), therefore removing or renaming environment variables is an incompatible change.</p>
<div class="section">
<h5>Policy<a name="Policy"></a></h5>
<p>Currently, there is NO policy on when the environment variables can change. Developers try to limit changes to major releases.</p></div></div>
<div class="section">
<h4>Build artifacts<a name="Build_artifacts"></a></h4>
<p>Hadoop uses maven for project management and changing the artifacts can affect existing user workflows.</p>
<div class="section">
<h5>Policy<a name="Policy"></a></h5>
<ul>
<li>Test artifacts: The test jars generated are strictly for internal use and are not expected to be used outside of Hadoop, similar to APIs annotated @Private, @Unstable.</li>
<li>Built artifacts: The hadoop-client artifact (maven groupId:artifactId) stays compatible within a major release, while the other artifacts can change in incompatible ways.</li></ul></div></div>
<div class="section">
<h4>Hardware/Software Requirements<a name="HardwareSoftware_Requirements"></a></h4>
<p>To keep up with the latest advances in hardware, operating systems, JVMs, and other software, new Hadoop releases or some of their features might require higher versions of the same. For a specific environment, upgrading Hadoop might require upgrading other dependent software components.</p>
<div class="section">
<h5>Policies<a name="Policies"></a></h5>
<ul>
<li>Hardware
<ul>
<li>Architecture: The community has no plans to restrict Hadoop to specific architectures, but can have family-specific optimizations.</li>
<li>Minimum resources: While there are no guarantees on the minimum resources required by Hadoop daemons, the community attempts to not increase requirements within a minor release.</li></ul></li>
<li>Operating Systems: The community will attempt to maintain the same OS requirements (OS kernel versions) within a minor release. Currently GNU/Linux and Microsoft Windows are the OSes officially supported by the community while Apache Hadoop is known to work reasonably well on other OSes such as Apple MacOSX, Solaris etc.</li>
<li>The JVM requirements will not change across point releases within the same minor release except if the JVM version under question becomes unsupported. Minor/major releases might require later versions of JVM for some/all of the supported operating systems.</li>
<li>Other software: The community tries to maintain the minimum versions of additional software required by Hadoop. For example, ssh, kerberos etc.</li></ul></div></div></div>
<div class="section">
<h3>References<a name="References"></a></h3>
<p>Here are some relevant JIRAs and pages related to the topic:</p>
<ul>
<li>The evolution of this document - <a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-9517">HADOOP-9517</a></li>
<li>Binary compatibility for MapReduce end-user applications between hadoop-1.x and hadoop-2.x - <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html"> MapReduce Compatibility between hadoop-1.x and hadoop-2.x</a></li>
<li>Annotations for interfaces as per interface classification schedule - <a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-7391">HADOOP-7391</a> <a href="./InterfaceClassification.html">Hadoop Interface Classification</a></li>
<li>Compatibility for Hadoop 1.x releases - <a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-5071">HADOOP-5071</a></li>
<li>The <a class="externalLink" href="http://wiki.apache.org/hadoop/Roadmap">Hadoop Roadmap</a> page that captures other release policies</li></ul></div></div>
</div>
</div>
<div class="clear">
<hr/>
</div>
<div id="footer">
<div class="xright">&#169; 2014
Apache Software Foundation
- <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a></div>
<div class="clear">
<hr/>
</div>
</div>
</body>
</html>
c4science · Help