Homec4science
Diffusion elastic-yarn (future) Browse

R3704/hadoop-2.3.0/share/hadoop/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html
c0a03c32f340future

Last Change
Raw File

HDFSHighAvailabilityWithNFS.html
View Options

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<!-- Generated by Apache Maven Doxia at 2014-02-11 -->
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Apache Hadoop 2.3.0 - Hadoop Distributed File System-2.3.0 - High Availability</title>
<style type="text/css" media="all">
@import url("./css/maven-base.css");
@import url("./css/maven-theme.css");
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
<meta name="Date-Revision-yyyymmdd" content="20140211" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body class="composite">
<div id="banner">
<a href="http://hadoop.apache.org/" id="bannerLeft">
<img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" />
</a>
<a href="http://www.apache.org/" id="bannerRight">
<img src="http://www.apache.org/images/asf_logo_wide.png" alt="" />
</a>
<div class="clear">
<hr/>
</div>
</div>
<div id="breadcrumbs">
<div class="xleft">
<a href="http://www.apache.org/" class="externalLink">Apache</a>
&gt;
<a href="http://hadoop.apache.org/" class="externalLink">Hadoop</a>
&gt;
<a href="../">hadoop-yarn</a>
&gt;
Apache Hadoop 2.3.0
</div>
<div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a>
|
<a href="https://svn.apache.org/repos/asf/hadoop/" class="externalLink">SVN</a>
|
<a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a>
&nbsp;| Last Published: 2014-02-11
&nbsp;| Version: 2.3.0
</div>
<div class="clear">
<hr/>
</div>
</div>
<div id="leftColumn">
<div id="navcolumn">
<h5>General</h5>
<ul>
<li class="none">
<a href="../../index.html">Overview</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/CommandsManual.html">Hadoop Commands Reference</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/FileSystemShell.html">File System Shell</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/Compatibility.html">Hadoop Compatibility</a>
</li>
</ul>
<h5>Common</h5>
<ul>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/Superusers.html">Superusers</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a>
</li>
</ul>
<h5>HDFS</h5>
<ul>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">HDFS User Guide</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html">High Availability With QJM</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html">High Availability With NFS</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">HDFS Snapshots</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">HDFS Architecture</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/Hftp.html">HFTP</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">C API libhdfs</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS REST API</a>
</li>
<li class="none">
<a href="../../hadoop-hdfs-httpfs/index.html">HttpFS Gateway</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">HDFS NFS Gateway</a>
</li>
</ul>
<h5>MapReduce</h5>
<ul>
<li class="none">
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibilty between Hadoop 1.x and Hadoop 2.x</a>
</li>
<li class="none">
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a>
</li>
<li class="none">
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a>
</li>
<li class="none">
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a>
</li>
</ul>
<h5>YARN</h5>
<ul>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/YARN.html">YARN Architecture</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">YARN Commands</a>
</li>
<li class="none">
<a href="../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a>
</li>
</ul>
<h5>YARN REST APIs</h5>
<ul>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/MapredAppMasterRest.html">MR Application Master</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/HistoryServerRest.html">History Server</a>
</li>
</ul>
<h5>Auth</h5>
<ul>
<li class="none">
<a href="../../hadoop-auth/index.html">Overview</a>
</li>
<li class="none">
<a href="../../hadoop-auth/Examples.html">Examples</a>
</li>
<li class="none">
<a href="../../hadoop-auth/Configuration.html">Configuration</a>
</li>
<li class="none">
<a href="../../hadoop-auth/BuildingIt.html">Building</a>
</li>
</ul>
<h5>Reference</h5>
<ul>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/releasenotes.html">Release Notes</a>
</li>
<li class="none">
<a href="../../api/index.html">API docs</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/CHANGES.txt">Common CHANGES.txt</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/CHANGES.txt">HDFS CHANGES.txt</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-mapreduce/CHANGES.txt">MapReduce CHANGES.txt</a>
</li>
</ul>
<h5>Configuration</h5>
<ul>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a>
</li>
<li class="none">
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a>
</li>
</ul>
<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
<img alt="Built by Maven" src="./images/logos/maven-feather.png"/>
</a>
</div>
</div>
<div id="bodyColumn">
<div id="contentBox">
<!-- Licensed under the Apache License, Version 2.0 (the "License"); --><!-- you may not use this file except in compliance with the License. --><!-- You may obtain a copy of the License at --><!-- --><!-- http://www.apache.org/licenses/LICENSE-2.0 --><!-- --><!-- Unless required by applicable law or agreed to in writing, software --><!-- distributed under the License is distributed on an "AS IS" BASIS, --><!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. --><!-- See the License for the specific language governing permissions and --><!-- limitations under the License. See accompanying LICENSE file. --><div class="section">
<h2>HDFS High Availability<a name="HDFS_High_Availability"></a></h2>
<ul>
<li><a href="#HDFS_High_Availability">HDFS High Availability</a>
<ul>
<li><a href="#Purpose">Purpose</a></li>
<li><a href="#Note:_Using_the_Quorum_Journal_Manager_or_Conventional_Shared_Storage">Note: Using the Quorum Journal Manager or Conventional Shared Storage</a></li>
<li><a href="#Background">Background</a></li>
<li><a href="#Architecture">Architecture</a></li>
<li><a href="#Hardware_resources">Hardware resources</a></li>
<li><a href="#Deployment">Deployment</a>
<ul>
<li><a href="#Configuration_overview">Configuration overview</a></li>
<li><a href="#Configuration_details">Configuration details</a></li>
<li><a href="#Deployment_details">Deployment details</a></li>
<li><a href="#Administrative_commands">Administrative commands</a></li></ul></li>
<li><a href="#Automatic_Failover">Automatic Failover</a>
<ul>
<li><a href="#Introduction">Introduction</a></li>
<li><a href="#Components">Components</a></li>
<li><a href="#Deploying_ZooKeeper">Deploying ZooKeeper</a></li>
<li><a href="#Before_you_begin">Before you begin</a></li>
<li><a href="#Configuring_automatic_failover">Configuring automatic failover</a></li>
<li><a href="#Initializing_HA_state_in_ZooKeeper">Initializing HA state in ZooKeeper</a></li>
<li><a href="#Starting_the_cluster_with_start-dfs.sh">Starting the cluster with start-dfs.sh</a></li>
<li><a href="#Starting_the_cluster_manually">Starting the cluster manually</a></li>
<li><a href="#Securing_access_to_ZooKeeper">Securing access to ZooKeeper</a></li>
<li><a href="#Verifying_automatic_failover">Verifying automatic failover</a></li></ul></li>
<li><a href="#Automatic_Failover_FAQ">Automatic Failover FAQ</a></li>
<li><a href="#BookKeeper_as_a_Shared_storage_EXPERIMENTAL">BookKeeper as a Shared storage (EXPERIMENTAL)</a></li></ul></li></ul>
<div class="section">
<h3><a name="Purpose">Purpose</a></h3>
<p>This guide provides an overview of the HDFS High Availability (HA) feature and how to configure and manage an HA HDFS cluster, using NFS for the shared storage required by the NameNodes.</p>
<p>This document assumes that the reader has a general understanding of general components and node types in an HDFS cluster. Please refer to the HDFS Architecture guide for details.</p></div>
<div class="section">
<h3><a name="Note:_Using_the_Quorum_Journal_Manager_or_Conventional_Shared_Storage">Note: Using the Quorum Journal Manager or Conventional Shared Storage</a></h3>
<p>This guide discusses how to configure and use HDFS HA using a shared NFS directory to share edit logs between the Active and Standby NameNodes. For information on how to configure HDFS HA using the Quorum Journal Manager instead of NFS, please see <a href="./HDFSHighAvailabilityWithQJM.html">this alternative guide.</a></p></div>
<div class="section">
<h3><a name="Background">Background</a></h3>
<p>Prior to Hadoop 2.0.0, the NameNode was a single point of failure (SPOF) in an HDFS cluster. Each cluster had a single NameNode, and if that machine or process became unavailable, the cluster as a whole would be unavailable until the NameNode was either restarted or brought up on a separate machine.</p>
<p>This impacted the total availability of the HDFS cluster in two major ways:</p>
<ul>
<li>In the case of an unplanned event such as a machine crash, the cluster would be unavailable until an operator restarted the NameNode.</li>
<li>Planned maintenance events such as software or hardware upgrades on the NameNode machine would result in windows of cluster downtime.</li></ul>
<p>The HDFS High Availability feature addresses the above problems by providing the option of running two redundant NameNodes in the same cluster in an Active/Passive configuration with a hot standby. This allows a fast failover to a new NameNode in the case that a machine crashes, or a graceful administrator-initiated failover for the purpose of planned maintenance.</p></div>
<div class="section">
<h3><a name="Architecture">Architecture</a></h3>
<p>In a typical HA cluster, two separate machines are configured as NameNodes. At any point in time, exactly one of the NameNodes is in an <i>Active</i> state, and the other is in a <i>Standby</i> state. The Active NameNode is responsible for all client operations in the cluster, while the Standby is simply acting as a slave, maintaining enough state to provide a fast failover if necessary.</p>
<p>In order for the Standby node to keep its state synchronized with the Active node, the current implementation requires that the two nodes both have access to a directory on a shared storage device (eg an NFS mount from a NAS). This restriction will likely be relaxed in future versions.</p>
<p>When any namespace modification is performed by the Active node, it durably logs a record of the modification to an edit log file stored in the shared directory. The Standby node is constantly watching this directory for edits, and as it sees the edits, it applies them to its own namespace. In the event of a failover, the Standby will ensure that it has read all of the edits from the shared storage before promoting itself to the Active state. This ensures that the namespace state is fully synchronized before a failover occurs.</p>
<p>In order to provide a fast failover, it is also necessary that the Standby node have up-to-date information regarding the location of blocks in the cluster. In order to achieve this, the DataNodes are configured with the location of both NameNodes, and send block location information and heartbeats to both.</p>
<p>It is vital for the correct operation of an HA cluster that only one of the NameNodes be Active at a time. Otherwise, the namespace state would quickly diverge between the two, risking data loss or other incorrect results. In order to ensure this property and prevent the so-called &quot;split-brain scenario,&quot; the administrator must configure at least one <i>fencing method</i> for the shared storage. During a failover, if it cannot be verified that the previous Active node has relinquished its Active state, the fencing process is responsible for cutting off the previous Active's access to the shared edits storage. This prevents it from making any further edits to the namespace, allowing the new Active to safely proceed with failover.</p></div>
<div class="section">
<h3><a name="Hardware_resources">Hardware resources</a></h3>
<p>In order to deploy an HA cluster, you should prepare the following:</p>
<ul>
<li><b>NameNode machines</b> - the machines on which you run the Active and Standby NameNodes should have equivalent hardware to each other, and equivalent hardware to what would be used in a non-HA cluster.</li>
<li><b>Shared storage</b> - you will need to have a shared directory which both NameNode machines can have read/write access to. Typically this is a remote filer which supports NFS and is mounted on each of the NameNode machines. Currently only a single shared edits directory is supported. Thus, the availability of the system is limited by the availability of this shared edits directory, and therefore in order to remove all single points of failure there needs to be redundancy for the shared edits directory. Specifically, multiple network paths to the storage, and redundancy in the storage itself (disk, network, and power). Beacuse of this, it is recommended that the shared storage server be a high-quality dedicated NAS appliance rather than a simple Linux server.</li></ul>
<p>Note that, in an HA cluster, the Standby NameNode also performs checkpoints of the namespace state, and thus it is not necessary to run a Secondary NameNode, CheckpointNode, or BackupNode in an HA cluster. In fact, to do so would be an error. This also allows one who is reconfiguring a non-HA-enabled HDFS cluster to be HA-enabled to reuse the hardware which they had previously dedicated to the Secondary NameNode.</p></div>
<div class="section">
<h3><a name="Deployment">Deployment</a></h3>
<div class="section">
<h4>Configuration overview<a name="Configuration_overview"></a></h4>
<p>Similar to Federation configuration, HA configuration is backward compatible and allows existing single NameNode configurations to work without change. The new configuration is designed such that all the nodes in the cluster may have the same configuration without the need for deploying different configuration files to different machines based on the type of the node.</p>
<p>Like HDFS Federation, HA clusters reuse the <tt>nameservice ID</tt> to identify a single HDFS instance that may in fact consist of multiple HA NameNodes. In addition, a new abstraction called <tt>NameNode ID</tt> is added with HA. Each distinct NameNode in the cluster has a different NameNode ID to distinguish it. To support a single configuration file for all of the NameNodes, the relevant configuration parameters are suffixed with the <b>nameservice ID</b> as well as the <b>NameNode ID</b>.</p></div>
<div class="section">
<h4>Configuration details<a name="Configuration_details"></a></h4>
<p>To configure HA NameNodes, you must add several configuration options to your <b>hdfs-site.xml</b> configuration file.</p>
<p>The order in which you set these configurations is unimportant, but the values you choose for <b>dfs.nameservices</b> and <b>dfs.ha.namenodes.[nameservice ID]</b> will determine the keys of those that follow. Thus, you should decide on these values before setting the rest of the configuration options.</p>
<ul>
<li><b>dfs.nameservices</b> - the logical name for this new nameservice
<p>Choose a logical name for this nameservice, for example &quot;mycluster&quot;, and use this logical name for the value of this config option. The name you choose is arbitrary. It will be used both for configuration and as the authority component of absolute HDFS paths in the cluster.</p>
<p><b>Note:</b> If you are also using HDFS Federation, this configuration setting should also include the list of other nameservices, HA or otherwise, as a comma-separated list.</p>
<div>
<pre>&lt;property&gt;
&lt;name&gt;dfs.nameservices&lt;/name&gt;
&lt;value&gt;mycluster&lt;/value&gt;
&lt;/property&gt;</pre></div></li>
<li><b>dfs.ha.namenodes.[nameservice ID]</b> - unique identifiers for each NameNode in the nameservice
<p>Configure with a list of comma-separated NameNode IDs. This will be used by DataNodes to determine all the NameNodes in the cluster. For example, if you used &quot;mycluster&quot; as the nameservice ID previously, and you wanted to use &quot;nn1&quot; and &quot;nn2&quot; as the individual IDs of the NameNodes, you would configure this as such:</p>
<div>
<pre>&lt;property&gt;
&lt;name&gt;dfs.ha.namenodes.mycluster&lt;/name&gt;
&lt;value&gt;nn1,nn2&lt;/value&gt;
&lt;/property&gt;</pre></div>
<p><b>Note:</b> Currently, only a maximum of two NameNodes may be configured per nameservice.</p></li>
<li><b>dfs.namenode.rpc-address.[nameservice ID].[name node ID]</b> - the fully-qualified RPC address for each NameNode to listen on
<p>For both of the previously-configured NameNode IDs, set the full address and IPC port of the NameNode processs. Note that this results in two separate configuration options. For example:</p>
<div>
<pre>&lt;property&gt;
&lt;name&gt;dfs.namenode.rpc-address.mycluster.nn1&lt;/name&gt;
&lt;value&gt;machine1.example.com:8020&lt;/value&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;dfs.namenode.rpc-address.mycluster.nn2&lt;/name&gt;
&lt;value&gt;machine2.example.com:8020&lt;/value&gt;
&lt;/property&gt;</pre></div>
<p><b>Note:</b> You may similarly configure the &quot;<b>servicerpc-address</b>&quot; setting if you so desire.</p></li>
<li><b>dfs.namenode.http-address.[nameservice ID].[name node ID]</b> - the fully-qualified HTTP address for each NameNode to listen on
<p>Similarly to <i>rpc-address</i> above, set the addresses for both NameNodes' HTTP servers to listen on. For example:</p>
<div>
<pre>&lt;property&gt;
&lt;name&gt;dfs.namenode.http-address.mycluster.nn1&lt;/name&gt;
&lt;value&gt;machine1.example.com:50070&lt;/value&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;dfs.namenode.http-address.mycluster.nn2&lt;/name&gt;
&lt;value&gt;machine2.example.com:50070&lt;/value&gt;
&lt;/property&gt;</pre></div>
<p><b>Note:</b> If you have Hadoop's security features enabled, you should also set the <i>https-address</i> similarly for each NameNode.</p></li>
<li><b>dfs.namenode.shared.edits.dir</b> - the location of the shared storage directory
<p>This is where one configures the path to the remote shared edits directory which the Standby NameNode uses to stay up-to-date with all the file system changes the Active NameNode makes. <b>You should only configure one of these directories.</b> This directory should be mounted r/w on both NameNode machines. The value of this setting should be the absolute path to this directory on the NameNode machines. For example:</p>
<div>
<pre>&lt;property&gt;
&lt;name&gt;dfs.namenode.shared.edits.dir&lt;/name&gt;
&lt;value&gt;file:///mnt/filer1/dfs/ha-name-dir-shared&lt;/value&gt;
&lt;/property&gt;</pre></div></li>
<li><b>dfs.client.failover.proxy.provider.[nameservice ID]</b> - the Java class that HDFS clients use to contact the Active NameNode
<p>Configure the name of the Java class which will be used by the DFS Client to determine which NameNode is the current Active, and therefore which NameNode is currently serving client requests. The only implementation which currently ships with Hadoop is the <b>ConfiguredFailoverProxyProvider</b>, so use this unless you are using a custom one. For example:</p>
<div>
<pre>&lt;property&gt;
&lt;name&gt;dfs.client.failover.proxy.provider.mycluster&lt;/name&gt;
&lt;value&gt;org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider&lt;/value&gt;
&lt;/property&gt;</pre></div></li>
<li><b>dfs.ha.fencing.methods</b> - a list of scripts or Java classes which will be used to fence the Active NameNode during a failover
<p>It is critical for correctness of the system that only one NameNode be in the Active state at any given time. Thus, during a failover, we first ensure that the Active NameNode is either in the Standby state, or the process has terminated, before transitioning the other NameNode to the Active state. In order to do this, you must configure at least one <b>fencing method.</b> These are configured as a carriage-return-separated list, which will be attempted in order until one indicates that fencing has succeeded. There are two methods which ship with Hadoop: <i>shell</i> and <i>sshfence</i>. For information on implementing your own custom fencing method, see the <i>org.apache.hadoop.ha.NodeFencer</i> class.</p>
<ul>
<li><b>sshfence</b> - SSH to the Active NameNode and kill the process
<p>The <i>sshfence</i> option SSHes to the target node and uses <i>fuser</i> to kill the process listening on the service's TCP port. In order for this fencing option to work, it must be able to SSH to the target node without providing a passphrase. Thus, one must also configure the <b>dfs.ha.fencing.ssh.private-key-files</b> option, which is a comma-separated list of SSH private key files. For example:</p>
<div>
<pre>&lt;property&gt;
&lt;name&gt;dfs.ha.fencing.methods&lt;/name&gt;
&lt;value&gt;sshfence&lt;/value&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;dfs.ha.fencing.ssh.private-key-files&lt;/name&gt;
&lt;value&gt;/home/exampleuser/.ssh/id_rsa&lt;/value&gt;
&lt;/property&gt;</pre></div>
<p>Optionally, one may configure a non-standard username or port to perform the SSH. One may also configure a timeout, in milliseconds, for the SSH, after which this fencing method will be considered to have failed. It may be configured like so:</p>
<div>
<pre>&lt;property&gt;
&lt;name&gt;dfs.ha.fencing.methods&lt;/name&gt;
&lt;value&gt;sshfence([[username][:port]])&lt;/value&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;dfs.ha.fencing.ssh.connect-timeout&lt;/name&gt;
&lt;value&gt;30000&lt;/value&gt;
&lt;/property&gt;</pre></div></li>
<li><b>shell</b> - run an arbitrary shell command to fence the Active NameNode
<p>The <i>shell</i> fencing method runs an arbitrary shell command. It may be configured like so:</p>
<div>
<pre>&lt;property&gt;
&lt;name&gt;dfs.ha.fencing.methods&lt;/name&gt;
&lt;value&gt;shell(/path/to/my/script.sh arg1 arg2 ...)&lt;/value&gt;
&lt;/property&gt;</pre></div>
<p>The string between '(' and ')' is passed directly to a bash shell and may not include any closing parentheses.</p>
<p>The shell command will be run with an environment set up to contain all of the current Hadoop configuration variables, with the '_' character replacing any '.' characters in the configuration keys. The configuration used has already had any namenode-specific configurations promoted to their generic forms -- for example <b>dfs_namenode_rpc-address</b> will contain the RPC address of the target node, even though the configuration may specify that variable as <b>dfs.namenode.rpc-address.ns1.nn1</b>.</p>
<p>Additionally, the following variables referring to the target node to be fenced are also available:</p>
<table border="1" class="bodyTable">
<tr class="a">
<td align="right">$target_host</td>
<td align="left">hostname of the node to be fenced</td></tr>
<tr class="b">
<td align="right">$target_port</td>
<td align="left">IPC port of the node to be fenced</td></tr>
<tr class="a">
<td align="right">$target_address</td>
<td align="left">the above two, combined as host:port</td></tr>
<tr class="b">
<td align="right">$target_nameserviceid</td>
<td align="left">the nameservice ID of the NN to be fenced</td></tr>
<tr class="a">
<td align="right">$target_namenodeid</td>
<td align="left">the namenode ID of the NN to be fenced</td></tr></table>
<p>These environment variables may also be used as substitutions in the shell command itself. For example:</p>
<div>
<pre>&lt;property&gt;
&lt;name&gt;dfs.ha.fencing.methods&lt;/name&gt;
&lt;value&gt;shell(/path/to/my/script.sh --nameservice=$target_nameserviceid $target_host:$target_port)&lt;/value&gt;
&lt;/property&gt;</pre></div>
<p>If the shell command returns an exit code of 0, the fencing is determined to be successful. If it returns any other exit code, the fencing was not successful and the next fencing method in the list will be attempted.</p>
<p><b>Note:</b> This fencing method does not implement any timeout. If timeouts are necessary, they should be implemented in the shell script itself (eg by forking a subshell to kill its parent in some number of seconds).</p></li></ul></li>
<li><b>fs.defaultFS</b> - the default path prefix used by the Hadoop FS client when none is given
<p>Optionally, you may now configure the default path for Hadoop clients to use the new HA-enabled logical URI. If you used &quot;mycluster&quot; as the nameservice ID earlier, this will be the value of the authority portion of all of your HDFS paths. This may be configured like so, in your <b>core-site.xml</b> file:</p>
<div>
<pre>&lt;property&gt;
&lt;name&gt;fs.defaultFS&lt;/name&gt;
&lt;value&gt;hdfs://mycluster&lt;/value&gt;
&lt;/property&gt;</pre></div></li></ul></div>
<div class="section">
<h4>Deployment details<a name="Deployment_details"></a></h4>
<p>After all of the necessary configuration options have been set, one must initially synchronize the two HA NameNodes' on-disk metadata.</p>
<ul>
<li>If you are setting up a fresh HDFS cluster, you should first run the format command (<i>hdfs namenode -format</i>) on one of NameNodes.</li>
<li>If you have already formatted the NameNode, or are converting a non-HA-enabled cluster to be HA-enabled, you should now copy over the contents of your NameNode metadata directories to the other, unformatted NameNode by running the command &quot;<i>hdfs namenode -bootstrapStandby</i>&quot; on the unformatted NameNode. Running this command will also ensure that the shared edits directory (as configured by <b>dfs.namenode.shared.edits.dir</b>) contains sufficient edits transactions to be able to start both NameNodes.</li>
<li>If you are converting a non-HA NameNode to be HA, you should run the command &quot;<i>hdfs -initializeSharedEdits</i>&quot;, which will initialize the shared edits directory with the edits data from the local NameNode edits directories.</li></ul>
<p>At this point you may start both of your HA NameNodes as you normally would start a NameNode.</p>
<p>You can visit each of the NameNodes' web pages separately by browsing to their configured HTTP addresses. You should notice that next to the configured address will be the HA state of the NameNode (either &quot;standby&quot; or &quot;active&quot;.) Whenever an HA NameNode starts, it is initially in the Standby state.</p></div>
<div class="section">
<h4>Administrative commands<a name="Administrative_commands"></a></h4>
<p>Now that your HA NameNodes are configured and started, you will have access to some additional commands to administer your HA HDFS cluster. Specifically, you should familiarize yourself with all of the subcommands of the &quot;<i>hdfs haadmin</i>&quot; command. Running this command without any additional arguments will display the following usage information:</p>
<div>
<pre>Usage: DFSHAAdmin [-ns &lt;nameserviceId&gt;]
[-transitionToActive &lt;serviceId&gt;]
[-transitionToStandby &lt;serviceId&gt;]
[-failover [--forcefence] [--forceactive] &lt;serviceId&gt; &lt;serviceId&gt;]
[-getServiceState &lt;serviceId&gt;]
[-checkHealth &lt;serviceId&gt;]
[-help &lt;command&gt;]</pre></div>
<p>This guide describes high-level uses of each of these subcommands. For specific usage information of each subcommand, you should run &quot;<i>hdfs haadmin -help &lt;command</i>&gt;&quot;.</p>
<ul>
<li><b>transitionToActive</b> and <b>transitionToStandby</b> - transition the state of the given NameNode to Active or Standby
<p>These subcommands cause a given NameNode to transition to the Active or Standby state, respectively. <b>These commands do not attempt to perform any fencing, and thus should rarely be used.</b> Instead, one should almost always prefer to use the &quot;<i>hdfs haadmin -failover</i>&quot; subcommand.</p></li>
<li><b>failover</b> - initiate a failover between two NameNodes
<p>This subcommand causes a failover from the first provided NameNode to the second. If the first NameNode is in the Standby state, this command simply transitions the second to the Active state without error. If the first NameNode is in the Active state, an attempt will be made to gracefully transition it to the Standby state. If this fails, the fencing methods (as configured by <b>dfs.ha.fencing.methods</b>) will be attempted in order until one succeeds. Only after this process will the second NameNode be transitioned to the Active state. If no fencing method succeeds, the second NameNode will not be transitioned to the Active state, and an error will be returned.</p></li>
<li><b>getServiceState</b> - determine whether the given NameNode is Active or Standby
<p>Connect to the provided NameNode to determine its current state, printing either &quot;standby&quot; or &quot;active&quot; to STDOUT appropriately. This subcommand might be used by cron jobs or monitoring scripts which need to behave differently based on whether the NameNode is currently Active or Standby.</p></li>
<li><b>checkHealth</b> - check the health of the given NameNode
<p>Connect to the provided NameNode to check its health. The NameNode is capable of performing some diagnostics on itself, including checking if internal services are running as expected. This command will return 0 if the NameNode is healthy, non-zero otherwise. One might use this command for monitoring purposes.</p>
<p><b>Note:</b> This is not yet implemented, and at present will always return success, unless the given NameNode is completely down.</p></li></ul></div></div>
<div class="section">
<h3><a name="Automatic_Failover">Automatic Failover</a></h3>
<div class="section">
<h4>Introduction<a name="Introduction"></a></h4>
<p>The above sections describe how to configure manual failover. In that mode, the system will not automatically trigger a failover from the active to the standby NameNode, even if the active node has failed. This section describes how to configure and deploy automatic failover.</p></div>
<div class="section">
<h4>Components<a name="Components"></a></h4>
<p>Automatic failover adds two new components to an HDFS deployment: a ZooKeeper quorum, and the ZKFailoverController process (abbreviated as ZKFC).</p>
<p>Apache ZooKeeper is a highly available service for maintaining small amounts of coordination data, notifying clients of changes in that data, and monitoring clients for failures. The implementation of automatic HDFS failover relies on ZooKeeper for the following things:</p>
<ul>
<li><b>Failure detection</b> - each of the NameNode machines in the cluster maintains a persistent session in ZooKeeper. If the machine crashes, the ZooKeeper session will expire, notifying the other NameNode that a failover should be triggered.</li>
<li><b>Active NameNode election</b> - ZooKeeper provides a simple mechanism to exclusively elect a node as active. If the current active NameNode crashes, another node may take a special exclusive lock in ZooKeeper indicating that it should become the next active.</li></ul>
<p>The ZKFailoverController (ZKFC) is a new component which is a ZooKeeper client which also monitors and manages the state of the NameNode. Each of the machines which runs a NameNode also runs a ZKFC, and that ZKFC is responsible for:</p>
<ul>
<li><b>Health monitoring</b> - the ZKFC pings its local NameNode on a periodic basis with a health-check command. So long as the NameNode responds in a timely fashion with a healthy status, the ZKFC considers the node healthy. If the node has crashed, frozen, or otherwise entered an unhealthy state, the health monitor will mark it as unhealthy.</li>
<li><b>ZooKeeper session management</b> - when the local NameNode is healthy, the ZKFC holds a session open in ZooKeeper. If the local NameNode is active, it also holds a special &quot;lock&quot; znode. This lock uses ZooKeeper's support for &quot;ephemeral&quot; nodes; if the session expires, the lock node will be automatically deleted.</li>
<li><b>ZooKeeper-based election</b> - if the local NameNode is healthy, and the ZKFC sees that no other node currently holds the lock znode, it will itself try to acquire the lock. If it succeeds, then it has &quot;won the election&quot;, and is responsible for running a failover to make its local NameNode active. The failover process is similar to the manual failover described above: first, the previous active is fenced if necessary, and then the local NameNode transitions to active state.</li></ul>
<p>For more details on the design of automatic failover, refer to the design document attached to HDFS-2185 on the Apache HDFS JIRA.</p></div>
<div class="section">
<h4>Deploying ZooKeeper<a name="Deploying_ZooKeeper"></a></h4>
<p>In a typical deployment, ZooKeeper daemons are configured to run on three or five nodes. Since ZooKeeper itself has light resource requirements, it is acceptable to collocate the ZooKeeper nodes on the same hardware as the HDFS NameNode and Standby Node. Many operators choose to deploy the third ZooKeeper process on the same node as the YARN ResourceManager. It is advisable to configure the ZooKeeper nodes to store their data on separate disk drives from the HDFS metadata for best performance and isolation.</p>
<p>The setup of ZooKeeper is out of scope for this document. We will assume that you have set up a ZooKeeper cluster running on three or more nodes, and have verified its correct operation by connecting using the ZK CLI.</p></div>
<div class="section">
<h4>Before you begin<a name="Before_you_begin"></a></h4>
<p>Before you begin configuring automatic failover, you should shut down your cluster. It is not currently possible to transition from a manual failover setup to an automatic failover setup while the cluster is running.</p></div>
<div class="section">
<h4>Configuring automatic failover<a name="Configuring_automatic_failover"></a></h4>
<p>The configuration of automatic failover requires the addition of two new parameters to your configuration. In your <tt>hdfs-site.xml</tt> file, add:</p>
<div>
<pre> &lt;property&gt;
&lt;name&gt;dfs.ha.automatic-failover.enabled&lt;/name&gt;
&lt;value&gt;true&lt;/value&gt;
&lt;/property&gt;</pre></div>
<p>This specifies that the cluster should be set up for automatic failover. In your <tt>core-site.xml</tt> file, add:</p>
<div>
<pre> &lt;property&gt;
&lt;name&gt;ha.zookeeper.quorum&lt;/name&gt;
&lt;value&gt;zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181&lt;/value&gt;
&lt;/property&gt;</pre></div>
<p>This lists the host-port pairs running the ZooKeeper service.</p>
<p>As with the parameters described earlier in the document, these settings may be configured on a per-nameservice basis by suffixing the configuration key with the nameservice ID. For example, in a cluster with federation enabled, you can explicitly enable automatic failover for only one of the nameservices by setting <tt>dfs.ha.automatic-failover.enabled.my-nameservice-id</tt>.</p>
<p>There are also several other configuration parameters which may be set to control the behavior of automatic failover; however, they are not necessary for most installations. Please refer to the configuration key specific documentation for details.</p></div>
<div class="section">
<h4>Initializing HA state in ZooKeeper<a name="Initializing_HA_state_in_ZooKeeper"></a></h4>
<p>After the configuration keys have been added, the next step is to initialize required state in ZooKeeper. You can do so by running the following command from one of the NameNode hosts.</p>
<div>
<pre>$ hdfs zkfc -formatZK</pre></div>
<p>This will create a znode in ZooKeeper inside of which the automatic failover system stores its data.</p></div>
<div class="section">
<h4>Starting the cluster with <tt>start-dfs.sh</tt><a name="Starting_the_cluster_with_start-dfs.sh"></a></h4>
<p>Since automatic failover has been enabled in the configuration, the <tt>start-dfs.sh</tt> script will now automatically start a ZKFC daemon on any machine that runs a NameNode. When the ZKFCs start, they will automatically select one of the NameNodes to become active.</p></div>
<div class="section">
<h4>Starting the cluster manually<a name="Starting_the_cluster_manually"></a></h4>
<p>If you manually manage the services on your cluster, you will need to manually start the <tt>zkfc</tt> daemon on each of the machines that runs a NameNode. You can start the daemon by running:</p>
<div>
<pre>$ hadoop-daemon.sh start zkfc</pre></div></div>
<div class="section">
<h4>Securing access to ZooKeeper<a name="Securing_access_to_ZooKeeper"></a></h4>
<p>If you are running a secure cluster, you will likely want to ensure that the information stored in ZooKeeper is also secured. This prevents malicious clients from modifying the metadata in ZooKeeper or potentially triggering a false failover.</p>
<p>In order to secure the information in ZooKeeper, first add the following to your <tt>core-site.xml</tt> file:</p>
<div>
<pre> &lt;property&gt;
&lt;name&gt;ha.zookeeper.auth&lt;/name&gt;
&lt;value&gt;@/path/to/zk-auth.txt&lt;/value&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;ha.zookeeper.acl&lt;/name&gt;
&lt;value&gt;@/path/to/zk-acl.txt&lt;/value&gt;
&lt;/property&gt;</pre></div>
<p>Please note the '@' character in these values -- this specifies that the configurations are not inline, but rather point to a file on disk.</p>
<p>The first configured file specifies a list of ZooKeeper authentications, in the same format as used by the ZK CLI. For example, you may specify something like:</p>
<div>
<pre>digest:hdfs-zkfcs:mypassword</pre></div>
<p>...where <tt>hdfs-zkfcs</tt> is a unique username for ZooKeeper, and <tt>mypassword</tt> is some unique string used as a password.</p>
<p>Next, generate a ZooKeeper ACL that corresponds to this authentication, using a command like the following:</p>
<div>
<pre>$ java -cp $ZK_HOME/lib/*:$ZK_HOME/zookeeper-3.4.2.jar org.apache.zookeeper.server.auth.DigestAuthenticationProvider hdfs-zkfcs:mypassword
output: hdfs-zkfcs:mypassword-&gt;hdfs-zkfcs:P/OQvnYyU/nF/mGYvB/xurX8dYs=</pre></div>
<p>Copy and paste the section of this output after the '-&gt;' string into the file <tt>zk-acls.txt</tt>, prefixed by the string &quot;<tt>digest:</tt>&quot;. For example:</p>
<div>
<pre>digest:hdfs-zkfcs:vlUvLnd8MlacsE80rDuu6ONESbM=:rwcda</pre></div>
<p>In order for these ACLs to take effect, you should then rerun the <tt>zkfc -formatZK</tt> command as described above.</p>
<p>After doing so, you may verify the ACLs from the ZK CLI as follows:</p>
<div>
<pre>[zk: localhost:2181(CONNECTED) 1] getAcl /hadoop-ha
'digest,'hdfs-zkfcs:vlUvLnd8MlacsE80rDuu6ONESbM=
: cdrwa</pre></div></div>
<div class="section">
<h4>Verifying automatic failover<a name="Verifying_automatic_failover"></a></h4>
<p>Once automatic failover has been set up, you should test its operation. To do so, first locate the active NameNode. You can tell which node is active by visiting the NameNode web interfaces -- each node reports its HA state at the top of the page.</p>
<p>Once you have located your active NameNode, you may cause a failure on that node. For example, you can use <tt>kill -9 &lt;pid of NN</tt>&gt; to simulate a JVM crash. Or, you could power cycle the machine or unplug its network interface to simulate a different kind of outage. After triggering the outage you wish to test, the other NameNode should automatically become active within several seconds. The amount of time required to detect a failure and trigger a fail-over depends on the configuration of <tt>ha.zookeeper.session-timeout.ms</tt>, but defaults to 5 seconds.</p>
<p>If the test does not succeed, you may have a misconfiguration. Check the logs for the <tt>zkfc</tt> daemons as well as the NameNode daemons in order to further diagnose the issue.</p></div></div>
<div class="section">
<h3>Automatic Failover FAQ<a name="Automatic_Failover_FAQ"></a></h3>
<ul>
<li><b>Is it important that I start the ZKFC and NameNode daemons in any particular order?</b>
<p>No. On any given node you may start the ZKFC before or after its corresponding NameNode.</p></li>
<li><b>What additional monitoring should I put in place?</b>
<p>You should add monitoring on each host that runs a NameNode to ensure that the ZKFC remains running. In some types of ZooKeeper failures, for example, the ZKFC may unexpectedly exit, and should be restarted to ensure that the system is ready for automatic failover.</p>
<p>Additionally, you should monitor each of the servers in the ZooKeeper quorum. If ZooKeeper crashes, then automatic failover will not function.</p></li>
<li><b>What happens if ZooKeeper goes down?</b>
<p>If the ZooKeeper cluster crashes, no automatic failovers will be triggered. However, HDFS will continue to run without any impact. When ZooKeeper is restarted, HDFS will reconnect with no issues.</p></li>
<li><b>Can I designate one of my NameNodes as primary/preferred?</b>
<p>No. Currently, this is not supported. Whichever NameNode is started first will become active. You may choose to start the cluster in a specific order such that your preferred node starts first.</p></li>
<li><b>How can I initiate a manual failover when automatic failover is configured?</b>
<p>Even if automatic failover is configured, you may initiate a manual failover using the same <tt>hdfs haadmin</tt> command. It will perform a coordinated failover.</p></li></ul></div>
<div class="section">
<h3>BookKeeper as a Shared storage (EXPERIMENTAL)<a name="BookKeeper_as_a_Shared_storage_EXPERIMENTAL"></a></h3>
<p>One option for shared storage for the NameNode is BookKeeper. BookKeeper achieves high availability and strong durability guarantees by replicating edit log entries across multiple storage nodes. The edit log can be striped across the storage nodes for high performance. Fencing is supported in the protocol, i.e, BookKeeper will not allow two writers to write the single edit log.</p>
<p>The meta data for BookKeeper is stored in ZooKeeper. In current HA architecture, a Zookeeper cluster is required for ZKFC. The same cluster can be for BookKeeper metadata.</p>
<p>For more details on building a BookKeeper cluster, please refer to the <a class="externalLink" href="http://zookeeper.apache.org/bookkeeper/docs/trunk/bookkeeperConfig.html ">BookKeeper documentation</a></p>
<p>The BookKeeperJournalManager is an implementation of the HDFS JournalManager interface, which allows custom write ahead logging implementations to be plugged into the HDFS NameNode.</p>
<ul>
<li><b>BookKeeper Journal Manager</b>
<p>To use BookKeeperJournalManager, add the following to hdfs-site.xml.</p>
<div>
<pre> &lt;property&gt;
&lt;name&gt;dfs.namenode.shared.edits.dir&lt;/name&gt;
&lt;value&gt;bookkeeper://zk1:2181;zk2:2181;zk3:2181/hdfsjournal&lt;/value&gt;
&lt;/property&gt;
&lt;property&gt;
&lt;name&gt;dfs.namenode.edits.journal-plugin.bookkeeper&lt;/name&gt;
&lt;value&gt;org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager&lt;/value&gt;
&lt;/property&gt;</pre></div>
<p>The URI format for bookkeeper is <tt>bookkeeper://[zkEnsemble]/[rootZnode] [zookkeeper ensemble]</tt> is a list of semi-colon separated, zookeeper host:port pairs. In the example above there are 3 servers, in the ensemble, zk1, zk2 &amp; zk3, each one listening on port 2181.</p>
<p><tt>[root znode]</tt> is the path of the zookeeper znode, under which the edit log information will be stored.</p>
<p>The class specified for the journal-plugin must be available in the NameNode's classpath. We explain how to generate a jar file with the journal manager and its dependencies, and how to put it into the classpath below.</p></li>
<li><b>More configuration options</b>
<ul>
<li><b>dfs.namenode.bookkeeperjournal.output-buffer-size</b> - Number of bytes a bookkeeper journal stream will buffer before forcing a flush. Default is 1024.
<div>
<pre> &lt;property&gt;
&lt;name&gt;dfs.namenode.bookkeeperjournal.output-buffer-size&lt;/name&gt;
&lt;value&gt;1024&lt;/value&gt;
&lt;/property&gt;</pre></div></li>
<li><b>dfs.namenode.bookkeeperjournal.ensemble-size</b> - Number of bookkeeper servers in edit log ensembles. This is the number of bookkeeper servers which need to be available for the edit log to be writable. Default is 3.
<div>
<pre> &lt;property&gt;
&lt;name&gt;dfs.namenode.bookkeeperjournal.ensemble-size&lt;/name&gt;
&lt;value&gt;3&lt;/value&gt;
&lt;/property&gt;</pre></div></li>
<li><b>dfs.namenode.bookkeeperjournal.quorum-size</b> - Number of bookkeeper servers in the write quorum. This is the number of bookkeeper servers which must have acknowledged the write of an entry before it is considered written. Default is 2.
<div>
<pre> &lt;property&gt;
&lt;name&gt;dfs.namenode.bookkeeperjournal.quorum-size&lt;/name&gt;
&lt;value&gt;2&lt;/value&gt;
&lt;/property&gt;</pre></div></li>
<li><b>dfs.namenode.bookkeeperjournal.digestPw</b> - Password to use when creating edit log segments.
<div>
<pre> &lt;property&gt;
&lt;name&gt;dfs.namenode.bookkeeperjournal.digestPw&lt;/name&gt;
&lt;value&gt;myPassword&lt;/value&gt;
&lt;/property&gt;</pre></div></li>
<li><b>dfs.namenode.bookkeeperjournal.zk.session.timeout</b> - Session timeout for Zookeeper client from BookKeeper Journal Manager. Hadoop recommends that this value should be less than the ZKFC session timeout value. Default value is 3000.
<div>
<pre> &lt;property&gt;
&lt;name&gt;dfs.namenode.bookkeeperjournal.zk.session.timeout&lt;/name&gt;
&lt;value&gt;3000&lt;/value&gt;
&lt;/property&gt;</pre></div></li></ul></li>
<li><b>Building BookKeeper Journal Manager plugin jar</b>
<p>To generate the distribution packages for BK journal, do the following.</p>
<p>$ mvn clean package -Pdist</p>
<p>This will generate a jar with the BookKeeperJournalManager, all the dependencies needed by the journal manager, hadoop-hdfs/src/contrib/bkjournal/target/hadoop-hdfs-bkjournal-<i>VERSION</i>.jar</p>
<p>Note that the -Pdist part of the build command is important, as otherwise the dependencies would not be packaged in the jar. The dependencies included in the jar are <a class="externalLink" href="http://maven.apache.org/plugins/maven-shade-plugin/">shaded</a> to avoid conflicts with other dependencies of the NameNode.</p></li>
<li><b>Putting the BookKeeperJournalManager in the NameNode classpath</b>
<p>To run a HDFS namenode using BookKeeper as a backend, copy the bkjournal jar, generated above, into the lib directory of hdfs. In the standard distribution of HDFS, this is at $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/</p>
<p>cp hadoop-hdfs/src/contrib/bkjournal/target/hadoop-hdfs-bkjournal-<i>VERSION</i>.jar $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/</p></li>
<li><b>Current limitations</b>
<p>1) NameNode format command will not format the BookKeeper data automatically. We have to clean the data manually from BookKeeper cluster and create the /ledgers/available path in Zookeeper. ---- $ zkCli.sh create /ledgers 0 $ zkCli.sh create /ledgers/available 0 ---- Note: bookkeeper://zk1:2181;zk2:2181;zk3:2181/hdfsjournal The final part /hdfsjournal specifies the znode in zookeeper where ledger metadata will be stored. Administrators may set this to anything they wish.</p>
<p>2) Security in BookKeeper. BookKeeper does not support SASL nor SSL for connections between the NameNode and BookKeeper storage nodes.</p>
<p>3) Auto-Recovery of storage node failures. Work inprogress <a class="externalLink" href="https://issues.apache.org/jira/browse/BOOKKEEPER-237 ">BOOKKEEPER-237</a>. Currently we have the tools to manually recover the data from failed storage nodes.</p></li></ul></div></div>
</div>
</div>
<div class="clear">
<hr/>
</div>
<div id="footer">
<div class="xright">&#169; 2014
Apache Software Foundation
- <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a></div>
<div class="clear">
<hr/>
</div>
</div>
</body>
</html>
c4science · Help