Page MenuHomec4science
Files F102925547

SingleNodeSetup.html
No OneTemporary
Actions

File Metadata

Created
Tue, Feb 25, 13:26

SingleNodeSetup.html
View Options

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<!-- Generated by Apache Maven Doxia at 2014-02-11 -->
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Apache Hadoop 2.3.0 - Single Node Setup</title>
<style type="text/css" media="all">
@import url("./css/maven-base.css");
@import url("./css/maven-theme.css");
@import url("./css/site.css");
</style>
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
<meta name="Date-Revision-yyyymmdd" content="20140211" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body class="composite">
<div id="banner">
<a href="http://hadoop.apache.org/" id="bannerLeft">
<img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" />
</a>
<a href="http://www.apache.org/" id="bannerRight">
<img src="http://www.apache.org/images/asf_logo_wide.png" alt="" />
</a>
<div class="clear">
<hr/>
</div>
</div>
<div id="breadcrumbs">
<div class="xleft">
<a href="http://www.apache.org/" class="externalLink">Apache</a>
&gt;
<a href="http://hadoop.apache.org/" class="externalLink">Hadoop</a>
&gt;
<a href="../">Apache Hadoop Project Dist POM</a>
&gt;
Apache Hadoop 2.3.0
</div>
<div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a>
|
<a href="https://svn.apache.org/repos/asf/hadoop/" class="externalLink">SVN</a>
|
<a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a>
&nbsp;| Last Published: 2014-02-11
&nbsp;| Version: 2.3.0
</div>
<div class="clear">
<hr/>
</div>
</div>
<div id="leftColumn">
<div id="navcolumn">
<h5>General</h5>
<ul>
<li class="none">
<a href="../../index.html">Overview</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/CommandsManual.html">Hadoop Commands Reference</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/FileSystemShell.html">File System Shell</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/Compatibility.html">Hadoop Compatibility</a>
</li>
</ul>
<h5>Common</h5>
<ul>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/Superusers.html">Superusers</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a>
</li>
</ul>
<h5>HDFS</h5>
<ul>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">HDFS User Guide</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html">High Availability With QJM</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html">High Availability With NFS</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">HDFS Snapshots</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">HDFS Architecture</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/Hftp.html">HFTP</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">C API libhdfs</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS REST API</a>
</li>
<li class="none">
<a href="../../hadoop-hdfs-httpfs/index.html">HttpFS Gateway</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">HDFS NFS Gateway</a>
</li>
</ul>
<h5>MapReduce</h5>
<ul>
<li class="none">
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibilty between Hadoop 1.x and Hadoop 2.x</a>
</li>
<li class="none">
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a>
</li>
<li class="none">
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a>
</li>
<li class="none">
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a>
</li>
</ul>
<h5>YARN</h5>
<ul>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/YARN.html">YARN Architecture</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">YARN Commands</a>
</li>
<li class="none">
<a href="../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a>
</li>
</ul>
<h5>YARN REST APIs</h5>
<ul>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/MapredAppMasterRest.html">MR Application Master</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-site/HistoryServerRest.html">History Server</a>
</li>
</ul>
<h5>Auth</h5>
<ul>
<li class="none">
<a href="../../hadoop-auth/index.html">Overview</a>
</li>
<li class="none">
<a href="../../hadoop-auth/Examples.html">Examples</a>
</li>
<li class="none">
<a href="../../hadoop-auth/Configuration.html">Configuration</a>
</li>
<li class="none">
<a href="../../hadoop-auth/BuildingIt.html">Building</a>
</li>
</ul>
<h5>Reference</h5>
<ul>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/releasenotes.html">Release Notes</a>
</li>
<li class="none">
<a href="../../api/index.html">API docs</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/CHANGES.txt">Common CHANGES.txt</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/CHANGES.txt">HDFS CHANGES.txt</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-mapreduce/CHANGES.txt">MapReduce CHANGES.txt</a>
</li>
</ul>
<h5>Configuration</h5>
<ul>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a>
</li>
<li class="none">
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a>
</li>
<li class="none">
<a href="../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a>
</li>
<li class="none">
<a href="../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a>
</li>
</ul>
<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
<img alt="Built by Maven" src="./images/logos/maven-feather.png"/>
</a>
</div>
</div>
<div id="bodyColumn">
<div id="contentBox">
<!-- Licensed under the Apache License, Version 2.0 (the "License"); --><!-- you may not use this file except in compliance with the License. --><!-- You may obtain a copy of the License at --><!-- --><!-- http://www.apache.org/licenses/LICENSE-2.0 --><!-- --><!-- Unless required by applicable law or agreed to in writing, software --><!-- distributed under the License is distributed on an "AS IS" BASIS, --><!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. --><!-- See the License for the specific language governing permissions and --><!-- limitations under the License. See accompanying LICENSE file. --><div class="section">
<h2>Single Node Setup<a name="Single_Node_Setup"></a></h2>
<ul>
<li><a href="#Single_Node_Setup">Single Node Setup</a>
<ul>
<li><a href="#Purpose">Purpose</a></li>
<li><a href="#Prerequisites">Prerequisites</a>
<ul>
<li><a href="#Supported_Platforms">Supported Platforms</a></li>
<li><a href="#Required_Software">Required Software</a></li>
<li><a href="#Installing_Software">Installing Software</a></li></ul></li>
<li><a href="#Download">Download</a></li>
<li><a href="#Prepare_to_Start_the_Hadoop_Cluster">Prepare to Start the Hadoop Cluster</a></li>
<li><a href="#Standalone_Operation">Standalone Operation</a></li>
<li><a href="#Pseudo-Distributed_Operation">Pseudo-Distributed Operation</a>
<ul>
<li><a href="#Configuration">Configuration</a></li>
<li><a href="#Setup_passphraseless_ssh">Setup passphraseless ssh</a></li>
<li><a href="#Execution">Execution</a></li></ul></li>
<li><a href="#Fully-Distributed_Operation">Fully-Distributed Operation</a></li></ul></li></ul>
<div class="section">
<h3>Purpose<a name="Purpose"></a></h3>
<p>This document describes how to set up and configure a single-node Hadoop installation so that you can quickly perform simple operations using Hadoop MapReduce and the Hadoop Distributed File System (HDFS).</p></div>
<div class="section">
<h3>Prerequisites<a name="Prerequisites"></a></h3>
<div class="section">
<h4>Supported Platforms<a name="Supported_Platforms"></a></h4>
<ul>
<li>GNU/Linux is supported as a development and production platform. Hadoop has been demonstrated on GNU/Linux clusters with 2000 nodes.</li>
<li>Windows is also a supported platform.</li></ul></div>
<div class="section">
<h4>Required Software<a name="Required_Software"></a></h4>
<p>Required software for Linux and Windows include:</p>
<ol style="list-style-type: decimal">
<li>Java^TM 1.6.x, preferably from Sun, must be installed.</li>
<li>ssh must be installed and sshd must be running to use the Hadoop scripts that manage remote Hadoop daemons.</li></ol></div>
<div class="section">
<h4>Installing Software<a name="Installing_Software"></a></h4>
<p>If your cluster doesn't have the requisite software you will need to install it.</p>
<p>For example on Ubuntu Linux:</p>
<div>
<pre> $ sudo apt-get install ssh
$ sudo apt-get install rsync</pre></div></div></div>
<div class="section">
<h3>Download<a name="Download"></a></h3>
<p>To get a Hadoop distribution, download a recent stable release from one of the Apache Download Mirrors.</p></div>
<div class="section">
<h3>Prepare to Start the Hadoop Cluster<a name="Prepare_to_Start_the_Hadoop_Cluster"></a></h3>
<p>Unpack the downloaded Hadoop distribution. In the distribution, edit the file <tt>conf/hadoop-env.sh</tt> to define at least <tt>JAVA_HOME</tt> to be the root of your Java installation.</p>
<p>Try the following command:</p>
<div>
<pre> $ bin/hadoop</pre></div>
<p>This will display the usage documentation for the hadoop script.</p>
<p>Now you are ready to start your Hadoop cluster in one of the three supported modes:</p>
<ul>
<li>Local (Standalone) Mode</li>
<li>Pseudo-Distributed Mode</li>
<li>Fully-Distributed Mode</li></ul></div>
<div class="section">
<h3>Standalone Operation<a name="Standalone_Operation"></a></h3>
<p>By default, Hadoop is configured to run in a non-distributed mode, as a single Java process. This is useful for debugging.</p>
<p>The following example copies the unpacked conf directory to use as input and then finds and displays every match of the given regular expression. Output is written to the given output directory.</p>
<div>
<pre> $ mkdir input
$ cp conf/*.xml input
$ bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'
$ cat output/*</pre></div></div>
<div class="section">
<h3>Pseudo-Distributed Operation<a name="Pseudo-Distributed_Operation"></a></h3>
<p>Hadoop can also be run on a single-node in a pseudo-distributed mode where each Hadoop daemon runs in a separate Java process.</p>
<div class="section">
<h4>Configuration<a name="Configuration"></a></h4>
<p>Use the following:</p>
<p>conf/core-site.xml:</p>
<div>
<pre>&lt;configuration&gt;
&lt;property&gt;
&lt;name&gt;fs.defaultFS&lt;/name&gt;
&lt;value&gt;hdfs://localhost:9000&lt;/value&gt;
&lt;/property&gt;
&lt;/configuration&gt;</pre></div>
<p>conf/hdfs-site.xml:</p>
<div>
<pre>&lt;configuration&gt;
&lt;property&gt;
&lt;name&gt;dfs.replication&lt;/name&gt;
&lt;value&gt;1&lt;/value&gt;
&lt;/property&gt;
&lt;/configuration&gt;</pre></div>
<p>conf/mapred-site.xml:</p>
<div>
<pre>&lt;configuration&gt;
&lt;property&gt;
&lt;name&gt;mapred.job.tracker&lt;/name&gt;
&lt;value&gt;localhost:9001&lt;/value&gt;
&lt;/property&gt;
&lt;/configuration&gt;</pre></div></div>
<div class="section">
<h4>Setup passphraseless ssh<a name="Setup_passphraseless_ssh"></a></h4>
<p>Now check that you can ssh to the localhost without a passphrase:</p>
<div>
<pre> $ ssh localhost</pre></div>
<p>If you cannot ssh to localhost without a passphrase, execute the following commands:</p>
<div>
<pre> $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub &gt;&gt; ~/.ssh/authorized_keys</pre></div></div>
<div class="section">
<h4>Execution<a name="Execution"></a></h4>
<p>Format a new distributed-filesystem:</p>
<div>
<pre> $ bin/hadoop namenode -format</pre></div>
<p>Start the hadoop daemons:</p>
<div>
<pre> $ bin/start-all.sh</pre></div>
<p>The hadoop daemon log output is written to the <tt>$<a name="HADOOP_LOG_DIR">HADOOP_LOG_DIR</a></tt> directory (defaults to <tt>$<a name="HADOOP_PREFIX">HADOOP_PREFIX</a>/logs</tt>).</p>
<p>Browse the web interface for the NameNode and the JobTracker; by default they are available at:</p>
<ul>
<li>NameNode - <tt>http://localhost:50070/</tt></li>
<li>JobTracker - <tt>http://localhost:50030/</tt></li></ul>
<p>Copy the input files into the distributed filesystem:</p>
<div>
<pre> $ bin/hadoop fs -put conf input</pre></div>
<p>Run some of the examples provided:</p>
<div>
<pre> $ bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'</pre></div>
<p>Examine the output files:</p>
<p>Copy the output files from the distributed filesystem to the local filesytem and examine them:</p>
<div>
<pre> $ bin/hadoop fs -get output output
$ cat output/*</pre></div>
<p>or</p>
<p>View the output files on the distributed filesystem:</p>
<div>
<pre> $ bin/hadoop fs -cat output/*</pre></div>
<p>When you're done, stop the daemons with:</p>
<div>
<pre> $ bin/stop-all.sh</pre></div></div></div>
<div class="section">
<h3>Fully-Distributed Operation<a name="Fully-Distributed_Operation"></a></h3>
<p>For information on setting up fully-distributed, non-trivial clusters see <a href="./ClusterSetup.html">Cluster Setup</a>.</p>
<p>Java and JNI are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries.</p></div></div>
</div>
</div>
<div class="clear">
<hr/>
</div>
<div id="footer">
<div class="xright">&#169; 2014
Apache Software Foundation
- <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a></div>
<div class="clear">
<hr/>
</div>
</div>
</body>
</html>

Event Timeline

c4science · Help