<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibilty between Hadoop 1.x and Hadoop 2.x</a>
<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
<img alt="Built by Maven" src="./images/logos/maven-feather.png"/>
</a>
</div>
</div>
<div id="bodyColumn">
<div id="contentBox">
<!-- Licensed under the Apache License, Version 2.0 (the "License"); --><!-- you may not use this file except in compliance with the License. --><!-- You may obtain a copy of the License at --><!-- --><!-- http://www.apache.org/licenses/LICENSE-2.0 --><!-- --><!-- Unless required by applicable law or agreed to in writing, software --><!-- distributed under the License is distributed on an "AS IS" BASIS, --><!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. --><!-- See the License for the specific language governing permissions and --><!-- limitations under the License. See accompanying LICENSE file. --><div class="section">
<td align="left">Used for commands, HTTP request and responses and code blocks.</td></tr>
<tr class="b">
<td align="left"><tt><Monospaced></tt></td>
<td align="left">User entered values.</td></tr>
<tr class="a">
<td align="left"><tt>[Monospaced]</tt></td>
<td align="left">Optional values. When the value is not specified, the default value is used.</td></tr>
<tr class="b">
<td align="left"><i>Italics</i></td>
<td align="left">Important phrases and words.</td></tr></table></div>
<div class="section">
<h3><a name="Introduction">Introduction</a></h3>
<p>The HTTP REST API supports the complete <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>/<a href="../../api/org/apache/hadoop/fs/FileContext.html">FileContext</a> interface for HDFS. The operations and the corresponding FileSystem/FileContext methods are shown in the next section. The Section <a href="#HTTP_Query_Parameter_Dictionary">HTTP Query Parameter Dictionary</a> specifies the parameter details such as the defaults and the valid values.</p>
<div class="section">
<h4><a name="Operations">Operations</a></h4>
<ul>
<li>HTTP GET
<ul>
<li><a href="#Open_and_Read_a_File"><tt>OPEN</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.open)</li>
<li><a href="#Status_of_a_FileDirectory"><tt>GETFILESTATUS</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.getFileStatus)</li>
<li><a href="#List_a_Directory"><tt>LISTSTATUS</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.listStatus)</li>
<li><a href="#Get_Content_Summary_of_a_Directory"><tt>GETCONTENTSUMMARY</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.getContentSummary)</li>
<li><a href="#Get_File_Checksum"><tt>GETFILECHECKSUM</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.getFileChecksum)</li>
<li><a href="#Get_Home_Directory"><tt>GETHOMEDIRECTORY</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.getHomeDirectory)</li>
<li><a href="#Get_Delegation_Token"><tt>GETDELEGATIONTOKEN</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.getDelegationToken)</li>
<li><a href="#Get_Delegation_Tokens"><tt>GETDELEGATIONTOKENS</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.getDelegationTokens)</li></ul></li>
<li>HTTP PUT
<ul>
<li><a href="#Create_and_Write_to_a_File"><tt>CREATE</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.create)</li>
<li><a href="#Make_a_Directory"><tt>MKDIRS</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.mkdirs)</li>
<li><a href="#Create_a_Symbolic_Link"><tt>CREATESYMLINK</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileContext.html">FileContext</a>.createSymlink)</li>
<li><a href="#Rename_a_FileDirectory"><tt>RENAME</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.rename)</li>
<li><a href="#Set_Replication_Factor"><tt>SETREPLICATION</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.setReplication)</li>
<li><a href="#Set_Owner"><tt>SETOWNER</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.setOwner)</li>
<li><a href="#Set_Permission"><tt>SETPERMISSION</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.setPermission)</li>
<li><a href="#Set_Access_or_Modification_Time"><tt>SETTIMES</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.setTimes)</li>
<li><a href="#Renew_Delegation_Token"><tt>RENEWDELEGATIONTOKEN</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.renewDelegationToken)</li>
<li><a href="#Cancel_Delegation_Token"><tt>CANCELDELEGATIONTOKEN</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.cancelDelegationToken)</li></ul></li>
<li>HTTP POST
<ul>
<li><a href="#Append_to_a_File"><tt>APPEND</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.append)</li>
<li><a href="#Concat_Files"><tt>CONCAT</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.concat)</li></ul></li>
<li>HTTP DELETE
<ul>
<li><a href="#Delete_a_FileDirectory"><tt>DELETE</tt></a> (see <a href="../../api/org/apache/hadoop/fs/FileSystem.html">FileSystem</a>.delete)</li></ul></li></ul></div>
<div class="section">
<h4><a name="FileSystem_URIs_vs_HTTP_URLs">FileSystem URIs vs HTTP URLs</a></h4>
<p>The FileSystem scheme of WebHDFS is "<tt>webhdfs://</tt>". A WebHDFS FileSystem URI has the following format.</p>
<p>In the REST API, the prefix "<tt>/webhdfs/v1</tt>" is inserted in the path and a query is appended at the end. Therefore, the corresponding HTTP URL has the following format.</p>
<td align="left">The HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint. The HTTP Kerberos principal MUST start with 'HTTP/' per Kerberos HTTP SPNEGO specification.</td></tr>
<td align="left">The Kerberos keytab file with the credentials for the HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint.</td></tr></table></div></div>
<p>When security is <i>off</i>, the authenticated user is the username specified in the <tt>user.name</tt> query parameter. If the <tt>user.name</tt> parameter is not set, the server may either set the authenticated user to a default web user, if there is any, or return an error response.</p>
<p>When security is <i>on</i>, authentication is performed by either Hadoop delegation token or Kerberos SPNEGO. If a token is set in the <tt>delegation</tt> query parameter, the authenticated user is the user encoded in the token. If the <tt>delegation</tt> parameter is not set, the user is authenticated by Kerberos SPNEGO.</p>
<p>Below are examples using the <tt>curl</tt> command tool.</p>
<p>See also: <a href="../hadoop-common/HttpAuthentication.html">Authentication for Hadoop HTTP web-consoles</a></p></div>
<div class="section">
<h3><a name="Proxy_Users">Proxy Users</a></h3>
<p>When the proxy user feature is enabled, a proxy user <i>P</i> may submit a request on behalf of another user <i>U</i>. The username of <i>U</i> must be specified in the <tt>doas</tt> query parameter unless a delegation token is presented in authentication. In such case, the information of both users <i>P</i> and <i>U</i> must be encoded in the delegation token.</p>
<p><b>Note</b> that the reason of having two-step create/append is for preventing clients to send out data before the redirect. This issue is addressed by the "<tt>Expect: 100-continue</tt>" header in HTTP/1.1; see <a class="externalLink" href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.2.3">RFC 2616, Section 8.2.3</a>. Unfortunately, there are software library bugs (e.g. Jetty 6 HTTP server and Java 6 HTTP client), which do not correctly implement "<tt>Expect: 100-continue</tt>". The two-step create/append is a temporary workaround for the software library bugs.</p>
<p>When an operation fails, the server may throw an exception. The JSON schema of error responses is defined in <a href="#RemoteException_JSON_Schema">RemoteException JSON Schema</a>. The table below shows the mapping from exceptions to HTTP response codes.</p>
"message" : "File does not exist: /foo/a.patch"
}
}</pre></div></div></div></div>
<div class="section">
<h3><a name="JSON_Schemas">JSON Schemas</a></h3>
<p>All operations, except for <a href="#Open_and_Read_a_File"><tt>OPEN</tt></a>, either return a zero-length response or a JSON response. For <a href="#Open_and_Read_a_File"><tt>OPEN</tt></a>, the response is an octet-stream. The JSON schemas are shown below. See <a class="externalLink" href="http://tools.ietf.org/id/draft-zyp-json-schema-03.html">draft-zyp-json-schema-03</a> for the syntax definitions of the JSON schemas.</p>
<p><b>Note</b> that the default value of <a class="externalLink" href="http://tools.ietf.org/id/draft-zyp-json-schema-03.html#additionalProperties"><tt>additionalProperties</tt></a> is an empty schema which allows any value for additional properties. Therefore, all WebHDFS JSON responses allow any additional property. However, if additional properties are included in the responses, they are considered as optional properties in order to maintain compatibility.</p>
<p>JavaScript syntax is used to define <tt>fileStatusProperties</tt> so that it can be referred in both <tt>FileStatus</tt> and <tt>FileStatuses</tt> JSON schemas.</p>
<p>JavaScript syntax is used to define <tt>tokenProperties</tt> so that it can be referred in both <tt>Token</tt> and <tt>Tokens</tt> JSON schemas.</p>
<div class="source">
<pre>var tokenProperties =
{
"type" : "object",
"properties":
{
"urlString":
{
"description": "A delegation token encoded as a URL safe string.",
<td align="left">The delegation token used for authentication.</td></tr>
<tr class="a">
<th align="left">Type</th>
<td align="left">String</td></tr>
<tr class="b">
<th align="left">Default Value</th>
<td align="left"><empty></td></tr>
<tr class="a">
<th align="left">Valid Values</th>
<td align="left">An encoded token.</td></tr>
<tr class="b">
<th align="left">Syntax</th>
<td align="left">See the note below.</td></tr></table>
<p><b>Note</b> that delegation tokens are encoded as a URL safe string; see <tt>encodeToUrlString()</tt> and <tt>decodeFromUrlString(String)</tt> in <tt>org.apache.hadoop.security.token.Token</tt> for the details of the encoding.</p>