Homec4science
Diffusion elastic-yarn (future) Browse

R3704/hadoop-2.3.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
2e9ad393f3f8future

Last Change
Raw File

HistoryServerRest.apt.vm
View Options

~~ Licensed under the Apache License, Version 2.0 (the "License");
~~ you may not use this file except in compliance with the License.
~~ You may obtain a copy of the License at
~~
~~ http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing, software
~~ distributed under the License is distributed on an "AS IS" BASIS,
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~~ See the License for the specific language governing permissions and
~~ limitations under the License. See accompanying LICENSE file.
---
History Server REST API's.
---
---
${maven.build.timestamp}
History Server REST API's.
\[ {{{./index.html}Go Back}} \]
%{toc|section=1|fromDepth=0|toDepth=3}
* Overview
The history server REST API's allow the user to get status on finished applications. Currently it only supports MapReduce and provides information on finished jobs.
* History Server Information API
The history server information resource provides overall information about the history server.
** URI
Both of the following URI's give you the history server information, from an application id identified by the appid value.
------
* http://<history server http address:port>/ws/v1/history
* http://<history server http address:port>/ws/v1/history/info
------
** HTTP Operations Supported
------
* GET
------
** Query Parameters Supported
------
None
------
** Elements of the <historyInfo> object
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| startedOn | long | The time the history server was started (in ms since epoch)|
*---------------+--------------+-------------------------------+
| hadoopVersion | string | Version of hadoop common |
*---------------+--------------+-------------------------------+
| hadoopBuildVersion | string | Hadoop common build string with build version, user, and checksum |
*---------------+--------------+-------------------------------+
| hadoopVersionBuiltOn | string | Timestamp when hadoop common was built |
*---------------+--------------+-------------------------------+
** Response Examples
<<JSON response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/info
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
{
"historyInfo" : {
"startedOn":1353512830963,
"hadoopVersionBuiltOn" : "Wed Jan 11 21:18:36 UTC 2012",
"hadoopBuildVersion" : "0.23.1-SNAPSHOT from 1230253 by user1 source checksum bb6e554c6d50b0397d826081017437a7",
"hadoopVersion" : "0.23.1-SNAPSHOT"
}
}
+---+
<<XML response>>
HTTP Request:
-----
GET http://<history server http address:port>/ws/v1/history/info
Accept: application/xml
-----
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 330
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<historyInfo>
<startedOn>1353512830963</startedOn>
<hadoopVersion>0.23.1-SNAPSHOT</hadoopVersion>
<hadoopBuildVersion>0.23.1-SNAPSHOT from 1230253 by user1 source checksum bb6e554c6d50b0397d826081017437a7</hadoopBuildVersion>
<hadoopVersionBuiltOn>Wed Jan 11 21:18:36 UTC 2012</hadoopVersionBuiltOn>
</historyInfo>
+---+
* MapReduce API's
The following list of resources apply to MapReduce.
** Jobs API
The jobs resource provides a list of the MapReduce jobs that have finished. It does not currently return a full list of parameters
*** URI
------
* http://<history server http address:port>/ws/v1/history/mapreduce/jobs
------
*** HTTP Operations Supported
------
* GET
------
*** Query Parameters Supported
Multiple paramters can be specified. The started and finished times have a begin and end parameter to allow you to specify ranges. For example, one could request all jobs that started between 1:00am and 2:00pm on 12/19/2011 with startedTimeBegin=1324256400&startedTimeEnd=1324303200. If the Begin parameter is not specfied, it defaults to 0, and if the End parameter is not specified, it defaults to infinity.
------
* user - user name
* state - the job state
* queue - queue name
* limit - total number of app objects to be returned
* startedTimeBegin - jobs with start time beginning with this time, specified in ms since epoch
* startedTimeEnd - jobs with start time ending with this time, specified in ms since epoch
* finishedTimeBegin - jobs with finish time beginning with this time, specified in ms since epoch
* finishedTimeEnd - jobs with finish time ending with this time, specified in ms since epoch
------
*** Elements of the <jobs> object
When you make a request for the list of jobs, the information will be returned as an array of job objects.
See also {{Job API}} for syntax of the job object. Except this is a subset of a full job. Only startTime,
finishTime, id, name, queue, user, state, mapsTotal, mapsCompleted, reducesTotal, and reducesCompleted are
returned.
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| job | array of job objects(json)/zero or more job objects(XML) | The collection of job objects |
*---------------+--------------+-------------------------------+
*** Response Examples
<<JSON response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
{
"jobs" : {
"job" : [
{
"submitTime" : 1326381344449,
"state" : "SUCCEEDED",
"user" : "user1",
"reducesTotal" : 1,
"mapsCompleted" : 1,
"startTime" : 1326381344489,
"id" : "job_1326381300833_1_1",
"name" : "word count",
"reducesCompleted" : 1,
"mapsTotal" : 1,
"queue" : "default",
"finishTime" : 1326381356010
},
{
"submitTime" : 1326381446500
"state" : "SUCCEEDED",
"user" : "user1",
"reducesTotal" : 1,
"mapsCompleted" : 1,
"startTime" : 1326381446529,
"id" : "job_1326381300833_2_2",
"name" : "Sleep job",
"reducesCompleted" : 1,
"mapsTotal" : 1,
"queue" : "default",
"finishTime" : 1326381582106
}
]
}
}
+---+
<<XML response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs
Accept: application/xml
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 1922
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<jobs>
<job>
<submitTime>1326381344449</submitTime>
<startTime>1326381344489</startTime>
<finishTime>1326381356010</finishTime>
<id>job_1326381300833_1_1</id>
<name>word count</name>
<queue>default</queue>
<user>user1</user>
<state>SUCCEEDED</state>
<mapsTotal>1</mapsTotal>
<mapsCompleted>1</mapsCompleted>
<reducesTotal>1</reducesTotal>
<reducesCompleted>1</reducesCompleted>
</job>
<job>
<submitTime>1326381446500</submitTime>
<startTime>1326381446529</startTime>
<finishTime>1326381582106</finishTime>
<id>job_1326381300833_2_2</id>
<name>Sleep job</name>
<queue>default</queue>
<user>user1</user>
<state>SUCCEEDED</state>
<mapsTotal>1</mapsTotal>
<mapsCompleted>1</mapsCompleted>
<reducesTotal>1</reducesTotal>
<reducesCompleted>1</reducesCompleted>
</job>
</jobs>
+---+
** {Job API}
A Job resource contains information about a particular job identified by {jobid}.
*** URI
------
* http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}
------
*** HTTP Operations Supported
------
* GET
------
*** Query Parameters Supported
------
None
------
*** Elements of the <job> object
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| id | string | The job id|
*---------------+--------------+-------------------------------+
| name | string | The job name |
*---------------+--------------+-------------------------------+
| queue | string | The queue the job was submitted to|
*---------------+--------------+-------------------------------+
| user | string | The user name |
*---------------+--------------+-------------------------------+
| state | string | the job state - valid values are: NEW, INITED, RUNNING, SUCCEEDED, FAILED, KILL_WAIT, KILLED, ERROR|
*---------------+--------------+-------------------------------+
| diagnostics | string | A diagnostic message |
*---------------+--------------+-------------------------------+
| submitTime | long | The time the job submitted (in ms since epoch)|
*---------------+--------------+-------------------------------+
| startTime | long | The time the job started (in ms since epoch)|
*---------------+--------------+-------------------------------+
| finishTime | long | The time the job finished (in ms since epoch)|
*---------------+--------------+-------------------------------+
| mapsTotal | int | The total number of maps |
*---------------+--------------+-------------------------------+
| mapsCompleted | int | The number of completed maps |
*---------------+--------------+-------------------------------+
| reducesTotal | int | The total number of reduces |
*---------------+--------------+-------------------------------+
| reducesCompleted | int | The number of completed reduces|
*---------------+--------------+-------------------------------+
| uberized | boolean | Indicates if the job was an uber job - ran completely in the application master|
*---------------+--------------+-------------------------------+
| avgMapTime | long | The average time of a map task (in ms)|
*---------------+--------------+-------------------------------+
| avgReduceTime | long | The average time of the reduce (in ms)|
*---------------+--------------+-------------------------------+
| avgShuffleTime | long | The average time of the shuffle (in ms)|
*---------------+--------------+-------------------------------+
| avgMergeTime | long | The average time of the merge (in ms)|
*---------------+--------------+-------------------------------+
| failedReduceAttempts | int | The number of failed reduce attempts |
*---------------+--------------+-------------------------------+
| killedReduceAttempts | int | The number of killed reduce attempts |
*---------------+--------------+-------------------------------+
| successfulReduceAttempts | int | The number of successful reduce attempts |
*---------------+--------------+-------------------------------+
| failedMapAttempts | int | The number of failed map attempts |
*---------------+--------------+-------------------------------+
| killedMapAttempts | int | The number of killed map attempts |
*---------------+--------------+-------------------------------+
| successfulMapAttempts | int | The number of successful map attempts |
*---------------+--------------+-------------------------------+
| acls | array of acls(json)/zero or more acls objects(xml)| A collection of acls objects |
*---------------+--------------+-------------------------------+
** Elements of the <acls> object
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| value | string | The acl value|
*---------------+--------------+-------------------------------+
| name | string | The acl name |
*---------------+--------------+-------------------------------+
*** Response Examples
<<JSON response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/json
Server: Jetty(6.1.26)
Content-Length: 720
+---+
Response Body:
+---+
{
"job" : {
"submitTime": 1326381446500,
"avgReduceTime" : 124961,
"failedReduceAttempts" : 0,
"state" : "SUCCEEDED",
"successfulReduceAttempts" : 1,
"acls" : [
{
"value" : " ",
"name" : "mapreduce.job.acl-modify-job"
},
{
"value" : " ",
"name" : "mapreduce.job.acl-view-job"
}
],
"user" : "user1",
"reducesTotal" : 1,
"mapsCompleted" : 1,
"startTime" : 1326381446529,
"id" : "job_1326381300833_2_2",
"avgMapTime" : 2638,
"successfulMapAttempts" : 1,
"name" : "Sleep job",
"avgShuffleTime" : 2540,
"reducesCompleted" : 1,
"diagnostics" : "",
"failedMapAttempts" : 0,
"avgMergeTime" : 2589,
"killedReduceAttempts" : 0,
"mapsTotal" : 1,
"queue" : "default",
"uberized" : false,
"killedMapAttempts" : 0,
"finishTime" : 1326381582106
}
}
+---+
<<XML response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2
Accept: application/xml
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 983
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<job>
<submitTime>1326381446500</submitTime>
<startTime>1326381446529</startTime>
<finishTime>1326381582106</finishTime>
<id>job_1326381300833_2_2</id>
<name>Sleep job</name>
<queue>default</queue>
<user>user1</user>
<state>SUCCEEDED</state>
<mapsTotal>1</mapsTotal>
<mapsCompleted>1</mapsCompleted>
<reducesTotal>1</reducesTotal>
<reducesCompleted>1</reducesCompleted>
<uberized>false</uberized>
<diagnostics/>
<avgMapTime>2638</avgMapTime>
<avgReduceTime>124961</avgReduceTime>
<avgShuffleTime>2540</avgShuffleTime>
<avgMergeTime>2589</avgMergeTime>
<failedReduceAttempts>0</failedReduceAttempts>
<killedReduceAttempts>0</killedReduceAttempts>
<successfulReduceAttempts>1</successfulReduceAttempts>
<failedMapAttempts>0</failedMapAttempts>
<killedMapAttempts>0</killedMapAttempts>
<successfulMapAttempts>1</successfulMapAttempts>
<acls>
<name>mapreduce.job.acl-modify-job</name>
<value> </value>
</acls>
<acls>
<name>mapreduce.job.acl-view-job</name>
<value> </value>
</acls>
</job>
+---+
** Job Attempts API
With the job attempts API, you can obtain a collection of resources that represent a job attempt. When you run a GET operation on this resource, you obtain a collection of Job Attempt Objects.
*** URI
------
* http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/jobattempts
------
*** HTTP Operations Supported
------
* GET
------
*** Query Parameters Supported
------
None
------
*** Elements of the <jobAttempts> object
When you make a request for the list of job attempts, the information will be returned as an array of job attempt objects.
jobAttempts:
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| jobAttempt | array of job attempt objects(JSON)/zero or more job attempt objects(XML) | The collection of job attempt objects |
*---------------+--------------+--------------------------------+
*** Elements of the <jobAttempt> object
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| id | string | The job attempt id |
*---------------+--------------+--------------------------------+
| nodeId | string | The node id of the node the attempt ran on|
*---------------+--------------+--------------------------------+
| nodeHttpAddress | string | The node http address of the node the attempt ran on|
*---------------+--------------+--------------------------------+
| logsLink | string | The http link to the job attempt logs |
*---------------+--------------+--------------------------------+
| containerId | string | The id of the container for the job attempt |
*---------------+--------------+--------------------------------+
| startTime | long | The start time of the attempt (in ms since epoch)|
*---------------+--------------+--------------------------------+
*** Response Examples
<<JSON response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/jobattempts
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
{
"jobAttempts" : {
"jobAttempt" : [
{
"nodeId" : "host.domain.com:8041",
"nodeHttpAddress" : "host.domain.com:8042",
"startTime" : 1326381444693,
"id" : 1,
"logsLink" : "http://host.domain.com:19888/jobhistory/logs/host.domain.com:8041/container_1326381300833_0002_01_000001/job_1326381300833_2_2/user1",
"containerId" : "container_1326381300833_0002_01_000001"
}
]
}
}
+---+
<<XML response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/jobattmpts
Accept: application/xml
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 575
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<jobAttempts>
<jobAttempt>
<nodeHttpAddress>host.domain.com:8042</nodeHttpAddress>
<nodeId>host.domain.com:8041</nodeId>
<id>1</id>
<startTime>1326381444693</startTime>
<containerId>container_1326381300833_0002_01_000001</containerId>
<logsLink>http://host.domain.com:19888/jobhistory/logs/host.domain.com:8041/container_1326381300833_0002_01_000001/job_1326381300833_2_2/user1</logsLink>
</jobAttempt>
</jobAttempts>
+---+
** Job Counters API
With the job counters API, you can object a collection of resources that represent al the counters for that job.
*** URI
------
* http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/counters
------
*** HTTP Operations Supported
------
* GET
------
*** Query Parameters Supported
------
None
------
*** Elements of the <jobCounters> object
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| id | string | The job id |
*---------------+--------------+-------------------------------+
| counterGroup | array of counterGroup objects(JSON)/zero or more counterGroup objects(XML) | A collection of counter group objects |
*---------------+--------------+-------------------------------+
*** Elements of the <counterGroup> objecs
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| counterGroupName | string | The name of the counter group |
*---------------+--------------+-------------------------------+
| counter | array of counter objects(JSON)/zero or more counter objects(XML) | A collection of counter objects |
*---------------+--------------+-------------------------------+
*** Elements of the <counter> object
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| name | string | The name of the counter |
*---------------+--------------+-------------------------------+
| reduceCounterValue | long | The counter value of reduce tasks |
*---------------+--------------+-------------------------------+
| mapCounterValue | long | The counter value of map tasks |
*---------------+--------------+-------------------------------+
| totalCounterValue | long | The counter value of all tasks |
*---------------+--------------+-------------------------------+
*** Response Examples
<<JSON response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/counters
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
{
"jobCounters" : {
"id" : "job_1326381300833_2_2",
"counterGroup" : [
{
"counterGroupName" : "Shuffle Errors",
"counter" : [
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "BAD_ID"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "CONNECTION"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "IO_ERROR"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "WRONG_LENGTH"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "WRONG_MAP"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "WRONG_REDUCE"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.FileSystemCounter",
"counter" : [
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 2483,
"name" : "FILE_BYTES_READ"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 108525,
"name" : "FILE_BYTES_WRITTEN"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "FILE_READ_OPS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "FILE_LARGE_READ_OPS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "FILE_WRITE_OPS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 48,
"name" : "HDFS_BYTES_READ"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "HDFS_BYTES_WRITTEN"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 1,
"name" : "HDFS_READ_OPS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "HDFS_LARGE_READ_OPS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "HDFS_WRITE_OPS"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.TaskCounter",
"counter" : [
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 1,
"name" : "MAP_INPUT_RECORDS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 1200,
"name" : "MAP_OUTPUT_RECORDS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 4800,
"name" : "MAP_OUTPUT_BYTES"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 2235,
"name" : "MAP_OUTPUT_MATERIALIZED_BYTES"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 48,
"name" : "SPLIT_RAW_BYTES"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "COMBINE_INPUT_RECORDS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "COMBINE_OUTPUT_RECORDS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 1200,
"name" : "REDUCE_INPUT_GROUPS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 2235,
"name" : "REDUCE_SHUFFLE_BYTES"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 1200,
"name" : "REDUCE_INPUT_RECORDS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "REDUCE_OUTPUT_RECORDS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 2400,
"name" : "SPILLED_RECORDS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 1,
"name" : "SHUFFLED_MAPS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "FAILED_SHUFFLE"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 1,
"name" : "MERGED_MAP_OUTPUTS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 113,
"name" : "GC_TIME_MILLIS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 1830,
"name" : "CPU_MILLISECONDS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 478068736,
"name" : "PHYSICAL_MEMORY_BYTES"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 2159284224,
"name" : "VIRTUAL_MEMORY_BYTES"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 378863616,
"name" : "COMMITTED_HEAP_BYTES"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter",
"counter" : [
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "BYTES_READ"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter",
"counter" : [
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "BYTES_WRITTEN"
}
]
}
]
}
}
+---+
<<XML response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/counters
Accept: application/xml
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 7030
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<jobCounters>
<id>job_1326381300833_2_2</id>
<counterGroup>
<counterGroupName>Shuffle Errors</counterGroupName>
<counter>
<name>BAD_ID</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>CONNECTION</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>IO_ERROR</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>WRONG_LENGTH</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>WRONG_MAP</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>WRONG_REDUCE</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
</counterGroup>
<counterGroup>
<counterGroupName>org.apache.hadoop.mapreduce.FileSystemCounter</counterGroupName>
<counter>
<name>FILE_BYTES_READ</name>
<totalCounterValue>2483</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>FILE_BYTES_WRITTEN</name>
<totalCounterValue>108525</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>FILE_READ_OPS</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>FILE_LARGE_READ_OPS</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>FILE_WRITE_OPS</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>HDFS_BYTES_READ</name>
<totalCounterValue>48</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>HDFS_BYTES_WRITTEN</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>HDFS_READ_OPS</name>
<totalCounterValue>1</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>HDFS_LARGE_READ_OPS</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>HDFS_WRITE_OPS</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
</counterGroup>
<counterGroup>
<counterGroupName>org.apache.hadoop.mapreduce.TaskCounter</counterGroupName>
<counter>
<name>MAP_INPUT_RECORDS</name>
<totalCounterValue>1</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>MAP_OUTPUT_RECORDS</name>
<totalCounterValue>1200</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>MAP_OUTPUT_BYTES</name>
<totalCounterValue>4800</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>MAP_OUTPUT_MATERIALIZED_BYTES</name>
<totalCounterValue>2235</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>SPLIT_RAW_BYTES</name>
<totalCounterValue>48</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>COMBINE_INPUT_RECORDS</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>COMBINE_OUTPUT_RECORDS</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>REDUCE_INPUT_GROUPS</name>
<totalCounterValue>1200</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>REDUCE_SHUFFLE_BYTES</name>
<totalCounterValue>2235</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>REDUCE_INPUT_RECORDS</name>
<totalCounterValue>1200</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>REDUCE_OUTPUT_RECORDS</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>SPILLED_RECORDS</name>
<totalCounterValue>2400</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>SHUFFLED_MAPS</name>
<totalCounterValue>1</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>FAILED_SHUFFLE</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>MERGED_MAP_OUTPUTS</name>
<totalCounterValue>1</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>GC_TIME_MILLIS</name>
<totalCounterValue>113</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>CPU_MILLISECONDS</name>
<totalCounterValue>1830</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>PHYSICAL_MEMORY_BYTES</name>
<totalCounterValue>478068736</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>VIRTUAL_MEMORY_BYTES</name>
<totalCounterValue>2159284224</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
<counter>
<name>COMMITTED_HEAP_BYTES</name>
<totalCounterValue>378863616</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
</counterGroup>
<counterGroup>
<counterGroupName>org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter</counterGroupName>
<counter>
<name>BYTES_READ</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
</counterGroup>
<counterGroup>
<counterGroupName>org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter</counterGroupName>
<counter>
<name>BYTES_WRITTEN</name>
<totalCounterValue>0</totalCounterValue>
<mapCounterValue>0</mapCounterValue>
<reduceCounterValue>0</reduceCounterValue>
</counter>
</counterGroup>
</jobCounters>
+---+
** Job Conf API
A job configuration resource contains information about the job configuration for this job.
*** URI
Use the following URI to obtain th job configuration information, from a job identified by the {jobid} value.
------
* http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/conf
------
*** HTTP Operations Supported
------
* GET
------
*** Query Parameters Supported
------
None
------
*** Elements of the <conf> object
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| path | string | The path to the job configuration file|
*---------------+--------------+-------------------------------+
| property | array of the configuration properties(JSON)/zero or more configuration properties(XML) | Collection of configuration property objects|
*---------------+--------------+-------------------------------+
*** Elements of the <property> object
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| name | string | The name of the configuration property |
*---------------+--------------+-------------------------------+
| value | string | The value of the configuration property |
*---------------+--------------+-------------------------------+
| source | string | The location this configuration object came from. If there is more then one of these it shows the history with the latest source at the end of the list. |
*---------------+--------------+-------------------------------+
*** Response Examples
<<JSON response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/conf
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
+---+
Response Body:
This is a small snippet of the output as the output if very large. The real output contains every property in your job configuration file.
+---+
{
"conf" : {
"path" : "hdfs://host.domain.com:9000/user/user1/.staging/job_1326381300833_0002/job.xml",
"property" : [
{
"value" : "/home/hadoop/hdfs/data",
"name" : "dfs.datanode.data.dir"
"source" : ["hdfs-site.xml", "job.xml"]
},
{
"value" : "org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer",
"name" : "hadoop.http.filter.initializers"
"source" : ["programatically", "job.xml"]
},
{
"value" : "/home/hadoop/tmp",
"name" : "mapreduce.cluster.temp.dir"
"source" : ["mapred-site.xml"]
},
...
]
}
}
+---+
<<XML response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/conf
Accept: application/xml
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 552
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<conf>
<path>hdfs://host.domain.com:9000/user/user1/.staging/job_1326381300833_0002/job.xml</path>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/hdfs/data</value>
<source>hdfs-site.xml</source>
<source>job.xml</source>
</property>
<property>
<name>hadoop.http.filter.initializers</name>
<value>org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer</value>
<source>programatically</source>
<source>job.xml</source>
</property>
<property>
<name>mapreduce.cluster.temp.dir</name>
<value>/home/hadoop/tmp</value>
<source>mapred-site.xml</source>
</property>
...
</conf>
+---+
** Tasks API
With the tasks API, you can obtain a collection of resources that represent a task within a job. When you run a GET operation on this resource, you obtain a collection of Task Objects.
*** URI
------
* http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/tasks
------
*** HTTP Operations Supported
------
* GET
------
*** Query Parameters Supported
------
* type - type of task, valid values are m or r. m for map task or r for reduce task.
------
*** Elements of the <tasks> object
When you make a request for the list of tasks , the information will be returned as an array of task objects.
See also {{Task API}} for syntax of the task object.
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| task | array of task objects(JSON)/zero or more task objects(XML) | The collection of task objects. |
*---------------+--------------+--------------------------------+
*** Response Examples
<<JSON response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
{
"tasks" : {
"task" : [
{
"progress" : 100,
"elapsedTime" : 6777,
"state" : "SUCCEEDED",
"startTime" : 1326381446541,
"id" : "task_1326381300833_2_2_m_0",
"type" : "MAP",
"successfulAttempt" : "attempt_1326381300833_2_2_m_0_0",
"finishTime" : 1326381453318
},
{
"progress" : 100,
"elapsedTime" : 135559,
"state" : "SUCCEEDED",
"startTime" : 1326381446544,
"id" : "task_1326381300833_2_2_r_0",
"type" : "REDUCE",
"successfulAttempt" : "attempt_1326381300833_2_2_r_0_0",
"finishTime" : 1326381582103
}
]
}
}
+---+
<<XML response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks
Accept: application/xml
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 653
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<tasks>
<task>
<startTime>1326381446541</startTime>
<finishTime>1326381453318</finishTime>
<elapsedTime>6777</elapsedTime>
<progress>100.0</progress>
<id>task_1326381300833_2_2_m_0</id>
<state>SUCCEEDED</state>
<type>MAP</type>
<successfulAttempt>attempt_1326381300833_2_2_m_0_0</successfulAttempt>
</task>
<task>
<startTime>1326381446544</startTime>
<finishTime>1326381582103</finishTime>
<elapsedTime>135559</elapsedTime>
<progress>100.0</progress>
<id>task_1326381300833_2_2_r_0</id>
<state>SUCCEEDED</state>
<type>REDUCE</type>
<successfulAttempt>attempt_1326381300833_2_2_r_0_0</successfulAttempt>
</task>
</tasks>
+---+
** {Task API}
A Task resource contains information about a particular task within a job.
*** URI
Use the following URI to obtain an Task Object, from a task identified by the {taskid} value.
------
* http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/tasks/{taskid}
------
*** HTTP Operations Supported
------
* GET
------
*** Query Parameters Supported
------
None
------
*** Elements of the <task> object
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| id | string | The task id |
*---------------+--------------+--------------------------------+
| state | string | The state of the task - valid values are: NEW, SCHEDULED, RUNNING, SUCCEEDED, FAILED, KILL_WAIT, KILLED
*---------------+--------------+--------------------------------+
| type | string | The task type - MAP or REDUCE|
*---------------+--------------+--------------------------------+
| successfulAttempt | string | The id of the last successful attempt |
*---------------+--------------+--------------------------------+
| progress | float | The progress of the task as a percent|
*---------------+--------------+--------------------------------+
| startTime | long | The time in which the task started (in ms since epoch) or -1 if it was never started |
*---------------+--------------+--------------------------------+
| finishTime | long | The time in which the task finished (in ms since epoch)|
*---------------+--------------+--------------------------------+
| elapsedTime | long | The elapsed time since the application started (in ms)|
*---------------+--------------+--------------------------------+
*** Response Examples
<<JSON response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
{
"task" : {
"progress" : 100,
"elapsedTime" : 6777,
"state" : "SUCCEEDED",
"startTime" : 1326381446541,
"id" : "task_1326381300833_2_2_m_0",
"type" : "MAP",
"successfulAttempt" : "attempt_1326381300833_2_2_m_0_0",
"finishTime" : 1326381453318
}
}
+---+
<<XML response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0
Accept: application/xml
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 299
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<task>
<startTime>1326381446541</startTime>
<finishTime>1326381453318</finishTime>
<elapsedTime>6777</elapsedTime>
<progress>100.0</progress>
<id>task_1326381300833_2_2_m_0</id>
<state>SUCCEEDED</state>
<type>MAP</type>
<successfulAttempt>attempt_1326381300833_2_2_m_0_0</successfulAttempt>
</task>
+---+
** Task Counters API
With the task counters API, you can object a collection of resources that represent all the counters for that task.
*** URI
------
* http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/tasks/{taskid}/counters
------
*** HTTP Operations Supported
------
* GET
------
*** Query Parameters Supported
------
None
------
*** Elements of the <jobTaskCounters> object
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| id | string | The task id |
*---------------+--------------+-------------------------------+
| taskcounterGroup | array of counterGroup objects(JSON)/zero or more counterGroup objects(XML) | A collection of counter group objects |
*---------------+--------------+-------------------------------+
*** Elements of the <counterGroup> object
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| counterGroupName | string | The name of the counter group |
*---------------+--------------+-------------------------------+
| counter | array of counter objects(JSON)/zero or more counter objects(XML) | A collection of counter objects |
*---------------+--------------+-------------------------------+
*** Elements of the <counter> object
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| name | string | The name of the counter |
*---------------+--------------+-------------------------------+
| value | long | The value of the counter |
*---------------+--------------+-------------------------------+
*** Response Examples
<<JSON response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/counters
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
{
"jobTaskCounters" : {
"id" : "task_1326381300833_2_2_m_0",
"taskCounterGroup" : [
{
"counterGroupName" : "org.apache.hadoop.mapreduce.FileSystemCounter",
"counter" : [
{
"value" : 2363,
"name" : "FILE_BYTES_READ"
},
{
"value" : 54372,
"name" : "FILE_BYTES_WRITTEN"
},
{
"value" : 0,
"name" : "FILE_READ_OPS"
},
{
"value" : 0,
"name" : "FILE_LARGE_READ_OPS"
},
{
"value" : 0,
"name" : "FILE_WRITE_OPS"
},
{
"value" : 0,
"name" : "HDFS_BYTES_READ"
},
{
"value" : 0,
"name" : "HDFS_BYTES_WRITTEN"
},
{
"value" : 0,
"name" : "HDFS_READ_OPS"
},
{
"value" : 0,
"name" : "HDFS_LARGE_READ_OPS"
},
{
"value" : 0,
"name" : "HDFS_WRITE_OPS"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.TaskCounter",
"counter" : [
{
"value" : 0,
"name" : "COMBINE_INPUT_RECORDS"
},
{
"value" : 0,
"name" : "COMBINE_OUTPUT_RECORDS"
},
{
"value" : 460,
"name" : "REDUCE_INPUT_GROUPS"
},
{
"value" : 2235,
"name" : "REDUCE_SHUFFLE_BYTES"
},
{
"value" : 460,
"name" : "REDUCE_INPUT_RECORDS"
},
{
"value" : 0,
"name" : "REDUCE_OUTPUT_RECORDS"
},
{
"value" : 0,
"name" : "SPILLED_RECORDS"
},
{
"value" : 1,
"name" : "SHUFFLED_MAPS"
},
{
"value" : 0,
"name" : "FAILED_SHUFFLE"
},
{
"value" : 1,
"name" : "MERGED_MAP_OUTPUTS"
},
{
"value" : 26,
"name" : "GC_TIME_MILLIS"
},
{
"value" : 860,
"name" : "CPU_MILLISECONDS"
},
{
"value" : 107839488,
"name" : "PHYSICAL_MEMORY_BYTES"
},
{
"value" : 1123147776,
"name" : "VIRTUAL_MEMORY_BYTES"
},
{
"value" : 57475072,
"name" : "COMMITTED_HEAP_BYTES"
}
]
},
{
"counterGroupName" : "Shuffle Errors",
"counter" : [
{
"value" : 0,
"name" : "BAD_ID"
},
{
"value" : 0,
"name" : "CONNECTION"
},
{
"value" : 0,
"name" : "IO_ERROR"
},
{
"value" : 0,
"name" : "WRONG_LENGTH"
},
{
"value" : 0,
"name" : "WRONG_MAP"
},
{
"value" : 0,
"name" : "WRONG_REDUCE"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter",
"counter" : [
{
"value" : 0,
"name" : "BYTES_WRITTEN"
}
]
}
]
}
}
+---+
<<XML response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/counters
Accept: application/xml
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 2660
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<jobTaskCounters>
<id>task_1326381300833_2_2_m_0</id>
<taskCounterGroup>
<counterGroupName>org.apache.hadoop.mapreduce.FileSystemCounter</counterGroupName>
<counter>
<name>FILE_BYTES_READ</name>
<value>2363</value>
</counter>
<counter>
<name>FILE_BYTES_WRITTEN</name>
<value>54372</value>
</counter>
<counter>
<name>FILE_READ_OPS</name>
<value>0</value>
</counter>
<counter>
<name>FILE_LARGE_READ_OPS</name>
<value>0</value>
</counter>
<counter>
<name>FILE_WRITE_OPS</name>
<value>0</value>
</counter>
<counter>
<name>HDFS_BYTES_READ</name>
<value>0</value>
</counter>
<counter>
<name>HDFS_BYTES_WRITTEN</name>
<value>0</value>
</counter>
<counter>
<name>HDFS_READ_OPS</name>
<value>0</value>
</counter>
<counter>
<name>HDFS_LARGE_READ_OPS</name>
<value>0</value>
</counter>
<counter>
<name>HDFS_WRITE_OPS</name>
<value>0</value>
</counter>
</taskCounterGroup>
<taskCounterGroup>
<counterGroupName>org.apache.hadoop.mapreduce.TaskCounter</counterGroupName>
<counter>
<name>COMBINE_INPUT_RECORDS</name>
<value>0</value>
</counter>
<counter>
<name>COMBINE_OUTPUT_RECORDS</name>
<value>0</value>
</counter>
<counter>
<name>REDUCE_INPUT_GROUPS</name>
<value>460</value>
</counter>
<counter>
<name>REDUCE_SHUFFLE_BYTES</name>
<value>2235</value>
</counter>
<counter>
<name>REDUCE_INPUT_RECORDS</name>
<value>460</value>
</counter>
<counter>
<name>REDUCE_OUTPUT_RECORDS</name>
<value>0</value>
</counter>
<counter>
<name>SPILLED_RECORDS</name>
<value>0</value>
</counter>
<counter>
<name>SHUFFLED_MAPS</name>
<value>1</value>
</counter>
<counter>
<name>FAILED_SHUFFLE</name>
<value>0</value>
</counter>
<counter>
<name>MERGED_MAP_OUTPUTS</name>
<value>1</value>
</counter>
<counter>
<name>GC_TIME_MILLIS</name>
<value>26</value>
</counter>
<counter>
<name>CPU_MILLISECONDS</name>
<value>860</value>
</counter>
<counter>
<name>PHYSICAL_MEMORY_BYTES</name>
<value>107839488</value>
</counter>
<counter>
<name>VIRTUAL_MEMORY_BYTES</name>
<value>1123147776</value>
</counter>
<counter>
<name>COMMITTED_HEAP_BYTES</name>
<value>57475072</value>
</counter>
</taskCounterGroup>
<taskCounterGroup>
<counterGroupName>Shuffle Errors</counterGroupName>
<counter>
<name>BAD_ID</name>
<value>0</value>
</counter>
<counter>
<name>CONNECTION</name>
<value>0</value>
</counter>
<counter>
<name>IO_ERROR</name>
<value>0</value>
</counter>
<counter>
<name>WRONG_LENGTH</name>
<value>0</value>
</counter>
<counter>
<name>WRONG_MAP</name>
<value>0</value>
</counter>
<counter>
<name>WRONG_REDUCE</name>
<value>0</value>
</counter>
</taskCounterGroup>
<taskCounterGroup>
<counterGroupName>org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter</counterGroupName>
<counter>
<name>BYTES_WRITTEN</name>
<value>0</value>
</counter>
</taskCounterGroup>
</jobTaskCounters>
+---+
** Task Attempts API
With the task attempts API, you can obtain a collection of resources that represent a task attempt within a job. When you run a GET operation on this resource, you obtain a collection of Task Attempt Objects.
*** URI
------
* http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/tasks/{taskid}/attempts
------
*** HTTP Operations Supported
------
* GET
------
*** Query Parameters Supported
------
None
------
*** Elements of the <taskAttempts> object
When you make a request for the list of task attempts, the information will be returned as an array of task attempt objects.
See also {{Task Attempt API}} for syntax of the task object.
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| taskAttempt | array of task attempt objects(JSON)/zero or more task attempt objects(XML) | The collection of task attempt objects |
*---------------+--------------+--------------------------------+
*** Response Examples
<<JSON response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/attempts
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
{
"taskAttempts" : {
"taskAttempt" : [
{
"assignedContainerId" : "container_1326381300833_0002_01_000002",
"progress" : 100,
"elapsedTime" : 2638,
"state" : "SUCCEEDED",
"diagnostics" : "",
"rack" : "/98.139.92.0",
"nodeHttpAddress" : "host.domain.com:8042",
"startTime" : 1326381450680,
"id" : "attempt_1326381300833_2_2_m_0_0",
"type" : "MAP",
"finishTime" : 1326381453318
}
]
}
}
+---+
<<XML response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/attempts
Accept: application/xml
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 537
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<taskAttempts>
<taskAttempt>
<startTime>1326381450680</startTime>
<finishTime>1326381453318</finishTime>
<elapsedTime>2638</elapsedTime>
<progress>100.0</progress>
<id>attempt_1326381300833_2_2_m_0_0</id>
<rack>/98.139.92.0</rack>
<state>SUCCEEDED</state>
<nodeHttpAddress>host.domain.com:8042</nodeHttpAddress>
<diagnostics/>
<type>MAP</type>
<assignedContainerId>container_1326381300833_0002_01_000002</assignedContainerId>
</taskAttempt>
</taskAttempts>
+---+
** {Task Attempt API}
A Task Attempt resource contains information about a particular task attempt within a job.
*** URI
Use the following URI to obtain an Task Attempt Object, from a task identified by the {attemptid} value.
------
* http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/tasks/{taskid}/attempt/{attemptid}
------
*** HTTP Operations Supported
------
* GET
------
*** Query Parameters Supported
------
None
------
*** Elements of the <taskAttempt> object
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| id | string | The task id |
*---------------+--------------+--------------------------------+
| rack | string | The rack |
*---------------+--------------+--------------------------------+
| state | string | The state of the task attempt - valid values are: NEW, UNASSIGNED, ASSIGNED, RUNNING, COMMIT_PENDING, SUCCESS_CONTAINER_CLEANUP, SUCCEEDED, FAIL_CONTAINER_CLEANUP, FAIL_TASK_CLEANUP, FAILED, KILL_CONTAINER_CLEANUP, KILL_TASK_CLEANUP, KILLED |
*---------------+--------------+--------------------------------+
| type | string | The type of task |
*---------------+--------------+--------------------------------+
| assignedContainerId | string | The container id this attempt is assigned to|
*---------------+--------------+--------------------------------+
| nodeHttpAddress | string | The http address of the node this task attempt ran on |
*---------------+--------------+--------------------------------+
| diagnostics| string | A diagnostics message |
*---------------+--------------+--------------------------------+
| progress | float | The progress of the task attempt as a percent|
*---------------+--------------+--------------------------------+
| startTime | long | The time in which the task attempt started (in ms since epoch)|
*---------------+--------------+--------------------------------+
| finishTime | long | The time in which the task attempt finished (in ms since epoch)|
*---------------+--------------+--------------------------------+
| elapsedTime | long | The elapsed time since the task attempt started (in ms)|
*---------------+--------------+--------------------------------+
For reduce task attempts you also have the following fields:
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| shuffleFinishTime | long | The time at which shuffle finished (in ms since epoch)|
*---------------+--------------+--------------------------------+
| mergeFinishTime | long | The time at which merge finished (in ms since epoch)|
*---------------+--------------+--------------------------------+
| elapsedShuffleTime | long | The time it took for the shuffle phase to complete (time in ms between reduce task start and shuffle finish)|
*---------------+--------------+--------------------------------+
| elapsedMergeTime | long | The time it took for the merge phase to complete (time in ms between the shuffle finish and merge finish)|
*---------------+--------------+--------------------------------+
| elapsedReduceTime | long | The time it took for the reduce phase to complete (time in ms between merge finish to end of reduce task)|
*---------------+--------------+--------------------------------+
*** Response Examples
<<JSON response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/attempts/attempt_1326381300833_2_2_m_0_0
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
{
"taskAttempt" : {
"assignedContainerId" : "container_1326381300833_0002_01_000002",
"progress" : 100,
"elapsedTime" : 2638,
"state" : "SUCCEEDED",
"diagnostics" : "",
"rack" : "/98.139.92.0",
"nodeHttpAddress" : "host.domain.com:8042",
"startTime" : 1326381450680,
"id" : "attempt_1326381300833_2_2_m_0_0",
"type" : "MAP",
"finishTime" : 1326381453318
}
}
+---+
<<XML response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/attempts/attempt_1326381300833_2_2_m_0_0
Accept: application/xml
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 691
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<taskAttempt>
<startTime>1326381450680</startTime>
<finishTime>1326381453318</finishTime>
<elapsedTime>2638</elapsedTime>
<progress>100.0</progress>
<id>attempt_1326381300833_2_2_m_0_0</id>
<rack>/98.139.92.0</rack>
<state>SUCCEEDED</state>
<nodeHttpAddress>host.domain.com:8042</nodeHttpAddress>
<diagnostics/>
<type>MAP</type>
<assignedContainerId>container_1326381300833_0002_01_000002</assignedContainerId>
</taskAttempt>
+---+
** Task Attempt Counters API
With the task attempt counters API, you can object a collection of resources that represent al the counters for that task attempt.
*** URI
------
* http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/tasks/{taskid}/attempt/{attemptid}/counters
------
*** HTTP Operations Supported
------
* GET
------
*** Query Parameters Supported
------
None
------
*** Elements of the <jobTaskAttemptCounters> object
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| id | string | The task attempt id |
*---------------+--------------+-------------------------------+
| taskAttemptcounterGroup | array of task attempt counterGroup objects(JSON)/zero or more task attempt counterGroup objects(XML) | A collection of task attempt counter group objects |
*---------------+--------------+-------------------------------+
*** Elements of the <taskAttemptCounterGroup> object
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| counterGroupName | string | The name of the counter group |
*---------------+--------------+-------------------------------+
| counter | array of counter objects(JSON)/zero or more counter objects(XML) | A collection of counter objects |
*---------------+--------------+-------------------------------+
*** Elements of the <counter> object
*---------------+--------------+-------------------------------+
|| Item || Data Type || Description |
*---------------+--------------+-------------------------------+
| name | string | The name of the counter |
*---------------+--------------+-------------------------------+
| value | long | The value of the counter |
*---------------+--------------+-------------------------------+
*** Response Examples
<<JSON response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/attempts/attempt_1326381300833_2_2_m_0_0/counters
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
{
"jobTaskAttemptCounters" : {
"taskAttemptCounterGroup" : [
{
"counterGroupName" : "org.apache.hadoop.mapreduce.FileSystemCounter",
"counter" : [
{
"value" : 2363,
"name" : "FILE_BYTES_READ"
},
{
"value" : 54372,
"name" : "FILE_BYTES_WRITTEN"
},
{
"value" : 0,
"name" : "FILE_READ_OPS"
},
{
"value" : 0,
"name" : "FILE_LARGE_READ_OPS"
},
{
"value" : 0,
"name" : "FILE_WRITE_OPS"
},
{
"value" : 0,
"name" : "HDFS_BYTES_READ"
},
{
"value" : 0,
"name" : "HDFS_BYTES_WRITTEN"
},
{
"value" : 0,
"name" : "HDFS_READ_OPS"
},
{
"value" : 0,
"name" : "HDFS_LARGE_READ_OPS"
},
{
"value" : 0,
"name" : "HDFS_WRITE_OPS"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.TaskCounter",
"counter" : [
{
"value" : 0,
"name" : "COMBINE_INPUT_RECORDS"
},
{
"value" : 0,
"name" : "COMBINE_OUTPUT_RECORDS"
},
{
"value" : 460,
"name" : "REDUCE_INPUT_GROUPS"
},
{
"value" : 2235,
"name" : "REDUCE_SHUFFLE_BYTES"
},
{
"value" : 460,
"name" : "REDUCE_INPUT_RECORDS"
},
{
"value" : 0,
"name" : "REDUCE_OUTPUT_RECORDS"
},
{
"value" : 0,
"name" : "SPILLED_RECORDS"
},
{
"value" : 1,
"name" : "SHUFFLED_MAPS"
},
{
"value" : 0,
"name" : "FAILED_SHUFFLE"
},
{
"value" : 1,
"name" : "MERGED_MAP_OUTPUTS"
},
{
"value" : 26,
"name" : "GC_TIME_MILLIS"
},
{
"value" : 860,
"name" : "CPU_MILLISECONDS"
},
{
"value" : 107839488,
"name" : "PHYSICAL_MEMORY_BYTES"
},
{
"value" : 1123147776,
"name" : "VIRTUAL_MEMORY_BYTES"
},
{
"value" : 57475072,
"name" : "COMMITTED_HEAP_BYTES"
}
]
},
{
"counterGroupName" : "Shuffle Errors",
"counter" : [
{
"value" : 0,
"name" : "BAD_ID"
},
{
"value" : 0,
"name" : "CONNECTION"
},
{
"value" : 0,
"name" : "IO_ERROR"
},
{
"value" : 0,
"name" : "WRONG_LENGTH"
},
{
"value" : 0,
"name" : "WRONG_MAP"
},
{
"value" : 0,
"name" : "WRONG_REDUCE"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter",
"counter" : [
{
"value" : 0,
"name" : "BYTES_WRITTEN"
}
]
}
],
"id" : "attempt_1326381300833_2_2_m_0_0"
}
}
+---+
<<XML response>>
HTTP Request:
------
GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/attempts/attempt_1326381300833_2_2_m_0_0/counters
Accept: application/xml
------
Response Header:
+---+
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 2735
Server: Jetty(6.1.26)
+---+
Response Body:
+---+
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<jobTaskAttemptCounters>
<id>attempt_1326381300833_2_2_m_0_0</id>
<taskAttemptCounterGroup>
<counterGroupName>org.apache.hadoop.mapreduce.FileSystemCounter</counterGroupName>
<counter>
<name>FILE_BYTES_READ</name>
<value>2363</value>
</counter>
<counter>
<name>FILE_BYTES_WRITTEN</name>
<value>54372</value>
</counter>
<counter>
<name>FILE_READ_OPS</name>
<value>0</value>
</counter>
<counter>
<name>FILE_LARGE_READ_OPS</name>
<value>0</value>
</counter>
<counter>
<name>FILE_WRITE_OPS</name>
<value>0</value>
</counter>
<counter>
<name>HDFS_BYTES_READ</name>
<value>0</value>
</counter>
<counter>
<name>HDFS_BYTES_WRITTEN</name>
<value>0</value>
</counter>
<counter>
<name>HDFS_READ_OPS</name>
<value>0</value>
</counter>
<counter>
<name>HDFS_LARGE_READ_OPS</name>
<value>0</value>
</counter>
<counter>
<name>HDFS_WRITE_OPS</name>
<value>0</value>
</counter>
</taskAttemptCounterGroup>
<taskAttemptCounterGroup>
<counterGroupName>org.apache.hadoop.mapreduce.TaskCounter</counterGroupName>
<counter>
<name>COMBINE_INPUT_RECORDS</name>
<value>0</value>
</counter>
<counter>
<name>COMBINE_OUTPUT_RECORDS</name>
<value>0</value>
</counter>
<counter>
<name>REDUCE_INPUT_GROUPS</name>
<value>460</value>
</counter>
<counter>
<name>REDUCE_SHUFFLE_BYTES</name>
<value>2235</value>
</counter>
<counter>
<name>REDUCE_INPUT_RECORDS</name>
<value>460</value>
</counter>
<counter>
<name>REDUCE_OUTPUT_RECORDS</name>
<value>0</value>
</counter>
<counter>
<name>SPILLED_RECORDS</name>
<value>0</value>
</counter>
<counter>
<name>SHUFFLED_MAPS</name>
<value>1</value>
</counter>
<counter>
<name>FAILED_SHUFFLE</name>
<value>0</value>
</counter>
<counter>
<name>MERGED_MAP_OUTPUTS</name>
<value>1</value>
</counter>
<counter>
<name>GC_TIME_MILLIS</name>
<value>26</value>
</counter>
<counter>
<name>CPU_MILLISECONDS</name>
<value>860</value>
</counter>
<counter>
<name>PHYSICAL_MEMORY_BYTES</name>
<value>107839488</value>
</counter>
<counter>
<name>VIRTUAL_MEMORY_BYTES</name>
<value>1123147776</value>
</counter>
<counter>
<name>COMMITTED_HEAP_BYTES</name>
<value>57475072</value>
</counter>
</taskAttemptCounterGroup>
<taskAttemptCounterGroup>
<counterGroupName>Shuffle Errors</counterGroupName>
<counter>
<name>BAD_ID</name>
<value>0</value>
</counter>
<counter>
<name>CONNECTION</name>
<value>0</value>
</counter>
<counter>
<name>IO_ERROR</name>
<value>0</value>
</counter>
<counter>
<name>WRONG_LENGTH</name>
<value>0</value>
</counter>
<counter>
<name>WRONG_MAP</name>
<value>0</value>
</counter>
<counter>
<name>WRONG_REDUCE</name>
<value>0</value>
</counter>
</taskAttemptCounterGroup>
<taskAttemptCounterGroup>
<counterGroupName>org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter</counterGroupName>
<counter>
<name>BYTES_WRITTEN</name>
<value>0</value>
</counter>
</taskAttemptCounterGroup>
</jobTaskAttemptCounters>
+---+
c4science · Help