@InterfaceAudience.Public @InterfaceStability.Stable public class FileOutputCommitter extends OutputCommitter
OutputCommitter that commits files specified
in job output directory i.e. ${mapreduce.output.fileoutputformat.outputdir}.| Modifier and Type | Field and Description |
|---|---|
static org.apache.commons.logging.Log |
LOG |
static String |
SUCCEEDED_FILE_NAME |
static String |
TEMP_DIR_NAME
Temporary directory name
|
| Constructor and Description |
|---|
FileOutputCommitter() |
| Modifier and Type | Method and Description |
|---|---|
void |
abortJob(JobContext context,
int runState)
For aborting an unsuccessful job's output.
|
void |
abortTask(TaskAttemptContext context)
Discard the task output.
|
void |
cleanupJob(JobContext context)
Deprecated.
|
void |
commitJob(JobContext context)
For committing job's output after successful job completion.
|
void |
commitTask(TaskAttemptContext context)
To promote the task's temporary output to final output location.
|
org.apache.hadoop.fs.Path |
getTaskAttemptPath(TaskAttemptContext context) |
org.apache.hadoop.fs.Path |
getWorkPath(TaskAttemptContext context,
org.apache.hadoop.fs.Path outputPath) |
boolean |
isCommitJobRepeatable(JobContext context)
Returns true if an in-progress job commit can be retried.
|
boolean |
isRecoverySupported()
Deprecated.
|
boolean |
isRecoverySupported(JobContext context)
Is task output recovery supported for restarting jobs?
If task output recovery is supported, job restart can be done more
efficiently.
|
boolean |
needsTaskCommit(TaskAttemptContext context)
Check whether task needs a commit.
|
void |
recoverTask(TaskAttemptContext context)
Recover the task output.
|
void |
setupJob(JobContext context)
For the framework to setup the job output during initialization.
|
void |
setupTask(TaskAttemptContext context)
Sets up output for the task.
|
abortJob, abortTask, cleanupJob, commitJob, commitTask, isCommitJobRepeatable, isRecoverySupported, needsTaskCommit, recoverTask, setupJob, setupTaskpublic static final org.apache.commons.logging.Log LOG
public static final String TEMP_DIR_NAME
public static final String SUCCEEDED_FILE_NAME
@InterfaceAudience.Private public org.apache.hadoop.fs.Path getTaskAttemptPath(TaskAttemptContext context) throws IOException
IOExceptionpublic org.apache.hadoop.fs.Path getWorkPath(TaskAttemptContext context, org.apache.hadoop.fs.Path outputPath) throws IOException
IOExceptionpublic void setupJob(JobContext context) throws IOException
OutputCommittersetupJob in class OutputCommittercontext - Context of the job whose output is being written.IOException - if temporary output could not be createdpublic void commitJob(JobContext context) throws IOException
OutputCommittercommitJob in class OutputCommittercontext - Context of the job whose output is being written.IOException@Deprecated public void cleanupJob(JobContext context) throws IOException
OutputCommittercleanupJob in class OutputCommittercontext - Context of the job whose output is being written.IOExceptionpublic void abortJob(JobContext context, int runState) throws IOException
OutputCommitterJobStatus.FAILED or
JobStatus.KILLED. This is called from the application
master process for the entire job. This may be called multiple times.abortJob in class OutputCommittercontext - Context of the job whose output is being written.runState - final runstate of the jobIOExceptionpublic void setupTask(TaskAttemptContext context) throws IOException
OutputCommittersetupTask in class OutputCommittercontext - Context of the task whose output is being written.IOExceptionpublic void commitTask(TaskAttemptContext context) throws IOException
OutputCommitterOutputCommitter.needsTaskCommit(TaskAttemptContext) returns true and this
task is the task that the AM determines finished first, this method
is called to commit an individual task's output. This is to mark
that tasks output as complete, as OutputCommitter.commitJob(JobContext) will
also be called later on if the entire job finished successfully. This
is called from a task's process. This may be called multiple times for the
same task, but different task attempts. It should be very rare for this to
be called multiple times and requires odd networking failures to make this
happen. In the future the Hadoop framework may eliminate this race.commitTask in class OutputCommittercontext - Context of the task whose output is being written.IOException - if commit is notpublic void abortTask(TaskAttemptContext context) throws IOException
OutputCommitterabortTask in class OutputCommitterIOExceptionpublic boolean needsTaskCommit(TaskAttemptContext context) throws IOException
OutputCommitterneedsTaskCommit in class OutputCommitterIOException@Deprecated public boolean isRecoverySupported()
OutputCommitterisRecoverySupported in class OutputCommittertrue if task output recovery is supported,
false otherwiseOutputCommitter.recoverTask(TaskAttemptContext)public boolean isCommitJobRepeatable(JobContext context) throws IOException
OutputCommitterisCommitJobRepeatable in class OutputCommittercontext - Context of the job whose output is being written.true repeatable job commit is supported,
false otherwiseIOExceptionpublic boolean isRecoverySupported(JobContext context) throws IOException
OutputCommitterisRecoverySupported in class OutputCommittercontext - Context of the job whose output is being written.true if task output recovery is supported,
false otherwiseIOExceptionOutputCommitter.recoverTask(TaskAttemptContext)public void recoverTask(TaskAttemptContext context) throws IOException
OutputCommitterMRConstants.APPLICATION_ATTEMPT_ID key in
JobContext.getConfiguration() for the
OutputCommitter. This is called from the application master
process, but it is called individually for each task.
If an exception is thrown the task will be attempted again.recoverTask in class OutputCommittercontext - Context of the task whose output is being recoveredIOExceptionCopyright © 2022 Apache Software Foundation. All rights reserved.