@InterfaceAudience.Private @InterfaceStability.Unstable public class AMRMClientImpl<T extends AMRMClient.ContainerRequest> extends AMRMClient<T>
AMRMClient.ContainerRequest| Modifier and Type | Field and Description |
|---|---|
protected String |
appHostName |
protected int |
appHostPort |
protected String |
appTrackingUrl |
protected Set<org.apache.hadoop.yarn.api.records.ResourceRequest> |
ask |
protected Set<String> |
blacklistAdditions |
protected Set<String> |
blacklistedNodes |
protected Set<String> |
blacklistRemovals |
protected Map<org.apache.hadoop.yarn.api.records.ContainerId,AbstractMap.SimpleEntry<org.apache.hadoop.yarn.api.records.Container,org.apache.hadoop.yarn.api.records.UpdateContainerRequest>> |
change |
protected org.apache.hadoop.yarn.api.records.Resource |
clusterAvailableResources |
protected int |
clusterNodeCount |
protected String |
newTrackingUrl |
protected Map<org.apache.hadoop.yarn.api.records.ContainerId,AbstractMap.SimpleEntry<org.apache.hadoop.yarn.api.records.Container,org.apache.hadoop.yarn.api.records.UpdateContainerRequest>> |
pendingChange |
protected Set<org.apache.hadoop.yarn.api.records.ContainerId> |
pendingRelease |
protected Set<org.apache.hadoop.yarn.api.records.ContainerId> |
release |
protected org.apache.hadoop.yarn.api.ApplicationMasterProtocol |
rmClient |
| Constructor and Description |
|---|
AMRMClientImpl() |
| Modifier and Type | Method and Description |
|---|---|
void |
addContainerRequest(T req)
Request containers for resources before calling
allocate |
org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse |
allocate(float progressIndicator)
Request additional containers and receive new container allocations.
|
org.apache.hadoop.yarn.api.records.Resource |
getAvailableResources()
Get the currently available resources in the cluster.
|
int |
getClusterNodeCount()
Get the current number of nodes in the cluster.
|
Collection<T> |
getMatchingRequests(long allocationRequestId)
Get outstanding
ContainerRequests matching the given
allocationRequestId. |
List<? extends Collection<T>> |
getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority,
String resourceName,
org.apache.hadoop.yarn.api.records.ExecutionType executionType,
org.apache.hadoop.yarn.api.records.Resource capability)
Get outstanding
ContainerRequests matching the given
parameters. |
List<? extends Collection<T>> |
getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority,
String resourceName,
org.apache.hadoop.yarn.api.records.Resource capability)
Get outstanding
ContainerRequests matching the given
parameters. |
protected void |
populateNMTokens(List<org.apache.hadoop.yarn.api.records.NMToken> nmTokens) |
org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse |
registerApplicationMaster(String appHostName,
int appHostPort,
String appTrackingUrl)
Register the application master.
|
void |
releaseAssignedContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)
Release containers assigned by the Resource Manager.
|
void |
removeContainerRequest(T req)
Remove previous container request.
|
protected void |
removePendingChangeRequests(List<org.apache.hadoop.yarn.api.records.UpdatedContainer> changedContainers) |
protected void |
removePendingReleaseRequests(List<org.apache.hadoop.yarn.api.records.ContainerStatus> completedContainersStatuses) |
void |
requestContainerUpdate(org.apache.hadoop.yarn.api.records.Container container,
org.apache.hadoop.yarn.api.records.UpdateContainerRequest updateContainerRequest)
Request a container update before calling
allocate. |
protected void |
serviceInit(org.apache.hadoop.conf.Configuration conf) |
protected void |
serviceStart() |
protected void |
serviceStop() |
void |
unregisterApplicationMaster(org.apache.hadoop.yarn.api.records.FinalApplicationStatus appStatus,
String appMessage,
String appTrackingUrl)
Unregister the application master.
|
void |
updateBlacklist(List<String> blacklistAdditions,
List<String> blacklistRemovals)
Update application's blacklist with addition or removal resources.
|
void |
updateTrackingUrl(String trackingUrl)
Update application's tracking url on next heartbeat.
|
createAMRMClient, getNMTokenCache, getRegisteredTimelineV2Client, registerTimelineV2Client, requestContainerResourceChange, setNMTokenCache, waitFor, waitFor, waitForclose, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStopprotected String appHostName
protected int appHostPort
protected String appTrackingUrl
protected String newTrackingUrl
protected org.apache.hadoop.yarn.api.ApplicationMasterProtocol rmClient
protected org.apache.hadoop.yarn.api.records.Resource clusterAvailableResources
protected int clusterNodeCount
protected final Set<org.apache.hadoop.yarn.api.records.ResourceRequest> ask
protected final Set<org.apache.hadoop.yarn.api.records.ContainerId> release
protected Set<org.apache.hadoop.yarn.api.records.ContainerId> pendingRelease
protected final Map<org.apache.hadoop.yarn.api.records.ContainerId,AbstractMap.SimpleEntry<org.apache.hadoop.yarn.api.records.Container,org.apache.hadoop.yarn.api.records.UpdateContainerRequest>> change
protected final Map<org.apache.hadoop.yarn.api.records.ContainerId,AbstractMap.SimpleEntry<org.apache.hadoop.yarn.api.records.Container,org.apache.hadoop.yarn.api.records.UpdateContainerRequest>> pendingChange
protected void serviceInit(org.apache.hadoop.conf.Configuration conf)
throws Exception
serviceInit in class org.apache.hadoop.service.AbstractServiceExceptionprotected void serviceStart()
throws Exception
serviceStart in class org.apache.hadoop.service.AbstractServiceExceptionprotected void serviceStop()
throws Exception
serviceStop in class org.apache.hadoop.service.AbstractServiceExceptionpublic org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse registerApplicationMaster(String appHostName, int appHostPort, String appTrackingUrl) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException
AMRMClientregisterApplicationMaster in class AMRMClient<T extends AMRMClient.ContainerRequest>appHostName - Name of the host on which master is runningappHostPort - Port master is listening onappTrackingUrl - URL at which the master info can be seenRegisterApplicationMasterResponseorg.apache.hadoop.yarn.exceptions.YarnExceptionIOExceptionpublic org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse allocate(float progressIndicator)
throws org.apache.hadoop.yarn.exceptions.YarnException,
IOException
AMRMClientaddContainerRequest are sent to the
ResourceManager. New containers assigned to the master are
retrieved. Status of completed containers and node health updates are also
retrieved. This also doubles up as a heartbeat to the ResourceManager and
must be made periodically. The call may not always return any new
allocations of containers. App should not make concurrent allocate
requests. May cause request loss.
Note : If the user has not removed container requests that have already been satisfied, then the re-register may end up sending the entire container requests to the RM (including matched requests). Which would mean the RM could end up giving it a lot of new allocated containers.
allocate in class AMRMClient<T extends AMRMClient.ContainerRequest>progressIndicator - Indicates progress made by the masterorg.apache.hadoop.yarn.exceptions.YarnExceptionIOExceptionprotected void removePendingReleaseRequests(List<org.apache.hadoop.yarn.api.records.ContainerStatus> completedContainersStatuses)
protected void removePendingChangeRequests(List<org.apache.hadoop.yarn.api.records.UpdatedContainer> changedContainers)
@InterfaceAudience.Private protected void populateNMTokens(List<org.apache.hadoop.yarn.api.records.NMToken> nmTokens)
public void unregisterApplicationMaster(org.apache.hadoop.yarn.api.records.FinalApplicationStatus appStatus,
String appMessage,
String appTrackingUrl)
throws org.apache.hadoop.yarn.exceptions.YarnException,
IOException
AMRMClientunregisterApplicationMaster in class AMRMClient<T extends AMRMClient.ContainerRequest>appStatus - Success/Failure status of the masterappMessage - Diagnostics message on failureappTrackingUrl - New URL to get master infoorg.apache.hadoop.yarn.exceptions.YarnExceptionIOExceptionpublic void addContainerRequest(T req)
AMRMClientallocateaddContainerRequest in class AMRMClient<T extends AMRMClient.ContainerRequest>req - Resource requestpublic void removeContainerRequest(T req)
AMRMClientremoveContainerRequest in class AMRMClient<T extends AMRMClient.ContainerRequest>req - Resource requestpublic void requestContainerUpdate(org.apache.hadoop.yarn.api.records.Container container,
org.apache.hadoop.yarn.api.records.UpdateContainerRequest updateContainerRequest)
AMRMClientallocate.
Any previous pending update request of the same container will be
removed.requestContainerUpdate in class AMRMClient<T extends AMRMClient.ContainerRequest>container - The container returned from the last successful resource
allocation or updateupdateContainerRequest - The UpdateContainerRequest.public void releaseAssignedContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)
AMRMClientreleaseAssignedContainer in class AMRMClient<T extends AMRMClient.ContainerRequest>public org.apache.hadoop.yarn.api.records.Resource getAvailableResources()
AMRMClientgetAvailableResources in class AMRMClient<T extends AMRMClient.ContainerRequest>public int getClusterNodeCount()
AMRMClientgetClusterNodeCount in class AMRMClient<T extends AMRMClient.ContainerRequest>public Collection<T> getMatchingRequests(long allocationRequestId)
AMRMClientContainerRequests matching the given
allocationRequestId. These ContainerRequests should have been added via
addContainerRequest earlier in the lifecycle. For performance,
the AMRMClient may return its internal collection directly without creating
a copy. Users should not perform mutable operations on the return value.
NOTE: This API only matches Container requests that were created by the
client WITH the allocationRequestId being set to a non-default value.getMatchingRequests in class AMRMClient<T extends AMRMClient.ContainerRequest>allocationRequestId - Allocation Request Idpublic List<? extends Collection<T>> getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority, String resourceName, org.apache.hadoop.yarn.api.records.Resource capability)
AMRMClientContainerRequests matching the given
parameters. These ContainerRequests should have been added via
addContainerRequest earlier in the lifecycle. For performance,
the AMRMClient may return its internal collection directly without creating
a copy. Users should not perform mutable operations on the return value.
Each collection in the list contains requests with identical
Resource size that fit in the given capability. In a
collection, requests will be returned in the same order as they were added.
NOTE: This API only matches Container requests that were created by the
client WITHOUT the allocationRequestId being set.getMatchingRequests in class AMRMClient<T extends AMRMClient.ContainerRequest>public List<? extends Collection<T>> getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority, String resourceName, org.apache.hadoop.yarn.api.records.ExecutionType executionType, org.apache.hadoop.yarn.api.records.Resource capability)
AMRMClientContainerRequests matching the given
parameters. These ContainerRequests should have been added via
addContainerRequest earlier in the lifecycle. For performance,
the AMRMClient may return its internal collection directly without creating
a copy. Users should not perform mutable operations on the return value.
Each collection in the list contains requests with identical
Resource size that fit in the given capability. In a
collection, requests will be returned in the same order as they were added.
specify an ExecutionType.
NOTE: This API only matches Container requests that were created by the
client WITHOUT the allocationRequestId being set.getMatchingRequests in class AMRMClient<T extends AMRMClient.ContainerRequest>priority - PriorityresourceName - LocationexecutionType - ExecutionTypecapability - Capabilitypublic void updateBlacklist(List<String> blacklistAdditions, List<String> blacklistRemovals)
AMRMClientupdateBlacklist in class AMRMClient<T extends AMRMClient.ContainerRequest>blacklistAdditions - list of resources which should be added to the
application blacklistblacklistRemovals - list of resources which should be removed from the
application blacklistpublic void updateTrackingUrl(String trackingUrl)
AMRMClientupdateTrackingUrl in class AMRMClient<T extends AMRMClient.ContainerRequest>trackingUrl - new tracking url for this applicationCopyright © 2022 Apache Software Foundation. All rights reserved.