@InterfaceAudience.Private @InterfaceStability.Unstable public class ZKRMStateStore extends RMStateStore
RMStateStore implementation backed by ZooKeeper.
The znode structure is as follows:
ROOT_DIR_PATH
|--- VERSION_INFO
|--- EPOCH_NODE
|--- RM_ZK_FENCING_LOCK
|--- RM_APP_ROOT
| |----- HIERARCHIES
| | |----- 1
| | | |----- (#ApplicationId barring last character)
| | | | |----- (#Last character of ApplicationId)
| | | | | |----- (#ApplicationAttemptIds)
| | | ....
| | |
| | |----- 2
| | | |----- (#ApplicationId barring last 2 characters)
| | | | |----- (#Last 2 characters of ApplicationId)
| | | | | |----- (#ApplicationAttemptIds)
| | | ....
| | |
| | |----- 3
| | | |----- (#ApplicationId barring last 3 characters)
| | | | |----- (#Last 3 characters of ApplicationId)
| | | | | |----- (#ApplicationAttemptIds)
| | | ....
| | |
| | |----- 4
| | | |----- (#ApplicationId barring last 4 characters)
| | | | |----- (#Last 4 characters of ApplicationId)
| | | | | |----- (#ApplicationAttemptIds)
| | | ....
| | |
| |----- (#ApplicationId1)
| | |----- (#ApplicationAttemptIds)
| |
| |----- (#ApplicationId2)
| | |----- (#ApplicationAttemptIds)
| ....
|
|--- RM_DT_SECRET_MANAGER_ROOT
|----- RM_DT_SEQUENTIAL_NUMBER_ZNODE_NAME
|----- RM_DELEGATION_TOKENS_ROOT_ZNODE_NAME
| |----- 1
| | |----- (#TokenId barring last character)
| | | |----- (#Last character of TokenId)
| | ....
| |----- 2
| | |----- (#TokenId barring last 2 characters)
| | | |----- (#Last 2 characters of TokenId)
| | ....
| |----- 3
| | |----- (#TokenId barring last 3 characters)
| | | |----- (#Last 3 characters of TokenId)
| | ....
| |----- 4
| | |----- (#TokenId barring last 4 characters)
| | | |----- (#Last 4 characters of TokenId)
| | ....
| |----- Token_1
| |----- Token_2
| ....
|
|----- RM_DT_MASTER_KEYS_ROOT_ZNODE_NAME
| |----- Key_1
| |----- Key_2
....
|--- AMRMTOKEN_SECRET_MANAGER_ROOT
|----- currentMasterKey
|----- nextMasterKey
|-- RESERVATION_SYSTEM_ROOT
|------PLAN_1
| |------ RESERVATION_1
| |------ RESERVATION_2
| ....
|------PLAN_2
....
Note: Changes from 1.1 to 1.2 - AMRMTokenSecretManager state has been saved
separately. The currentMasterkey and nextMasterkey have been stored.
Also, AMRMToken has been removed from ApplicationAttemptState.
Changes from 1.2 to 1.3, Addition of ReservationSystem state.
Changes from 1.3 to 1.4 - Change the structure of application znode by
splitting it in 2 parts, depending on a configurable split index. This limits
the number of application znodes returned in a single call while loading
app state.
Changes from 1.4 to 1.5 - Change the structure of delegation token znode by
splitting it in 2 parts, depending on a configurable split index. This limits
the number of delegation token znodes returned in a single call while loading
tokens state.RMStateStore.RMDTSecretManagerState, RMStateStore.RMState, RMStateStore.RMStateStoreState| Modifier and Type | Field and Description |
|---|---|
protected static org.apache.hadoop.yarn.server.records.Version |
CURRENT_VERSION_INFO |
protected int |
delegationTokenNodeSplitIndex |
static String |
RM_APP_ROOT_HIERARCHIES |
static String |
ROOT_ZNODE_NAME |
protected String |
znodeWorkingPath |
AM_CLIENT_TOKEN_MASTER_KEY_NAME, AM_RM_TOKEN_SERVICE, AMRMTOKEN_SECRET_MANAGER_ROOT, baseEpoch, DELEGATION_KEY_PREFIX, DELEGATION_TOKEN_PREFIX, DELEGATION_TOKEN_SEQUENCE_NUMBER_PREFIX, EPOCH_NODE, RESERVATION_SYSTEM_ROOT, resourceManager, RM_APP_ROOT, RM_DT_SECRET_MANAGER_ROOT, rmStateStoreEventHandler, VERSION_NODE| Constructor and Description |
|---|
ZKRMStateStore() |
| Modifier and Type | Method and Description |
|---|---|
protected void |
closeInternal()
Derived classes close themselves using this method.
|
protected List<org.apache.zookeeper.data.ACL> |
constructZkRootNodeACL(org.apache.hadoop.conf.Configuration conf,
List<org.apache.zookeeper.data.ACL> sourceACLs)
Given the
Configuration and ACLs used (sourceACLs) for
ZooKeeper access, construct the ACLs for the store's root node. |
void |
deleteStore()
Derived classes must implement this method to delete the state store
|
long |
getAndIncrementEpoch()
Get the current epoch of RM and increment the value.
|
protected org.apache.hadoop.yarn.server.records.Version |
getCurrentVersion()
Get the current version of the underlying state store.
|
void |
initInternal(org.apache.hadoop.conf.Configuration conf)
Derived classes initialize themselves using this method.
|
RMStateStore.RMState |
loadState()
Blocking API
The derived class must recover state from the store and return a new
RMState object populated with that state
This must not be called on the dispatcher thread
|
protected org.apache.hadoop.yarn.server.records.Version |
loadVersion()
Derived class use this method to load the version information from state
store.
|
void |
removeApplication(org.apache.hadoop.yarn.api.records.ApplicationId removeAppId)
Derived classes must implement this method to remove application from the
state store
|
protected void |
removeApplicationAttemptInternal(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)
Blocking API
Derived classes must implement this method to remove the state of specified
attempt.
|
protected void |
removeApplicationStateInternal(ApplicationStateData appState)
Blocking API
Derived classes must implement this method to remove the state of an
application and its attempts
|
protected void |
removeReservationState(String planName,
String reservationIdName)
Blocking API
Derived classes must implement this method to remove the state of
a reservation allocation.
|
protected void |
removeRMDelegationTokenState(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier)
Blocking API
Derived classes must implement this method to remove the state of RMDelegationToken
|
protected void |
removeRMDTMasterKeyState(org.apache.hadoop.security.token.delegation.DelegationKey delegationKey)
Blocking API
Derived classes must implement this method to remove the state of
DelegationToken Master Key
|
void |
startInternal()
Derived classes start themselves using this method.
|
protected void |
storeApplicationAttemptStateInternal(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId,
ApplicationAttemptStateData attemptStateDataPB)
Blocking API
Derived classes must implement this method to store the state of an
application attempt
|
void |
storeApplicationStateInternal(org.apache.hadoop.yarn.api.records.ApplicationId appId,
ApplicationStateData appStateDataPB)
Blocking API
Derived classes must implement this method to store the state of an
application.
|
protected void |
storeOrUpdateAMRMTokenSecretManagerState(AMRMTokenSecretManagerState amrmTokenSecretManagerState,
boolean isUpdate)
Blocking API Derived classes must implement this method to store or update
the state of AMRMToken Master Key
|
protected void |
storeReservationState(org.apache.hadoop.yarn.proto.YarnProtos.ReservationAllocationStateProto reservationAllocation,
String planName,
String reservationIdName)
Blocking API
Derived classes must implement this method to store the state of
a reservation allocation.
|
protected void |
storeRMDelegationTokenState(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier,
Long renewDate)
Blocking API
Derived classes must implement this method to store the state of
RMDelegationToken and sequence number
|
protected void |
storeRMDTMasterKeyState(org.apache.hadoop.security.token.delegation.DelegationKey delegationKey)
Blocking API
Derived classes must implement this method to store the state of
DelegationToken Master Key
|
protected void |
storeVersion()
Derived class use this method to store the version information.
|
protected void |
updateApplicationAttemptStateInternal(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId,
ApplicationAttemptStateData attemptStateDataPB) |
protected void |
updateApplicationStateInternal(org.apache.hadoop.yarn.api.records.ApplicationId appId,
ApplicationStateData appStateDataPB) |
protected void |
updateRMDelegationTokenState(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier,
Long renewDate)
Blocking API
Derived classes must implement this method to update the state of
RMDelegationToken and sequence number
|
checkVersion, getCredentialsFromAppAttempt, getRMStateStoreEventHandler, getRMStateStoreState, handleStoreEvent, isFencedState, nextEpoch, notifyStoreOperationFailed, removeApplication, removeApplicationAttempt, removeReservation, removeRMDelegationToken, removeRMDTMasterKey, serviceInit, serviceStart, serviceStop, setResourceManager, setRMDispatcher, storeNewApplication, storeNewApplicationAttempt, storeNewReservation, storeOrUpdateAMRMTokenSecretManager, storeRMDelegationToken, storeRMDTMasterKey, updateApplicationAttemptState, updateApplicationState, updateApplicationState, updateApplicationStateSynchronously, updateFencedState, updateRMDelegationTokenclose, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStoppublic static final String ROOT_ZNODE_NAME
protected static final org.apache.hadoop.yarn.server.records.Version CURRENT_VERSION_INFO
public static final String RM_APP_ROOT_HIERARCHIES
protected String znodeWorkingPath
protected int delegationTokenNodeSplitIndex
@InterfaceAudience.Private @InterfaceStability.Unstable protected List<org.apache.zookeeper.data.ACL> constructZkRootNodeACL(org.apache.hadoop.conf.Configuration conf, List<org.apache.zookeeper.data.ACL> sourceACLs) throws NoSuchAlgorithmException
Configuration and ACLs used (sourceACLs) for
ZooKeeper access, construct the ACLs for the store's root node.
In the constructed ACL, all the users allowed by sourceACLs are
given read-write-admin access, while the current RM has exclusive
create-delete access.
To be called only when HA is enabled and the configuration doesn't set an
ACL for the root node.conf - the configurationsourceACLs - the source ACLsNoSuchAlgorithmException - thrown if the digest
algorithm used by Zookeeper cannot be foundpublic void initInternal(org.apache.hadoop.conf.Configuration conf)
throws IOException,
NoSuchAlgorithmException
RMStateStoreinitInternal in class RMStateStoreIOExceptionNoSuchAlgorithmExceptionpublic void startInternal()
throws Exception
RMStateStorestartInternal in class RMStateStoreExceptionprotected void closeInternal()
throws Exception
RMStateStorecloseInternal in class RMStateStoreExceptionprotected org.apache.hadoop.yarn.server.records.Version getCurrentVersion()
RMStateStoregetCurrentVersion in class RMStateStoreprotected void storeVersion()
throws Exception
RMStateStorestoreVersion in class RMStateStoreExceptionprotected org.apache.hadoop.yarn.server.records.Version loadVersion()
throws Exception
RMStateStoreloadVersion in class RMStateStoreExceptionpublic long getAndIncrementEpoch()
throws Exception
RMStateStoregetAndIncrementEpoch in class RMStateStoreExceptionpublic RMStateStore.RMState loadState() throws Exception
RMStateStoreloadState in class RMStateStoreExceptionpublic void storeApplicationStateInternal(org.apache.hadoop.yarn.api.records.ApplicationId appId,
ApplicationStateData appStateDataPB)
throws Exception
RMStateStorestoreApplicationStateInternal in class RMStateStoreExceptionprotected void updateApplicationStateInternal(org.apache.hadoop.yarn.api.records.ApplicationId appId,
ApplicationStateData appStateDataPB)
throws Exception
updateApplicationStateInternal in class RMStateStoreExceptionprotected void storeApplicationAttemptStateInternal(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId,
ApplicationAttemptStateData attemptStateDataPB)
throws Exception
RMStateStorestoreApplicationAttemptStateInternal in class RMStateStoreExceptionprotected void updateApplicationAttemptStateInternal(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId,
ApplicationAttemptStateData attemptStateDataPB)
throws Exception
updateApplicationAttemptStateInternal in class RMStateStoreExceptionprotected void removeApplicationAttemptInternal(org.apache.hadoop.yarn.api.records.ApplicationAttemptId appAttemptId)
throws Exception
RMStateStoreremoveApplicationAttemptInternal in class RMStateStoreExceptionprotected void removeApplicationStateInternal(ApplicationStateData appState) throws Exception
RMStateStoreremoveApplicationStateInternal in class RMStateStoreExceptionprotected void storeRMDelegationTokenState(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier,
Long renewDate)
throws Exception
RMStateStorestoreRMDelegationTokenState in class RMStateStoreExceptionprotected void removeRMDelegationTokenState(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier)
throws Exception
RMStateStoreremoveRMDelegationTokenState in class RMStateStoreExceptionprotected void updateRMDelegationTokenState(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier,
Long renewDate)
throws Exception
RMStateStoreupdateRMDelegationTokenState in class RMStateStoreExceptionprotected void storeRMDTMasterKeyState(org.apache.hadoop.security.token.delegation.DelegationKey delegationKey)
throws Exception
RMStateStorestoreRMDTMasterKeyState in class RMStateStoreExceptionprotected void removeRMDTMasterKeyState(org.apache.hadoop.security.token.delegation.DelegationKey delegationKey)
throws Exception
RMStateStoreremoveRMDTMasterKeyState in class RMStateStoreExceptionpublic void deleteStore()
throws Exception
RMStateStoredeleteStore in class RMStateStoreExceptionpublic void removeApplication(org.apache.hadoop.yarn.api.records.ApplicationId removeAppId)
throws Exception
RMStateStoreremoveApplication in class RMStateStoreExceptionprotected void storeOrUpdateAMRMTokenSecretManagerState(AMRMTokenSecretManagerState amrmTokenSecretManagerState, boolean isUpdate) throws Exception
RMStateStorestoreOrUpdateAMRMTokenSecretManagerState in class RMStateStoreExceptionprotected void removeReservationState(String planName, String reservationIdName) throws Exception
RMStateStoreremoveReservationState in class RMStateStoreExceptionprotected void storeReservationState(org.apache.hadoop.yarn.proto.YarnProtos.ReservationAllocationStateProto reservationAllocation,
String planName,
String reservationIdName)
throws Exception
RMStateStorestoreReservationState in class RMStateStoreExceptionCopyright © 2008–2022 Apache Software Foundation. All rights reserved.