Package org.apache.storm.scheduler
Class Cluster
- java.lang.Object
-
- org.apache.storm.scheduler.Cluster
-
- All Implemented Interfaces:
ISchedulingState
- Direct Known Subclasses:
SingleTopologyCluster
public class Cluster extends Object implements ISchedulingState
The current state of the storm cluster. Cluster is not currently thread safe.
-
-
Constructor Summary
Constructors Constructor Description Cluster(Cluster src)Copy constructor.Cluster(Cluster src, Topologies topologies)Testing Constructor that takes an existing cluster and replaces the topologies in it.Cluster(INimbus nimbus, ResourceMetrics resourceMetrics, Map<String,SupervisorDetails> supervisors, Map<String,? extends SchedulerAssignment> assignments, Topologies topologies, Map<String,Object> conf)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description protected voidassertValidTopologyForModification(String topologyId)Check if the given topology is allowed for modification right now.voidassign(SchedulerAssignment assignment, boolean ignoreSingleExceptions)Assign everything for the given topology.voidassign(WorkerSlot slot, String topologyId, Collection<ExecutorDetails> executors)Assign the slot to the executors for this topology.voidblacklistHost(String host)voidfreeSlot(WorkerSlot slot)Free the specified slot.voidfreeSlots(Collection<WorkerSlot> slots)free the slots.NormalizedResourceRequestgetAllScheduledResourcesForNode(String nodeId)Get all scheduled resources for node.Set<Integer>getAssignablePorts(SupervisorDetails supervisor)Get the ports that are not blacklisted.List<WorkerSlot>getAssignableSlots()Get all non-blacklisted slots in the cluster.List<WorkerSlot>getAssignableSlots(SupervisorDetails supervisor)Return all non-blacklisted slots on this supervisor.static doublegetAssignedMemoryForSlot(Map<String,Object> topConf)Get heap memory usage for a worker's main process and logwriter process.intgetAssignedNumWorkers(TopologyDetails topology)Get the number of workers assigned to a topology.SchedulerAssignmentgetAssignmentById(String topologyId)get the current assignment for the topology.Map<String,SchedulerAssignment>getAssignments()Get all the assignments.Set<Integer>getAvailablePorts(SupervisorDetails supervisor)Return the available ports of this supervisor.NormalizedResourceOffergetAvailableResources(SupervisorDetails sd)Get the resources on the supervisor that are available to be scheduled.List<WorkerSlot>getAvailableSlots()Get all the available worker slots in the cluster.List<WorkerSlot>getAvailableSlots(SupervisorDetails supervisor)Return all the available slots on this supervisor.Set<String>getBlacklistedHosts()Get all of the hosts that are blacklisted.doublegetClusterTotalCpuResource()Get the total amount of CPU resources in cluster.Map<String,Double>getClusterTotalGenericResources()Get the total amount of generic resources (excluding CPU and memory) in cluster.doublegetClusterTotalMemoryResource()Get the total amount of memory resources in cluster.Map<String,Object>getConf()Get the nimbus configuration.List<String>getGreyListedSupervisors()StringgetHost(String supervisorId)Map a supervisor to a given host.INimbusgetINimbus()doublegetMinWorkerCpu()Map<String,List<ExecutorDetails>>getNeedsSchedulingComponentToExecutors(TopologyDetails topology)Get the component name to executor list for executors that need to be scheduled.Map<ExecutorDetails,String>getNeedsSchedulingExecutorToComponents(TopologyDetails topology)Get the executor to component name map for executors that need to be scheduled.Map<String,List<String>>getNetworkTopography()Get the network topography (rackId -> nodes in the rack).List<WorkerSlot>getNonBlacklistedAvailableSlots(List<String> blacklistedSupervisorIds)Get all the available worker slots in the cluster, that are not blacklisted.NormalizedResourceOffergetNonBlacklistedClusterAvailableResources(Collection<String> blacklistedSupervisorIds)Get the resources in the cluster that are available for scheduling.ResourceMetricsgetResourceMetrics()doublegetScheduledCpuForNode(String nodeId)Get the total cpu currently scheduled on a node.doublegetScheduledMemoryForNode(String nodeId)Get the total memory currently scheduled on a node.StringgetStatus(String topoId)Map<String,String>getStatusMap()Get all topology scheduler statuses.SupervisorDetailsgetSupervisorById(String nodeId)Get a specific supervisor with thenodeId.Map<String,SupervisorDetails>getSupervisors()Get all the supervisors.List<SupervisorDetails>getSupervisorsByHost(String host)Get all the supervisors on the specifiedhost.Map<String,SupervisorResources>getSupervisorsResourcesMap()Get the amount of used and free resources on a supervisor.TopologiesgetTopologies()Get all of the topologies.Map<String,TopologyResources>getTopologyResourcesMap()Get the amount of resources used by topologies.Collection<ExecutorDetails>getUnassignedExecutors(TopologyDetails topology)get the unassigned executors of the topology.Set<Integer>getUsedPorts(SupervisorDetails supervisor)Get all the used ports of this supervisor.Collection<WorkerSlot>getUsedSlots()Get all currently occupied slots.Collection<WorkerSlot>getUsedSlotsByTopologyId(String topologyId)get slots used by a topology.WorkerResourcesgetWorkerResources(WorkerSlot ws)Get the resources for a given slot.Map<String,Map<WorkerSlot,WorkerResources>>getWorkerResourcesMap()Gets the reference to the full topology->worker resource map.booleanisBlackListed(String supervisorId)Check is a given supervisor is on a blacklisted host.booleanisBlacklistedHost(String host)Check if a given host is blacklisted.booleanisSlotOccupied(WorkerSlot slot)Check if a slot is occupied or not.booleanneedsScheduling(TopologyDetails topology)Does the topology need scheduling.booleanneedsSchedulingRas(TopologyDetails topology)LikeISchedulingState.needsScheduling(TopologyDetails)but does not take into account the number of workers requested.List<TopologyDetails>needsSchedulingTopologies()Get all of the topologies that need scheduling.voidsetAssignments(Map<String,? extends SchedulerAssignment> newAssignments, boolean ignoreSingleExceptions)Set assignments for cluster.voidsetBlacklistedHosts(Set<String> hosts)Set the list of hosts that are blacklisted.voidsetGreyListedSupervisors(Set<String> greyListedSupervisors)voidsetNetworkTopography(Map<String,List<String>> networkTopography)voidsetStatus(String topologyId, String statusMessage)set scheduler status for a topology.voidsetStatus(TopologyDetails td, String statusMessage)set scheduler status for a topology.voidsetStatusIfAbsent(String topologyId, String statusMessage)voidsetStatusMap(Map<String,String> statusMap)set scheduler status map.voidunassign(String topoId)Unassign everything for the given topology id.voidupdateFrom(Cluster other)Update the assignments and status from the other cluster.booleanwouldFit(WorkerSlot ws, ExecutorDetails exec, TopologyDetails td, NormalizedResourceOffer resourcesAvailable, double maxHeap)Would scheduling exec on ws fit? With a heap <= maxHeap total memory added <= memoryAvailable and cpu added <= cpuAvailable.-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface org.apache.storm.scheduler.ISchedulingState
getAssignedRacks, getHostToRack
-
-
-
-
Constructor Detail
-
Cluster
public Cluster(INimbus nimbus, ResourceMetrics resourceMetrics, Map<String,SupervisorDetails> supervisors, Map<String,? extends SchedulerAssignment> assignments, Topologies topologies, Map<String,Object> conf)
-
Cluster
public Cluster(Cluster src)
Copy constructor.
-
Cluster
public Cluster(Cluster src, Topologies topologies)
Testing Constructor that takes an existing cluster and replaces the topologies in it.- Parameters:
src- the original clustertopologies- the new topolgoies to use
-
-
Method Detail
-
getAssignedMemoryForSlot
public static double getAssignedMemoryForSlot(Map<String,Object> topConf)
Get heap memory usage for a worker's main process and logwriter process.- Parameters:
topConf- - the topology config- Returns:
- the assigned memory (in MB)
-
assertValidTopologyForModification
protected void assertValidTopologyForModification(String topologyId)
Check if the given topology is allowed for modification right now. If not throw an IllegalArgumentException else go on.- Parameters:
topologyId- the id of the topology to check
-
getTopologies
public Topologies getTopologies()
Description copied from interface:ISchedulingStateGet all of the topologies.- Specified by:
getTopologiesin interfaceISchedulingState- Returns:
- all of the topologies that are a part of the cluster.
-
getBlacklistedHosts
public Set<String> getBlacklistedHosts()
Description copied from interface:ISchedulingStateGet all of the hosts that are blacklisted.- Specified by:
getBlacklistedHostsin interfaceISchedulingState- Returns:
- all of the hosts that are blacklisted
-
setBlacklistedHosts
public void setBlacklistedHosts(Set<String> hosts)
Set the list of hosts that are blacklisted.- Parameters:
hosts- the new hosts that are blacklisted.
-
blacklistHost
public void blacklistHost(String host)
-
isBlackListed
public boolean isBlackListed(String supervisorId)
Description copied from interface:ISchedulingStateCheck is a given supervisor is on a blacklisted host.- Specified by:
isBlackListedin interfaceISchedulingState- Parameters:
supervisorId- the id of the supervisor- Returns:
- true if it is else false
-
isBlacklistedHost
public boolean isBlacklistedHost(String host)
Description copied from interface:ISchedulingStateCheck if a given host is blacklisted.- Specified by:
isBlacklistedHostin interfaceISchedulingState- Parameters:
host- the name of the host- Returns:
- true if it is else false.
-
getHost
public String getHost(String supervisorId)
Description copied from interface:ISchedulingStateMap a supervisor to a given host.- Specified by:
getHostin interfaceISchedulingState- Parameters:
supervisorId- the id of the supervisor- Returns:
- the actual host name the supervisor is on
-
needsSchedulingTopologies
public List<TopologyDetails> needsSchedulingTopologies()
Description copied from interface:ISchedulingStateGet all of the topologies that need scheduling.- Specified by:
needsSchedulingTopologiesin interfaceISchedulingState- Returns:
- all of the topologies that are not fully scheduled.
-
needsScheduling
public boolean needsScheduling(TopologyDetails topology)
Description copied from interface:ISchedulingStateDoes the topology need scheduling.A topology needs scheduling if one of the following conditions holds:
- Although the topology is assigned slots, but is squeezed. i.e. the topology is assigned less slots than desired.
- There are unassigned executors in this topology
- Specified by:
needsSchedulingin interfaceISchedulingState
-
needsSchedulingRas
public boolean needsSchedulingRas(TopologyDetails topology)
Description copied from interface:ISchedulingStateLikeISchedulingState.needsScheduling(TopologyDetails)but does not take into account the number of workers requested. This is because the number of workers is ignored in RAS- Specified by:
needsSchedulingRasin interfaceISchedulingState- Parameters:
topology- the topology to check- Returns:
- true if the topology needs scheduling else false.
-
getNeedsSchedulingExecutorToComponents
public Map<ExecutorDetails,String> getNeedsSchedulingExecutorToComponents(TopologyDetails topology)
Description copied from interface:ISchedulingStateGet the executor to component name map for executors that need to be scheduled.- Specified by:
getNeedsSchedulingExecutorToComponentsin interfaceISchedulingState- Parameters:
topology- the topology this is for- Returns:
- a executor -> component-id map which needs scheduling in this topology.
-
getNeedsSchedulingComponentToExecutors
public Map<String,List<ExecutorDetails>> getNeedsSchedulingComponentToExecutors(TopologyDetails topology)
Description copied from interface:ISchedulingStateGet the component name to executor list for executors that need to be scheduled.- Specified by:
getNeedsSchedulingComponentToExecutorsin interfaceISchedulingState- Parameters:
topology- the topology this is for- Returns:
- a component-id -> executors map which needs scheduling in this topology.
-
getUsedPorts
public Set<Integer> getUsedPorts(SupervisorDetails supervisor)
Description copied from interface:ISchedulingStateGet all the used ports of this supervisor.- Specified by:
getUsedPortsin interfaceISchedulingState
-
getAvailablePorts
public Set<Integer> getAvailablePorts(SupervisorDetails supervisor)
Description copied from interface:ISchedulingStateReturn the available ports of this supervisor.- Specified by:
getAvailablePortsin interfaceISchedulingState
-
getAssignablePorts
public Set<Integer> getAssignablePorts(SupervisorDetails supervisor)
Description copied from interface:ISchedulingStateGet the ports that are not blacklisted.- Specified by:
getAssignablePortsin interfaceISchedulingState- Parameters:
supervisor- the supervisor- Returns:
- the ports that are not blacklisted
-
getNonBlacklistedAvailableSlots
public List<WorkerSlot> getNonBlacklistedAvailableSlots(List<String> blacklistedSupervisorIds)
Description copied from interface:ISchedulingStateGet all the available worker slots in the cluster, that are not blacklisted.- Specified by:
getNonBlacklistedAvailableSlotsin interfaceISchedulingState- Parameters:
blacklistedSupervisorIds- list of supervisor ids that should also be considered blacklisted.
-
getAvailableSlots
public List<WorkerSlot> getAvailableSlots()
Description copied from interface:ISchedulingStateGet all the available worker slots in the cluster.- Specified by:
getAvailableSlotsin interfaceISchedulingState
-
getAvailableSlots
public List<WorkerSlot> getAvailableSlots(SupervisorDetails supervisor)
Description copied from interface:ISchedulingStateReturn all the available slots on this supervisor.- Specified by:
getAvailableSlotsin interfaceISchedulingState
-
getAssignableSlots
public List<WorkerSlot> getAssignableSlots(SupervisorDetails supervisor)
Description copied from interface:ISchedulingStateReturn all non-blacklisted slots on this supervisor.- Specified by:
getAssignableSlotsin interfaceISchedulingState- Parameters:
supervisor- the supervisor- Returns:
- the non-blacklisted slots
-
getAssignableSlots
public List<WorkerSlot> getAssignableSlots()
Description copied from interface:ISchedulingStateGet all non-blacklisted slots in the cluster.- Specified by:
getAssignableSlotsin interfaceISchedulingState
-
getUnassignedExecutors
public Collection<ExecutorDetails> getUnassignedExecutors(TopologyDetails topology)
Description copied from interface:ISchedulingStateget the unassigned executors of the topology.- Specified by:
getUnassignedExecutorsin interfaceISchedulingState- Parameters:
topology- the topology to check- Returns:
- the unassigned executors of the topology.
-
getAssignedNumWorkers
public int getAssignedNumWorkers(TopologyDetails topology)
Description copied from interface:ISchedulingStateGet the number of workers assigned to a topology.- Specified by:
getAssignedNumWorkersin interfaceISchedulingState- Parameters:
topology- the topology this is for- Returns:
- the number of workers assigned to this topology.
-
getAvailableResources
public NormalizedResourceOffer getAvailableResources(SupervisorDetails sd)
Description copied from interface:ISchedulingStateGet the resources on the supervisor that are available to be scheduled.- Specified by:
getAvailableResourcesin interfaceISchedulingState- Parameters:
sd- the supervisor.- Returns:
- the resources available to be scheduled.
-
wouldFit
public boolean wouldFit(WorkerSlot ws, ExecutorDetails exec, TopologyDetails td, NormalizedResourceOffer resourcesAvailable, double maxHeap)
Description copied from interface:ISchedulingStateWould scheduling exec on ws fit? With a heap <= maxHeap total memory added <= memoryAvailable and cpu added <= cpuAvailable.- Specified by:
wouldFitin interfaceISchedulingState- Parameters:
ws- the slot to put it inexec- the executor to investigatetd- the topology detains for this executorresourcesAvailable- all the available resourcesmaxHeap- the maximum heap size for ws- Returns:
- true it fits else false
-
assign
public void assign(WorkerSlot slot, String topologyId, Collection<ExecutorDetails> executors)
Assign the slot to the executors for this topology.- Throws:
RuntimeException- if the specified slot is already occupied.
-
assign
public void assign(SchedulerAssignment assignment, boolean ignoreSingleExceptions)
Assign everything for the given topology.- Parameters:
assignment- the new assignment to make
-
freeSlot
public void freeSlot(WorkerSlot slot)
Free the specified slot.- Parameters:
slot- the slot to free
-
freeSlots
public void freeSlots(Collection<WorkerSlot> slots)
free the slots.- Parameters:
slots- multiple slots to free
-
isSlotOccupied
public boolean isSlotOccupied(WorkerSlot slot)
Description copied from interface:ISchedulingStateCheck if a slot is occupied or not.- Specified by:
isSlotOccupiedin interfaceISchedulingState- Parameters:
slot- the slot be to checked.- Returns:
- true if the specified slot is occupied.
-
getAssignmentById
public SchedulerAssignment getAssignmentById(String topologyId)
Description copied from interface:ISchedulingStateget the current assignment for the topology.- Specified by:
getAssignmentByIdin interfaceISchedulingState
-
getUsedSlotsByTopologyId
public Collection<WorkerSlot> getUsedSlotsByTopologyId(String topologyId)
Description copied from interface:ISchedulingStateget slots used by a topology.- Specified by:
getUsedSlotsByTopologyIdin interfaceISchedulingState
-
getSupervisorById
public SupervisorDetails getSupervisorById(String nodeId)
Description copied from interface:ISchedulingStateGet a specific supervisor with thenodeId.- Specified by:
getSupervisorByIdin interfaceISchedulingState
-
getUsedSlots
public Collection<WorkerSlot> getUsedSlots()
Description copied from interface:ISchedulingStateGet all currently occupied slots.- Specified by:
getUsedSlotsin interfaceISchedulingState
-
getSupervisorsByHost
public List<SupervisorDetails> getSupervisorsByHost(String host)
Description copied from interface:ISchedulingStateGet all the supervisors on the specifiedhost.- Specified by:
getSupervisorsByHostin interfaceISchedulingState- Parameters:
host- hostname of the supervisor- Returns:
- the
SupervisorDetailsobject.
-
getAssignments
public Map<String,SchedulerAssignment> getAssignments()
Description copied from interface:ISchedulingStateGet all the assignments.- Specified by:
getAssignmentsin interfaceISchedulingState
-
setAssignments
public void setAssignments(Map<String,? extends SchedulerAssignment> newAssignments, boolean ignoreSingleExceptions)
Set assignments for cluster.
-
getSupervisors
public Map<String,SupervisorDetails> getSupervisors()
Description copied from interface:ISchedulingStateGet all the supervisors.- Specified by:
getSupervisorsin interfaceISchedulingState
-
getNonBlacklistedClusterAvailableResources
public NormalizedResourceOffer getNonBlacklistedClusterAvailableResources(Collection<String> blacklistedSupervisorIds)
Description copied from interface:ISchedulingStateGet the resources in the cluster that are available for scheduling.- Specified by:
getNonBlacklistedClusterAvailableResourcesin interfaceISchedulingState- Parameters:
blacklistedSupervisorIds- other ids that are tentatively blacklisted.
-
getClusterTotalCpuResource
public double getClusterTotalCpuResource()
Description copied from interface:ISchedulingStateGet the total amount of CPU resources in cluster.- Specified by:
getClusterTotalCpuResourcein interfaceISchedulingState
-
getClusterTotalMemoryResource
public double getClusterTotalMemoryResource()
Description copied from interface:ISchedulingStateGet the total amount of memory resources in cluster.- Specified by:
getClusterTotalMemoryResourcein interfaceISchedulingState
-
getClusterTotalGenericResources
public Map<String,Double> getClusterTotalGenericResources()
Description copied from interface:ISchedulingStateGet the total amount of generic resources (excluding CPU and memory) in cluster.- Specified by:
getClusterTotalGenericResourcesin interfaceISchedulingState
-
getNetworkTopography
public Map<String,List<String>> getNetworkTopography()
Description copied from interface:ISchedulingStateGet the network topography (rackId -> nodes in the rack).- Specified by:
getNetworkTopographyin interfaceISchedulingState
-
setStatus
public void setStatus(TopologyDetails td, String statusMessage)
set scheduler status for a topology.
-
setStatus
public void setStatus(String topologyId, String statusMessage)
set scheduler status for a topology.
-
getStatusMap
public Map<String,String> getStatusMap()
Description copied from interface:ISchedulingStateGet all topology scheduler statuses.- Specified by:
getStatusMapin interfaceISchedulingState
-
getTopologyResourcesMap
public Map<String,TopologyResources> getTopologyResourcesMap()
Description copied from interface:ISchedulingStateGet the amount of resources used by topologies. Used for displaying resource information on the UI.- Specified by:
getTopologyResourcesMapin interfaceISchedulingState- Returns:
- a map that contains multiple topologies and the resources the topology requested and assigned. Key: topology id Value: an array that describes the resources the topology requested and assigned in the following format: {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, assignedMemOnHeap, assignedMemOffHeap, assignedCpu}
-
getSupervisorsResourcesMap
public Map<String,SupervisorResources> getSupervisorsResourcesMap()
Description copied from interface:ISchedulingStateGet the amount of used and free resources on a supervisor. Used for displaying resource information on the UI- Specified by:
getSupervisorsResourcesMapin interfaceISchedulingState- Returns:
- a map where the key is the supervisor id and the value is a map that represents resource usage for a supervisor in the following format: {totalMem, totalCpu, usedMem, usedCpu}
-
getWorkerResourcesMap
public Map<String,Map<WorkerSlot,WorkerResources>> getWorkerResourcesMap()
Description copied from interface:ISchedulingStateGets the reference to the full topology->worker resource map.- Specified by:
getWorkerResourcesMapin interfaceISchedulingState- Returns:
- map of topology -> map of worker slot ->resources for that worker
-
getWorkerResources
public WorkerResources getWorkerResources(WorkerSlot ws)
Description copied from interface:ISchedulingStateGet the resources for a given slot.- Specified by:
getWorkerResourcesin interfaceISchedulingState- Parameters:
ws- the slot- Returns:
- the resources currently assigned
-
getResourceMetrics
public ResourceMetrics getResourceMetrics()
-
getAllScheduledResourcesForNode
public NormalizedResourceRequest getAllScheduledResourcesForNode(String nodeId)
Description copied from interface:ISchedulingStateGet all scheduled resources for node.- Specified by:
getAllScheduledResourcesForNodein interfaceISchedulingState
-
getScheduledMemoryForNode
public double getScheduledMemoryForNode(String nodeId)
Description copied from interface:ISchedulingStateGet the total memory currently scheduled on a node.- Specified by:
getScheduledMemoryForNodein interfaceISchedulingState- Parameters:
nodeId- the id of the node- Returns:
- the total memory currently scheduled on the node
-
getScheduledCpuForNode
public double getScheduledCpuForNode(String nodeId)
Description copied from interface:ISchedulingStateGet the total cpu currently scheduled on a node.- Specified by:
getScheduledCpuForNodein interfaceISchedulingState- Parameters:
nodeId- the id of the node- Returns:
- the total cpu currently scheduled on the node
-
getINimbus
public INimbus getINimbus()
-
getConf
public Map<String,Object> getConf()
Description copied from interface:ISchedulingStateGet the nimbus configuration.- Specified by:
getConfin interfaceISchedulingState
-
unassign
public void unassign(String topoId)
Unassign everything for the given topology id.- Parameters:
topoId- the is of the topology to unassign
-
updateFrom
public void updateFrom(Cluster other)
Update the assignments and status from the other cluster.- Parameters:
other- the cluster to get the assignments and status from
-
getMinWorkerCpu
public double getMinWorkerCpu()
-
-