Volcano Resource Reservation For Queue

@qiankunli; Oct 11rd, 2021

Motivation

In my case, we use volcano to schedule training job(tfjob/pytorchjob/vcjob) in k8s cluster, there are many groups such as ad/recommend/tts, a queue represents a group. In order to ensure the full utilization of resources, we generally do not configure queue.capability. but this will cause a queue to running out all resources, and make the new job of other queue unable to execute. so we want to reserve some resources for a queue, so that any new job in the queue can be submitted immediately.

As issue 1101 mentioned, Volcano should support resource reservation for specified queue. Requirement detail as follows:

• Support reserving specified resources for specified queue
• We only Consider non-preemption reservation.
• Support enable and disable resource reservation for specified queue dynamically without restarting Volcano.
• Support hard reservation resource specified and percentage reservation resource specified.

@Thor-wl already provide a design doc Volcano Resource Reservation For Queue I do not implement all features above, supported feature are as follows:

• Support reserving specified resources for specified queue
• We only Consider non-preemption reservation.
• Support enable and disable resource reservation for specified queue dynamically without restarting Volcano.
• Support hard reservation resource specified

Consideration

Resource Request

• The reserved resource cannot be more than the total resource amount of cluster at all dimensions.
• If capability is set in a queue, the reserved resource must be no more than it at all dimensions.

Safety

• Malicious application for large amount of resource reservation will cause jobs in other queue to block.

Design

API

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: q1
spec:
  reclaimable: true
  weight: 1
  guarantee:             // reservation key word
    resource:            // specified reserving resource
      cpu: 2c
      memory: 4G

guarantee.resource list of reserving resource categories and amount.

Implementation

In order to support guarantee mechanism, there are two scenarios to consider

1. support spec.guarantee during scheduling
2. create a new queue whose spec.guarantee is not nil or an existed queue's spec.guarantee becomes bigger

support spec.guarantee during scheduling

if there are three queues and 30 GPUs in cluster.

queue/attr	guarantee GPUs	capability GPUs	realCapability GPUs
queue1	5	nil	30
queue2	nil	nil	25
queue3	nil	10	10

// /volcano/pkg/scheduler/plugins/proportion/proportion.go
type queueAttr struct {
	queueID api.QueueID
	name    string
	
	deserved  *api.Resource
	allocated *api.Resource
	request   *api.Resource
	// inqueue represents the resource request of the inqueue job
	inqueue    *api.Resource
	capability *api.Resource
	realCapability *api.Resource
	guarantee      *api.Resource
}

on each schedule cycle, proportion plugin will calculate queueAttr.deserved for a queue which means how many resources the queue can use. when consider a new task, if queueAttr.deserved is bigger than queueAttr.allocated, the new task can be scheduled.

1. queueAttr.deserved must be bigger than queueAttr.guarantee
2. if queueAttr.guarantee is not nil(like queue1), it means the 5 GPUs only can be used by queue1 even there is no job running in queue1. we use queueAttr.realCapability to represent the upper limit resources that a queue can use.
   1. if queueAttr.capability is nil(like queue2), realCapability = total resources - sum(other-queue.guarantee)
   2. if queueAttr.capability is not nil(like queue3), realCapability = min(capability,total resources - sum(other-queue.guarantee))
3. replace queueAttr.capability with queueAttr.realCapability everywhere

After doing this, a queue owns the resources which is bigger than queueAttr.guarantee and less than queueAttr.realCapability

create a new queue whose spec.guarantee is not nil

if there are three queues and 30 GPUs in cluster, there are many task in queue1/queue2/queue3 and running out the 30 GPUs,

queue/attr	weight	deserved GPUs	guarantee GPUs	capability GPUs	realCapability GPUs
queue1	1	10	5	nil	30
queue2	1	10	nil	nil	25
queue3	1	10	nil	10	10

then we create queue4 and submit a new job(request 2GPUs) in queue4

queue/attr	weight	deserved GPUs	guarantee GPUs	capability GPUs	realCapability GPUs
queue1	1	6	5	nil	20
queue2	1	6	nil	nil	15
queue3	1	6	nil	10	10
queue4	2	12	10	nil	25

1. the overcommit plugin will deny the new job in queue4 because there is no free GPUs in cluster. so,we should change the logic, if job.request < queue4.guarantee, the job can be Inqueue whether there are free GPUs or not.
2. we should enable the reclaim action, so that volcano can reclaim the task in overused queue

Usage

Configure guarantee for queue

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: q1
spec:
  reclaimable: true
  weight: 1
  guarantee:             // reservation key word
    resource:            // specified reserving resource
      cpu: 2c
      memory: 4G

Enable reclaim action for scheduler.

apiVersion: v1
kind: ConfigMap
metadata:
  name: volcano-scheduler-configmap
  namespace: volcano-system
data:
  volcano-scheduler.conf: |
    actions: "enqueue,allocate,reclaim,backfill"
    tiers:
    - plugins:
      - name: priority
      - name: gang
      - name: conformance
    - plugins:
      - name: overcommit
      - name: drf
      - name: predicates
      - name: proportion
      - name: nodeorder
      - name: binpack