wiki:schedule

Version 11 (modified by liaojiaohe, 14 years ago) (diff)

--

现在我们使用的是Capacity Scheduler ,在下面的场景存在问题:

当有大任务(晚上定向广告,数据组多数据查询) 占用主队列(default)时,一些比较快的定时任务会阻塞

设想方案

  • 研究Capacity Scheduler 参数
  • 如果Capacity Scheduler 不能慢速,修改代码让系统把job分配到不同的queue

假设又两条queue:default,highPrior,当default被占满后,而highPrior比较空闲[[BR]]

优先级是HIGH的会被分配到highPrior队列

修改JobQueueManager,jobAdded 函数

  @Override
  public void jobAdded(JobInProgress job) throws IOException {
    LOG.info("Job " + job.getJobID() + " submitted to queue " + 
        job.getProfile().getQueueName());
    
    // add job to the right queue
    CapacitySchedulerQueue queue = getQueue(job.getProfile().getQueueName());
    if (null == queue) {
      // job was submitted to a queue we're not aware of
      LOG.warn("Invalid queue " + job.getProfile().getQueueName() + 
          " specified for job" + job.getProfile().getJobID() + 
          ". Ignoring job.");
      return;
    }
    // add job to waiting queue. It will end up in the right place, 
    // based on priority. 
    queue.addWaitingJob(job);
    // let scheduler know. 
    scheduler.jobAdded(job);
  }

Attachments