| Version 11 (modified by liaojiaohe, 14 years ago) (diff) |
|---|
现在我们使用的是Capacity Scheduler ,在下面的场景存在问题:
当有大任务(晚上定向广告,数据组多数据查询) 占用主队列(default)时,一些比较快的定时任务会阻塞
设想方案
- 研究Capacity Scheduler 参数
- 如果Capacity Scheduler 不能慢速,修改代码让系统把job分配到不同的queue
假设又两条queue:default,highPrior,当default被占满后,而highPrior比较空闲[[BR]]
优先级是HIGH的会被分配到highPrior队列
修改JobQueueManager,jobAdded 函数
@Override
public void jobAdded(JobInProgress job) throws IOException {
LOG.info("Job " + job.getJobID() + " submitted to queue " +
job.getProfile().getQueueName());
// add job to the right queue
CapacitySchedulerQueue queue = getQueue(job.getProfile().getQueueName());
if (null == queue) {
// job was submitted to a queue we're not aware of
LOG.warn("Invalid queue " + job.getProfile().getQueueName() +
" specified for job" + job.getProfile().getJobID() +
". Ignoring job.");
return;
}
// add job to waiting queue. It will end up in the right place,
// based on priority.
queue.addWaitingJob(job);
// let scheduler know.
scheduler.jobAdded(job);
}
Attachments
-
src.zip
(38.5 KB) -
added by liaojiaohe 13 years ago.
-
hadoop-capacity-scheduler-1.0.3.jar
(63.9 KB) -
added by liaojiaohe 13 years ago.
![(please configure the [header_logo] section in trac.ini)](http://www1.pconline.com.cn/hr/2009/global/images/logo.gif)