jsp编译内存不够问题

[chenchongqi]

现象

8.27报价库前台更新jsp的时候，237.45这台虚拟报编译错误：

[08-27 15:27:29.812] com.caucho.java.JavaCompileException: Resin can't execute the compiler `/bin/sh'.  This usually means that the compiler is not in the op
erating system's PATH or the compiler is incorrectly specified in the configuration.  You may need to add the full path to <java compiler='/bin/sh'/>.
[08-27 15:27:29.812]
[08-27 15:27:29.812] java.io.IOException: Cannot run program "/bin/sh": java.io.IOException: error=12, Cannot allocate memory
[08-27 15:27:29.812]    at com.caucho.java.ExternalCompiler.executeCompiler(ExternalCompiler.java:435)
[08-27 15:27:29.812]    at com.caucho.java.ExternalCompiler.compileInt(ExternalCompiler.java:151)
[08-27 15:27:29.812]    at com.caucho.java.AbstractJavaCompiler.run(AbstractJavaCompiler.java:102)
[08-27 15:27:29.812]    at java.lang.Thread.run(Thread.java:619)

被监控脚本重启时的系统状态：

system stat:
top - 11:52:19 up 38 days, 5 min,  2 users,  load average: 1.23, 1.51, 2.00
Tasks:  84 total,   1 running,  82 sleeping,   1 stopped,   0 zombie
Cpu(s): 14.9%us,  1.2%sy,  0.0%ni, 76.9%id,  1.2%wa,  2.1%hi,  3.6%si,  0.0%st
Mem:   5072544k total,  5025172k used,    47372k free,     5468k buffers
Swap:  2096472k total,    10868k used,  2085604k free,   653936k cached
        
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
19153 root      21   0 4515m 4.0g 9.8m S 99.2 82.9 212:19.36 java
    1 root      15   0 10352  580  544 S  0.0  0.0   0:06.58 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:40.73 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.54 ksoftirqd/0
    4 root      RT  -5     0    0    0 S  0.0  0.0   1:05.93 migration/1
    5 root      34  19     0    0    0 S  0.0  0.0   0:00.50 ksoftirqd/1
    6 root      RT  -5     0    0    0 S  0.0  0.0   1:13.14 migration/2
    7 root      34  19     0    0    0 S  0.0  0.0   0:00.48 ksoftirqd/2
    8 root      RT  -5     0    0    0 S  0.0  0.0   1:11.65 migration/3
    9 root      34  19     0    0    0 S  0.0  0.0   0:00.52 ksoftirqd/3
   10 root      10  -5     0    0    0 S  0.0  0.0   0:22.16 events/0
   11 root      10  -5     0    0    0 S  0.0  0.0   0:05.40 events/1
   12 root      10  -5     0    0    0 S  0.0  0.0   0:05.29 events/2
   13 root      10  -5     0    0    0 S  0.0  0.0   0:05.33 events/3
   14 root      10  -5     0    0    0 S  0.0  0.0   0:00.28 khelper
   63 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kthread
   70 root      10  -5     0    0    0 S  0.0  0.0   0:56.25 kblockd/0
   71 root      10  -5     0    0    0 S  0.0  0.0   0:13.40 kblockd/1
   72 root      10  -5     0    0    0 S  0.0  0.0   0:13.51 kblockd/2
   73 root      10  -5     0    0    0 S  0.0  0.0   0:13.39 kblockd/3 
   74 root      14  -5     0    0    0 S  0.0  0.0   0:00.00 kacpid
  307 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/0
  308 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/1
  309 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/2
  310 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/3
  313 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 khubd
  315 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kseriod
  407 root      10  -5     0    0    0 S  0.0  0.0  30:25.70 kswapd0
  408 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0
  409 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 aio/1
  410 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 aio/2
  411 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 aio/3
  595 root      15   0  132m 3740 1560 S  0.0  0.1   0:43.84 python
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ 
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0  10868  47364   5468 653936    0    0   130   273    2    2 15  7 77  1  0
 1  0  10868  47460   5480 653908    0    0     0   177  987  188 25  1 74  0  0
 1  0  10868  46912   5480 653624    0    0     0     0  975  230 24  4 72  0  0
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT   
  0.00 100.00  77.18  53.45  98.76   3863  549.840    30   66.716  616.556
  0.00 100.00  95.12  53.45  99.71   3863  549.840    30   66.716  616.556
  0.00  64.81   0.00  54.63 100.00   3865  551.103    31   66.716  617.819

分析

• 老谢之前有过总结： http://bbs.pconline.cn/topic-175.html
• 现象上看是jsp编译的时候，操作系统分配不到内存
• jsp复杂度高导致编译需要的资源多
• 应用的压力大，导致jvm本身占用的内存多，操作系统的内存余量有影响

初步处理

• 237.45增加了1g内存
• 准备dump一次内存看看应用占内存的大头是哪部分，resin是否可以减少内存配置
• 测试环境模拟压力测试，看看能否重现并做动态监控和分析

长期目标

• 报价应用前台拆分：索引服务 + SSI服务
• 报价服务器拆分：静态文件 + mysql + squid