Status for Andrew DeFaria

As you know there have been service interruptions in CQWeb. I keep looking at the logs for clues. About the only consistent thing is an error similar to this:

2006-09-28 00:10:20 Ajp13Processor[8009][15] process: invoke
java.net.SocketException: Connection reset by peer: socket write error
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(Unknown Source)
        at java.net.SocketOutputStream.write(Unknown Source)
        at org.apache.ajp.Ajp13.send(Ajp13.java:525)
        at org.apache.ajp.RequestHandler.finish(RequestHandler.java:495)
        at org.apache.ajp.Ajp13.finish(Ajp13.java:395)
        at org.apache.ajp.tomcat4.Ajp13Response.finishResponse(Ajp13Response.java:196)
        at org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:464)
        at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:551)
        at java.lang.Thread.run(Unknown Source)

Now "Connection reset by peer" could be an error that the process gets because the service has stopped so this could be more of a symptom than a cure. However searching for "Ajp13Processor socket write error" points me to this post which suggests increasing the stack and heap sizes for the JVM. Problems that are intermittent can be consistent with running out of stack or heap size.<./p>

According to the Clearquest Web Administration Guide:

---

Controlling Java VM Memory Consumption

You can configure the memory consumption of Java processes used by New ClearQuest Web by adjusting the parameters in property files under the various components.

Windows

This section describes the configuration changes for New ClearQuest Web Java VM memory consumption for processes running on Microsoft Windows. To specify the VM memory consumption:

1. Open the appropriate configuration file for the New ClearQuest Web component whose memory consumption you want to reconfigure. For the ClearQuest Web application:

   Component	Configuration file
   Apache Tomcat Server	C:\Program Files\Rational\Common\rwp\bin\jk_service2.in.properties
   Rational Web Platform	C:\Program Files\Rational\Common\rwp\bin\jk_service2.properties

   
   

   For the ClearQuest server:

   Component	Configuration file
   ClearQuest Request Manager	C:\Program Files\Rational\ClearQuest\cqweb\cqserver\requestmgr_service.properties
   ClearQuest Registry Server	C:\Program Files\Rational\ClearQuest\cqweb\cqregsvr\cqregsvr_service.properties

2. Modify the section shown below:
   
   

   # # JVM Options #
   # Useful Options:
   # -Xms2m = Initial heap size, modify for desired size
   # -Xmx256m = Maximum heap size, modify for desired size
   # -Xrs = Available in Jdk1.3.1 to avoid JVM termination during logoff
   #
   wrapper.jvm.options=-Xrs -Xms2m -Xmx256m
       

---

I looked at these config files on the three machines (dfls83-85) and they were pretty much set to the default:

Ltx0062320:for server in 83 84 85; do grep
wrapper.jvm.options= //dfls$server/Rational/Common/rwp/bin/jk_service2*properties
//dfls$server/Rational/ClearQuest/cqweb/cqserver/requestmgr_service.properties
//dfls
$server/Rational/ClearQuest/cqweb/cqregsvr/cqregsvr_service.properties;
done
//dfls83/Rational/Common/rwp/bin/jk_service2.default.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls83/Rational/Common/rwp/bin/jk_service2.in.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls83/Rational/Common/rwp/bin/jk_service2.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls83/Rational/ClearQuest/cqweb/cqserver/requestmgr_service.properties:wrapper.jvm.options=-Xrs

//dfls83/Rational/ClearQuest/cqweb/cqregsvr/cqregsvr_service.properties:wrapper.jvm.options=-Xrs

//dfls84/Rational/Common/rwp/bin/jk_service2.default.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls84/Rational/Common/rwp/bin/jk_service2.in.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls84/Rational/Common/rwp/bin/jk_service2.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls84/Rational/ClearQuest/cqweb/cqserver/requestmgr_service.properties:wrapper.jvm.options=-Xrs

//dfls84/Rational/ClearQuest/cqweb/cqregsvr/cqregsvr_service.properties:wrapper.jvm.options=-Xrs

//dfls85/Rational/Common/rwp/bin/jk_service2.default.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls85/Rational/Common/rwp/bin/jk_service2.in.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls85/Rational/Common/rwp/bin/jk_service2.properties:wrapper.jvm.options=-Xrs
-Xms2m -Xmx256m

//dfls85/Rational/ClearQuest/cqweb/cqserver/requestmgr_service.properties:wrapper.jvm.options=-Xrs

//dfls85/Rational/ClearQuest/cqweb/cqregsvr/cqregsvr_service.properties:wrapper.jvm.options=-Xrs

 

All these machines have 2 gig of main memory and largely just serve CQWeb. Indeed the CQWeb service processes are consuming most of the memory:

dfls83
dfls84
dfls85

I think we should try setting at least the following:

wrapper.jvm.options=-Xrs -Xms128m -Xmx512m

A restart of all CQ Web services would probably be needed for the changes to become effective. The above settings start off the jvm @ 128m for all 4 processes thus a total memory footprint of 512 Meg and limit each process to 512 Meg max for a total footprint of 2 Gig (when full). We might want to bounce this idea off IBM/Rational support to see if all 4 process should have the same settings or if we show vary them.