Load Balancing CQ Web Servers based on Number of CQ Web Users
The task at hand was to write a redirector that load balances amongst a number of CQ Web servers based on the number of CQ Web Users currently on each server. Additionally, based on how the user came into the CQ Web server farm, redirect them to the proper schema.
Determining Load
The old IIS CQ Web Server used to allow you to query the number of active CQ Web Users. The new Apache/Tomcat server only allows admins to do this. Additionally the admin need to be logged in, thus have a valid token. IBM/Rational suggests using Apache's server-status URL to determine load. However that only displays number of Apache requests in progress not number of CQ Web Users.
If ExtendedStatus is turned on then Apache lists each connection and the URL they are working on. By filtering "GET /cqweb" we can get a rough estimate of the number of CQ Web Users. There is a problem in that the redirector script cannot query the same web server that it's running on. Additionally this information can only be obtained if ExtendedStatus is turned on.
Algorithm for selecting a server
The algorithm for selecting a non busy server is described as:
Pick a lightly loaded server out of the pool. Note that if a server is not running with ExtendedStatus on then $cq_users will be undef. This is different than the case where the server has ExtendedStatus on but there just aren't any CQ Web users (which would be denoted by $cq_users = 0). Thus we may have the condition where:
| Server |
cq_users |
ExtendedStatus |
| server1 |
undef |
off |
| server2 |
20 |
on |
| server3 |
0 |
on |
| server4 |
10 |
on |
In such a case we wish to pick server3 since it has no current CQ web users.
The algorithm used here will be to remove all servers from the pool who are not running with ExtendedStatus on since we cannot reliably tell how loaded the server is from the standpoint of CQ Web users. If, however, no servers have ExtendedStatus on (thus all $cq_users return as undef) then we will consider the $nbr_apache_requests. IOW $nbr_apache_requests is not equivalent with $cq_users and thus they cannot be compared together. But if no server is running with ExtendedStatus on we need to pick something!
Note: If $random then a server is simply randomly chosen.
Unfortunately, given this algorithm, if we had the following situation:
| Server |
cq_users |
ExtendedStatus |
| server1 |
undef |
on |
| server2 |
undef |
off |
| server3 |
undef |
off |
| server4 |
10 |
on |
Then this algorithm will always return server4.
Important Note: The web server doing the redirection cannot be queried. Attempting to do so hangs! Therefore it cannot participate in the server pool. It is recommended that another web server be set up as the redirector and the DNS name cqweb assigned to it. This web server can, however, participate by being a Clearquest Request Manager.
Random Redirection
The script can also redirect randomly instead of relying on load of CQ Web Users. Currently there are 3 servers in the pool. Only one of them has ExtendedStatus turned on. As such redirecting by load will always resolve to the one server using, the one running with ExtendedStatus on. This is not good. So currently it just picks a server randomly from the pool. This behavior is controlled by the lb parameter (currently defaulted to off meaning pick server randomly).
Defining the Server Pool
The server pool is defined by a small file, servers.cfg, which simply list the servers participating in the pool. Servers can be added or removed dynamically.
Mapping Redirection
In the past users went to http://cqweb/<area>. These were HTML files in the DocumentRoot which redirected to a series of redirection scripts. It was hoped that HTTP_REFERER could be used to determine where to redirect the visitor. Unfortunately HTTP_REFERER is not guaranteed and indeed it's undefined on the web servers!
Instead one must specify the group parameter to the redirector script. The script then maintains a map between <areas> -> Schema/ContextIDs. If the group is not specified or not in the map then the user is redirected to the main login page. This is not viewed as a hardship because we need redirecting <area> files anyway. The new form of redirecting <area> file is:
<html>
<head>
<mdeta http-equiv="refresh" content="0; url=http://cqweb.itg.ti.com/cgi-bin/redirect.pl?group=<area>>
</head>
Redirect Map
The redirect map, stored in redirect.map, is a file of key/value pairs. For example:
CMDT: &schema=CMDT.2003.06.00&contextid=CMDT
CSSD: &schema=omap.2002.05.00&contextid=OMAPS
DLP-Play: &schema=DLP.2003.06.00&contextid=Play
DLP: &schema=DLP.2003.06.00&contextid=DLP
DMD-p: &schema=DLP.2003.06.00&contextid=DMD-p
DMD: &schema=DLP.2003.06.00&contextid=DMD
GCM: &schema=CMDT.2003.06.00&contextid=GCM
HPALP: &schema=HPA_MKT_LP&contextid=HPALP
LDM: &schema=CMDT.2003.06.00&contextid=LDM
NV: &schema=CMDT.2003.06.00&contextid=NV
SDO: &schema=SDS.2003.06.00&contextid=SDSCM
SDO_TEST: &schema=SDS_TST_DEV&contextid=SDSCM
WiMax: &schema=WiMax.SR5&contextid=WiMax
mDTV: &schema=mDTV.2003.06.00&contextid=MDTV
mDTV_play: &schema=mDTV.2003.06.00&contextid=PLAY
Parameters for redirect.pl
The following parameters, specified in the URL, are supported by redirect.pl:
- group
- Specifies the key into the redirect.map for the schema/contextid. If not specified then defaults to main login page of the selected server
- lb
- If set then load balancing is attempted based on ExtendedStatus and CQ Web Users as described above. Default: undefined (off)
- debug
- If specified the user is not redirected rather debugging information is output.