LDN - Luminis Developer Network

I decided that I really needed to cluster the resource tier of Luminis, especially if we were going to start looking at closing other avenues to resources off and forcing people to go through Luminis on our campus. So I found some test hardware and decided to have at it. This is installed with Luminis III.2 as a base platform to test failover, then I’ve also upgraded to III.3.1 patch 21 to see how patching in a clustered environment works. Its accomplished by using a couple of Sun Solaris Boxes (V240’s) and Solaris 9 with Sun Cluster 3.1. I won’t go into too much detail about Sun cluster, as I’ll assume if you are attempting this you are familiar with it, but as always I’m happy to answer any questions. :)

1. Install & patch Solaris on both nodes.
2. Install & patch Sun Cluster on both nodes (after this step you should have a two node cluster)
3. On our systems, luminis is installed in /opt/pipeline. For convenience, I’ve soft-linked /opt/pipeline to the global storage where luminis will ultimately reside (/global/luminis). This soft-link should exist on both nodes.
4. Create a resource group for luminis (i.e. luminis-rg). This can be accomplished with the command:
# scrgadm –a –g luminis-rg –y nodelist=node1,node2

5. Add the logical hostname to the resource and bring it online on node 1. This can be done with the commands:
# scrgadm –a –t SUNW.LogicalHost
# scrgadm –a –L –g luminis-rg –j logicalhostname
# scswitch –Z –g luminis-rg

6. Add the necessary users and groups for the luminis install (typical install stuff).
7. Install Luminis III.2 on Node 1 making sure your custom.conf uses the logical hostname and not the physical hostname and make sure you install to global storage that is accessible to both node 1 and node 2. (i.e. /global/luminis)
8. After the install is done, make sure you can bring it up on that node.
9. If everything looks good, go ahead and bring luminis down on node 1.
10. Fail the logical hostname over to Node 2 with the command:
# scswitch –z –g luminis-rg –h node2

11. Install Luminis III.2 on Node 2 making sure your custom.conf uses the logical hostname and not the physical hostname and make sure you install to global storage that is accessible to both node 1 and node 2. (i.e. /global/luminis) The global storage here can be the same as node 1. It will just overwrite the stuff that’s in global (this should still be a base install anyway, so no biggie there).
12. Again bring up luminis on node 2 and make sure everything is ok. If it looks good, bring it down again.
13. At this point you should have /global/luminis filled with the portal, but there are pieces that get installed outside of /opt/pipeline. From either node 1 or node 2 you should copy /var/opt/SUNWics5 to global storage somewhere. We used /global/luminis/var/opt/SUNWics5. We tried to keep the directory structure similar.
14. After you have the calendaring stuff copied to global storage, you need to create soft-links on both node 1 and node 2 from /var/opt/SUNWics5 to wherever you copied the directory to on global storage (i.e. /global/luminis/var/opt/SUNWics5)
15. At this point everything (except for message queue) is on global storage. I tried moving message queue to global storage, but I think its way to embedded in the OS to do so which is why we had to install luminis on both nodes.
16. Now you need to make calendar server use the logical name. For whatever reason all of luminis is happy using the logical hostname except for calendar server. It likes to use the physical hostname. So edit your ics.conf file (/global/luminis/products/SUNWics5/cal/bin/config/ics.conf) and change the following two lines:
local.servername = "node1" --> local.servername = “logicalhostname”
service.ens.host = "node1" --> service.ens.host = “logicalhostname”

Make sure that you keep the original lines in the file (just add a ! in front of them.) You’ll need to change these files back to perform any upgrades/patching.
17. You will also need to edit the ics.conf.bak that’s in the same directory and change the following entries (again keeping originals handy for upgrades):
local.hostname = "node1" --> local.hostname = “logicalhostname”
local.servername = "node1" --> local.servername = “logicalhostname”
service.ens.host = "node1" --> service.ens.host = “logicalhostname”
service.http.calendarhostname = "node1.myschool.edu" --> service.http.calendarhostname = “logicalhostname.myschool.edu”
18. In your web servers alias directory (/global/luminis/products/ws/alias) you have cert db files. You will need to make copies of all the db files that are in there so that there is a set for node 1 and a set for node 2. (i.e. You should have https-cp-node1-cert7.db and https-cp-node2-cert7.db for every db file in there, including the mb server.) A simple cp will work for this. When you renew certs for your portal, you will need to remember to copy these db files after you’ve installed the updated cert.
19. At this point, you can manually bring up the portal on either node manually. Add data to the portal while running on node 1 and then bring it down on node 1 and bring it up on node 2. The data you added should be there just as if you were still running on node 1.
20. We took this one step further and made the portal a resource within sun cluster. To do this, you can use the Generic Data Service cluster module for Sun Cluster. You give it a start script and a stop script (again these should be on global storage like /global/luminis/scripts/start.ksh or something similar.) Your portal HAS to be configured to use sudo for this to work. Sungard HE has a doc note on how to do this. You should remember to configure sudo on both nodes. Commands to do this are:
# scrgadm –a –t SUNW.gds
# scrgadm –a –j Portal –g luminis-rg –y Port_list=”80/tcp 636/tcp 443/tcp” –x Failover_enabled=true –x Start_command=”/global/luminis/scripts/cluster_agent_start.ksh” –x Stop_command=”/global/luminis/scripts/cluster_agent_stop.ksh”

The port_list is just what we ended up using. I think the GDS module just wants SOMETHING in there. It doesn’t matter much what we put in there because we turn off the pmf monitoring after the startup script is done anyway. There are many other options with the gds that you can tailor and customize to your environment including custom monitoring, etc.
21. With your startup script, I found I had to do a couple of things. I had to bring in all of cpadmins environment (.cprc and any customizations you’ve added). You also need to remember that cluster stuff runs as root. So in your startup script, all your normal lines (like startcp, runsesev, runsesau and runevents) need to be preceeded by su – cpadmin –c “” And finally with the startup script, because of the way luminis works, when the portal is done starting, the actually startup process pid goes away. Cluster doesn’t like this and thinks that the resource has failed. So you need to turn off pmf:
# pmfadm -s luminis-rg,Portal,0.svc HUP

luminis-rg is my resource group and Portal is the actual portal gds. Luminis-rg contains the logical hostname and the portal resource, and this is what actually fails over. You need to have the logical hostname on the same box that the portal is trying to run on, or it wont work, so it makes sense to group them together.
22. The stop script is much easier, you just put all your stop commands in there preceeded by the same su – cpadmin –c “”
23. After all this is done, you finally have a portal that is resource clustered. If you do a scstat –g you should see something like this:
# scstat -g

-- Resource Groups and Resources --

Group Name Resources
---------- ---------
Resources: luminis-rg logicalhostname Portal

-- Resource Groups --

Group Name Node Name State
---------- --------- -----
Group: luminis-rg Node1 Offline
Group: luminis-rg Node2 Online

-- Resources --

Resource Name Node Name State Status Message
------------- --------- ----- --------------
Resource: logicalhostname node1 Offline Offline - LogicalHostname offline.
Resource: logicalhostname node2 Online Online - LogicalHostname online.

Resource: Portal node1 Offline Offline
Resource: Portal node2 Online Online

24. A couple notes on upgrading when you are in this configuration:
-- You can leave your resource online, but remember to unmonitor it anytime you are trying to patch (or even manually restart) the portal. That way cluster is not trying to restart the portal while you are trying to take it down.
--Remember to undo sudo before you attempt to patch/upgrade
-- Remember to change back the ics.conf and ics.conf.back files to use the physical hostname
-- Remember when you update your certs, to copy the db files in the alias directory.
-- I’ve upgraded from III.2 to III.3.1 patch level 234209342 and it all seems to work quite well as far as only having to patch it once on one node and taking it taking the patches with it when it fails to the other node. So it’s a little more setup in the beginning but not so much more maintenance in the long run.

There you have it! This will be put into production here at DU in the next coming months. New hardware is arriving and we’ll be migrating to this clustered solution. The nice thing is since you have two nodes, you can have your master node as one resource group and if you want to run parallel deployment as another group, then they can both fail over to each other. Makes hardware maintenance or entire compute room outages very easy to handle (we have a main computer room and a redundant computer room in another building on campus). Just fail it over y0! :)

I imagine something similar could be done for those of you running in the windows world. I believe windows has the notion of a “global” file system, although I don’t know about the whole resource group/resource stuff with windows.

Anyway, there it is in a nutshell (more like a coconut shell, but hey whatever works ) I’d be happy to answer any questions you guys might have.