A successful Luminis III to Luminis 4.2.1 migration
I am a little late in posting this but we successfully migrated from Luminis III to Luminis 4.2.1 at Centennial College over the weekend of July 4th. Some details:
Timelines:
- Started Friday July 3rd at 5:00 pm
- Finished migration Sunday July 5th
- Tested and updated channels and content on Monday July 6th
- Released to users Tuesday July 7th
Hardware used:
- 2 T5220 Sun Servers with 2 x 8 core processors, 32 Gigs of ram. With cool threads the server shows 64 processors or threads running.
- 2 Server setup with Resource Tier running Web and Calendar and a Mail Tier
- Solaris 05/09 with ZFS root file system. Applications Tiers on their own drives in another ZFS pool.
- Tiers are in Solaris Containers or Zones
- Migration from two v490s with 8 Gigs of RAM and two hardware RAID stacks.
- Reused the RAID stacks to create one large ZFS pool, something which was a problem using UFS. If you create a UFS file system over 1 TB, the number of inodes available to it drops significantly. We ran our Luminis III system with two seperate 999 Gig file systems on Solaris 9.
Preparation:
- Pruned 356,000 users down to 196,000.
- Test test test on development migrations.
- Ran a mock live upgrade using data from our production Luminis III server. This is where the true problems will get encountered and it is wise to find out what they are before going ahead with a full migration. To keep the servers from interfering with each other during this mock upgrade, we sandboxed the Luminis IV migration box by pointing it's hosts files to itself and using a fake DNS with fudged entries.
- Researched lumdev.net (very helpful in our migration. Someone always had the answer to something we were looking for.
Issues encountered:
- SQL Server to Oracle conversion caused a lot of problems. If we were going from Oracle to Oracle or MSSQL to MSSQL the time to finish would have been a lot shorter. As it happens we went with SQL Server to SQL Server with a migration strategy to Oracle at a later date.
- One user complained their email was missing. Restored from backup so no issue there.
- Final mail backup took 24 hours. Mail restore took 18 hours. (Faster machine)
- Portal was slow and unstable when we first released it to end users. Tracked the problem down to the table ta_x_user on the Database Tier not being indexed properly. There is a compound index for columns taID and userId but the TA channel queries were only querying by userId so every time someone logged in a full table scan of 6 million rows was performed. As soon as an additional index of the userId column alone was created the server contention was gone and there have been no slowdowns since. If any schools are experiencing slow logins, double check your indexes for the ta_x_user table which can get very large. It was originally 16 million rows until we pruned it down to 6 million for the migration. ta_x_user is used for targeted announcements and if you send to large populations of users it can get big very fast. Removing users does not seem to shrink it. There were old users there even after we properly deleted users from Luminis III using cptool remove user. not sure why SunGardHE missed that.
- Calendar had no exit link, but a logout button. This was confusing so we changed the templates to make it an exit link.
Installation Notes:
- The ZFS rollback feature did wonders for us while we upgraded. If we messed up an import we just rolled back the instance to its preimport state. If the DB was touched it was a simple matter of running the resetrdb script in $CP_ROOT/installs/bin.
- Installing in zones allows for concurrent imports if you can dance around the DNS trickery. We ran one import from SQL Server to Oracle (worked in DEV not PROD) and one running with SQL Server to SQL Server.
- Old system was ready to be reinstated if it turned out we were going to abort the upgrade.
- Needed to download the latest a_mig.jar file because there were lots of import problems using the standalone Luminis 4.0.0.0 system to import. How did anyone ever finish a successful migration without this file? Version we used: a_mig-3.7.jar
- Created ZFS snapshots at every major point in the migration, including the post migration upgrade to 4.2.1
- Oh yeah, the migration has to be done in 4.0.0.0 unless this has changed recently
Post Migration:
- Installed custom home pages but the location was no longer valid. We used /cp/ipx/student and /cp/ipx/staff for our student and staff pages respectively.
- A lot of users had the direct shortcut to the old URL and were getting an error page when hitting the old URL. Had to create a custom Java class to capture the above URLs and redirect them to the new landing page which is now /cp/home/displaylogin. I'll post the details for that later.
- Spell check does not work if the Region is not set to US - English. We have both Canada - English and US - English. Submitted ticket.
- Wrong license key installed. SunGardHE gave us a new one to reenable file sharing in group tools.
- Empty migrated admin group caused Targeted Announcement error. Fixed after a ticket submitted.
- Email quota was not showing. Eventually started working.
- Targeted content errors started showing moniker errors. - Defect.
- Javascript Error when signed into Self Service Banner (SSB) via luminis - Defect
- Norton internet security seems to be the culprit for some users getting accessed denied or dropping to PIN screen when accessing SSB. Ticket still open
- IMSBackup groups and scripts for backing up Iplanet email are useless from Luminis III since it was sorted by login id. With the immutable ID in Luminis IV users have to be grouped via immutable ID or we will end up with one large backup file.
- Recently submitted a ticket for session counts growing but not shrinking. Number hit over 2400 concurrent logins which we have never reached before.
Post Mortem:
- When you discount the few bugs we found and the ta_x_user table problem I described above, our system is running better than it has before. Faster to login, faster to render channels and layouts, faster to get email. We love our system! Or maybe it is just because we took a month preparing and four days toiling until finally we saw the birth of our new baby.
Congratulations on the Upgrade !
Tha't's great news, and gives hope to those of us cowering in the shadows, fearing the migration process. I have a similar situation III.x to 4.x - but I'm sticking with Oracle - however going from Windows to Linux :(
We're moving to Banner 8 and now I'm forced to upgrage to Luminis 4. It was one thing when it was my own choice to upgrade, when I wanted to, how I wanted to ... but now it'll have to be done if we want the Banner Channels to operate.
I think B-8 is getting rolled out over Christmas break, so I have 5 months to plan and execute.
-Jon
Second testing a migration with live data
I wish I had done this before attempting to go live with Luminis IV. After months of running migrations against our test system, our first attempt at migrating to IV failed due to differences between prod and test. A second migration a month later went smoothly once we fixed all the problems encountered with the first one.
If I could replan my migration, I would have cloned our production zones to a second set of hardware and cloned the Luminis DB, then performed migrations against that snapshot. Unfortunately I came up with the idea two weeks before go-live. I'll try this when we move to V, many years from now.
Banner 8 and Luminis 4
Thanks Jon. We were quite comfortable with Luminis III but we too are rolling out Banner 8 this October and the push was on to get us to Luminis 4. After some of the horror stories it turned out that we have read, it turned out to not be such a bad thing after all. It did help that I took the time to install Luminis 4 way before we even talked about the project.
One thing I remember in my install/upgrade travels, SunGardHE said they do not support cross platform upgrades. Are you enlisting their help for the Windows to Linux move?
Tony
Targeted Announcement and Empty Migrated Groups
Hi,
After migration to L4 from L3 it looks like it has migrated the group names but not the members. When I select Employee role from target audience it says "Employee currently does not have any members".
Has anybody got this problem of empty groups ?
thanks
Kirti Bodhmage
Problem with Luminis IV upgrade
Hi ,
University of Greenwich have recently migrated to LuminisIV from III and its not a easy run. Portal is very unstable and needs restart very often. We are still digging into logs.
I was reading your post
****
A lot of users had the direct shortcut to the old URL and were getting an error page when hitting the old URL. Had to create a custom Java class to capture the above URLs and redirect them to the new landing page which is now /cp/home/displaylogin. I'll post the details for that later.
******
We are getting lot of error Access security violation for the login page as it has been changed to /cp/home/displaylogin instead of /cp/home/loginf. Have you got solution for this error.
We are getting following error in log.
org.apache.tomcat.util.threads.ThreadPool logFull
SEVERE: All threads (300) are currently busy, waiting. Increase maxThreads (300) or check the servlet status
Has someone came across this error.
thanks
Kirti Bodhmage
Redirects
Your change in login page is due to the login page being SSL enabled (address bar shows https). When you use the framed SSL (address bar shows http) it will still point to /cp/home/loginf. I am curious though, when I point to /cp/home/loginf on our portal, it automatically redirects to /cp/home/displaylogin without me having to use my redirect code. The page I was referring to in my post was actually /cp/ipx/student which was a custom page to appease our marketing folks. I had to create a redirect from /cp/ipx/student to /cp/home/displaylogin and in order to redirect from a non-existant path (/cp/ipx/ did not exist anymore) to a real one I had to create a redirect servlet.
From what I have read about that error it may be a memory issue (need to allocate more memory to tomcat). Check your $CP_HOME/bin/tomcat-cp-conf configuration for memory allocation. Here is what we have set:
Tony
Luminis IV Tomcat issue
Hi,
I have done new memory setting as below and increase our maxthread to 300. Tomcat is quite stable now.
We got few errors in catalina logs.
Aug 18, 2009 8:46:41 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: Failed shutdown of Apache Portable Runtime
When this occurs , tomcat is still running even when stopcp shows the CP Webserver is down. But when you startcp,
It won't start Luminis as Tomcat hasn't been shutdown properly.
2.
Aug 18, 2009 8:50:10 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/li
Has anybody got these errors ?
Everythings looks fine now but finger cross when student returns in September.
thanks
Kirti
maxThreads
The "all threads are currently busy" indicates you need to increase your maxThreads value in your tomcat-cp server.xml file.
That is at $CP_ROOT/products/tomcat/tomcat-cp/conf/server.xml
Both on the 80 and 443 connectors
<Connector port="80"
maxKeepAliveRequests="-1"
maxHttpHeaderSize="16384"
maxThreads="600"
minSpareThreads="25"
maxSpareThreads="75"
enableLookups="false"
redirectPort="443"
acceptCount="100"
connectionTimeout="3000"
disableUploadTimeout="true" />
Sungard has a tuning guide for LumIV that you should look at.
maxThreads and memory
Just remember that increasing the number of threads also requires increasing the memory allocated to tomcat.
We just finished a migration from P3 to P4
We just finished a migration, over the weekend. I will post the details later. I am still busy fixing a few outstanding items, but overall it went well.
How did you make the calendar logout an exit link. I agree that this will be confusing to users. Thanks.
-Mike
Calendar logout exit link
Mike,
Look in "/opt/luminis/products/uwc/data/uwc/common" and modify the file "MasterHeadNormal.jsp" and either remove the HTML for the logout link or replace it with code to act like an exit button.
Search for "uwc-common-logout" to find the appropriate section of code to modify.
Also, if you don't see your change immediately, you can run "touch" via the command-line on:
"/opt/luminis/products/uwc/data/uwc/calclient/*"
Cheers!
That worked perfectly
David,
Thanks for the information on the calendar logout link.
Mike
Different MasterHeadNormal.jsp file
This is tricky, there are more than one MasterHeadNormal.jsp files...
According to the Lum IV.0.1 Upgrage Guide, page 19, it should be:
/var/opt/pipeline/products/uwc/uwc/common/MasterHeadNormal.jsp
But I don't have that directory on my calendar box... I used:
/var/opt/luminis/products/uwc/uwc/common/MasterHeadNormal.jsp
(or for windows D:\Sun\WebServer\https-comexp\webapps\https-comexp\uwc\uwc\common\MasterHeadNormal.jsp)
... This is from the Lum IV.0.1 Upgrade Guide, page 19.