You are here

Luminis Statistics from Uportal

Submitted by zbtirrell on Tue, 04/12/2005 - 15:25

As many of us who use and administer Luminis know, the stats are quite inadequate straight out of the box.  There are some improvements in III.2, but they are still not enough.  Some posts on here have suggested some other solutions that gain some ground.  I have been instead researching Uportal's build in StatsRecorderFactory support.  After a lot of trouble initially getting this enabled, Grace Francisco at SCT support was finally able to get me what I was missing to try this out.  The following is an account of how to enable these stats outlining what I like and don't like about this solution.  In short this is useful, but short of perfect.

1) cd CP_WEBINF/config
2) jar xf ../lib/uPortal.jar properties/portal.properties --> this actually will get picked up by our classpath, and much easier than editing the file, and rejarring it back to uPortal.jar
3) cd properties
4) edit portal.properties
5) change

#org.jasig.portal.services.stats.StatsRecorderFactory.implementation=org.jasig.portal.services.stats.DoNothingStatsRecorderFactory
to
org.jasig.portal.services.stats.StatsRecorderFactory.implementation=org.jasig.portal.services.stats.LoggingStatsRecorderFactory

6) turn the stat recorder settings ON, if you scroll down a few lines you can choose which ones you want.

7) bounce the webserver:
CP_ROOT/bin/rc/70-webserver stop
CP_ROOT/bin/rc/70-webserver start

8) add logging settings to CP_WEBINF/config/cplog4j.properties
# - STATS-RECORDER
log4j.logger.uportal=INFO, stats
log4j.appender.stats=org.apache.log4j.RollingFileAppender
log4j.appender.stats.File=${util.logservice.log4j.directory}/stats.log
log4j.appender.stats.MaxFileSize=10000KB
log4j.appender.stats.MaxBackupIndex=10
log4j.appender.stats.layout=org.apache.log4j.PatternLayout
log4j.appender.stats.layout.ConversionPattern=[%d{ISO8601}] %m%n

These are some of the log entries created in stats.log:
[2005-03-18 17:10:11,828] [INFO] WebServlet [uportal]: STATS-RECORDER: (cpadmin) logged in successfully at Fri Mar 18 17:10:11 MST 2005
[2005-03-18 17:10:30,937] [INFO] WebServlet [uportal]: STATS-RECORDER: Channel [E-mail Channel, 221, u11l1n13] was rendered in layout 1 by (cpadmin) at Fri Mar 18 17:10:30 MST 2005
[2005-03-18 17:10:30,937] [INFO] WebServlet [uportal]: STATS-RECORDER: Channel [Personal Announcements, 210, u11l1n14] was rendered in layout 1 by (cpadmin) at Fri Mar 18 17:10:30 MST 2005
[2005-03-18 17:10:30,937] [INFO] WebServlet [uportal]: STATS-RECORDER: Channel [Campus Announcements, 211, u11l1n15] was rendered in layout 1 by (cpadmin) at Fri Mar 18 17:10:30 MST 2005
[2005-03-18 17:10:30,937] [INFO] WebServlet [uportal]: STATS-RECORDER: Channel [My Headlines, 212, u11l1n17] was rendered in layout 1 by (cpadmin) at Fri Mar
[2005-03-18 17:12:43,015] [INFO] WebServlet [uportal]: STATS-RECORDER: Session destroyed for (cpadmin) at Fri Mar 18 17:12:43 MST 2005

The first major limitation of this is that it is logging all the stats in a file.  So in my instance I am running a cron'd process hourly to go in and pick up the stats files, parse them, and load them into Oracle.  My table structure is attached so that you can see the sort of info I'm am acquiring.  The script I've written is in PHP and if anyone is really interested, I could share that as well.  In a perfect world, there would be a Stats Recorder for Oracle, so that it could go straight in without the extra steps.  Someone has written RDBMStatsRecorder which hits Postgres.  That may be a good base for building an Oracle version.  (Google search RDMBStatsRecorder to find out more).

The second limitation is that it is logging all the STATS-RECORDER entries into the cp.log as well as stats.log.  This makes for a very dirty cp.log.  Someone who knows more about log4j may be able to explain how to resolve this.  I unfortunately do not...

The third limitation is that there is not a way to track what tabs are being accessed.  This is an important statistic for a bunch of reasons.  I reported this desire to support and they entered Enhancement Request #26422 on my behalf.  Another option would be to try and glean these from the webserver logs, but of course then you would not have a username and associated role to tie to these stats, so I'm not certain how useful that might be... Better than nothing though.

UPDATE: You can see my PHP code here: http://my.plymouth.edu:81/code/stats.phps

Modification:

Attachments: 

Comments

Having no experience with this at all, I'm just shooting in the dark.

If you set your log4j properties to ERROR will they still record in the stats.log file or does that turn them off all together?  I seem to remember there are levels of logging and if you keep the ",stats" and set the other property to a higher level, maybe it'll stop pumping them into the cp.log.  Just a thought. I'll have to try this out.

Like this - log4j.logger.uportal=ERROR, stats

again just a thought, I have no idea whether I'm blowing smoke or not.

If you set it to ERROR, then you don't get any of the STATS-RECORDER entries. They are INFO level messages... I did set "log4j.rootLogger=ERROR, file" which is the one that I thought should be going into cp.log

Thanks for your post - it's another way of helping fill the hole for the missing stats functionality.

To get around logging to both cp.log AND stats.log, add the line emboldened below to cplog4j.properties - It removes logging to cp.log

# DAS: STATS-RECORDER
log4j.logger.uportal=INFO, stats
log4j.appender.stats=org.apache.log4j.RollingFileAppender
log4j.appender.stats.File=${util.logservice.log4j.directory}/stats.log
log4j.appender.stats.MaxFileSize=10000KB
log4j.appender.stats.MaxBackupIndex=10
log4j.appender.stats.layout=org.apache.log4j.PatternLayout
log4j.appender.stats.layout.ConversionPattern=[%d{ISO8601}] %m%n
log4j.additivity.stats=false

What exactly does your cron job do & would you post an example of it please? The PHP code would also be very useful.

Also, do you get a whole load of extra lines that aren't related to the stats-recorder and just grep for line that include the following string - STATS-RECORDER - ?

I too would be interessted in seeing your code for parsing this data if it is possible to share it with us.

Check the main article.

Also, the log4j.additivity.stats=false change did not stop the logging for me to cp.log  Is that the only thing that needs to be changed?

did you implement this stats function specifically so that you could track the channel and tab renders?

i noticed on my III.3 test install that we have a sessions.log file that is recording logins and logouts with a timestamp and i didn't have to change anything with the log4j file or any other uportal files.

so, i guess i'm wondering if i would be better off using your directions and recommendations if my current main concern is displaying statistics of simple logons/logoffs or if i should just adapt your php file to use the sessions.log file instead. not sure if that made sense but hopefully it did.

We originally implemented this stats tracking in an attempt at getting what channels and tabs were most popular. Frankly, it's the login/logout data that we use most frequently.

Following your suggestion, I have modified our code and broken it into two seperate scripts. One that picks up the tab/channel stats. The other picks up only logins and logouts from the delivered session.log files. (http://my.plymouth.edu:81/code/stats_session.phps)

I'm trying to get this working, having some difficulties, and have some questions that seem to me rather dumb, but I'll go ahead with anyways:

1) How is this useful? I see a 'rendering' entry for every channel on a tab when I switch to that tab. I was hoping for something that told when someone used a channel. Is there some other setting in portal.properties to turn on? I've turned rendered, and targeted on.

2) OK. Where's the stats.log file? All I can see are entries in cp.log (additivity.stats=false does not seem to affect it).

Thanks, Bob.