Subversion: pushing dev to test to prod
During summit I was reading up on Subversion (http://subversion.tigris.org/). I came across a blog post that said "the blog you are reading now, is a checkout from a subversion repository.
It got me thinking about how I maintain my test and production systems. In particular, the challenge of keeping track of all the changes across the systems. Several times I've had the installation of a patch on our test systems act differently than the same patch installed on our production systems.
In an ideal world, I'd like to be able to test something extensively on our test system, then when it is ready to go, simple hit a button and have all of it pushed to production. In researching it a bit, there doesn't appear to be any one way of doing this that is favored over another.
I ended up installing subversion served via Apache on our Sol10 test machines. sunfreeware.com was able to provide me all the packages and dependencies needed to get it working. It took me a bit longer than it should have, because I downloaded the wrong version of Apache heh. After that was done, googling for a few commands got me up and running with SVN pretty easily.
The first thing I did was move all the big files that I didn't want in my repository, out to a safe area. (Like ldap db files, and logs). Then I imported our entire $CP_ROOT into the repository. Next I tar'd the $CP_ROOT directory, moved it to a safe area, and then deleted $CP_ROOT. Then I used the svn command checkout, to extract the portal directory back into its original location.
The last step took me a while to figure out, and that was how to make svn ignore my log and data directories from further import/checkout/or update commands. I moved all the ldap and log data back into place, and then went to each parent directory containing a directory that I wanted to ignore. The command was:
svn propset svn:ignore '*' somedirectory
Then I started playing with it. I was using a windows machine at the time, so I downloaded tortoiseSVN. It was really easy to work with. I could checkout a file from the repo, edit it locally, right click on it (or a directory), and tortoise would commit the file back to the repo. Then I'd need to ssh to the server and tell it to 'svn checkout http://myrepo /myCP_ROOT' to see the change.
The nice thing is, once my development zones are up and running, I'll just install the svn client, and svn checkout the repository to them.
I'll be experimenting for a while, patching, editing, and pushing changes back and forth between development and test boxes, but hopefully and can use this method to push entire Luminis patches, or major skin/channel upgrades with a simple command.
The ability to rollback to different versions, as well as see who edited what file and when, is just icing on the cake really.
I'd be curious to hear if anyone else is already using some sort of CVS to push changes from test to production servers, and what tips you might have about this.
One thing I know that I'll need to do, is expand my initial import to include areas outside of CP_ROOT. I might just end up importing the root "/" on each box to make sure that I get everything, then going to each directory I don't want and setting ignore on them. For instance, I know that even though our calendar server is installed at /mnt/calendar, Luminis writes to /etc/luminis, as well as various areas in /var.
If no one else is currently doing something like this, I'll be sure to post my steps if I get it working smoothly.
- Jason's blog
- Login or register to post comments
Hack Your Knowledge
Recent blog posts
- Adding a Tab - disable this feature
- Cleaning entered username at login
- Cross platform migration
- Banner SSB Channels
- Alternative for inline frame channels?
- Luminis channel errors
- Facebook Channel - Non-Portlet
- AppServer Installation Errors
- Luminis 3.3 to 4 Migration properties questions/thoughts...
- Proxy Settings on Luminis
Developer Links
Poll
Syndicate
Comments
versions
how did you handle (a) different machines (b) different luminis versions ?
thanks
Rich
PS we got CVS for Luminis III - I'd thought that $CP_DOC_ROOT was a good place to import - it missed a few customizations (Java certificates files etc) but saved the import being too big - and sorting out lots of "ignores" - given space import everything would be better.
re: different machines and versions
I'm still learning/testing this whole thing out, but for now, I'm focusing only on the resource tier.
What I've done with subversion is setup the repository to allow for all the servers and version by simply providing a different import path for them.
For instance, my test Luminis IV resource tier is stored like this:
server:port/svn/test/resource-tier/mnt/portal
My test calendar server will be:
server:port/svn/test/calendar/mnt/calendar
With /mnt/portal and /mnt/calendar being the actual CP_ROOT areas of those respective servers.
When I setup the dev servers, I'll be importing them to /svn/dev/server/.....
When I want to "push" dev to "test", I'll just svn checkout /svn/dev on the test box. So I guess its actually like pulling the data where I want it, not pushing it.
The basic strat that I'm using so far, is to:
1. Determine the list of things that I know I do not want to import into subversion, like: CP_ROOT/products/slapd-cp/db/"everything except luminisconfig", or CP_ROOT/logs, etc..
2. I shutdown luminis, and with a script, I move every directory I want to ignore, to /mnt/backup.
3. I make a directory in the same place, with the same name, but empty. Like an empty CP_ROOT/logs for instance.
4. I do a svn import -m "Initial import" CP_ROOT http://server:port/svn/test/server/cp_root
5. This is the scary part:), and I'm glad I have zfs snapshot hehe, I do a rm -rf cp_root
6. I do a svn checkout http://server:port/svn/test/server/mnt/portal /mnt/portal
7. Run a script that does 'svn propset svn:ignore '*' ignoredDirectory on each directory to be ignored.
8. I move my data from /mnt/backup back into the appropriate directories.
9. I startcp to see if it still works.
The areas I put into svn so far, are cp_root, /var/imq, and /etc/luminis for the resource/ldap server.
As soon as I verify that I can patch a dev box, like the resource/ldap, and then svn checkout that patch to a test box, I'll post up my scripts I used.
just track changes
Is there a proper strategy for using something like Subversion just to track changes - and not for deploying the software? I have a snapshot of the standard IV.0.2 install and I also have a modified install (with our customizations). We'll do future mods, too, of course.
Is there a way to retroactively bring those into subversion to keep track of the changes, etc?
re: into subversion to keep track
I believe so. Like I said above, svn is fairly new to me, so I'm still learning.
But if you were to import an area of your portal into subversion, then delete that local copy, then do an svn checkout of what you just imported, the .svn directories that subversion create would keep track of the changes.
For instance, using the steps above, I imported my CP_ROOT of one server into subversion.
If another developer (or me, as a test I did this) edits a file, you can always see whats changed by getting on the server, and checking in the local copy back to the repo.
Like, if your CP_ROOT is at /mnt/portal, and you imported that CP_ROOT to the svn repository at http://server:port/svn/myserver/mnt/portal and someone edited /mnt/portal/webapps/luminis/site/login.html a couple days later, if you were to type:
svn commit /mnt/portal
It would return the line:
Commited Revision 2.
subversion would upload the changes that occured in /mnt/portal since the last commit (or check-in). Then using a svn enabled client, like Eclipse for instance, you could view revision 2, and it would highlight the exact lines of the file that were changed.
Eclipse has been great for me so far. The "Team SVN" plugin has a history and revision window, so I can click each revision and see what files were changed in that commit.
If two people edit the same file and try to commit it, SVN tells you that there is a conflict, and were it is, and, if using something like Eclipse, you can pop the conflicted files into a comparison editor and see what the differences are.
Yes
Yes, We have been using Subversion throughout our development/deployment and our strategy works like this:
We make a local copy of the current Luminis root files.
As we go to modify a file, we first check it into subversion, then make our modification to it, then check that in. (Our initial population was more than a single file, but we only check in modified files).
We then go along merrily making changes to those files, checking them in, rolling them back, and comparing to the original (Eclipse has a great compare editor). We tag our releases, and move along.
Now when a new version of Luminis comes out and we want to install it... Our sysadmins install it on dev, and send me the root files. I then pop those all in over the top of my modifications, remove the files that now want to be added to he repository, and check in those files. Now I have a repository full of Sungard code in the fiels that we have modifications for. I then replace all of those files with the version before that (the modifications from the last release), and compare/merge them down, and re-check them in (then I have a set of new version files with our mods in them). This works like a charm , and it usually takes about 2hours to update our very extensive modifications to the new version.
Thanks. Have you ever run
Thanks.
Have you ever run into trouble with changes in the ldap or database that might not be captured by subversion?
Or do you have a way to compare changes, like configman changes to the ldap, with subversion?
One of the things I was trying to avoid, was having to sync with any data that is changing due to basic portal use. But in doing that, I fear I might miss something that a patch could do, like adding a database index, or changing ldap acl's, etc..
Subversion hook_script
Jason,
Subversion has "hook-scripts" that can accomplish your needs.
There are several scripts under your SVN_REPO/hooks directory.
Example: post-commit
post-commit will be execute each time there's a commit to your svn.
You then can write a script to checkout to your production and call it from post-commit
Anyway, you can take a look here:
http://subversion.tigris.org/tools_contrib.html#hook_scripts
and read the comment in the SVN_REPO/hooks/post-comment.tmpl
For those that would like to track the changes with SVN, I would suggest to install "Trac". It integrates nicely with SVN and you can track the changes from the browser.
Check it out here: http://trac.edgewall.org/
Have fun!
--
Thai Nguyen
Version Control Concerns
I would love to move Luminis into a repo, however my concern is with configuration files, etc that are specific to the environment. Database settings, and LDAP (categorically "configman" settings) are stored outside of Luminis' files that would be checked in to the repository.
However, I would assume that there are hostnames, usernames, passwords, paths, etc that are all hardcoded into $CP_ROOT inside files. For example $CP_ROOT/install/install.properties. Is this the case?
Also, I know that particular files like "jaas.conf" are stored outside of the $CP_ROOT directory. How do you handle this?
If it had been programmed better, this would be great, as our production and test environments are really out of sync.
cvs concerns
Well what we do right now in order to be able to copy our live production zones (virtual zones) into test zones, is to make a dns server that basically fakes all the test zones into thinking that our db connections, ldap locations, site names, etc.. is the test zones.
That way, we can directly copy a production server to new IP addresses, and everything works fine. We tried using just host files, but it seems that parts of the luminis code rely on DNS over hosts.
Currently my rep is pulling in cp_root, /usr/jdk, and the shared directories. As I find more areas that Luminis needs, I'll probably pull those in also. I can pull in the configman settings by pulling in just the Luminis Configuration db files from the ldap area.
I should have a dev system in place soon, so I'll do an experiment and see if I can patch a 4.0.0.0 luminis to 4.0.2.0 in dev, and "push" it to a 4.0.0.0 test system without error.
The only thing it might miss, is if the Luminis patches updated the actual structure of the ldap, or added db indexes.
So.... long story short. All the names of our systems are identical between prod/test/dev. We just make the servers use separate DNS and localhost files to control them.
On my desktop, I have some buttons linked to sh scripts that swap my own host file out depending on which "my.pcc.edu" I want to see.
jwhitene at pcc edu