I’m a proud father. Here’s proof.
All posts by ajohnson
Automated Application Deployment with Tomcat: Part II with Subversion And Ant
Got around to writing the deployment scripts that go with the Subversion work I did last week. As part of the scripts, I also worked on the problem I wrote about a couple weeks ago (ie: needing different log4j properties files per environment). Once you have a source control solution and a master build file, it’s pretty easy. I stripped out the explicit log4j category and appender directives and instead used the Ant <propertyfile> tag to write out the appropriate category / appender directives based on the presence (or lack thereof) of the project.debug property in my build file. When I want to create a build that outputs logging information to the console, this block of code is fired:
<target name="build.resources.debug" if="project.debug">
<propertyfile
file="WEB-INF/classes/log4j.properties">
<entry key="log4j.rootCategory" value="INFO, STDOUT"/>
</propertyfile>
</target>
Otherwise, this block is run:
<target name="build.resources.production" unless="project.debug">
<propertyfile
file="WEB-INF/classes/log4j.properties">
<entry key="log4j.rootCategory" value="ERROR, SMTPAPPENDER"/>
<entry key="log4j.category.com.mycompany" value="ERROR, SMTPAPPENDER"/>
</propertyfile>
</target>
In Eclipse, you can easily create the project.debug property using the Arguments text box provided in Run –> External Tools –> $your ant build –> Main Tab –> Arguments: -Dproject.debug=true. By default then, log4j will only log error conditions and will send them to the SMTP appender.
The second thing I did was to create a deployment Ant script that wrapped my main build.xml. It checks out the source code from Subversion:
<exec executable="svn">
<arg line="co svn://sourcecodeserver/myproject/branches/v1.0 . --username username --password password"/>
</exec>
<ant antfile="build.xml" />
<exec executable="net">
<arg line='stop "Apache Tomcat"'/>
</exec>
<delete dir="${tomcat.install}myapp"/>
<mkdir dir="${tomcat.install}myapp"/>
<unzip src="myapp.war" dest="${tomcat.install}myapp"/>
<exec executable="net">
<arg line='start "Apache Tomcat"'/>
</exec>
We’ll see how well it works once this goes into production. If nothing else, at least I’ll have a couple more hours in my day.
Source Control with Subversion on Windows
We’re getting around to hiring an additional me at work so I needed to get a source control solution in place before he/she arrives and starts hacking away at the codebases. At the last place I worked we used Visual SourceSafe for a long time (which was pretty worthless) and then started using CVS, although not on the projects I worked on. A couple weeks ago I asked a bunch of questions about automating Java web application deployment and Erki suggested that I buy and read the Pragmatic Project Automation book, which I did and which I loved. I read it from start to finish (which isn’t an easy thing to do with a technical book) and took a ton of notes. Anyway, I’m pretty sure the book didn’t mention Subversion even once, but it did drill home the importance of setting up your code in source control and then doing your manual, scheduled, automated (using CruiseControl) or production builds directly from the source control system using Ant or Groovy, which led to alot of ideas about how you could automate builds to production Tomcat systems. I’ve heard alot about Subversion; it’s described as CVS with none of the warts and all of the features, so I tried it out. Following are the notes I kept during the installation process. Hopefully this helps out someone somewhere down the road:
a) Download Subversion (obviously) I downloaded version 1.1.1 ( svn-1.1.1-setup-2.exe)
b) Run the install. I chose to install to c:\subversion\.
c) Test the Subversion install by going to the install directory and then going to the /bin directory. Run the svnadmin from the command line:
CD c:\subversion\bin\
svnadmin
If you get an error that says that says “svnadmin.exe – Unable To Locate DLL”, you’ll need to either a) install the Visual C++ 6.0 VCREDIST.exe, which is available here (but only if you’re using Windows 98 or Windows ME) or b) find someone or some system that has msvcp60.dll and copy that file to c:\$windows\system32\ (where $windows is the directory that contains your Windows installation).
d) Create a repository on the machine that is going to host your source code. From the command line on the server machine:
svnadmin create d:\$projects\$projectname
where $projects is the name of a folder that will contain all your Subversion source code repositories and $projectname is the name of the specific project you’re going to create a repository for.
e) Start the server either from the command line:
svnserve -d -r d:\$projects
or use the SVN Service Wrapper (http://dark.clansoft.dk/~mbn/svnservice/), note that the r switch limits directory browsing by specifying the root of your repositories.
f) Create a users database (which is nothing more than a text file, I created a file called users.conf) and save it in the d:\$projects\$projectname\conf directory and then add your usernames & passwords. An example user database might look like this:[users]
aaron = mypassword
g) Within the d:\$projects\$projectname\directory, open the conf/svnserve.conf file using your text editor of choice and add the following to allow reading and writing to the users in the database we just created:
[general]
anon-access = none
auth-access = write
password-db = users.conf
realm = $projectname
where $projectname is the name of the project you’re setting up and users.conf is the name of the file you created in step f.
h) Finally, restart the server and you’re ready to start using the client and import your project.
i) To import an existing project, invoke the svn executable from the command line on the client:
svn import c:\temp\myproject svn://myserver/myproject -m "initial import" aaron mypassword
where c:\temp\myproject is the directory that contains the existing project you want to import (and should have 3 folders: trunk, tags and branches, your source code should be in the trunk folder; more on repository layout here), myserver is the name of the server that contains the Subversion repository you created in step d, myproject is the name of the project (ie: $projectname from step d) you’re importing into, the m switch signifies the message you want to tag this import with, followed by the username and password you setup in the users database in step f.
j) Finally, checkout the project from the server (invoking the svn executable from the directory that you want the trunk checked out from…, ie: c:\projects\myproject)
svn co svn://myserver/myproject/trunk . aaron mypassword
k) To add, move, delete, or edit and then commit any changes back to subversion:
Add: svn add src/myfile.java -m "adding a file"
Move: svn move src/myfile.java src/mynewfile.java -m "moved myfile.java to mynewfile.java"
Delete: svn delete src/myfile.java -m "removing a file"
Commit: svn commit src/myfile.java -m "the message"
That’s it… there’s a alot more to read. One of the coolest things about Subversion is that the documentation is all packaged up into a free PDF published by Oreilly, which you can download here. I printed out the manual and breezed through it in a couple hours. You should too.
I did try to get the Subclipse Eclipse Subversion plugin working, but for the life of me… I couldn’t. I think it ended up for the better anyhow because while the command line is a bit more tedious, I understand more about what’s going on in the system, which is helpful. I’d recommend that you get familiar with the command line client before attempting to use a GUI plugin.
After you’re all set with Subversion and you’re ready to get started with CruiseControl, note that while Cruisecontrol contains an <svn> tag, you still need to drop down into command line mode in Ant to checkout / update your local repository when doing an automated build. For example:
<exec executable="svn">
<arg line="update"/>
</exec>
MT Captcha Installed
I installed and configured MT Captcha on all six blogs this server hosts. While it does present a problem for a blind person who wants to post a comment (quick fix: send me an email if you want to post a comment), it hopefully will stop the comment spam.
Never heard of Captcha? It stands for a “completely automated public Turing test to tell computers and humans apart”, naturally. Read more about on wikipedia.
Java static initializer question
I ran into a problem with a section of code that lives within a Java static initializer; I’m not sure that the problem is the static init. Let me step back and frame the problem first. I have a Struts action class that presents a form for collecting date information; said form includes a drop-down menu that is bound to an ArrayList of LabelValueBean objects that contains the months of the year (Jan|1,Feb|2,Mar|3, etc..). The ArrayList of LabelValueBean objects is retrieved from a data provider class using a static method:
Collection months = MyDateProvider.getMonths();
Here’s the pseudo-code inside the date provider class:
public class MyDateProvider {
private static Collection months = new ArrayList();
static {
Calendar c = Calendar.getInstance();
SimpleDateFormat formatter = new SimpleDateFormat("MMM");
SimpleDateFormat monthformatter = new SimpleDateFormat("MM");
for (int i=0; i
The problem is that somehow the ArrayList of LabelValueBean objects contained two instances of March and no February, so the dropdown list would look like this:
January
March
March
April
May
June
July
August
September
October
November
December
Restarting Tomcat temporarily fixed the problem (obviously because MyDateProvider was then reloaded), but my question is how did is it possible for the ArrayList to be missing February and contain two instances of March? Any takers?
Update: the problem has nothing to do with the static init and everything to do with the way I used the Calendar class. The Calendar.getInstance() method returns a Calendar object initialized with the current date and time, so when the web application was restarted on 11/30/2004, the Calendar object reflected that date/time. Looping from 0 to 11 and setting the month (c.set(java.util.Calendar.MONTH, i);) resulted in the month changing correctly EXCEPT during the month of February , which has only 29 days. The date "02/30/2004" is illegal and so the Calendar class helpfully bumps the month up, which resulted in two instances of March in my drop down. Thanks to Paul Hastings for pointing this out.
Blog Spam and Parasites
So I run a couple blogs and like many people I haven’t solved the blog spam problem yet (even MT-Blacklist only works some of the time and it’s a losing battle). There obviously have been alot of people trying to solve the problem, but it’s interesting to watch from another angle: blog spammers (I guess all spammers) are effectively parasites living off a host system. For the most part, they don’t bother the host system too much. One, two, maybe ten comments per day and we both keep on living. It’s the times like last night when I got 418 spam comments on my blog that I think I should turn off comments entirely but killing the host doesn’t seem like the right answer.
A couple of weeks ago I listened to Janice Benyus’ presentation on biomimicry (courtesy of Jason Kottke, you can download and listen to the presentation in MP3 format). In it she talks about 12 ways in which nature does things better than we do: self assembly, the power of shape, natural selection as innovation (amazing to hear how nature in many ways is still light years ahead of our man-made technology). It’s interesting stuff if you’re into that kind of thing, but then you probably wouldn’t haven’t gotten this far if you weren’t. Anyway, that presentation popped back into my head today while I was deleting all 418 comments: how does nature deal with the problem of parasites? I guess it goes without saying that nature hasn’t solved the problem either because the earth is really old… and we’ve still got a ton of parasites. Malaria, giardia, fleas and ticks are all rampant abusers of animals and humans (reading the Wikipedia entry for Malaria reminds me of the malaria pills I had to take when my pop took my brother and I to Kenya for 2 weeks… I chewed one by accident, talk about a bitter pill!) But their existence doesn’t mean that nature (or humans) haven’t tried to get rid of parasites. Doctors and researchers have been trying to eradicate malaria from much of the world for many years using a variety of methods:
· attacking the mosquito population: Reminds me of a post by Tim Bray (Crooks in Plain Sight), why not hunt down the people who do the blog spam (we certainly know where they live on the web)? Tim, or perhaps someone else, suggests that no one wants too. Better yet, no one has the time or energy to do so. Probably true.
· genetically modifying mosquitoes so that they don’t carry malaria: I think this would be analogous to asking the search engines (Google, MSN, Yahoo, etc..) to do something about the problem (ie: modify the Google search algorithm so that even if comments gets on your blog and stays there, it doesn’t influence the PageRank of the link they post). Would this happen? Unlikely. Any change Google makes would be countered by changes from the parasite/blog spammer. That’s a losing battle.
· distributing mosquito nets to areas of the world where the problem is most severe: A mosquito net? I think this is the MT-Blacklist / scode of our disease control toolkit. Put a net over our blogs so that the parasites can’t get it. Works for someone, but complete eradication will only work if everyone has a net. The parasites will continue to flourish on all the unprotected blogs.
· releasing millions of sterile mosquitoes into the wild (Sterile insect technique):
: The idea here is that by releasing millions of sterile mosquitoes, males will fruitlessly attempt to fertilize eggs.. females will lay eggs that can’t ever hatch. I think this would be analogous to a bunch of people getting together and making sites about incest, kid porn and beastiality. Neutralize the blog spammers efforts by beating them at their own game (which we’ve proven we can do).
·through chemicals like DDT: No idea what the corollary would be in the blog world, most likely like the first (attacking the mosquito population)
·by developing a vaccine for malaria: Here’s an interesting idea: instead of distributing nets like MT-Blacklist or scode (which block spam at the point of attack), why not modify our systems that they become immune to the affects of the parasite? Maybe this is modifying the comment URL so that it’s a redirect rather than a direct link (already being done), adding comment registration (already being done), or a comment approval process.
I’m not sure a solution will ever be found (regular spam doesn’t seem to be getting any better)… if nothing else, it’s an interesting problem. Oh, and if you found it interesting, drop me a comment. 😉
Mikedotnet.com v2.0
In my continuing quest for free stuff, I *helped* a buddy transfer his ASP based website to my movable type installation and we had the launch party… um.. actually there was no launch party. No free beer or pizza. Nada. Ahem. Anyways, you can check out the new Mikedotnet.com v2.0 now. Also, I figure by mentioning Mikedotnet a couple times on my blog, I will continue to rank higher than he does for his own site, which means he must continue to give me free stuff. You too Patrick Owens!
New Feature on KarensRecipes.com
The Thanksgiving 2004 Cooking Adventure is online now courtesy of yours truly and some nagging by my mom to put something new on the homepage. Oh, and she wrote part of it, so I guess she deserves some credit.
Commons Net FTP listFiles() returns null
Just bringing this to the top of the Google QueueTM, if you’re using the listFiles() method of the org.apache.commons.net.ftp.FTPFile class and you’re seeing that listFiles() returns null when it shouldn’t be, make sure to upgrade to the latest and greatest version of commons-net (which is 1.2.2 right now). Using version 1.0.0 was causing NPE’s all over the place. I think this bug on issues.apache.org discusses the issue, but it’s more specific than what I was seeing.
Automating Application Deployment with Tomcat
I’m asking a bunch of questions here so if you’re expecting answers, look elsewhere. In the past couple years, I’ve waded through a bunch of different ways to configure and build Java web applications. At first, I hardcoded connection strings, settings and message strings in the source code, compiled using Eclipse and copied over the JSP and class files to the web server (lame, I know, but you gotta start somewhere). As the applications I wrote got more complex and as I got smarter, I started using Ant to perform the build and I learned about properties files and web.xml. After unpacking alot of other open source Java applications, I’m now using log4j for logging and error messaging, JNDI for configuration (datasource and application configuration stored in Tomcat’s server.xml), resource bundles for storing internationalized message strings, JUnit for running tests and Ant for cleaning, testing and building my apps into a deployable format (war files).
And that’s where it stops being easy. Deployments suck. We have a pretty small environment (a couple test servers, a couple staging servers, a couple live servers) and deploying changes to those servers is tedious. For example, in my development environment, I like to have log4J append any and all information to the console, which lets me watch the system as it starts up and runs:
log4j.category.com.mycompany=INFO, STDOUT
But once I build the application and deploy it to the live environment, I only want error messages and I want them sent to the SMTPAPPENDER:
log4j.category.com.mycompany=ERROR, SMTPAPPENDER
so I’m stuck editing a text file every time we deploy an application. It’s not that big a of a deal, but I also have applications on each server that need to have the appropriate entries in Tomcat’s server.xml (environment entries, JDBC connections, etc..), sometimes Tomcat needs to be restarted after your deploy the war, applications are deployed to different directories on different machines, sometimes the application being rolled out doesn’t work and you need to roll back to the previous version, how do you keep track of all the live / staging / development servers? The environment I work in is pretty small, so all of this can be done by hand, but it’s tedious, boring and error prone. How do you guys that work in larger environments do it? How do you move the .war files from your staging environment to the live environment? Using Ant? Do you trigger Ant tasks on the live servers that check out source code from CVS and build the apps there? Do you restart Tomcat every time? Do you do one half of your machines at a time and then the next half? You can’t be doing this by hand! Any tips?