Douglas Hofstadter (Le Ton beau de Marot, Goedel, Escher, Bach, The Mind’s I, etc..) will be speaking at MIT on April 12. More information is available at on the MIT Language, Cognition, and Computation Seminar Series website. I wish I could go to this one.
Category Archives: Software Development
Doug Cutting interview on TSS
The guys at TSS did a pretty indepth interview with Doug Cutting (who wrote Lucene and is now involved in the Nutch project). You can read the transcript here; there’s video there somewhere, I just couldn’t find it.
Donald Knuth on NPR
Donuld Knuth was on the Morning Edition this morning, you can still catch it via Windows Media Player or Real Player. He’s a real character (he leaves his bike helmet on inside because… eventually he’ll have to put it back on anyway).
Using Betwixt, Struts and REST
My latest project at work launched a couple weeks ago, unfortunately there isn’t much to look at, but it’s not like I’m a Flash developer, so unless I start writing Windows apps, no one will ever be able to see my work anyway. Alas, a major portion of the project involved consolidating some business logic behind a SOAP API (I’ll go into that some other time). We were supposed to be moving our front end to ASP.NET but at the last minute, the ASP.NET migration didn’t happen so we ended up needing something that ASP could use as well, which meant REST. With the business logic wrapped up behind business delegates, all that was needed as a Servlet Filter to map requests for /myapp/xml to a Struts Action class, which invoked the business delegates and then returned XML to the client. Now on the SOAP end of things (Axis embedded in Tomcat), I didn’t have to deal with XML (Axis handles all the XML serialization and deserialization). But with REST you’re in conundrum: do you create toXML() methods on all of your JavaBeans? do you put the JavaBean or collection of beans in the Request scope and then generate the XML using JSP? Those options all seemed error prone and harder to test. I looked at JAXB (makes more sense when you have XML that you want to turn into a JavaBean) and Castor (same thing) but finally decided on using Betwixt, a Jakarta Commons library that provide a simple way of turning a JavaBean into XML. The end result looked something like this:
public class RESTAction extends DispatchAction {
public ActionForward doSomething(...) {
MyBean bean = BusinessLogic.getBean();
response.setContentType("text/xml");
OutputStream sos = response.getOutputStream();
BeanWriter beanWriter = new BeanWriter(sos);
BindingConfiguration config = beanWriter.getBindingConfiguration();
config.setMapIDs(false);
beanWriter.enablePrettyPrint();
String declr = "<?xml version='1.0' encoding='UTF-8' ?>";
beanWriter.writeXmlDeclaration(declr);
beanWriter.write("mybean", bean);
beanWriter.flush();
beanWriter.close();
return null;
}
}
which when combined with this action mapping:
<action
path="/xml"
type="com.mycompany.web.actions.RESTAction"
name="xmlForm"
scope="request"
parameter="method"
validate="false">
<forward name="error" path="/WEB-INF/views/xml/error.jsp" />
</action>
is invoked like so:
http://myserver/myapp/xml?method=doSomething
I’m not 100% sold on this as the “way”, but I do like how simple Betwixt is. If you’ve got a different idea, comment away.
Maps.google.com Deconstructed
Joel Webber took some time to analyze the new google mapping application (which is amazing, every person I’ve showed it too thinks it’s one of the coolest things they’ve seen on the web recently). They do some really cool stuff with IFrames, JavaScript, XML, XSLT and PNG. Read it here.
Retrieving an RSS feed protected by Basic Authentication using ROME
Today I worked on a feature in a Struts web application where I needed to retrieve an RSS feed that is protected by Basic Authentication and then display the the results of the feed in a web page. I’ve heard alot about ROME in the past couple weeks, so I decided to try it out. It quickly passed the 10 minute test (downloaded rome-0.5.jar, downloaded jdom.jar, used the sample code from the tutorial on the wiki); I was able to retrieve, parse and display the results of my own feed in no time:
String feed = "http://cephas.net/blog/index.rdf";
URL feedUrl = new URL(feed);
SyndFeedInput input = new SyndFeedInput();
SyndFeed feed = input.build(new XmlReader(feedUrl));
System.out.println(feed);
Easy. But that wasn’t my problem. I needed to be able to set the Basic Authentication header which is usually done like this:
String feed = "http://yoursite.com/index.rdf";
URL feedUrl = new URL(feed)
HttpURLConnection httpcon = (HttpURLConnection)feedUrl.openConnection();
String encoding = new sun.misc.BASE64Encoder().
encode("username:password".getBytes());
httpcon.setRequestProperty ("Authorization", "Basic " + encoding);
httpcon.connect();
.. // do stuff
httpcon.disconnect();
Turns out that the designers of the ROME library were pretty smart. In addition to including the XmlReader(URL url) constructor, they also included a XmlReader(URLConnection connection) constructor, which allows you to combine the two blocks of code I wrote above to make this:
String feed = "http://yoursite.com/index.rdf";
URL feedUrl = new URL(feed)
HttpURLConnection httpcon = (HttpURLConnection)feedUrl.openConnection();
String encoding = new sun.misc.BASE64Encoder().
encode("username:password".getBytes());
httpcon.setRequestProperty ("Authorization", "Basic " + encoding);
SyndFeedInput input = new SyndFeedInput();
SyndFeed feed = input.build(new XmlReader(httpcon));
Add this code to your Struts action, put the resulting SyndFeed in the request scope (request.setAttribute("feed", feed);) and then in the JSP:
<c:forEach var="entry" items="${feed.entries}">
<strong>${entry.title}</strong><br />
${entry.description.value}<br />
<fmt:formatDate value="${entry.publishedDate}" type="both" pattern="MMMM dd, yyyy" />
by ${entry.author} | <a href="${entry.link}">link</a>
</c:forEach>
So there you have it. I hope that makes it easier for someone else!
Paradigmatic Software Development: Week 1
I started my first UMass Dartmouth class last week, it’s called “Paradigmatic Software Development”. You can read all about the syllabus and course objectives here if you’re so inclined. Following are my notes from the first reading assignment (Chapter 1 of Object-Oriented and Classical Software Engineering):
· The book starts off with a story about a man who received a bill for $0 and after receiving it multiple times, he sends in a check for $0, which then proceeds to crash his bank account. Funnier still is that this story appears to be an urban legend. I’m sure the bigger point is that errors like the one mentioned are all too common, hence this course. In case the author ever reads this, there is a much better list of famous bugs on wikipedia; see also Off-by-one error, fencepost error.
· Definition: software engineering is a discipline whose aim is the production of fault free software, delivered on time and within budget, that satisfies the clients need. Additionally the software must be easy to modify when the users needs change.
· There are 5 aspects of software engineering: historical (a very low percentage of software products have been completed sucessfully), economic (tech ‘a’ is 10% better than tech ‘b’ so if we switch to using ‘b’ we should be able to complete project ‘x’ in 10% less time), maintenance (requirements, analysis, design, implementation, post delivery maintenance, retirement)
· Real life example of the life cycle model mentioned above: at Mindseye we a) met with the client to get the project [requirements], b) produced a features and functions document which was agreed upon before work started [analyis], c) produced the black lines / specifications [design], d) implemented the site [implementation], e) entered into support contract [post delivery maintenance] or existing resources took over the project. The one thing we didn’t do was design of the software itself, our design / specification document was made up mostly of an object model and black lines (similar to what most people would call information architecture).
· Maintenance often happens before the project is released ( because of changes to the design etc…). Only good software actually makes it to post delivery maintenance phase because bad software is thrown away well before that time (or is never implemented)
· The book at this points presents a trivial example of how encapsulation is “a good thing” by showing how it would be much better to use this:
public static final float salestax = 6.0;
than to insert 6.0 in multiple places in your code (obviously). The funny thing is that by using a float to represent sales tax, he unwittingly introduced a bug into his code.
· Average cost percentages per phase (requirements, analysis, design, etc..) has hardly changed over the 20 or so years (~20% to requirements, ~20% to design, ~35% to coding / unit testing, ~25% to integration) but much more interesting is the fact that more than 75% of the total cost of a project is related to post delivery, so anything that can reduce the amount of time and money that you spend in that phase should probably be closely looked at.
· requirements / design / analysis:
— fixing costs more when you find it later (if you’re using the classical software development model)
— techniques that help you complete the requirements / analysis / design phase better will help save quite a bit of money and time up front
· planning phase: Not included because planning is usually carried out all through the life cycle; mostly during the beginning of the project and the software project management plan
· testing: needs to be happening all the time althoguh the book seems to think more of a QA group than of unit / automated testing
· documentation phase? No because the documentation should be kept up at all times so there ahould never be a time when it is out of date
· Object Oriented
— strengths of OO: post delivery maintenance is easier because of information hiding, OO makes more sense when modeling (which should lead to higher quality software), encapsulation (OO is sometimes referred to as responsibility-driven design or design by contract), classical paradigm results in a set of modules that is conceptually a single uit, OO results in smaller largely independent units ,OO promotes reuse
— diff between OO and classical:
— OO has workflows, classical has phases
— OO starts integrating classes almost right away while classical doesn’t integrate modules until the design phase.
Automated Application Deployment with Tomcat: Part II with Subversion And Ant
Got around to writing the deployment scripts that go with the Subversion work I did last week. As part of the scripts, I also worked on the problem I wrote about a couple weeks ago (ie: needing different log4j properties files per environment). Once you have a source control solution and a master build file, it’s pretty easy. I stripped out the explicit log4j category and appender directives and instead used the Ant <propertyfile> tag to write out the appropriate category / appender directives based on the presence (or lack thereof) of the project.debug property in my build file. When I want to create a build that outputs logging information to the console, this block of code is fired:
<target name="build.resources.debug" if="project.debug">
<propertyfile
file="WEB-INF/classes/log4j.properties">
<entry key="log4j.rootCategory" value="INFO, STDOUT"/>
</propertyfile>
</target>
Otherwise, this block is run:
<target name="build.resources.production" unless="project.debug">
<propertyfile
file="WEB-INF/classes/log4j.properties">
<entry key="log4j.rootCategory" value="ERROR, SMTPAPPENDER"/>
<entry key="log4j.category.com.mycompany" value="ERROR, SMTPAPPENDER"/>
</propertyfile>
</target>
In Eclipse, you can easily create the project.debug property using the Arguments text box provided in Run –> External Tools –> $your ant build –> Main Tab –> Arguments: -Dproject.debug=true. By default then, log4j will only log error conditions and will send them to the SMTP appender.
The second thing I did was to create a deployment Ant script that wrapped my main build.xml. It checks out the source code from Subversion:
<exec executable="svn">
<arg line="co svn://sourcecodeserver/myproject/branches/v1.0 . --username username --password password"/>
</exec>
<ant antfile="build.xml" />
<exec executable="net">
<arg line='stop "Apache Tomcat"'/>
</exec>
<delete dir="${tomcat.install}myapp"/>
<mkdir dir="${tomcat.install}myapp"/>
<unzip src="myapp.war" dest="${tomcat.install}myapp"/>
<exec executable="net">
<arg line='start "Apache Tomcat"'/>
</exec>
We’ll see how well it works once this goes into production. If nothing else, at least I’ll have a couple more hours in my day.
Source Control with Subversion on Windows
We’re getting around to hiring an additional me at work so I needed to get a source control solution in place before he/she arrives and starts hacking away at the codebases. At the last place I worked we used Visual SourceSafe for a long time (which was pretty worthless) and then started using CVS, although not on the projects I worked on. A couple weeks ago I asked a bunch of questions about automating Java web application deployment and Erki suggested that I buy and read the Pragmatic Project Automation book, which I did and which I loved. I read it from start to finish (which isn’t an easy thing to do with a technical book) and took a ton of notes. Anyway, I’m pretty sure the book didn’t mention Subversion even once, but it did drill home the importance of setting up your code in source control and then doing your manual, scheduled, automated (using CruiseControl) or production builds directly from the source control system using Ant or Groovy, which led to alot of ideas about how you could automate builds to production Tomcat systems. I’ve heard alot about Subversion; it’s described as CVS with none of the warts and all of the features, so I tried it out. Following are the notes I kept during the installation process. Hopefully this helps out someone somewhere down the road:
a) Download Subversion (obviously) I downloaded version 1.1.1 ( svn-1.1.1-setup-2.exe)
b) Run the install. I chose to install to c:\subversion\.
c) Test the Subversion install by going to the install directory and then going to the /bin directory. Run the svnadmin from the command line:
CD c:\subversion\bin\
svnadmin
If you get an error that says that says “svnadmin.exe – Unable To Locate DLL”, you’ll need to either a) install the Visual C++ 6.0 VCREDIST.exe, which is available here (but only if you’re using Windows 98 or Windows ME) or b) find someone or some system that has msvcp60.dll and copy that file to c:\$windows\system32\ (where $windows is the directory that contains your Windows installation).
d) Create a repository on the machine that is going to host your source code. From the command line on the server machine:
svnadmin create d:\$projects\$projectname
where $projects is the name of a folder that will contain all your Subversion source code repositories and $projectname is the name of the specific project you’re going to create a repository for.
e) Start the server either from the command line:
svnserve -d -r d:\$projects
or use the SVN Service Wrapper (http://dark.clansoft.dk/~mbn/svnservice/), note that the r switch limits directory browsing by specifying the root of your repositories.
f) Create a users database (which is nothing more than a text file, I created a file called users.conf) and save it in the d:\$projects\$projectname\conf directory and then add your usernames & passwords. An example user database might look like this:[users]
aaron = mypassword
g) Within the d:\$projects\$projectname\directory, open the conf/svnserve.conf file using your text editor of choice and add the following to allow reading and writing to the users in the database we just created:
[general]
anon-access = none
auth-access = write
password-db = users.conf
realm = $projectname
where $projectname is the name of the project you’re setting up and users.conf is the name of the file you created in step f.
h) Finally, restart the server and you’re ready to start using the client and import your project.
i) To import an existing project, invoke the svn executable from the command line on the client:
svn import c:\temp\myproject svn://myserver/myproject -m "initial import" aaron mypassword
where c:\temp\myproject is the directory that contains the existing project you want to import (and should have 3 folders: trunk, tags and branches, your source code should be in the trunk folder; more on repository layout here), myserver is the name of the server that contains the Subversion repository you created in step d, myproject is the name of the project (ie: $projectname from step d) you’re importing into, the m switch signifies the message you want to tag this import with, followed by the username and password you setup in the users database in step f.
j) Finally, checkout the project from the server (invoking the svn executable from the directory that you want the trunk checked out from…, ie: c:\projects\myproject)
svn co svn://myserver/myproject/trunk . aaron mypassword
k) To add, move, delete, or edit and then commit any changes back to subversion:
Add: svn add src/myfile.java -m "adding a file"
Move: svn move src/myfile.java src/mynewfile.java -m "moved myfile.java to mynewfile.java"
Delete: svn delete src/myfile.java -m "removing a file"
Commit: svn commit src/myfile.java -m "the message"
That’s it… there’s a alot more to read. One of the coolest things about Subversion is that the documentation is all packaged up into a free PDF published by Oreilly, which you can download here. I printed out the manual and breezed through it in a couple hours. You should too.
I did try to get the Subclipse Eclipse Subversion plugin working, but for the life of me… I couldn’t. I think it ended up for the better anyhow because while the command line is a bit more tedious, I understand more about what’s going on in the system, which is helpful. I’d recommend that you get familiar with the command line client before attempting to use a GUI plugin.
After you’re all set with Subversion and you’re ready to get started with CruiseControl, note that while Cruisecontrol contains an <svn> tag, you still need to drop down into command line mode in Ant to checkout / update your local repository when doing an automated build. For example:
<exec executable="svn">
<arg line="update"/>
</exec>
Blog Spam and Parasites
So I run a couple blogs and like many people I haven’t solved the blog spam problem yet (even MT-Blacklist only works some of the time and it’s a losing battle). There obviously have been alot of people trying to solve the problem, but it’s interesting to watch from another angle: blog spammers (I guess all spammers) are effectively parasites living off a host system. For the most part, they don’t bother the host system too much. One, two, maybe ten comments per day and we both keep on living. It’s the times like last night when I got 418 spam comments on my blog that I think I should turn off comments entirely but killing the host doesn’t seem like the right answer.
A couple of weeks ago I listened to Janice Benyus’ presentation on biomimicry (courtesy of Jason Kottke, you can download and listen to the presentation in MP3 format). In it she talks about 12 ways in which nature does things better than we do: self assembly, the power of shape, natural selection as innovation (amazing to hear how nature in many ways is still light years ahead of our man-made technology). It’s interesting stuff if you’re into that kind of thing, but then you probably wouldn’t haven’t gotten this far if you weren’t. Anyway, that presentation popped back into my head today while I was deleting all 418 comments: how does nature deal with the problem of parasites? I guess it goes without saying that nature hasn’t solved the problem either because the earth is really old… and we’ve still got a ton of parasites. Malaria, giardia, fleas and ticks are all rampant abusers of animals and humans (reading the Wikipedia entry for Malaria reminds me of the malaria pills I had to take when my pop took my brother and I to Kenya for 2 weeks… I chewed one by accident, talk about a bitter pill!) But their existence doesn’t mean that nature (or humans) haven’t tried to get rid of parasites. Doctors and researchers have been trying to eradicate malaria from much of the world for many years using a variety of methods:
· attacking the mosquito population: Reminds me of a post by Tim Bray (Crooks in Plain Sight), why not hunt down the people who do the blog spam (we certainly know where they live on the web)? Tim, or perhaps someone else, suggests that no one wants too. Better yet, no one has the time or energy to do so. Probably true.
· genetically modifying mosquitoes so that they don’t carry malaria: I think this would be analogous to asking the search engines (Google, MSN, Yahoo, etc..) to do something about the problem (ie: modify the Google search algorithm so that even if comments gets on your blog and stays there, it doesn’t influence the PageRank of the link they post). Would this happen? Unlikely. Any change Google makes would be countered by changes from the parasite/blog spammer. That’s a losing battle.
· distributing mosquito nets to areas of the world where the problem is most severe: A mosquito net? I think this is the MT-Blacklist / scode of our disease control toolkit. Put a net over our blogs so that the parasites can’t get it. Works for someone, but complete eradication will only work if everyone has a net. The parasites will continue to flourish on all the unprotected blogs.
· releasing millions of sterile mosquitoes into the wild (Sterile insect technique):
: The idea here is that by releasing millions of sterile mosquitoes, males will fruitlessly attempt to fertilize eggs.. females will lay eggs that can’t ever hatch. I think this would be analogous to a bunch of people getting together and making sites about incest, kid porn and beastiality. Neutralize the blog spammers efforts by beating them at their own game (which we’ve proven we can do).
·through chemicals like DDT: No idea what the corollary would be in the blog world, most likely like the first (attacking the mosquito population)
·by developing a vaccine for malaria: Here’s an interesting idea: instead of distributing nets like MT-Blacklist or scode (which block spam at the point of attack), why not modify our systems that they become immune to the affects of the parasite? Maybe this is modifying the comment URL so that it’s a redirect rather than a direct link (already being done), adding comment registration (already being done), or a comment approval process.
I’m not sure a solution will ever be found (regular spam doesn’t seem to be getting any better)… if nothing else, it’s an interesting problem. Oh, and if you found it interesting, drop me a comment. 😉