Category Archives: SOAP

Debugging SOAP / XFire with ethereal

I’ve spent way more time than I should have the last couple weeks working to help migrate a website built against Jive Forums to run against a Clearspace X instance. As part of the migration, one of the things I did was to move all the data syndication that had been done with RSS and custom namespaces to use the Clearspace SOAP API, which is built on a technology called XFire. The first problem I ran into was that production website was configured so that requests to http://example.com were redirected to http://www.example.com/, which resulted in errors like this in the logs:

Jul 5, 2007 11:30:11 PM org.apache.commons.httpclient.HttpMethodDirector isRedirectNeeded
INFO: Redirect requested but followRedirects is disabled

That error was pretty easy to fix (swap in http://www.example.com in place of http://example.com), but the next thing I ran into was way less intuitive. When I invoked a certain service, I’d get a stack trace that looked like this:

Exception in thread "main" org.codehaus.xfire.XFireRuntimeException: Could not invoke service.. 
Nested exception is org.codehaus.xfire.fault.XFireFault: Unexpected character '-' (code 45) in prolog; expected '<'
 at [row,col {unknown-source}]: [2,1]
org.codehaus.xfire.fault.XFireFault: Unexpected character '-' (code 45) in prolog; expected '<'
 at [row,col {unknown-source}]: [2,1]
	at org.codehaus.xfire.fault.XFireFault.createFault(XFireFault.java:89)
	at org.codehaus.xfire.client.Client.onReceive(Client.java:386)

which was troubling because the exact same SOAP method invocation worked fine on both my local machine and in the test environment. What was different? Two things: the production system was running on Java 6 and the production system was configured to run behind an Apache HTTP server proxied by mod_caucho versus no Apache HTTP server / proxy in development or on my machine. I needed to see what was going on between the server and the client (one of the things that makes SOAP so hard is that you can't just GET a URL to see what's being returned) so I fired up ethereal at the behest of one of my coworkers. I kicked off a couple of SOAP requests with ethereal running, recorded the packets and then analyzed the capture. Said coworker then pointed out the key to debugging HTTP requests with ethereal: right click on the TCP packet you're interested in and then click 'Follow TCP Stream'. The invocation response looked like this when run against the development environment:

HTTP/1.1 200 OK
Date: Mon, 02 Jul 2007 21:59:30 GMT
Server: Resin/3.0.14
Content-Type: multipart/related; type="application/xop+xml"; start=""; start-info="text/xml"; .boundary="----=_Part_5_25686393.1183413571061"
Connection: close
Transfer-Encoding: chunked

1dce

------=_Part_5_25686393.1183413571061
Content-Type: application/xop+xml; charset=UTF-8; type="text/xml"
Content-Transfer-Encoding: 8bit
Content-ID: 
...

and looked like this when invoked against the production instance:

HTTP/1.1 200 OK
Date: Mon, 02 Jul 2007 21:41:56 GMT
Server: Apache/2.0.52 (Red Hat)
Vary: Accept-Encoding,User-Agent
Cache-Control: max-age=0
Expires: Mon, 02 Jul 2007 21:41:56 GMT
Transfer-Encoding: chunked
Content-Type: text/plain; charset=UTF-8
X-Pad: avoid browser bug

24e

------=_Part_29_31959705.1183412516805
Content-Type: application/xop+xml; charset=UTF-8; type="text/xml"
Content-Transfer-Encoding: 8bit
Content-ID: 
...

Notice the different content type returned by the production server? So then the mystery became not 'what?' but 'who?' I googled around for a bit and found a bug filed against JIRA that had all the same symptoms as the problem I was running into: the solution posted in the comments of the bug said that the problem was with mod_caucho. I worked with the ISP that hosts the production instance of Clearspace, got them to remove mod_caucho and use mod_proxy to isolate that piece of the puzzle and sure enough, the problem went away. Our ISP recommended that we not settle for mod_proxy for the entire site and instead wrote up a nifty solution using mod_rewrite and mod_proxy, which I've pasted below:

 RewriteRule ^/clearspace/rpc/soap(/?(.*))$ to://www.example.com:8080/clearspace/rpc/soap$1
 RewriteRule ^to://([^/]+)/(.*)    http://$1/$2   [E=SERVER:$1,P,L]
 ProxyPassReverse /community/rpc/soap/ http://www.example.com/clearspace/rpc/soap/

Hope that helps someone down the road!

Fiddler, .NET and SOAP

This morning I spent some time playing with WSE, .NET and Java. I knew Fiddler could listen in on conversations between IE and the world, but I never was able to get it to listen to conversations between me and well, me (ie: localhost). Turns out, according to the well named help page, that

“….NET will always bypass the Fiddler proxy for URLs containing localhost. So, rather than using localhost, change your code to refer to the machine name.”

Easy.

Update to embedded Axis application in Tomcat

I got a great email from Tamás in response to my last post who pointed out that the straight copy of deploy.wsdd to server-config.wsdd doesn’t cut it. More importantly, he mentioned that there is a utility that ships with Axis that allows you to generate server-config.wsdd from your deploy.wsdd (or from multiple deploy.wsdd if you have multiple web service end points). From the command line it looks like this:

> java -cp axis.jar;jaxrpc.jar;commons-logging.jar;commons-discovery.jar;saaj.jar;
org.apache.axis.utils.Admin server dir1\deploy.wsdd dir2\deploy.wsdd

But if you’re using the Ant build.xml I provided in the previous example, you’d use this:

<java
  classname="org.apache.axis.utils.Admin"
  fork="true"
  failonerror="true"
  classpathref="compile.classpath"
  dir="${basedir}\WEB-INF\">
   <arg value="server" />
    <arg file="${basedir}\deploy.wsdd" />
</java>

I updated the source code example (embeddedaxis.zip), you can download it here.

NOTE: The source code for the Admin class is available here, where you can see (but the documentation doesn’t mention) that the Admin class accepts multiple WSDD files from the command line.

Embeddding an Apache Axis application in Tomcat

One of the applications I’ve been heading up at my 8-to-5 needed a SOAP API that fronted a Java application deployed on Tomcat. If you’ve spent any time with Axis you know that it’s not the simplest thing to deal with; in fact it’s downright complex if you want do anything more than the simplest thing using SOAP. The simplest? Write your source code, rename the .java file to have a .jws extension and then copy the file into a public directory inside your servlet container (see ‘Deploy a Java Class as a Web Service’). Easy. But this method leaves alot to be desired: you can’t use packages in the source code and the code is compiled at runtime which means you don’t find out about compilation errors until after deployment (one way around this would be to first compile the .java file to make sure that it works and then use Ant to copy / rename the .java file to a .jws) and you can’t specify custom type mappings, among other things. Lucky for you, the jws method isn’t the only way you can do it.

The next two options give you flexibility with the additional cost of complexity. The first method is well covered in the documentation: Axis comes packaged with a web application that you can deploy to your servlet container and then add your custom services using a remote administration client also provided with the war file that you deploy. The downside (at least in my environment) is that this means you now have to maintain and deploy two separate applications: my Java based web application would be deployed to Tomcat and then the same business logic would be deployed to the Axis engine running inside of Tomcat. I felt that it would be simpler to maintain to instead deploy the web application and the SOAP application together as one application in one war file. That of course, is not well documented (in fact other than a PDF file that’s part of the ‘Java Development with Ant‘ written by Erik Hatcher, there is no mention of deploying your web application alongside an Axis application without using the Axis AdminClient). Hence the article you’re reading now.

step 1: setup application / dependent libraries: I’ll make the assumption that you already have a web application that you want to expose SOAP webservices with which means you probably have a file structure that looks like something like this:

index.jsp
  - WEB-INF
    web.xml
    -classes
    -lib

You’ll need to add the following libraries to your WEB-INF/lib directory:
* axis.jar
* axis-ant.jar
* jaxrpc.jar
* wsdl4j.jar
* commons-logging.jar
* commons-discovery.jar
* saaj.jar

If you don’t, you can download the sample application I wrote that exposes a single hello world webservice.

Alright, so either you’ve downloaded the sample application or you’ve got your own application and you’ve added the above libraries to your WEB-INF/lib directory. Now you’re ready to write some script to expose your existing classes.

step 2: use WSDL2Java / Java2WSDL to generate the server side wrapper & deployment descriptors for the classes you want to expose. Write a interface:

package net.cephas.soap;
public interface HelloWorld extends java.rmi.Remote {
  public java.lang.String sayHello(java.lang.String in0)
    throws java.rmi.RemoteException;
}

and then a corresponding implementation which should be named $InterfaceName + SoapBindingImpl

package net.cephas.soap;
import java.rmi.RemoteException;
public class HelloworldSoapBindingImpl implements HelloWorld {
  public String sayHello(String name) throws RemoteException {
    return "hello " + name;
  }
}

Next, you can use Ant (but you could easily run this from the command line as well) to create the WSDL and generate the server side wrapper & deployment descriptors. You’ll need to define two tasks in Ant:

<taskdef name="axis-java2wsdl" classname="org.apache.axis.tools.ant.wsdl.Java2WsdlAntTask">
  <classpath refid="compile.classpath" />
</taskdef>
<taskdef name="axis-wsdl2java" classname="org.apache.axis.tools.ant.wsdl.Wsdl2javaAntTask">
  <classpath refid="compile.classpath" />
</taskdef>

and then you can generate the WSDL:

<axis-java2wsdl
  classname="net.cephas.soap.HelloWorld"
 style="rpc"
  namespace="urn:soap.cephas.net"
  location="http://localhost:8080/${project.distname}/soap/helloworld"
  output="${basedir}\generated\helloworld.wsdl">
</axis-java2wsdl>

and generate the server side wrappers:

<axis-wsdl2java
  output="${basedir}\generated"
  serverside="true"
  skeletondeploy="true"
  url="${basedir}\generated\helloworld.wsdl"
  verbose="true"
  debug="true">
</axis-wsdl2java>

Copy the generated web service deployment descriptor (deploy.wsdd) file to WEB-INF/server-config.wsdd:

<copy file="generated/net/cephas/soap/deploy.wsdd"
  tofile="WEB-INF/server-config.wsdd" />

and the *SoapBindingSkeleton.java file to your source tree:

<copy todir="src/net/cephas/soap" includeEmptyDirs="no">
  <fileset dir="generated/net/cephas/soap">
    <patternset>
    <include name="*SoapBindingSkeleton.java" />
    </patternset>
    </fileset>
</copy>

Finally, compile your source and either jar it up to the WEB-INF/lib directory or deploy the compiled classes to the WEB-INF/classes directory.

step 3: configure web.xml with the appropriate servlet mappings. The last thing you need to do is to map a resource path in your application to Axis, you accomplish this by adding a servlet and servlet-mapping element to your web.xml:

<servlet>
  <servlet-name>AxisServlet</servlet-name>
  <display-name>Apache-Axis Servlet</display-name>
  <servlet-class>
    org.apache.axis.transport.http.AxisServlet
  </servlet-class>
</servlet>
<servlet-mapping>
  <servlet-name>AxisServlet</servlet-name>
  <url-pattern>/soap/*</url-pattern>
</servlet-mapping>

You can see that I’ve selected ‘/soap/’ as the resource path which in combination with the server-config.wsdd means that I’ll invoke the SOAP web service using a URL like this:

/embeddedaxis/soap/helloworld?wsdl

where embeddedaxis is the name I’ve the sample application. It will probably be different for your application.

Phew! That’s an awful lot of scripting and configuration to deploy hello world, but at least now you don’t have to rely on .jws files or deploying your application separately from the SOAP service. If you’re trying to do the same thing and something you read above doesn’t make sense, ping me.

Download embeddedaxis.zip.

eBay Java / C# SOAP Examples

At the start of this year I worked with some guys at eBay to further develop their code samples. Some, but not all of the twelve examples I wrote went live in the recently launched Community Codebase. You can download all the Java examples (of which I wrote three) or browse the Subversion repository. I wrote the SOAPAddItem, SOAPGetItem, and SOAPGetUser items in the Java source tree.

The examples I wrote were different from the majority of the (then) existing examples in that I didn’t make any IDE assumptions (most of the Java examples have JBuilder .jpx files and Eclipse .project files, the .NET projects contain the Visual Studio artifacts.. tsk tsk.) and as such, all the examples I wrote contain comprehensive Ant / NAnt build files which means you can get up and running without having to setup your fancy schmancy IDE. But the biggest difference was that all the examples I wrote used either Ant (with WSDL2Java) or NAnt (with NAntContrib) tasks to conditionally download the eBay WSDL, generate the client stub(s), and compile the resulting code, which makes for a prettier source code repository (generated stubs aren’t checked into source) and gives you compile time checking of your code against the API.

If you’re interested in how you can use Ant or NAnt in a build environment where you access SOAP services, you should check it out!

Fun with Axis / SOAP, Java and Calendar

The fun with dates didn’t stop at the last post. This one took longer to track down but I learned alot more from it. I’m working on both sides of an Apache Axis SOAP implementation where the server side system persists events and client applications can query to find events and CRUD events against this server side application. So the server presents an event:

...
public class EventRemote {
private String name;
private String address;
private Calendar eventdate;
...
// getters & setters omitted
}

and methods that allow client applications to search for events and create / retrieve / update and delete them:

...
public class EventManagerRemote {
public void updateEvent(EventRemote event) {
...
}
public void createEvent(EventRemote event) {
...
}
public EventRemote findEventByID(long eventid) {
...
}
// other methods omitted
}

For the last couple weeks the clients and servers had no problems, create, retrieve, update, delete and search all worked as expected. But recently a user requested a change to the UI of the client system which resulted in a change to the way the client application needed to format the eventdate (a Calendar object). In the past the client would retrieve the event and then format the eventdate property using a SimpleDateFormat:

EventRemote event = manager.findEventByID(12);
Calendar eventdate = event.getEventdate();
SimpleDateFormat formatter = new SimpleDateFormat("M/d/yyyy h:m aa");
String strdate = formatter.format(eventdate.getTime());

So if my event was today (June 30, 2005 at 7:30pm), the above snippet would print:

6/30/2005 7:30 PM

The change to the client I mentioned above meant that I needed to access each element of the Calendar object separately, so instead of having one big datetime string, I need to know that the month was June, the date ’30’, the year ‘2005’ and so on. Since the getMonth()/Day()/Year() methods of the Date class have been deprecated since v1.1, I had to use the Calendar class get(int field) methods. Not a problem, there’s some different things you have to work through in the Calendar class (ie: the month field has values that range from 0 to 11 instead of 1 to 12), but it’s doable:

EventRemote event = manager.findEventByID(12);
Calendar eventdate = event.getEventdate();
int day = eventdate.get(Calendar.DATE);
int year = eventdate.get(Calendar.YEAR);
...
int hour = eventdate.get(Calendar.HOUR);
int minute = eventdate.get(Calendar.MINUTE);
int am_pm = eventdate.get(Calendar.AM_PM);

I printed out the results (again assuming a date of June 30,2005 at 7:30pm) and instead of getting that same date I got this:

6/30/2005 11:30 PM

which was exactly 4 hours ahead of the time I wanted. I dug into Axis source code and saw that the CalendarDeserializer class explicity sets the TimeZone of the returned Calendar object to GMT (to make it UTC right?). Understandably, this causes the HOUR property of the Calendar object to jump four hours ahead (I’m in EST, GMT/UTC is currently 4 hours ahead of EST, it’ll be 5 hours during the winter months, explanation). The workaround (although I’m not sure it’s a good idea), was to set the TimeZone of the returned Calendar object back to EST. This probably doesn’t make alot of sense without seeing some sample code:

Calendar calendar = new GregorianCalendar(2005, 5, 30, 19, 30);
String utcformat = "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'";
SimpleDateFormat zulu = new SimpleDateFormat(utcformat);
String serializeddate = zulu.format(calendar.getTime());
// string gets sent along the wire via soap
// CalendarDeserializer parses date string
Calendar deserial = Calendar.getInstance();
Date deserialdate = zulu.parse(serializeddate);
deserial.setTime(deserialdate);
// CalendarDeserializer explicitly sets TZ to GMT
deserial.setTimeZone(TimeZone.getTimeZone("GMT"));
SimpleDateFormat formatter = new SimpleDateFormat(utcformat);
// will print '2005-06-30T19:30:00.000Z'
System.out.println(formatter.format(deserial.getTime()));
// will print '11', should print '7'
System.out.println(deserial.get(Calendar.HOUR));

So I understand that the date gets sent along the wire in UTC format, which strips out timezone information the date was originally encoded with, my question is this: why does the HOUR property of the Calendar object reflect UTC time (ie: 11:30pm) while the formatted time reflects EST time (7:30pm)?

Using Betwixt, Struts and REST

My latest project at work launched a couple weeks ago, unfortunately there isn’t much to look at, but it’s not like I’m a Flash developer, so unless I start writing Windows apps, no one will ever be able to see my work anyway. Alas, a major portion of the project involved consolidating some business logic behind a SOAP API (I’ll go into that some other time). We were supposed to be moving our front end to ASP.NET but at the last minute, the ASP.NET migration didn’t happen so we ended up needing something that ASP could use as well, which meant REST. With the business logic wrapped up behind business delegates, all that was needed as a Servlet Filter to map requests for /myapp/xml to a Struts Action class, which invoked the business delegates and then returned XML to the client. Now on the SOAP end of things (Axis embedded in Tomcat), I didn’t have to deal with XML (Axis handles all the XML serialization and deserialization). But with REST you’re in conundrum: do you create toXML() methods on all of your JavaBeans? do you put the JavaBean or collection of beans in the Request scope and then generate the XML using JSP? Those options all seemed error prone and harder to test. I looked at JAXB (makes more sense when you have XML that you want to turn into a JavaBean) and Castor (same thing) but finally decided on using Betwixt, a Jakarta Commons library that provide a simple way of turning a JavaBean into XML. The end result looked something like this:

public class RESTAction extends DispatchAction {
  public ActionForward doSomething(...) {
    MyBean bean = BusinessLogic.getBean();
    response.setContentType("text/xml");
    OutputStream sos = response.getOutputStream();
    BeanWriter beanWriter = new BeanWriter(sos);
    BindingConfiguration config = beanWriter.getBindingConfiguration();
    config.setMapIDs(false);
    beanWriter.enablePrettyPrint();
    String declr = "<?xml version='1.0' encoding='UTF-8' ?>";
    beanWriter.writeXmlDeclaration(declr);
    beanWriter.write("mybean", bean);
    beanWriter.flush();
    beanWriter.close();
    return null;
  }
}

which when combined with this action mapping:

<action
  path="/xml"
  type="com.mycompany.web.actions.RESTAction"
  name="xmlForm"
  scope="request"
  parameter="method"
  validate="false">
  <forward name="error" path="/WEB-INF/views/xml/error.jsp" />
</action>

is invoked like so:

http://myserver/myapp/xml?method=doSomething

I’m not 100% sold on this as the “way”, but I do like how simple Betwixt is. If you’ve got a different idea, comment away.

ebay web services talk by Jeffrey McManus

A couple weeks ago I attended the Boston .NET User Group to hear Jeffrey McManus give a talk on how ebay is using web services. I took a bunch of notes during his presentation, which I’m putting up here semi-edited. The first part of the talk was mainly about ebay the company; how much business they do per quarter, how many users they have,etc.. The second part was about the ebay API, which I was interested in because the company I’m working for now is exploring a similar program, so you’ll see more detailed notes there:

developer.ebay.com

45000 product categories
collectibles was the first product category on ebay, it is category id = 1
ebay did approximately $5.6 billion in cars last year

105 million registered users as of 3/1/2004
rate of growth is accelerating quarter over quarter
28 countries with a physical presence and localized website
20 million items for sale at any one time…
1 billion items were listed last year
7 gigabits per second outgoing traffic
10 billion api hits in 2004

in december 2003, 1 of every 3 people on the internet came to ebay

1 in 3 used golf clubs bought in the US are purchased on ebay
valueguide.pga.com uses ebay data to help determine the price of used golf equipment

Ebay isn’t just for knick knacks: more than 125,000 keyword searches for Louis Vuitton, Coach and Prada on ebay every day

superpawn.com
— they build a proprietary POS with ebay and significantly reduced costs
— used to cost $23 per item to list & describe each item on ebay (person costs), now approx $.25

highline auctions
— custom system integrator with a business around ebay integration, specifically in cars
— reduced the time it takes to create a professional listing from 30 minutes to under 3 minutes

auctionworks
— solution provider on ebay

use cases
— integration w/ business tools (quickbooks, excel, sap, etc..) (ie: a donation can be estimated for taxes using a data licensing application)
— integration w/ back ends for medium & large retailers, manufacturers

‘we make inefficient markets efficient’
scarce –> in-season reatail –> end of life–> used/vintage
— end of life scenario for retailers
— products at the end of life can be put on ebay programmatically saving companies the most of having to get rid of excess product

top level meta categories
— sports, tickets, etc…
— ‘browseability’ is a one of the challenges
— most likely use cases are sellers listing items programatically

data licensing services
— you can purchase data feeds from ebay so that you can find out what the given price for a given car in a given location is
andale.com: provides ebay pricing and selling recommendations

API

Amazon Web Services Newletter cool links

The February Amazon Web Services Newsletter had some pretty cool applications:

Blogfuel: “Use blogfuel to create a link to an individual item at amazon (using ASIN Search) to illustrate your book review. Or place the code in the sidebar of your blog as a “What I am currently reading feature”. You have control over the width, colour, number of columns, image and text size.

PocketBooks – Amazon: “Check prices and availability of books at Amazon.com and Amazon.co.uk from your Java enabled phone

Books of note, from Sybex: Mining Amazon Web Services: Building Applications with the Amazon API and from Dimensions: Google, Amazon, and Beyond: Creating and Consuming Web Services