Till almost 2 years back, Google was synonymous with Search. For my brother and father, both doctors, people who have become Net savvy very recently - Net means one of their Medical site and Google for anything that they cannot find otherwise.
However, if you just look around today, you hear Google coming up in far too many conversations - apart from the Search domain. You cannot help but wonder - What is Google upto?
It is amazing to see the vast array of tools Google is providing and the technologies Google is dabbling with:
GMail - has totally redefined e-mail applications
Google Talk - Like many others, I am taken in by its simplicity and useful features
GTalk integration with Gmail - cool idea!
Google Calendar
Google Spreadsheets
Google Checkout - haven't really checked it out :-)
Google Search API - now you can integrate Google Search into your web applications
Google Web Toolkit - awesome engineering idea
Google Maps
Google Earth - great, despite all the furore over security and privacy
So much more than the great Search Engine they are famous for!
Tuesday, August 29, 2006
There is so much out there to learn!
This post isn't about any specific task that was giving me problems and about how and what I did to work around the problem..
Instead this is just a very humble confession - there is so much, so much out there that I don't know about. So much happening on so many fronts, and I as a developer / engineer sitting in the front of a desktop machine in a company in Pune, India have not yet seen even the tip of the iceberg for that matter.
It is overwhelming at times to just log onto google and see the terabytes and gigabytes of data Google throws at you to read from. Over the last few days / months, I have been doing what every engineer's dream could perhaps be (especially when he is bogged down by deadlines and not finding any extra time for himself). I have been reading up stuff, checking out new technologies, searching extensively on the Net and reading new terms almost every day.
What I need to do in the coming few days - Look at SWT, Eclipse plug-in framework, Graphical Editor framework, Draw 2D (I know this isn't new, but GEF seems to be a very good piece of design work). If this isn't enough, I am now also hooked onto Web 2.0 (whatever that means!) - AJAX, Google Web Toolkit and everything that comes along with it, DOJO, Scriptaculous, AJAX Comet - phew!
One caveat is however to not get lost by this bombardment of information and be able to instead have a definite path carved for myself and my team.
Instead this is just a very humble confession - there is so much, so much out there that I don't know about. So much happening on so many fronts, and I as a developer / engineer sitting in the front of a desktop machine in a company in Pune, India have not yet seen even the tip of the iceberg for that matter.
It is overwhelming at times to just log onto google and see the terabytes and gigabytes of data Google throws at you to read from. Over the last few days / months, I have been doing what every engineer's dream could perhaps be (especially when he is bogged down by deadlines and not finding any extra time for himself). I have been reading up stuff, checking out new technologies, searching extensively on the Net and reading new terms almost every day.
What I need to do in the coming few days - Look at SWT, Eclipse plug-in framework, Graphical Editor framework, Draw 2D (I know this isn't new, but GEF seems to be a very good piece of design work). If this isn't enough, I am now also hooked onto Web 2.0 (whatever that means!) - AJAX, Google Web Toolkit and everything that comes along with it, DOJO, Scriptaculous, AJAX Comet - phew!
One caveat is however to not get lost by this bombardment of information and be able to instead have a definite path carved for myself and my team.
Thursday, August 24, 2006
Back to the basics of isolation levels and locking
In one of my older projects, we were using WebSphere's Scheduler in order to schedule tasks that will run at regular intervals or just once at the specified time. The WebSphere Scheduler exposes certain APIs and mandates that the task be implemented in a certain way. The business logic to be executed at the scheduled time (i.e. what you want the task to do) is to be written in the "process" method of a stateless session bean that implements the "TaskHandler" interface. The information about this stateless session bean is registered with WebSphere at the time of scheduling the task. WebSphere in turn invokes this process method at the scheduled time in order to invoke the task. WebSphere stores the information about the tasks in its own tables, the main one being the TASK table.
For a very long time, we were facing a peculiar, consistent problem when it came to actually starting the tasks at the scheduled time. The WebSphere Server used to hang whenever the WebSphere Scheduler tried to start a scheduled task. The only option and a way out used to be restarting the server and telling everyone around - hey, you can test the remaining application all you want, but please do not schedule any tasks to run immediately!
We did figure the problem later on, and it really turned out to be a very computer science basics issue.
It had to do with locking of the TASK table. At the time of starting a scheduled task, WebSphere Scheduler does the following:
1. Gets a "ROW" level "write" lock on the row of the TASK table that contains information about the current task to be executed. It updates the table with the new state and the next fire time of the task
2. Invokes the task's process method in order to actually execute the business logic therein.
3. Once the task completes and the process method returns, the WebSphere Scheduler commits the transaction and then releases the lock.
Thus, all the calls to the Scheduler API are transactional.
Now, what was happening in our case was:
In our process() method implementation, we were trying to update our internal data structures with the next fire time of the task and hence trying to read the task information from the TASK table. This is when it was trying to get a "ROW" level "read" lock on the TASK table. However, since the row was already locked by WebSphere Scheduler and the transaction was not yet complete, our request for a read operation was getting blocked. It was a typical "deadlock" situation, ultimately resulting in the server threads getting hung.
So the solution to our problem was as simple as not reading the information from the TASK table at that point of time and moving that code to a later point in time. It all boiled down really to the basics of isolation levels, locks, transactions, levels of locks etc.
Overall - time well spent that made me refresh my "database locking" concepts once again!
For a very long time, we were facing a peculiar, consistent problem when it came to actually starting the tasks at the scheduled time. The WebSphere Server used to hang whenever the WebSphere Scheduler tried to start a scheduled task. The only option and a way out used to be restarting the server and telling everyone around - hey, you can test the remaining application all you want, but please do not schedule any tasks to run immediately!
We did figure the problem later on, and it really turned out to be a very computer science basics issue.
It had to do with locking of the TASK table. At the time of starting a scheduled task, WebSphere Scheduler does the following:
1. Gets a "ROW" level "write" lock on the row of the TASK table that contains information about the current task to be executed. It updates the table with the new state and the next fire time of the task
2. Invokes the task's process method in order to actually execute the business logic therein.
3. Once the task completes and the process method returns, the WebSphere Scheduler commits the transaction and then releases the lock.
Thus, all the calls to the Scheduler API are transactional.
Now, what was happening in our case was:
In our process() method implementation, we were trying to update our internal data structures with the next fire time of the task and hence trying to read the task information from the TASK table. This is when it was trying to get a "ROW" level "read" lock on the TASK table. However, since the row was already locked by WebSphere Scheduler and the transaction was not yet complete, our request for a read operation was getting blocked. It was a typical "deadlock" situation, ultimately resulting in the server threads getting hung.
So the solution to our problem was as simple as not reading the information from the TASK table at that point of time and moving that code to a later point in time. It all boiled down really to the basics of isolation levels, locks, transactions, levels of locks etc.
Overall - time well spent that made me refresh my "database locking" concepts once again!
Tuesday, August 22, 2006
J2ME and Mappoint integration
I happened to read a very good article by Michael Yuan, titled: Let the mobile games begin. It is a pretty old article actually, but contains a nice comparison of J2ME and .Net CF platforms for mobile application development. One example in this article is about using Microsoft's Mappoint web service for getting directions, routes and maps for desired locations, thus providing location based services from the convenience of a mobile phone.
I decided to try it out, with my basic knowledge of J2ME and especially about web services support in J2ME. As described in the article, I did the following:
1. Generated Java stubs using WSDL2Java tool of Apache Axis from the Mappoint Staging WSDL (available at http://staging.mappoint.net/standard-30/mappoint.wsdl
2. Wrote a wrapper class MappointClient that used these stubs to invoke the Mappoint web service
3. Exposed this wrapper class as a web service on Apache Axis (This is a nice idea actually that helps save a lot of wireless bandwidth and time. Explained well in the article. This wrapper class serves as a facade, which in turn makes multiple calls to the Mappoint web service to get the required information, over the relatively faster wired network).
4. Used custom Axis HTTP Connection handler that supports digest authentication (required by Mappoint web service, but not supported by default by Axis, again explained well in the article)
5. Used a third party library ksoap to invoke the facade web service from a J2ME MIDlet.
Well, you will think that things should just work fine, since they are explained so well in the article.. Yes - they should have, but they did not.
When I ran the whole set up, and invoked the facade web service from my J2ME MIDlet using ksoap, I got the following exception, that made no sense to me for a very long time:
Server redirected too many times (20)
To debug the problem then, I added a few debug statements to the custom HTTP Connection handler to check out the values of the HTTP headers. I realized that I was getting a text/html response back, where as I should have been getting a text/xml content in the response. On digging a little deeper, I figured that I was getting a Proxy-Authenticate header in the response with the following HTTP status code:
407 Proxy Authentication Required
Thus, this clearly was a proxy server issue. My machine was behind my company's firewall, and something was going wrong with the proxy settings.
First Try:
I set the system properties: http.proxyHost, http.proxyPort, http.proxyUser and http.proxyPassword to the correct values.
Did not work.
Second Try:
May be, my tomcat server was not picking up the System properties, hence I explicitly passed these values as runtime parameters to Tomcat. (In Tomcat 5.5, you do this by opening the Java tab and specifying the runtime parameters)
This did not work either.
Third and a successful try:
Stumbled upon this link http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html that explained the meaning of the various HTTP status codes.
The explanation of error code 407, as given in the site mentioned above:
"Indicates that the client must first authenticate itself with the proxy. The proxy MUST return a Proxy-Authenticate header field containing a challenge applicable to the proxy for the requested resource. The client MAY repeat the request with a suitable Proxy-Authorization header field."
So, this clearly was how the proxy server was behaving in my case. I included a small fix given below in the custom HTTPConnection handler's writeConnection method:
String userPassword = "username:password";
String encoded = Base64Coder.encode(userPassword);
conn.setRequestProperty("Proxy-Authorization", "Basic "+encoded);
Tested with:
1. Apache Axis 1.2 (but should work just fine with Axis 1.4)
2. J2ME: CLDC 1.1 and MIDP 2.0
3. ksoap for web services support
4. Tomcat 5.5
It was fun doing this exercise. Learnt quite a few things and in the end got a nice looking, impressing J2ME MIDlet up and running - one which shows turn by turn directions given the start and end locations and shows a nice map with the route highlighted :-)
I decided to try it out, with my basic knowledge of J2ME and especially about web services support in J2ME. As described in the article, I did the following:
1. Generated Java stubs using WSDL2Java tool of Apache Axis from the Mappoint Staging WSDL (available at http://staging.mappoint.net/standard-30/mappoint.wsdl
2. Wrote a wrapper class MappointClient that used these stubs to invoke the Mappoint web service
3. Exposed this wrapper class as a web service on Apache Axis (This is a nice idea actually that helps save a lot of wireless bandwidth and time. Explained well in the article. This wrapper class serves as a facade, which in turn makes multiple calls to the Mappoint web service to get the required information, over the relatively faster wired network).
4. Used custom Axis HTTP Connection handler that supports digest authentication (required by Mappoint web service, but not supported by default by Axis, again explained well in the article)
5. Used a third party library ksoap to invoke the facade web service from a J2ME MIDlet.
Well, you will think that things should just work fine, since they are explained so well in the article.. Yes - they should have, but they did not.
When I ran the whole set up, and invoked the facade web service from my J2ME MIDlet using ksoap, I got the following exception, that made no sense to me for a very long time:
Server redirected too many times (20)
To debug the problem then, I added a few debug statements to the custom HTTP Connection handler to check out the values of the HTTP headers. I realized that I was getting a text/html response back, where as I should have been getting a text/xml content in the response. On digging a little deeper, I figured that I was getting a Proxy-Authenticate header in the response with the following HTTP status code:
407 Proxy Authentication Required
Thus, this clearly was a proxy server issue. My machine was behind my company's firewall, and something was going wrong with the proxy settings.
First Try:
I set the system properties: http.proxyHost, http.proxyPort, http.proxyUser and http.proxyPassword to the correct values.
Did not work.
Second Try:
May be, my tomcat server was not picking up the System properties, hence I explicitly passed these values as runtime parameters to Tomcat. (In Tomcat 5.5, you do this by opening the Java tab and specifying the runtime parameters)
This did not work either.
Third and a successful try:
Stumbled upon this link http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html that explained the meaning of the various HTTP status codes.
The explanation of error code 407, as given in the site mentioned above:
"Indicates that the client must first authenticate itself with the proxy. The proxy MUST return a Proxy-Authenticate header field containing a challenge applicable to the proxy for the requested resource. The client MAY repeat the request with a suitable Proxy-Authorization header field."
So, this clearly was how the proxy server was behaving in my case. I included a small fix given below in the custom HTTPConnection handler's writeConnection method:
String userPassword = "username:password";
String encoded = Base64Coder.encode(userPassword);
conn.setRequestProperty("Proxy-Authorization", "Basic "+encoded);
Tested with:
1. Apache Axis 1.2 (but should work just fine with Axis 1.4)
2. J2ME: CLDC 1.1 and MIDP 2.0
3. ksoap for web services support
4. Tomcat 5.5
It was fun doing this exercise. Learnt quite a few things and in the end got a nice looking, impressing J2ME MIDlet up and running - one which shows turn by turn directions given the start and end locations and shows a nice map with the route highlighted :-)
Get the WSDL styles basics right!
This blog is really a continuation of the earlier blog about J2ME Web Services, and document/literal support of Axis.
As it turns out, Apache Axis was generating the correct response - as far as the document/literal WSDL-SOAP binding is concerned. In my particular example, what I needed to use was the document/literal wrapped WSDL binding. Wish I had read this article before:
http://www-128.ibm.com/developerworks/webservices/library/ws-whichwsdl/
A very good article that really clears the confusion a novice like me might have around the different WSDL "style" and "use" attribtues and their meaning. A must read!
As it turns out, Apache Axis was generating the correct response - as far as the document/literal WSDL-SOAP binding is concerned. In my particular example, what I needed to use was the document/literal wrapped WSDL binding. Wish I had read this article before:
http://www-128.ibm.com/developerworks/webservices/library/ws-whichwsdl/
A very good article that really clears the confusion a novice like me might have around the different WSDL "style" and "use" attribtues and their meaning. A must read!
Monday, August 21, 2006
Playing around with J2ME Web Services Optional Package (JSR 172)
Problem at hand:
I have been exploring J2ME for the last few months. One of the things I wanted to do was to check out the J2ME Web Services Optional Package (JSR 172) and write a simple sample POC MIDlet that uses the J2ME Web Service Client APIs to invoke a web service. I had already tried using the third party library ksoap and had been successful at it.
One thing to note about J2ME Web Services Optional Package is that it supports only document/literal type of web services for now. So that meant figuring out how to write a document/literal service using Apache Axis.
The set-up I had at the server side was:
1. Tomcat 5.5
2. Apache Axis 1.2
The set-up at the client side was:
1. Sun Wireless Toolkit for CLDC version 2.5
2. Eclipse 3.2
3. Supported versions: CLDC 1.1 and MIDP 2.0
Server side:
1. Wrote a very simple HelloWorld class in Eclipse, with the following three methods:
a. String sayHello()
b. String sayHelloWithTime()
c. String sayHelloToMe(String me)
2. Exposed this simple HelloWorld class as a web service in Apache Axis 1.2. To make sure the web service is of type document/literal, I made the following settings in the wsdd file (apart from the usual ones): provider="java:RPC" style="document" use="literal"
3. Used to AdminClient tool to deploy the web service onto Axis.
And lo - my web service was deployed and accessible in no time.
Client side:
1. J2ME Web Services Optional Package does not have support for dynamic invocation of a web service (the way ksoap has). Therefore, it is necessary to generate static stubs (which is not that much of a problem really).
2. Sun Wireless Toolkit provides a Stub Generator utility that, given the location of the WSDL file, generates the J2ME client stubs.
3. Once the stub was generated, I wrote a simple MIDlet with only one form to use this stub class (that acts just like a normal local class) to invoke the web service.
Having done all this (of course I had achieved this after having spent some time on the Net), I was expecting things to work fine, although I soon realized that I had just opened a Pandora's box! I kept on hitting problems one after the other, and I kept on solving one at a time. Lately I have realized that I actually enjoy such small struggles, at the end of it they almost invariably give me some sense of an achievement, however small that might be..
OK - so lets see what all came out of the Pandora's box:
1. I kept on getting weird useless exceptions such as: Invalid element in server response, Invalid namespace, found xxx, expected yyy
2. While trying to find some logical explanation to what was happening, my fertile mind came up with the following few conclusions (which obviously proved wrong later):
a. That Apache Axis Server is generating a different response when the web service is invoked from a J2ME web service client (Yes - so you see, blame it on the JSR 172 implementation!)
b. The Apache Axis Server is not able to handle methods with the same signature (OK - so if nothing works, I can always pass dummy parameters to have all methods have different signatures, can't I?)
3. Something was definitely going wrong with the response, since I was not getting one that I expected (or rather the one that my stub was expecting), so I decided to use the oh-so-useful SOAPMonitor of Apache Axis.
Enabling the SOAPMonitor:
I enabled the SOAPMonitor by following the steps given here: http://www-scf.usc.edu/~csci571/2006Spring/axisinstall.html
My first conclusion (2 a) was soon proved wrong when I checked out the response I received from the server using both a J2ME and a normal J2SE web service client. In both cases, I was getting a wrong response. To give an example, the response I received was:
<soapenv:Body>
<meReturn xmlns="http://helloworldsample">Hello Arati</meReturn>
</soapenv:Body>
Where as according to the WSDL, I should have got the following: (Note the differences in the element name and the namespace)
<soapenv:Body>
<sayHelloToMeReturn xmlns="http://localhost:8081/axis/services/HelloWorldService">Hello Arati</sayHelloToMeReturn>
</soapenv:Body>
Some desperate attempts to get it working:
1. After cursing Axis 1.2 too much, I moved on to Axis 2. This however, turned out to be a wrong decision - did not really help, Axis 2 has been completed rearchitected, redesigned and I unnecessarily spent almost one day figuring out how to deploy a very simple web service on Axis 2 and then how to enable the SOAPMonitor on it.
2. I posted my queries, my doubts on the J2ME Forums and a few other forums (ex. Artima).
3. Having received no help from there, I moved on to the safe Axis 1.4 version. And then once again started fiddling around with the SOAPMonitor. One thing I realized (which I should have seen earlier) was that the SOAP request itself was not being formatted correctly. For example, on invoking the String sayHelloToMe(String me) method, the SOAP request was as follows:
<soapenv:Body>
<me xmlns="http://helloworldsample">Arati</me>
</soapenv:Body>
Ideally, with my basic knowledge of SOAP, XML et al, I knew that this is wrong. The response should definitely have contained the method name (sayHelloToMe) - how the hell is the Axis server otherwise going to understand which method to invoke?
4. This finally put me on the right track. I then doubted the document/literal support of Axis for the first time (that was a correct doubt) and on reading up a little realized the difference between the style "document" and style "wrapped". Check out this in detail in Axis's user guide at Service styles section
5. I made changes to the wsdd file to use style="wrapped" instead of document style, redeployed the web service, regenerated the stub classes using the stub generator and then checked out the MIDlet once again. This time I had hit the jackpot! The J2ME web service client worked beautifully, and the SOAPMonitor also started showing me expected results!
Well - so after a little fight and desperate searching on the Net for about 2 days (duh - I know I should have got this sooner than this) - I have cracked J2ME Web services + Axis combination, at least for the simple things..
Three cheers for the SOAPMonitor - Hip hip hurray, Hip hip hurray, Hip hip hurray!
I have been exploring J2ME for the last few months. One of the things I wanted to do was to check out the J2ME Web Services Optional Package (JSR 172) and write a simple sample POC MIDlet that uses the J2ME Web Service Client APIs to invoke a web service. I had already tried using the third party library ksoap and had been successful at it.
One thing to note about J2ME Web Services Optional Package is that it supports only document/literal type of web services for now. So that meant figuring out how to write a document/literal service using Apache Axis.
The set-up I had at the server side was:
1. Tomcat 5.5
2. Apache Axis 1.2
The set-up at the client side was:
1. Sun Wireless Toolkit for CLDC version 2.5
2. Eclipse 3.2
3. Supported versions: CLDC 1.1 and MIDP 2.0
Server side:
1. Wrote a very simple HelloWorld class in Eclipse, with the following three methods:
a. String sayHello()
b. String sayHelloWithTime()
c. String sayHelloToMe(String me)
2. Exposed this simple HelloWorld class as a web service in Apache Axis 1.2. To make sure the web service is of type document/literal, I made the following settings in the wsdd file (apart from the usual ones): provider="java:RPC" style="document" use="literal"
3. Used to AdminClient tool to deploy the web service onto Axis.
And lo - my web service was deployed and accessible in no time.
Client side:
1. J2ME Web Services Optional Package does not have support for dynamic invocation of a web service (the way ksoap has). Therefore, it is necessary to generate static stubs (which is not that much of a problem really).
2. Sun Wireless Toolkit provides a Stub Generator utility that, given the location of the WSDL file, generates the J2ME client stubs.
3. Once the stub was generated, I wrote a simple MIDlet with only one form to use this stub class (that acts just like a normal local class) to invoke the web service.
Having done all this (of course I had achieved this after having spent some time on the Net), I was expecting things to work fine, although I soon realized that I had just opened a Pandora's box! I kept on hitting problems one after the other, and I kept on solving one at a time. Lately I have realized that I actually enjoy such small struggles, at the end of it they almost invariably give me some sense of an achievement, however small that might be..
OK - so lets see what all came out of the Pandora's box:
1. I kept on getting weird useless exceptions such as: Invalid element in server response, Invalid namespace, found xxx, expected yyy
2. While trying to find some logical explanation to what was happening, my fertile mind came up with the following few conclusions (which obviously proved wrong later):
a. That Apache Axis Server is generating a different response when the web service is invoked from a J2ME web service client (Yes - so you see, blame it on the JSR 172 implementation!)
b. The Apache Axis Server is not able to handle methods with the same signature (OK - so if nothing works, I can always pass dummy parameters to have all methods have different signatures, can't I?)
3. Something was definitely going wrong with the response, since I was not getting one that I expected (or rather the one that my stub was expecting), so I decided to use the oh-so-useful SOAPMonitor of Apache Axis.
Enabling the SOAPMonitor:
I enabled the SOAPMonitor by following the steps given here: http://www-scf.usc.edu/~csci571/2006Spring/axisinstall.html
My first conclusion (2 a) was soon proved wrong when I checked out the response I received from the server using both a J2ME and a normal J2SE web service client. In both cases, I was getting a wrong response. To give an example, the response I received was:
<soapenv:Body>
<meReturn xmlns="http://helloworldsample">Hello Arati</meReturn>
</soapenv:Body>
Where as according to the WSDL, I should have got the following: (Note the differences in the element name and the namespace)
<soapenv:Body>
<sayHelloToMeReturn xmlns="http://localhost:8081/axis/services/HelloWorldService">Hello Arati</sayHelloToMeReturn>
</soapenv:Body>
Some desperate attempts to get it working:
1. After cursing Axis 1.2 too much, I moved on to Axis 2. This however, turned out to be a wrong decision - did not really help, Axis 2 has been completed rearchitected, redesigned and I unnecessarily spent almost one day figuring out how to deploy a very simple web service on Axis 2 and then how to enable the SOAPMonitor on it.
2. I posted my queries, my doubts on the J2ME Forums and a few other forums (ex. Artima).
3. Having received no help from there, I moved on to the safe Axis 1.4 version. And then once again started fiddling around with the SOAPMonitor. One thing I realized (which I should have seen earlier) was that the SOAP request itself was not being formatted correctly. For example, on invoking the String sayHelloToMe(String me) method, the SOAP request was as follows:
<soapenv:Body>
<me xmlns="http://helloworldsample">Arati</me>
</soapenv:Body>
Ideally, with my basic knowledge of SOAP, XML et al, I knew that this is wrong. The response should definitely have contained the method name (sayHelloToMe) - how the hell is the Axis server otherwise going to understand which method to invoke?
4. This finally put me on the right track. I then doubted the document/literal support of Axis for the first time (that was a correct doubt) and on reading up a little realized the difference between the style "document" and style "wrapped". Check out this in detail in Axis's user guide at Service styles section
5. I made changes to the wsdd file to use style="wrapped" instead of document style, redeployed the web service, regenerated the stub classes using the stub generator and then checked out the MIDlet once again. This time I had hit the jackpot! The J2ME web service client worked beautifully, and the SOAPMonitor also started showing me expected results!
Well - so after a little fight and desperate searching on the Net for about 2 days (duh - I know I should have got this sooner than this) - I have cracked J2ME Web services + Axis combination, at least for the simple things..
Three cheers for the SOAPMonitor - Hip hip hurray, Hip hip hurray, Hip hip hurray!
Subscribe to:
Posts (Atom)
