Everyone that has done server side programming on Java knows about the redeploy hell and all of that. What AS to choose? Some front end perhaps? Do they work together and how can we minimize downtime?
I have gone through the same questions and I did this one task which idea was to study available Java AS and choose which one we should use, it was a study task. One of the motivation was that the old system that we will one day hopefully replace is running on JBoss 6.0 and it was a pain to test and a real pain in the ass to redeploy. We did find out that in production environment it didn’t take so long to restart it, but in test environment and on developers machine it was a real pain. The conclusion of this task was that the AS doesn’t really matter and we must develop our web application without any dependencies to the AS. The old application is coupled with JBoss so the main developer said that updating JBoss is very hard and changing the AS is also out of the question. For what I have seen so far it might be that decoupling from JBoss would not be that big of a task if the only dependency to JBoss are the JBoss specific EJB annotations that are used to setup the Java EE container. The new project that we are working on is based on Spring and the war contains everything we need. It runs on Tomcat, but is not coupled with it. Redeployment is not that difficult and currently we are using Jenkins CI to automatically build it and deploy it to our smoke test Tomcat. The redeployment did cause issues and due to time constraints and quite a lot of depending on the knowledge of one developer at the time, thank you very much, the current solution calls a script after the build process which shuts down the Tomcat, cleans the webapps folder, copies the newly build war into the webapps and starts up the Tomcat. Now this isn’t very intuitive, but we can manage with it. Development of the new application is also much much faster. It is a maven project and we develop it with Jetty so it doesn’t take too long to start up the web application, at least compared to the other project. I have thought about JRebel because it would make things even faster if we can use it. IT, computers and all that can really be easy, working and fast if we do it right, but most of the times it is not done right. You can do things the correct way, or the wrong way or completely f***ed up way that makes no sense and complicates things and unfortunately thats how things are usually done.
One day I thought that why do we need the AS? We are developing Spring application and the war package has everything we need, Tomcat really doesn’t provide anything extra. I already knew at this point that Jetty could be embedded and we could make a standalone application which would start the web server and everything. Using the embedded Jetty we could eliminate the AS. One thing that this solution would give us is the easier redeployment. We simply kill the process and restart the new Java application, easy! No need to redeploy the war in the AS and hope that it works. Tomcat for example has memory leak issues and most web applications suffer from these issue because people do not probably know about them. These issue for example caused that the third time our web application was redeployed to Tomcat through the manager Tomcat would hang completely.
My colleague found this Java framework/library called Dropwizard that would be one option to simplify our web application. There are same tutorials about using Dropwizard and Spring together that points out that they kind of step on each other toes. We are most likely not going to use it for now, but what I have read about it is very useful for creating web services for example. So if and when our web application becomes distributed Dropwizard probably will be very useful in developing services. Think of the ease that you just start an application giving the port is listens to as parameter and in few seconds you have a fully running web services listening at the defined port. Think load balancers, think message consumers, think ten different web servers on a single computer servicing clients with just starting with different port parameter instead of setting up application server and deploying to that.
Spring Boot. I just came across this and it seemed to have everything I was planning. And it is a Spring project so no extra hassle to get the Spring application context up and running. I think this is the way we will go. I am Spring enthusiastic so I am very excited about this. I hope this will work as I have thought that it will work.
Now one problem solved, no need for application servers and war package to deploy into them. Maybe this will be a new trend that will reduce the number of AS with deployed war packages, time will tell. I had this idea that if we can control the start up process we and do it so that if the port is already in use the application will sleep for few second and try again. And it will keep doing this until the port become available. Now thing about it like this. You have the old version running and you need to update it. You start the new version which will not start completely because the older version still has the port reserved. Now you shutdown this old version of the application and the newer version will reserve the port and starts serving clients. At least theoretically the redeployment should be quite fast. Start the new application, shutdown the old. Probably not very suitable in production environment as such, but in test environment could be very useful.
How hard it is to create SSH server in your application? With Java not very hard. Your web application could also start SSH server that could be used to log into your web application and do all kinds of stuff. Tell it to gracefully shutdown for example. Now if you were to do this you would of course do the mechanism to trigger the graceful shutdown and have access to it through some service or such. Now your command in the SSH terminal would of course just call this service to initiate the shutdown process. If you think about it, it really doesn’t matter who initiates the shutdown process meaning who calls the service.
Sticky HTTP sessions makes it possible to keep session data in single server and no need store the session data and reload it on every request. This works as long as the server is running and does not crash or shutdown for some other reason. The proxy keeps track which session is directed to which backend server. Personally I think this feature eases the application development because there is no need to serialize the session data if you do not prepare for the server crash.
Now for a moment imagine that a proxy with load balancing and everything something like HAProxy would have some messaging capabilities. Now each server instance connects to the proxy and informs about their existence. The proxy can dynamically start directing requests to new servers. What about if the server could notify through a message that it will go down as soon as possible. The proxy takes that server instance out of the list of servers that requests can be directed so the instance will not serve new sessions. The proxy would however still server the sticky sessions to this server that were active when the server notified about its shutdown. When the last sticky session timeouts the proxy would notify the server that all sticky sessions are now expired and no more requests are directed to the server, now the server could shut itself down.
Now the servers can communicate with the proxy through some messaging protocol. The server can also communicate with anything else through that channel. Some application could send message that would request the server to shutdown. This would start the chain described earlier.
Hmm.. basically existing technologies (except for the proxy I haven’t yet found one that would have this capability and be open source) that does various things. How about I take all of this, combine them and give them some buzz word name? Like Cloud Computing.. wait thats already taken and it has done in this way, combining existing technologies and giving the combination a buzz word.
Maybe I would have some service or CI system plugin that copies the new build somewhere and starts it. It would wait for the port to become free because it is still reserved. Now I the service could send message to the old server requesting it to shutdown. Now the old server send message to the proxy that it will want to shutdown. The proxy sends the old server a message that sticky sessions are served and then the old server could shutdown. Now the new server gets the port and is up and running. It will send the proxy a message that it is ready to receive requests. What now then? Well think that you have two instances of the server would have zero downtime on the request serving because the server update would be done automatically and the restart would only happen after sticky HTTP sessions are served. If there are no sticky sessions that server can just shutdown. You could have ten instances of the server and they could all be updated without brake in the service. Sticky sessions would be servers out using the old version of the server and new requests would be directed to the new versions of the server. You could even nicely coordinate the process with configurable application that would keep track that all servers are updated. If the whole process would be triggered when new successful build finishes then basically updating the test, staging and production environments could be done automatically without any interaction from the admin or such. A vision that could be worth gaining at least in my opinion.
Now there are lot of pitfalls and problems and issues that I know that are not discussed here. This is about the theoretical idea that would create a automatically updating server environment with zero downtime. I am not a web expert and I know that there are also lot of things that I have not thought out when coming up with this idea. Perhaps you could point out issues that might turn up when trying to develop this kind of system. Or maybe this kind of system has been in development, but there have been some problems that could not be solved. I would really like to know because this straight forward explanation how the system would work is really a straight forward explanation. To me it sounds great and achievable, but I am not an expert on the subject so enlighten me.
It is also possible that this kind of system exists, but I haven’t found it. I say that Google is f***ing with me, but I know at the same time that real problem is that I am using wrong words while searching. So it’s not really Google f****ing with me, but lack of my intelligence while trying to come up with the correct search term.