Evolving persistence, SQL and ORMs to the next generation

Recently there has been a lot of discussion on the web describing the ORMs as a thing of the past, and its associated discussions that agreed and disagreed with their arguments presented to various degrees. The discussions even took a long conversations across the twitter space too.

Persistence models and application evolution

Persistence is important for any application. Over the years this has been done in various ways. While RDBMS and SQL were the most obvious choice till few years from now, recent times have also seen successful practical usage of other persistence models which include Amazon's SimpleDB, and Google's BigTable among others. These non-relational alternatives fall under the NoSQL (or “post-relational”) persistence models.

Naturally it becomes quite obvious that over the time, relational databases have proven inadequate to satisfy the growing needs of the Web2.0 applications. The advent of the new-generation non-relational databases are a proof to this. Every persistence model has a niche area to which they cater. The newer databases have has also made people to think about "mix-n-match" use of the various persistence models as people talk more and more about polyglot persistence.

The relation-object impedance mismatch

Since RDBMS has a solid theoretical foundation, and has been used across various applications, its existence for another century cannot be ruled out. Yet new advances in persistence like CouchDB, BigTable, SimpleDB, Graph databases, etc are most welcome steps in advancing the persistence mechanism.

Even with newer persistence models, a lot of the enterprise applications still use, and will continue to use relational databases. Applications that use up this data are largely represented as objects. While object model seem to be the most obvious choice for application development, the mismatch between the relational model and the object model comes as the reason for a lot of hues and cries that have given rise to ORMs. The learning curve and the pain in using these ORMs have now given rise to debates questioning the existence of ORMs in the future.

More than debating over the existence of ORMs itself, rather a deep look is needed, to the problem which the ORM is trying to solve. Why does the ORM exist? The most popular answer would be impedance mismatch. But the mismatch is pointed out between relational data representation versus the object model.

Overhauling the persistence mechanisms

Often the data storage mechanism and its interfacing API are seen as an integral part, including the RDBMS data storage and SQL. They are often seen as one, but are actually two separate components of the modern day relational databases. They were created at a time when the existence and usage of OO was questionable. Even now when they are used for a long time, it can be argued that SQL and RDBMS themselves have a mismatch in the paradigms. Now when OO has gained so much importance, it is time that the RDBMS and SQL should be re-looked from the OO perspective.

We can consider two aspects in any persistence mechanism:

1. The way data is stored (relational, key-value, hierarchical , column, JSON etc) using various data structures and storage mechanisms. Closely associated to this data storage will be the related methods of storing and fetching the stored data (algorithms used to fetch data based on persistent store itself - eg relational algebra) that could be probably exposed through function calls. The function calls will be closely associated to the data store mechanism itself.
2. Interfacing provided to interact with persistent store (SQL, NoSQL, ObjectQL, programming language API etc).

Often these two aspects are baked together (in our minds most of the time). This thinking should evolve and these two should essentially be seen as separate, and ideally pluggable. We talk about adapter pattern in programming languages, but the same concept could probably be applied to persistence mechanisms as well. The persistent storage and its storing and fetching mechanism should allow widely different interfacing provided to interact with it. DB vendors can provide few interfacing mechanism, and also allow others to write other interfacing mechanism that should be pluggable.

Interfacing will expose the persistent store in various forms - relational model, or object model or even other models used now and may come in future. Mapping of the datastore persistence model to the exposed model should reside with the interfacing mechanism used.

Concluding thoughts

Achieving this practically may not be very easy, but not impossible. Once this is achieved, ORMs will rather shed off the mapping weight and evolve as frameworks that enhance developer productivity on issues like caching, application scaling etc. Rather, ORMs will no longer be ORMs and probably a new breed of softwares will evolve that will take care of the cross-cutting concerns for an application when dealing with persistent stores.

This may look crap to some, but then that is what I wish. I am an application developer presenting my ideas on what a database itself can provide, and how applications can leverage the new capabilities. The issue needs to be looked at by db vendors and research scholars. DB vendors and db developers need to give deeper thoughts to the problems faced by application developers in their need to persist application data. How many of us use the bare databases (using only SQLs probably) without an application? The current forms of persistence may be right in their own theoretical framework, but it should evolve to fit in seamlessly with the application development space, where it is used.

And till that happens, I will have to carefully weigh my choices among NoSQL, SQL and ORMs, and muddle with the problems that have already heated up a lot of discussions in the application development space.

My state of affairs - part 2 (looking for new projects)

In my previous blog post, I mentioned about the my state of affairs, of how things have been moving in last 2-3 years, and why I was so irregular at posting blogs. The state of affairs was actually not finished yet. I continue it here.

Looking out for new job/consulting opportunities

My current employer is finding it difficult to sustain its employees. While some employees quit at the start of the turbulent times, some of us including me decided to stay strong with the company. And now more than one year after showing signs of trouble, the company is not yet able to fully recover and get back to normal business. The management is so kind to ask us to look out for other jobs as an open offer to us. We can take our own time to move out. Although the management has plans to bring it up again back to business, it doesn't want its employees to be dragged across the difficult times any longer. As our work is tapering off, I will be looking out to see if some good offers can come along.

So if you are looking out for an experienced developer or consultant, with proven track record in providing effective business solutions with good exposure to various technologies, try me - I will be happy to help you out. I specialize in Java/EE, Spring, Maven and OSGi (thats about building truly modular Java applications). And if you want few more experience hands, may be my company can help you in your IT endeavors, specializing in creating robust modular and scalable Java applications.

In the meanwhile I will be working on some OSS projects and writing blogs. So, keep an eye on this blog. My blog posts will give a good understanding of various technologies and how the technology can be best put to use in solving real-life business problems.

I am also working out on a project (of my own) that is a good mix of some technologies I have already learned, or have scratched the surface. It will provide the ground for me to build depth of my technical knowledge as well as serve as a ground to build up applications for my client that require ground-up work. Some time down the line, I will also give out this code to the OSS space.

If you would like to know more about me, follow the links below:

• LinkedIn profile
• Resume (pdf)
• Twitter profile
• Open Source contributions

My state of affairs - part 1

Now it has been a while that I have blogged, and I understand that not many people are there who follow my blog. Though some of the topics on which I have blogged in the past have been getting some good hits, I cannot maintain a regular list of people who will visit my blog often, probably because I am not regular at blogging. But thanks to those 3 feed-readers who have been with me, and probably would like to know, what keeps me busy that prevents me to write blogs, and now why is it that I am giving out an explanation. Also, probably in every other serious post that I write, I almost mention that the next post will be coming very soon, and still that does not happen.

Last 3 years

While it is a fact that I have been dealing with so many topics in the past, especially in the last 3 years or so, and every time I feel like posting about it on the blog, and yet that does not happen. all this time, I have been dealing with a vast breadth of topics in the Java space - Rules Engine/Drools, OSGi (Pax, Felix, Equinox, Karaf), Maven, ESB (ServiceMix), Message Routers (Camel), JPA/Hibernate, Spring, Webworks/Struts2, Spring Webflow, JSF, Facelets, RichFaces, REST, SOA, GAE (Google App Engine), GWT, Spring Security and its various tools, implementations and their integrations on how these topics can play with each other. It was during this time that I got more into the OSS space as well. Initially moved by the JBoss professional open sourcing and then more impressed and involved with the OPS4J philosophy. The single factor that made me meet OPS4J was its PAX line of products that dealt with OSGi. And as I started learning OSGi, I was happy to take out some of my personal time to make few commits into the same products in PAX - that made me smile better as I struggled with my initial steps with OSGi.

The journey begins

This journey begins as I complete almost one year with my previous employer, when the client and the employer impressed with my performance handed over the implementation of intelligence in an insurance quoting system to me. Thanks to them, it was a great experience. I implemented the system using Drools and I was able to deliver the system that could be taken out to production systems within 7 months from the date I was assigned the task, with hardly any help from anybody else. Whats more, the system was almost zero on bugs as of my knowledge. Here I started my journey of exploring technologies beyond the regular Java/J2EE developer. Though in the past I had done it on my personal itch, this was a better experience, since could see it working on a real system. Before this, I had experience with the regular Java/J2EE experience like the majority developers with just Servlets, JSP, Struts, SQL, EJB etc etc.

Last 2 years

Now it is almost 2 years that I've been working with my current employer, and most of this time I was into R&D primarily with OSGi and its related aspects. While I personally like to dig past bleeding edge technologies, and playing with it, a similar work profile as my job profile was something that was very exciting. I was always excited about how technology could be used constructively to provide least effort and long term solutions to real-life business problems. I was the prime responsible for bringing up a framework for the company that could be used by its products/services to cater its clients in the coming future. The power and dynamism of OSGi was a major push to this. During this time, I had the opportunity to deal with a variety of technologies to make up the framework as mentioned above. I did made serious advances in setting up the technological stack for my company based on OSGi. But I still feel very sorry that I could do very less on contributing this knowledge to the OSS, as I see people trying to struggle with OSGi and web .

But why should all that prevent from posting blogs?

All this made me so much involved in understanding the technologies that I considered it worthwhile to spend that 2-5 hours to understanding the other technology for integration to the stack, than to write a blog on what I had just learned. I even spent my personal time for understanding aspects of technology that was outside the scope in my official research. Also, its not that I did not write blogs. While a very few appeared here on my public blog, most of them went on to my corporate blog that is not a public one. Also, all this time, my understanding about any technology is something, that I think would be better to blog about as I learn more. But over the time, that never happened as I piled up more and more technologies.

So, now whats there for the future?

Guessed, it right! I am again going to say that I will try to be more regular at blogging. Blogging is a tool to showcase yourself, express your ideas and to market yourself. Now this time, there are reasons too why I should blog, about which I will be explaining in my immediately next blog (mmmh..., you will not have to wait, its ready as I write this. This blog was bloating too much, so I just decided to split it into two).