Thrive or Survive - the changing rules for databases

Not since the late seventies, when Larry Ellison’s Relational Software Inc. (RSI) turned out the first commerically available RDBMS - Oracle, has there been such rapid changing of the rules (read disruption) in the database industry.
With Web 2.0 pushing enterprise adoption, and the ensuing information explosion in the maze of audio, video, data and ever-growing data warehouses, it seems that the conventional relational database systems are growing tired. With estimates of unstructured data being anywhere between 80% to 95% of all business data, and the ever changing requirements imposed by Web 2.0 - storage of pictures, audio and video, the demands being made on conventional RDBMS technology are monstrous. With the load window available being fixed due to availability and uptime requirements, the ever increasing data to be loaded into data warehouses, the bulking-up of the data due to usage of XML based formats, conflicting requirements of SQL and XQuery, the database is also being challenged by the demands of business intelligence.

And so the time has never been better for start-ups with innovative technologies. From the self-tuning column databases of Vertica to the data compression technologies of Infobright and the lock-free database from Ants Software , there is renewed interest in reinventing the RDBMS to optimize performance.

The competition has already hotted up in the high-end data warehousing segment with the introduction of appliances. With both Netezza and DATAllegro gaining traction rapidly, incumbent Teradata is feeling the heat.

Another entrant to the Web 2.0 database race is Database as a service (DBaaS). With Google opening up its cloud computing platform and making available Bigtable , and Microsoft offering a beta release of SQL Server Data Services (SSDS), the incumbent Amazon’s S3 and SimpleDB is getting some competition.

The incumbent conventional RDBMS vendors like Oracle are not resting on their laurels. With its aggresive acquisition strategy, Oracle has acquired Sleepycat’s open-source embedded database BerkeleyDB in early 2006 and now licenses it commercially. With its other acquisition (2005) - TimesTen, now integrated as an in-memory database, Oracle has been targeting SaaS ISVs to sell its Oracle SaaS platform. IBM has already moved into the BI infrastructure segment since 2005 with its pSeries Data Warehousing Balanced Configuration Units and evolved on to the Infosphere Balanced Warehouse applicances. Apart from SSDS, Microsoft has designed SQL Server 2008 with data warehouse features like the star join query, SSIS persistent lookups or the MERGE SQL statement. Most of the biggies (e.g. BOBJ acquiring Inxight) have acquired small companies with technologies to search and analyze unstructured data.

Yet with all this “new and improved”, the newer and innovative technologies are gaining a lot of traction at least in the data warehousing/ETL space. It remains to be seen if the trend catches on in the OLTP segment as well. But for now, the staid and bland database segment is on fire.