Time to get this blog back into activity. The past few years have been so busy with projects the size of 10s of PB, that I did not get a chance to do any public blogging. We started this site when #BigData was not a term but an over simplification. How times have changed. We are working on brand new massive scale projects and are turning more and more into real time analytical processing. Time to writeup what we have been up to. Check back over the next weeks as I will update the site with new stories and insights we found along the

Today I ran across some discussion about Agile Development in Data Warehousing, and note that we talk about this in the context of the DW development, but not in relation to the Business. I believe there is a need to discriminate some of these processes quite differently. Most simply put - One is applying Agile to DW development; the other is applying Agile to Business Analysis.

Core DW foundations involve modeling root components of business data needs and implementing a data model which allows for flexibility to answer questions of the data - a concept I call "Designing for the Unknown". The more renormalization and change from the source system, typically the more transformation logic and less flexibility, and ergo higher cost and less organizational agility.

Effective Agile development of the DW infrastructure itself involves delineating the methodologies which can be used for what types of development functions. For example, creating a core or "root key" entity in the data model

Over the last year I have spent a good amount of time thinking about the cost of analytics, and a few things worry me about our industry and how vendors price in this industry.  In the Data Warehousing and BI industry, we're starting to see pricing models based on data volume or size. I know of one vendor which prices by specint, so - get a bigger system or virtualize your systems - get a big bill.

 The problem with these licensing schemes is that they actually make the cost of doing analytics more expensive over time.  If I am a good technology user, over time I'm driving down the costs per unit of data volume to create and maintain the data flowing into my DW.  In the simple case you could say that the cost to generate data is declining at the inverse rate of Moore's law, particularly for a web business where most of the cost is in hardware.

Back to the point, using Moores Law as the example, the cost of data generation declines per unit of volume at a rate of ~50% every 18 months. 

Finally I got to complete a high level systems overview. I realize it does not contain too much detail, but as you can imagine, we are bound by pretty strict NDAs.

Nevertheless, it should give you a good feel for how much data we process any given day. The stats are pretty much going into 2008 figures and are growing rapidly.

Here is a link to the article: Our Systems

Enjoy the reading and post your comments!

Oliver

Welcome to the xlmpp blog!

 

Its time to kick off an exciting blog about ultra large scale information processing architecture and real world analytical systems.

Over the next weeks and months Michael, Darren and myself will be blogging about the largest data processing systems, common issues, scalability of large scale mpp clusters, time to market, analytics, ultra high data volumes and lots more.

Be sure you bookmark the site and subscribe to our news feed.