Data Warehousing 101

Posted Jul 21, 2011 17:37 UTC (Thu) by jeremiah (subscriber, #1221)
Parent article: Data Warehousing 101

Nice article. Now if I could find some kind of DB that's good at storing deconstructed XML or JSON with version information I'd be a happy camper. I'm always drawn to map-reduce, but always find myself using going back to RDBMS w/ recursive queries.

I've got a situation where every attribute and element is version-ed and stored separately. Everything is treated as a node, and the relationships between nodes are defined by the query. Needless to say, it's a little trippy. Or to say it another way, we can store to different directed graphs, and define the structure of a third, and have the DB/code fill in the values of the 3rd, using the first two.

Anyone know of some kind of db architecture that would be inherently good at this?

Data Warehousing 101

Posted Jul 21, 2011 19:41 UTC (Thu) by jberkus (guest, #55561) [Link] (1 responses)

Jeremiah,

If only there were! There's a few different implementations of graph databases, of which Neo4J is probably the leader, but every one I've seen is hard to use and has some substantial scalability limitations.

I think Postgres & WITH RECURSIVE, plus pl/perl or pl/python stored procedures for the decomposition may be your current best bet.

Data Warehousing 101

Posted Jul 21, 2011 20:28 UTC (Thu) by jeremiah (subscriber, #1221) [Link]

Except for the languages listed, that's what I've got. I found Neo4J difficult because of the lack of flexibility in defining relationships, ie they couldn't be dynamic. At times I even toy with the idea of doing something on my own at the file system level, but it's just not worth the effort, esp trying to redo a lot of the features that the Postgres folks have worked so hard over the years to develop. And I'm sure I could never approach their level of scalability on my own. Thanks for the input.

Data Warehousing 101

Posted Aug 4, 2011 0:12 UTC (Thu) by Lennie (subscriber, #49641) [Link]

Have you considered a "noSQL" solution like ?: http://couchdb.apache.org/docs/intro.html http://couchdb.apache.org/docs/overview.html