LWN: Comments on "Data Warehousing 101"

Refreshing overview

lab — Sat, 06 Aug 2011 09:36:40 +0000

Can only second that! Very informative article. Thanks Josh.

Data Warehousing 101

Lennie — Thu, 04 Aug 2011 00:12:17 +0000

Have you considered a "noSQL" solution like ?: http://couchdb.apache.org/docs/intro.html http://couchdb.apache.org/docs/overview.html

Refreshing overview

mathstuf — Sat, 30 Jul 2011 21:46:04 +0000

Better: Make buttons for +1 and -1. Then have the ability to get git --diffstat-like output for a thread.

Refreshing overview

oak — Sat, 30 Jul 2011 10:06:19 +0000

Maybe the LWN commenting system should add "+1" comments as "+" characters to the header or footer of the previous post instead?

Queuing platforms for ETL?

gramosalvarez — Thu, 28 Jul 2011 11:06:01 +0000

Great article, indeed.
Still, I don't see where ActiveMQ or RabbitMQ stand as alternatives for ETL tools.
Could you elaborate on this, please?
Thanks!

Data Warehousing 101

jberkus — Thu, 28 Jul 2011 01:37:14 +0000

KVAML,

Huh. See, every time somebody thinks they've invented new DB technology ...

;-)

KDB

Jaquatech — Tue, 26 Jul 2011 16:29:58 +0000

http://kx.com/products.php sounds like it
-Jan

Data Warehousing 101

kvaml — Sun, 24 Jul 2011 22:06:26 +0000

"Column-store databases (were) Invented in 1999"

Actually the old Dartmouth Time Sharing System (DTSS) had a column store database known as project FIND in the 1970's.

Data Warehousing 101

gerv — Sat, 23 Jul 2011 05:50:12 +0000

Just the sort of thing I subscribe to LWN for. Great :-)

Gerv

Refreshing overview

ballombe — Fri, 22 Jul 2011 22:34:31 +0000

Well, some people take that as a hint that +1 are not informative enough to warrant a comment.

KDB

jberkus — Fri, 22 Jul 2011 17:12:55 +0000

Macson,

Link? That's not one I've heard of before.

KDB

macson_g — Fri, 22 Jul 2011 09:35:09 +0000

Interesting article!

I'd like to add KDB to to the list; its ubiquitous in the finance industry.

KDB is a C-store database with embedded Q interpreter. Q is a powerful and flexible programming language and having it running at the core of the DB allows for complex statistical analysis to be executed fast even on very large datasets (think terabytes of data and billions of records).

Data Warehousing 101

jeremiah — Thu, 21 Jul 2011 20:28:39 +0000

Except for the languages listed, that's what I've got. I found Neo4J difficult because of the lack of flexibility in defining relationships, ie they couldn't be dynamic. At times I even toy with the idea of doing something on my own at the file system level, but it's just not worth the effort, esp trying to redo a lot of the features that the Postgres folks have worked so hard over the years to develop. And I'm sure I could never approach their level of scalability on my own. Thanks for the input.

Data Warehousing 101

jberkus — Thu, 21 Jul 2011 19:41:09 +0000

Jeremiah,

If only there were! There's a few different implementations of graph databases, of which Neo4J is probably the leader, but every one I've seen is hard to use and has some substantial scalability limitations.

I think Postgres & WITH RECURSIVE, plus pl/perl or pl/python stored procedures for the decomposition may be your current best bet.

Refreshing overview

louie — Thu, 21 Jul 2011 19:27:05 +0000

+1.

(Esteemed Editor, your comment system's "less than ten characters" rule should have an exception for +1s ;)

Data Warehousing 101

jeremiah — Thu, 21 Jul 2011 17:37:06 +0000

Nice article. Now if I could find some kind of DB that's good at storing deconstructed XML or JSON with version information I'd be a happy camper. I'm always drawn to map-reduce, but always find myself using going back to RDBMS w/ recursive queries.

I've got a situation where every attribute and element is version-ed and stored separately. Everything is treated as a node, and the relationships between nodes are defined by the query. Needless to say, it's a little trippy. Or to say it another way, we can store to different directed graphs, and define the structure of a third, and have the DB/code fill in the values of the 3rd, using the first two.

Anyone know of some kind of db architecture that would be inherently good at this?

Refreshing overview

sumanah — Thu, 21 Jul 2011 15:42:40 +0000

I feel way more ready to think about data warehousing challenges (and opportunities) now! Thanks, Josh & LWN.