LWN: Comments on "Four years of Go"

Fortran no more

mathstuf — Wed, 11 Dec 2013 23:01:44 +0000

Maybe it's a mis-translation (I don't know dakas' mother tongue), but I call my apartment (which is a floor of a house, but much closer to an apartment) "home" (I also call my childhood house "home" depending on the context) which can easily transform into "house" in speech.

Fortran no more

nix — Wed, 11 Dec 2013 22:01:05 +0000

Then it would be an apartment, surely, not a "house"?

Apartments with elevator shafts I expect. Houses, not so much, unless you're a supervillain (in which case I hope there's a piranha pit or glowing radioactive rubble at the bottom of it).

Fortran no more

jwakely — Mon, 09 Dec 2013 18:46:10 +0000

Oops, yes thanks!

Fortran no more

mathstuf — Mon, 09 Dec 2013 13:39:56 +0000

An apartment complex maybe?

Fortran no more

mpr22 — Mon, 09 Dec 2013 02:42:40 +0000

std::string has no inherent encoding; the data it contains really is just a bunch of bytes not containing NUL. (And because people chickened out over the constraints for wchar_t, std::wstring might also be such a bunch of bytes and cannot be relied on to store Unicode.)

QString, on the other hand, is a UTF-16 (ugh!) Unicode string.

Fortran no more

nix — Mon, 09 Dec 2013 01:14:19 +0000

I think you mean 'implicit conversions are bad' :)

Fortran no more

nix — Mon, 09 Dec 2013 01:13:08 +0000

If you need to contain a NULL to handle Unicode, your representation is wrong (i.e. not UTF-8).

Fortran no more

nix — Mon, 09 Dec 2013 01:12:05 +0000

That's what malloc() is for (assuming we're still talking C). Of course in that case you have to pass around sizes as well, and if you were using them a lot I'd be thinking about doing a nice opaque ADT (yes, C can do ADTs perfectly well).

Fortran no more

nix — Mon, 09 Dec 2013 01:07:26 +0000

In fact, for a while she owned the main server in the house (the weaker machines being used as X terminals via RG-58 Ethernet through an unused elevator shaft).

Your house had an unused elevator shaft?! Where were you living, an abandoned power station?

Four years of Go

nix — Sun, 08 Dec 2013 21:14:07 +0000

Well, one advantage of the relational approach is that it lets systems evolve. The systems I used at $oldwork were a typical large system: i.e. the relational models were all in people's heads and there was a lot to absorb by osmosis before you got started. But if you wanted to pile a new sort of query in there, it was a matter of define the appropriate indexes and do it. Sure, we tried to avoid adding new indexes on particularly huge or highly-updated tables, and the client would have to wait for a while while the indexer chewed through TiB of data, but we *could* add new cross-system joins as the system evolved, and routinely did, all the time, without having to do any sort of major rethink of our data model.

Now the downside here of course is that we never had to do a major rethink of our data model, so it got more and more crufty over the years. But the upside is that this system evolved into something wildly larger than the original *without* a rewrite or nontrivial data model change in more than twenty years.

I kinda doubt Pick (or any graph-traversal database) can do that. They all seem to be about encoding what your application wants out of the system into the depths of the data model -- which is fine, until that changes, which it always does, and then you're screwed.

(Disclaimer: yes, I do work for Oracle -- but this stuff, and these opinions, all predate my Oracle employment. In fact, since coming to work for Oracle I've had nothing at all to do with databases, after spending my entire working life before that SQL-bashing. Strange.)

Four years of Go

nix — Sun, 08 Dec 2013 21:04:19 +0000

Pick guarantees a direct hit pretty much every time, if you know the primary key, giving an O(1) performance.

You still have to do a hash lookup, I think, so O(log n).

Fortran no more

dakas — Thu, 28 Nov 2013 12:10:58 +0000

Uh, you can of course put a QtMatrix, as a black box into standard data structures. The claim was that two matrix-using applications can't talk to each other regarding the contents of their matrices. Sure, you can store one's library complete matrix objects into matrices of the other library. But that's just storage. The other library can't do anything meaningful with the stored objects.

It's fascinating how a wagonload of posters here cannot even recognize the basic problem.

In a nutshell: one library will not multiply matrices (or do more complex numeric work) stored in another library's matrix.

Fortran no more

dakas — Thu, 28 Nov 2013 12:03:56 +0000

Still not run-time sized.

Amazingly productive programmers

thoeme — Sun, 24 Nov 2013 09:02:35 +0000

>I've discovered that some amazingly talented programmers are orders of >magnitude better and more productive than the average programmer
Yes, (luckily) I discovered that fact myself, me being a *not* talented programmer in comparision to my class mates when I was in university. Just in time to switch the main tpic ;-)

Four years of Go

raven667 — Sun, 24 Nov 2013 02:10:34 +0000

Edit: that should be Bruce Banner, my brain isn't firing on all cylinders either

Four years of Go

raven667 — Sun, 24 Nov 2013 02:02:07 +0000

That's like David Banner saying "don't make me angry". 8-)

Four years of Go

wtanksleyjr — Sat, 23 Nov 2013 20:05:01 +0000

+1 for "ah-doc queries".

Fortran no more

hummassa — Sat, 23 Nov 2013 09:23:58 +0000

iirc, *a pointed to a[0][0]. Which confused me immensely. Which led me to suspect that the MS C compiler was doing something funny in the code in the subroutine in which a was declared.

Yes, a is an

int (*a)[3];

So, *a is an array of three integers, and in an expression, it is again demoted to "a pointer to the first of three integers".

Maybe


#include <stdio.h>

int a[2][3] = { { 1, 2, 3 }, { 4, 5, 6 } };
int b[3][3] = { { 7, 8, 9 }, { 10, 11, 12 }, { 13, 14, 15 } };

void f(int *x) {
  printf("%p %d\n", x, *x);
}

void g(int (*x)[3]) {
  printf("%p %d\n", x, **x);
  printf("%p %d\n", x, (*x)[1]);
  printf("%p %d\n", x, x[1][1]);
}

void y(int x[2][3]) {
  printf("%p %d\n", x, **x);
  printf("%p %d\n", x, (*x)[1]);
  printf("%p %d\n", x, x[1][1]);
}

void t(int **x) {
  printf("%p %d\n", x, **x);
}

int main(int argc, char** argv) {
  //f(a);
  g(a);
  y(a);
  //t(a);
  f(a[1]);
  //f(b);
  g(b);
  y(b);
  //t(b);
  f(b[1]);
  return 0;
}

with its output:


0x600b20 1
0x600b20 2
0x600b20 5
0x600b20 1
0x600b20 2
0x600b20 5
0x600b2c 4
0x600b40 7
0x600b40 8
0x600b40 11
0x600b40 7
0x600b40 8
0x600b40 11
0x600b4c 10

explains better what I am saying (commented out lines are syntax errors). As I told you, you were seeing the "side of weird" that passing matrices as parameters in C presents... because the type of an array of array parameter is pointer to array, but the type of star-pointer-to-array is an array, that itself in an expression is demoted to... pointer-to-the-first-element of the array!

Fortran no more

Wol — Sat, 23 Nov 2013 08:42:17 +0000

That's not what I found ...

iirc, *a pointed to a[0][0]. Which confused me immensely. Which led me to suspect that the MS C compiler was doing something funny in the code in the subroutine in which a was declared.

Which, when I read the language spec, I couldn't find anything saying that wasn't okay!

> Yes. I don't even think multidimensional arrays are an afterthought, I think they are kind of an accident. "This kind of makes sense, let's not touch it" stuff.

Which leads me to suspect it's implementation-dependent, not spec'd, and not portable.

Anyways, it looks like we used different compilers, got different results, and (knowing MS's attitude to following the spec) if either of us is right it's probably you :-)

I know I actually ended up writing a whole bunch of code that manually built an array of "pointers to pointers" in order to try and get things to work as I thought they should.

Cheers,
Wol

Four years of Go

k8to — Sat, 23 Nov 2013 07:11:09 +0000

Not necessarily.

My job is working on an entirely ad-hoc focused engine (primarily keyword search) which could be fairly described as nosql store. It doesn't offer the raw entries scanned speed of the fixed-nature queries, so we have a SECOND datastore for that. They're both nosql, though in terms of attitude, representing relatively simple structure enormous tables.

But I'll give you that the common goal of most noSQL developers is building somsething for the relatively fixed qeries at enormous scale.

Four years of Go

k8to — Sat, 23 Nov 2013 07:05:25 +0000

I've dabbled in the bindings for both Tcl and Python back in the 90s. That's a far cry from providing a built-in language that everyone can rely on.

Fortran no more

hummassa — Sat, 23 Nov 2013 00:05:42 +0000

> Have you actually tried to DO that?

As would Yoda say, not tried -- did that. :D

> Okay, I was using MS C v5, but when I started playing around with defining things like

> int a[3][2];

> sub(a);

This usually botches because most people think: "oh, arrays are passed as pointers, so "sub(a)" is a "sub(int**a)". NO. Arrays of integer are passed as pointers to integers, but arrays of arrays of integers are passed as pointers to arrays of integers. This is so that the body of "sub" can calculate AT COMPILE TIME how long is ONE ELEMENT (one array of two integers = two "natural words") of the passed-by-pointer array. Sorry about the caps, they are just for emphasis, not shouting.

Yes. I don't even think multidimensional arrays are an afterthought, I think they are kind of an accident. "This kind of makes sense, let's not touch it" stuff.

That's why usually, in C++, we wrap a class around any matrix thing. That way, you can pass them by value or reference, and you lose the operator[](int x) in favour of the operator()(int row, int col)...

Four years of Go

Wol — Fri, 22 Nov 2013 23:16:15 +0000

> In any event there is no point in arguing about peoples personal preferences, Wol likes Pick and how it handles the kind of data he works with, all the more power to him.

In my original post that set all this off, I did say "don't get me started", didn't I? Or words to that effect ...

Anyways, it's GMT23:15 here, time I was in bed...

Cheers,
Wol

Four years of Go

Wol — Fri, 22 Nov 2013 23:12:43 +0000

>> To which I have to add, joins are part of the program layer above the data store in a relational database. What's to stop some enterprising vendor from adding the SAME layer to a Pick database - oh but they have - it's called SQL.
>There's a joke in Russian: "If a grandmother had balls then she would be a grandfather". It applies here.

But seriously. The SQL query engine sits above the datastore in a relational database. It's been added to Pick databases to sit at exactly the same level. So we match relational databases in flexibility. I can write an ad-hoc SQL query just the same as you.

Problem is, if you try and change your relational datastore to match Pick, you have to break all the rules that make your database a relational one. So you can't match me for speed. My worst-case scenario for the speed at which my query runs is pretty much your typical expected scenario. Catch me if you can ... :-) and if the query makes real-world sense you'll be left eating my smoke!

Cheers,
Wol

Four years of Go

Wol — Fri, 22 Nov 2013 23:01:33 +0000

> Nope. Relational algebra doesn't care about "attributes" or links between them. It operates on tuples produced by the Cartesian multiplication of relations (tables). There are NO limits on the ways to combine data and there's no notion of 'linked attributes' anywhere in the relational algebra.

And this has what relationship with the real world? You are talking maths - "welcome to the dark side, Anakin" :-) Engineers use maths to *describe* the real world - you seem to be a Computer "Scientist" (ie a mathematician) who is interested in pure maths with no interest in the real world as it really is...

> You are totally free to create a join between Martian temperatures table and a table with the poets of the Victorian age. And then filter it by matching the age of poets at the time of their death with the temperature on Mars. I have no idea why you would do THAT, but it's possible.

Perfect! You have just hit on the reason why relational databases run like treacle on tranquilizers! Relational theory says, iirc, that all queries are equal. And precisely because a stupid query like that is possible AND EXPENSIVE, the equality theorem says that all normal regular queries must be EQUALLY as expensive!!!

The problem here is that, because relational is MATHS, and stupid queries make just as much mathematical sense as reasonable queries, the maths ignores the real world cost of actually carrying out the query! Which is why query optimisers are so important in the relational world (and why they don't make sense in the Pick world - any Pick query that makes real world sense can easily be proven to be "almost optimal").

>> I'll just end with a challenge I throw out regularly - how do you STORE a LIST in a relational database (and I won't accept an answer that MODELs a list!)
>Answer: "you don't".

You do realise you have just stated categorically that a relational database IS USELESS AS A DATABASE? How much data in the real world comes as lists? 90%? And you can't STORE that in a relational database?

There are two big problems here, imho. The first is that marketing dollars have driven most competitors to relational into obscurity. In a world where everyone is taught the same viewpoint, any dissent is seen as heresy.

And the second problem is the disconnect between mathematicians - who think that the world should behave according to their maths, and the engineers - who know damn well the world does what it wants and blow the maths!

I'm very firmly in the second category, and as I see it, the majority of data comes as lists, and if a "database" can't store lists, then it's not fit for real world use!

To quote Einstein, "make things as simple as possible, BUT NO SIMPLER". Because relational databases make things too simple, the applications have to be overly complex, such that the final solution is forced to be much more complicated than necessary. If my application uses lists, then the DATABASE should handle looking after the lists. With a relational database the APPLICATION needs to worry about managing the lists, keeping them in order, looking after the metadata (ie which item is first, second, etc). All stuff which belongs in the database!

> Please, read about "normalization" (about the third normal form, in particular).

You said above that relational algebra doesn't care about "attributes". Why then is the wikipedia article on 3rd normal form full of references to attributes? And while I'm not up on relational terms, I thought a tuple was implemented as a row in a table - which is a bunch of related attributes stored in cells. Indeed, re-reading what you wrote I'm not sure you understand (or certainly can't explain) what you're talking about. Are you seriously telling me a relational relationship is represented by a table? Even if I don't properly understand the language, that sounds nonsense to me!

Oh - and Pick (done properly) IS a normal form database. It's just that (1) it's not a first normal form (relational) database. And (2) Pick doesn't enforce good design - it's up to the programmer to make sure the FILEs are properly normalised. But if they are, it is ABSOLUTELY TRIVIAL to convert a Pick file into a group of joined relational first normal form tables. So trivial, in fact, that it's done as a matter of course to present a SQL interface to those apps that expect a relational database, and done using generic tools in the database.

The problem with relational databases is they were designed by mathematicians, not engineers. Pick was designed by engineers. And imho implement Pick using relational insights and you end up with a dream database. But relational - because it is pure maths and ignores the real world - doesn't actually work!

Look at dakas post elsewhere in these talkbacks. He took a program written by computer scientists that ran for more than a day, and rewrote it. The rewrite terminated after a couple of minutes ... Pick was written by real-world engineers to solve real-world problems, and it shows. Relational was designed by mathematicians, and that shows too!

Combine the two, and it will blow everything else out of the water, but unfortunately, certain fundamental assumptions of Pick and Relational are incompatible. And it's the Pick assumptions that result in the speed advantage - such that a Pentium 90 can give a dual-processor Xeon 800 the fright of its life!

Oh - and I would have NO QUALMS WHATSOEVER at taking a client-server relational system and replacing the relational database back end with a Pick system. A bit of effort "denormalising" the system to a properly designed Pick data schema and the only thing the clients would notice is that the back end suddenly started behaving like a speed demon on rocket fuel.

Cheers,
Wol

Four years of Go

Cyberax — Fri, 22 Nov 2013 22:47:59 +0000

> That's because you're thinking relational when using Pick. OF COURSE it's going to suck. With a Pick database why would you WANT to do a join? ANY join? It's like talking French in a Mexican restaurant in New York.
Why not? Especially if you are trying to help a tourist from France, who is lost in the menu.

That's the power of relational databases - I can reasonably expect ad-hoc queries to be fast enough if I design a schema good enough. In Pick (and most of NoSQL databases) you have to design your data schema for all of the queries, if you want to do something ad-hoc that doesn't map to your data well - you're screwed.

> To which I have to add, joins are part of the program layer above the data store in a relational database. What's to stop some enterprising vendor from adding the SAME layer to a Pick database - oh but they have - it's called SQL.
There's a joke in Russian: "If a grandmother had balls then she would be a grandfather". It applies here.

Fortran no more

Wol — Fri, 22 Nov 2013 22:44:57 +0000

Have you actually tried to DO that?

Okay, I was using MS C v5, but when I started playing around with defining things like

int a[3][2];

sub(a);

then sub screwed up accessing a something horrid. And when I tried to work out what on earth was happening, by reading the language spec, it was pretty clear that the implementation was undefined for anything more than a one-dimensional array, and you were just asking for trouble expecting it to survive being passed as an argument.

C assumes that you are using pointers. Arrays are an after-thought. Which makes using 2 (or more) dimensional arrays rather problematic ...

Cheers,
Wol

Four years of Go

Wol — Fri, 22 Nov 2013 21:07:39 +0000

>> But if you need to do ad-hoc joins then Pick totally sucks.

>> And it turns out that most problems belong to the second category.

That's because you're thinking relational when using Pick. OF COURSE it's going to suck. With a Pick database why would you WANT to do a join? ANY join? It's like talking French in a Mexican restaurant in New York.

I've just looked up graph databases in Wikipedia. And guess what - it says "graph databases typically don't need expensive joins".

To which I have to add, joins are part of the program layer above the data store in a relational database. What's to stop some enterprising vendor from adding the SAME layer to a Pick database - oh but they have - it's called SQL.

But as I keep saying, what you need to do when using Pick is think of each FILE as containing a relational view of an object. Then you get the speed of Pick with the flexibility of relational - if you want the worst of both worlds all you have to do is map each relational table to a Pick file and query it with SQL. Dead easy, but I wouldn't recommend it ...

Cheers,
Wol

Four years of Go

Wol — Fri, 22 Nov 2013 20:07:15 +0000

EXACTLY!

What was that I said? On a 386-equivalent system, thrashing like mad, the response for THIRTY-TWO users was perfectly acceptable ...

I've seen repeated stuff that says Pick systems scale - and apart from file and file-system limits I've never heard anybody say that the system can't cope.

Cheers,
Wol

Fortran no more

hummassa — Fri, 22 Nov 2013 16:24:50 +0000

I concur 100% with the forementioned explanation.

I program a lot in C++, and ALL the problems I have with it lately is when I need to do something Unicode-related.

Fortran no more

mathstuf — Fri, 22 Nov 2013 15:00:50 +0000

<speculation>
Well, it could probably be due to the C++ committee not wanting to split platforms (Windows and everyone else) by choosing either UTF-16 or UTF-8 as an internal representation. Languages like Python have the benefit of solving these problems by fiat (because there is a BDFL or it's the only implementation), but C++ is committee-run, so any one-sided solution ends up not being implemented and everyone gets screwed with a no-sided solution ("it's just bytes").
</speculation>

Four years of Go

ibukanov — Fri, 22 Nov 2013 14:52:05 +0000

> Ad-hoc queries might certainly suck, but they will work.

It depends on the size of the data. With 5TB Oracle DB cannot be used in practice for anything but searches that hit indexes. And this is the problem with the relational DB - it all starts well with medium-sized data and nice custom queries, but then the data grow and the DB hits the implementation limit much faster than anticipated. And then one just wonder what exactly was the purpose of those license fees and hours spent on installing, tuning and monitoring the DB...

Fortran no more

ceplm — Fri, 22 Nov 2013 14:47:55 +0000

OK, let me restate: language which in this time doesn't contain in its standard library good support for Unicode string is quite suspicious (yes, I know about how problematic sometimes this support is, and UTF-16 v. UTF-8, Han Unification, and combining characters, etc.).

Fortran no more

hummassa — Fri, 22 Nov 2013 14:47:32 +0000

Well, actually, C++ has std::string nowadays but it did not have it when Qt was started (hence my remark that things are getting better in this front, especially in Qt5).

What I found really missing last time I tested the C++11 libs: the encoding classes were not up to spec (in this respect, QtString was really more complete than std::string).

Fortran no more

mathstuf — Fri, 22 Nov 2013 13:40:04 +0000

I had the same questions when looking at Qt. It turns out that Qt is older than the STL (1991 vs. 1995 or 1996 when the STL could actually be used). Knowing that and that QString is amenable to Unicode makes using std::string a hard sell. QString is also implicitly shared where std::string doesn't have to be (though at least libstdc++ seems to do so).

Fortran no more

ceplm — Fri, 22 Nov 2013 13:34:41 +0000

What I meant is that language which pretends to be a bit higher level than C and doesn't have standard string type (why there is QtString at all?) seems to me to be a bit problematic.

Fortran no more

mathstuf — Fri, 22 Nov 2013 13:11:40 +0000

Well, QString supports Unicode well whereas std::string is basically a bytestring which cannot contain NUL (well it can, but then it acts funny with C code). They're fundamentally different, so interop *shouldn't* be automagically done by the compiler.

Fortran no more

jwakely — Fri, 22 Nov 2013 12:24:55 +0000

I still don't see what difficulty you're referring to.

If you mean this part:

> Additionally it seems that in Qt conversion operators are frowned
> upon. For example, QString doesn't define conversion operators to and
> from std::string but uses explicit methods for that.

That's not a "difficulty" it's a design choice, and one most good C++ programmers agree with. Explicit conversions are bad, mmkay?

If there is some difficulty described in that thread that I'm failing to see then it's not caused by libstdc++ (which is just one implementation of the standard) but by the interfaces required by the standard.

Four years of Go

khim — Fri, 22 Nov 2013 09:45:34 +0000

Yup. This is about right dichotomy: NoSQL for pre-defined fast queries, SQL for ah-hoc, slow (but working!) queries. Both people who are using SQL to process pre-defined queries bazillion times and people who are trying to process ah-doc queries with NoSQL are “doing it wrong”™. Note that with some preprocessing and with some limitation SQL can even be used for large datasets.

Four years of Go

Cyberax — Fri, 22 Nov 2013 03:36:03 +0000

Ad-hoc queries might certainly suck, but they will work. And on the general amount of data in a typical company they work well enough. And often the query execution time is not that important, a marketing analyst can often wait for a few minutes for a report. And given that companies were the major buyer of software in 80-90s it's no wonder that SQL databases ruled.

Now, if we're talking about a highly-loaded website where users expect pages to appear in milliseconds then it's another story. De-normalizing data and making complex documents accessible as atomic objects often becomes necessary, so graph and hierarchical databases (now known as NoSQL) are reappearing.