> The designers arbitrarily decided that 32 bit fixed length addresses would be good enough indefinitely, when in practice both the fixed length constraint and the 32 bit constraint were causing serious problems within less than fifteen years.
Please do not throw out the baby with the bath water: the fact that addresses are 32bits wide is not a core design principle of IP! It is just a minor implementation detail gone really wrong.
It is much more difficult to have no mid-layer at all when designing communication protocols than when designing filesystems or other kernel subsystems. Simply because you are not alone. Upgrading your kernel is easy. Having everyone upgrading its kernel is obviously much harder. Already today you CAN avoid TCP/IP entirely and send raw packets on the wire! But they will obviously not go any further than your network neighbours. To go any further you MUST agree on a minimum set of conventions (that is: a protocol), including a fixed format for addresses. Else please explain how to route hundreds of gigabit per seconds with free form addresses.
I am definitely not pretending that IP is the perfect layer 3, far from it. But it is a *minimal* one, really. This is actually both its strength and weakness.
Please name a layer 3 lighter and with less constraints than IP (I did not say ¨better¨).
> So to fix this enormous mess, the IETF goes out and designs another
protocol, IPv6, a protocol which is incompatible in almost every way
imaginable with the protocol it is trying to replace,
This is only because of IPv4´s original sin which was never designed to be ¨upgradable¨. Easy to blame 40 years later. Since you have to give up on compatibility anyway, then better start from scratch and not copy/paste IPv4 past mistakes. And by the way, IPv6 is still a ¨minimal¨ layer 3.
> to the point that many think widespread migration is never going to happen.
Every operating system already supports IPv6, and a number of consumer ISPs are already offering IPv6. Many people are already using it (I do), it already works. Except in the US maybe. The reason withholding IPv6 deployment is not incompatibility but laziness and wealth of IPv4 addresses (and only for some countries).
> To say nothing of the standard BSD socket interface, which makes it more or less impossible to write a layer 3 transparent application program,
It is not pretty but possible: please look at getaddrinfo() examples. Anyway I fully agree that the BSD socket API sucks, but this is a different topic.