Moving past TCP in the data center, part 1

You're thinking of the 'enterprise' type server world. I'm thinking of the much, much, much bigger FANG-type DC world. At at least 1 of the biggest DC operators in the world, the (newer) servers do not have an ethernet PHY. The host has a PCIe connection to a special shared-NIC (which is like a very cut-down, hacky L2+ switching ASIC). 3 or more hosts connect to the shared-NIC. The shared NIC then switches the host ethernet packets, which are sent over hardware specific messaging to the shared-NIC, onto the PHY.

There is one more PHY, to the BMC, which controls the chassis and can give serial access to the hosts (and some very slow, hacky IP/TCP to the host, IIRC).

There are some large enterprise "blade" type systems which also use onboard switching ASICS I think, but the ones I know of use more traditional switching ASICs, (e.g. HPE were using Intel RRC10k) with actual NIC blocks for each host included into the ASIC. So these actually look and work a lot more like a traditional network, with proper buffering and flow-control between the hosts and the switching logic and the upstream (the shared NICs above did not do this properly - least in earlier iterations - causing significant issues).