vs. VSIPL++?

I love MPI, but it definitely leaks out. You generally have to run the program with a special launcher and every process runs main. Cache coherency is a big killer for SMP, MPI deals with that at the expense of needing to move more memory around (for example, copying between differnet processes address spaces). Explicit threads with affinity and awareness of the cache behavior of their neighbors can be better yet, but that model is really only possible for a few very special kernels (sparse matrix-vector products for instance) and even then isn't widely available.