It would not work. There are two use cases that the pointer/page combination doesn't cover, namely: removable storage and named data. Both are very important, so a system that doesn't support them will not be very useful.
And as Viro said above, the moment you add a name->page mapping... voilà! the filesystem is back.