Debian is already there, and I've been running it on my smartphone as primary OS for years ;) It's just about where to pour one's interest in so far as to productize things for not only individuals themselves but more to ordinary end-users, in addition to hw availability issues. I've only scratched the surface of that with my instructions and (poor and minimal) preinstalled image. One can argue Debian hasn't done that for ordinary desktop users either, but it was Ubuntu that did it.
So, Debian for smartphones already exists and welcomes people, but it's the more narrowly specified and more productized Ubuntu for smartphones that's missing. Mer + MeeGo CE + Plasma Active? Tizen?