Well don't forget that opening malicious documents in applications can have the same effect as executing a binary.
Another one is using built in languages for applications to execute virus-like things. For example Word Macro viruses were very very popular.
So... say in Linux there is a flaw in Envice's PDF rendering method some were. So you could send a legal PDF over email that when executed by Evince it would exploit that flaw and run some shell code.
Since there is no security internal to a user account or desktop any program, even the most trivial and unimportant, with a exploitable flaw can be used to gain full access to anything and everything on that user's desktop..
This example is just targeting Nautilus or Konquerer to do a oversight in the .desktop standard... but any program that is commonly used to handle files downloaded from the internet has the same potential problems.
With Linux having a generic attack like with Word Macros or Win32 HTML rendering flaws probably won't work.
So worms and viruses won't spread. The environments are to diverse and are patched too quickly for a generic attack to work in that manner.
This is, in fact, what makes Linux resistant to viruses even though the Linux binary formats makes it very easy to write viruses.
HOWEVER this does not make Linux resistant against focused attacks. If a attacker knows what desktop your using and knows that there are flaws in some of the software your using on your desktop then they could create a focused attack that, if they are targeting a corporate desktop install (for example) with hundreds of users, then they can have a very high probability of success.