Frequently you just need to run one command after another, and nothing is simpler than shell for doing this.
Really complex shell programs aren't started as complex programs, they start off as simple programs and get incrementally enhanced until they are big complex programs, at each step of the way it's substantially less work to just tweak this one thing then it is to re-write the entire thing in some other language.
There are also a LOT of tools available to do things for you, so you don't do them in shell, you do them by invoking a program and it does the work for you.
If most of the work you are needing to do can be done by these programs, your 'shell program' can outperform the equivalent program in the 'better' shell languages.
using these external tools is frequently simpler than programming the same thing in one of the 'better' languages
you want to clear a directory before you start doing other things, in shell rm -r dir does it, how many lines of code does it take to do it in Perl or Python?
Least common denominator is a factor, but I think it's a small factor. It's an unusual server that doesn't have both Perl and Python on it, so lack of availability of the language is not an issue (you may not have the libraries you want installed, that can be a real issue)
I do a lot in shell, and a lot in Perl (I do a little in Python). If I am doing something quick and simple, I'll use shell, if I know it's going to be doing a lot of data manipulation, I'll use Perl.
ALthough I'll point out that sed, cut, grep allow you do solve a lot of 'data manipulation' issues fairly easily, so Perl comes into play more when calculations or non-trivial logic come into play.
Yes, Perl has the system() call (and Python has the equivalent), and everything I'm saying above can be done in these languages with a series of such calls. But it's hard to say that the resulting program is any better than the shell program it replaced. I once had a 20 line, ugly shell script that I wrote turned over to the software engineering department for "official maintenance" and to bring it up to "professional standards". In three months they re-wrote the 20 line shell script into a 200 line Perl script, with everything that actually did something being a system() call that invoked the shell commands that were in my initial script (long ; joined lines remaining intact). But this was now a "professional" program, it had lots of comments (none of which explained what any of the calls were doing, or why), every function was less than one screen, etc. It's the best example I've ever seen of how "coding standards" can make a program hard to deal with.