You know, FTP doesn't give file listings in a fixed format either, but as a plaintext file, which the client must parse differently depending on what FTP server you are running, and sometimes based on the LC_xxx settings on the server...
The closest thing to a standard there is is "do like POSIX `ls -la`", but that is not universally followed, especially by ftp servers on non-POSIX hosts...
For HTTP there is at least webdav, which can provide a standardized machine parseable directory listing...