Perspective

from what I understand, you don't have to get the corners of the building, any rectangle corners can do. you may not be able to get the building corners, but you can usually get window corners that are lined up vertically and horizonatally. I agree that the idea of being able to specify vertical and horizontal lines without them having to be a rectangle would be a nice option, but most of the time they will form a rectangle

It's also worth remembering that this data is going to be used to distort the image. The camera lens also distorts lines (some more than others, fisheye lenses being the extreme example). So the result you get is not going to be perfect anyway. You are distorting the image to counter other distortion to remove the distraction. If you want perfect correction, then you need to map out the exact distortion created by the lens (and this will change based on zoom setting) and use that map for the correction.