This type of thing is what went wrong in libpng. Their worst sin was /inserting/ new structure members into an existing structure definition which was part of their public ABI. But there were many others, including changing the declaration of ABI functions depending on your build flags.
For a while its authors tried being indignant, insisting that you should only use a libpng program with the precise build of the precise version of the library against which it was compiled.
Eventually we managed to explain to them how ABI compatibility works, and the modern libpng takes roughly the getter/setter approach with the caller having only an opaque handle to the internal structures.