It's not my favorite API, but for a cross-platform (incl Windows) VoIP app, I probably would go with Portaudio.
SDL has a nice and friendly audio API, is cross-platform and has worked very well in my experience but it doesn't do recording.
The PulseAudio API is OK to work with as well. It probably would be my choice for a VoIP app if I didn't have to care about portability.
GStreamer is extremely focused on media-player like applications, and all API documentation is built around the assumption that data comes from somewhere else and you're just building a pipeline. Using an application as the data source seems so to be rare and it's not obvious how to do it.