You still own the code you release under GPL. the restriction you are describing is actually caused by the non-copyleft licences you claim to prefer. If you choose to use MIT, you are limiting which libraries you can use. If you had picked GPL to begin with, you can use any library.