Extended family? I would use the name no matter how much they hated it.
Spouse? No, I would not use the name. I think both parents should be in agreement on the name so I crossed off names that J did not like and he did the same for me.
That said, J's mother, J's Grandmother as well as my mother all hated Tobias as a name. Now that Toby is here they all seem ok with it.