I had a connector in the 3.5 mm headphone jack, but when I pulled it out, the connector broke off and is now stuck in the jack.

There are some things you can try, but it is entirely possible that this jack will never be usable again. Each of the steps below could potentially render the jack unusable, but the first step is the least likely to do so.

1. Take apart a ballpoint pen with a plastic ink cartridge. Use the back end of the plastic ink cartridge (without the pen tip) and stick that in the hole and try to catch it on the end of the metal plug and pull it out.
2. If step 1 didn't work, use the same plastic ink cartridge, but hold the back tip over a flame for 1-2 seconds to soften the plastic (do not melt it) and try to stick it in the hole again while the plastic is still soft.
3. If neither of the above steps work, try the superglue method: apply a very small dab of superglue to the end of the plastic pen ink cartridge, stick it in the hole, wait for it to dry, and pull it out. Just a warning: the vast majority of people who try this report that it does not work, but you might get lucky.