Your six- or seven-digit passcode is unique. Only those who key in the code can enter your conference.
Additionally, everyone hears a brief tone when someone enters or exits the call. As the Host, you can press *5 to lock (or unlock) the conference, which prevents others from joining. Press *# at any time to hear how many are connected to the call.