When you hit publish they will not be invited automatically, you need to invite them manually.
You can select only those users who you want to test the app. From Users tab you will need to tick the boxes for the desired emails and press on re-send invite button from Actions section.