What happens when someone joins a group with the entry check on?

It depends on whether your group is closed or open, and Telegram decides which. In a closed group where people ask to join, the bot sends the question to the applicant in a private chat and approves them only after a correct answer.

In an open group anyone can walk in, so the bot lets them in and immediately mutes them until they answer; if they never started the bot, they see a message visible only to them with a button that opens the private chat.

Either way the newcomer gets a simple arithmetic question with six options and two attempts. A correct answer lets them in or unmutes them at once. After two wrong answers the join request is declined, or in an open group the person stays muted until an admin unmutes them in Telegram. If they never answer, they are removed from the group after 60 seconds.

That removal is not a ban: the person can come back and try again, so a real player who simply missed the message is never locked out. A correct answer gives the newcomer a limited set of rights: writing, media and reactions. It does not hand over group management, so passing an arithmetic check never lets someone invite people, pin messages or rename the group.