In case of in-lecture quizzes, is it unreasonable to fail students who are late or absent?

Do you think it's me being unreasonable?

Sadly, yes. I see two problems here:

1. You said in a comment that the policy of "the quiz can happen at any time" was not articulated at all. You gave the quiz near the end of the class a couple of times, and the students naturally assumed that that's when you'll always have it. Now you are trying to argue legalistically that since you didn't say when the quiz will occur, you were reserving the right to hold it at any time. However, in my opinion if you are going to argue based on legalisms, the burden is on you to dot all your i's and cross all your t's and make sure that you gave your students such amazingly clear information that they couldn't even imagine arguing with you.

   The thing to keep in mind here is that you are an educator, not a prosecutor trying to entrap a criminal in court. The students deserve to have clearly articulated rules so that they can focus their time and energy on the material; in particular, if there's a quiz, they deserve to know the precise time when it will be held, or at the very least to have an explicit announcement that the quiz can be held at variable times. Overall, your approach to dealing with the students who came late as I'm understanding it from your question seems to me to place you in an adversarial, combative position relative to your students, which is not where you want to be as a teacher - it can only serve to distract from your educational objectives and provides for a poor learning environment for your students.

2. A second problem is that I'm getting a strong feeling from reading your question that you're trying to use the quiz as a crowd control mechanism, which feels wrong to me (and partially as a result has gotten you into the current messy situation). You've devised an elaborate strategy that consists of holding a quiz at a randomly selected time during the class as a solution to the problem of making students come to class, and come on time. The problem of students coming in late may be a very real concern, but random quizzes are simply an inappropriate and ineffective way to address it (and one that can potentially be perceived quite negatively by the students).

   A quiz is a form of assessment, and is a legitimate device to use for that purpose, but let's keep things in their right place: deal with the problem of disruptive late-coming students in the appropriate way, and have your assessment in the way that makes sense from an educational standpoint, whether it be quizzes, exams, homework, or even mandatory class attendance if that's important to you and your institution's policies allow it.