How do I answer a question on graduate school applications on textbooks used in my classes, when my class didn't use a textbook?

Then say that! Tell the instructor's name, and give a link to the instructor's notes. If the notes are somehow not publicly accessible, then, yes, it'll be impossible for anyone else to know what is in them.

The point of such a requirement is to give much more detail than saying that you studied "algebra" and "analysis" and "combinatorics" and ... since these labels are far too vague, and also do not indicate the level at which the course was conducted.

EDIT: Contrary to a comment, I think that being able to describe the source(s) for one's study is very important. E.g., the author. If the notes are/were in a language other than English, that hardly matters, if the note-writer is known to the international mathematics community, and, again, if you can provide a link to the notes. Most older mathematicians can read French, German, and fake a bit of Dutch, Italian, Spanish, maybe slavic languages. Just be forthright: tell who the instructor was/is, and show the notes.

Again, the real point is to describe what was actually studied, since there is no universal syllabus, nor universal standard for level of rigor, abstraction, etc. Just saying "Algebra" communicates almost nothing, and admissions committees want more. An answer of the form "I can't describe what I studied/learned" is unhelpful, and will not gain you much, as you can imagine.