Yeah I do it, and you have to be relentless. Mark every video you don't want to see.
And god help you if you watch even one off-topic video - like another comment mentions, YouTube algorithm then concludes you want to see a million more. Reminds me of a Weird Al lyric about TiVo: "I watched Will and Grace one time, one day. Now TiVo thinks I'm gay."