People Aren't Thinking About You, Except When They Are

There's this piece of advice that gets bandied around a lot about how "nobody is thinking about you half as much as you think they are."

Rough idea being: don't worry so much, you imagine everyone is obsessed with your mistakes and quirks, but actually they're too busy being obsessed with their own mistakes and quirks.

I'm sure this is mostly right, but there is a flipside I encounter occasionally: sometimes, people will remember a small stray comment you made 12 years ago. Sometimes they loved it, sometimes they hated it, sometimes they misinterpreted it, often you've forgotten it.

I'm not sure what the right synthesis of these two facts is.