"There's too many aspects to keep track of!"

Okay, so in another conversation with +Cameron Corniuk, this came up, and I was looking for information on it.  Mostly because I've heard this a few times, but it's not something I've ever personally encountered, and am trying to understand if I'm missing something, or if I have something to share, or what the case may be.

One of Cameron's examples was that in a game with four players, you automatically have twenty aspects in play.  So I'll start with that.  And he's absolutely right, and that's too many to keep track of.

Which is why I don't, as either a player or a GM.  As a player, I've never felt any need whatsoever to keep track of the aspects that my fellow players have.  I trust that they'll inform me if there's something applicable.

But even as a GM, I don't "keep track of" aspects, per se.  I write them down.  And since aspects don't really "do" anything until triggered in some way (mostly), I don't really worry about them.

So what happens then is that I'll either look for something to invoke or compel.  Possibly Create Advantage of, as well.  Then I do a quick scan over what's available, see if there's anything appropriate, and trigger off of that.

I'll usually keep a few situation aspects in mind, that are the most relevant, but that's about it.  At no point do I ever try to keep 20+ aspects in mind.

What experiences have others had with this?