We can afford to do that (which should stress less the [matrix] homeservers) because we no longer drop all the user displaynames across all channels) when we see a Leave or Ban membership event