These days, I’m not sure that there’s any one element of music that, alone, will make a piece of music widely inaccessible. A piece with screeching dissonance and complete lack of meter, sure: I think a random person off the street will have a hard time relating. But just screeching dissonance? Plenty of pop music has harsh dissonance; it just ameliorates it with a strong sense of meter. Lack of meter? Floaty ambient music gets away with that, because it rests on a cushion of consonance.
Anyway, I’m pretty sure that you all (by which I refer to my wide, wide readership*) can come up with plenty of counterexamples to this over-ambitious generalization. Still, I think it points to something meaningful. At least, I hope it does.