For one, what’s the minimum storage size of a distribution? If you manually specified “all” of the percentiles of the exponential distribution, it could take an infinite amount of storage space, right? But you only need one parameter (or should I say three: base, λ, and the function definition) to generate it. On the other hand what’s so special about the functions that we know? A sum of exponentials is simple in some other linear basis but I can’t think of a cheap way to store that in a computer.

For two, every probability distribution is just a 1-generalised function s.t. ∫ƒ is a function while ƒ is not guaranteed to be one. Which is why it makes sense to store the CDF’s (they’re also monotonic which has to be good for making them small in some way). So now chase the diagram backwards: your solution with storing percentiles of CDF’s is applicable to any 1-generalised function. Does that buy us anything for free?