There is a lot of latency because there is a memory controller between everything that works out how to send commands across and get data and put it in the CPU, mostly because just accessing RAM outright by attaching it to CPU pins does not work anymore (and partly because the memory controller adds features, but he is become less of an effect).