Speed is a big issue, as well as address space. Caches can't be huge and really fast, partly because a fast cache is crazy power hungry, and a big cache has large distances to travel. So your on-die caches are hierarchical in part so that you can have a fast cache and a big cache.
The second problem, address space- larger address space requires more addressing bits, which makes the CPU bigger, and bigger means slower. This constrains cache as well as memory. The final level, disk, has that as an advantage over DRAM because it is not memory-mapped. It's not frequently a problem, but for example Nehalem, despite being a "64-bit" architecture, actually only supports 44-bits of address, setting a hard limit at 16 tebibytes of RAM (not considering MMIO etc)
The second problem, address space- larger address space requires more addressing bits, which makes the CPU bigger, and bigger means slower. This constrains cache as well as memory. The final level, disk, has that as an advantage over DRAM because it is not memory-mapped. It's not frequently a problem, but for example Nehalem, despite being a "64-bit" architecture, actually only supports 44-bits of address, setting a hard limit at 16 tebibytes of RAM (not considering MMIO etc)