Device drivers and processor architectures might make the source tree big, but they don't make the kernel slower or its memory footprint larger. I believe you can even still compile the kernel without SMP support.