To build or not to build un-needed kernel modules

I have built a working kernel with the settings needed [*] built into the kernel. There are still many settings that are left as [M]odule and those modules are built as the kernel is built. My board is headless, no display ports, no graphics, no sound. The built modules are not copied to my target board. The rootfs (built from scratch) and custom apps/executable(s) do not complain about missing modules.
The kernel size won’t be reduced. The savings will be in kernel build time.

Is it worth taking time to edit the kernel config and remove all the [M]odules?

I look forward to your comments.
Thank you,