Generally speaking, it is a sound idea to keep the default packages, kernel, and drives installed on your server when you first purchased it or when you first installed your Linux-based operating system. Recompiling a kernel is usually something taken up by adventurers and computer science students. Despite that general trend, there are some rare cases when compiling the Linux kernel from source might be beneficial or necessary.
1. Kernels compiled for specific hardware are generally faster.
2. You can exclude modules and other components you do not need.
3. If you have specific requirements, you can add them or even change parts of the source code completely.
1. You become responsible for updates, security patches, etc.
2. Something could go terribly wrong if you do not compile a good kernel.
3. It takes a long time and requires a good deal of technological expertise.
It is truly up to the system administrator to decide whether a custom kernel is necessary on a dedicated server. Choose wisely.
Image Source: Wikimedia Commons