Answer: When you are running a dedicated server, especially one with live websites, there is always a lot to learn. From domain name servers (DNS) to mail server configuration, you may find yourself with little time for anything else. One security question that sometimes crops up is whether or not it is beneficial or even safe to run an X-server on top of your Linux operating system.
The first answer to the question is that you probably do not have a reason to run X.org at all. Most servers do not need X.org to operate. Even if you want a GUI interface for your server applications, most are web-based rather than X-based. Furthermore, if you are administering your server remotely, X might not even be practical. There are only two situations where you really might need X: 1. Your server is not only a web server but serves other functions specifically requiring desktop applications, or 2. for some reason, you need to forward X.org applications over SSH.
The next question to answer is whether or not it is secure. The simple answer is that most Linux experts would tell you no and that you should avoid it at all costs. The real risk, however, comes from X11 connection forwarding. This is a built-in component of X.org that allows a user to log into it remotely. It is primarily used for internal networks, and you have no need for it. Disabling that and its ports in your firewall will make X.org more secure, but you still should not use it unless you absolutely need it.