m64.pl
06/08/14How I learned that the default settings are not production settings
I got the rare Linode Alert in my email today, claiming 111.8% CPU usage on my modest little 2048 box. When I followed the link to Linode's dashboard, I saw on my charts that it had actually risen to around 600% and had been at that level for about 3 hours. What the fuck? What am I doing that could possibly have gone so wrong all of a sudden? I serve a couple webapps on this box but they have always been well-oiled machines.
As soon as I ssh in and run top
, I'm even more confused.
User www
, which as far as I know doesn't do anything at all, is throwing a party with a perl script called "m64."
I killed that shit as soon as I had collected evidence. I'm surprised and glad Linode didn't shut me down for clobbering their machine like that. Then I started Googling for this script name, to see what it could have been doing. The only thing I could find was this Feb 2014 post which indicates that it's a Bitcoin mining script. That makes sense, since one of the two things that box serves is my Bitcoin market data tool for traders. It probably drew some attention.
Naturally I was curious how someone got a file on my server. It was running as an unprivileged user,
which was a slight relief, but nothing compared to the sinking feeling that my beloved Linode had a malevolent
stranger creeping around in it. I went through the ssh logs at /var/log/secure
and was horrified.
In the last 12 hours, there were thousands of failed ssh login attempts mostly as root
, admin
, and apache
, but also many others.
Literally tens of thousands of lines of this. A login request every second.
Jun 8 16:01:20 meowset sshd[15833]: Failed password for root from 117.21.191.210 port 2643 ssh2 Jun 8 16:01:22 meowset sshd[15841]: Failed password for root from 117.21.191.210 port 1382 ssh2 Jun 8 16:01:24 meowset sshd[15833]: Failed password for root from 117.21.191.210 port 2643 ssh2 Jun 8 16:01:25 meowset sshd[15841]: Failed password for root from 117.21.191.210 port 1382 ssh2 Jun 8 16:01:26 meowset sshd[15833]: Failed password for root from 117.21.191.210 port 2643 ssh2 Jun 8 16:01:27 meowset sshd[15841]: Failed password for root from 117.21.191.210 port 1382 ssh2 Jun 8 16:01:29 meowset sshd[15833]: Failed password for root from 117.21.191.210 port 2643 ssh2 Jun 8 16:01:29 meowset sshd[15841]: Failed password for root from 117.21.191.210 port 1382 ssh2 Jun 8 16:01:31 meowset sshd[15833]: Failed password for root from 117.21.191.210 port 2643 ssh2 Jun 8 16:01:31 meowset sshd[15841]: Failed password for root from 117.21.191.210 port 1382 ssh2 Jun 8 16:01:31 meowset sshd[15842]: Disconnecting: Too many authentication failures for root Jun 8 16:01:31 meowset sshd[15841]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=117.21.191.210 user=root Jun 8 16:01:31 meowset sshd[15841]: PAM service(sshd) ignoring max retries; 6 > 3 Jun 8 16:01:33 meowset sshd[15833]: Failed password for root from 117.21.191.210 port 2643 ssh2 Jun 8 16:01:33 meowset sshd[15834]: Disconnecting: Too many authentication failures for root Jun 8 16:01:33 meowset sshd[15833]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=117.21.191.210 user=root Jun 8 16:01:33 meowset sshd[15833]: PAM service(sshd) ignoring max retries; 6 > 3 Jun 8 16:01:36 meowset sshd[15961]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=117.21.191.210 user=root Jun 8 16:01:37 meowset sshd[15940]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=117.21.191.210 user=root Jun 8 16:01:39 meowset sshd[15961]: Failed password for root from 117.21.191.210 port 1931 ssh2 Jun 8 16:01:39 meowset sshd[15940]: Failed password for root from 117.21.191.210 port 2359 ssh2 Jun 8 16:01:41 meowset sshd[15961]: Failed password for root from 117.21.191.210 port 1931 ssh2
An obvious brute-force attempt, but not completely automated. These are interlaced
with seemingly human attempts at logging in as users like shit
, and shit2
.
Pretty futile and confusing.
Jun 8 16:34:55 meowset sshd[27949]: Invalid user shit from 195.158.109.185 Jun 8 16:34:55 meowset sshd[27957]: input_userauth_request: invalid user shit Jun 8 16:34:55 meowset sshd[27949]: pam_unix(sshd:auth): check pass; user unknown Jun 8 16:34:55 meowset sshd[27949]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=mail.sterling.com.mt Jun 8 16:34:55 meowset sshd[27949]: pam_succeed_if(sshd:auth): error retrieving information about user shit Jun 8 16:35:02 meowset sshd[27949]: Failed password for invalid user shit from 195.158.109.185 port 56864 ssh2 Jun 8 16:35:02 meowset sshd[27957]: Received disconnect from 195.158.109.185: 11: Bye Bye Jun 8 16:35:04 meowset sshd[28011]: Invalid user shit2 from 195.158.109.185 Jun 8 16:35:04 meowset sshd[28012]: input_userauth_request: invalid user shit2
Here's a full list of invalid users for which login attempts were made, which I find really interesting. It looks like a lot of these are coming off Centos's standard users, which apparently my installation didn't include.
admin ajay ale anti asterisk avahi bszsimon bwadmin cacti chem cosmos csaba csanyip cskiraly cvsadmin db2inst1 deploy desiree dev devteam garrysmod gdm gergely git gnats guest guest1 guest2 ielm jboss jessie jim jira johnky karakai konsti krejczinger kris lhc lswang magento manager manon master meres mihaly minecraft mysql nagios nam ndbao news nikos no-reply noreply nouser oracle pappd periodic photo photos postgres postgres2 postgress postgrestest postgresuser postmaster redmine research rpcuser rsync rtorrent rvm sabayon share shit shit2 support supports supporttest supportuser temp test test1 test2 tom tomcat tsumura upload user user01 user1 user123 user2 userfetch userftp userguest userhome users usr usr1 usr2 vwalker william www-data xochitl yuji zabbix zimbra zsofi zspandi
There were only a handful attempts for each of these. Apparently if an attacker detects that a box accepts ssh using a password, they try to log in as every possible user that could exist in the hopes of an easy password or an empty password.
Also, for fun, here are all the IPs that tried brute-forcing in as root
.
The majority of them seem to be Chinese.
1.93.33.77 100.2.156.3 113.171.10.1 115.239.248.61 116.10.191.163 116.10.191.170 116.10.191.196 116.10.191.232 116.10.191.234 116.10.191.253 117.21.191.210 122.225.103.118 195.158.109.185 211.25.206.34 220.177.198.26 220.177.198.43 222.186.40.170 222.186.56.33 61.142.106.5 61.153.105.69 62.212.141.128
Seeing all this made me a little anxious that perhaps they would manage to knock down the gates any minute. The ssh logs seem to be truncated, so I don't even know how long this has been going on for - relentlessly for all of today at the least.
But what about that Bitcoin miner? As the logs show, apparently many different IPs
had managed to log in as www
. I'm not even sure where that user came from, or what its password was.
Jun 8 10:54:03 meowset sshd[2083]: Accepted password for www from 23.102.134.179 port 1168 ssh2 ... Jun 8 14:39:51 meowset sshd[18941]: Accepted password for www from 191.238.227.219 port 1048 ssh2 ... Jun 8 15:03:28 meowset sshd[27443]: Accepted password for www from 191.236.20.93 port 1056 ssh2 ... Jun 8 16:36:25 meowset sshd[28548]: Accepted password for www from 195.158.109.185 port 33915 ssh2 ... Jun 8 17:55:00 meowset sshd[24431]: Accepted password for www from 23.96.51.35 port 1072 ssh2 ... Jun 8 18:37:27 meowset sshd[7806]: Accepted password for www from 23.96.53.244 port 1160 ssh2 ... Jun 8 19:42:04 meowset sshd[31758]: Accepted password for www from 23.102.134.202 port 1064 ssh2 ... Jun 8 19:46:13 meowset sshd[765]: Accepted password for www from 188.25.126.228 port 64247 ssh2
And the last successful login attempt was followed by the password being changed. Then shortly after, the miner started.
Jun 8 19:47:27 meowset passwd: pam_unix(passwd:chauthtok): password changed for www
The moral of this story
What did I learn today? Set up ssh keys! Then in /etc/ssh/sshd_config
,
change PasswordAuthentication yes
to
PasswordAuthentication no
:)
I hope whoever that was enjoys the three cents they made today.
For anyone interested, you can download the file that was running. It's a binary. I'm stuck trying to learn more about it, but I'd love to hear it if you can. me@artur.co.