On 2021-03-02 9:31 a.m., Marcus Meissner wrote:

Let me guess the test user has password "test"?

Nope.  I'm not that dumb.  I recall the days when the computer operators would leave the default user & passwords on VAX 11/780 computers.

On 02/03/2021 15.38, James Knott wrote:

On 2021-03-02 9:27 a.m., Josef Moellers wrote:

...

On 02.03.21 15:17, James Knott wrote:

...

Today, I noticed my computer performance was very poor.  Top showed something called .dhcpd running under user test and using almost 400% of my CPU.  In ~/test, I found an executable .dhcpd.  I have not used the test account for a long time.  That .dhcpd also survived a reboot, so I'll have to determine how it's starting. You can have a look at what it's actually doing with "strace": strace -f -p $(pidof .dhcpd)

Also, have a look if it has any suspicious strings (attached;-)  ): strings .dhcpd

I have already removed the execute bits from it and allowed only root rw access.  If someone wants to examine it, I can provide the file.  It's about 3.5 MB, so I'll leave it on Google drive.

This could have only come from one of the updates that arrived yesterday, as I haven't installed any other software recently, other than OpenVPN 2.5.0, which came from https://download.opensuse.org/repositories/home:/stroeder:/network/openSUSE_... last week.

Try rpm -f /home/test/.dhcpd to find out if it came in an rpm. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 02.03.21 16:44, James Knott wrote:

On 2021-03-02 10:41 a.m., Carlos E. R. wrote:

...

rpm -f/home/test/.dhcpd

to find out if it came in an rpm.

That didn't work.  I just got a usage display.

Put a space between the '-f' and the pathname Josef -- SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nürnberg Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer

On 02.03.21 16:53, James Knott wrote:

On 2021-03-02 10:46 a.m., Josef Moellers wrote:

...

...

That didn't work.  I just got a usage display. Put a space between the '-f' and the pathname

Same thing.  I've never had to worry about a space with the options before.

Fired but forgot ... it should be "rpm -qf <pathname>" <redface> Josef -- SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nürnberg Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer

On 2021-03-02 10:49 a.m., Per Jessen wrote:

...

That didn't work.  I just got a usage display. Try "rpm -qf/home/test/.dhcpd"

rpm -qf /home/test/.dhcpd error: file /home/test/.dhcpd: No such file or directory However, I noticed it seems to be looking in whatever directory I run it from.

Per Jessen wrote:

James Knott wrote:

...

On 2021-03-02 10:49 a.m., Per Jessen wrote:

...

...

That didn't work.  I just got a usage display. Try "rpm -qf/home/test/.dhcpd"

rpm -qf /home/test/.dhcpd error: file /home/test/.dhcpd: No such file or directory

My typo -

rpm -qf /home/test/.dhpcd

Try googling ".dhpcd", there are mentions going back to 2019. Also with user 'test'. -- Per Jessen, Zürich (14.1°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland.

James Knott wrote:

On 2021-03-02 11:01 a.m., Per Jessen wrote:

...

Try googling ".dhpcd", there are mentions going back to 2019. Also with user 'test'.

I didn't see anything relevant. I even tried .dhcpd virus.

it has to be ".dhpcd", not ".dhcpd". at askubuntu, I saw people discussing malware. -- Per Jessen, Zürich (11.8°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland.

On 02/03/2021 17.37, Knurpht-openSUSE wrote:

Op dinsdag 2 maart 2021 17:34:44 CET schreef Per Jessen:

...

it has to be ".dhpcd", not ".dhcpd". at askubuntu, I saw people discussing malware. https://askubuntu.com/questions/1100467/re-what-is-dhpcd-command-and-how-to-...

But they don't know how it got installed. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 2021-03-02 11:53 a.m., Lew Wolfgang wrote:

...

I didn't see anything relevant. I even tried .dhcpd virus.

Did you try to upload the file to virustotal.com?

Marcus Meissner did.

On 3/2/21 9:33 AM, Marcus Meissner wrote:

On Tue, Mar 02, 2021 at 12:11:56PM -0500, James Knott wrote:

...

On 2021-03-02 11:53 a.m., Lew Wolfgang wrote:

...

...

I didn't see anything relevant. I even tried .dhcpd virus. Did you try to upload the file to virustotal.com? Marcus Meissner did. https://www.virustotal.com/gui/file/364c6b2dcd55840f9c121c4131024965ef44dd18...

It reports as some generic Bitcoin Miner malware.

Of course everyone knows that the infected host can never be fully trusted again, right? James:  have you tried running rkhunter on the infected host? It can be loaded via YaST if necessary. Hopefully you have off-line backups.  If I were you, after trying to determine how this happened, I'd slick the disk and reinstall everything from scratch. Regards, Lew

James Knott wrote:

...

James:  have you tried running rkhunter on the infected host? It can be loaded via YaST if necessary.

Done that.

and it found nothing ?

...

Hopefully you have off-line backups.  If I were you, after trying to determine how this happened, I'd slick the disk and reinstall everything from scratch.

I was already thinking about that.  Any chance 15.3 will be available soon?

btw, the shadow password file was changed at the same time as that .dhcpd file was created.  Since there were only 3 login password in it and I know 2 of them still work, I assume the test account password was changed.  I changed the password and set the account to not allow login.  My next step is to remove the test user entirely.

sometimes just grepping for 'ssh' in /var/log/messages will show you a login from an unknown ip address. -- Per Jessen, Zürich (9.0°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland.

On 2021-03-02 20:21, James Knott wrote:

I just realized I had ssh opened in my firewall for testing recently and forgot to close it. I have changed the password and disabled login for that account.  I will be removing the account entirely.

in /etc/ssh/sshd_config PasswordAuthentication no But first put an authorized_keys in your ~/.ssh/ Works like a charm. No scumbags for over a decade on our servers. All our servers have sshd port (on a different port than 22) open on public internet. -- /bengan

On 02/03/2021 20.21, James Knott wrote:

On 2021-03-02 1:55 p.m., Per Jessen wrote:

...

James Knott wrote:

...

sometimes just grepping for 'ssh' in /var/log/messages will show you a login from an unknown ip address.

That appears to be the way they got in.  I just realized I had ssh opened in my firewall for testing recently and forgot to close it. I have changed the password and disabled login for that account.  I will be removing the account entirely.

It came from 61.177.173.3, which is in China.

He guessed the login/pass? -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 02/03/2021 21.52, James Knott wrote:

On 2021-03-02 3:50 p.m., Carlos E. R. wrote:

...

He guessed the login/pass?

Apparently.  According to what I saw about that site, they try lots of passwords and there were lots of attemps.

And what user did they get in, root or test? -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

James Knott wrote:

On 2021-03-02 3:54 p.m., Carlos E. R. wrote:

...

...

Apparently.  According to what I saw about that site, they try lots of passwords and there were lots of attemps. And what user did they get in, root or test?

Test.

Don't feel too comfortable yet, but that is a good(!) sign. They may have been after nothing but your cpu cycles. -- Per Jessen, Zürich (5.1°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland.

On 02/03/2021 21.56, James Knott wrote:

On 2021-03-02 3:54 p.m., Carlos E. R. wrote:

...

...

Apparently.  According to what I saw about that site, they try lots of passwords and there were lots of attemps. And what user did they get in, root or test?

Test.

Ok, so the user did exist and had actually a password and they guessed it, via ssh and many attempts. Ok, that's the important information to know. They did not make "root", so the machine probably was not compromised. Check "crontab -e" (uses vi by default) to see if he created a cronjob. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 02/03/2021 22.14, James Knott wrote:

On 2021-03-02 4:12 p.m., Carlos E. R. wrote:

...

Ok, so the user did exist and had actually a password and they guessed it, via ssh and many attempts. Ok, that's the important information to know. They did not make "root", so the machine probably was not compromised.

Check "crontab -e"  (uses vi by default) to see if he created a cronjob.

Nothing there.

A systemd user timer, then? Something in /etc/cron.d/? -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 02/03/2021 22.18, James Knott wrote:

On 2021-03-02 4:17 p.m., Carlos E. R. wrote:

...

...

Nothing there. A systemd user timer, then?

Something in/etc/cron.d/?

No.

Then maybe they connected via ssh to start it again? Doesn't make sense. A user systemd service? A desktop autostart? No, that would be a different user. Oh, I meant "crontab -e - u test" (you must be root). -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 02/03/2021 22.41, James Knott wrote:

On 2021-03-02 4:24 p.m., Carlos E. R. wrote:

...

Then maybe they connected via ssh to start it again? Doesn't make sense.

A user systemd service?

A desktop autostart? No, that would be a different user.

Oh, I meant "crontab -e - u test" (you must be root).

2 * * * * /home/test/.dhpcd -o ca.minexmr.com:4444 -B >/dev/null 2>/dev/null

However, I cleared the executable bits for that file.

Yes, sure, but it was important for us to know that they set a crontab entry and which.

btw, no space between - and u.

Yeah, finger error :-) You could also firewall ca.minexmr.com or in your dns setup. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 02/03/2021 23.04, James Knott wrote:

On 2021-03-02 4:57 p.m., Carlos E. R. wrote:

...

You could also firewall ca.minexmr.com or in your dns setup.

I blocked the entire /8 network that it came from.  Also, I closed port 22, which I had open for testing.

All that do not have effect on ca.minexmr.com, because they would be outgoing connections from your computer. In fact, it would be good for everybody. Even the FBI. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 3/2/21 1:21 PM, James Knott wrote:

On 2021-03-02 1:55 p.m., Per Jessen wrote:

...

James Knott wrote:

...

...

James:  have you tried running rkhunter on the infected host? It can be loaded via YaST if necessary. Done that. and it found nothing ?

I found several warnings.  I'm checking them now.

...

...

...

Hopefully you have off-line backups.  If I were you, after trying to determine how this happened, I'd slick the disk and reinstall everything from scratch. I was already thinking about that.  Any chance 15.3 will be available soon?

btw, the shadow password file was changed at the same time as that .dhcpd file was created.  Since there were only 3 login password in it and I know 2 of them still work, I assume the test account password was changed.  I changed the password and set the account to not allow login.  My next step is to remove the test user entirely. sometimes just grepping for 'ssh' in /var/log/messages will show you a login from an unknown ip address.

That appears to be the way they got in.  I just realized I had ssh opened in my firewall for testing recently and forgot to close it. I have changed the password and disabled login for that account.  I will be removing the account entirely.

It came from 61.177.173.3, which is in China.

Would have never gotten it... iptables is your friend: -A INPUT -s 61.0.0.0/8 -j DROP -- David C. Rankin, J.D.,P.E.

...

...

...

...

"JK" == James Knott writes:

JK> On 2021-03-02 11:36 p.m., David C. Rankin wrote:

...

Would have never gotten it... iptables is your friend:

-A INPUT -s 61.0.0.0/8 -j DROP

JK> My firewall is pfsense and I have blocked the entire 61.0.0.0 /8 network. Maybe better to use geoip based filtering and block the whole China, unless you have something to do.

...

...

...

...

"Carlos" == Carlos E R writes:

Carlos> On 03/03/2021 13.16, Togan Muftuoglu wrote:

...

...

...

...

...

> "JK" == James Knott writes:

JK> On 2021-03-02 11:36 p.m., David C. Rankin wrote:

...

...

...

Would have never gotten it... iptables is your friend:

-A INPUT -s 61.0.0.0/8 -j DROP

JK> My firewall is pfsense and I have blocked the entire 61.0.0.0 /8 network.

...

Maybe better to use geoip based filtering and block the whole China, unless you have something to do.

Carlos> China is not the only country with hackers. When one understands the concept then one can block wtf countries. Try not so hard to read everything literally. ;)

On 2021-03-03 7:24 a.m., Togan Muftuoglu wrote:

...

...

Maybe better to use geoip based filtering and block the whole China, unless you have something to do.

Carlos> China is not the only country with hackers.

When one understands the concept then one can block wtf countries. Try not so hard to read everything literally.;)

Maybe I can allow only Canada or perhaps just Ontario.

On 3/3/21 4:04 AM, James Knott wrote:

On 2021-03-02 11:36 p.m., David C. Rankin wrote:

...

Would have never gotten it... iptables is your friend:

  -A INPUT -s 61.0.0.0/8 -j DROP

My firewall is pfsense and I have blocked the entire 61.0.0.0 /8 network.

But China has many networks.  Then you have the proxy issue... Why not block everything by default if you're not running a pubic server?   Just open needed ports to specific addresses, not even subnets. Regards, Lew

On 02/03/2021 19.46, James Knott wrote:

btw, the shadow password file was changed at the same time as that .dhcpd file was created.  Since there were only 3 login password in it and I know 2 of them still work, I assume the test account password was changed.  I changed the password and set the account to not allow login.  My next step is to remove the test user entirely.

Correlate that point in time with all logs. What were you doing at the time. It is very important to find out how that thing got in. To create /home/test the first time they need root access of some kind, so anything you do to disable that account is pointless. Then you say that the thing would restart itself. I have read on google that it restarts every halfhour, so there will be some cronjob entry somewhere. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, March 2, 2021 10:46 AM, James Knott wrote:

btw, the shadow password file was changed at the same time as that .dhcpd file was created.  Since there were only 3 login password in it and I know 2 of them still work, I assume the test account password was changed.

Game over. He's rooted you. Wish you'd respond to my questions so it may help the rest of us.

On 2021-03-02 3:53 p.m., Carlos E. R. wrote:

Maybe not, if he directly loged in as "test". Should be in the log. If he were root, he would have deleted the traces.

It was test.  It's been over an hour since I rebooted and no sign of .dhcpd yet.

On 2021-03-02 4:18 p.m., Carlos E. R. wrote:

...

It was test.  It's been over an hour since I rebooted and no sign of .dhcpd yet. Check the log to see if something tried to start it and failed (because it is not executable).

It's not executable now, because I cleared the execute bits.  It was running this morning.

On 2021-03-02 4:25 p.m., Carlos E. R. wrote:

...

It's not executable now, because I cleared the execute bits.  It was running this morning. I know you did, which is precisely why I ask you to scan the logs after that instant to see if something tried and failed.

Well, I have removed that crontab line, so it won't be trying to start again.

On 2021-03-02 4:46 p.m., Carlos E.R. wrote:

...

Well, I have removed that crontab line, so it won't be trying to start again. Ah, you removed a crontab line! You did not tell us that. What line, where exactly?

2 * * * * /home/test/.dhpcd -o ca.minexmr.com:4444 -B >/dev/null 2>/dev/null I mentioned it in another message to the list.  It was in the crontab for the user test.

On 03/03/2021 07.48, Per Jessen wrote:

James Knott wrote:

...

On 2021-03-02 4:46 p.m., Carlos E.R. wrote:

...

...

Well, I have removed that crontab line, so it won't be trying to start again. Ah, you removed a crontab line! You did not tell us that. What line, where exactly?

2 * * * * /home/test/.dhpcd -o ca.minexmr.com:4444 -B >/dev/null ^^^^^^^^^^^^^^^^^^^

That is a useful hint - it led me to this:

https://blogs.akamai.com/sitr/2020/04/a-brief-history-of-a-rootable-docker-i...

As also expected when this project first started, the docker image was a prime target for crypto-mining malware. An installation script is run via SSH that monitors the process tree with "top" - a process monitoring tool - checking for when the malware has successfully been executed. Once the process is detected, in this case it is "dhcpcd", the malware adds two new users to the system - test and test1. It changes the root password to ""(blank) and creates entries in cron to start the mining software up after a reboot. It also attempts to set the immutable bit (+i) on various files including /etc/shadow and the malware binary in /sbin/dhcpcd.

The malware also adds a .ssh key to /root/.ssh/authorized_keys:

ssh-rsa ..........

The mining software reports as XMRig v5.2.0.

That attack is different, they start with root powers. Much more dangerous. Interesting read that link, thanks :-) -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 03/03/2021 14.42, James Knott wrote:

On 2021-03-03 1:48 a.m., Per Jessen wrote:

...

...

2 * * * * /home/test/.dhpcd -o ca.minexmr.com:4444 -B >/dev/null                                   ^^^^^^^^^^^^^^^^^^^

That is a useful hint - it led me to this:

https://blogs.akamai.com/sitr/2020/04/a-brief-history-of-a-rootable-docker-i...

As also expected when this project first started, the docker image was a prime target for crypto-mining malware. An installation script is run via SSH that monitors the process tree with "top" - a process monitoring tool - checking for when the malware has successfully been executed. Once the process is detected, in this case it is "dhcpcd", the malware adds two new users to the system - test and test1. It changes the root password to ""(blank) and creates entries in cron to start the mining software up after a reboot. It also attempts to set the immutable bit (+i) on various files including /etc/shadow and the malware binary in /sbin/dhcpcd.

I haven't seen that.  For example: -rw-r----- 1 root shadow 1060 Mar  2 17:00 /etc/shadow

...

The malware also adds a .ssh key to /root/.ssh/authorized_keys: Nothing there.

Do you allow root login via ssh? Did they attempt to login as root (check your logs)? If "yes" then perhaps your root's password is harder to guess. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, March 2, 2021 10:48 PM, Per Jessen > > 2 * * * * /home/test/.dhpcd -o ca.minexmr.com:4444 -B >/dev/null

^^^^^^^^^^^^^^^^^^^

That is a useful hint - it led me to this:

https://blogs.akamai.com/sitr/2020/04/a-brief-history-of-a-rootable-docker-i...

As also expected when this project first started, the docker image was a prime target for crypto-mining malware. An installation script is run via SSH that monitors the process tree with "top" - a process monitoring tool - checking for when the malware has successfully been executed. Once the process is detected, in this case it is "dhcpcd", the malware adds two new users to the system - test and test1. It changes the root password to ""(blank) and creates entries in cron to start the mining software up after a reboot. It also attempts to set the immutable bit (+i) on various files including /etc/shadow and the malware binary in /sbin/dhcpcd.

The malware also adds a .ssh key to /root/.ssh/authorized_keys:

ssh-rsa ..........

The mining software reports as XMRig v5.2.0.

Per Jessen, Zürich (5.5°C)

This is all why I have switched over to an immutable OS. (MicroOS) Every security layer counts.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, March 2, 2021 1:42 PM, jdd@dodin.org wrote:

Le 02/03/2021 à 20:07, colony.three@protonmail.ch a écrit :

...

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Tuesday, March 2, 2021 10:46 AM, James Knott james.knott@jknott.net wrote:

...

btw, the shadow password file was changed at the same time as that .dhcpd file was created.  Since there were only 3 login password in it and I know 2 of them still work, I assume the test account password was changed.

Game over. He's rooted you. Wish you'd respond to my questions so it may help the rest of us.

user test can change his own passwd

jdd

Sheesh. I give up.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Wednesday, March 3, 2021 2:00 AM, Bengt Gördén wrote:

On 2021-03-03 06:29, colony.three@protonmail.ch wrote:

...

Sheesh. I give up.

You have written three messages. Different recipients each time. It's not easy to figure out who you ask. In your first message to Jonathan Arsenault you ask three questions. In your second message to James Knott you ask him to answer your question to Jonathan. In your third message to Jean-Daniel Dodin you give up. Are you a bot?

/bengan

'Am I a 'bot'... That's an amusing conclusion. Do I sound like a 'bot? If so, you have not been understanding my questions. However your most fundamental misunderstanding... would. a. 'bot. give. up.... Maybe something to think about. No, not different recipients. The listserv each time (unless one time I slipped up). No, I have just been asking questions which get to the heart of this, but no one seems to understand them. Everyone has zeroed on on SSH being the vector, which I think is nonsense unless the OP is an idiot, which he professes not to be. Now we are left -without- the vital information needed to fully investigate this... and instead a pile-on game of who is smarter. Another dreary race to the bottom. I am just a refugee from CentOS. But my experiences on #suse and here have been disappointing.

On 2021-03-02 11:33 p.m., David C. Rankin wrote:

The link Gertjan provide shows the final solution for the Ubuntu user:

sudo deluser --system --remove-all-files test

I used the Yast User and Group Management to remove the user and files.  Is that any different?

On 05/03/2021 05.26, David C. Rankin wrote:

On 3/3/21 6:03 AM, James Knott wrote:

...

On 2021-03-02 11:33 p.m., David C. Rankin wrote:

...

The link Gertjan provide shows the final solution for the Ubuntu user:

   sudo deluser --system --remove-all-files test

I used the Yast User and Group Management to remove the user and files.  Is that any different?

Late-reply,

No, but make sure you look for a check box that says "remove all user file" or something similar.

Notice that this means "/home/[user]", not any other file in the filesystem that may belong to the user. It doesn't do a search. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 05/03/2021 14.45, James Knott wrote:

On 2021-03-05 7:47 a.m., Carlos E. R. wrote:

...

...

   No, but make sure you look for a check box that says "remove all user file" or something similar. Notice that this means "/home/[user]", not any other file in the filesystem that may belong to the user. It doesn't do a search.

Where else is a user allowed to write?

/tmp, for instance. There is a "find" concoction that will find any file belonging to a certain user, but I don't have it handy -- meaning I did not find the note :-) -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 2021-03-05 9:02 a.m., Per Jessen wrote:

...

There is a "find" concoction that will find any file belonging to a certain user, but I don't have it handy -- meaning I did not find the note:-) Try this: find / -type f -user carlos

-user carlos didn't work. ;-) Actually, since I have deleted the user, I had to search on UID and came up with this: /tmp/sora.arm5 /tmp/sora.x86 /tmp/sora.m68k /tmp/sora.mips /tmp/sora.ppc /tmp/sora.sh4 /tmp/sora.arm4 /tmp/uYtea.x86 /tmp/sora.arm6 /tmp/sora.mpsl /tmp/uYtea.x86.3 /tmp/uYtea.x86.1 /tmp/uYtea.x86.4 /tmp/uYtea.x86.2 /tmp/sora.arm7 linux:~ # ls -l /tmp/sora* -rwxr-xr-x 1 1001 users   212 Feb 28 15:29 /tmp/sora.arm4 -rwxr-xr-x 1 1001 users 22140 Feb 28 15:29 /tmp/sora.arm5 -rwxr-xr-x 1 1001 users 29448 Feb 28 15:29 /tmp/sora.arm6 -rwxr-xr-x 1 1001 users 48664 Feb 28 15:29 /tmp/sora.arm7 -rwxr-xr-x 1 1001 users 53072 Feb 28 15:29 /tmp/sora.m68k -rwxr-xr-x 1 1001 users 26204 Feb 28 15:29 /tmp/sora.mips -rwxr-xr-x 1 1001 users 27220 Feb 28 15:29 /tmp/sora.mpsl -rwxr-xr-x 1 1001 users 23928 Feb 28 15:29 /tmp/sora.ppc -rwxr-xr-x 1 1001 users 51616 Feb 28 15:29 /tmp/sora.sh4 -rwxr-xr-x 1 1001 users 24760 Feb 28 15:29 /tmp/sora.x86 linux:~ # ls -l /tmp/uYtea* -rwxr-xr-x 1 1001 users 63540 Mar  1 16:44 /tmp/uYtea.x86 -rwxr-xr-x 1 1001 users 63540 Feb 28 21:18 /tmp/uYtea.x86.1 -rwxr-xr-x 1 1001 users 63540 Feb 28 21:18 /tmp/uYtea.x86.2 -rwxr-xr-x 1 1001 users 63540 Feb 28 21:18 /tmp/uYtea.x86.3 -rwxr-xr-x 1 1001 users 63540 Feb 28 21:18 /tmp/uYtea.x86.4 I have deleted all those.

On 2021-03-05 9:39 a.m., Josef Moellers wrote:

...

Where else is a user allowed to write? The user doesn't have to write the file himself, it can also be created on his behalf, eg mailboxes, crontabs, atjobs, to name but a few.

I have deleted all files with that user's UID.

Having read through this long conversation, and, yes, coming in late (sorry for that), I seem to be missing something. How did the hacker get in? Was it simply a brute force at the SSH login? On March 5, 2021 2:41:35 PM UTC, James Knott wrote:

On 2021-03-05 9:39 a.m., Josef Moellers wrote:

...

...

Where else is a user allowed to write? The user doesn't have to write the file himself, it can also be created on his behalf, eg mailboxes, crontabs, atjobs, to name but a few.

I have deleted all files with that user's UID.

On 2021-03-14 10:43 a.m., ken wrote:

Having read through this long conversation, and, yes, coming in late (sorry for that), I seem to be missing something. How did the hacker get in? Was it simply a brute force at the SSH login?

It appears to be.  I had been doing some testing and accidentally left ssh open through my firewall.  I normally use OpenVPN to connect to my system.

Drop the link I want to throw that thing at IDA, sounds like cryptominer. Do you have bind/smb enabled with no master? On Tue, Mar 2, 2021 at 10:59 AM Per Jessen wrote:

James Knott wrote:

...

On 2021-03-02 10:49 a.m., Per Jessen wrote:

...

...

That didn't work. I just got a usage display. Try "rpm -qf/home/test/.dhcpd"

rpm -qf /home/test/.dhcpd error: file /home/test/.dhcpd: No such file or directory

My typo -

rpm -qf /home/test/.dhpcd

-- Per Jessen, Zürich (14.0°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland.

On 02.03.2021 19:07, James Knott wrote:

On 2021-03-02 11:04 a.m., Jonathan Arsenault wrote:

...

Drop the link I want to throw that thing at IDA, sounds like cryptominer. Do you have bind/smb enabled with no master?

The link doesn't work.  The file gets identified as a virus, which only the owner can download.  If you have some other safe way to provide the file I can try that.

Password protected zip or any other archive format.

On 02/03/2021 16.57, James Knott wrote:

On 2021-03-02 10:49 a.m., Per Jessen wrote:

...

...

That didn't work.  I just got a usage display. Try "rpm -qf/home/test/.dhcpd"

rpm -qf /home/test/.dhcpd error: file /home/test/.dhcpd: No such file or directory

Is that the actual path to the malware? Or have you deleted it? If you have deleted it, create it again with touch, then repeat the rpm query.

However, I noticed it seems to be looking in whatever directory I run it from.

-- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 02/03/2021 17.58, James Knott wrote:

On 2021-03-02 11:55 a.m., Carlos E. R. wrote:

...

...

error: file/home/test/.dhcpd: No such file or directory Is that the actual path to the malware? Or have you deleted it? If you have deleted it, create it again with touch, then repeat the rpm query.

That is where I found it.

Weird. I would have expected this: cer@Telcontar:~> touch .dhcpd cer@Telcontar:~> rpm -qf /home/cer/.dhcpd file /home/cer/.dhcpd is not owned by any package cer@Telcontar:~> Are you running rpm as user or as root? Perhaps you do not have permissions to read the file. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 2021-03-02 10:41 a.m., Carlos E. R. wrote:

rpm -f/home/test/.dhcpd

to find out if it came in an rpm.

BTW, I have determined it is a virus.

On 2021-03-02 11:58 a.m., Carlos E.R. wrote:

...

BTW, I have determined it is a virus. It is malware. To be a virus it has to match certain criteria that I don't see here. I think it is a trojan.

Whichever.  That's the message I got when I tried to download it from Google drive onto Windows.  Also, I purposely used malware in the message title, as I didn't know what it was.

On 3/2/21 4:03 PM, Carlos E.R. wrote:

Thus, no vulnerability in the system, no compromised update. The root account was not compromised.

Fortunately!  I wonder if they tried to get root?

Mitigations for the future:

- Don't open ssh port 22, use a high port on a strange number (not 50000, for instance).

Changing ports might help a bit, but dedicated hackers can discover moved ports easily.  Consider shodan.io for example.  They have crawlers that map all ports on all servers.

- in /etc/ssh/sshd_config set "PasswordAuthentication no" after adding appropriate keys to ~/.ssh/

The file is ~/.ssh/authorized_keys by default. Setting up something like fail2ban would also be a good thing. Regards, Lew

Lew Wolfgang wrote:

On 3/2/21 4:03 PM, Carlos E.R. wrote:

...

Thus, no vulnerability in the system, no compromised update. The root account was not compromised.

Fortunately!  I wonder if they tried to get root?

...

Mitigations for the future:

- Don't open ssh port 22, use a high port on a strange number (not 50000, for instance).

Changing ports might help a bit, but dedicated hackers can discover moved ports easily. 

Absolutely. It is only a matter of time. If ssh is to be open publicly, setting up public key authentication is the way to go.

Setting up something like fail2ban would also be a good thing.

Possibly - for 14-15 years, our firewall has automatically blocked new external accesses after a certain rate per minute, both for ssh, sip, and ftp traffic. That has worked well, but hackers have become very patient - brute force attacks was yesterday. Today it is slow, thorough, distributed - maybe 50 machines slowly trying out passwords, once a minute, one machine after the other. Such slow, patient attacks usually don't trigger any traps or fail2ban. -- Per Jessen, Zürich (5.8°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland.

On 3/3/21 3:52 AM, Carlos E. R. wrote:

...

...

...

Mitigations for the future:

- Don't open ssh port 22, use a high port on a strange number (not 50000, for instance). Changing ports might help a bit, but dedicated hackers can discover moved ports easily. Absolutely. It is only a matter of time. They have not found mine...:-)

How are you so sure?  Maybe they're using an nmap "stealth" scan and know about your open ports?  Maybe they just haven't tried to connect yet? Regards, Lew

Carlos E. R. wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Wednesday, 2021-03-03 at 08:04 -0800, Lew Wolfgang wrote:

...

On 3/3/21 3:52 AM, Carlos E. R. wrote:

Mitigations for the future:

- Don't open ssh port 22, use a high port on a strange number (not 50000, for instance).

Changing ports might help a bit, but dedicated hackers can discover moved ports easily.

Absolutely. It is only a matter of time.

They have not found mine... :-)

How are you so sure?  Maybe they're using an nmap "stealth" scan and know about your open ports?  Maybe they just haven't tried to connect yet?

Because an ssh attempt is logged.

Maybe re-read what Lew wrote :-)

The only strange entries are some like this:

<4.6> 2020-03-07T09:39:57.547577+01:00 Isengard sshd 6958 - - Bad protocol version identification '\003' from 45.148.121.4 port 63970 [snip]

Someone knows about an open port. -- Per Jessen, Zürich (8.2°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland.

Lew Wolfgang wrote:

Well, that you have any entries at all from sshd means that something is poking at the port it's listening on, right? 

I would say so.

I just checked and I've got 942 entries like the following in the past four days, most from different IP's.

Oh yeah, the threat is certainly real. In February, our external (rented) servers had about 4-5000 such attempts, each. All configured for public key only, even so listening on a high port. Always with big time gaps, 5-10-15 minutes - true brute force was yesteryear. 44'244 attempts over twelve specific servers, 14'963 unique ipv4 addresses from 141 countries. -- Per Jessen, Zürich (6.2°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland.

Carlos E. R. wrote:

On 03/03/2021 20.24, Per Jessen wrote:

...

Lew Wolfgang wrote:

...

Well, that you have any entries at all from sshd means that something is poking at the port it's listening on, right?

I would say so.

...

I just checked and I've got 942 entries like the following in the past four days, most from different IP's.

Oh yeah, the threat is certainly real. In February, our external (rented) servers had about 4-5000 such attempts, each. All configured for public key only, even so listening on a high port. Always with big time gaps, 5-10-15 minutes - true brute force was yesteryear.

44'244 attempts over twelve specific servers, 14'963 unique ipv4 addresses from 141 countries.

What do you use to analyze that? Ie, how do you know?

The plain old logfile - /var/log/messages. sshd will log failed login attempts. To analyze it, I grep through the files, then load the data into a database, makes it easier to query. -- Per Jessen, Zürich (5.6°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland.

Carlos E.R. wrote:

A tool that would collect the failed connection attempts, then would automatically search for those IPs in the logs, would be nice. Cross referencing.

I don't think I quite understand. Finding the failed connection attempts and the origin IP-addresses is easy, but which logs do you then want to search? -- Per Jessen, Zürich (12.2°C)

Carlos E. R. wrote:

On 04/03/2021 10.53, Per Jessen wrote:

...

Carlos E.R. wrote:

...

A tool that would collect the failed connection attempts, then would automatically search for those IPs in the logs, would be nice. Cross referencing.

I don't think I quite understand. Finding the failed connection attempts and the origin IP-addresses is easy, but which logs do you then want to search?

/var/log/messages, for instance. Locate any other occurrence of the same IP, possibly in a time range, to find out if that IP that tried to connect to something else. If you have running services that use other logs, like apache, then read those, too.

Oh okay. I guess it might be worth a try, it's easily done if you load the extracted data into a database.

If the machine is external, not NATted like mine, it will receive scans directly. Thus, suppose you get a failed connect attempt at some service. The tool registers that IP, and finds out that he is trying other services at one hour intervals. Well, the tool would then detect slow scans.

Well, the analysis already does, but after the fact. I don't know what other services might be open to an attack though. -- Per Jessen, Zürich (13.2°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland.

On 03/03/2021 18.33, Lew Wolfgang wrote:

On 3/3/21 8:42 AM, Carlos E. R. wrote:

...

On Wednesday, 2021-03-03 at 08:04 -0800, Lew Wolfgang wrote:

...

On 3/3/21 3:52 AM, Carlos E. R. wrote:

...

Mitigations for the future:

...

   - Don't open ssh port 22, use a high port on a strange number (not 50000, for instance).

...

Changing ports might help a bit, but dedicated hackers can discover moved ports easily.

...

Absolutely. It is only a matter of time. They have not found mine... :-)

...

How are you so sure?  Maybe they're using an nmap "stealth" scan and know about your open ports?  Maybe they just haven't tried to connect yet?

Because an ssh attempt is logged.

grep shd  /var/log/messages*z | grep port | egrep -v "192.168" | less

But nmap stealth scans don't leave log entries.   I'm sure that someone has swept all your ports without you knowing about it.

All the ports of the external router, it is possible and I will not know because my router says nothing. But that is not where ssh is. If something attempts to actually connect to my ssh machine at the correct port, the ssh machine logs the attempt and the failure or success. And that (at the time I last looked) had not happened. attempts on other ports will not reach the ssh daemon on another machine...

...

The only strange entries are some like this:

<4.6> 2020-03-21T17:05:21.639712+01:00 Isengard sshd 25854 - -  Bad protocol version identification '\003' from 194.61.26.163 port 1145 <4.6> 2020-04-07T14:35:53.062798+02:00 Isengard sshd 31626 - -  Bad protocol version identification '\003' from 80.89.234.58 port 913

I have no idea what that is.

These are the last:

<4.6> 2021-01-02T02:58:24.470179+01:00 Isengard sshd 6839 - -  Did not receive identification string from 194.36.85.126 port 12770 <4.6> 2021-01-05T19:19:46.729422+01:00 Isengard sshd 14602 - -  Bad protocol version identification '\003' from 89.248.165.25 port 63861

(all are from Europe, not China ;-) )

Well, that you have any entries at all from sshd means that something is poking at the port it's listening on, right?  Maybe these are shodan scans?

I don't know what they are. The entries list an IP and a port. At that port there is nothing listening on my machine, so maybe it is the egress port. This is just the thing: https://unix.stackexchange.com/questions/573275/what-kind-of-an-sshd-attack-... Basically, they are saying that they are scanning hhtp servers, and when they hit by chance the sshd NATted port, the sshd server sees it and complains. The other one: "Did not receive identification string from" seems to happen from providers of virtual servers checking if you are alive - but this is not a virtual server. It can be simply an aborted connection attempt (nmap) or an attempt at a known vulnerability, documented in "/etc/fail2ban/filter.d/sshd-ddos.conf" https://serverfault.com/questions/483852/sshd-log-full-of-did-not-receive-id...

Another thing you might be able to do is use the "AllowUsers" parameter in /etc/ssh/sshd_config.  Any login attempts to users not listed are rejected by sshd.   You can even list an IP address, as in "wolfgang@w.x.t.z", or even "wolfgang@w.x.y.0/24" as examples.

No, I can not list an IP address, as I do not know in advance what address I would use to connect from. The daemon is using publickkey, so I don't think they can guess easily the correct user, but yes, you are right, I could limit them. Problem is limiting from outside and not from inside. Ah, of course, connection is impossible if the attempted user has no publickey defined, which in effect is the same as listing the users.

I just checked and I've got 942 entries like the following in the past four days, most from different IP's.

"sshd[6236]: User root from 220.248.95.178 not allowed because not listed in AllowUsers"

This was from China, btw.

Regards, Lew

-- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2021-03-03 at 12:46 -0800, Lew Wolfgang wrote:

On 3/3/21 11:47 AM, Carlos E. R. wrote:

...

All the ports of the external router, it is possible and I will not know because my router says nothing. But that is not where ssh is. If something attempts to actually connect to my ssh machine at the correct port, the ssh machine logs the attempt and the failure or success. And that (at the time I last looked) had not happened.

But something woke up your ssh daemon, you showed us the log entries.  I'd guess that you're seeing aborted login attempts that don't get to the point of supplying a userid.  Maybe our Chinese friends are just sniffing around?

No, not really. The "Bad protocol version identification" are, apparently, attempts to find an http server on other ports, and hitting instead the ssh port. There is no attempt later to login on ssh. None of those comes from China: several come from the Netherlands :-p This is the one that tries ssh: Did not receive identification string from 194.36.85.126 port 12770 and does not try to login because that error is produced earlier in the negotiation. It is rather a probe of the existence and alivenes of an ssh server (and my IP is not fixed, so it doesn't matter). There are few in the entire log: cer@Isengard:~> zgrep sshd /var/log/messages*z | grep "Did not receive identification string from" /var/log/messages-20200909.xz:<4.6> 2020-07-31T16:16:13.521585+02:00 Isengard sshd 1137 - - Did not receive identification string from 185.176.27.190 port 62000 /var/log/messages-20200909.xz:<4.6> 2020-08-01T19:27:06.749153+02:00 Isengard sshd 11473 - - Did not receive identification string from 185.176.27.86 port 62000 /var/log/messages-20200909.xz:<4.6> 2020-08-15T18:50:07.476866+02:00 Isengard sshd 4601 - - Did not receive identification string from 142.93.116.72 port 51906 /var/log/messages-20201025.xz:<4.6> 2020-10-06T10:56:30.354699+02:00 Isengard sshd 30444 - - Did not receive identification string from 185.193.90.86 port 62000 /var/log/messages-20201025.xz:<4.6> 2020-10-10T19:09:05.197476+02:00 Isengard sshd 30599 - - Did not receive identification string from 80.82.65.74 port 34970 /var/log/messages-20201025.xz:<4.6> 2020-10-10T19:09:05.415243+02:00 Isengard sshd 30601 - - Did not receive identification string from 80.82.65.74 port 37092 /var/log/messages-20201025.xz:<4.6> 2020-10-10T19:09:05.637049+02:00 Isengard sshd 30602 - - Did not receive identification string from 80.82.65.74 port 39174 /var/log/messages-20210106.xz:<4.6> 2020-12-08T00:57:49.171469+01:00 Isengard sshd 20188 - - Did not receive identification string from 80.82.65.74 port 35436 /var/log/messages-20210106.xz:<4.6> 2020-12-08T00:57:49.392584+01:00 Isengard sshd 20189 - - Did not receive identification string from 80.82.65.74 port 38132 /var/log/messages-20210106.xz:<4.6> 2020-12-08T00:57:49.623665+01:00 Isengard sshd 20190 - - Did not receive identification string from 80.82.65.74 port 40424 /var/log/messages-20210106.xz:<4.6> 2021-01-02T02:58:24.470179+01:00 Isengard sshd 6839 - - Did not receive identification string from 194.36.85.126 port 12770 cer@Isengard:~> Two from Russia, one from the USA, another from Turkey. And this one is interesting: cer@Isengard:~> host 80.82.65.74 74.65.82.80.in-addr.arpa domain name pointer scanner.openportstats.com. cer@Isengard:~> That one might be me requesting a scanner on myself. Possibly there might be a reference to this in some mail list at those days, but I have not looked :-) - -- Cheers, Carlos E. R. (from openSUSE 15.2 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iHIEARECADIWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCYEAHChQccm9iaW4ubGlz dGFzQGdteC5lcwAKCRC1MxgcbY1H1b0OAKCQJGog5j3/cz6D3YA3UnIlwo2UrwCg iJvf12XibDF/y+xMciZTk0BOZIM= =cG58 -----END PGP SIGNATURE-----

On 03/03/2021 23.52, Lew Wolfgang wrote:

On 3/3/21 2:00 PM, Carlos E. R. wrote:

...

cer@Isengard:~> host 80.82.65.74 74.65.82.80.in-addr.arpa domain name pointer scanner.openportstats.com. cer@Isengard:~>

That one might be me requesting a scanner on myself. Possibly there might be a reference to this in some mail list at those days, but I have not looked :-)

Try going to openportstats.com and enter your router's wan IP address, then let us know what you see.

"No results" They don't do an actual scan when I ask, but look on their database - that is useless, the IP is dynamic, so they could have scanned me or any other Telefónica client.

They couldn't find anything on my home router, but they have lots of info on the open ports of my server.

BTW, they don't need permission to scan you.

Well... I'm not sure about that. Maybe not in the country they are based, but they are trespassing on other countries and jurisdictions. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 04/03/2021 05.22, Lew Wolfgang wrote:

On 3/3/21 5:05 PM, Carlos E.R. wrote:

...

On 03/03/2021 23.52, Lew Wolfgang wrote:

...

...

cer@Isengard:~> host 80.82.65.74 74.65.82.80.in-addr.arpa domain name pointer scanner.openportstats.com. cer@Isengard:~>

That one might be me requesting a scanner on myself. Possibly there might be a reference to this in some mail list at those days, but I have

not looked :-) Try going to openportstats.com and enter your router's wan IP address,

On 3/3/21 2:00 PM, Carlos E. R. wrote: then let us know what you see. "No results"

They don't do an actual scan when I ask, but look on their database - that is useless, the IP is dynamic, so they could have scanned me or any other Telefónica client.

How often does your IP change?  I've seen mine not change for years.

Any time the router restarts. The other day the router was working but the connection was down, so I had to power cycle the entire setup, and the IP changed again. For instance, this year it has changed four times. By the way. The other day I could not reach my own apache server (on a dynamic name). After many tests, I found that it failed with Firefox, but worked with Chrome. Whaaat? Apparently, Firefox (on phone) was using its own dns over http service and hitting the IP I had a week ago, while Chrome was hitting the current IP. It was a scare, because my testing revealed open ports in "my" router that shouldn't be. Well, they were somebody else's router (in the same ISP, same router brand). Meaning that the the DNS over http service of Firefox was using stale DNS information.

...

...

They couldn't find anything on my home router, but they have lots of info on the open ports of my server.

BTW, they don't need permission to scan you. Well... I'm not sure about that. Maybe not in the country they are based, but they are trespassing on other countries and jurisdictions.

The commercial scan services collect only publicly available data, like which ports are open, and banners from open ports.  You or me scanning would probably violate our use agreements with our ISP's, but that wouldn't be a problem for a business who's their own ISP.    Even if a jurisdiction had a specific law regarding scanning, I doubt if it could, or would, be enforced.  I could fat-finger a url and accidentally scan someone.  Would an arrest warrant for my IP address be issued? Shodan.io never asked me for permission.

An arrest no... a reprimand more likely. And very difficult to enforce, practically impossible.

And of course botnets also scan for open ports, and not just the obvious ones.  I just checked on my underused server with a direct and static connection, and over a 75-minute period I had 869 ports scanned, 637 of them were unique ephemeral high-numbered ports.  Botnets don't care about laws, and if your IP is static for very long, your hidden ssh port could be exposed.

It can, certainly. Which is why I also use publickey. I can not detect scans, my router is too dumb for that. I can only see them if they hit directly the right port, as you saw in my logs. And it happens sparsely. I thought it was never, because the last time I studied the logs I did not see them. Having a dynamic IP helps. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 04/03/2021 14.36, James Knott wrote:

On 2021-03-04 7:31 a.m., Carlos E. R. wrote:

...

...

How often does your IP change?  I've seen mine not change for years. Any time the router restarts. The other day the router was working but the connection was down, so I had to power cycle the entire setup, and the IP changed again.

WOW!  Mine changes so seldom it's virtually static.  Also, my host name is based on my firewall & modem MAC addresses, so it doesn't change unless I change hardware.  My IPv6 prefix is also consistent.  It took replacing my firewall computer & fresh pfsense install to make it change.

My ISP is capable of claiming they do it for backward compatibility with ADSL and modems :-p I suspect they will do the same when they implement IPv6. I would even bet on it, except that I never bet :-) -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 04/03/2021 14.12, Per Jessen wrote:

Lew Wolfgang wrote:

...

And of course botnets also scan for open ports, and not just the obvious ones.  I just checked on my underused server with a direct and static connection, and over a 75-minute period I had 869 ports scanned, 637 of them were unique ephemeral high-numbered ports.  Botnets don't care about laws, and if your IP is static for very long, your hidden ssh port could be exposed.

Yep, I can assure you they will be. Even if we use public key authentication, we still run ssh on a high port - on one server installed in 2015, there were no ssh attacks until 7 February 2021. Since 2907 login attempts.

On another, also installed in 2015, attempts started 1 November 2020, since then 136680.

Static IPs? We might have to go one step further: migrate the port randomly and periodically. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 04/03/2021 14.33, Per Jessen wrote:

Carlos E. R. wrote:

...

On 04/03/2021 14.12, Per Jessen wrote:

...

Lew Wolfgang wrote:

...

And of course botnets also scan for open ports, and not just the obvious ones.  I just checked on my underused server with a direct and static connection, and over a 75-minute period I had 869 ports scanned, 637 of them were unique ephemeral high-numbered ports. Botnets don't care about laws, and if your IP is static for very long, your hidden ssh port could be exposed.

Yep, I can assure you they will be. Even if we use public key authentication, we still run ssh on a high port - on one server installed in 2015, there were no ssh attacks until 7 February 2021. Since 2907 login attempts.

On another, also installed in 2015, attempts started 1 November 2020, since then 136680.

Static IPs?

They are servers, so yes.

...

We might have to go one step further: migrate the port randomly and periodically.

As I have been trying to explain, it does not add anything. Just use public key auth.

On my case, it does help. Hits are very rare. Yes, of course I use public key auth, since day one. Changing the port periodically just reduces noise in the logs. Would be akin to changing the password periodically :-) -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 04/03/2021 17.49, Lew Wolfgang wrote:

On 3/4/21 5:26 AM, Carlos E. R. wrote:

...

On 04/03/2021 14.12, Per Jessen wrote:

...

Lew Wolfgang wrote:

...

And of course botnets also scan for open ports, and not just the obvious ones.  I just checked on my underused server with a direct and static connection, and over a 75-minute period I had 869 ports scanned, 637 of them were unique ephemeral high-numbered ports. Botnets don't care about laws, and if your IP is static for very long, your hidden ssh port could be exposed. Yep, I can assure you they will be.  Even if we use public key authentication, we still run ssh on a high port - on one server installed in 2015, there were no ssh attacks until 7 February 2021. Since 2907 login attempts.

On another, also installed in 2015, attempts started 1 November 2020, since then 136680. Static IPs?

We might have to go one step further: migrate the port randomly and periodically.

If you're really that paranoid, take a look at port knocking:

https://www.howtogeek.com/442733/how-to-use-port-knocking-on-linux-and-why-y...

Interesting. sequence: The sequence of ports someone must access to open or close port 22. The default ports are 7000, 8000, and 9000 to open it, and 9000, 8000, and 7000 to close it. You can change these or add more ports to the list. For our purposes, we’ll stick with the defaults. However, there is a router in front of my server machine. The current configuration is that, from internet I connect using ssh to some high port, and the router changes that incoming connection to another on port 22 to the correct machine on the LAN. To do what that article suggests, I would have to add three more rules to my router, which has a limited and arbitrarily small number of rules I can write. Not very feasible, but something to consider if the situation worsens.

Even that is security through obscurity, but an argument could be made that multiple levels of obscurity can only help.

Sure. :-) Another design could be sending innocuous looking pings to some predefined port and in some time sequence, perhaps with certain payload. If things get bad, we could change altogether the entry system to the ssh to some type of challenge response method, where what is sent changes every time, and doesn't rely on encryption (but can use it). Or combine with a phone. Two factor security. One method used long ago was to call (via phone) the server, the server would hang and call you back on a predefined number. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)

On 2021/03/02 23:05, Per Jessen wrote:

Today it is slow, thorough, distributed - maybe 50 machines slowly trying out passwords, once a minute, one machine after the other. Such slow, patient attacks usually don't trigger any traps or fail2ban.

Today or rather from about five or more years ago, most sensitive institutions give about 3 invalid password attempts against an account and then lock the account, requiring a call to support with your personally identifying info. Seems like that would stop such attacks as the trigger is any 3 bad attempts and then the account is locked, no?

On 3/5/21 8:36 AM, Per Jessen wrote:

L A Walsh wrote:

...

On 2021/03/02 23:05, Per Jessen wrote:

...

Today it is slow, thorough, distributed - maybe 50 machines slowly trying out passwords, once a minute, one machine after the other. Such slow, patient attacks usually don't trigger any traps or fail2ban.

---

Today or rather from about five or more years ago, most sensitive institutions give about 3 invalid password attempts against an account and then lock the account, requiring a call to support with your personally identifying info.

Seems like that would stop such attacks as the trigger is any 3 bad attempts and then the account is locked, no? I'm not sure if I would even contemplate disabling a Linux user account based on 3 bad attempts.

Others do. Also, Smartcards will lock themselves after three failed pin attempts, requiring that the card be brought to a facility to unlock it after the bearer proves who they are. Regards, Lew

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Friday, March 5, 2021 9:10 AM, Per Jessen wrote:

Besides, with public key auth, why bother?

Word. There should be no question. Just. be. sure. to. disable. password. auth.

On 2021/03/05 08:58, jdd@dodin.org wrote:

Le 05/03/2021 à 17:36, Per Jessen a écrit :

...

I'm not sure if I would even contemplate disabling a Linux user account based on 3 bad attempts.

possible if the login is a private long thing, where it's unlikely it can be found, else any attempt by a non authorized user lock out the real user

---- The financial institutions really don't seem to have a problem with some random hacker trying to DOS all their customers. Maybe they have other policies in place to detect and block such attempts. Or maybe their customers prefer having to call in to support rather than losing a few million dollars. Note these are web-logins/password, not ones where they can hammer a specific port. It's a bit harder to automate a web-account password when you have to remember your secure picture and answer one of your secret questions in addition to the password. Instead of locking you out, it might simply force you to re-enter your some account information -- like 3 attempts on an ssh password, and you have to login to the web-access page and identify your secure picture after receiving a 1-time code at your local email address. Just having to switch modalities from host+port to web and email to re-enable host+port login is sufficiently messy to dissuade very many attempts against a specific account. ======== On 2021/03/05 09:10, Per Jessen wrote: I just foresee the situation where and 'admin' account is locked out and support has gone home for the weekend. --- First thing -- the institutions/organizations that have such security measures have teams answering phones 24/7, holidays included. There are no weekends. Second thing -- as for someone having their admin account open to password cracking facing the web -- that shouldn't happen to begin with. With a minimal security policy, you couldn't access an admin account directly from the external web. You'd have to first login to a internet access account to a bastion machine, then try for a user login to an admin-capable machine and from there elevate to an admin account. The admin account would never be exposed to the outside internet such that it would be able to be probed from the outside net -- actually not likely from the inside net either -- only an internal user account would be able to acquire admin privs on some specific machine.

On 2021/03/05 10:03, Per Jessen wrote:

Is there a reason public key auth isn't good enough? Dunno, I didn't design it. Just describing security systems used as an adjunct to ssh access.

...

======== On 2021/03/05 09:10, Per Jessen wrote: I just foresee the situation where and 'admin' account is locked out and support has gone home for the weekend. ---

First thing -- the institutions/organizations that have such security measures have teams answering phones 24/7, holidays included.

So why are we talking about here on this list?

---- Why are you taking about support going home for the weekend?

...

There are no weekends. Second thing -- as for someone having their admin account open to password cracking facing the web -- that shouldn't happen to begin

We were not talking about the web, but about ssh.

Same same. In this context, facing open web = facing public internet. Talking about disabling accounts after 3 broken password attempts and why admin accounts wouldn't be vulnerable -- mainly, they aren't accessible via a public facing internet port.

L A Walsh wrote:

On 2021/03/05 10:03, Per Jessen wrote:

...

Is there a reason public key auth isn't good enough?

Dunno, I didn't design it.

Me neither, but for some fifteeen years, it has been very good at preventing unauthorized access to my systems. If you don't know if it is good enough, maybe you need to explore it?

...

...

======== On 2021/03/05 09:10, Per Jessen wrote: I just foresee the situation where and 'admin' account is locked out and support has gone home for the weekend. ---

First thing -- the institutions/organizations that have such security measures have teams answering phones 24/7, holidays included.

So why are we talking about here on this list?

---- why are you taking about support going home for the weekend?

Well, because they do - I don't pay them to work on weekends. If a user account were to be locked due to third parties trying to gain access, that would become a simple support case. For Monday morning.

...

...

There are no weekends. Second thing -- as for someone having their admin account open to password cracking facing the web -- that shouldn't happen to begin

We were not talking about the web, but about ssh.

Same same. In this context, facing open web = facing public internet.

Plonk. -- Per Jessen, Zürich (0.7°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland.

On 2021-03-04 11:53 a.m., Adam Majer wrote:

If you want to have a quiet life for SSH admin only connections, limit to some IPv6 address unique for SSH. Then you don't have to worry about high ports. It will be quiet.

I don't know what you mean by unique for ssh but, with privacy addresses, the address used for outgoing connections is not the same as the one used for incoming.  There is one consistent address, often based on the MAC address, and up to seven temporary privacy addresses, with a new one every day.  There's no way anyone could determine the ssh server address from any privacy address.  The best they could do is narrow(?) it down to 1 in 18.4 billion, billion.

it should be possible to use random port, based on algorithm like the one used by banks and other dealers that send us a random number by sms: listen to a port with a given number, only at a given date&time if the key is common between client and server, one can know what port it is, and as this port is used only once in a while, can't be guessed, and the rest of the time ssh is not even listening... jdd -- http://dodin.org

On 2021-03-04 12:00 p.m., James Knott wrote:

On 2021-03-04 11:53 a.m., Adam Majer wrote:

...

If you want to have a quiet life for SSH admin only connections, limit to some IPv6 address unique for SSH. Then you don't have to worry about high ports. It will be quiet.

I don't know what you mean by unique for ssh but, with privacy addresses, the address used for outgoing connections is not the same as the one used for incoming.  There is one consistent address, often based on the MAC address, and up to seven temporary privacy addresses, with a new one every day.  There's no way anyone could determine the ssh server address from any privacy address.  The best they could do is narrow(?) it down to 1 in 18.4 billion, billion

Another thing you could do is use a different /64 for your servers. For example, I get a /56 prefix from my ISP.  This means I have 256 /64s.  Others provide a /48 for 65536 /64s.  Just pick one different from your outgoing connections.

On 2021-03-04 12:46 p.m., Per Jessen wrote:

...

If you want to have a quiet life for SSH admin only connections, limit to some IPv6 address unique for SSH. Then you don't have to worry about high ports. It will be quiet. Yup, I have not seen any ssh login attempts on IPv6 for at least ten years.

One reason for that is the address space is so huge.  A single /64 has as many addresses as the entire IPv4 address space squared!

On 2021-03-04 2:49 p.m., Carlos E.R. wrote:

Unfortunately, IPv6 is not available to me.

You should be able to get it through tunnelling, as I did for the first six years. Hurricane Electric (he.net) is popular.