odd style of URL on download.o.o (but causing a lot of traffic)
I am seeing a lot of traffic with GET URLs such as these: GET /repositories/./Apache:/MirrorBrain/SLE_15_SP2/x86_64/http://build.opensuse.org/ GET /repositories/Apache/openSUSE_Leap_15.2/x86_64/https://software.opensuse.org/ GET /repositories/./Apache:/Shibboleth/SLE_12_SP2/repodata/http://build.opensuse.org/ GET /repositories/./Apache:/MirrorBrain/SLE_15_SP2/https://software.opensuse.org/ Of course they all result in a 404, but currently such requests take up almost 10% of our total http traffic. Does anyone recognise that unusual request format, with 'https://software.opensuse.org/' or 'http://build.opensuse.org/' appended ? This started on 6 December with a few hundred thousand requests, but grew to 2million by 13 December, now around 3.5-4million a day. -- Per Jessen, Zürich (7.9°C) Member, openSUSE Heroes
On 26/12/2021 12.43, Per Jessen wrote:
I am seeing a lot of traffic with GET URLs such as these:
GET /repositories/./Apache:/MirrorBrain/SLE_15_SP2/x86_64/http://build.opensuse.org/ GET /repositories/Apache/openSUSE_Leap_15.2/x86_64/https://software.opensuse.org/ GET /repositories/./Apache:/Shibboleth/SLE_12_SP2/repodata/http://build.opensuse.org/ GET /repositories/./Apache:/MirrorBrain/SLE_15_SP2/https://software.opensuse.org/
Of course they all result in a 404, but currently such requests take up almost 10% of our total http traffic.
Does anyone recognise that unusual request format, with 'https://software.opensuse.org/' or 'http://build.opensuse.org/' appended ?
This started on 6 December with a few hundred thousand requests, but grew to 2million by 13 December, now around 3.5-4million a day.
Maybe you should translate those log entries to what a web browser would have in the address bar to generate those entries. Me, I can not figure it out, not that familiar with apache. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)
Carlos E. R. wrote:
On 26/12/2021 12.43, Per Jessen wrote:
I am seeing a lot of traffic with GET URLs such as these:
GET /repositories/./Apache:/MirrorBrain/SLE_15_SP2/x86_64/http://build.opensuse.org/ GET /repositories/Apache/openSUSE_Leap_15.2/x86_64/https://software.opensuse.org/ GET /repositories/./Apache:/Shibboleth/SLE_12_SP2/repodata/http://build.opensuse.org/ GET /repositories/./Apache:/MirrorBrain/SLE_15_SP2/https://software.opensuse.org/
Of course they all result in a 404, but currently such requests take up almost 10% of our total http traffic.
Does anyone recognise that unusual request format, with 'https://software.opensuse.org/' or 'http://build.opensuse.org/' appended ?
This started on 6 December with a few hundred thousand requests, but grew to 2million by 13 December, now around 3.5-4million a day.
Maybe you should translate those log entries to what a web browser would have in the address bar to generate those entries.
Well, that is exactly what a browser would show too, but they are not coming from browsers, only from curl.
Me, I can not figure it out, not that familiar with apache.
Apache is not relevant. It could be any webserver. -- Per Jessen, Zürich (7.9°C) Member, openSUSE Heroes
On 26/12/2021 13.14, Per Jessen wrote:
Carlos E. R. wrote:
On 26/12/2021 12.43, Per Jessen wrote:
I am seeing a lot of traffic with GET URLs such as these:
GET /repositories/./Apache:/MirrorBrain/SLE_15_SP2/x86_64/http://build.opensuse.org/ GET /repositories/Apache/openSUSE_Leap_15.2/x86_64/https://software.opensuse.org/ GET /repositories/./Apache:/Shibboleth/SLE_12_SP2/repodata/http://build.opensuse.org/ GET /repositories/./Apache:/MirrorBrain/SLE_15_SP2/https://software.opensuse.org/
Of course they all result in a 404, but currently such requests take up almost 10% of our total http traffic.
Does anyone recognise that unusual request format, with 'https://software.opensuse.org/' or 'http://build.opensuse.org/' appended ?
This started on 6 December with a few hundred thousand requests, but grew to 2million by 13 December, now around 3.5-4million a day.
Maybe you should translate those log entries to what a web browser would have in the address bar to generate those entries.
Well, that is exactly what a browser would show too, but they are not coming from browsers, only from curl.
Me, I can not figure it out, not that familiar with apache.
Apache is not relevant. It could be any webserver.
Ok, I mean I'm not familiar with apache logs, I can not interpret them. I can not figure out what actual URL curl tried. Or do you mean that curl tried exactly: curl /repositories/./Apache:/MirrorBrain/SLE_15_SP2/https: \ //software.opensuse.org/ That is really weird. I tried this: Telcontar:/var/log/YaST2 # zgrep "/repositories/Apache \ /openSUSE_Leap_15.2/" y2log*gz Telcontar:/var/log/YaST2 # No hits. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)
Carlos E. R. wrote:
Ok, I mean I'm not familiar with apache logs, I can not interpret them. I can not figure out what actual URL curl tried.
Or do you mean that curl tried exactly:
curl /repositories/./Apache:/MirrorBrain/SLE_15_SP2/https: \ //software.opensuse.org/
Well, it would still have to correspond to the standard scheme, e.g. curl http://download.opensuse.org/repositories/./Apache:/MirrorBrain/SLE_15_SP2/h... -- Per Jessen, Zürich (8.5°C) Member, openSUSE Heroes
On 26/12/2021 13.39, Per Jessen wrote:
Carlos E. R. wrote:
Ok, I mean I'm not familiar with apache logs, I can not interpret them. I can not figure out what actual URL curl tried.
Or do you mean that curl tried exactly:
curl /repositories/./Apache:/MirrorBrain/SLE_15_SP2/https: \ //software.opensuse.org/
Well, it would still have to correspond to the standard scheme, e.g.
curl http://download.opensuse.org/repositories/./Apache:/MirrorBrain/SLE_15_SP2/h...
Anyway, I grepped my logs, no hit: Telcontar:/var/log/YaST2 # zgrep "./Apache" y2log*gz Telcontar:/var/log/YaST2 # zgrep "Apache:" y2log*gz Telcontar:/var/log/YaST2 # grep "Apache:" y2log Telcontar:/var/log/YaST2 # grep "./Apache" y2log Telcontar:/var/log/YaST2 # grep "Apache" y2log Telcontar:/var/log/YaST2 # zgrep "Apache" y2log*gz Telcontar:/var/log/YaST2 # I would suspect a malformed download string related to the Apache repository, so some script in a package there. I don't use that repository, but perhaps someone has the toolset to analyze that entire repo for scripts in packages there for curl calls. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)
Carlos E. R. wrote:
I would suspect a malformed download string related to the Apache repository,
It affects all kinds of repositories, although all under repositories/ - I only published a couple of sample URLs. home/ libreoffice/ cloud/ hamradio/ -- Per Jessen, Zürich (8.9°C) Member, openSUSE Heroes
On 26/12/2021 14.02, Per Jessen wrote:
Carlos E. R. wrote:
I would suspect a malformed download string related to the Apache repository,
It affects all kinds of repositories, although all under repositories/ - I only published a couple of sample URLs. home/ libreoffice/ cloud/ hamradio/
Oh. Well, perhaps you could post more samples we can grep in our logs to see if they are there. Of course, there could be bad luck and they are not logged at all :-( -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)
Carlos E. R. wrote:
On 26/12/2021 14.02, Per Jessen wrote:
Carlos E. R. wrote:
I would suspect a malformed download string related to the Apache repository,
It affects all kinds of repositories, although all under repositories/ - I only published a couple of sample URLs. home/ libreoffice/ cloud/ hamradio/
Oh.
Well, perhaps you could post more samples we can grep in our logs to see if they are there.
Of course, there could be bad luck and they are not logged at all :-(
I doubt it - they are not coming from anywhere in Spain for instance and they are all being fetched with curl. https://progress.opensuse.org/issues/104328 -- Per Jessen, Zürich (10.2°C) Member, openSUSE Heroes
Am Sonntag, 26. Dezember 2021, 12:43:21 CET schrieb Per Jessen:
This started on 6 December with a few hundred thousand requests, but grew to 2million by 13 December, now around 3.5-4million a day.
Where is it coming from? Always the same addresses (Means: someone probing a (D)DNS)? Cheers Axel
Axel Braun wrote:
Am Sonntag, 26. Dezember 2021, 12:43:21 CET schrieb Per Jessen:
This started on 6 December with a few hundred thousand requests, but grew to 2million by 13 December, now around 3.5-4million a day.
Where is it coming from? Always the same addresses (Means: someone probing a (D)DNS)?
I did analyse the traffic from 21 or 22 December - 79 unique IPs, from e.g. Linode. I have a gut feeling it might be caused by something we've released ? -- Per Jessen, Zürich (8.3°C) Member, openSUSE Heroes
On 26/12/2021 13.26, Per Jessen wrote:
Axel Braun wrote:
Am Sonntag, 26. Dezember 2021, 12:43:21 CET schrieb Per Jessen:
This started on 6 December with a few hundred thousand requests, but grew to 2million by 13 December, now around 3.5-4million a day.
Where is it coming from? Always the same addresses (Means: someone probing a (D)DNS)?
I did analyse the traffic from 21 or 22 December - 79 unique IPs, from e.g. Linode.
I have a gut feeling it might be caused by something we've released ?
Maybe on apache repo. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)
On Sun, 26 Dec 2021 13:26:11 +0100 Per Jessen wrote:
Am Sonntag, 26. Dezember 2021, 12:43:21 CET schrieb Per Jessen:
This started on 6 December with a few hundred thousand requests, but grew to 2million by 13 December, now around 3.5-4million a day.
Where is it coming from? Always the same addresses (Means: someone probing a (D)DNS)?
I did analyse the traffic from 21 or 22 December - 79 unique IPs, from e.g. Linode.
I have a gut feeling it might be caused by something we've released ? 79 source IP addresses at least seems not to be something widely used. Maybe these are 79 public IP addresses of clouds and the requests come from many instances of e.g. docker containers running in these clouds?
Regards, Dieter -- Unencrypted and unsigned email is like a postcard written in pencil.
dieter wrote:
On Sun, 26 Dec 2021 13:26:11 +0100 Per Jessen wrote:
I have a gut feeling it might be caused by something we've released ? 79 source IP addresses at least seems not to be something widely used.
Maybe these are 79 public IP addresses of clouds and the requests come from many instances of e.g. docker containers running in these clouds?
Yeah, that is also my hunch, but I have no idea where to look. -- Per Jessen, Zürich (8.2°C) Member, openSUSE Heroes
On Sunday 2021-12-26 12:43, Per Jessen wrote:
I am seeing a lot of traffic with GET URLs such as these:
GET /repositories/./Apache:/MirrorBrain/SLE_15_SP2/x86_64/http://build.opensuse.org/ GET /repositories/Apache/openSUSE_Leap_15.2/x86_64/https://software.opensuse.org/ GET /repositories/./Apache:/Shibboleth/SLE_12_SP2/repodata/http://build.opensuse.org/ GET /repositories/./Apache:/MirrorBrain/SLE_15_SP2/https://software.opensuse.org/
Of course they all result in a 404, but currently such requests take up almost 10% of our total http traffic.
This started on 6 December with a few hundred thousand requests, but grew to 2million by 13 December, now around 3.5-4million a day.
Based on the growth pattern, the jump-to idea is libzypp. (That also happens to be using curl, so, ... go figure) The repositories that are targeted apparently only concern 12.2/15.2 systems, Factory/Tumbleweed URLs is prominently absent. Hurr. openSUSE:Leap:15.2:Update/libzypp: >---------------------------------------------------------------------------- >r13 | maintenance-robot | 2021-12-06 13:06:45 | 81c554ade0548b4ec3c309f5be693d99 | unknown | rq935137 > >Set link to libzypp.17215 via maintenance_release request Well well well. Botched update?
Jan Engelhardt wrote:
On Sunday 2021-12-26 12:43, Per Jessen wrote:
I am seeing a lot of traffic with GET URLs such as these:
GET /repositories/./Apache:/MirrorBrain/SLE_15_SP2/x86_64/http://build.opensuse.org/ GET /repositories/Apache/openSUSE_Leap_15.2/x86_64/https://software.opensuse.org/ GET /repositories/./Apache:/Shibboleth/SLE_12_SP2/repodata/http://build.opensuse.org/ GET /repositories/./Apache:/MirrorBrain/SLE_15_SP2/https://software.opensuse.org/
Of course they all result in a 404, but currently such requests take up almost 10% of our total http traffic.
This started on 6 December with a few hundred thousand requests, but grew to 2million by 13 December, now around 3.5-4million a day.
Based on the growth pattern, the jump-to idea is libzypp. (That also happens to be using curl, so, ... go figure)
The repositories that are targeted apparently only concern 12.2/15.2 systems, Factory/Tumbleweed URLs is prominently absent. Hurr.
Thanks for the input Jan - the URLs above were just samples, I'll check if I see any other patterns.
openSUSE:Leap:15.2:Update/libzypp:
----------------------------------------------------------------------------
r13 | maintenance-robot | 2021-12-06 13:06:45 | 81c554ade0548b4ec3c309f5be693d99 | unknown | rq935137 > Set link to libzypp.17215 via maintenance_release request
Well well well. Botched update?
The first such requests started at 21 December 10:21 UTC - from a CloudVsp system in Beijing. They also included Leap 42.3, Leap 15.0, SLE15 though. -- Per Jessen, Zürich (6.4°C) Member, openSUSE Heroes
On Sunday 2021-12-26 20:22, Per Jessen wrote:
I am seeing a lot of traffic with GET URLs such as these:
GET /repositories/./Apache:/MirrorBrain/SLE_15_SP2/x86_64/http://build.opensuse.org/
This started on 6 December with a few hundred thousand requests, but grew to 2million by 13 December, now around 3.5-4million a day.
----------------------------------------------------------------------------
r13 | maintenance-robot | 2021-12-06 13:06:45 | 81c554ade0548b4ec3c309f5be693d99 | unknown | rq935137 > Set link to libzypp.17215 via maintenance_release request
Botched update?
The first such requests started at 21 December 10:21 UTC
So what was that about Dec 6?
Jan Engelhardt wrote:
On Sunday 2021-12-26 20:22, Per Jessen wrote:
I am seeing a lot of traffic with GET URLs such as these:
GET /repositories/./Apache:/MirrorBrain/SLE_15_SP2/x86_64/http://build.opensuse.org/
This started on 6 December with a few hundred thousand requests, but grew to 2million by 13 December, now around 3.5-4million a day.
----------------------------------------------------------------------------
r13 | maintenance-robot | 2021-12-06 13:06:45 | 81c554ade0548b4ec3c309f5be693d99 | unknown | rq935137 > Set link to libzypp.17215 via maintenance_release request
Botched update?
The first such requests started at 21 December 10:21 UTC
So what was that about Dec 6?
Arrggh! Typo ...... December 6 10:21 UTC. -- Per Jessen, Zürich (5.8°C) Member, openSUSE Heroes
On Sunday 2021-12-26 21:04, Per Jessen wrote:
Jan Engelhardt wrote:
On Sunday 2021-12-26 20:22, Per Jessen wrote:
I am seeing a lot of traffic with GET URLs such as these:
GET /repositories/./Apache:/MirrorBrain/SLE_15_SP2/x86_64/http://build.opensuse.org/
This started on 6 December with a few hundred thousand requests, but grew to 2million by 13 December, now around 3.5-4million a day.
----------------------------------------------------------------------------
r13 | maintenance-robot | 2021-12-06 13:06:45 | 81c554ade0548b4ec3c309f5be693d99 | unknown | rq935137 > Set link to libzypp.17215 via maintenance_release request
Botched update?
The first such requests started at 21 December 10:21 UTC
So what was that about Dec 6?
Arrggh! Typo ...... December 6 10:21 UTC.
The update for SLE 15.2 was committed a few days prior to Leap 15.2 ---------------------------------------------------------------------------- r14 | mauriziogalli | 2021-12-02 06:11:46 | b1f6fd015223da484cb7595b2bb1359c | unknown | Set link to libzypp.21847 via maintenance_release request
Jan Engelhardt wrote:
The update for SLE 15.2 was committed a few days prior to Leap 15.2
---------------------------------------------------------------------------- r14 | mauriziogalli | 2021-12-02 06:11:46 | b1f6fd015223da484cb7595b2bb1359c | unknown |
Set link to libzypp.21847 via maintenance_release request
Okay - any suggestion as to what do I need to to do to get this fixed to take a load of our infrastructure? -- Per Jessen, Zürich (12.8°C) Member, openSUSE Heroes
On Thursday 2021-12-30 21:54, Per Jessen wrote:
Jan Engelhardt wrote:
The update for SLE 15.2 was committed a few days prior to Leap 15.2
---------------------------------------------------------------------------- r14 | mauriziogalli | 2021-12-02 06:11:46 | b1f6fd015223da484cb7595b2bb1359c | unknown |
Set link to libzypp.21847 via maintenance_release request
Okay - any suggestion as to what do I need to to do to get this fixed to take a load of our infrastructure?
Can we get a full line of the httpd log? https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/7... just said "GET /repositories/./Apache:/MirrorBrain/SLE_15_SP2/x86_64/http://build.opensuse.org/" and you mentioned curl without furthere detail. zypp would use an agent like "ZYpp 17.28.8 (curl 7.66.0) openSUSE-Leap-15.2-x86_64", so one can see that it is zypp - or, in the absence, that it is not. Also what crossed my mind is that redundant "./" in the request URI. This I would attribute to apt, but then again, apt does not convey curl in the user-agent. A different thought is that someone's having a stupid shell script somewhere, with plenty of people having a copy..
On 30/12/2021 23.36, Jan Engelhardt wrote:
Can we get a full line of the httpd log?
here are some:
pontifex2 (download.o.o):/var/log/apache2/download.opensuse.org # grep /build.opensuse.org access_log|head 139.180.217.245 - - [30/Dec/2021:23:59:59 +0000] "GET /repositories/./Apache:/Shibboleth/SLE_15/x86_64/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:20473 P:139.180.192.0/19 864 6757 size:- - "-" "-" 123.59.120.132 - - [30/Dec/2021:23:59:59 +0000] "GET /repositories/./Apache:/Modules/SLE_15/x86_64/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:4808 P:123.59.0.0/16 808 7207 size:- - "-" "-" 45.33.42.112 - - [30/Dec/2021:23:59:59 +0000] "GET /repositories/./Apache:/Modules/Apache_SLE_15_SP1/src/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:63949 P:45.33.0.0/17 869 6757 size:- - "-" "-" 223.166.174.4 - - [30/Dec/2021:23:59:59 +0000] "GET /repositories/./Apache:/Shibboleth/openSUSE_Leap_42.3/x86_64/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:17621 P:223.166.0.0/16 823 7207 size:- - "-" "-" 192.248.154.55 - - [30/Dec/2021:23:59:59 +0000] "GET /repositories/Archiving/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:20473 P:192.248.128.0/18 839 6757 size:- - "-" "-" 139.162.219.171 - - [30/Dec/2021:23:59:59 +0000] "GET /repositories/./Application:/./ERP:/./GNUHealth:/3.6/openSUSE_Leap_15.2/repodata/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:63949 P:139.162.0.0/16 896 6757 size:- - "-" "-" 45.33.42.112 - - [31/Dec/2021:00:00:00 +0000] "GET /repositories/./Apache:/MirrorBrain/openSUSE_Factory/x86_64/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:63949 P:45.33.0.0/17 875 6757 size:- - "-" "-" 123.59.120.156 - - [31/Dec/2021:00:00:00 +0000] "GET /repositories/Apache/openSUSE_Leap_15.2/x86_64/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:4808 P:123.59.0.0/16 809 7207 size:- - "-" "-" 45.33.116.69 - - [31/Dec/2021:00:00:00 +0000] "GET /repositories/./Application:/./ERP:/./Tryton:/5.0/openSUSE_Leap_15.3/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:63949 P:45.33.0.0/17 884 6757 size:- - "-" "-" 172.104.98.170 - - [31/Dec/2021:00:00:00 +0000] "GET /repositories/./Apache:/Modules/openSUSE_Factory/repodata/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:63949 P:172.104.64.0/18 873 6757 size:- - "-" "-" pontifex2 (download.o.o):/var/log/apache2/download.opensuse.org # grep /build.opensuse.org access_log|tail 139.180.217.245 - - [31/Dec/2021:01:57:36 +0000] "GET /repositories/./Apache:/Modules/Apache_SLE_15_SP1/src/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:20473 P:139.180.192.0/19 869 6757 size:- - "-" "-" 172.105.232.137 - - [31/Dec/2021:01:57:36 +0000] "GET /repositories/./Apache:/MirrorBrain/openSUSE_Leap_15.1/noarch/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:63949 P:172.105.192.0/18 877 6757 size:- - "-" "-" 173.230.131.60 - - [31/Dec/2021:01:57:36 +0000] "GET /repositories/./Apache:/Modules/openSUSE_Factory/src/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "Wget/1.17.1 (linux-curl)" want:- give:- r:- - -:- ASN:63949 P:173.230.128.0/19 881 6757 size:- - "-" "-" 172.105.232.137 - - [31/Dec/2021:01:57:36 +0000] "GET /repositories/Apache/SLE_15_SP2/src/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:63949 P:172.105.192.0/18 851 6757 size:- - "-" "-" 139.162.176.152 - - [31/Dec/2021:01:57:36 +0000] "GET /repositories/./Apache:/Modules/Apache_openSUSE_Leap_15.1/src/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:63949 P:139.162.0.0/16 877 6757 size:- - "-" "-" 95.179.217.30 - - [31/Dec/2021:01:57:36 +0000] "GET /repositories/./Application:/./ERP:/./Tryton:/5.0/openSUSE_Leap_15.3/noarch/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:20473 P:95.179.128.0/17 891 6757 size:- - "-" "-" 104.237.135.81 - - [31/Dec/2021:01:57:36 +0000] "GET /repositories/./Apache:/MirrorBrain/openSUSE_Factory/src/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:63949 P:104.237.128.0/19 872 6757 size:- - "-" "-" 139.162.225.134 - - [31/Dec/2021:01:57:36 +0000] "GET /repositories/./Apache:/Test/openSUSE_Tumbleweed/repodata/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:63949 P:139.162.0.0/16 873 6757 size:- - "-" "-" 123.59.120.200 - - [31/Dec/2021:01:57:36 +0000] "GET /repositories/./Apache:/MirrorBrain/Debian_10/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:4808 P:123.59.0.0/16 808 7207 size:- - "-" "-" 149.248.53.80 - - [31/Dec/2021:01:57:36 +0000] "GET /repositories/./Apache:/Shibboleth/openSUSE_Tumbleweed/i586/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:20473 P:149.248.0.0/18 875 6757 size:- - "-" "-" pontifex2 (download.o.o):/var/log/apache2/download.opensuse.org # grep /software.opensuse.org/ access_log|wc 580274 14582099 158685814 pontifex2 (download.o.o):/var/log/apache2/download.opensuse.org # grep /build.opensuse.org/ access_log|wc 582160 14628988 156865540
one interesting fact is that it seems to do requests with both incorrect suffixes at the same rate of around 84/s
pontifex2 (download.o.o):/var/log/apache2/download.opensuse.org # grep /software.opensuse.org/ access_log|cut -d\ -f1|sort|uniq -c|sort -n|tail -40 7700 45.63.124.224 8031 123.59.120.44 8480 172.105.232.137 8926 223.166.174.30 9281 172.104.163.142 9838 180.153.180.97 9929 172.104.49.212 9991 223.166.174.4 10089 223.166.174.27 11108 45.33.42.112 11145 172.104.98.170 11244 123.59.120.253 11716 173.230.131.60 11795 123.59.120.176 11820 139.162.71.138 12067 223.166.174.39 12162 123.59.120.132 12202 123.59.120.200 12368 180.153.180.102 12861 149.248.53.80 12920 123.59.211.81 12940 45.79.150.80 12949 139.162.225.134 13032 123.59.120.201 13093 123.59.120.35 13132 123.59.120.156 13489 172.105.17.61 13859 223.166.174.34 15366 45.33.116.69 15584 198.58.105.17 15695 45.33.110.152 16294 123.59.120.135 16388 95.179.217.30 16773 104.237.135.81 16878 123.59.120.73 17073 45.77.60.139 17180 123.59.120.240 18870 123.59.120.230 26303 139.162.176.152 41749 139.162.219.171
pontifex2 (download.o.o):/var/log/apache2/download.opensuse.org # grep /software.opensuse.org/ access_log|cut -d\ -f1|sort|uniq -c|wc -l 112
only 66 of those did more than 100 requests Interestingly, there are webservers responding on all IPs I checked. All with some login form. Some said P-660R-T1 v2 PMG5317-T20B which is a Zyxel home router, so maybe these are some hacked servers scanning the web for more stuff to hack? I temporarily blacklisted those 66 IPs and now we are down to (more normal?) 20000 requests per minute. Somewhat unrelated: there are 6% of requests like this:
"HEAD /update/leap/15.3/oss/media.1/media HTTP/2.0" 404 1083 "-" "ZYpp 17.27.0 (curl 7.66.0) "
On 31/12/2021 03.39, Bernhard M. Wiedemann wrote:
which is a Zyxel home router, so maybe these are some hacked servers scanning the web for more stuff to hack?
I temporarily blacklisted those 66 IPs and now we are down to (more normal?) 20000 requests per minute.
Maybe it is even more interesting. Accessing different ports/URLs on these IPs lets me believe these are honeypots. That we see these log entries, means that these honeypots run actual untrusted code from strangers on the internet. When watching all requests from a single IP, it becomes very obvious that it is a (very stupid) crawler and the Apache repos were requested, because they come first in http://download.opensuse.org/repositories/ . grep -o "GET /[^ ]*" single-ip.log
GET /repositories/ GET /repositories/http://build.opensuse.org/ GET /repositories/https://software.opensuse.org/ GET /repositories/Apache/ GET /repositories/Apache/http://build.opensuse.org/ GET /repositories/Apache/https://software.opensuse.org/ GET /repositories/Apache//repositories/ GET /repositories/Apache//repositories/ GET /repositories/Apache/openSUSE_Factory/ GET /repositories/Apache/openSUSE_Factory//repositories/Apache/ GET /repositories/Apache/openSUSE_Factory/http://build.opensuse.org/ GET /repositories/Apache/openSUSE_Factory/https://software.opensuse.org/ GET /repositories/Apache/openSUSE_Factory//repositories/Apache/ GET /repositories/Apache/openSUSE_Factory/i586/ GET /repositories/Apache/openSUSE_Factory/i586//repositories/Apache/openSUSE_Factory/ GET /repositories/Apache/openSUSE_Factory/i586/http://build.opensuse.org/ GET /repositories/Apache/openSUSE_Factory/i586/https://software.opensuse.org/ GET /repositories/Apache/openSUSE_Factory/i586/ GET /repositories/Apache/openSUSE_Factory/i586//repositories/Apache/openSUSE_Factory/ GET /repositories/Apache/openSUSE_Factory/i586//repositories/Apache/openSUSE_Factory/ GET /repositories/Apache/openSUSE_Factory/i586/http://build.opensuse.org/ GET /repositories/Apache/openSUSE_Factory/i586/https://software.opensuse.org/ GET /repositories/Apache/openSUSE_Factory/i586//repositories/Apache/openSUSE_Factory/ GET /repositories/Apache/openSUSE_Factory/noarch/ GET /repositories/Apache/openSUSE_Factory/noarch//repositories/Apache/openSUSE_Factory/
On Friday 2021-12-31 03:39, Bernhard M. Wiedemann wrote:
here are some:
139.180.217.245 - - [30/Dec/2021:23:59:59 +0000] "GET /repositories/./Apache:/Shibboleth/SLE_15/x86_64/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:20473 P:139.180.192.0/19 864 6757 size:- - "-" "-" 139.180.217.245 - - [31/Dec/2021:01:57:36 +0000] "GET /repositories/./Apache:/Modules/Apache_SLE_15_SP1/src/http://build.opensuse.org/ HTTP/1.1" 404 1083 "-" "curl/7.54.0" want:- give:- r:- - -:- ASN:20473 P:139.180.192.0/19 869 6757 size:- - "-" "-"
* So indeed just plain curl/libcurl, no zypp. * Who runs curl-7.54 anyway? - there is no curl-7.54 in any openSUSE product and state to be found - there is no curl-7.54 in any contemporary, proliferated distro: https://repology.org/project/curl/versions - could be your Zyxel systems
one interesting fact is that it seems to do requests with both incorrect suffixes at the same rate of around 84/s
pontifex2 (download.o.o):/var/log/apache2/download.opensuse.org # grep /software.opensuse.org/ access_log|cut -d\ -f1|sort|uniq -c|sort -n|tail -40 7700 45.63.124.224 JP, office-ten router 8031 123.59.120.44 CN, office-ten 8480 172.105.232.137 CN, office-ten 8926 223.166.174.30 CN, Huawei EchoLife HG8546M, china mobile 9281 172.104.163.142 US, zyxel mgmt 9838 180.153.180.97 CN, moxa console mgmt 9929 172.104.49.212 CN, "bai cells" mgmt 11108 45.33.42.112 bai cells 11145 172.104.98.170 Huawei USG6390 11244 123.59.120.253 HG8546M 11716 173.230.131.60 US, IIS/"redcamera" 12162 123.59.120.132 cn, TL-WR842N router 12202 123.59.120.200 cn, IIS/"redcamera" 12920 123.59.211.81 cn, "DVR components" 13032 123.59.120.201 zimbra 13093 123.59.120.35 cn, ZTE ZXHN H168N 13132 123.59.120.156 13489 172.105.17.61 moxa
only 66 of those did more than 100 requests
Interestingly, there are webservers responding on all IPs I checked. All with some login form. Some said P-660R-T1 v2 PMG5317-T20B which is a Zyxel home router, so maybe these are some hacked servers scanning the web for more stuff to hack?
We won't know ultimately. Oh well.
Somewhat unrelated: there are 6% of requests like this:
"HEAD /update/leap/15.3/oss/media.1/media HTTP/2.0" 404 1083 "-" "ZYpp 17.27.0 (curl 7.66.0) "
Guess someone has 15.3 configured with type=yast2 in .repo files from earlier days and never noticed. This scenario _is_ possible. In 15.3, openSUSE-release.rpm started forcing .repo files on us. Having a wrong URL in one of your _own_ (rpm-untracked) .repo files therefore can go unnoticed especially when a graphic frontend is used (thinking PackageKit here). The behavior of shipping .repo is unique in that 15.2 did not do it, and neither does tumbleweed.
Bernhard M. Wiedemann wrote:
I temporarily blacklisted those 66 IPs and now we are down to (more normal?) 20000 requests per minute.
Looking at the last couple of days, there is only one IPv4 address left, I have added that one too, temporarily. -- Per Jessen, Zürich (13.2°C) Member, openSUSE Heroes
Am 03.01.22 um 10:52 schrieb Per Jessen:
Bernhard M. Wiedemann wrote:
I temporarily blacklisted those 66 IPs and now we are down to (more normal?) 20000 requests per minute.
Looking at the last couple of days, there is only one IPv4 address left, I have added that one too, temporarily. Since the aforementioned IPs have been blacklisted, I am not experiencing "curl error 16" anymore. As a matter of fact, the updates to TW 20220101 and TW20220102 went like a charm. Thx.
Regards, Frank
participants (7)
-
Axel Braun -
Bernhard M. Wiedemann -
Carlos E. R. -
dieter -
Frank Krüger -
Jan Engelhardt -
Per Jessen