[heroes] pfx2asn not up to date? (olaf/scanner)

newer
[heroes] mirror.hostsuisse.com -...

older
[heroes] Planned power outage of...

Per Jessen

27 Sep 2017 27 Sep '17

13:14

On olaf, there is a cronjob that is meant to refresh the pfx2asn table every day. It doesn't look as if it has been done for maybe 2 years. Some scripts or binaries are missing 'asn_get_routeviews' and 'asn_import' ? These are supplied by apache2-mod_asn-tools, which is not installed. I was just wondering - olaf is new, but that data in the pfx2asn table is least 2 years old ? About a week ago, I wrote to the mirroring list about some stats we discovered on our public mirror: ------------------------ ... these are some interesting observations. Looking only at traffic from user-agent "ZYpp <something>" : [snip] 123 different countries, 50 with more than 10 unique addresses. (IPv4 only). These are the top scorers: country addrs | us | 1115 | | de | 768 | | it | 658 | | ch | 639 | | es | 462 | | fr | 315 | | br | 312 | | ru | 234 | | ca | 143 | | in | 131 | | cn | 128 | | mx | 120 | | uk | 119 | | za | 111 | | pl | 105 | ---------------------------- I think this is due to no asn numbers for our IP range (185.85.248.0/22). Before I go ahead and install apache2-mod_asn-tools, can anyone think of a reason I shouldn't ? afaict, all it does is retrieve the daily snapshot from mirrorbrain.org and load that into pfx2asn. -- Per Jessen, Zürich (19.1°C) openSUSE mailing list admin -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org

Show replies by date

Lars Vogdt

27 Sep 27 Sep

14:31

Am Wed, 27 Sep 2017 15:14:34 +0200 schrieb Per Jessen :

...

Before I go ahead and install apache2-mod_asn-tools, can anyone think of a reason I shouldn't ? afaict, all it does is retrieve the daily snapshot from mirrorbrain.org and load that into pfx2asn.

I have to admit that the missing RPM was my fault: I left out all apache* packages on olaf by intention, to have a very small footprint - not having in mind that the *tools* might be needed. Sorry for that. :-/ I installed the package now and initially triggered an update. But I did not check if this changes something in the database. Regards, Lars -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org

Per Jessen

16:05

Lars Vogdt wrote:

...

Am Wed, 27 Sep 2017 15:14:34 +0200 schrieb Per Jessen :

...
Before I go ahead and install apache2-mod_asn-tools, can anyone think of a reason I shouldn't ? afaict, all it does is retrieve the daily snapshot from mirrorbrain.org and load that into pfx2asn.

I have to admit that the missing RPM was my fault: I left out all apache* packages on olaf by intention, to have a very small footprint - not having in mind that the *tools* might be needed. Sorry for that. :-/

No big deal Lars. I was just wondering as the data seemed to be quite old - olaf hasn't been running for very long?

...

I installed the package now and initially triggered an update. But I did not check if this changes something in the database.

I've checked the database, pfx2asn was updated. That should be enough. -- Per Jessen, Zürich (20.4°C) openSUSE mailing list admin -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org

Lars Vogdt

18:45

Am 27. September 2017 18:05:33 MESZ schrieb Per Jessen :

...

No big deal Lars. I was just wondering as the data seemed to be quite old - olaf hasn't been running for very long?

The database is very old - and Olaf is currently not doing the scans: these are still running on the old machine. Dunno why the data hasn't been updated from there since such a long time - and honestly I just want to get rid of the old machine sonner than later. => time to sent an announcement to the mirror last that our scanner now scans from another IP (which is still in the documented range, so normally the should no change be needed on the mirrors side). My problem: atm Olaf just has IPv4 - I need to think how we can manage to let him also scan via IPv6...

...

I've checked the database, pfx2asn was updated. That should be enough.

Thanks for checking! But as I don't want to make the mistake twice: can you tell me the SQL statement you used to verify? I want to write a monitoring check for this... Regards, Lars -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org

Per Jessen

19:07

Lars Vogdt wrote:

...

Am 27. September 2017 18:05:33 MESZ schrieb Per Jessen :

...
No big deal Lars. I was just wondering as the data seemed to be quite old - olaf hasn't been running for very long?

The database is very old - and Olaf is currently not doing the scans: these are still running on the old machine.

Aha, I was not aware.

...

Dunno why the data hasn't been updated from there since such a long time - and honestly I just want to get rid of the old machine sonner than later.

=> time to sent an announcement to the mirror last that our scanner now scans from another IP (which is still in the documented range, so normally the should no change be needed on the mirrors side). My problem: atm Olaf just has IPv4 - I need to think how we can manage to let him also scan via IPv6...

To get scanning on ipv6 would be good, agree. Our mirror is taken offline for about 3 minutes when olaf tries the ipv6 address first, then the ipv4 address a little later. Mind you, the pfx2asn table is ipv4 only ...

...

...
I've checked the database, pfx2asn was updated. That should be enough.

Thanks for checking!

Just to make sure I got the right place - mirrordb3, right?

...

But as I don't want to make the mistake twice: can you tell me the SQL statement you used to verify? I want to write a monitoring check for this...

It's not so easy - I happened to notice 'asn 0' and 'prefix 0' being listed for our mirror. "mb iplookup 185.85.248.0". I checked the pfx2asn table, and the prefix (185.95.248.0/22) wasn't listed. That range was allocated in Jan 2015, so I knew some data had to be out of date. Maybe you can check for the downloaded file being out of date? /home/mirrorbrain/oix-full-snapshot-latest.dat.bz2 -- Per Jessen, Zürich (16.2°C) openSUSE mailing list admin -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org

Lars Vogdt

19:56

Am 27. September 2017 21:07:28 MESZ schrieb Per Jessen :

...

...
let him also scan via IPv6...

To get scanning on ipv6 would be good, agree. Our mirror is taken offline for about 3 minutes when olaf tries the ipv6 address first, then the ipv4 address a little later.

Jip. I'm wondering, if scar could either do some NATv6 or route/accept IPv6 from Olaf directly. At the moment, my favorite is NAT, as this would not only have a security plus, but also the benefit of having everything coming from the same DNS source. But I'm open for suggestions...

...

Mind you, the pfx2asn table is ipv4 only ...

There should be some IPv6 tables, too (and now the question is how they get updated). But I did not look into the DB for a long time.

...

...
Thanks for checking!

Just to make sure I got the right place - mirrordb3, right?

Yes. mirrordb3 is currently the master (you can see this in /etc/mirrorbrain.conf) and mirrordb4 the (read only) slave.

...

...
I want to write a monitoring check for this...

It's not so easy - I happened to notice 'asn 0' and 'prefix 0' being listed for our mirror. "mb iplookup 185.85.248.0". I checked the pfx2asn table, and the prefix (185.95.248.0/22) wasn't listed. That range was allocated in Jan 2015, so I knew some data had to be out of date. Maybe you can check for the downloaded file being out of date?

That happens IMHO already, but is a good first step. We might also check some known entries to be correct. And I'll see what I can get out of the DB next week, when I'm back in the office. Regards from SUSECON ;-) Lars -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org

Per Jessen

28 Sep 28 Sep

07:32

Per Jessen wrote:

...

Lars Vogdt wrote:

...
Am Wed, 27 Sep 2017 15:14:34 +0200 schrieb Per Jessen :

...
Before I go ahead and install apache2-mod_asn-tools, can anyone think of a reason I shouldn't ? afaict, all it does is retrieve the daily snapshot from mirrorbrain.org and load that into pfx2asn.

I have to admit that the missing RPM was my fault: I left out all apache* packages on olaf by intention, to have a very small footprint - not having in mind that the *tools* might be needed. Sorry for that. :-/

No big deal Lars. I was just wondering as the data seemed to be quite old - olaf hasn't been running for very long?

...
I installed the package now and initially triggered an update. But I did not check if this changes something in the database.

I've checked the database, pfx2asn was updated. That should be enough.

Well, "mb iplookup 185.85.248.0" returns the appropriate information now: # mb iplookup 185.85.248.0 185.85.248.0/22 (AS13030) None That's good, but "mb show mirror.hostsuisse.com" says: # mb show mirror.hostsuisse.com identifier : mirror.hostsuisse.com operatorName : Hostsuisse (ENIDAN Technologies GmbH) operatorUrl : http://www.hostsuisse.com/ baseurl : http://mirror.hostsuisse.com/opensuse/ baseurlFtp : baseurlRsync : rsync://mirror.hostsuisse.com/opensuse/ region : eu country : ch asn : 0 prefix : 0.0.0.0/0 lat,lng : 47.145,8.155 [snip] I don't know how this mirror definition is updated, I had expected it to happen automagically. -- Per Jessen, Zürich (14.8°C) openSUSE mailing list admin -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org

Per Jessen

07:42

Per Jessen wrote:

...

Per Jessen wrote:

That's good, but "mb show mirror.hostsuisse.com" says:

# mb show mirror.hostsuisse.com identifier : mirror.hostsuisse.com operatorName : Hostsuisse (ENIDAN Technologies GmbH) operatorUrl : http://www.hostsuisse.com/ baseurl : http://mirror.hostsuisse.com/opensuse/ baseurlFtp : baseurlRsync : rsync://mirror.hostsuisse.com/opensuse/ region : eu country : ch asn : 0 prefix : 0.0.0.0/0 lat,lng : 47.145,8.155 [snip]

I don't know how this mirror definition is updated, I had expected it to happen automagically.

Okay, I found it: # mb update --asn --prefix mirror.hostsuisse.com mirror.hostsuisse.com: updating network prefix (0.0.0.0/0 -> 185.85.248.0/22) mirror.hostsuisse.com: updating autonomous system number (0 -> 13030) There are a few others mirrors with asn=0,prefix=0 - it's easy to update them, but I can't help thinking it ought to be automatic? Also, ASNs do change. I'll do some dry-runs and see how outdated our database is. -- Per Jessen, Zürich (15.1°C) openSUSE mailing list admin -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org

Per Jessen

08:34

Per Jessen wrote:

...

There are a few others mirrors with asn=0,prefix=0 - it's easy to update them, but I can't help thinking it ought to be automatic? Also, ASNs do change.

I'll do some dry-runs and see how outdated our database is.

I checked 324 mirror addresses - 118 network updates 60 ASN updates 37 DNS lookup for hostname failed: Name or service not known 12 STRANGE! There's no ASN containing this hosts IP address 11 warnings about multiple IP addresses 99 Not found (not sure what 'Not found' actually means). 107 Okay. I don't know how important this data is. -- Per Jessen, Zürich (16.0°C) openSUSE mailing list admin -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org

Per Jessen

19:04

Per Jessen wrote:

...

------------------------ ... these are some interesting observations. Looking only at traffic from user-agent "ZYpp <something>" : [snip] 123 different countries, 50 with more than 10 unique addresses. (IPv4 only). These are the top scorers:

country addrs | us | 1115 | | de | 768 | | it | 658 | | ch | 639 | | es | 462 | | fr | 315 | | br | 312 | | ru | 234 | | ca | 143 | | in | 131 | | cn | 128 | | mx | 120 | | uk | 119 | | za | 111 | | pl | 105 |

----------------------------

I think this is due to no asn numbers for our IP range (185.85.248.0/22).

A quick look at the logs today indicate far more traffic from CH, and hardly any from e.g. US, DE, BR etc. Getting pfx2asn up-to-date did the job. This was never a problem for the mirror, but it didn't make much sense for US|DE|BR|etc clients to be downloading anything from a CH mirror. -- Per Jessen, Zürich (17.7°C) openSUSE mailing list admin -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org

2419

Age (days ago)

2420

Last active (days ago)

List overview

Download

9 comments

2 participants

participants (2)

Lars Vogdt
Per Jessen

[heroes] pfx2asn not up to date? (olaf/scanner)

tags

participants (2)