What role does lkv-srvapp39 serve? Is it your postgresql database server? That of a second Uyuni postgresql server you are trying to take data from to merge into a HUB reporting DB? It looks like your application server may be failing to connect to it. There could be a whole bunch of things that could be causing that: a) DNS resolution? Maybe the lkv-srvapp39 server doesn't have a proper DNS entry. Maybe the Uyuni server has the wrong DNS server configured for resolving that name. Can you use nslookup to resolve the name from the Uyuni app server? b) typo on the name in the configuration file? c) missing domain name (not FQDN) and no DNS domain search configured in your resolver configuration? Or it's a different internal domain from your HUB server and there's no trust/forwarding setup for DNS between those domains? d) routing issue? e) host-based firewall on the (remote?) PG server not allowing connections on the TCP/5432 port? f) physical firewall or router ACLs preventing the connection? You're going to have to identify any other possibilities for your topology and then test all of them using other means (nslookup, ping, traceroute, psql, etc) to see what's not working. From: Crefeld, Tobias LKV Bayern e.V. Sent: Tuesday, August 2, 2022 3:29 AM To: General discussion related to the openSUSE Uyuni project Subject: AW: Subtask mgr-update-reporting failed. Hi Simon, thanks for your hints! I followed them and received a lot of messages but actually I'm no java developer. I tried to reduce the log output by leaving out the "at"-lines but it is still large. Hope that no one gets annoyed too much: 2022-08-02 09:56:00,022 [DefaultQuartzScheduler_Worker-18] WARN org.hibernate.orm.deprecation - HHH90000022: Hibernate's legacy org.hibernate.Criteria API is deprecated; use the JPA javax.persistence.criteria.CriteriaQuery instead 2022-08-02 09:56:02,718 [DefaultQuartzScheduler_Worker-1] WARN org.hibernate.orm.deprecation - HHH020100: The Ehcache second-level cache provider for Hibernate is deprecated. See https://hibernate.atlassian.net/browse/HHH-12441 for details. 2022-08-02 09:56:33,348 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#0] WARN com.mchange.v2.resourcepool.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@1c8f4b1fmailto:com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@1c... -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: org.postgresql.util.PSQLException: The hostname lkv-srvapp39 could not be verified by hostnameverifier PgjdbcHostnameVerifier. 2022-08-02 09:56:33,348 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#0] WARN com.mchange.v2.resourcepool.BasicResourcePool - Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@411ce9ecmailto:com.mchange.v2.resourcepool.BasicResourcePool@411ce9ec is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests. 2022-08-02 09:56:33,348 [DefaultQuartzScheduler_Worker-1] WARN org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator - HHH000342: Could not obtain connection to query metadata : Connections could not be acquired from the underlying database! 2022-08-02 09:56:33,354 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#1] WARN com.mchange.v2.resourcepool.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5ab931ddmailto:com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5a... -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: org.postgresql.util.PSQLException: The hostname lkv-srvapp39 could not be verified by hostnameverifier PgjdbcHostnameVerifier. 2022-08-02 09:56:33,354 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#1] WARN com.mchange.v2.resourcepool.BasicResourcePool - Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@411ce9ecmailto:com.mchange.v2.resourcepool.BasicResourcePool@411ce9ec is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests. 2022-08-02 09:56:33,360 [DefaultQuartzScheduler_Worker-1] WARN net.sf.ehcache.DiskStorePathManager - diskStorePath '/tmp' is already used by an existing CacheManager either in the same VM or in a different process. The diskStore path for this CacheManager will be set to /tmp/ehcache_auto_created2394383558487899130diskstore. To avoid this warning consider using the CacheManager factory methods to create a singleton CacheManager or specifying a separate ehcache configuration (ehcache.xml) for each CacheManager instance. 2022-08-02 09:56:33,361 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#2] WARN com.mchange.v2.resourcepool.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@68e07fffmailto:com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@68... -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: org.postgresql.util.PSQLException: The hostname lkv-srvapp39 could not be verified by hostnameverifier PgjdbcHostnameVerifier. 2022-08-02 09:56:33,362 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#2] WARN com.mchange.v2.resourcepool.BasicResourcePool - Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@411ce9ecmailto:com.mchange.v2.resourcepool.BasicResourcePool@411ce9ec is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests. 2022-08-02 09:56:33,374 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#1] WARN com.mchange.v2.resourcepool.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3b518b25mailto:com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3b... -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: org.postgresql.util.PSQLException: The hostname lkv-srvapp39 could not be verified by hostnameverifier PgjdbcHostnameVerifier. 2022-08-02 09:56:33,374 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#0] WARN com.mchange.v2.resourcepool.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5599538emailto:com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@55... -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: org.postgresql.util.PSQLException: The hostname lkv-srvapp39 could not be verified by hostnameverifier PgjdbcHostnameVerifier. 2022-08-02 09:56:33,374 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#1] WARN com.mchange.v2.resourcepool.BasicResourcePool - Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@411ce9ecmailto:com.mchange.v2.resourcepool.BasicResourcePool@411ce9ec is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests. 2022-08-02 09:56:33,375 [DefaultQuartzScheduler_Worker-1] WARN org.hibernate.engine.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: null 2022-08-02 09:56:33,375 [DefaultQuartzScheduler_Worker-1] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - Connections could not be acquired from the underlying database! 2022-08-02 09:56:33,375 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#0] WARN com.mchange.v2.resourcepool.BasicResourcePool - Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@411ce9ecmailto:com.mchange.v2.resourcepool.BasicResourcePool@411ce9ec is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests. 2022-08-02 09:56:33,384 [DefaultQuartzScheduler_Worker-1] ERROR com.redhat.rhn.taskomatic.task.ReportDbUpdateTask - Executing a task threw an exception: org.quartz.JobExecutionException 2022-08-02 09:56:33,385 [DefaultQuartzScheduler_Worker-1] ERROR com.redhat.rhn.taskomatic.task.ReportDbUpdateTask - Message: Unable to update reporting db 2022-08-02 09:56:33,385 [DefaultQuartzScheduler_Worker-1] ERROR com.redhat.rhn.taskomatic.task.ReportDbUpdateTask - Cause: {} org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database! Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source. Caused by: org.postgresql.util.PSQLException: The hostname lkv-srvapp39 could not be verified by hostnameverifier PgjdbcHostnameVerifier. 2022-08-02 09:56:33,387 [DefaultQuartzScheduler_Worker-1] ERROR com.redhat.rhn.taskomatic.task.ReportDbUpdateTask - Stack trace:org.quartz.JobExecutionException: Unable to update reporting db [See nested exception: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection] Caused by: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database! Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source. Caused by: org.postgresql.util.PSQLException: The hostname lkv-srvapp39 could not be verified by hostnameverifier PgjdbcHostnameVerifier. 2022-08-02 09:57:00,022 [DefaultQuartzScheduler_Worker-2] WARN org.hibernate.orm.deprecation - HHH90000022: Hibernate's legacy org.hibernate.Criteria API is deprecated; use the JPA javax.persistence.criteria.CriteriaQuery instead Maybe someone has a better understanding of this stuff and can explain what's going wrong? BTW: The messages with "QuartzScheduler" are repeated every minute, independent of the status of "mgr-update-reporting-bunch". TIA! Regards. Tobias Crefeld.

org.postgresql.util.PSQLException: The hostname lkv-srvapp39 could not be verified by hostnameverifier PgjdbcHostnameVerifier. This sounds to me, that you configuration is using the "short" name to access PostgreSQL. Try from your server, if this is resolvable. Maybe you're missing a proper "search" DNS config option. If the internal connection to your PostgreSQL is TLS encrypted, then the short name also needs to be added to the TLS SAN (Subject Alternative Name) the PostgreSQL is running, of the default TLS verification options are used 1. As mentioned... Is a simple "ping lkv-srvapp39" resolvable from within your uyuni server? 2. Try to change the host setting that defines your PostgreSQL and change it to use the full FQDN. Robert Am 6. Aug. 2022, 00:37, um 00:37, Paul-Andre Panon via Uyuni Users schrieb:

What role does lkv-srvapp39 serve? Is it your postgresql database server? That of a second Uyuni postgresql server you are trying to take data from to merge into a HUB reporting DB? It looks like your application server may be failing to connect to it. There could be a whole bunch of things that could be causing that: a) DNS resolution? Maybe the lkv-srvapp39 server doesn't have a proper DNS entry. Maybe the Uyuni server has the wrong DNS server configured for resolving that name. Can you use nslookup to resolve the name from the Uyuni app server? b) typo on the name in the configuration file? c) missing domain name (not FQDN) and no DNS domain search configured in your resolver configuration? Or it's a different internal domain from your HUB server and there's no trust/forwarding setup for DNS between those domains? d) routing issue? e) host-based firewall on the (remote?) PG server not allowing connections on the TCP/5432 port? f) physical firewall or router ACLs preventing the connection?

You're going to have to identify any other possibilities for your topology and then test all of them using other means (nslookup, ping, traceroute, psql, etc) to see what's not working.

From: Crefeld, Tobias LKV Bayern e.V. Sent: Tuesday, August 2, 2022 3:29 AM To: General discussion related to the openSUSE Uyuni project Subject: AW: Subtask mgr-update-reporting failed.

Hi Simon,

thanks for your hints! I followed them and received a lot of messages but actually I'm no java developer.

I tried to reduce the log output by leaving out the "at"-lines but it is still large. Hope that no one gets annoyed too much:

2022-08-02 09:56:00,022 [DefaultQuartzScheduler_Worker-18] WARN org.hibernate.orm.deprecation - HHH90000022: Hibernate's legacy org.hibernate.Criteria API is deprecated; use the JPA javax.persistence.criteria.CriteriaQuery instead

2022-08-02 09:56:02,718 [DefaultQuartzScheduler_Worker-1] WARN org.hibernate.orm.deprecation - HHH020100: The Ehcache second-level cache provider for Hibernate is deprecated. See https://hibernate.atlassian.net/browse/HHH-12441 for details.

2022-08-02 09:56:33,348 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#0] WARN com.mchange.v2.resourcepool.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@1c8f4b1fmailto:com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@1c... -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: org.postgresql.util.PSQLException: The hostname lkv-srvapp39 could not be verified by hostnameverifier PgjdbcHostnameVerifier.

2022-08-02 09:56:33,348 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#0] WARN com.mchange.v2.resourcepool.BasicResourcePool - Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@411ce9ecmailto:com.mchange.v2.resourcepool.BasicResourcePool@411ce9ec is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.

2022-08-02 09:56:33,348 [DefaultQuartzScheduler_Worker-1] WARN org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator - HHH000342: Could not obtain connection to query metadata : Connections could not be acquired from the underlying database!

2022-08-02 09:56:33,354 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#1] WARN com.mchange.v2.resourcepool.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5ab931ddmailto:com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5a... -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: org.postgresql.util.PSQLException: The hostname lkv-srvapp39 could not be verified by hostnameverifier PgjdbcHostnameVerifier.

2022-08-02 09:56:33,354 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#1] WARN com.mchange.v2.resourcepool.BasicResourcePool - Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@411ce9ecmailto:com.mchange.v2.resourcepool.BasicResourcePool@411ce9ec is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.

2022-08-02 09:56:33,360 [DefaultQuartzScheduler_Worker-1] WARN net.sf.ehcache.DiskStorePathManager - diskStorePath '/tmp' is already used by an existing CacheManager either in the same VM or in a different process. The diskStore path for this CacheManager will be set to /tmp/ehcache_auto_created2394383558487899130diskstore. To avoid this warning consider using the CacheManager factory methods to create a singleton CacheManager or specifying a separate ehcache configuration (ehcache.xml) for each CacheManager instance.

2022-08-02 09:56:33,361 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#2] WARN com.mchange.v2.resourcepool.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@68e07fffmailto:com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@68... -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: org.postgresql.util.PSQLException: The hostname lkv-srvapp39 could not be verified by hostnameverifier PgjdbcHostnameVerifier.

2022-08-02 09:56:33,362 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#2] WARN com.mchange.v2.resourcepool.BasicResourcePool - Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@411ce9ecmailto:com.mchange.v2.resourcepool.BasicResourcePool@411ce9ec is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.

2022-08-02 09:56:33,374 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#1] WARN com.mchange.v2.resourcepool.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3b518b25mailto:com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3b... -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: org.postgresql.util.PSQLException: The hostname lkv-srvapp39 could not be verified by hostnameverifier PgjdbcHostnameVerifier.

2022-08-02 09:56:33,374 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#0] WARN com.mchange.v2.resourcepool.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5599538emailto:com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@55... -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: org.postgresql.util.PSQLException: The hostname lkv-srvapp39 could not be verified by hostnameverifier PgjdbcHostnameVerifier.

2022-08-02 09:56:33,374 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#1] WARN com.mchange.v2.resourcepool.BasicResourcePool - Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@411ce9ecmailto:com.mchange.v2.resourcepool.BasicResourcePool@411ce9ec is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.

2022-08-02 09:56:33,375 [DefaultQuartzScheduler_Worker-1] WARN org.hibernate.engine.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: null

2022-08-02 09:56:33,375 [DefaultQuartzScheduler_Worker-1] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - Connections could not be acquired from the underlying database!

2022-08-02 09:56:33,375 [C3P0PooledConnectionPoolManager[identityToken->z8kfluaqp0t6ro20pjuq|3ff5aba]-HelperThread-#0] WARN com.mchange.v2.resourcepool.BasicResourcePool - Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@411ce9ecmailto:com.mchange.v2.resourcepool.BasicResourcePool@411ce9ec is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.

2022-08-02 09:56:33,384 [DefaultQuartzScheduler_Worker-1] ERROR com.redhat.rhn.taskomatic.task.ReportDbUpdateTask - Executing a task threw an exception: org.quartz.JobExecutionException 2022-08-02 09:56:33,385 [DefaultQuartzScheduler_Worker-1] ERROR com.redhat.rhn.taskomatic.task.ReportDbUpdateTask - Message: Unable to update reporting db 2022-08-02 09:56:33,385 [DefaultQuartzScheduler_Worker-1] ERROR com.redhat.rhn.taskomatic.task.ReportDbUpdateTask - Cause: {} org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection

Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!

Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.

Caused by: org.postgresql.util.PSQLException: The hostname lkv-srvapp39 could not be verified by hostnameverifier PgjdbcHostnameVerifier.

2022-08-02 09:56:33,387 [DefaultQuartzScheduler_Worker-1] ERROR com.redhat.rhn.taskomatic.task.ReportDbUpdateTask - Stack trace:org.quartz.JobExecutionException: Unable to update reporting db [See nested exception: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection]

Caused by: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection

Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!

Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.

Caused by: org.postgresql.util.PSQLException: The hostname lkv-srvapp39 could not be verified by hostnameverifier PgjdbcHostnameVerifier.

2022-08-02 09:57:00,022 [DefaultQuartzScheduler_Worker-2] WARN org.hibernate.orm.deprecation - HHH90000022: Hibernate's legacy org.hibernate.Criteria API is deprecated; use the JPA javax.persistence.criteria.CriteriaQuery instead

Maybe someone has a better understanding of this stuff and can explain what's going wrong?

BTW: The messages with "QuartzScheduler" are repeated every minute, independent of the status of "mgr-update-reporting-bunch".

TIA!

Regards. Tobias Crefeld.