HTTP Status 500 - persistence layer problem

Hi Everyone,

I just noticed that I cannot log into my Aggregate instance via my browser. I already searched the forums but this exact report doesn't match on any thread.
I get the following report on each log in attempt.

What is the problem? Please be detailed.

HTTP Status 500 - persistence layer problem; nested exception is org.opendatakit.common.persistence.exception.ODKDatastoreException: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

description The server encountered an internal error that prevented it from fulfilling this request.

exception

org.springframework.dao.TransientDataAccessResourceException: persistence layer problem; nested exception is org.opendatakit.common.persistence.exception.ODKDatastoreException: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
	org.opendatakit.common.security.spring.UserDetailsServiceImpl.loadUserByUsername(UserDetailsServiceImpl.java:206)
	org.springframework.security.web.authentication.www.DigestAuthenticationFilter.doFilter(DigestAuthenticationFilter.java:159)
	org.opendatakit.common.security.spring.DigestAuthenticationFilter.doFilter(DigestAuthenticationFilter.java:40)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.opendatakit.common.security.spring.OutOfBandUserFilter.doFilter(OutOfBandUserFilter.java:105)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.opendatakit.common.security.spring.Oauth2ResourceFilter.doFilter(Oauth2ResourceFilter.java:352)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:157)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
	org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
	org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
root cause

org.opendatakit.common.persistence.exception.ODKDatastoreException: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
	org.opendatakit.common.persistence.engine.mysql.QueryImpl.executeQuery(QueryImpl.java:273)
	org.opendatakit.common.security.spring.RegisteredUsersTable.getUniqueUserByUsername(RegisteredUsersTable.java:310)
	org.opendatakit.common.security.spring.UserDetailsServiceImpl.loadUserByUsername(UserDetailsServiceImpl.java:159)
	org.springframework.security.web.authentication.www.DigestAuthenticationFilter.doFilter(DigestAuthenticationFilter.java:159)
	org.opendatakit.common.security.spring.DigestAuthenticationFilter.doFilter(DigestAuthenticationFilter.java:40)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.opendatakit.common.security.spring.OutOfBandUserFilter.doFilter(OutOfBandUserFilter.java:105)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.opendatakit.common.security.spring.Oauth2ResourceFilter.doFilter(Oauth2ResourceFilter.java:352)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:157)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
	org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
	org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
root cause

org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
	org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
	org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:619)
	org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:684)
	org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:716)
	org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:726)
	org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:776)
	org.opendatakit.common.persistence.engine.mysql.QueryImpl.executeQuery(QueryImpl.java:266)
	org.opendatakit.common.security.spring.RegisteredUsersTable.getUniqueUserByUsername(RegisteredUsersTable.java:310)
	org.opendatakit.common.security.spring.UserDetailsServiceImpl.loadUserByUsername(UserDetailsServiceImpl.java:159)
	org.springframework.security.web.authentication.www.DigestAuthenticationFilter.doFilter(DigestAuthenticationFilter.java:159)
	org.opendatakit.common.security.spring.DigestAuthenticationFilter.doFilter(DigestAuthenticationFilter.java:40)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.opendatakit.common.security.spring.OutOfBandUserFilter.doFilter(OutOfBandUserFilter.java:105)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.opendatakit.common.security.spring.Oauth2ResourceFilter.doFilter(Oauth2ResourceFilter.java:352)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:157)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
	org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
	org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
root cause

com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
	sun.reflect.GeneratedConstructorAccessor154.newInstance(Unknown Source)
	sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
	com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
	com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:341)
	com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2192)
	com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2225)
	com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2024)
	com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
	com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
	sun.reflect.GeneratedConstructorAccessor30.newInstance(Unknown Source)
	sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
	com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
	com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
	org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:39)
	org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:256)
	org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868)
	org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
	org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
	org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
	org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1533)
	org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
	org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
	org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:619)
	org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:684)
	org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:716)
	org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:726)
	org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:776)
	org.opendatakit.common.persistence.engine.mysql.QueryImpl.executeQuery(QueryImpl.java:266)
	org.opendatakit.common.security.spring.RegisteredUsersTable.getUniqueUserByUsername(RegisteredUsersTable.java:310)
	org.opendatakit.common.security.spring.UserDetailsServiceImpl.loadUserByUsername(UserDetailsServiceImpl.java:159)
	org.springframework.security.web.authentication.www.DigestAuthenticationFilter.doFilter(DigestAuthenticationFilter.java:159)
	org.opendatakit.common.security.spring.DigestAuthenticationFilter.doFilter(DigestAuthenticationFilter.java:40)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.opendatakit.common.security.spring.OutOfBandUserFilter.doFilter(OutOfBandUserFilter.java:105)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.opendatakit.common.security.spring.Oauth2ResourceFilter.doFilter(Oauth2ResourceFilter.java:352)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:157)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
	org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
	org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
root cause

java.net.ConnectException: Connection refused (Connection refused)
	java.net.PlainSocketImpl.socketConnect(Native Method)
	java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
	java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
	java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
	java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	java.net.Socket.connect(Socket.java:580)
	com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)
	com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
	com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2192)
	com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2225)
	com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2024)
	com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
	com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
	sun.reflect.GeneratedConstructorAccessor30.newInstance(Unknown Source)
	sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
	com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
	com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
	org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:39)
	org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:256)
	org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868)
	org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
	org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
	org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
	org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1533)
	org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
	org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
	org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:619)
	org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:684)
	org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:716)
	org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:726)
	org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:776)
	org.opendatakit.common.persistence.engine.mysql.QueryImpl.executeQuery(QueryImpl.java:266)
	org.opendatakit.common.security.spring.RegisteredUsersTable.getUniqueUserByUsername(RegisteredUsersTable.java:310)
	org.opendatakit.common.security.spring.UserDetailsServiceImpl.loadUserByUsername(UserDetailsServiceImpl.java:159)
	org.springframework.security.web.authentication.www.DigestAuthenticationFilter.doFilter(DigestAuthenticationFilter.java:159)
	org.opendatakit.common.security.spring.DigestAuthenticationFilter.doFilter(DigestAuthenticationFilter.java:40)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.opendatakit.common.security.spring.OutOfBandUserFilter.doFilter(OutOfBandUserFilter.java:105)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.opendatakit.common.security.spring.Oauth2ResourceFilter.doFilter(Oauth2ResourceFilter.java:352)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:157)
	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
	org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
	org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
note The full stack trace of the root cause is available in the Apache Tomcat/8.0.43 logs.

What steps can we take to reproduce the problem?

Not sure. I could log in perfectly well a few weeks ago but now I can't.

I have attached the same error report on a text file.

Thanks for the assistance.

SpaceTurtle

Hi, @SpaceTurtle!

The error is telling us that there's some communication problem between Aggregate and your MySQL database server.

Could you check if it's up, running, and accepting connections?

You could always try rebooting the server if that's something you can do.

2 Likes

Hi. You may have already found this in your searching, but does this help?