Projects tigase _server server-core Issues #819
Failing to initialize repository could force-stop server? (#819)
Closed
wojciech.kapcia@tigase.net opened 7 years ago
Due Date
2017-05-01
2017-04-28 17:03:23.028 [main]             Kernel.injectDependencies()             WARNING:  Can't inject dependency to bean default (class: class tigase.db.beans.AuthRepositoryMDPoolBean$AuthRepositoryConfigBean) unloading bean default
RootCause:
   -> java.lang.RuntimeException: Could not initialize tigase.db.jdbc.TigaseCustomAuth for name 'default'
      [tigase.db.beans.MDPoolConfigBean.setInstances(MDPoolConfigBean.java:169)]
      -> tigase.db.DBInitException: Problem initializing jdbc connection: jdbc:derby:/Users/wojtek/dev/tigase/tigase-testsuite/tigase_test_db;create=true
         [tigase.db.jdbc.TigaseCustomAuth.initRepository(TigaseCustomAuth.java:621)]
         -> java.sql.SQLSyntaxErrorException: No method was found that matched the method call tigase.db.derby.StoredProcedures.tigAccountStatus(java.lang.String, java.sql.ResultSet[]), tried all combinations of object and primitive types and any possible type conversion for any  parameters the method call may have. The method might exist but it is not public and/or static, or the parameter types are not method invocation convertible.
            [org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)]
            -> ERROR 42X50: No method was found that matched the method call tigase.db.derby.StoredProcedures.tigAccountStatus(java.lang.String, java.sql.ResultSet[]), tried all combinations of object and primitive types and any possible type conversion for any  parameters the method call may have. The method might exist but it is not public and/or static, or the parameter types are not method invocation convertible.
               [org.apache.derby.iapi.error.StandardException.newException(Unknown Source)]
…
2017-04-28 17:03:49.813 [jabber:iq:register Queue Worker 6]  WorkerThread.run()    SEVERE:   tigase.server.xmppsession.SessionManager$ProcessorWorkerThread,(jabber:iq:register Queue Worker 6) Exception during packet processing: from=c2s@atlantiscity.local/127.0.0.1_5222_127.0.0.1_52400, to=sess-man@atlantiscity.local, DATA=<iq type="set" id="reg2" xmlns="jabber:client"><query xmlns="jabber:iq:register"><username>admin</username><password>stats</password><email>test_user@localhost</email></query></iq>, SIZE=180, XMLNS=jabber:client, PRIORITY=NORMAL, PERMISSION=LOCAL, TYPE=set
java.lang.NullPointerException
	at tigase.db.jdbc.TigaseCustomAuth.addUser(TigaseCustomAuth.java:346)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at tigase.stats.StatisticsInvocationHandler.invoke(StatisticsInvocationHandler.java:80)
	at com.sun.proxy.$Proxy31.addUser(Unknown Source)
	at tigase.db.AuthRepositoryMDImpl.addUser(AuthRepositoryMDImpl.java:66)
	at tigase.xmpp.impl.JabberIqRegister.createAccount(JabberIqRegister.java:161)
	at tigase.xmpp.impl.JabberIqRegister.doRegisterNewAccount(JabberIqRegister.java:335)
	at tigase.xmpp.impl.JabberIqRegister.process(JabberIqRegister.java:642)
	at tigase.server.xmppsession.SessionManager$ProcessorWorkerThread.process(SessionManager.java:2442)
	at tigase.util.WorkerThread.run(WorkerThread.java:128)

We already have a couple of cases where failing to initialise repository stops the server with an error -- should we consider above as well?

Andrzej Wójcik (Tigase) commented 7 years ago

There are two cases here:

  • failure to connect to default data source (default repository)

Without this many components and features may fail to work properly (including clustering) so we should stop a server.

  • failure to connect to other data sources

This may return some errors but I think that it should work.

I'm going to review code responsible for initialization of repositories as it may be handy to start a server before DB is started (or while it is starting) and connect to a database as it will become available.

Andrzej Wójcik (Tigase) commented 7 years ago

I've found that even though the instance of a repository is not initialized properly it is injected to repository pool which leads to NPE during runtime. I made changes in a kernel to make sure it will not happen again. This should fix above NPE.

Moreover, I've added check during bootstrapping of Tigase XMPP Server to make sure that data sources are initialized before we will start loading components - to make it fail-fast with fewer errors being logged.

Please verify that this works for you. We cannot stop server at this point as it may happen that this exception will be thrown during server reconfiguration and stopping server at this point could be very inconvenient.

wojciech.kapcia@tigase.net commented 7 years ago

Andrzej Wójcik wrote:

There are two cases here:

  • failure to connect to default data source (default repository)

Without this many components and features may fail to work properly (including clustering) so we should stop a server.

+1

  • failure to connect to other data sources

This may return some errors but I think that it should work.

I'm not sure about that - it leaves the server in rather half-working state (and I was under the impression that you are in general more in favour of failing that allowing that, for example our discussion in DSL where we consider configuration "bad" if any of the items are wrong instead of ignoring them)

I'm going to review code responsible for initialization of repositories as it may be handy to start a server before DB is started (or while it is starting) and connect to a database as it will become available.

Andrzej Wójcik wrote:

I've found that even though the instance of a repository is not initialized properly it is injected to repository pool which leads to NPE during runtime. I made changes in a kernel to make sure it will not happen again. This should fix above NPE.

Moreover, I've added check during bootstrapping of Tigase XMPP Server to make sure that data sources are initialized before we will start loading components - to make it fail-fast with fewer errors being logged.

Please verify that this works for you.

It works for me, however the is a small but - 'config-type' = 'setup' results in stopping the server:

==========
STARTED Tigase Tue May  2 13:10:02 -03 2017 using:
    ./scripts/tigase.sh start etc/tigase.conf 
==========
Picked up JAVA_TOOL_OPTIONS: -Djava.awt.headless=true
componentInfo{Title=Tigase XMPP Server, Version=7.2.0-SNAPSHOT-b4809/7b560498 (2017-05-01/23:10:40), Class=tigase.xml.XMLUtils}
componentInfo{Title=Tigase XMPP Server, Version=7.2.0-SNAPSHOT-b4809/7b560498 (2017-05-01/23:10:40), Class=tigase.util.ClassUtil}
componentInfo{Title=Tigase XMPP Server, Version=7.2.0-SNAPSHOT-b4809/7b560498 (2017-05-01/23:10:40), Class=tigase.server.XMPPServer}
2017-05-02 13:10:02.390 [main]             ConfiguratorAbstract.parseArgs()        CONFIG:   Setting defaults: --property-file = etc/init.properties
2017-05-02 13:10:02.486 [main]             ConfigHolder.loadConfiguration()        CONFIG:   Loaded configuration:
--test = false
'config-type' = 'setup'
http () {
    setup () {
        'admin-password' = 'tigase'
        'admin-user' = 'admin'
    }
}





  =============================================================================
  ERROR! Terminating the server process.
  Problem initializing the server: tigase.kernel.KernelException: Can't find bean implementing class tigase.db.beans.DataSourceBean
  Please fix the problem and start the server again.
  =============================================================================





ShutdownThread started...

Total number of threads: 5
No locked threads.

Save thread-dump to file: logs/thread-dump.log, size: 1689
ShutdownThread finished...

We cannot stop server at this point as it may happen that this exception will be thrown during server reconfiguration and stopping server at this point could be very inconvenient.

Agreed, and before we considered the same:

  • if the server couldn't establish connection to repository during startup = bad state;

  • exception accessing repository after correct startup = ok state (temporary network problem for example that could be recovered).

Andrzej Wójcik (Tigase) commented 7 years ago

Wojciech Kapcia wrote:

Andrzej Wójcik wrote:

There are two cases here:

  • failure to connect to default data source (default repository)

Without this many components and features may fail to work properly (including clustering) so we should stop a server.

+1

  • failure to connect to other data sources

This may return some errors but I think that it should work.

I'm not sure about that - it leaves the server in rather half-working state (and I was under the impression that you are in general more in favour of failing that allowing that, for example our discussion in DSL where we consider configuration "bad" if any of the items are wrong instead of ignoring them)

Yes, you are right. I was just thinking about a possible use case.

I'm going to review code responsible for initialization of repositories as it may be handy to start a server before DB is started (or while it is starting) and connect to a database as it will become available.

Andrzej Wójcik wrote:

I've found that even though the instance of a repository is not initialized properly it is injected to repository pool which leads to NPE during runtime. I made changes in a kernel to make sure it will not happen again. This should fix above NPE.

Moreover, I've added check during bootstrapping of Tigase XMPP Server to make sure that data sources are initialized before we will start loading components - to make it fail-fast with fewer errors being logged.

Please verify that this works for you.

It works for me, however the is a small but - 'config-type' = 'setup' results in stopping the server:

[...]

It is already fixed. I've fixed this issue today so it is not part of latest snapshot build.

We cannot stop server at this point as it may happen that this exception will be thrown during server reconfiguration and stopping server at this point could be very inconvenient.

Agreed, and before we considered the same:

  • if the server couldn't establish connection to repository during startup = bad state;
  • exception accessing repository after correct startup = ok state (temporary network problem for example that could be recovered).

Yes, this is how it works now.

wojciech.kapcia@tigase.net commented 7 years ago

Works OK now.

issue 1 of 1
Type
Bug
Priority
Normal
Assignee
RedmineID
5450
Version
tigase-server-8.0.0
Spent time
32h 30m
Issue Votes (0)
Watchers (0)
Reference
tigase/_server/server-core#819
Please wait...
Page is in error, reload to recover