%wojtek We can do that, however in most test cases accounts may differ between each test in same test case. So we should annotate each method then?

In fact in this case I'm more in a favour of creation base test class (ie. AbstractTest) and in it add methods to create accounts and clean them - actual test should only call them in method annotated with @Before and clean them in method annotated with @@After@. I do not see point in moving creation of account to annotations.

Waiting for clarification from Wojciech.

Andrzej Wójcik wrote:

In fact in this case I'm more in a favour of creation base test class (ie. AbstractTest) and in it add methods to create accounts and clean them - actual test should only call them in method annotated with @Before and clean them in method annotated with @@After@. I do not see point in moving creation of account to annotations.

OK, the biggest problem is lack of admin account. I think that in general we need at least one admin account (that does all the work) - this could be default and created before running a full test (hence idea with annotations - test case/method could be annoted with "requiredJID" and it could be ensured - i.e. created if missing), and in addition each test should (must?) create accounts/items it needs.

It can be done with Before/After, but... it could be possible already but not implemented.

I've rewritten our base classes used for testing and I've implemented few managers.

Unfortunately, it was not possible to use annotations to provide accounts, vhosts, pubsub nodes as each of this items require different options and a lot of them may be used.

Instead of using annotations I created managers with methods returning builders. Using this builders you may pass settings and call build() to create object. This (if possible) should be done in methods annotated with @BeforeClass to execute before any test in this class, or with @BeforeMethod to execute before every test method will be executed. If not it may be used within test method.

Thanks to managers and builders it is not required to unregister accounts, delete pubsub nodes or remove vhosts - it will be done automatically by managers when proper test scope will be left (ie. objects created in @BeforeClass will be destroyed after execution of every test method in this class).

I also added verification that admin account exists on server and if not then tests will automatically create this account (done once per test suite).

Andrzej Wójcik wrote:

I also added verification that admin account exists on server and if not then tests will automatically create this account (done once per test suite).

%andrzej.wojcik

Will this work for individual tests? For example:

mvn surefire:test -Dtest=tigase.tests.util.RetrieveVersion

%andrzej.wojcik

OK, it seems that tigase.tests.AbstractTest#ensureAdminAccountExists is called only @BeforeSuite so single method/package test won't work.

Would it make sense to call it always but store the result so the actual check would be done only on first call?

You are right, but @BeforeSute should be called event if a single test is started. I've found out that this was a bug in TestNG see https://github.com/cbeust/testng/issues/420 and we were using version 6.8.8 which was still affected. To solve this issue I updated TestNG to newest available version 6.11.

Thank you for the update - now it's run, but seems to fail:

java.lang.AssertionError: expected [null] but found [tigase.jaxmpp.core.client.exceptions.JaxmppException: tigase.jaxmpp.core.client.exceptions.JaxmppException: Unauthorized with condition=not_authorized]
Expected :null
Actual   :tigase.jaxmpp.core.client.exceptions.JaxmppException: tigase.jaxmpp.core.client.exceptions.JaxmppException: Unauthorized with condition=not_authorized
 <Click to see difference>


	at org.testng.Assert.fail(Assert.java:93)
	at org.testng.Assert.failNotSame(Assert.java:508)
	at org.testng.Assert.assertNull(Assert.java:444)
	at org.testng.Assert.assertNull(Assert.java:433)
	at tigase.tests.AbstractTest.ensureAdminAccountExists(AbstractTest.java:199)
	at tigase.tests.AbstractTest.setupSuite(AbstractTest.java:536)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:523)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:224)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:146)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:326)
	at org.testng.SuiteRunner.run(SuiteRunner.java:289)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1301)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1226)
	at org.testng.TestNG.runSuites(TestNG.java:1144)
	at org.testng.TestNG.run(TestNG.java:1115)
	at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
	at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:127)

I've checked and there was a wrong implementation of exception handling and I've changed it. Now it works fine.

After finishing the test there is a contradictory exception:

tearDownSuite(tigase.tests.util.RetrieveVersion)  Time elapsed: 0.491 sec  <<< FAILURE!
java.lang.RuntimeException: Better not to remove user 'admin', don't you think?
	at tigase.tests.utils.AccountsManager.unregisterAccount(AccountsManager.java:242)
	at tigase.tests.utils.AccountsManager.unregisterAccount(AccountsManager.java:223)
	at tigase.tests.utils.AccountsManager.lambda$scopeFinished$2(AccountsManager.java:127)
	at java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:890)
	at java.util.concurrent.CopyOnWriteArraySet.forEach(CopyOnWriteArraySet.java:404)
	at tigase.tests.utils.AccountsManager.scopeFinished(AccountsManager.java:125)
	at tigase.tests.AbstractTest.tearDownSuite(AbstractTest.java:542)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

It seems that removal of the admin account was called yet account manager complains about it? Should we delete admin account (or any other automatically created account) only if it was created when starting the test?

As we discussed, I've allowed removal of admin account if account was created by TTS-NG.

Works correctly now.