DefaultLogic.removeExpired removal of slot failed (#111)

Wojciech Kapcia (Tigase) opened 4 years ago

[2021-03-31 17:37:42:739] [FINE    ] [  ConnectionOpenThread ] DefaultLogic.removeExpired()     : removal of slot failed
java.io.IOException: Could not remove file 689c2aca-9fb5-4884-805a-2801a667b7d2 from S3
	at tigase.extras.http.upload.S3Store.remove(S3Store.java:140)
	at tigase.http.upload.logic.DefaultLogic.removeExpired(DefaultLogic.java:161)
	at tigase.http.upload.ExpirationTask.run(ExpirationTask.java:50)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The XML you provided was not well-formed or did not validate against our published schema (Service: Amazon S3; Status Code: 400; Error Code: MalformedXML; Request ID: WZ0KP1NV2CMA4S30; S3 Extended Request ID: nGMd09CP/6HQon+9Zre3p5ORVSqWN85XS4pwNGia4Z9J54mStNSisCBGnLcidwLpR08CpBX7EFs=; Proxy: null), S3 Extended Request ID: nGMd09CP/6HQon+9Zre3p5ORVSqWN85XS4pwNGia4Z9J54mStNSisCBGnLcidwLpR08CpBX7EFs=
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1811)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1395)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1371)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5155)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5101)
	at com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2279)
	at tigase.extras.http.upload.S3Store.remove(S3Store.java:138)
	... 8 more

Activities

Andrzej Wójcik (Tigase) commented 4 years ago

Might be caused by files with incorrect filenames being uploaded a while ago. It would be good to increase logging level and log list of keys which we wanted to delete to verify that.
Andrzej Wójcik (Tigase) commented 4 years ago

I've checked S3 and it looks like there is no folder named 689c2aca-9fb5-4884-805a-2801a667b7d2 while there should be if the file was uploaded. Most likely file failed to upload by we still have old entries in slots repository which is used as a source of the list of files to remove.
Andrzej Wójcik (Tigase) commented 4 years ago

It looks like the issue was caused by "deleting" an empty list of files with the use of the DeleteObjects AWS S3 action which expects a list of files. The exception reported is about XML not well-formed, so most likely not matching schema (object to remove tag may be required to appear at least one time).

I've applied a fix to our code to skip requests to AWS if there are no files in the slot (were not uploaded or were already removed).
Login to comment

Type	Bug
Priority	Normal
Assignee	Wojciech Kapcia (Tigase)
Version	tigase-server-8.2.0

Issue Votes (0)

Watchers (0)

Reference

tigase/_server/tigase-http-api#111