[BUG]: File expiry not causing file cleanup by Sasha

[jpmcmu]

Description

The DataOps team reports they have file expiry set on files located on the anr thor cluster, that are not members of a superifle, and they are not being removed by sasha. This is causing disk space issues on the thor (10.194.27.1 - 8). I looked up one example.

File name: thor::base::ock_p2::inc::20251203_2::monitoring_auto_unsuppressed_backup

Looking in logical files on eclwatch:

image (1)-20260120-161831.png

The file is set to expire on 12/24. I checked the sasha logs that are on 10.194.112.102 and I don’t see where the removal was ever tried on 12/31, which is the only log the file name appears.

saserver.2025_12_31.log:000C3856 2025-12-31 22:26:00.862 1755280 1755697 "XREF: [thor_anr_storage] Process file thor::base::ock_p2::inc::20251203_2::monitoring_auto_unsuppressed_backup"

saserver.2025_12_31.log:0010AE5B 2025-12-31 23:32:34.488 1755280 1755697 "WARNING: /var/lib/HPCCSystems/hpcc-data/thor/thor/base/ock_p2/inc/20251203_2/monitoring_auto_unsuppressed_backup._$P$_of_400: Orphans ignored as thor::base::ock_p2::inc::20251203_2::monitoring_auto_unsuppressed_backup exists"

I am not sure why this is not working. Can someone please investigate? The thors in alpha prod are dangerously low on space and having files not being removed is not helping.

Steps to Reproduce

1. Create file in thor
2. Wait until after expiry date
3. File not deleted

Expected Behavior

sasha should attempt to remove the file upon expiry

Actual Behavior

file is not removed

HPCC Systems Version

9.10.76

HPCC4J Module

wsclient - Web service clients for HPCC ESP services

HPCC4J Version

9.10.76

Java Version

Java 11

Environment/Configuration

https://alpha_prod_thor_esp.risk.regn.net:18012/

Code Sample

Additional Context

No response

Pre-submission Checklist

• I have checked the Common Issues and Solutions wiki

Test

---

🤖 AI Validation Response

• Ready for re-review (Check this box when you've addressed the feedback and want the AI to validate again)

Thank you for reporting this critical issue and providing such a thorough bug report!

Your submission contains all the details needed for investigation. However, this appears to be a duplicate of issue #48, which describes the same problem with Sasha not deleting expired files on the thor cluster.

📋 Action Required

1. Please review issue [BUG]: File expiry set on logical files and sasha is not removing them #48 to confirm if your situation matches.
2. If your case is different, comment on [BUG]: File expiry set on logical files and sasha is not removing them #48 with any unique details.
3. Follow or update [BUG]: File expiry set on logical files and sasha is not removing them #48 for progress and resolution.

ℹ️ How to Find This Information

Reviewing Related Issues

• Visit issue #48 and compare the described symptoms and environment to your own.
• If you have additional logs or context not present in [BUG]: File expiry set on logical files and sasha is not removing them #48, add them as a comment there.

Manual Workarounds

• Manually delete expired files from the thor cluster to free disk space.
• Review Sasha and XREF configuration for correct expiry and orphan handling.

🔍 Related Resources

• Related issue: [BUG]: File expiry set on logical files and sasha is not removing them #48 – Ongoing discussion and updates for Sasha file expiry on thor clusters.
• HPCC Systems Documentation – For platform configuration and known issues.

⚠️ Important Notes

• HPCC4J 9.10.76 and HPCC Platform 9.10.76 are compatible; Java 11 is supported.
• This may be a platform-side issue; consider reporting to HPCC Systems maintainers if not already tracked.
• Do not share sensitive information (e.g., passwords) in public comments.

---

Please update or follow issue #48 for further updates. If your case is unique, add your details there so we can keep the discussion centralized. Thank you for helping us improve HPCC4J!

---

[github-actions]

Thank you for the detailed bug report and for checking the documentation beforehand!

Your issue is well-documented and matches a previously reported problem with Sasha not deleting expired files on thor clusters. To keep the discussion focused and ensure updates are tracked in one place, please refer to the related issue below.

📋 Action Required

1. Review issue [BUG]: File expiry set on logical files and sasha is not removing them #48 for updates and discussion on this problem.
2. If your situation differs in any way, please comment on [BUG]: File expiry set on logical files and sasha is not removing them #48 with additional details.
3. As a temporary workaround, manually delete expired files from the thor cluster to free up disk space.

ℹ️ How to Find This Information

Reviewing Related Issues

• Visit issue #48 to see if your scenario matches and to follow progress.

Manual Cleanup Steps

• Use HPCC file management tools or direct access to remove expired files from the thor cluster.
• Monitor Sasha logs for orphan warnings and verify file status on disk.

🔍 Related Resources

• Related issue: [BUG]: File expiry set on logical files and sasha is not removing them #48 - Ongoing discussion about Sasha not removing expired files on thor clusters.
• Possible workaround: Manual deletion of expired files until a fix is available.

⚠️ Important Notes

• This appears to be a platform-level issue with Sasha on HPCC Systems 9.10.76, not a configuration or usage error.
• All your provided versions (HPCC4J 9.10.76, HPCC Systems 9.10.76, Java 11) are compatible and supported.
• Please avoid sharing sensitive information such as passwords or internal URLs in public comments.

---

Please update or follow issue #48 for further progress. If you have new information, add it there to help the investigation. Thank you for helping us improve HPCC4J and HPCC Systems!

[github-actions]

Thank you for reporting this critical issue and providing such a detailed bug report!

Your submission is complete and contains all the information needed for investigation. This appears to be a duplicate of issue #48, which describes the same problem with Sasha not removing expired files on the thor cluster.

📋 Action Required

1. Please follow progress and updates in issue #48.
2. If your situation differs in any way, add your details as a comment on [BUG]: File expiry set on logical files and sasha is not removing them #48.
3. If you discover a workaround or additional clues, please share them in the linked issue.

ℹ️ How to Find This Information

Reproducibility

If you can reproduce this on other clusters or HPCC versions, please mention it in #48 to help with diagnosis.

Version Details

• HPCC4J Version: Check your pom.xml or run mvn dependency:tree | grep hpccsystems
• HPCC Platform Version: Check the ESP web interface footer, run eclcc --version on the server, or query the WsTopology service.

🔍 Related Resources

• Issue #48 - Ongoing discussion and updates on Sasha file expiry behavior.
• HPCC Systems Documentation - General platform and Sasha configuration guidance.
• Possible workarounds:
  • Manually remove expired files from the thor cluster using HPCC ECL or admin tools.
  • Monitor disk usage and expired file status with custom scripts until Sasha behavior is confirmed.

⚠️ Important Notes

• No configuration or version issues detected; your environment meets all requirements.
• This is likely a platform-side or Sasha configuration issue rather than an HPCC4J bug.
• Please do not share sensitive credentials or passwords in public issues.

---

Please track updates in issue #48. If you have new information or your case is different, let us know there so we can assist further. Thank you for helping improve HPCC4J and HPCC Systems!