Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

extensions: add sync, download functionalities #11814

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shwstppr wants to merge 29 commits into apache:main
base: main
Choose a base branch
Loading
from shwstppr:feature-sync-ext

Conversation

@shwstppr
Copy link
Contributor

@shwstppr shwstppr commented Oct 9, 2025 •
edited
Loading

Description

This change introduces support for:

  • synchronizing extension files across management servers in a clustered environment.
  • download extension files as an archive (from a desired management server in a clustered environment). If needed (cases when MS is not directly accessible), download can be made to work via SSVM and secondary storage.

Key changes:

  • Adds new syncExtension API to trigger synchronization for a selected extension.
  • Implements util to create .tgz archive of the extension directory or selected files for sync.
  • For sync, sends DownloadAndSyncExtensionFilesCommand to peer management servers. Handles archive download, staging, and extraction on the receiver side.
  • Adds new downloadExtension API to allow downloading extension files as an archive.
  • Implements util to create .zip archive of the extension directory for download.
  • For forcing download via secondary storage, global configuration - extension.share.download.use.secondary.storage can be used and set to true.
  • Generates signed share URL with HMAC signature and expiry.
  • URL expiry and file cleanup can be controlled via global configuration - extension.share.link.validity.interval
  • Adds Sync Extension and Download Extension actions in the UI Extensions view.
  • Updates events, logging, and cleanup of temporary share files.
  • Refactors all filesystem-related code to the extensions framework layer.
  • Checksum for extensions is now calculated and compared for all files in the extension directory.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

For Sync:
image 

(localcloud) 🐱 > sync extension id=3cf82f18-89c5-4acc-8f46-8d42da139f30 sourcemanagementserverid=0d8d73fa-9241-49aa-8d7e-7642f143ce66
{
 "success": true
}
(localcloud) 🐱 > sync extension id=3cf82f18-89c5-4acc-8f46-8d42da139f30
💩 Missing required parameters: sourcemanagementserverid
(localcloud) 🐱 > sync extension -h
syncExtension: To sync the extension files from one management server to other management server(s)
This API is asynchronous.
Required params: id, sourcemanagementserverid, 
API Params Type Description
========== ==== ===========
files list List of files to sync. Specify absolute o
 r relative paths. If not provided, all e
 xtension files will be synced.
id uuid ID of the extension
sourcemanagementserverid uuid ID of the management server from which fi
 les are to be synced
targetmanagementserverids list the IDs of the management servers to whic
 h the extension files are to be synced. 
 If not specified, the files will be sync
 ed to all management servers

For Download:

image image 
(localcloud) 🐱 > download extension -h
downloadExtension: To download the extension files as an archive
This API is asynchronous.
Required params: id, 
API Params Type Description
========== ==== ===========
id uuid ID of the extension
managementserverid uuid ID of the management server from which fi
 les are to be downloaded
(localcloud) 🐱 > download extension id=3cf82f18-89c5-4acc-8f46-8d42da139f30
{
 "extension": {
 "id": "3cf82f18-89c5-4acc-8f46-8d42da139f30",
 "managementserverid": "0d8d73fa-9241-49aa-8d7e-7642f143ce66",
 "managementservername": "qa1-4.22-test-61f54c88-kvm-mgmt1",
 "name": "test",
 "url": "http://172.120.0.228:8080/share/extensions/test-1760334975686.zip?exp=1760338575"
 }
}
(localcloud) 🐱 > download extension id=3cf82f18-89c5-4acc-8f46-8d42da139f30 managementserverid=5ccb810f-4a31-4b8d-a36c-2ebb7e87cb26 
{
 "extension": {
 "id": "3cf82f18-89c5-4acc-8f46-8d42da139f30",
 "managementserverid": "5ccb810f-4a31-4b8d-a36c-2ebb7e87cb26",
 "managementservername": "422-test-mgmt2",
 "name": "test",
 "url": "http://172.120.0.83:8080/share/extensions/test-1760334994612.zip?exp=1760338594"
 }
}

How Has This Been Tested?

How did you try to break this feature and the system with this change?

sudo87 reacted with rocket emoji
Copy link

codecov bot commented Oct 9, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 61.69031% with 621 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.02%. Comparing base (a5b6bc3) to head (21419d1).

Files with missing lines Patch % Lines
...extensions/manager/ExtensionsShareManagerImpl.java 76.47% 66 Missing and 50 partials ⚠️
...sions/manager/ExtensionsFilesystemManagerImpl.java 65.35% 64 Missing and 33 partials ⚠️
...work/extensions/manager/ExtensionsManagerImpl.java 73.14% 24 Missing and 34 partials ⚠️
...amework/extensions/ExtensionArchiveDataObject.java 12.50% 48 Missing and 1 partial ⚠️
...dstack/storage/to/DownloadableArchiveObjectTO.java 0.00% 48 Missing ⚠️
.../cloudstack/utils/server/ServerPropertiesUtil.java 0.00% 39 Missing ⚠️
...om/cloud/storage/download/DownloadMonitorImpl.java 0.00% 31 Missing ⚠️
...oudstack/storage/template/DownloadManagerImpl.java 0.00% 31 Missing ⚠️
...dstack/storage/image/BaseImageStoreDriverImpl.java 0.00% 28 Missing ⚠️
...pache/cloudstack/utils/filesystem/ArchiveUtil.java 69.04% 20 Missing and 6 partials ⚠️
... and 17 more
Additional details and impacted files 
@@ Coverage Diff @@
## main #11814 +/- ##
============================================
+ Coverage 17.89% 18.02% +0.12% 
- Complexity 16092 16313 +221 
============================================
 Files 5936 5951 +15 
 Lines 532734 534054 +1320 
 Branches 65165 65307 +142 
============================================
+ Hits 95347 96263 +916 
- Misses 426711 426984 +273 
- Partials 10676 10807 +131
Flag Coverage Δ
uitests 3.67% <ø> (-0.01%) ⬇️
unittests 19.13% <61.69%> (+0.13%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
 

This change introduces support for synchronizing extension files across management servers in a clustered environment.
- Adds new syncExtension API to trigger synchronization for a selected extension.
- Implements service to create .tgz archive of the extension directory or selected files.
- Generates signed share URL with HMAC signature and expiry.
- Sends DownloadAndSyncExtensionFilesCommand to peer management servers.
- Handles archive download, staging, and extraction on receiver side.
- Adds Sync Extension action in the UI Extensions view.
- Updates events, logging, and cleanup of temporary share files.
Refactors all filesystem related code to extensions framework layer.
Checksum for extensions is now calculated and compared for all files in
the extension directory.
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
shwstppr added 11 commits October 10, 2025 10:33 
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
@shwstppr shwstppr changed the title (削除) extensions: add sync functionality (削除ここまで) (追記) extensions: add sync, download functionalities (追記ここまで) Oct 13, 2025
@apache apache deleted a comment from blueorangutan Oct 13, 2025
@apache apache deleted a comment from blueorangutan Oct 13, 2025 
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Copy link

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

 

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Copy link

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

 

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Copy link
Contributor Author

@blueorangutan package

Copy link

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16208

Copy link

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

 

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Copy link
Contributor

@blueorangutan package

Copy link

@borisstoyanov a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16643

Copy link
Contributor

@harikrishna-patnala harikrishna-patnala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shwstppr this looks great. Got a chance to test, sync and downloads are working fine. Though link expiry does not seem to be working as expected.
I've enabled this "extension.share.download.use.secondary.storage" and set "extension.share.link.validity.interval" to 30.

I've made 3 attempts of download, folders and the last file remains exists after expiry time

Image

Copy link

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

Copy link
Contributor Author

@harikrishna-patnala thanks a lot for review/testing. Will have a look at the cleanup flow.

 

url
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
Copy link
Contributor Author

@shwstppr this looks great. Got a chance to test, sync and downloads are working fine. Though link expiry does not seem to be working as expected. I've enabled this "extension.share.download.use.secondary.storage" and set "extension.share.link.validity.interval" to 30.

I've made 3 attempts of download, folders and the last file remains exists after expiry time
Image

@harikrishna-patnala this is a generic issue with download url cleanup on SSVM. While the symlink was removed, temp directory was not getting removed.

2026年01月31日T07:36:09,043 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) Request:Seq 4-7227995926952935435: { Cmd , MgmtId: 90520731363323, via: 4, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.storage.DeleteEntityDownloadURLCommand":{"path":"volumes/2/4/97b241b4-4d31-469f-b4a4-a2fcab2606ff.qcow2","extractUrl":"http://172.120.1.2/userdata/88a99588-c0e0-4691-96c7-c7e6534d9fbe/t-dd.qcow2","type":"VOLUME","parentPath":"c794fe3b-bedb-3e8e-a5c6-c5a2b28a2065","accountId":"0","wait":"0","bypassHostMaintenance":"false"}}] }
2026年01月31日T07:36:09,044 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) Processing command: com.cloud.agent.api.storage.DeleteEntityDownloadURLCommand
2026年01月31日T07:36:09,044 DEBUG [storage.resource.NfsSecondaryStorageResource] (AgentRequest-Handler-5:[]) Executing command "DeleteEntityDownloadURLCommand" [{"path":"volumes/2/4/97b241b4-4d31-469f-b4a4-a2fcab2606ff.qcow2","extractUrl":"http://172.120.1.2/userdata/88a99588-c0e0-4691-96c7-c7e6534d9fbe/t-dd.qcow2","type":"VOLUME","parentPath":"c794fe3b-bedb-3e8e-a5c6-c5a2b28a2065","accountId":0,"contextMap":{},"wait":0,"bypassHostMaintenance":false}].
2026年01月31日T07:36:09,044 WARN [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) handleDeleteEntityDownloadURLCommand Path:volumes/2/4/97b241b4-4d31-469f-b4a4-a2fcab2606ff.qcow2 Type:VOLUME
2026年01月31日T07:36:09,045 DEBUG [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) Executing command [/bin/bash -c unlink /var/www/html/userdata/88a99588-c0e0-4691-96c7-c7e6534d9fbe/t-dd.qcow2 ].
2026年01月31日T07:36:09,054 DEBUG [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) Successfully executed process [5938] for command [/bin/bash -c unlink /var/www/html/userdata/88a99588-c0e0-4691-96c7-c7e6534d9fbe/t-dd.qcow2 ].
2026年01月31日T07:36:09,054 WARN [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) Deleted volume /mnt/SecStorage/c794fe3b-bedb-3e8e-a5c6-c5a2b28a2065/volumes/2/4/97b241b4-4d31-469f-b4a4-a2fcab2606ff.qcow2
2026年01月31日T07:36:09,055 DEBUG [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) Executing command [/bin/bash -c rm -rf /mnt/SecStorage/c794fe3b-bedb-3e8e-a5c6-c5a2b28a2065/volumes/2/4/97b241b4-4d31-469f-b4a4-a2fcab2606ff.qcow2 ].
2026年01月31日T07:36:09,063 DEBUG [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) Successfully executed process [5939] for command [/bin/bash -c rm -rf /mnt/SecStorage/c794fe3b-bedb-3e8e-a5c6-c5a2b28a2065/volumes/2/4/97b241b4-4d31-469f-b4a4-a2fcab2606ff.qcow2 ].
2026年01月31日T07:36:09,063 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) Seq 4-7227995926952935435: { Ans: , MgmtId: 90520731363323, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":"true","details":"","wait":"0","bypassHostMaintenance":"false"}}]
root@s-2-VM:/var/www/html/userdata# ls -lrt
total 20
drwxr-xr-x 2 www-data www-data 4096 Jan 31 06:08 9142b6e3-f6c6-48de-86b6-6003eb147d2c
drwxr-xr-x 2 www-data www-data 4096 Jan 31 06:10 85378760-8b68-4c60-81fc-11ddbe59fd85
drwxr-xr-x 2 www-data www-data 4096 Jan 31 06:28 a3276a6b-03aa-44d0-aeef-183da3b214c4
drwxr-xr-x 2 www-data www-data 4096 Jan 31 06:35 0cfcf0a0-58f8-471b-8864-9b8a2b754da3
drwxr-xr-x 2 www-data www-data 4096 Jan 31 07:36 88a99588-c0e0-4691-96c7-c7e6534d9fbe
root@s-2-VM:/var/www/html/userdata# ls -lrt 88a99588-c0e0-4691-96c7-c7e6534d9fbe
total 0

I've added a fix and created #12562

Copy link
Contributor Author

@blueorangutan package

Copy link

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16662

Copy link
Contributor Author

@blueorangutan test

Copy link

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

Copy link

[SF] Trillian test result (tid-15347)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 53990 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11814-t15347-kvm-ol8.zip
Smoke tests completed. 147 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestListIdsParams>:teardown Error 1.12 test_list_ids_parameter.py
test_01_snapshot_root_disk Error 5.03 test_snapshots.py
test_02_list_snapshots_with_removed_data_store Error 47.81 test_snapshots.py
test_02_list_snapshots_with_removed_data_store Error 47.82 test_snapshots.py
ContextSuite context=TestSnapshotStandaloneBackup>:teardown Error 31.70 test_snapshots.py
test_01_snapshot_usage Error 25.73 test_usage.py
test_01_vpn_usage Error 1.10 test_usage.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@harikrishna-patnala harikrishna-patnala harikrishna-patnala left review comments

Copilot code review Copilot Copilot left review comments

Assignees

No one assigned

Projects

None yet

Milestone

4.23.0

Development

Successfully merging this pull request may close these issues.

AltStyle によって変換されたページ (->オリジナル) /