-
Notifications
You must be signed in to change notification settings - Fork 47
importers: high cpu usage during git clone phase #2182
helio-frota
started this conversation in
General
-
not 100% sure but I think we can use perf + hotspot to indentify who is causing this cpu high usage
- enable the
osv-githubimporter - edit
Cargo.tomladding this content at the end of the file
[profile.release] debug = true
- ofc build with
--release - run the importer and then
perf record -F99 --call-graph dwarf -p PID_here, collect that until the cpu decrease sudo dnf install hotspotor download and run https://github.com/KDAB/hotspot/releases appImage using/opening the generatedperf.datafile- see what happened
Beta Was this translation helpful? Give feedback.
All reactions
Replies: 3 comments
-
Beta Was this translation helpful? Give feedback.
All reactions
0 replies
-
I played a bit with those global git2 options and unfortunatelly I found no differences when watching the cpu usage
Changes:
➜ trustify git:(main) ✗ git diff diff --git a/modules/importer/src/runner/common/walker/git.rs b/modules/importer/src/runner/common/walker/git.rs index 0934c948..0cd76355 100644 --- a/modules/importer/src/runner/common/walker/git.rs +++ b/modules/importer/src/runner/common/walker/git.rs @@ -492,7 +492,11 @@ mod test { #[test_log::test(tokio::test)] async fn test_walker() -> Result<(), anyhow::Error> { - const SOURCE: &str = "https://github.com/RConsortium/r-advisory-database"; + git2::opts::enable_caching(false); + git2::opts::strict_hash_verification(false); + git2::opts::strict_object_creation(false); + // const SOURCE: &str = "https://github.com/RConsortium/r-advisory-database"; + const SOURCE: &str = "https://github.com/github/advisory-database"; let path = PathBuf::from(format!( "{}target/test.data/test_walker.git", env!("CARGO_WORKSPACE_ROOT")
Run the test and take a look on the CPU usage
➜ trustify git:(main) ✗ cargo nextest run test_walker Compiling trustify-module-importer v0.4.0-beta.1 (/home/heliofrota/Desktop/tc/trustify/modules/importer) Compiling trustify-server v0.4.0-beta.1 (/home/heliofrota/Desktop/tc/trustify/server) Compiling xtask v0.4.0-beta.1 (/home/heliofrota/Desktop/tc/trustify/xtask) Compiling trustify-trustd v0.4.0-beta.1 (/home/heliofrota/Desktop/tc/trustify/trustd) Finished `test` profile [unoptimized + debuginfo] target(s) in 7.85s ──────────── Nextest run ID a941fd1b-b3ce-40c6-b24a-a27439586106 with nextest profile: default Starting 2 tests across 36 binaries (670 tests skipped) PASS [ 1.231s] trustify-module-importer runner::common::walker::git::test::test_walker_fail_escape SLOW [> 60.000s] trustify-module-importer runner::common::walker::git::test::test_walker SLOW [>120.000s] trustify-module-importer runner::common::walker::git::test::test_walker SLOW [>180.000s] trustify-module-importer runner::common::walker::git::test::test_walker SLOW [>240.000s] trustify-module-importer runner::common::walker::git::test::test_walker
Beta Was this translation helpful? Give feedback.
All reactions
0 replies
-
feel free to try https://github.com/helio-frota/git2BBQ?tab=readme-ov-file#git2bbq 👍
Beta Was this translation helpful? Give feedback.
All reactions
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment