-
-
Notifications
You must be signed in to change notification settings - Fork 422
Improve json decoding #1402
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve json decoding #1402
Conversation
21db43c to
ee5eeca
Compare
ee5eeca to
470e2b9
Compare
470e2b9 to
fdfb53b
Compare
Results: $ go test -v -benchmem -bench BenchmarkIndexParsing github.com/arduino/arduino-cli/arduino/libraries/librariesindex === RUN TestIndexer --- PASS: TestIndexer (0.16s) goos: linux goarch: amd64 pkg: github.com/arduino/arduino-cli/arduino/libraries/librariesindex cpu: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz BenchmarkIndexParsingStdJSON BenchmarkIndexParsingStdJSON-8 5 214872730 ns/op 94.52 MB/s 58956539 B/op 418973 allocs/op BenchmarkIndexParsingEasyJSON BenchmarkIndexParsingEasyJSON-8 16 69215472 ns/op 293.42 MB/s 56162664 B/op 418966 allocs/op PASS ok github.com/arduino/arduino-cli/arduino/libraries/librariesindex 4.442s easyjson is 3x faster.
fdfb53b to
878168e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be possible to set up some infrastructure for the code generation?
- Task that runs the generation command
- GitHub Actions workflow that runs the task and then checks for a diff (which would indicate that the generated code has gone out of sync)
I did that for the generated code in Arduino Lint and found it to be very useful. The benefits should be even more here where there are more contributors.
Probably should also be documented in the contributor guide:
https://github.com/arduino/arduino-cli/blob/master/docs/CONTRIBUTING.md
@ubidefeo
ubidefeo
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have run the build from this PR for about one hour on several projects and have not been able to break it.
From a usage point of view it LGTM
f4d814a to
ab68a8e
Compare
ab68a8e to
7f393b1
Compare
Co-authored-by: per1234 <accounts@perglass.com>
For the record: we are using a patched version of easyjson until this patch is accepted upstream: mailru/easyjson#372
I found that the standard golang JSON decoder is very slow. This is particularly noticeable because the library_index.json is constantly growing (currently we are at ~20Mb) and the parsing time adds up at every invocation of the cli.
This PR:
easyjsonlibrary for parsing the library_index.jsonHere the result: