Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Comments

Feature/parallel fetch data#267

Open
tstelzer wants to merge 13 commits intocontentlayerdev:main from
tstelzer:feature/parallel-fetch-data
Open

Feature/parallel fetch data #267
tstelzer wants to merge 13 commits intocontentlayerdev:main from
tstelzer:feature/parallel-fetch-data

Conversation

@tstelzer
Copy link
Contributor

@tstelzer tstelzer commented Jul 12, 2022
edited
Loading

This PR begins to parallelize the (presumably) computationally most expensive step: parsing source files. It does so by introducing a worker pool, which the main thread delegates to. Summarized changes:

  • Wrapped makeCacheItemFromFilePath with a thread worker pool via piscina.
  • Rewrote signature of makeCacheItemFromFilePath to make input/result serializable.
  • (style) Introduced prettier plugin to eslint. Arguably, this doesn't need to be part of this PR 🤔

This PR is considered WIP, with some things left to do:

Mandatory:

  • Serialize/deserialize user-supplied plugins. Currently, if plugins are configured, every thread will panic, as plugins can't be loaded.
  • Serialize/deserialize failures caused in the thread, specifically, anything that is nested in unknown errors (explicit errors should already be handled).
  • Optimize. We're only getting ("on my machine") a ~2x speedup compared to single-threaded. Might be related to my machine, but there's probably much more we can get here. Maybe related to the overhead of serialization? Playing around with thread-pool size or queue size might be profitable as well.

Optional:

  • Generalize worker pool code. Right now, the code is very specific to makeCacheItemFromFilePath.
  • Improve otel tracing in thread. Currently, it's starting an entirely separate trace.

schickling and goustaff reacted with hooray emoji
tstelzer added 13 commits July 7, 2022 12:21
EXPERIMENTAL: Consider this commit a chain of experimental changes,
which likely break the pacakge.
This fixes an issue where the piscina worker pool crashes with
`FATAL ERROR: v8::FromJust Maybe value is Nothing`
Supposedly, this is the worker running out of memory? No idea. As a fix,
we explicitly pick boundaries for queue and thread size in piscina.
Copy link

changeset-bot bot commented Jul 12, 2022

⚠️ No Changeset found

Latest commit: 5c827a3

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

help wanted Extra attention is needed meta: never-stale

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

AltStyle によって変換されたページ (->オリジナル) /