cc @mclark as well

Thank you for this really thoughtful and interesting analysis! I'm collecting my thoughts and chatting more with my team, and I see a ton of opportunity here! I'll follow up with more thoughts soon, and I'd personally be excited to help solve this problem you're facing.

I am adding this as a placeholder comment for folks to add a 👍 if they use privacy protections today

I love option 2 for a variety of reasons, most importantly maybe that it elevates work on a public API in visibility.

The reason I have been ok with what we have today is that I actually believe that the investment in a public API shouldn't just be making certain constants public (for many packages). Rather, I believe there should be a deliberate investment in designing the API for the use-cases of other parts of the app.

I'd explicitly vote against 4 -- even if some changes made to eliminate a privacy violation are suboptimal (just moving files around, etc.), it still provides a mechanism to be more explicit about a package's public API, and we've found value in that even before intentionally redesigning that public API. It's a core part of Packwerk's value proposition for us.

We're currently working around the junk drawer problem by defining subpackages -- we'll have a top-level (e.g.) UserService pack, then one in user_service/app/models (with public_path set to .) that other packs can explicitly depend on if they need direct access to the models outside of the UserService's public API. This points to a possible 5th option, where Packwerk has better subpackage support -- the main change required being a way to inherit package membership. Maybe something like:

# UserService package.yml
enforce_privacy: true
enforce_dependencies: true
# don't need to specify that UserService depends on UserService models, b/c inheritance
# UserService app/models package.yml
public_path: .
enforce_privacy: true
enforce_dependencies: true
subpackage: true # activates inheritance behavior; inheritance would flow upwards until it hits a package.yml that _doesn't_ have subpackage: true
# don't need to specify that the models pack depends on other UserService subpackages, etc.
# BookService package.yml
enforce_privacy: true
enforce_dependencies: true
dependencies:
- packs/user_service/app/models # BookService depends only on UserService models, not on its public API endpoints or other subpackages
# ReviewService package.yml
enforce_privacy: true
enforce_dependencies: true
dependencies:
- packs/user_service # ReviewService depends on UserService and its subpackages

This doesn't completely address the issue, though, since it'd still be exposing all the models (in the case above) as part of the subpackage's public API 🤔

So far at Shopify, we didn't see much value in the privacy checks. Most of our packages have this option disabled and when we do have them enabled, people mostly fix this kind of violation mostly by moving files around, without improving the APIs, or, even worse, people just record the deprecations and forget about them.

This sounds to me like a cultural issue.
There are 2 main reasons why we decided to start using Packwerk in my org:

1. Have clear ownership over the code (i.e. each package is owned by one team)
2. Have clear boundaries between packages.

The former is a great reason to embrace Packwerk and it also helps reduce the cognitive load for new starters, but if that's all you're trying to achieve, then I agree that enforce_privacy isn't going to be interesting/useful for you.

In our case though, we also wanted to take this opportunity to apply Hexagonal Architecture patterns inside our application and decouple the different packages to reduce the entanglement in our application.
I always tell my team that, in an ideal world, if we decided to take one of our packages/components and make it a (micro)service, that should be a really easy thing to do.

But if the app/public folder is full of models, controllers and other framework-specific stuff, then I really don't see how this could possibly be achieved.
Resolving each privacy violation without simply moving files around is exactly how the codebase can evolve and improve over time, and the team can learn and experiment with different patterns to properly decouple the packages.

I love this image from the Packwerk USAGE doc, because I think it illustrates very clearly the ultimate objective of using Packwerk to achieve a healthier codebase:

Caveat: I designed the first versions of packwerk, but I am not currently using it. You may treat my opinions as historical context.

Origins

Packwerk was first intended to be purely a dependency management tool. We added privacy enforcement to drive adoption, because that was what people were asking for. In the metaphor of carrots and sticks, privacy is sugar. It's easy to understand has broad appeal, but it may not actually be good for you. There are a lot of intricacies to making things private and public within a Rails application that can be, even excluding cultural issues, difficult to resolve.

However, I've been convinced back then and still am that dependency management is by far the more useful mechanism. It is a lot less likely to lead to bad design decisions. Also, it is much easier to establish conventions for public/private (e.g. in naming) than it is for dependencies. To emphasize, I do think the value proposition of packwerk is in dependencies, and privacy as an option is a nice cherry on top.

My Preference

For the above reasons, my preferred solution is 4. It allows us to focus the core of packwerk on the core value proposition, while also establishing the plugin mechanism and having people use the privacy implementation of their choice if they feel like it's the best option for them.

Explicitly Exporting Constants

I think explicitly exporting constants would only exacerbate the problem lined out by Rafael. In my opinion, having all kinds of things in the public folder is a smell. If we intentionally design and implement public interfaces, things in the public folder will be dedicated to that purpose. So there shouldn't be a broad array of types of things in there. Unless we're talking about cross-package inheritance - but I think that warrants a separate discussion.

If we change the mechanism from a public folder to "export" annotations, we will make the public interface a lot more implicit. IMO it will actually make it easier to say "I want to use this thing but I can't, so I'll just export it". Someone trying to judge the quality of the interface will not easily see that there are different kinds of things in the interface, because there isn't one place to look at. There also isn't one place to look at for people that want to use a package.

Cultural Problems

It was brought up that the problem discussed here is a cultural one.

IMO there are no purely cultural problems in software development, just as there are no purely technical ones above a certain scale.

Yes, there is a cultural component here. But tools shape culture (and vice versa), and packwerk was very much designed for that purpose. Part of this discussion should be how we want the privacy mechanism to shape people's behavior.

This has been a really interesting and thoughtful discussion, thank you everyone for participating and sharing your thoughts!

I have a proposal that I think might be satisfying based on what I'm reading in this thread. What if, as a first step, we support a plugin architecture to packwerk checkers and move privacy protection, with its behavior unchanged, into a gem. This would require no behavioral change for consumers – they simply bump packwerk to 3.0 and add the gem to their gem file, at which point packwerk should continue to operate as normal. This does a couple of things:

1. It can support a variety of other kinds of packwerk/modularization checks between packages, which is something Gusto has been invested in for a while (https://github.com/rubyatscale/package_protections) and could allow folks to extend packwerk in new and interesting ways.
2. It can allow shopify, or other organizations that don't find privacy enforcement valuable to not need to maintain its complexity. Clients have flexibility to choose what they want.
3. We can continue to have the discussion about how the privacy checker could work AND if there are strongly differing opinions on how it should work, packwerk could support different implementations of it without needing to fork all of packwerk. (In fact, privacy checking already supports multiple implementations of noting something as private or public via the public path and via enforce_privacy being set to a list of constants. No reason we couldn't support several options with a reasonable default if that is what best serves the community.)

How do people feel about having packwerk work with a plugin system and pulling privacy enforcement into a gem and iterating on its behavior independently?

Separately, I wanted to share a bit more what we discussed as a team at Gusto. Just to clarify -- I think that this is independent of the decision to move it into a plugin which is why this is a separate thread.

At Gusto we are really invested in the privacy checking component of packwerk, and want to continue to improve it. We feel like there are multiple separate, independent problems we are trying to solve and requirements we are trying to enforce:

1. It should be obvious and clear to consumers what public API is available.
2. The mechanism to mark something as public should be easy to use while also discouraging spurious, thoughtless promotion to public API.
3. Packwerk should not be hard to use in a vanilla Rails app, and it should at least have a configuration option/path that works with a vanilla Rails app.
4. We would like to be opinionated about how public API is expressed while...
5. ... while offering different clients with different needs a rich way to express their public API

I like this framing because it decouples the problems from the solutions (annotations, public folder, etc.). Are there other problems/requirements that I'm missing here?

If not, what we generally landed on as a team was the use of a magic comment at the top of a file to denote a file as public. Here's how it does or can solve the problems above:

1. I was imagining a CLI like bin/not_sure_where_the_cli_would_live public_api path/to/package which would open up ruby-doc like website that lists out the fils that are public.
2. As discussed, this is a cultural AND tooling issue. Appropriately, at Gusto we've approached this from a sociotechnical point of view. This means (A) we have (or want) tooling to prevent certain types of public API, such as no ActiveRecord at the public boundary and (B) invest heavily in social and cultural work to educate and evangelize what quality APIs look like. More importantly, use_packwerk already makes moving something to public trivial, so I don't think this tactic changes that.
3. This approach is invisible to Rails and all Rails conventions can be followed as normal.
4. Well this is the opinion after all
5. I'm not 100% convinced yet, but I am thinking we might want to keep the public folder around as an option, but perhaps migrate it to option (2). My preference would be to see if we can solve everything that people want from a public folder using other tactics. I'm very open to the idea of keeping around the behavior as is and I can see it as still being valuable for a lot of organizations.

@rafaelfranca

I've created three PRs to move towards extracting privacy out as a plugin:
#232
#233
#234

Once these land, I think I'll have enough to extract it out as a gem plugin, then I can put up a PR in packwerk that exposes some simple interface(s) for registering the plugin. At that point I can delete the implementation from packwerk and reference the gem.

At that point if we want, we could choose to remove it from packwerk entirely and publish a major version bump letting folks know they need to include the extracted plugin in their Gemfile to preserve the existing behavior.

This is an old issue - but one that I've pointed people to recently for reasons to reconsider privacy enforcment. If you still want to pursue it and the folder structure of the second proposal is interesting - you can achieve it with a snippet like this:

# config/initializers/public_zeitwerk_components.rb
# assuming packages are stored in a components/ directory
Rails.application.config.before_initialize do
 Dir.glob(Rails.root.join("components/*/public/*/").each do |dir|
 Rails.autoloaders.main.push_dir(dir)
 end
end