hey @ctrueden, I'm struggling to run the jep tests locally (on my mac). I have openjdk version "11.0.27" 2025年04月15日 but still get:

--> tests/it/java_heap.py [OK]
The operation couldn’t be completed. Unable to locate a Java Runtime.
Please visit http://www.java.com for information on installing Java.
Traceback (most recent call last):
 File "/Users/talley/dev/self/scyjava/tests/it/jvm_version.py", line 11, in <module>
 before_version = scyjava.jvm_version()
 File "/Users/talley/dev/self/scyjava/src/scyjava/_jvm.py", line 70, in jvm_version
 default_jvm_path = jpype.getDefaultJVMPath()
 File "/Users/talley/dev/self/scyjava/.venv/lib/python3.13/site-packages/jpype/_jvmfinder.py", line 70, in getDefaultJVMPath
 return finder.get_jvm_path()
 ~~~~~~~~~~~~~~~~~~~^^
 File "/Users/talley/dev/self/scyjava/.venv/lib/python3.13/site-packages/jpype/_jvmfinder.py", line 184, in get_jvm_path
 jvm = method()
 File "/Users/talley/dev/self/scyjava/.venv/lib/python3.13/site-packages/jpype/_jvmfinder.py", line 311, in _javahome_binary
 return subprocess.check_output(
 ~~~~~~~~~~~~~~~~~~~~~~~^
 ['/usr/libexec/java_home']).strip()
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/Users/talley/.local/share/uv/python/cpython-3.13.3-macos-aarch64-none/lib/python3.13/subprocess.py", line 472, in check_output
 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
 ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 **kwargs).stdout
 ^^^^^^^^^
 File "/Users/talley/.local/share/uv/python/cpython-3.13.3-macos-aarch64-none/lib/python3.13/subprocess.py", line 577, in run
 raise CalledProcessError(retcode, process.args,
 output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/libexec/java_home']' returned non-zero exit status 1.

it's a bit unclear to me whether I should be trying to get these to work at all with jep, or just add a skip to the test

As a workaround, you could try setting JAVA_HOME? Then maybe jpype wouldn't try to invoke the java_home command. Out of curiosity: what does /usr/libexec/java_home -V say when you run it from the CLI?

got that working... and commented out the "don't run on macos cause it's flaky" bit, and now get this:

-------------------------------------------
| Testing Jep mode (Python inside Java) |
-------------------------------------------
DEBUG 2025年05月01日 08:20:23,154: Using settings: {'m2repo': '/Users/talley/.m2/repository', 'cachedir': '/Users/talley/.jgo', 'links': 'auto'}
DEBUG 2025年05月01日 08:20:23,155: Using repositories: {'scijava.public': 'https://maven.scijava.org/content/groups/public'}
DEBUG 2025年05月01日 08:20:23,155: Using shortcuts: {}
DEBUG 2025年05月01日 08:20:23,155: Returning expanded coordinate black.ninia:jep:jep.Run.
DEBUG 2025年05月01日 08:20:23,155: Returning expanded coordinate org.scijava:scijava-table.
INFO 2025年05月01日 08:20:23,155: First time start-up may be slow. Downloaded dependencies will be cached for shorter start-up times in subsequent executions.
DEBUG 2025年05月01日 08:20:23,155: Executing: ('/Users/talley/Library/Caches/cjdk/v0/misc-dirs/98cdba9371f93e1b5b8b95941b562d1647aecc21/apache-maven-3.9.9/bin/mvn', '-B', '-f', '/Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/pom.xml', 'dependency:resolve', '-X')
DEBUG 2025年05月01日 08:20:31,600: Relevant mvn output: [INFO] black.ninia:jep:jar:4.2.2:compile -- module jep (auto)
DEBUG 2025年05月01日 08:20:31,600: Linking source /Users/talley/.m2/repository/black/ninia/jep/4.2.2/jep-4.2.2.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/jep-4.2.2.jar with link_type auto
DEBUG 2025年05月01日 08:20:31,600: Linking source /Users/talley/.m2/repository/black/ninia/jep/4.2.2/jep-4.2.2.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/jep-4.2.2.jar with link_type hard
DEBUG 2025年05月01日 08:20:31,601: Relevant mvn output: [INFO] org.scijava:scijava-table:jar:1.0.2:compile -- module org.scijava.table [auto]
DEBUG 2025年05月01日 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/scijava-table/1.0.2/scijava-table-1.0.2.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-table-1.0.2.jar with link_type auto
DEBUG 2025年05月01日 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/scijava-table/1.0.2/scijava-table-1.0.2.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-table-1.0.2.jar with link_type hard
DEBUG 2025年05月01日 08:20:31,601: Relevant mvn output: [INFO] org.scijava:scijava-common:jar:2.89.0:compile -- module org.scijava [auto]
DEBUG 2025年05月01日 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/scijava-common/2.89.0/scijava-common-2.89.0.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-common-2.89.0.jar with link_type auto
DEBUG 2025年05月01日 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/scijava-common/2.89.0/scijava-common-2.89.0.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-common-2.89.0.jar with link_type hard
DEBUG 2025年05月01日 08:20:31,601: Relevant mvn output: [INFO] org.scijava:parsington:jar:3.0.0:compile -- module org.scijava.parsington [auto]
DEBUG 2025年05月01日 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/parsington/3.0.0/parsington-3.0.0.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/parsington-3.0.0.jar with link_type auto
DEBUG 2025年05月01日 08:20:31,601: Linking source /Users/talley/.m2/repository/org/scijava/parsington/3.0.0/parsington-3.0.0.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/parsington-3.0.0.jar with link_type hard
DEBUG 2025年05月01日 08:20:31,601: Relevant mvn output: [INFO] org.bushe:eventbus:jar:1.4:compile -- module eventbus (auto)
DEBUG 2025年05月01日 08:20:31,601: Linking source /Users/talley/.m2/repository/org/bushe/eventbus/1.4/eventbus-1.4.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/eventbus-1.4.jar with link_type auto
DEBUG 2025年05月01日 08:20:31,601: Linking source /Users/talley/.m2/repository/org/bushe/eventbus/1.4/eventbus-1.4.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/eventbus-1.4.jar with link_type hard
DEBUG 2025年05月01日 08:20:31,602: Relevant mvn output: [INFO] org.scijava:scijava-optional:jar:1.0.1:compile -- module org.scijava.optional [auto]
DEBUG 2025年05月01日 08:20:31,602: Linking source /Users/talley/.m2/repository/org/scijava/scijava-optional/1.0.1/scijava-optional-1.0.1.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-optional-1.0.1.jar with link_type auto
DEBUG 2025年05月01日 08:20:31,602: Linking source /Users/talley/.m2/repository/org/scijava/scijava-optional/1.0.1/scijava-optional-1.0.1.jar to target /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/scijava-optional-1.0.1.jar with link_type hard
DEBUG 2025年05月01日 08:20:31,602: class path: /Users/talley/.jgo/black.ninia/jep/RELEASE/598a6cd55c0501d03b71b81bc3431f66189c760bf00c14fed5e4f9e4b66a9b83/*
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
Current thread 0x00000001716a3000 (most recent call first):
 <no Python frame>

that's probably not the flaky bit right? looks like a poor setup of the python environment

i think i see the issue, i'll work on the jgo ... jep_test.py command directly and see if I can figure out what assumptions it's making about my environment setup

after digging a bit deeper into how jgo and jep itself is working, I'm skipping stubgen tests on jep for now. it seems extremely dependent on some careful manual setup of the python environment. I was able to get it to find my standard library, but then it lost the pure python parts of jep. After fixing that, it was unable to find my (editable) install of scyjava because it's not following .pth files in a standard way. I don't understand all the variables well enough yet (I don't understand who's doing the magic, whether it's jep.Run itself or the jgo command line), so don't know where to attack it

code-wise, this is ready to go... however, I don't expect it to be "human-interpretable" yet. There are many ways we could choose to document this and encourage/discourage its usage in various scenarios. Might be best to have a zoom about it so we can tinker with it together and discuss

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/fiji-friends-weekly-dev-update-thread/103718/94

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/poll-which-priorities-are-most-urgent-for-fijis-python-support/113172/1

Just a quick update: I removed the testing of jep mode awhile back, so that should no longer gum up the works here. @tlambert03 I'm hoping we can make a little time to revisit this PR some time next week.

yes that would be great! i do think a little chat would be super useful here

Writing some notes to self here (after having not looked at it for a while) that might help others understand what's going on here. This PR...

• adds a new CLI command called scyjava-stubgen. It's worth mentioning this is the first CLI command offered by scyjava (i.e. this PR adds project.scripts to pyproject.toml for the first time). This command can be used to generate type stubs for any maven endpoint, for example: scyjava-stubgen org.scijava:parsington:3.1.0.

• That CLI command just parses the arguments and calls the function scyjava._stubs.generate_stubs. (generate_stubs is a good candidate for a public function but I haven't touched the scyjava public API yet). Importantly, when called via the CLI it writes stubs by default to scyjava.types namespace itself. In other words it permantently writes stub files inside of the scyjava package (this could have permissions issues on some computers). For example, that parsington call above would create scyjava.types.org.scijava.parsington...

• After running that command (and having generated stubs) you can now import those stubs directly from the namespace:

  from scyjava.types.org.scijava.parsington import Function

  with full static type support in IDEs. Which is great! that's the main goal here. 🎉

• The important question to ask here, of course, is "when is the JVM started". That import alone does not start the JVM, instead, it returns a thin Proxy object that will start the JVM when an instance of that class is created (i.e. in the __new__ method). The logic for that is defined in scyjava._stubs.setup_java_imports. (That setup_java_imports function is used heavily by the generated type stubs, it generates the module level __getattr__ function that returns the Proxy objects imported from scyjava.types...)

• an important detail here (as usual) is that all things would need to be imported before any of them are instantiated

open questions:

to me, the biggest open question here is "who is responsible for generating stubs, and where do they go?".

possibilities include:

1. after the environment and scyjava is setup. the user could call that command line argument. That's fine and good, but not "portable" since it requires action after pip install to get the type stubs.
2. Which brings the second part of this PR (which should probably be broken into a new PR), which are hatch and setuptools plugins to allow developers to build and include stubs in their own packages. More on that elsewhere

after talking with @ctrueden ... something I will try to implement:

1. implement a sys.meta_path importer that sees imports from scyjava.types and dynamically generates types
2. implement a entry-point specification that allows packages to declare which endpoints they need (in order to generate dynamically imported types)

for this PR I will pull out everything by the stub generating mechanism, and do the above in another PR

ok @ctrueden, I think this is ready for final review. I've removed the hatch/setuptools plugins, and double checked all the documentation. will follow up on usage patterns in another PR. Let me know if you have any questions!

@ctrueden, any thoughts on this step? I think it stands alone as useful, without imposing/deciding too much on who is going to use it

@tlambert03 Yes, sorry! I started working on a revamp of jgo a few days ago, and it's been eating my entire brain. I will try to review and merge this PR this week!

Just a quick followup to say that I'm looking at this today. I am also considering whether this feature would be best in scyjava or jgo. It will definitely land somewhere, hopefully today.

Yeah I think it could still easily live anywhere... the only thing that might eventually pin it somewhere is a decision to establish a shared public namespace like "scyjava.types". Which is still a big question mark anyway (ie, it make stay forever as a local dev tool)

OK, after reflecting on it, this work is much more suitable here in scyjava than it would be in jgo. The jgo project does not depend on jpype, whereas scyjava is, in a nutshell, unioning the benefits of jgo and jpype. (And now with the stub generation: +stubgenj as well) The only other places I could imagine this work to go would be one of jpype or stubgenj—but we can always explore pushing it upstream later after we run it through its paces.

So, in testing this work, I ran into problems making the type completions work in VSCode and PyCharm. The issue was a mismatch between the prefixed classes e.g. scyjava.types.org.scijava.Context that get stubified with stubgenj's assumption that the Python class names will match the Java class names in the toplevel Python namespace.

Or to put it another way: the stub file at e.g. src/scyjava/types/org/scijava/__init__.pyi had imports beginning with import org.scijava., rather than import scyjava.types.org.scijava.. So the type reasoning logic would have problems, like hitting dead ends for every method return type. So you couldn't type e.g.:

from scyjava.types.org.scijava import Context
from scyjava.types.org.scijava.ui import UIService
ctx = Context()
ui = ctx.service(UIService).show<ctrl+space>

and see the UIService's show methods.

With 4f54611, this now works, because all the type references in the stubs now get rewritten to have the proper Python package prefix preceding the Java code's own package prefix.

@tlambert03 I'm not actually sure though: is the above code how you intended this feature to be used? Or did you want to write:

from org.scijava import Context
from org.scijava.ui import UIService
ctx = Context()
ui = ctx.service(UIService).show<ctrl+space>

without the scyjava.types Python package prefixes? If the latter, we would need to generate the stubs at the top level src, not at src/scyjava/types, right?

Secondly: there remains an issue with core Java classes: this commit does not rewrite imports like import java.lang to import scyjava.types.java.lang. I think maybe it should, but wanted your opinion.

Thirdly: due to my lack of knowledge in this area, I leaned rather heavily on Claude to write this patch, and it may have errors or stupidities. What do you think? Are we on the right track here?