Skip to content

Freeze the resolved interpreter for child processes #796

Description

@CarloLucibello

When PythonCall resolves its interpreter via CondaPkg, each child process that loads
PythonCall re-resolves the same CondaPkg environment from scratch. Spawn several at once —
e.g. Distributed workers for parallel data loading (MLUtils DataLoader(...; num_workers=N)
over a PythonCall-backed dataset) — and they serialize on CondaPkg's lock, spamming
CondaPkg: Waiting for lock to be freed and redoing work the parent already did.

PythonCall already has the fix, but gates it behind CI=true and flags it as a hack
(src/C/context.jl, v0.9.35):

# HACK: If we are using CondaPkg, prevent child processes from using it by explicitly
# setting the executable. ...
# A better solution may be to use some environment variable to "freeze" CondaPkg in
# child processes.
# Only done when CI=true since it's a hack.
if (get(ENV, "CI", "false") == "true") && (CTX.which === :CondaPkg)
    ENV["JULIA_PYTHONCALL_EXE"] = CTX.exe_path::String
end

Since JULIA_PYTHONCALL_EXE is exactly the "freeze" env var the comment asks for (child
processes inherit ENV, and getpref_exe reads it), the same line solves the general case.

Proposal

Set JULIA_PYTHONCALL_EXE whenever CTX.which === :CondaPkg, not only under CI — ideally
behind a preference (e.g. freeze_child_processes, default true) so it can be disabled.
get! already leaves a user-set value untouched.

Workaround

A downstream package can do this itself after init:

if PythonCall.C.CTX.which === :CondaPkg
    get!(ENV, "JULIA_PYTHONCALL_EXE", PythonCall.python_executable_path()::String)
end

I've done this in JuliaGenAI/HuggingFaceDatasets.jl#64

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions