de_pipe_template error

[sunzhen123-lab]

Hi, I'm currently attempting to run the code for Plan / Render pipelined mode from the quick start guide, shown below:
/isaac-sim/python.sh launcher.py \ --config configs/simbox/de_pipe_template.yaml
After running this program, it gets stuck due to certain issues – after a prolonged period of being unresponsive, it either exits automatically or the GPU utilization drops to zero. I’m struggling to troubleshoot this and suspect it might be related to Ray’s scheduling mechanism. Could you please help me figure out how to resolve this? All my other YAML scripts run without any issues; only this one is problematic. I haven’t modified any of the code or YAML files at all, so I’d be extremely grateful if you, the amazing author, could look into this for me!My computer is equipped with four NVIDIA RTX 4090 GPUs.
code 6 (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,168ms] [Warning] [omni.hydra] Rendering failed. (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,175ms] [Warning] [omni.hydra] Rendering failed. (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,182ms] [Warning] [omni.hydra] Rendering failed. (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,188ms] [Warning] [omni.hydra] Rendering failed. (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,193ms] [Warning] [omni.hydra] Rendering failed. (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,197ms] [Warning] [omni.hydra] Rendering failed. (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,210ms] [Warning] [omni.hydra] Rendering failed. (InnerPipe pid=15316) ndowing-glfw.plugin] GLFW initialization failed. (InnerPipe pid=15316) [3.114s] [ext: omni.kit.widget.options_menu-1.1.4] sta (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,222ms] [Error] [gpu.foundation.plugin] CUDA interop cannot be used! (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,223ms] [Error] [gpu.foundation.plugin] Texture creation failed for the device: 0. (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,223ms] [Error] [carb.scenerenderer-rtx.plugin] Failed to allocate 1280x720 LdrColor resource for device mask 0x1 (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,223ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6 (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,223ms] [Warning] [omni.hydra] Rendering failed. (InnerPipe pid=15322) t: omni.kit.actions.window-1.1.1] startup (InnerPipe pid=15322) 2026-03-23 12:48:32 [7,274ms] [Warning] [omni.kit.widget.cache_indicator.cache_state_menu] Unable to detect Omniverse Cache Server. Consider installing it for better IO performance. (InnerPipe pid=15322) 2026-03-23 12:48:33 [7,438ms] [Warning] [omni.kit.profiler.window] remove _SpanInstance.__lt__ and use insort 'key' arg instead (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,236ms] [Error] [gpu.foundation.plugin] CUDA interop cannot be used! (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,236ms] [Error] [gpu.foundation.plugin] Texture creation failed for the device: 0. (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,236ms] [Error] [carb.scenerenderer-rtx.plugin] Failed to allocate 1280x720 LdrColor resource for device mask 0x1 (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,236ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6 (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,248ms] [Error] [gpu.foundation.plugin] CUDA interop cannot be used! (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,248ms] [Error] [gpu.foundation.plugin] Texture creation failed for the device: 0. (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,248ms] [Error] [carb.scenerenderer-rtx.plugin] Failed to allocate 1280x720 LdrColor resource for device mask 0x1 (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,248ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6 (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,255ms] [Error] [gpu.foundation.plugin] CUDA interop cannot be used! (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,255ms] [Error] [gpu.foundation.plugin] Texture creation failed for the device: 0. (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,255ms] [Error] [carb.scenerenderer-rtx.plugin] Failed to allocate 1280x720 LdrColor resource for device mask 0x1 (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,255ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6 (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,269ms] [Error] [gpu.foundation.plugin] CUDA interop cannot be used! (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,269ms] [Error] [gpu.foundation.plugin] Texture creation failed for the device: 0. (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,269ms] [Error] [carb.scenerenderer-rtx.plugin] Failed to allocate 1280x720 LdrColor resource for device mask 0x1 (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,269ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6 (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,236ms] [Warning] [omni.hydra] Rendering failed. (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,248ms] [Warning] [omni.hydra] Rendering failed. (InnerPipe pid=15314) [13.268s] Simulation App Startup Complete (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,255ms] [Warning] [omni.hydra] Rendering failed. (InnerPipe pid=15314) 2026-03-23 12:48:33 [13,269ms] [Warning] [omni.hydra] Rendering failed. (InnerPipe pid=15316) rtup (InnerPipe pid=15316) [4.025s] [ext: omni.kit.viewport.menubar.di (InnerPipe pid=15322) (InnerPipe pid=15322) 2026-03-23 12:48:33 [8,001ms] [Warning] [omni.kvdb.plugin] Disabling key-value database because another kit process is locking it (InnerPipe pid=15322) 2026-03-23 12:48:34 [9,364ms] [Error] [carb.cudainterop.plugin] CUDA error 101: cudaErrorInvalidDevice - invalid device ordinal) [repeated 3x across cluster] (InnerPipe pid=15322) 2026-03-23 12:48:34 [9,364ms] [Error] [carb.cudainterop.plugin] Failed to set the CUDA device: -1 [repeated 3x across cluster] (InnerPipe pid=15316) Warp 1.2.1 initialized: (InnerPipe pid=15316) CUDA Toolkit 11.8, Driver 13.0 (InnerPipe pid=15316) Devices: (InnerPipe pid=15316) "cpu" : "x86_64" (InnerPipe pid=15316) "cuda:0" : "NVIDIA GeForce RTX 4090" (24 GiB, sm_89, mempool enabled) (InnerPipe pid=15316) Kernel cache: (InnerPipe pid=15316) /root/.cache/warp/1.2.1 (InnerPipe pid=15322) [7.935s] Simulation App Starting (InnerPipe pid=15316) |---------------------------------------------------------------------------------------------| [repeated 7x across cluster] (InnerPipe pid=15316) |=============================================================================================| [repeated 2x across cluster] (InnerPipe pid=15316) | 0 | NVIDIA GeForce RTX 4090 | Yes: 0 | | 24810 MB | 10de | 0 | (InnerPipe pid=15316) | | | | | | 0 | | [repeated 8x across cluster] (InnerPipe pid=15316) splay-106.0.2] startup (InnerPipe pid=15316) 2026-03-23 12:48:35 [4,985ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template (InnerPipe pid=15316) 2026-03-23 12:48:36 [5,894ms] [Warning] [omni.kit.widget.cache_indicator.cache_state_menu] Unable to detect Omniverse Cache Server. Consider installing it for better IO performance. (InnerPipe pid=15316) [5.918s] [e (InnerPipe pid=15316) xt: omni.kit.selection-0.1.4] startup (InnerPipe pid=15316) 2026-03-23 12:48:36 [6,034ms] [Warning] [omni.kit.profiler.window] remove _SpanInstance.__lt__ and use insort 'key' arg instead (InnerPipe pid=15316) [6.472s] [ext: omni.i (InnerPipe pid=15314) 2026-03-23 12:48:37 [16,991ms] [Fatal] [drake] Could not initialize pydrake: UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 6104: ordinal not in range(128) (InnerPipe pid=15314) (InnerPipe pid=15314) At: (InnerPipe pid=15314) /isaac-sim/kit/python/lib/python3.10/encodings/ascii.py(26): decode (InnerPipe pid=15314) /isaac-sim/kit/python/lib/python3.10/site-packages/pydrake/__init__.py(56): _execute_extra_python_code (InnerPipe pid=15314) <frozen importlib._bootstrap>(241): _call_with_frames_removed (InnerPipe pid=15314) <frozen importlib._bootstrap_external>(1176): create_module (InnerPipe pid=15314) <frozen importlib._bootstrap>(571): module_from_spec (InnerPipe pid=15314) <frozen importlib._bootstrap>(674): _load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1006): _find_and_load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1027): _find_and_load (InnerPipe pid=15314) <frozen importlib._bootstrap>(241): _call_with_frames_removed (InnerPipe pid=15314) <frozen importlib._bootstrap>(1078): _handle_fromlist (InnerPipe pid=15314) /isaac-sim/kit/python/lib/python3.10/site-packages/pydrake/__init__.py(147): <module> (InnerPipe pid=15314) <frozen importlib._bootstrap>(241): _call_with_frames_removed (InnerPipe pid=15314) <frozen importlib._bootstrap_external>(883): exec_module (InnerPipe pid=15314) <frozen importlib._bootstrap>(688): _load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1006): _find_and_load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1027): _find_and_load (InnerPipe pid=15314) <frozen importlib._bootstrap>(241): _call_with_frames_removed (InnerPipe pid=15314) <frozen importlib._bootstrap>(992): _find_and_load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1027): _find_and_load (InnerPipe pid=15314) /workspace/InternDataEngine/workflows/simbox/solver/kpam/optimization_problem.py(2): <module> (InnerPipe pid=15314) <frozen importlib._bootstrap>(241): _call_with_frames_removed (InnerPipe pid=15314) <frozen importlib._bootstrap_external>(883): exec_module (InnerPipe pid=15314) <frozen importlib._bootstrap>(688): _load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1006): _find_and_load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1027): _find_and_load (InnerPipe pid=15314) /workspace/InternDataEngine/workflows/simbox/solver/kpam/mp_terms.py(3): <module> (InnerPipe pid=15314) <frozen importlib._bootstrap>(241): _call_with_frames_removed (InnerPipe pid=15314) <frozen importlib._bootstrap_external>(883): exec_module (InnerPipe pid=15314) <frozen importlib._bootstrap>(688): _load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1006): _find_and_load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1027): _find_and_load (InnerPipe pid=15314) /workspace/InternDataEngine/workflows/simbox/solver/planner.py(9): <module> (InnerPipe pid=15314) <frozen importlib._bootstrap>(241): _call_with_frames_removed (InnerPipe pid=15314) <frozen importlib._bootstrap_external>(883): exec_module (InnerPipe pid=15314) <frozen importlib._bootstrap>(688): _load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1006): _find_and_load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1027): _find_and_load (InnerPipe pid=15314) /workspace/InternDataEngine/workflows/simbox/core/skills/artpreplan.py(11): <module> (InnerPipe pid=15314) <frozen importlib._bootstrap>(241): _call_with_frames_removed (InnerPipe pid=15314) <frozen importlib._bootstrap_external>(883): exec_module (InnerPipe pid=15314) <frozen importlib._bootstrap>(688): _load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1006): _find_and_load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1027): _find_and_load (InnerPipe pid=15314) /workspace/InternDataEngine/workflows/simbox/core/skills/__init__.py(6): <module> (InnerPipe pid=15314) <frozen importlib._bootstrap>(241): _call_with_frames_removed (InnerPipe pid=15314) <frozen importlib._bootstrap_external>(883): exec_module (InnerPipe pid=15314) <frozen importlib._bootstrap>(688): _load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1006): _find_and_load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1027): _find_and_load (InnerPipe pid=15314) <frozen importlib._bootstrap>(241): _call_with_frames_removed (InnerPipe pid=15314) <frozen importlib._bootstrap>(992): _find_and_load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1027): _find_and_load (InnerPipe pid=15314) /workspace/InternDataEngine/workflows/simbox/core/skills/__init__.py(3): <module> (InnerPipe pid=15314) <frozen importlib._bootstrap>(241): _call_with_frames_removed (InnerPipe pid=15314) <frozen importlib._bootstrap_external>(883): exec_module (InnerPipe pid=15314) <frozen importlib._bootstrap>(688): _load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1006): _find_and_load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1027): _find_and_load (InnerPipe pid=15314) /workspace/InternDataEngine/workflows/simbox_dual_workflow.py(29): <module> (InnerPipe pid=15314) <frozen importlib._bootstrap>(241): _call_with_frames_removed (InnerPipe pid=15314) <frozen importlib._bootstrap_external>(883): exec_module (InnerPipe pid=15314) <frozen importlib._bootstrap>(688): _load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1006): _find_and_load_unlocked (InnerPipe pid=15314) <frozen importlib._bootstrap>(1027): _find_and_load (InnerPipe pid=15314) /workspace/InternDataEngine/workflows/__init__.py(5): import_extensions (InnerPipe pid=15314) /workspace/InternDataEngine/nimbus_extension/components/load/env_loader.py(86): __init__ (InnerPipe pid=15314) /workspace/InternDataEngine/nimbus/scheduler/instructions.py(29): run (InnerPipe pid=15314) /workspace/InternDataEngine/nimbus/scheduler/stages.py(48): run (InnerPipe pid=15314) /workspace/InternDataEngine/nimbus/scheduler/inner_pipe.py(257): run (InnerPipe pid=15314) /isaac-sim/kit/python/lib/python3.10/site-packages/ray/util/tracing/tracing_helper.py(461): _resume_span (InnerPipe pid=15314) /isaac-sim/kit/python/lib/python3.10/site-packages/ray/_private/function_manager.py(704): actor_method_executor (InnerPipe pid=15314) /isaac-sim/kit/python/lib/python3.10/site-packages/ray/_private/worker.py(1028): main_loop (InnerPipe pid=15314) /isaac-sim/kit/python/lib/python3.10/site-packages/ray/_private/workers/default_worker.py(323): <module> (InnerPipe pid=15314) (InnerPipe pid=15322) 2026-03-23 12:48:36 [10,911ms] [Error] [gpu.foundation.plugin] No CUDA device found for graphics device 1. (InnerPipe pid=15322) 2026-03-23 12:48:36 [11,047ms] [Error] [gpu.foundation.plugin] CUDA interop cannot be used! [repeated 136x across cluster] (InnerPipe pid=15322) 2026-03-23 12:48:36 [11,047ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 3. [repeated 136x across cluster] (InnerPipe pid=15322) 2026-03-23 12:48:36 [11,047ms] [Error] [gpu.foundation.plugin] Failed to allocate buffers for block-compression [repeated 68x across cluster] (InnerPipe pid=15316) Warning: Possible version incompatibility. Attempting to load omni::fabric::IPath with version v0.2 against v0.1. [repeated 2x across cluster] (InnerPipe pid=15316) [6.463s] [ext: omni.isaac.utils-1.0.1] startup [repeated 403x across cluster] (InnerPipe pid=15316) 2026-03-23 12:48:37 [6,376ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed. [repeated 9x across cluster] (InnerPipe pid=15316) 2026-03-23 12:48:37 [6,376ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin) [repeated 10x across cluster] (InnerPipe pid=15322) 2026-03-23 12:48:38 [13,128ms] [Error] [gpu.foundation.plugin] Texture creation failed for the device: 0. [repeated 4x across cluster] (InnerPipe pid=15322) 2026-03-23 12:48:38 [13,128ms] [Error] [carb.scenerenderer-rtx.plugin] Failed to allocate 1280x720 LdrColor resource for device mask 0x1 [repeated 4x across cluster] (InnerPipe pid=15322) 2026-03-23 12:48:38 [13,128ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6 [repeated 4x across cluster] (InnerPipe pid=15322) 2026-03-23 12:48:38 [13,128ms] [Warning] [omni.hydra] Rendering failed. [repeated 4x across cluster] (InnerPipe pid=15322) [12.117s] app ready (InnerPipe pid=15322) 2026-03-23 12:48:38 [12,551ms] [Warning] [omni.kit.imgui_renderer.plugin] _createExtendCursor: No windowing. [repeated 2x across cluster] (InnerPipe pid=15322) [13.379s] Simulation App Startup Complete (InnerPipe pid=15316) saac.cortex.sample_behaviors-1.0.5] startup (InnerPipe pid=15316) [6.555s] Simulation App Starting (InnerPipe pid=15316) [10.672s] app ready (InnerPipe pid=15316) 2026-03-23 12:48:37 [6,691ms] [Warning] [omni.kvdb.plugin] Disabling key-value database because another kit process is locking it (InnerPipe pid=15322) (InnerPipe pid=15322) (InnerPipe pid=15322) 2026-03-23 12:48:42 [17,028ms] [Fatal] [drake] Could not initialize pydrake: UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 6104: ordinal not in range(128) (InnerPipe pid=15322) At: (InnerPipe pid=15322) /isaac-sim/kit/python/lib/python3.10/encodings/ascii.py(26): decode (InnerPipe pid=15322) /isaac-sim/kit/python/lib/python3.10/site-packages/pydrake/__init__.py(56): _execute_extra_python_code (InnerPipe pid=15322) <frozen importlib._bootstrap>(241): _call_with_frames_removed [repeated 12x across cluster] (InnerPipe pid=15322) <frozen importlib._bootstrap_external>(1176): create_module (InnerPipe pid=15322) <frozen importlib._bootstrap>(571): module_from_spec (InnerPipe pid=15322) <frozen importlib._bootstrap>(688): _load_unlocked [repeated 9x across cluster] (InnerPipe pid=15322) <frozen importlib._bootstrap>(1006): _find_and_load_unlocked [repeated 11x across cluster] (InnerPipe pid=15322) <frozen importlib._bootstrap>(1027): _find_and_load [repeated 11x across cluster] (InnerPipe pid=15322) <frozen importlib._bootstrap>(1078): _handle_fromlist (InnerPipe pid=15322) /isaac-sim/kit/python/lib/python3.10/site-packages/ray/_private/workers/default_worker.py(323): <module> [repeated 9x across cluster] (InnerPipe pid=15322) <frozen importlib._bootstrap_external>(883): exec_module [repeated 8x across cluster] (InnerPipe pid=15322) /workspace/InternDataEngine/workflows/__init__.py(5): import_extensions (InnerPipe pid=15322) /workspace/InternDataEngine/nimbus_extension/components/load/env_loader.py(86): __init__ (InnerPipe pid=15322) /workspace/InternDataEngine/nimbus/scheduler/inner_pipe.py(257): run [repeated 3x across cluster] (InnerPipe pid=15322) /isaac-sim/kit/python/lib/python3.10/site-packages/ray/util/tracing/tracing_helper.py(461): _resume_span (InnerPipe pid=15322) /isaac-sim/kit/python/lib/python3.10/site-packages/ray/_private/function_manager.py(704): actor_method_executor (InnerPipe pid=15322) /isaac-sim/kit/python/lib/python3.10/site-packages/ray/_private/worker.py(1028): main_loop (InnerPipe pid=15322) 2026-03-23 12:48:38 [13,377ms] [Error] [gpu.foundation.plugin] CUDA interop cannot be used! [repeated 14x across cluster] (InnerPipe pid=15316) [11.720s] Simulation App Startup Complete (InnerPipe pid=15316) [6.551s] [ext: omni.isaac.sim.python-4.1.0] startup [repeated 3x across cluster] (InnerPipe pid=15322) 2026-03-23 12:48:38 [13,377ms] [Warning] [omni.hydra] Rendering failed. [repeated 10x across cluster] (InnerPipe pid=15316) 2026-03-23 12:48:41 [11,079ms] [Warning] [omni.kit.imgui_renderer.plugin] _createExtendCursor: No windowing. [repeated 2x across cluster] (InnerPipe pid=15316) 2026-03-23 12:48:46 [15,360ms] [Fatal] [drake] Could not initialize pydrake: UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 6104: ordinal not in range(128) (InnerPipe pid=15316) (InnerPipe pid=15316) At: (InnerPipe pid=15316) /isaac-sim/kit/python/lib/python3.10/encodings/ascii.py(26): decode (InnerPipe pid=15316) /isaac-sim/kit/python/lib/python3.10/site-packages/pydrake/__init__.py(56): _execute_extra_python_code (InnerPipe pid=15316) <frozen importlib._bootstrap_external>(1176): create_module (InnerPipe pid=15316) <frozen importlib._bootstrap>(571): module_from_spec (InnerPipe pid=15316) <frozen importlib._bootstrap>(1078): _handle_fromlist (InnerPipe pid=15316) /workspace/InternDataEngine/workflows/__init__.py(5): import_extensions (InnerPipe pid=15316) /workspace/InternDataEngine/nimbus_extension/components/load/env_loader.py(86): __init__ (InnerPipe pid=15316) /isaac-sim/kit/python/lib/python3.10/site-packages/ray/util/tracing/tracing_helper.py(461): _resume_span (InnerPipe pid=15316) /isaac-sim/kit/python/lib/python3.10/site-packages/ray/_private/function_manager.py(704): actor_method_executor (InnerPipe pid=15316) /isaac-sim/kit/python/lib/python3.10/site-packages/ray/_private/worker.py(1028): main_loop (InnerPipe pid=15316) (InnerPipe pid=15322) 2026-03-23 12:48:38 [13,377ms] [Error] [gpu.foundation.plugin] Texture creation failed for the device: 0. [repeated 10x across cluster] (InnerPipe pid=15322) 2026-03-23 12:48:38 [13,377ms] [Error] [carb.scenerenderer-rtx.plugin] Failed to allocate 1280x720 LdrColor resource for device mask 0x1 [repeated 10x across cluster] (InnerPipe pid=15322) 2026-03-23 12:48:38 [13,377ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6 [repeated 10x across cluster] ^CTraceback (most recent call last): File "/workspace/InternDataEngine/launcher.py", line 49, in <module> main() File "/workspace/InternDataEngine/launcher.py", line 43, in main run_data_engine(config, args.random_seed) File "/workspace/InternDataEngine/nimbus/__init__.py", line 20, in run_data_engine data_engine.run() File "/workspace/InternDataEngine/nimbus/data_engine.py", line 64, in run value.wait_stop() File "/workspace/InternDataEngine/nimbus/dist_sim/head_node.py", line 161, in wait_stop sleep(0.1) KeyboardInterrupt (InnerPipe pid=15325) /isaac-sim/kit/python/lib/python3.10/site-packages/ray/runtime_context.py:457: RayDeprecationWarning: This API is deprecated and may be removed in future Ray releases. You could suppress this warning by setting env variable PYTHONWARNINGS="ignore::DeprecationWarning" (InnerPipe pid=15325) Use get_actor_id() instead (InnerPipe pid=15325) not self.actor_id.is_nil() (InnerPipe pid=15316) <frozen importlib._bootstrap>(241): _call_with_frames_removed [repeated 12x across cluster] (InnerPipe pid=15316) <frozen importlib._bootstrap>(688): _load_unlocked [repeated 9x across cluster] (InnerPipe pid=15316) <frozen importlib._bootstrap>(1006): _find_and_load_unlocked [repeated 11x across cluster] (InnerPipe pid=15316) <frozen importlib._bootstrap>(1027): _find_and_load [repeated 11x across cluster] (InnerPipe pid=15316) /isaac-sim/kit/python/lib/python3.10/site-packages/ray/_private/workers/default_worker.py(323): <module> [repeated 9x across cluster] (InnerPipe pid=15316) <frozen importlib._bootstrap_external>(883): exec_module [repeated 8x across cluster] (InnerPipe pid=15316) /workspace/InternDataEngine/nimbus/scheduler/inner_pipe.py(255): run [repeated 3x across cluster]

[zyhestu]

Well. I suspect this may be related to your machine having 4 GPUs configured. The pipeline mode launches multiple Isaac Sim instances, so our recommended usage for this mode is to start it inside a container bound to only one GPU.

Could you let me know how much memory your machine has? You can try launching a single-GPU container on your computer and allocating as much memory as possible. It should then start normally.

Our default pipeline configuration includes 2 planners and 1 renderer. In our experimental environment, this runs properly with 1 GPU、16CPU and 120GB of memory. If you encounter processes being killed due to insufficient memory, consider reducing the number of planners and trying a configuration of 1 planner and 1 renderer.

[sunzhen123-lab]

Our 4090 GPU has 24GB of VRAM, and we only have one CPU socket with 46 physical cores / 92 logical CPU threads, and about 460GB of system RAM.Can this machine run the full pipeline with a single GPU?
Another question: we want to run multi‑GPU pipeline parallelism inside a Docker container — is there no good solution for this right now?

[zyhestu]

Our 4090 GPU has 24GB of VRAM, and we only have one CPU socket with 46 physical cores / 92 logical CPU threads, and about 460GB of system RAM.Can this machine run the full pipeline with a single GPU? Another question: we want to run multi‑GPU pipeline parallelism inside a Docker container — is there no good solution for this right now?

Based on your setup, my suggestion for production mode is to launch 4 containers on your machine, assigning 1 GPU, 24 logical cores, and 110GB memory to each container. You can then run different generation tasks in pipeline mode within each container separately.

Additionally, regarding running multi-GPU parallel tasks within a single container, my recommendation is to avoid this approach. In our practice, we have observed that Isaac Sim can introduce some unexpected bugs in this scenario.