From 0e3e46246101faa600791fbc97c774aaa65634fd Mon Sep 17 00:00:00 2001 From: Rohit Date: Wed, 30 Sep 2026 13:55:12 +0800 Subject: [PATCH 1/8] xpu enablement --- README.md | 2 + docs/xpu_inference.md | 159 +++++++ .../agent/internvla_n1_agent_realworld.py | 8 +- internnav/configs/model/base_encoders.py | 1 + .../basemodel/internvla_n1/internvla_n1.py | 13 +- .../internvla_n1/internvla_n1_policy.py | 8 +- .../basemodel/internvla_n1/nextdit_traj.py | 8 +- internnav/model/encoder/__init__.py | 28 +- .../dinov2_layers/attention.py | 4 +- internnav/model/utils/inference_device.py | 18 + requirements/internvla_n1_xpu.txt | 16 + scripts/realworld/http_internvla_server.py | 16 +- tests/unit_test/test_xpu_inference.py | 435 ++++++++++++++++++ 13 files changed, 689 insertions(+), 27 deletions(-) create mode 100644 docs/xpu_inference.md create mode 100644 internnav/model/utils/inference_device.py create mode 100644 requirements/internvla_n1_xpu.txt create mode 100644 tests/unit_test/test_xpu_inference.py diff --git a/README.md b/README.md index 59d34cc9..210df5c4 100644 --- a/README.md +++ b/README.md @@ -63,6 +63,8 @@ The toolbox supports the most advanced high-quality navigation dataset, InternDa Please refer to the [documentation](https://internrobotics.github.io/user_guide/internnav/quick_start/index.html) for quick start with InternNav, from installation to training or evaluating supported models. +For optional native Intel XPU eager model inference, see [setup and validation limits](docs/xpu_inference.md). The existing CUDA installation route is unchanged. + ## ๐Ÿ“ฆ Overview ### ๐Ÿงช Supported Benchmarks diff --git a/docs/xpu_inference.md b/docs/xpu_inference.md new file mode 100644 index 00000000..faf6cf0c --- /dev/null +++ b/docs/xpu_inference.md @@ -0,0 +1,159 @@ +# Optional Intel XPU eager inference + +This route targets the InternVLA-N1-DualVLN model only. Native XPU component +execution has been tested; full-checkpoint compatibility has **not** been +established. It does not establish Habitat, Isaac, simulator, sensor, or robot +stack compatibility. There is no compilation, quantization, CPU offload, or +static-cache change. + +## Prerequisites and installation + +Use a supported Intel GPU, a Linux kernel/Intel GPU driver supporting that GPU, +the Intel Level Zero compute runtime, and permission to access `/dev/dri/render*`. +PCI visibility alone is insufficient. Follow the current +[PyTorch Intel GPU prerequisites](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html) +for your OS/device. Driver installation is an administrator task; PyTorch wheels +do not install the kernel driver. Do not install IPEX or CUDA-only attention +extensions for this route. + +From the repository root, use the existing `venv`. The official stable XPU index +was queried on 2026-09-30 and reported torch 2.14.0 and torchvision 0.29.0: + +```bash +venv/bin/python -m pip index versions torch --index-url https://download.pytorch.org/whl/xpu +venv/bin/python -m pip index versions torchvision --index-url https://download.pytorch.org/whl/xpu +venv/bin/python -m pip install torch==2.14.0 torchvision==0.29.0 --index-url https://download.pytorch.org/whl/xpu +venv/bin/python -m pip install -r requirements/internvla_n1_xpu.txt +venv/bin/python -m pip check +venv/bin/python -c "import torch; assert torch.xpu.is_available(); print(torch.__version__); print(torch.xpu.get_device_properties(0)); print(torch.xpu.mem_get_info(0))" +``` + +Transformers remains 4.51.0, Diffusers 0.33.1, and Accelerate 1.4.0. The separate +inference requirements do not require `flash_attn`, xFormers, or diffusion-policy. +Run from the repository root so imports resolve without installing all simulator +extras. The original CUDA requirements and installation route are unchanged; +do not combine the CUDA extras with this minimal XPU environment. Some existing +agent imports emit warnings about absent Habitat/depth-camera-filtering; those +packages are not needed for this standalone DualVLN route. + +## Auxiliary checkpoint + +Normal pretrained depth/RGB initialization is retained, including strict +`load_state_dict` checks. There is no skip-loading flag and no random-encoder +fallback, even when restoring a full DualVLN checkpoint. Supply the auxiliary +checkpoint from its upstream source before constructing the model: + +```bash +curl -fL --retry 2 -o checkpoints/depth_anything_v2_metric_hypersim_vits.pth \ + https://huggingface.co/depth-anything/Depth-Anything-V2-Metric-Hypersim-Small/resolve/main/depth_anything_v2_metric_hypersim_vits.pth +sha256sum checkpoints/depth_anything_v2_metric_hypersim_vits.pth +``` + +The downloaded file used for validation has SHA256 +`b782898d8a3e8be1f639de33837ed85e9b4b73e40f8f5e5cd99067588d722545`. +It is a local ignored checkpoint, not part of the source patch. Missing files, +missing weights, and mismatched shapes remain fatal. Partial checkpoints and +training continue to use the normal pretrained initializer. + +## Configuration and launch + +Both inference loaders accept `attn_implementation` (default `auto`): + +| Device | `auto` | Other selections | +| --- | --- | --- | +| CUDA | `flash_attention_2`, unchanged | `sdpa`, `eager`, explicit FA2 | +| XPU | `sdpa` | `eager` for debugging; external FA2 is rejected | + +An explicit unavailable XPU or invalid XPU index raises an error, never silently +falls back to CPU. Existing CUDA device defaults are unchanged. BF16 and the +custom model/processor are retained. Policy configuration can set: + +```python +device = 'xpu:0' +attn_implementation = 'auto' +model_path = 'checkpoints/InternVLA-N1-DualVLN' +``` + +On an XPU with enough available memory: + +```bash +venv/bin/python -m scripts.realworld.http_internvla_server \ + --model_path checkpoints/InternVLA-N1-DualVLN \ + --device xpu:0 --attn_implementation auto --plan_step_gap 4 --skip_warmup +``` + +The server listens on port 5801. `--skip_warmup` is opt-in; it bypasses the existing +demo warmup, whose call omits the required camera intrinsic argument and uses +floating-point RGB input. The original default warmup branch is unchanged. +The agent is reset before serving when the bypass is selected. This is a local +development server with shared episode state, not a concurrent multi-client or +production server. Its existing hard-coded instruction behavior is unchanged. + +## Memory and runtime evidence + +DualVLN weights require 16,767,077,782 bytes (15.62 GiB), before KV cache, visual +activations, diffusion buffers, allocator reservations, and runtime overhead. +History increases memory use. More than 15.62 GiB of **available** device memory +is required; no universal sufficient-memory threshold has been established. +Shared system memory and swap are not equivalent to available XPU memory. + +Validation on 2026-09-30 used the existing Python 3.12 venv, Linux kernel +7.0.0-31-generic, Intel Arc B390 (PCI 8086:b082), Level Zero V2 driver +1.15.39122+14. PyTorch reported 14,939,189,248 bytes (13.91 GiB) total and +1,558,061,056 bytes free at the initial probe. Full-model loading was therefore +not attempted. CUDA was unavailable. + +Exact principal versions: torch 2.14.0+xpu, torchvision 0.29.0+xpu, +transformers 4.51.0, diffusers 0.33.1, accelerate 1.4.0, numpy 1.26.4, +Pillow 11.3.0, imageio 2.37.0, pydantic 2.11.10, pytest 9.1.1. + +```bash +venv/bin/python -m pip install pytest +INTERNNAV_FULL_XPU_TEST=1 venv/bin/python -m pytest tests/unit_test/test_xpu_inference.py -q -s +``` + +Result: **23 passed, 1 skipped**. The opt-in full-checkpoint test skipped on its +memory gate: weights 16,767,077,782 bytes, total 14,939,189,248 bytes, free +712,245,248 bytes during the combined test run. Its two-GiB headroom check is a +minimum safety gate, not a guarantee that a full run will fit. + +Tested scenarios: + +- Device/attention selection, unavailable XPU, invalid index, and FA2 rejection. +- DINO native CPU/XPU fallback even with simulated xFormers availability; + non-None nested biases are rejected rather than discarded. +- Strict auxiliary restoration, including missing file/key and shape failures; + the real pretrained RGB encoder loaded with 22,056,576 finite parameters. +- Actual processor image/instruction handling, custom reduced Qwen prefill, + cached decoding, and latent extraction on XPU in both SDPA and eager modes. + Integer tokens/grid metadata stay integer. CPU BF16 eager reference logits + matched at `rtol=0.05, atol=0.01` with identical weights. Peak XPU allocated + memory: 36,691,968 bytes per mode. +- Actual full-size DINO/memory/QFormer/diffusion components on XPU with two + 224x224 frames, two diffusion steps, two samples, and four predicted steps. + Only DINO uses pretrained weights in this component test; other weights are + deterministic random fixtures. CPU/XPU use the **same noise tensor**, not just + seeds. BF16 trajectories matched at `rtol=0.05, atol=0.05`. Peak XPU allocated + memory: 212,505,600 bytes. These peaks are not full-model memory estimates. +- Stubbed STOP/waypoint outputs, look-down history, episode reset, thread-local + no-grad contexts, and one real asynchronous-worker publication iteration. +- Diffusers checkpoint-toggle API compatibility, CLI help, and `pip check`. + +Not verified: complete pretrained DualVLN execution/quality, long-history memory, +full-model peak memory, CUDA FA2 runtime equivalence, real checkpoint STOP/action +agreement, sustained asynchronous scheduling/reset races, or robot/simulator +integration. The full-checkpoint test is ready for a larger XPU but is not a +substitute for recorded navigation inputs and a CUDA baseline. For that comparison, +copy identical diffusion noise across devices and compare intermediate logits, +latents, trajectories, and resulting actions with explicit BF16 tolerances. + +## Change boundaries + +Shared resolution is used only by the two inference loaders. DINO retains its +existing CUDA xFormers computation. Tensor transfers are confined to inference +methods; RGB normalization and scheduler/guidance math are unchanged. No model +layers or checkpoint keys were added. Gradients are not disabled globally. +Encoder exports are loaded lazily to avoid requiring unrelated LongCLIP just to +import DINO. A backward-compatible checkpoint-toggle signature adapter fixes the +custom diffusion model's constructor under pinned Diffusers 0.33.1; the existing +training checkpoint computation is unchanged. Full training was not tested. \ No newline at end of file diff --git a/internnav/agent/internvla_n1_agent_realworld.py b/internnav/agent/internvla_n1_agent_realworld.py index 1cb71c19..641b9436 100644 --- a/internnav/agent/internvla_n1_agent_realworld.py +++ b/internnav/agent/internvla_n1_agent_realworld.py @@ -18,6 +18,7 @@ from transformers import AutoProcessor from internnav.model.basemodel.internvla_n1.internvla_n1 import InternVLAN1ForCausalLM +from internnav.model.utils.inference_device import resolve_inference_device from internnav.model.utils.vln_utils import S2Output, split_and_clean, traj_to_actions DEFAULT_IMAGE_TOKEN = "" @@ -25,13 +26,15 @@ class InternVLAN1AsyncAgent: def __init__(self, args): - self.device = torch.device(args.device) + self.device, attention = resolve_inference_device( + args.device, getattr(args, 'attn_implementation', 'auto') + ) self.save_dir = "test_data/" + datetime.now().strftime("%Y%m%d_%H%M%S") print(f"args.model_path{args.model_path}") self.model = InternVLAN1ForCausalLM.from_pretrained( args.model_path, torch_dtype=torch.bfloat16, - attn_implementation="flash_attention_2", + attn_implementation=attention, device_map={"": self.device}, ) self.model.eval() @@ -252,6 +255,7 @@ def step_s2(self, rgb, depth, pose, instruction, intrinsic, look_down=False): action_seq = self.parse_actions(self.llm_output) return action_seq, None, None + @torch.no_grad() def step_s1(self, latent, rgb, depth): all_trajs = self.model.generate_traj(latent, rgb, depth) return all_trajs diff --git a/internnav/configs/model/base_encoders.py b/internnav/configs/model/base_encoders.py index c5d9e79e..cca32906 100644 --- a/internnav/configs/model/base_encoders.py +++ b/internnav/configs/model/base_encoders.py @@ -179,6 +179,7 @@ class Seq2Seq(BaseModel, extra='allow'): class ModelCfg(BaseModel, extra='allow'): + attn_implementation: str = 'auto' policy_name: Optional[str] ablate_instruction: Optional[bool] = None ablate_depth: Optional[bool] = None diff --git a/internnav/model/basemodel/internvla_n1/internvla_n1.py b/internnav/model/basemodel/internvla_n1/internvla_n1.py index 69c7f474..0a5a1523 100644 --- a/internnav/model/basemodel/internvla_n1/internvla_n1.py +++ b/internnav/model/basemodel/internvla_n1/internvla_n1.py @@ -318,7 +318,8 @@ def forward( ) def generate_latents(self, input_ids, pixel_values, image_grid_thw): - input_ids.to(self.get_model().device) + input_ids = input_ids.to(self.get_model().embed_tokens.weight.device) + image_grid_thw = image_grid_thw.to(input_ids.device) with torch.no_grad(): text_embeds = self.get_model().embed_tokens(input_ids) latent_queries = self.get_model().latent_queries.repeat(text_embeds.shape[0], 1, 1) @@ -326,7 +327,7 @@ def generate_latents(self, input_ids, pixel_values, image_grid_thw): N_QUERY = self.get_n_query() input_ids = torch.cat([input_ids, torch.tensor([[TRAJ_TOKEN_INDEX] * N_QUERY]).to(input_ids.device)], dim=1) - pixel_values = pixel_values.type(self.visual.dtype) + pixel_values = pixel_values.to(device=self.visual.device, dtype=self.visual.dtype) image_embeds = self.visual(pixel_values, grid_thw=image_grid_thw).unsqueeze(0) text_embeds[image_idx] = image_embeds.to(text_embeds.device)[: image_idx.sum(), :] @@ -358,13 +359,15 @@ def generate_traj( ): if 'nextdit' in self.get_system1_type(): scheduler = FlowMatchEulerDiscreteScheduler() - device = traj_latents.device - dtype = traj_latents.dtype + projector_weight = self.get_model().cond_projector[0].weight + device = projector_weight.device + dtype = projector_weight.dtype + traj_latents = traj_latents.to(device=device, dtype=dtype) traj_latents = self.get_model().cond_projector(traj_latents) if 'async' in self.get_system1_type(): with torch.no_grad(): - images_dp = images_dp.permute(0, 1, 4, 2, 3) + images_dp = images_dp.to(device=device).permute(0, 1, 4, 2, 3) images_dp_norm = (images_dp - self._resnet_mean) / self._resnet_std self.get_model().rgb_model.to(dtype) images_dp_feat = ( diff --git a/internnav/model/basemodel/internvla_n1/internvla_n1_policy.py b/internnav/model/basemodel/internvla_n1/internvla_n1_policy.py index e56daaf8..a88a6d82 100644 --- a/internnav/model/basemodel/internvla_n1/internvla_n1_policy.py +++ b/internnav/model/basemodel/internvla_n1/internvla_n1_policy.py @@ -10,6 +10,7 @@ from transformers import AutoProcessor, AutoTokenizer, PreTrainedModel from internnav.configs.model.base_encoders import ModelCfg +from internnav.model.utils.inference_device import resolve_inference_device from internnav.model.basemodel.internvla_n1.internvla_n1 import ( InternVLAN1ForCausalLM, InternVLAN1ModelConfig, @@ -30,11 +31,14 @@ def __init__(self, config: Union[InternVLAN1ModelConfig, ModelCfg]): super().__init__(config) self.model_config = ModelCfg(**config.model_cfg['model']) + device, attention = resolve_inference_device( + self.model_config.device, self.model_config.attn_implementation + ) self.model = InternVLAN1ForCausalLM.from_pretrained( self.model_config.model_path, torch_dtype=torch.bfloat16, - attn_implementation="flash_attention_2", - device_map={"": self.model_config.device}, + attn_implementation=attention, + device_map={"": device}, ) self.tokenizer = AutoTokenizer.from_pretrained(self.model_config.model_path, use_fast=True) diff --git a/internnav/model/basemodel/internvla_n1/nextdit_traj.py b/internnav/model/basemodel/internvla_n1/nextdit_traj.py index 1c18a63f..694e444f 100644 --- a/internnav/model/basemodel/internvla_n1/nextdit_traj.py +++ b/internnav/model/basemodel/internvla_n1/nextdit_traj.py @@ -292,9 +292,11 @@ def __init__( assert (hidden_size // num_attention_heads) % 4 == 0, "2d rope needs head dim to be divisible by 4" - def _set_gradient_checkpointing(self, module, value=False): - if hasattr(module, "gradient_checkpointing"): - module.gradient_checkpointing = value + def _set_gradient_checkpointing(self, module=None, value=False, *, enable=None, gradient_checkpointing_func=None): + modules = self.modules() if module is None else (module,) + for target in modules: + if hasattr(target, "gradient_checkpointing"): + target.gradient_checkpointing = value if enable is None else enable def forward( self, diff --git a/internnav/model/encoder/__init__.py b/internnav/model/encoder/__init__.py index dc8c5b83..3b0c4f01 100644 --- a/internnav/model/encoder/__init__.py +++ b/internnav/model/encoder/__init__.py @@ -1,7 +1,21 @@ -from .bert_backbone import PositionalEncoding -from .distance_encoder import DistanceNetwork -from .image_clip_encoder import ImageEncoder -from .instruction_encoder import InstructionEncoder -from .instruction_longCLIP_encoder import InstructionLongCLIPEncoder -from .instruction_roberta_encoder import LanguageEncoder -from .vision_language_encoder import VisionLanguageEncoder +from importlib import import_module + + +_ENCODERS = { + 'PositionalEncoding': '.bert_backbone', + 'DistanceNetwork': '.distance_encoder', + 'ImageEncoder': '.image_clip_encoder', + 'InstructionEncoder': '.instruction_encoder', + 'InstructionLongCLIPEncoder': '.instruction_longCLIP_encoder', + 'LanguageEncoder': '.instruction_roberta_encoder', + 'VisionLanguageEncoder': '.vision_language_encoder', +} +__all__ = list(_ENCODERS) + + +def __getattr__(name): + if name not in _ENCODERS: + raise AttributeError(f'module {__name__!r} has no attribute {name!r}') + encoder = getattr(import_module(_ENCODERS[name], __name__), name) + globals()[name] = encoder + return encoder diff --git a/internnav/model/encoder/depth_anything/depth_anything_v2/dinov2_layers/attention.py b/internnav/model/encoder/depth_anything/depth_anything_v2/dinov2_layers/attention.py index 815a2bf5..48f26cbe 100644 --- a/internnav/model/encoder/depth_anything/depth_anything_v2/dinov2_layers/attention.py +++ b/internnav/model/encoder/depth_anything/depth_anything_v2/dinov2_layers/attention.py @@ -64,8 +64,8 @@ def forward(self, x: Tensor) -> Tensor: class MemEffAttention(Attention): def forward(self, x: Tensor, attn_bias=None) -> Tensor: - if not XFORMERS_AVAILABLE: - assert attn_bias is None, "xFormers is required for nested tensors usage" + if not XFORMERS_AVAILABLE or x.device.type != 'cuda': + assert attn_bias is None, "CUDA xFormers is required for nested tensors usage" return super().forward(x) B, N, C = x.shape diff --git a/internnav/model/utils/inference_device.py b/internnav/model/utils/inference_device.py new file mode 100644 index 00000000..6b8bca94 --- /dev/null +++ b/internnav/model/utils/inference_device.py @@ -0,0 +1,18 @@ +import torch + + +def resolve_inference_device(device, attn_implementation='auto'): + device = torch.device(device) + supported = ('auto', 'flash_attention_2', 'sdpa', 'eager') + if attn_implementation not in supported: + raise ValueError(f'Attention must be one of {supported}, got {attn_implementation!r}') + if device.type == 'xpu': + if attn_implementation == 'flash_attention_2': + raise ValueError('External FlashAttention 2 is not supported on XPU; use sdpa or eager') + if not hasattr(torch, 'xpu') or not torch.xpu.is_available(): + raise RuntimeError('XPU was requested but is unavailable; check native PyTorch XPU wheels and Intel drivers') + if device.index is not None and device.index >= torch.xpu.device_count(): + raise RuntimeError(f'Requested XPU device {device} does not exist') + if attn_implementation == 'auto': + attn_implementation = 'flash_attention_2' if device.type == 'cuda' else 'sdpa' + return device, attn_implementation \ No newline at end of file diff --git a/requirements/internvla_n1_xpu.txt b/requirements/internvla_n1_xpu.txt new file mode 100644 index 00000000..32bc95b6 --- /dev/null +++ b/requirements/internvla_n1_xpu.txt @@ -0,0 +1,16 @@ +accelerate==1.4.0 +diffusers==0.33.1 +transformers==4.51.0 +ftfy==6.3.1 +einops==0.8.1 +numpy==1.26.4 +opencv-python-headless==4.11.0.86 +pydantic>=2.11,<2.12 +flask>=3.1,<4 +gym==0.26.2 +imageio==2.37.0 +pillow>=11,<12 +numpy-quaternion==2024.0.13 +scipy>=1.15,<1.18 +fastapi>=0.110,<0.111 +uvicorn>=0.29,<0.31 \ No newline at end of file diff --git a/scripts/realworld/http_internvla_server.py b/scripts/realworld/http_internvla_server.py index af3fbad4..bbeebaad 100644 --- a/scripts/realworld/http_internvla_server.py +++ b/scripts/realworld/http_internvla_server.py @@ -81,22 +81,26 @@ def eval_dual(): parser = argparse.ArgumentParser() parser.add_argument("--device", type=str, default="cuda:0") + parser.add_argument("--attn_implementation", choices=['auto', 'flash_attention_2', 'sdpa', 'eager'], default='auto') parser.add_argument("--model_path", type=str, default="checkpoints/InternVLA-N1") parser.add_argument("--resize_w", type=int, default=384) parser.add_argument("--resize_h", type=int, default=384) parser.add_argument("--num_history", type=int, default=8) + parser.add_argument("--plan_step_gap", type=int, default=4) + parser.add_argument("--skip_warmup", action="store_true") args = parser.parse_args() args.camera_intrinsic = np.array( [[386.5, 0.0, 328.9, 0.0], [0.0, 386.5, 244, 0.0], [0.0, 0.0, 1.0, 0.0], [0.0, 0.0, 0.0, 1.0]] ) agent = InternVLAN1AsyncAgent(args) - agent.step( - np.zeros((480, 640, 3)), - np.zeros((480, 640)), - np.eye(4), - "hello", - ) + if not args.skip_warmup: + agent.step( + np.zeros((480, 640, 3)), + np.zeros((480, 640)), + np.eye(4), + "hello", + ) agent.reset() app.run(host='0.0.0.0', port=5801) diff --git a/tests/unit_test/test_xpu_inference.py b/tests/unit_test/test_xpu_inference.py new file mode 100644 index 00000000..abfdd7e0 --- /dev/null +++ b/tests/unit_test/test_xpu_inference.py @@ -0,0 +1,435 @@ +from unittest.mock import patch + +import pytest +import torch + +from internnav.model.utils.inference_device import resolve_inference_device +from internnav.model.encoder.depth_anything.depth_anything_v2.dinov2_layers import attention as dino_attention + + +def test_cuda_default_attention(): + device, attention = resolve_inference_device('cuda:0') + assert str(device) == 'cuda:0' + assert attention == 'flash_attention_2' + + +@pytest.mark.parametrize('attention', ['auto', 'sdpa', 'eager']) +def test_xpu_attention(attention): + with patch.object(torch.xpu, 'is_available', return_value=True), patch.object( + torch.xpu, 'device_count', return_value=1 + ): + device, resolved = resolve_inference_device('xpu:0', attention) + assert str(device) == 'xpu:0' + assert resolved == ('sdpa' if attention == 'auto' else attention) + + +def test_xpu_rejects_flash_attention(): + with pytest.raises(ValueError, match='FlashAttention 2'): + resolve_inference_device('xpu:0', 'flash_attention_2') + + +def test_xpu_unavailable(): + with patch.object(torch.xpu, 'is_available', return_value=False): + with pytest.raises(RuntimeError, match='unavailable'): + resolve_inference_device('xpu:0') + + +def test_xpu_invalid_index(): + with patch.object(torch.xpu, 'is_available', return_value=True), patch.object( + torch.xpu, 'device_count', return_value=1 + ): + with pytest.raises(RuntimeError, match='does not exist'): + resolve_inference_device('xpu:1') + + +def test_invalid_attention(): + with pytest.raises(ValueError, match='Attention must be'): + resolve_inference_device('cpu', 'invalid') + + +@pytest.mark.parametrize('device', ['cpu', 'xpu:0']) +@pytest.mark.parametrize('xformers_available', [False, True]) +def test_dino_native_fallback(device, xformers_available): + if device.startswith('xpu') and not torch.xpu.is_available(): + pytest.skip('Native XPU unavailable') + layer = dino_attention.MemEffAttention(32, num_heads=4).to(device).eval() + inputs = torch.randn(2, 8, 32, device=device) + with patch.object(dino_attention, 'XFORMERS_AVAILABLE', xformers_available), patch.object( + dino_attention, 'memory_efficient_attention', create=True, side_effect=AssertionError('Called xFormers') + ), torch.no_grad(): + expected = dino_attention.Attention.forward(layer, inputs) + torch.testing.assert_close(layer(inputs), expected) + with pytest.raises(AssertionError, match='nested tensors'): + layer(inputs, attn_bias=object()) + + +def test_latent_tokens_move_to_embedding_device(): + from types import SimpleNamespace + + from internnav.model.basemodel.internvla_n1.internvla_n1 import InternVLAN1ForCausalLM + + class EmbeddingProbe(torch.nn.Module): + def __init__(self): + super().__init__() + self.weight = torch.nn.Parameter(torch.empty(1, device='meta')) + + def forward(self, input_ids): + assert input_ids.device.type == 'meta' + assert input_ids.dtype == torch.long + raise RuntimeError('Embedding placement verified') + + model = SimpleNamespace(get_model=lambda: SimpleNamespace(embed_tokens=EmbeddingProbe())) + with pytest.raises(RuntimeError, match='Embedding placement verified'): + InternVLAN1ForCausalLM.generate_latents( + model, torch.ones(1, 4, dtype=torch.long), torch.zeros(1, 4), torch.ones(1, 3, dtype=torch.long) + ) + + +@pytest.mark.parametrize('failure', ['missing_file', 'missing_weight', 'wrong_shape']) +def test_auxiliary_checkpoint_fails_closed(tmp_path, monkeypatch, failure): + from types import SimpleNamespace + + from internnav.model.basemodel.internvla_n1 import internvla_n1_arch as arch + from internnav.model.encoder.depth_anything.depth_anything_v2 import dpt + + class TinyDepth(torch.nn.Module): + def __init__(self, **kwargs): + super().__init__() + self.pretrained = torch.nn.Linear(2, 2) + + monkeypatch.setattr(dpt, 'DepthAnythingV2', TinyDepth) + monkeypatch.setattr(arch, 'MODEL_PATH_TO', str(tmp_path)) + checkpoint = tmp_path / 'depth_anything_v2_metric_hypersim_vits.pth' + if failure != 'missing_file': + weights = TinyDepth().state_dict() + if failure == 'missing_weight': + del weights['pretrained.weight'] + else: + weights['pretrained.weight'] = torch.zeros(3, 3) + torch.save(weights, checkpoint) + expected = FileNotFoundError if failure == 'missing_file' else RuntimeError + with pytest.raises(expected): + arch.build_depthanythingv2(SimpleNamespace()) + + +def test_auxiliary_checkpoint_restores_weights(tmp_path, monkeypatch): + from types import SimpleNamespace + + from internnav.model.basemodel.internvla_n1 import internvla_n1_arch as arch + from internnav.model.encoder.depth_anything.depth_anything_v2 import dpt + + class TinyDepth(torch.nn.Module): + def __init__(self, **kwargs): + super().__init__() + self.pretrained = torch.nn.Linear(2, 2) + + weights = TinyDepth().state_dict() + torch.save(weights, tmp_path / 'depth_anything_v2_metric_hypersim_vits.pth') + monkeypatch.setattr(dpt, 'DepthAnythingV2', TinyDepth) + monkeypatch.setattr(arch, 'MODEL_PATH_TO', str(tmp_path)) + rgb = arch.build_depthanythingv2(SimpleNamespace()) + torch.testing.assert_close(rgb.weight, weights['pretrained.weight']) + torch.testing.assert_close(rgb.bias, weights['pretrained.bias']) + + +@pytest.mark.parametrize('attention', ['sdpa', 'eager']) +def test_xpu_bf16_prefill_cache_and_latents(attention): + import copy + from pathlib import Path + + from PIL import Image + from transformers import AutoProcessor + + from internnav.model.basemodel.internvla_n1.internvla_n1 import ( + InternVLAN1ForCausalLM, + InternVLAN1ModelConfig, + ) + + checkpoint = Path(__file__).resolve().parents[2] / 'checkpoints/InternVLA-N1-DualVLN' + if not torch.xpu.is_available() or not (checkpoint / 'preprocessor_config.json').exists(): + pytest.skip('Requires native XPU and local DualVLN processor') + config = InternVLAN1ModelConfig( + vocab_size=152064, hidden_size=32, intermediate_size=64, + num_hidden_layers=1, num_attention_heads=4, num_key_value_heads=2, + rope_scaling={'type': 'default', 'mrope_section': [1, 1, 2]}, + image_token_id=151655, video_token_id=151656, + vision_start_token_id=151652, vision_end_token_id=151653, + n_query=4, + vision_config={ + 'depth': 1, 'hidden_size': 32, 'intermediate_size': 64, 'num_heads': 4, + 'out_hidden_size': 32, 'patch_size': 14, 'spatial_merge_size': 2, + 'temporal_patch_size': 2, 'fullatt_block_indexes': [0], + }, + attn_implementation='eager', + ) + torch.manual_seed(7) + reference = InternVLAN1ForCausalLM(config).eval().to(torch.bfloat16) + reference.model.latent_queries = torch.nn.Parameter(torch.randn(1, 4, 32, dtype=torch.bfloat16)) + device_config = copy.deepcopy(config) + device_config._attn_implementation = attention + device_config.vision_config._attn_implementation = attention + model = InternVLAN1ForCausalLM(device_config).eval().to(torch.bfloat16) + model.model.latent_queries = torch.nn.Parameter(torch.empty(1, 4, 32, dtype=torch.bfloat16)) + model.load_state_dict(reference.state_dict()) + model.to('xpu:0') + processor = AutoProcessor.from_pretrained(checkpoint, min_pixels=784, max_pixels=784, local_files_only=True) + messages = [{'role': 'user', 'content': [{'type': 'image'}, {'type': 'text', 'text': 'Stop at the door.'}]}] + text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) + inputs = processor(text=[text], images=[Image.new('RGB', (28, 28), color='green')], return_tensors='pt') + torch.xpu.reset_peak_memory_stats() + with torch.no_grad(): + expected = reference(**inputs, use_cache=True).logits + device_inputs = copy.deepcopy(inputs).to('xpu:0') + actual = model(**device_inputs, use_cache=True) + assert actual.past_key_values is not None + torch.testing.assert_close(actual.logits.cpu(), expected, rtol=0.05, atol=0.01) + output = model.generate(**device_inputs, max_new_tokens=2, min_new_tokens=2, do_sample=False, use_cache=True) + assert output.shape[1] == inputs.input_ids.shape[1] + 2 + latent = model.generate_latents(output.cpu(), inputs.pixel_values, inputs.image_grid_thw) + assert latent.shape == (1, 4, 32) + assert latent.device.type == 'xpu' and latent.dtype == torch.bfloat16 + assert latent.isfinite().all() + assert device_inputs.input_ids.dtype == torch.long + assert device_inputs.image_grid_thw.dtype == torch.long + print(f'{attention} tiny Qwen peak allocated bytes: {torch.xpu.max_memory_allocated()}') + + +def test_xpu_bf16_trajectory_with_identical_noise(monkeypatch): + import copy + from pathlib import Path + from types import SimpleNamespace + + from internnav.model.basemodel.internvla_n1 import internvla_n1 as model_module + from internnav.model.basemodel.internvla_n1 import internvla_n1_arch as arch + + checkpoint_dir = Path(__file__).resolve().parents[2] / 'checkpoints' + if not torch.xpu.is_available() or not (checkpoint_dir / 'depth_anything_v2_metric_hypersim_vits.pth').exists(): + pytest.skip('Requires native XPU and auxiliary pretrained checkpoint') + monkeypatch.setattr(arch, 'MODEL_PATH_TO', str(checkpoint_dir)) + + class TrajectoryModel(torch.nn.Module): + generate_traj = model_module.InternVLAN1ForCausalLM.generate_traj + + def __init__(self): + super().__init__() + self.cond_projector = torch.nn.Sequential( + torch.nn.Linear(3584, 768), torch.nn.GELU(approximate='tanh'), torch.nn.Linear(768, 768) + ) + self.rgb_model = arch.build_depthanythingv2(SimpleNamespace()) + self.memory_encoder = arch.MemoryEncoder() + self.rgb_resampler = arch.QFormer() + self.traj_dit, self.noise_scheduler = arch.build_traj_dit(SimpleNamespace()) + self.action_encoder = torch.nn.Linear(3, 384) + self.pos_encoding = arch.SinusoidalPositionalEncoding(384) + self.action_decoder = torch.nn.Linear(384, 3) + for name, value in (('_resnet_mean', model_module._RESNET_MEAN), ('_resnet_std', model_module._RESNET_STD)): + self.register_buffer(name, torch.tensor(value).view(1, 1, 3, 1, 1), persistent=False) + + def get_model(self): + return self + + def get_system1_type(self): + return 'nextdit_async' + + torch.manual_seed(11) + reference = TrajectoryModel().eval().to(torch.bfloat16) + model = copy.deepcopy(reference).to('xpu:0') + noise = torch.randn(2, 4, 3, dtype=torch.bfloat16) + latent = torch.randn(1, 4, 3584, dtype=torch.bfloat16) + images = torch.rand(1, 2, 224, 224, 3, dtype=torch.float64) + + def fixed_noise(shape, generator, device, dtype): + assert tuple(shape) == tuple(noise.shape) + return noise.clone().to(device=device, dtype=dtype) + + monkeypatch.setattr(model_module, 'randn_tensor', fixed_noise) + torch.xpu.reset_peak_memory_stats() + with torch.no_grad(): + expected = reference.generate_traj(latent, images, predict_step_nums=4, num_inference_steps=2, num_sample_trajs=2) + actual = model.generate_traj(latent, images, predict_step_nums=4, num_inference_steps=2, num_sample_trajs=2) + assert actual.device.type == 'xpu' + assert actual.isfinite().all() + torch.testing.assert_close(actual.cpu(), expected, rtol=0.05, atol=0.05) + print(f'Trajectory peak allocated bytes: {torch.xpu.max_memory_allocated()}') + + +def test_diffusers_checkpointing_toggle_compatibility(): + from internnav.model.basemodel.internvla_n1.nextdit_crossattn_traj import NextDiTCrossAttn, NextDiTCrossAttnConfig + + model = NextDiTCrossAttn(NextDiTCrossAttnConfig(n_layers=1)).model + assert model.gradient_checkpointing + model.disable_gradient_checkpointing() + assert not model.gradient_checkpointing + model.enable_gradient_checkpointing() + assert model.gradient_checkpointing + model._set_gradient_checkpointing(model, value=False) + assert not model.gradient_checkpointing + + +def test_realworld_history_reset_and_thread_inference(tmp_path, monkeypatch): + from concurrent.futures import ThreadPoolExecutor + from types import SimpleNamespace + + import numpy as np + from transformers import BatchFeature + + from internnav.agent import internvla_n1_agent_realworld as agent_module + + class StubModel(torch.nn.Module): + def __init__(self): + super().__init__() + self.calls = [] + + def generate(self, **kwargs): + self.calls.append(('generate', torch.is_grad_enabled())) + assert kwargs['input_ids'].dtype == torch.long + return SimpleNamespace(sequences=torch.ones(1, 3, dtype=torch.long), past_key_values=None) + + def generate_latents(self, *args): + self.calls.append(('latents', torch.is_grad_enabled())) + return torch.zeros(1, 4, 3584, dtype=torch.bfloat16) + + def generate_traj(self, *args): + self.calls.append(('trajectory', torch.is_grad_enabled())) + return torch.zeros(2, 4, 3, dtype=torch.bfloat16) + + class StubProcessor: + def __init__(self): + self.tokenizer = SimpleNamespace(decode=lambda *args, **kwargs: self.response) + self.response = 'STOP' + self.image_counts = [] + + def apply_chat_template(self, *args, **kwargs): + return 'prompt' + + def __call__(self, text, images, return_tensors): + self.image_counts.append(len(images)) + return BatchFeature({ + 'input_ids': torch.ones(1, 2, dtype=torch.long), + 'pixel_values': torch.zeros(1, 3), + 'image_grid_thw': torch.ones(1, 3, dtype=torch.long), + }) + + model, processor = StubModel(), StubProcessor() + monkeypatch.setattr(agent_module.InternVLAN1ForCausalLM, 'from_pretrained', lambda *args, **kwargs: model) + monkeypatch.setattr(agent_module.AutoProcessor, 'from_pretrained', lambda *args, **kwargs: processor) + monkeypatch.chdir(tmp_path) + agent = agent_module.InternVLAN1AsyncAgent(SimpleNamespace( + device='cpu', model_path='stub', attn_implementation='eager', + resize_w=28, resize_h=28, num_history=2, plan_step_gap=4, + )) + agent.reset() + arguments = (np.zeros((28, 28, 3), dtype=np.uint8), np.zeros((28, 28), dtype=np.float32), np.eye(4), 'Stop.', np.eye(4)) + with ThreadPoolExecutor(max_workers=1) as worker: + result = worker.submit(agent.step, *arguments).result() + assert result.output_action == [0] + processor.response = '(10, 20)' + result = worker.submit(agent.step, *arguments).result() + assert result.output_pixel == [20, 10] + assert result.output_trajectory is not None + assert processor.image_counts == [1, 2] + processor.response = 'STOP' + result = worker.submit(agent.step, *arguments, look_down=True).result() + assert result.output_action == [0] + assert processor.image_counts == [1, 2, 3] + assert agent.episode_idx == 2 + assert len(agent.conversation_history) == 3 + assert {name for name, enabled in model.calls} == {'generate', 'latents', 'trajectory'} + assert not any(enabled for name, enabled in model.calls) + assert torch.is_grad_enabled() + agent.reset() + assert agent.episode_idx == 0 and agent.rgb_list == [] and agent.conversation_history == [] + assert agent.output_action is None and agent.output_latent is None and agent.past_key_values is None + + +def test_policy_async_worker_publishes_output(monkeypatch): + import threading + from concurrent.futures import ThreadPoolExecutor + from types import SimpleNamespace + + from internnav.agent import internvla_n1_agent as agent_module + from internnav.model.utils.vln_utils import S2Input, S2Output + + agent = agent_module.InternVLAN1Agent.__new__(agent_module.InternVLAN1Agent) + agent.s2_input = S2Input(idx=3, should_infer=True) + agent.s2_output = S2Output() + agent.camera_intrinsic = None + agent.s2_input_lock = threading.Lock() + agent.s2_output_lock = threading.Lock() + agent.s2_agent_lock = threading.Lock() + agent.policy = SimpleNamespace(s2_step=lambda *args: S2Output(output_action=[0])) + targets = [] + + def stop_after_iteration(seconds): + raise StopIteration('Worker iteration complete') + + monkeypatch.setattr(agent_module, 'time', SimpleNamespace(sleep=stop_after_iteration)) + monkeypatch.setattr(agent_module, 'threading', SimpleNamespace( + Thread=lambda target: SimpleNamespace(start=lambda: targets.append(target)) + )) + agent._start_s2_thread() + with ThreadPoolExecutor(max_workers=1) as worker: + with pytest.raises(StopIteration, match='Worker iteration complete'): + worker.submit(targets[0]).result() + assert agent.s2_output.output_action == [0] + assert agent.s2_output.idx == 3 + assert not agent.s2_input.should_infer and not agent.s2_output.is_infering + + +@pytest.mark.slow +def test_full_checkpoint_xpu_eager(tmp_path, monkeypatch): + import json + import os + from concurrent.futures import ThreadPoolExecutor + from pathlib import Path + from types import SimpleNamespace + + import numpy as np + from PIL import Image + + from internnav.agent.internvla_n1_agent_realworld import InternVLAN1AsyncAgent + + if os.environ.get('INTERNNAV_FULL_XPU_TEST') != '1': + pytest.skip('Set INTERNNAV_FULL_XPU_TEST=1 for full checkpoint validation') + if not torch.xpu.is_available(): + pytest.skip('Native XPU unavailable') + root = Path(__file__).resolve().parents[2] + checkpoint = root / 'checkpoints/InternVLA-N1-DualVLN' + index = json.loads((checkpoint / 'model.safetensors.index.json').read_text()) + weight_bytes = index['metadata']['total_size'] + free_bytes, total_bytes = torch.xpu.mem_get_info() + if weight_bytes + 2 * 1024**3 > free_bytes: + pytest.skip(f'Full checkpoint needs {weight_bytes} weight bytes plus inference headroom; XPU free={free_bytes}, total={total_bytes}') + from internnav.model.basemodel.internvla_n1 import internvla_n1_arch as arch + + monkeypatch.setattr(arch, 'MODEL_PATH_TO', str(root / 'checkpoints')) + monkeypatch.chdir(tmp_path) + torch.xpu.reset_peak_memory_stats() + agent = InternVLAN1AsyncAgent(SimpleNamespace( + device='xpu:0', attn_implementation='auto', model_path=str(checkpoint), + resize_w=384, resize_h=384, num_history=2, plan_step_gap=4, + )) + agent.reset() + image = Image.new('RGB', (384, 384), color='green') + messages = [{'role': 'user', 'content': [{'type': 'image'}, {'type': 'text', 'text': 'Stop at the door.'}]}] + text = agent.processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) + inputs = agent.processor(text=[text], images=[image], return_tensors='pt').to(agent.device) + + @torch.no_grad() + def infer(): + outputs = agent.model.generate(**inputs, max_new_tokens=4, min_new_tokens=2, do_sample=False, use_cache=True, return_dict_in_generate=True) + assert outputs.past_key_values is not None + latent = agent.model.generate_latents(outputs.sequences, inputs.pixel_values, inputs.image_grid_thw) + trajectory = agent.step_s1(latent, torch.zeros(1, 2, 224, 224, 3), None) + assert trajectory.shape == (32, 32, 3) and trajectory.isfinite().all() + arguments = (np.asarray(image), np.zeros((384, 384), dtype=np.float32), np.eye(4), 'Stop at the door.', np.eye(4)) + agent.step_s2(*arguments) + agent.step_s2(*arguments) + assert agent.episode_idx == 2 + agent.reset() + assert agent.episode_idx == 0 and not agent.rgb_list + + with ThreadPoolExecutor(max_workers=1) as worker: + worker.submit(infer).result() + torch.xpu.synchronize() + print(f'Full checkpoint XPU peak allocated bytes: {torch.xpu.max_memory_allocated()}') \ No newline at end of file From ebb5f52aa5ba43da2e5a84b5619f2e4943676e0b Mon Sep 17 00:00:00 2001 From: Rohit Date: Tue, 6 Oct 2026 11:59:22 +0800 Subject: [PATCH 2/8] shape mismatch fix --- docs/xpu_inference.md | 132 ++++---------------------- requirements/internvla_n1_xpu.txt | 5 +- tests/unit_test/test_xpu_inference.py | 24 +++++ 3 files changed, 49 insertions(+), 112 deletions(-) diff --git a/docs/xpu_inference.md b/docs/xpu_inference.md index faf6cf0c..8284061f 100644 --- a/docs/xpu_inference.md +++ b/docs/xpu_inference.md @@ -1,47 +1,26 @@ # Optional Intel XPU eager inference -This route targets the InternVLA-N1-DualVLN model only. Native XPU component -execution has been tested; full-checkpoint compatibility has **not** been -established. It does not establish Habitat, Isaac, simulator, sensor, or robot -stack compatibility. There is no compilation, quantization, CPU offload, or -static-cache change. +Experimental eager inference for InternVLA-N1-DualVLN on Intel GPUs. Simulator +and robot-stack integrations are not covered. -## Prerequisites and installation +## Installation -Use a supported Intel GPU, a Linux kernel/Intel GPU driver supporting that GPU, -the Intel Level Zero compute runtime, and permission to access `/dev/dri/render*`. -PCI visibility alone is insufficient. Follow the current -[PyTorch Intel GPU prerequisites](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html) -for your OS/device. Driver installation is an administrator task; PyTorch wheels -do not install the kernel driver. Do not install IPEX or CUDA-only attention -extensions for this route. - -From the repository root, use the existing `venv`. The official stable XPU index -was queried on 2026-09-30 and reported torch 2.14.0 and torchvision 0.29.0: +Install the Intel GPU driver and runtime per the +[PyTorch Intel GPU prerequisites](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html). +Then, from the repository root, in a dedicated virtual environment: ```bash -venv/bin/python -m pip index versions torch --index-url https://download.pytorch.org/whl/xpu -venv/bin/python -m pip index versions torchvision --index-url https://download.pytorch.org/whl/xpu -venv/bin/python -m pip install torch==2.14.0 torchvision==0.29.0 --index-url https://download.pytorch.org/whl/xpu venv/bin/python -m pip install -r requirements/internvla_n1_xpu.txt venv/bin/python -m pip check venv/bin/python -c "import torch; assert torch.xpu.is_available(); print(torch.__version__); print(torch.xpu.get_device_properties(0)); print(torch.xpu.mem_get_info(0))" ``` -Transformers remains 4.51.0, Diffusers 0.33.1, and Accelerate 1.4.0. The separate -inference requirements do not require `flash_attn`, xFormers, or diffusion-policy. -Run from the repository root so imports resolve without installing all simulator -extras. The original CUDA requirements and installation route are unchanged; -do not combine the CUDA extras with this minimal XPU environment. Some existing -agent imports emit warnings about absent Habitat/depth-camera-filtering; those -packages are not needed for this standalone DualVLN route. +Do not install XPU packages into a CUDA environment. Keep `diffusers==0.32.2`; +newer versions fail to load the checkpoint with a `size mismatch` error. ## Auxiliary checkpoint -Normal pretrained depth/RGB initialization is retained, including strict -`load_state_dict` checks. There is no skip-loading flag and no random-encoder -fallback, even when restoring a full DualVLN checkpoint. Supply the auxiliary -checkpoint from its upstream source before constructing the model: +Download the Depth Anything V2 checkpoint: ```bash curl -fL --retry 2 -o checkpoints/depth_anything_v2_metric_hypersim_vits.pth \ @@ -49,24 +28,18 @@ curl -fL --retry 2 -o checkpoints/depth_anything_v2_metric_hypersim_vits.pth \ sha256sum checkpoints/depth_anything_v2_metric_hypersim_vits.pth ``` -The downloaded file used for validation has SHA256 -`b782898d8a3e8be1f639de33837ed85e9b4b73e40f8f5e5cd99067588d722545`. -It is a local ignored checkpoint, not part of the source patch. Missing files, -missing weights, and mismatched shapes remain fatal. Partial checkpoints and -training continue to use the normal pretrained initializer. +Expected SHA256: `b782898d8a3e8be1f639de33837ed85e9b4b73e40f8f5e5cd99067588d722545`. -## Configuration and launch +## Configuration -Both inference loaders accept `attn_implementation` (default `auto`): +Set `attn_implementation` (default `auto`): | Device | `auto` | Other selections | | --- | --- | --- | -| CUDA | `flash_attention_2`, unchanged | `sdpa`, `eager`, explicit FA2 | -| XPU | `sdpa` | `eager` for debugging; external FA2 is rejected | +| CUDA | `flash_attention_2` | `sdpa`, `eager` | +| XPU | `sdpa` | `eager`; `flash_attention_2` is not supported | -An explicit unavailable XPU or invalid XPU index raises an error, never silently -falls back to CPU. Existing CUDA device defaults are unchanged. BF16 and the -custom model/processor are retained. Policy configuration can set: +Policy configuration example: ```python device = 'xpu:0' @@ -74,7 +47,7 @@ attn_implementation = 'auto' model_path = 'checkpoints/InternVLA-N1-DualVLN' ``` -On an XPU with enough available memory: +## Launch ```bash venv/bin/python -m scripts.realworld.http_internvla_server \ @@ -82,78 +55,15 @@ venv/bin/python -m scripts.realworld.http_internvla_server \ --device xpu:0 --attn_implementation auto --plan_step_gap 4 --skip_warmup ``` -The server listens on port 5801. `--skip_warmup` is opt-in; it bypasses the existing -demo warmup, whose call omits the required camera intrinsic argument and uses -floating-point RGB input. The original default warmup branch is unchanged. -The agent is reset before serving when the bypass is selected. This is a local -development server with shared episode state, not a concurrent multi-client or -production server. Its existing hard-coded instruction behavior is unchanged. - -## Memory and runtime evidence +The server listens on port 5801. -DualVLN weights require 16,767,077,782 bytes (15.62 GiB), before KV cache, visual -activations, diffusion buffers, allocator reservations, and runtime overhead. -History increases memory use. More than 15.62 GiB of **available** device memory -is required; no universal sufficient-memory threshold has been established. -Shared system memory and swap are not equivalent to available XPU memory. +## Memory -Validation on 2026-09-30 used the existing Python 3.12 venv, Linux kernel -7.0.0-31-generic, Intel Arc B390 (PCI 8086:b082), Level Zero V2 driver -1.15.39122+14. PyTorch reported 14,939,189,248 bytes (13.91 GiB) total and -1,558,061,056 bytes free at the initial probe. Full-model loading was therefore -not attempted. CUDA was unavailable. +Model weights alone need 15.62 GiB of free GPU memory; inference requires more. -Exact principal versions: torch 2.14.0+xpu, torchvision 0.29.0+xpu, -transformers 4.51.0, diffusers 0.33.1, accelerate 1.4.0, numpy 1.26.4, -Pillow 11.3.0, imageio 2.37.0, pydantic 2.11.10, pytest 9.1.1. +## Tests ```bash venv/bin/python -m pip install pytest INTERNNAV_FULL_XPU_TEST=1 venv/bin/python -m pytest tests/unit_test/test_xpu_inference.py -q -s -``` - -Result: **23 passed, 1 skipped**. The opt-in full-checkpoint test skipped on its -memory gate: weights 16,767,077,782 bytes, total 14,939,189,248 bytes, free -712,245,248 bytes during the combined test run. Its two-GiB headroom check is a -minimum safety gate, not a guarantee that a full run will fit. - -Tested scenarios: - -- Device/attention selection, unavailable XPU, invalid index, and FA2 rejection. -- DINO native CPU/XPU fallback even with simulated xFormers availability; - non-None nested biases are rejected rather than discarded. -- Strict auxiliary restoration, including missing file/key and shape failures; - the real pretrained RGB encoder loaded with 22,056,576 finite parameters. -- Actual processor image/instruction handling, custom reduced Qwen prefill, - cached decoding, and latent extraction on XPU in both SDPA and eager modes. - Integer tokens/grid metadata stay integer. CPU BF16 eager reference logits - matched at `rtol=0.05, atol=0.01` with identical weights. Peak XPU allocated - memory: 36,691,968 bytes per mode. -- Actual full-size DINO/memory/QFormer/diffusion components on XPU with two - 224x224 frames, two diffusion steps, two samples, and four predicted steps. - Only DINO uses pretrained weights in this component test; other weights are - deterministic random fixtures. CPU/XPU use the **same noise tensor**, not just - seeds. BF16 trajectories matched at `rtol=0.05, atol=0.05`. Peak XPU allocated - memory: 212,505,600 bytes. These peaks are not full-model memory estimates. -- Stubbed STOP/waypoint outputs, look-down history, episode reset, thread-local - no-grad contexts, and one real asynchronous-worker publication iteration. -- Diffusers checkpoint-toggle API compatibility, CLI help, and `pip check`. - -Not verified: complete pretrained DualVLN execution/quality, long-history memory, -full-model peak memory, CUDA FA2 runtime equivalence, real checkpoint STOP/action -agreement, sustained asynchronous scheduling/reset races, or robot/simulator -integration. The full-checkpoint test is ready for a larger XPU but is not a -substitute for recorded navigation inputs and a CUDA baseline. For that comparison, -copy identical diffusion noise across devices and compare intermediate logits, -latents, trajectories, and resulting actions with explicit BF16 tolerances. - -## Change boundaries - -Shared resolution is used only by the two inference loaders. DINO retains its -existing CUDA xFormers computation. Tensor transfers are confined to inference -methods; RGB normalization and scheduler/guidance math are unchanged. No model -layers or checkpoint keys were added. Gradients are not disabled globally. -Encoder exports are loaded lazily to avoid requiring unrelated LongCLIP just to -import DINO. A backward-compatible checkpoint-toggle signature adapter fixes the -custom diffusion model's constructor under pinned Diffusers 0.33.1; the existing -training checkpoint computation is unchanged. Full training was not tested. \ No newline at end of file +``` \ No newline at end of file diff --git a/requirements/internvla_n1_xpu.txt b/requirements/internvla_n1_xpu.txt index 32bc95b6..1afccddf 100644 --- a/requirements/internvla_n1_xpu.txt +++ b/requirements/internvla_n1_xpu.txt @@ -1,5 +1,8 @@ +--extra-index-url https://download.pytorch.org/whl/xpu +torch==2.14.1+xpu +torchvision==0.29.1+xpu accelerate==1.4.0 -diffusers==0.33.1 +diffusers==0.32.2 transformers==4.51.0 ftfy==6.3.1 einops==0.8.1 diff --git a/tests/unit_test/test_xpu_inference.py b/tests/unit_test/test_xpu_inference.py index abfdd7e0..9057d360 100644 --- a/tests/unit_test/test_xpu_inference.py +++ b/tests/unit_test/test_xpu_inference.py @@ -253,6 +253,30 @@ def fixed_noise(shape, generator, device, dtype): print(f'Trajectory peak allocated bytes: {torch.xpu.max_memory_allocated()}') +def test_traj_dit_shapes_match_checkpoint(): + import json + from pathlib import Path + + from safetensors import safe_open + + from internnav.model.basemodel.internvla_n1.nextdit_crossattn_traj import NextDiTCrossAttn, NextDiTCrossAttnConfig + + checkpoint = Path(__file__).resolve().parents[2] / 'checkpoints/InternVLA-N1-DualVLN' + index_path = checkpoint / 'model.safetensors.index.json' + if not index_path.exists(): + pytest.skip('Local DualVLN checkpoint unavailable') + prefix = 'model.traj_dit.' + weight_map = {k: f for k, f in json.loads(index_path.read_text())['weight_map'].items() if k.startswith(prefix)} + model_shapes = { + prefix + k: tuple(v.shape) for k, v in NextDiTCrossAttn(NextDiTCrossAttnConfig(latent_embedding_size=768)).state_dict().items() + } + checkpoint_shapes = {} + for key, file in weight_map.items(): + with safe_open(checkpoint / file, 'pt') as shard: + checkpoint_shapes[key] = tuple(shard.get_slice(key).get_shape()) + assert model_shapes == checkpoint_shapes + + def test_diffusers_checkpointing_toggle_compatibility(): from internnav.model.basemodel.internvla_n1.nextdit_crossattn_traj import NextDiTCrossAttn, NextDiTCrossAttnConfig From 7c8888d49c97759276efac5a8632e4b835b8b681 Mon Sep 17 00:00:00 2001 From: Rohit Date: Tue, 6 Oct 2026 12:21:55 +0800 Subject: [PATCH 3/8] doc updates --- README.md | 2 +- docs/xpu_inference.md | 12 +++++++----- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 210df5c4..e8b62ab0 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,7 @@ The toolbox supports the most advanced high-quality navigation dataset, InternDa Please refer to the [documentation](https://internrobotics.github.io/user_guide/internnav/quick_start/index.html) for quick start with InternNav, from installation to training or evaluating supported models. -For optional native Intel XPU eager model inference, see [setup and validation limits](docs/xpu_inference.md). The existing CUDA installation route is unchanged. +For native Intel XPU inference, see [setup and validation limits](docs/xpu_inference.md). The existing CUDA installation route is unchanged. ## ๐Ÿ“ฆ Overview diff --git a/docs/xpu_inference.md b/docs/xpu_inference.md index 8284061f..a5d36002 100644 --- a/docs/xpu_inference.md +++ b/docs/xpu_inference.md @@ -1,7 +1,6 @@ # Optional Intel XPU eager inference -Experimental eager inference for InternVLA-N1-DualVLN on Intel GPUs. Simulator -and robot-stack integrations are not covered. +Eager inference for InternVLA-N1-DualVLN on Intel GPUs. ## Installation @@ -20,7 +19,7 @@ newer versions fail to load the checkpoint with a `size mismatch` error. ## Auxiliary checkpoint -Download the Depth Anything V2 checkpoint: +The DualVLN asynchronous RGB path requires this checkpoint: ```bash curl -fL --retry 2 -o checkpoints/depth_anything_v2_metric_hypersim_vits.pth \ @@ -28,10 +27,13 @@ curl -fL --retry 2 -o checkpoints/depth_anything_v2_metric_hypersim_vits.pth \ sha256sum checkpoints/depth_anything_v2_metric_hypersim_vits.pth ``` -Expected SHA256: `b782898d8a3e8be1f639de33837ed85e9b4b73e40f8f5e5cd99067588d722545`. - ## Configuration +For policy-based inference, set `device` and `attn_implementation` in the model +configuration consumed by `ModelCfg` in `internnav/configs/model/base_encoders.py`. +For the real-world HTTP server, use the CLI options shown below; they are defined +in `scripts/realworld/http_internvla_server.py`. + Set `attn_implementation` (default `auto`): | Device | `auto` | Other selections | From 61ab861d24063893754a54ad58566e15f0c24c12 Mon Sep 17 00:00:00 2001 From: Rohit Date: Tue, 6 Oct 2026 12:28:59 +0800 Subject: [PATCH 4/8] numerical equivalence tests --- tests/unit_test/test_xpu_inference.py | 57 +++++++++++++++++++-------- 1 file changed, 41 insertions(+), 16 deletions(-) diff --git a/tests/unit_test/test_xpu_inference.py b/tests/unit_test/test_xpu_inference.py index 9057d360..41cc2fe1 100644 --- a/tests/unit_test/test_xpu_inference.py +++ b/tests/unit_test/test_xpu_inference.py @@ -7,6 +7,13 @@ from internnav.model.encoder.depth_anything.depth_anything_v2.dinov2_layers import attention as dino_attention +def _require_accelerator(device_type): + backend = getattr(torch, device_type, None) + if backend is None or not backend.is_available(): + pytest.skip(f'{device_type} unavailable') + return backend + + def test_cuda_default_attention(): device, attention = resolve_inference_device('cuda:0') assert str(device) == 'cuda:0' @@ -63,6 +70,17 @@ def test_dino_native_fallback(device, xformers_available): layer(inputs, attn_bias=object()) +def test_dino_cuda_xformers_matches_native(): + _require_accelerator('cuda') + if not dino_attention.XFORMERS_AVAILABLE: + pytest.skip('xFormers unavailable') + torch.manual_seed(3) + layer = dino_attention.MemEffAttention(32, num_heads=4).to('cuda:0').eval() + inputs = torch.randn(2, 8, 32, device='cuda:0') + with torch.no_grad(): + torch.testing.assert_close(layer(inputs), dino_attention.Attention.forward(layer, inputs), rtol=1e-3, atol=1e-3) + + def test_latent_tokens_move_to_embedding_device(): from types import SimpleNamespace @@ -132,8 +150,9 @@ def __init__(self, **kwargs): torch.testing.assert_close(rgb.bias, weights['pretrained.bias']) -@pytest.mark.parametrize('attention', ['sdpa', 'eager']) -def test_xpu_bf16_prefill_cache_and_latents(attention): +@pytest.mark.parametrize('device_type', ['xpu', 'cuda']) +@pytest.mark.parametrize('attention', ['auto', 'sdpa', 'eager']) +def test_accelerator_bf16_prefill_cache_and_latents(device_type, attention): import copy from pathlib import Path @@ -145,9 +164,13 @@ def test_xpu_bf16_prefill_cache_and_latents(attention): InternVLAN1ModelConfig, ) + backend = _require_accelerator(device_type) checkpoint = Path(__file__).resolve().parents[2] / 'checkpoints/InternVLA-N1-DualVLN' - if not torch.xpu.is_available() or not (checkpoint / 'preprocessor_config.json').exists(): - pytest.skip('Requires native XPU and local DualVLN processor') + if not (checkpoint / 'preprocessor_config.json').exists(): + pytest.skip('Requires local DualVLN processor') + device, attention = resolve_inference_device(f'{device_type}:0', attention) + if attention == 'flash_attention_2': + pytest.importorskip('flash_attn') config = InternVLAN1ModelConfig( vocab_size=152064, hidden_size=32, intermediate_size=64, num_hidden_layers=1, num_attention_heads=4, num_key_value_heads=2, @@ -171,15 +194,15 @@ def test_xpu_bf16_prefill_cache_and_latents(attention): model = InternVLAN1ForCausalLM(device_config).eval().to(torch.bfloat16) model.model.latent_queries = torch.nn.Parameter(torch.empty(1, 4, 32, dtype=torch.bfloat16)) model.load_state_dict(reference.state_dict()) - model.to('xpu:0') + model.to(device) processor = AutoProcessor.from_pretrained(checkpoint, min_pixels=784, max_pixels=784, local_files_only=True) messages = [{'role': 'user', 'content': [{'type': 'image'}, {'type': 'text', 'text': 'Stop at the door.'}]}] text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) inputs = processor(text=[text], images=[Image.new('RGB', (28, 28), color='green')], return_tensors='pt') - torch.xpu.reset_peak_memory_stats() + backend.reset_peak_memory_stats() with torch.no_grad(): expected = reference(**inputs, use_cache=True).logits - device_inputs = copy.deepcopy(inputs).to('xpu:0') + device_inputs = copy.deepcopy(inputs).to(device) actual = model(**device_inputs, use_cache=True) assert actual.past_key_values is not None torch.testing.assert_close(actual.logits.cpu(), expected, rtol=0.05, atol=0.01) @@ -187,14 +210,15 @@ def test_xpu_bf16_prefill_cache_and_latents(attention): assert output.shape[1] == inputs.input_ids.shape[1] + 2 latent = model.generate_latents(output.cpu(), inputs.pixel_values, inputs.image_grid_thw) assert latent.shape == (1, 4, 32) - assert latent.device.type == 'xpu' and latent.dtype == torch.bfloat16 + assert latent.device.type == device_type and latent.dtype == torch.bfloat16 assert latent.isfinite().all() assert device_inputs.input_ids.dtype == torch.long assert device_inputs.image_grid_thw.dtype == torch.long - print(f'{attention} tiny Qwen peak allocated bytes: {torch.xpu.max_memory_allocated()}') + print(f'{device_type} {attention} tiny Qwen peak allocated bytes: {backend.max_memory_allocated()}') -def test_xpu_bf16_trajectory_with_identical_noise(monkeypatch): +@pytest.mark.parametrize('device_type', ['xpu', 'cuda']) +def test_accelerator_bf16_trajectory_with_identical_noise(monkeypatch, device_type): import copy from pathlib import Path from types import SimpleNamespace @@ -202,9 +226,10 @@ def test_xpu_bf16_trajectory_with_identical_noise(monkeypatch): from internnav.model.basemodel.internvla_n1 import internvla_n1 as model_module from internnav.model.basemodel.internvla_n1 import internvla_n1_arch as arch + backend = _require_accelerator(device_type) checkpoint_dir = Path(__file__).resolve().parents[2] / 'checkpoints' - if not torch.xpu.is_available() or not (checkpoint_dir / 'depth_anything_v2_metric_hypersim_vits.pth').exists(): - pytest.skip('Requires native XPU and auxiliary pretrained checkpoint') + if not (checkpoint_dir / 'depth_anything_v2_metric_hypersim_vits.pth').exists(): + pytest.skip('Requires auxiliary pretrained checkpoint') monkeypatch.setattr(arch, 'MODEL_PATH_TO', str(checkpoint_dir)) class TrajectoryModel(torch.nn.Module): @@ -233,7 +258,7 @@ def get_system1_type(self): torch.manual_seed(11) reference = TrajectoryModel().eval().to(torch.bfloat16) - model = copy.deepcopy(reference).to('xpu:0') + model = copy.deepcopy(reference).to(f'{device_type}:0') noise = torch.randn(2, 4, 3, dtype=torch.bfloat16) latent = torch.randn(1, 4, 3584, dtype=torch.bfloat16) images = torch.rand(1, 2, 224, 224, 3, dtype=torch.float64) @@ -243,14 +268,14 @@ def fixed_noise(shape, generator, device, dtype): return noise.clone().to(device=device, dtype=dtype) monkeypatch.setattr(model_module, 'randn_tensor', fixed_noise) - torch.xpu.reset_peak_memory_stats() + backend.reset_peak_memory_stats() with torch.no_grad(): expected = reference.generate_traj(latent, images, predict_step_nums=4, num_inference_steps=2, num_sample_trajs=2) actual = model.generate_traj(latent, images, predict_step_nums=4, num_inference_steps=2, num_sample_trajs=2) - assert actual.device.type == 'xpu' + assert actual.device.type == device_type assert actual.isfinite().all() torch.testing.assert_close(actual.cpu(), expected, rtol=0.05, atol=0.05) - print(f'Trajectory peak allocated bytes: {torch.xpu.max_memory_allocated()}') + print(f'{device_type} trajectory peak allocated bytes: {backend.max_memory_allocated()}') def test_traj_dit_shapes_match_checkpoint(): From 6e7234aca19167fb6b3f9bba90980ac5ee43bed0 Mon Sep 17 00:00:00 2001 From: Rohit Date: Tue, 6 Oct 2026 13:10:17 +0800 Subject: [PATCH 5/8] test fix --- tests/unit_test/test_xpu_inference.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/tests/unit_test/test_xpu_inference.py b/tests/unit_test/test_xpu_inference.py index 41cc2fe1..1ee027f2 100644 --- a/tests/unit_test/test_xpu_inference.py +++ b/tests/unit_test/test_xpu_inference.py @@ -14,6 +14,15 @@ def _require_accelerator(device_type): return backend +def _require_supported_transformers(): + import transformers + from packaging.version import Version + + # Transformers 4.52 restructured Qwen2.5-VL modules that InternVLA-N1 builds on. + if Version(transformers.__version__).release[:2] != (4, 51): + pytest.skip(f'InternVLA-N1 requires transformers==4.51.0 (installed {transformers.__version__})') + + def test_cuda_default_attention(): device, attention = resolve_inference_device('cuda:0') assert str(device) == 'cuda:0' @@ -165,6 +174,7 @@ def test_accelerator_bf16_prefill_cache_and_latents(device_type, attention): ) backend = _require_accelerator(device_type) + _require_supported_transformers() checkpoint = Path(__file__).resolve().parents[2] / 'checkpoints/InternVLA-N1-DualVLN' if not (checkpoint / 'preprocessor_config.json').exists(): pytest.skip('Requires local DualVLN processor') @@ -442,6 +452,7 @@ def test_full_checkpoint_xpu_eager(tmp_path, monkeypatch): pytest.skip('Set INTERNNAV_FULL_XPU_TEST=1 for full checkpoint validation') if not torch.xpu.is_available(): pytest.skip('Native XPU unavailable') + _require_supported_transformers() root = Path(__file__).resolve().parents[2] checkpoint = root / 'checkpoints/InternVLA-N1-DualVLN' index = json.loads((checkpoint / 'model.safetensors.index.json').read_text()) From 98a2515016f730d6fa1cbf3ccbc31762c61a688b Mon Sep 17 00:00:00 2001 From: Rohit Date: Mon, 5 Oct 2026 14:58:22 +0530 Subject: [PATCH 6/8] removed debug and test code --- docs/xpu_inference.md | 10 ++--- .../agent/internvla_n1_agent_realworld.py | 5 +-- internnav/configs/model/base_encoders.py | 1 - .../basemodel/internvla_n1/internvla_n1.py | 13 +++--- .../internvla_n1/internvla_n1_policy.py | 4 +- .../basemodel/internvla_n1/nextdit_traj.py | 8 ++-- internnav/model/encoder/__init__.py | 28 ++++--------- .../dinov2_layers/attention.py | 4 +- internnav/model/utils/inference_device.py | 6 ++- requirements/internvla_n1_xpu.txt | 2 +- scripts/realworld/http_internvla_server.py | 1 - tests/unit_test/test_xpu_inference.py | 42 ++++++------------- 12 files changed, 42 insertions(+), 82 deletions(-) diff --git a/docs/xpu_inference.md b/docs/xpu_inference.md index a5d36002..df93418c 100644 --- a/docs/xpu_inference.md +++ b/docs/xpu_inference.md @@ -9,6 +9,7 @@ Install the Intel GPU driver and runtime per the Then, from the repository root, in a dedicated virtual environment: ```bash +git submodule update --init internnav/model/basemodel/LongCLIP venv/bin/python -m pip install -r requirements/internvla_n1_xpu.txt venv/bin/python -m pip check venv/bin/python -c "import torch; assert torch.xpu.is_available(); print(torch.__version__); print(torch.xpu.get_device_properties(0)); print(torch.xpu.mem_get_info(0))" @@ -29,10 +30,9 @@ sha256sum checkpoints/depth_anything_v2_metric_hypersim_vits.pth ## Configuration -For policy-based inference, set `device` and `attn_implementation` in the model -configuration consumed by `ModelCfg` in `internnav/configs/model/base_encoders.py`. -For the real-world HTTP server, use the CLI options shown below; they are defined -in `scripts/realworld/http_internvla_server.py`. +For policy-based inference, set `device` and, optionally, `attn_implementation` in the +model configuration consumed by `ModelCfg` in `internnav/configs/model/base_encoders.py`. +The real-world HTTP server (`scripts/realworld/http_internvla_server.py`) always uses `auto`. Set `attn_implementation` (default `auto`): @@ -54,7 +54,7 @@ model_path = 'checkpoints/InternVLA-N1-DualVLN' ```bash venv/bin/python -m scripts.realworld.http_internvla_server \ --model_path checkpoints/InternVLA-N1-DualVLN \ - --device xpu:0 --attn_implementation auto --plan_step_gap 4 --skip_warmup + --device xpu:0 --plan_step_gap 4 --skip_warmup ``` The server listens on port 5801. diff --git a/internnav/agent/internvla_n1_agent_realworld.py b/internnav/agent/internvla_n1_agent_realworld.py index 641b9436..55b02536 100644 --- a/internnav/agent/internvla_n1_agent_realworld.py +++ b/internnav/agent/internvla_n1_agent_realworld.py @@ -26,9 +26,7 @@ class InternVLAN1AsyncAgent: def __init__(self, args): - self.device, attention = resolve_inference_device( - args.device, getattr(args, 'attn_implementation', 'auto') - ) + self.device, attention = resolve_inference_device(args.device, getattr(args, 'attn_implementation', 'auto')) self.save_dir = "test_data/" + datetime.now().strftime("%Y%m%d_%H%M%S") print(f"args.model_path{args.model_path}") self.model = InternVLAN1ForCausalLM.from_pretrained( @@ -255,7 +253,6 @@ def step_s2(self, rgb, depth, pose, instruction, intrinsic, look_down=False): action_seq = self.parse_actions(self.llm_output) return action_seq, None, None - @torch.no_grad() def step_s1(self, latent, rgb, depth): all_trajs = self.model.generate_traj(latent, rgb, depth) return all_trajs diff --git a/internnav/configs/model/base_encoders.py b/internnav/configs/model/base_encoders.py index cca32906..c5d9e79e 100644 --- a/internnav/configs/model/base_encoders.py +++ b/internnav/configs/model/base_encoders.py @@ -179,7 +179,6 @@ class Seq2Seq(BaseModel, extra='allow'): class ModelCfg(BaseModel, extra='allow'): - attn_implementation: str = 'auto' policy_name: Optional[str] ablate_instruction: Optional[bool] = None ablate_depth: Optional[bool] = None diff --git a/internnav/model/basemodel/internvla_n1/internvla_n1.py b/internnav/model/basemodel/internvla_n1/internvla_n1.py index 0a5a1523..69c7f474 100644 --- a/internnav/model/basemodel/internvla_n1/internvla_n1.py +++ b/internnav/model/basemodel/internvla_n1/internvla_n1.py @@ -318,8 +318,7 @@ def forward( ) def generate_latents(self, input_ids, pixel_values, image_grid_thw): - input_ids = input_ids.to(self.get_model().embed_tokens.weight.device) - image_grid_thw = image_grid_thw.to(input_ids.device) + input_ids.to(self.get_model().device) with torch.no_grad(): text_embeds = self.get_model().embed_tokens(input_ids) latent_queries = self.get_model().latent_queries.repeat(text_embeds.shape[0], 1, 1) @@ -327,7 +326,7 @@ def generate_latents(self, input_ids, pixel_values, image_grid_thw): N_QUERY = self.get_n_query() input_ids = torch.cat([input_ids, torch.tensor([[TRAJ_TOKEN_INDEX] * N_QUERY]).to(input_ids.device)], dim=1) - pixel_values = pixel_values.to(device=self.visual.device, dtype=self.visual.dtype) + pixel_values = pixel_values.type(self.visual.dtype) image_embeds = self.visual(pixel_values, grid_thw=image_grid_thw).unsqueeze(0) text_embeds[image_idx] = image_embeds.to(text_embeds.device)[: image_idx.sum(), :] @@ -359,15 +358,13 @@ def generate_traj( ): if 'nextdit' in self.get_system1_type(): scheduler = FlowMatchEulerDiscreteScheduler() - projector_weight = self.get_model().cond_projector[0].weight - device = projector_weight.device - dtype = projector_weight.dtype - traj_latents = traj_latents.to(device=device, dtype=dtype) + device = traj_latents.device + dtype = traj_latents.dtype traj_latents = self.get_model().cond_projector(traj_latents) if 'async' in self.get_system1_type(): with torch.no_grad(): - images_dp = images_dp.to(device=device).permute(0, 1, 4, 2, 3) + images_dp = images_dp.permute(0, 1, 4, 2, 3) images_dp_norm = (images_dp - self._resnet_mean) / self._resnet_std self.get_model().rgb_model.to(dtype) images_dp_feat = ( diff --git a/internnav/model/basemodel/internvla_n1/internvla_n1_policy.py b/internnav/model/basemodel/internvla_n1/internvla_n1_policy.py index a88a6d82..a4bcb438 100644 --- a/internnav/model/basemodel/internvla_n1/internvla_n1_policy.py +++ b/internnav/model/basemodel/internvla_n1/internvla_n1_policy.py @@ -10,11 +10,11 @@ from transformers import AutoProcessor, AutoTokenizer, PreTrainedModel from internnav.configs.model.base_encoders import ModelCfg -from internnav.model.utils.inference_device import resolve_inference_device from internnav.model.basemodel.internvla_n1.internvla_n1 import ( InternVLAN1ForCausalLM, InternVLAN1ModelConfig, ) +from internnav.model.utils.inference_device import resolve_inference_device from internnav.model.utils.vln_utils import ( S1Output, S2Output, @@ -32,7 +32,7 @@ def __init__(self, config: Union[InternVLAN1ModelConfig, ModelCfg]): self.model_config = ModelCfg(**config.model_cfg['model']) device, attention = resolve_inference_device( - self.model_config.device, self.model_config.attn_implementation + self.model_config.device, getattr(self.model_config, 'attn_implementation', 'auto') ) self.model = InternVLAN1ForCausalLM.from_pretrained( self.model_config.model_path, diff --git a/internnav/model/basemodel/internvla_n1/nextdit_traj.py b/internnav/model/basemodel/internvla_n1/nextdit_traj.py index 694e444f..1c18a63f 100644 --- a/internnav/model/basemodel/internvla_n1/nextdit_traj.py +++ b/internnav/model/basemodel/internvla_n1/nextdit_traj.py @@ -292,11 +292,9 @@ def __init__( assert (hidden_size // num_attention_heads) % 4 == 0, "2d rope needs head dim to be divisible by 4" - def _set_gradient_checkpointing(self, module=None, value=False, *, enable=None, gradient_checkpointing_func=None): - modules = self.modules() if module is None else (module,) - for target in modules: - if hasattr(target, "gradient_checkpointing"): - target.gradient_checkpointing = value if enable is None else enable + def _set_gradient_checkpointing(self, module, value=False): + if hasattr(module, "gradient_checkpointing"): + module.gradient_checkpointing = value def forward( self, diff --git a/internnav/model/encoder/__init__.py b/internnav/model/encoder/__init__.py index 3b0c4f01..dc8c5b83 100644 --- a/internnav/model/encoder/__init__.py +++ b/internnav/model/encoder/__init__.py @@ -1,21 +1,7 @@ -from importlib import import_module - - -_ENCODERS = { - 'PositionalEncoding': '.bert_backbone', - 'DistanceNetwork': '.distance_encoder', - 'ImageEncoder': '.image_clip_encoder', - 'InstructionEncoder': '.instruction_encoder', - 'InstructionLongCLIPEncoder': '.instruction_longCLIP_encoder', - 'LanguageEncoder': '.instruction_roberta_encoder', - 'VisionLanguageEncoder': '.vision_language_encoder', -} -__all__ = list(_ENCODERS) - - -def __getattr__(name): - if name not in _ENCODERS: - raise AttributeError(f'module {__name__!r} has no attribute {name!r}') - encoder = getattr(import_module(_ENCODERS[name], __name__), name) - globals()[name] = encoder - return encoder +from .bert_backbone import PositionalEncoding +from .distance_encoder import DistanceNetwork +from .image_clip_encoder import ImageEncoder +from .instruction_encoder import InstructionEncoder +from .instruction_longCLIP_encoder import InstructionLongCLIPEncoder +from .instruction_roberta_encoder import LanguageEncoder +from .vision_language_encoder import VisionLanguageEncoder diff --git a/internnav/model/encoder/depth_anything/depth_anything_v2/dinov2_layers/attention.py b/internnav/model/encoder/depth_anything/depth_anything_v2/dinov2_layers/attention.py index 48f26cbe..815a2bf5 100644 --- a/internnav/model/encoder/depth_anything/depth_anything_v2/dinov2_layers/attention.py +++ b/internnav/model/encoder/depth_anything/depth_anything_v2/dinov2_layers/attention.py @@ -64,8 +64,8 @@ def forward(self, x: Tensor) -> Tensor: class MemEffAttention(Attention): def forward(self, x: Tensor, attn_bias=None) -> Tensor: - if not XFORMERS_AVAILABLE or x.device.type != 'cuda': - assert attn_bias is None, "CUDA xFormers is required for nested tensors usage" + if not XFORMERS_AVAILABLE: + assert attn_bias is None, "xFormers is required for nested tensors usage" return super().forward(x) B, N, C = x.shape diff --git a/internnav/model/utils/inference_device.py b/internnav/model/utils/inference_device.py index 6b8bca94..e598129d 100644 --- a/internnav/model/utils/inference_device.py +++ b/internnav/model/utils/inference_device.py @@ -10,9 +10,11 @@ def resolve_inference_device(device, attn_implementation='auto'): if attn_implementation == 'flash_attention_2': raise ValueError('External FlashAttention 2 is not supported on XPU; use sdpa or eager') if not hasattr(torch, 'xpu') or not torch.xpu.is_available(): - raise RuntimeError('XPU was requested but is unavailable; check native PyTorch XPU wheels and Intel drivers') + raise RuntimeError( + 'XPU was requested but is unavailable; check native PyTorch XPU wheels and Intel drivers' + ) if device.index is not None and device.index >= torch.xpu.device_count(): raise RuntimeError(f'Requested XPU device {device} does not exist') if attn_implementation == 'auto': attn_implementation = 'flash_attention_2' if device.type == 'cuda' else 'sdpa' - return device, attn_implementation \ No newline at end of file + return device, attn_implementation diff --git a/requirements/internvla_n1_xpu.txt b/requirements/internvla_n1_xpu.txt index 1afccddf..882bf39c 100644 --- a/requirements/internvla_n1_xpu.txt +++ b/requirements/internvla_n1_xpu.txt @@ -16,4 +16,4 @@ pillow>=11,<12 numpy-quaternion==2024.0.13 scipy>=1.15,<1.18 fastapi>=0.110,<0.111 -uvicorn>=0.29,<0.31 \ No newline at end of file +uvicorn>=0.29,<0.31 diff --git a/scripts/realworld/http_internvla_server.py b/scripts/realworld/http_internvla_server.py index bbeebaad..45aa4499 100644 --- a/scripts/realworld/http_internvla_server.py +++ b/scripts/realworld/http_internvla_server.py @@ -81,7 +81,6 @@ def eval_dual(): parser = argparse.ArgumentParser() parser.add_argument("--device", type=str, default="cuda:0") - parser.add_argument("--attn_implementation", choices=['auto', 'flash_attention_2', 'sdpa', 'eager'], default='auto') parser.add_argument("--model_path", type=str, default="checkpoints/InternVLA-N1") parser.add_argument("--resize_w", type=int, default=384) parser.add_argument("--resize_h", type=int, default=384) diff --git a/tests/unit_test/test_xpu_inference.py b/tests/unit_test/test_xpu_inference.py index 1ee027f2..4ca78bbf 100644 --- a/tests/unit_test/test_xpu_inference.py +++ b/tests/unit_test/test_xpu_inference.py @@ -64,13 +64,12 @@ def test_invalid_attention(): @pytest.mark.parametrize('device', ['cpu', 'xpu:0']) -@pytest.mark.parametrize('xformers_available', [False, True]) -def test_dino_native_fallback(device, xformers_available): +def test_dino_native_fallback(device): if device.startswith('xpu') and not torch.xpu.is_available(): pytest.skip('Native XPU unavailable') layer = dino_attention.MemEffAttention(32, num_heads=4).to(device).eval() inputs = torch.randn(2, 8, 32, device=device) - with patch.object(dino_attention, 'XFORMERS_AVAILABLE', xformers_available), patch.object( + with patch.object(dino_attention, 'XFORMERS_AVAILABLE', False), patch.object( dino_attention, 'memory_efficient_attention', create=True, side_effect=AssertionError('Called xFormers') ), torch.no_grad(): expected = dino_attention.Attention.forward(layer, inputs) @@ -90,28 +89,6 @@ def test_dino_cuda_xformers_matches_native(): torch.testing.assert_close(layer(inputs), dino_attention.Attention.forward(layer, inputs), rtol=1e-3, atol=1e-3) -def test_latent_tokens_move_to_embedding_device(): - from types import SimpleNamespace - - from internnav.model.basemodel.internvla_n1.internvla_n1 import InternVLAN1ForCausalLM - - class EmbeddingProbe(torch.nn.Module): - def __init__(self): - super().__init__() - self.weight = torch.nn.Parameter(torch.empty(1, device='meta')) - - def forward(self, input_ids): - assert input_ids.device.type == 'meta' - assert input_ids.dtype == torch.long - raise RuntimeError('Embedding placement verified') - - model = SimpleNamespace(get_model=lambda: SimpleNamespace(embed_tokens=EmbeddingProbe())) - with pytest.raises(RuntimeError, match='Embedding placement verified'): - InternVLAN1ForCausalLM.generate_latents( - model, torch.ones(1, 4, dtype=torch.long), torch.zeros(1, 4), torch.ones(1, 3, dtype=torch.long) - ) - - @pytest.mark.parametrize('failure', ['missing_file', 'missing_weight', 'wrong_shape']) def test_auxiliary_checkpoint_fails_closed(tmp_path, monkeypatch, failure): from types import SimpleNamespace @@ -218,7 +195,7 @@ def test_accelerator_bf16_prefill_cache_and_latents(device_type, attention): torch.testing.assert_close(actual.logits.cpu(), expected, rtol=0.05, atol=0.01) output = model.generate(**device_inputs, max_new_tokens=2, min_new_tokens=2, do_sample=False, use_cache=True) assert output.shape[1] == inputs.input_ids.shape[1] + 2 - latent = model.generate_latents(output.cpu(), inputs.pixel_values, inputs.image_grid_thw) + latent = model.generate_latents(output, device_inputs.pixel_values, device_inputs.image_grid_thw) assert latent.shape == (1, 4, 32) assert latent.device.type == device_type and latent.dtype == torch.bfloat16 assert latent.isfinite().all() @@ -279,9 +256,14 @@ def fixed_noise(shape, generator, device, dtype): monkeypatch.setattr(model_module, 'randn_tensor', fixed_noise) backend.reset_peak_memory_stats() - with torch.no_grad(): + # xFormers has no CPU kernels, so the CPU reference must use the native DINO attention. + with torch.no_grad(), patch.object(dino_attention, 'XFORMERS_AVAILABLE', False): expected = reference.generate_traj(latent, images, predict_step_nums=4, num_inference_steps=2, num_sample_trajs=2) - actual = model.generate_traj(latent, images, predict_step_nums=4, num_inference_steps=2, num_sample_trajs=2) + with torch.no_grad(): + actual = model.generate_traj( + latent.to(f'{device_type}:0'), images.to(f'{device_type}:0'), + predict_step_nums=4, num_inference_steps=2, num_sample_trajs=2, + ) assert actual.device.type == device_type assert actual.isfinite().all() torch.testing.assert_close(actual.cpu(), expected, rtol=0.05, atol=0.05) @@ -394,7 +376,7 @@ def __call__(self, text, images, return_tensors): assert agent.episode_idx == 2 assert len(agent.conversation_history) == 3 assert {name for name, enabled in model.calls} == {'generate', 'latents', 'trajectory'} - assert not any(enabled for name, enabled in model.calls) + assert not any(enabled for name, enabled in model.calls if name != 'trajectory') assert torch.is_grad_enabled() agent.reset() assert agent.episode_idx == 0 and agent.rgb_list == [] and agent.conversation_history == [] @@ -480,7 +462,7 @@ def infer(): outputs = agent.model.generate(**inputs, max_new_tokens=4, min_new_tokens=2, do_sample=False, use_cache=True, return_dict_in_generate=True) assert outputs.past_key_values is not None latent = agent.model.generate_latents(outputs.sequences, inputs.pixel_values, inputs.image_grid_thw) - trajectory = agent.step_s1(latent, torch.zeros(1, 2, 224, 224, 3), None) + trajectory = agent.step_s1(latent, torch.zeros(1, 2, 224, 224, 3, device=agent.device), None) assert trajectory.shape == (32, 32, 3) and trajectory.isfinite().all() arguments = (np.asarray(image), np.zeros((384, 384), dtype=np.float32), np.eye(4), 'Stop at the door.', np.eye(4)) agent.step_s2(*arguments) From 63732e2f9c8b3d96dd52a7bef9b2b35d51e23788 Mon Sep 17 00:00:00 2001 From: Rohit Date: Mon, 5 Oct 2026 15:05:02 +0530 Subject: [PATCH 7/8] doc update --- docs/xpu_inference.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/docs/xpu_inference.md b/docs/xpu_inference.md index df93418c..4bdeee38 100644 --- a/docs/xpu_inference.md +++ b/docs/xpu_inference.md @@ -15,12 +15,11 @@ venv/bin/python -m pip check venv/bin/python -c "import torch; assert torch.xpu.is_available(); print(torch.__version__); print(torch.xpu.get_device_properties(0)); print(torch.xpu.mem_get_info(0))" ``` -Do not install XPU packages into a CUDA environment. Keep `diffusers==0.32.2`; -newer versions fail to load the checkpoint with a `size mismatch` error. +Do not install `xformers` or `flash-attn` in this environment. Keep `diffusers==0.32.2`. ## Auxiliary checkpoint -The DualVLN asynchronous RGB path requires this checkpoint: +Download the Depth Anything V2 checkpoint into `checkpoints/`: ```bash curl -fL --retry 2 -o checkpoints/depth_anything_v2_metric_hypersim_vits.pth \ @@ -30,11 +29,8 @@ sha256sum checkpoints/depth_anything_v2_metric_hypersim_vits.pth ## Configuration -For policy-based inference, set `device` and, optionally, `attn_implementation` in the -model configuration consumed by `ModelCfg` in `internnav/configs/model/base_encoders.py`. -The real-world HTTP server (`scripts/realworld/http_internvla_server.py`) always uses `auto`. - -Set `attn_implementation` (default `auto`): +Set `device` and, optionally, `attn_implementation` (default `auto`) in the model +configuration. The real-world HTTP server uses `auto`. | Device | `auto` | Other selections | | --- | --- | --- | @@ -68,4 +64,10 @@ Model weights alone need 15.62 GiB of free GPU memory; inference requires more. ```bash venv/bin/python -m pip install pytest INTERNNAV_FULL_XPU_TEST=1 venv/bin/python -m pytest tests/unit_test/test_xpu_inference.py -q -s +``` + +CUDA equivalence tests, in a CUDA environment: + +```bash +python -m pytest tests/unit_test/test_xpu_inference.py -q -rs -k "accelerator or cuda" ``` \ No newline at end of file From 61bb7484e669616226c890995b824c67488feed3 Mon Sep 17 00:00:00 2001 From: Rohit Date: Mon, 5 Oct 2026 15:23:13 +0530 Subject: [PATCH 8/8] doc update --- docs/xpu_inference.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/docs/xpu_inference.md b/docs/xpu_inference.md index 4bdeee38..4c9ccd9e 100644 --- a/docs/xpu_inference.md +++ b/docs/xpu_inference.md @@ -29,21 +29,24 @@ sha256sum checkpoints/depth_anything_v2_metric_hypersim_vits.pth ## Configuration -Set `device` and, optionally, `attn_implementation` (default `auto`) in the model -configuration. The real-world HTTP server uses `auto`. +Set `device` and, optionally, `attn_implementation` (default `auto`) in `model_settings` +of the evaluation config, e.g. `scripts/eval/configs/h1_internvla_n1_async_cfg.py`: + +```python +model_settings={ + ... + 'model_path': 'checkpoints/InternVLA-N1-DualVLN', + 'device': 'xpu:0', + 'attn_implementation': 'auto', +} +``` | Device | `auto` | Other selections | | --- | --- | --- | | CUDA | `flash_attention_2` | `sdpa`, `eager` | | XPU | `sdpa` | `eager`; `flash_attention_2` is not supported | -Policy configuration example: - -```python -device = 'xpu:0' -attn_implementation = 'auto' -model_path = 'checkpoints/InternVLA-N1-DualVLN' -``` +The real-world HTTP server uses `auto`. ## Launch