支持yarn扩展rollout模型上下文长度吗

[tingjun-cs]

请问支持扩展 推理/rollout模型的上下文长度吗？在许多场景，模型预训练的上下文长度（比如Qwen3 32k）是不够用的，需要拓展，类似下面的方式

[chenyushuo]

目前有两种方法支持使用rope-scaling对上下文长途进行扩展：

1. 你可以直接修改模型的config.json文件，在里面修改rope-scaling等相关参数，例如这样：

{
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "hidden_act": "silu",
  "hidden_size": 1536,
  "initializer_range": 0.02,
  "intermediate_size": 8960,
  "max_position_embeddings": 32768,
  "max_window_layers": 21,
  "model_type": "qwen2",
  "num_attention_heads": 12,
  "num_hidden_layers": 28,
  "num_key_value_heads": 2,
  "rms_norm_eps": 1e-06,
  "rope_scaling": {"rope_type":"yarn","factor":2.0,"original_max_position_embeddings": 16384},
  "rope_theta": 10000.0,
  "sliding_window": 32768,
  "tie_word_embeddings": true,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.43.1",
  "use_cache": true,
  "use_sliding_window": false,
  "vocab_size": 151936
}

3. 我们在 Add rope_scaling and rope_theta to config #390 中增加了rope相关的config，你也可以修改训练配置文件中的model.rope_scaling参数，例如：

# model configs
model:
  model_path: Qwen/Qwen2.5-1.5B-Instruct
  max_prompt_tokens: null
  min_response_tokens: 1
  max_response_tokens: 1024
  max_model_len: 4096
  rope_scaling:
    rope_type: yarn
    factor: 2.0
    original_max_position_embeddings: 16384
    # mrope_section: [24,20,20]  # for vl
    # mrope_interleaved: true  # for vl
  rope_theta: 10000

[tingjun-cs]

@chenyushuo 看到了呢，十分感谢