Open
Conversation
- Add FSDP layer wrapping fallback for InternVL HF models - Fix empty videos list causing IndexError in processor - Fix list of tensors not being stacked in multimodal_train_inputs - Add torch_dtype=bfloat16 for model loading - Add WandB environment variables passthrough to Ray job - Add InternVL image processing utilities - Add KIE training shell script with cleanup and checks Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add automatic checkpoint conversion from HuggingFace format (training) to SGLang format (inference) for InternVL models. Changes: - Add model_converter.py with HF->SGLang key mapping and QKV concatenation - Add --convert-to-sglang and --sglang-model-path arguments - Integrate conversion into train.py save() function - Add convert_internvl_checkpoint.py CLI tool - Add start_sglang_internvl.sh helper script - Add model_conversion_guide.md documentation Key features: - Handles vision_tower -> vision_model naming - Handles multi_modal_projector -> mlp1 naming - Concatenates separate q/k/v_proj weights into qkv - Automatically converts after each checkpoint save - Preserves both HF and SGLang formats Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add debugging tools and documentation for InternVL multimodal training. Changes: - Add multimodal logging in sglang_rollout.py - Add request dumping for debugging - Convert <IMG_CONTEXT> to <image> placeholder for SGLang - Add test_sglang_request.py for testing dumped requests - Add internvl_training_lifecycle.md documentation Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…g-fixes Feature/internvl kie training fixes
- Use non-HF InternVL model for both training and inference - Add img_context_token_id setting for non-HF InternVL models - Update load_processor to detect and handle non-HF InternVL models - Fix image placeholder handling (<image> vs <IMG_CONTEXT>) - Convert numpy arrays to lists in build_processor_kwargs - Add InternVL-specific data loading to preserve <image> placeholders - Remove HF-to-SGLang checkpoint conversion (no longer needed) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…g-fixes feat: support non-HF format InternVL training Final
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…g-fixes docs: add InternVL training documentation and summary
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.