| --- |
| title: CoDA Fine-tuning |
| emoji: 🚀 |
| colorFrom: blue |
| colorTo: purple |
| sdk: gradio |
| sdk_version: 4.44.0 |
| app_file: app.py |
| pinned: false |
| license: apache-2.0 |
| hf_oauth: true |
| hf_oauth_scopes: |
| - read-repos |
| - write-repos |
| --- |
| |
| # CoDA Model Fine-tuning Space |
|
|
| This Space allows you to fine-tune the **Salesforce/CoDA-v0-Instruct** text generation diffusion model on the **baseten-admin/gpt-oss120b-generated-perfectblend** dataset. |
|
|
| ## Features |
|
|
| - 🎯 **Full Fine-tuning**: Complete parameter fine-tuning (not LoRA) |
| - 💬 **ChatML Format**: Processes conversation data with question-answer pairs |
| - 🔄 **Auto Upload**: Automatically uploads trained model to your Hugging Face account |
| - 📊 **Progress Tracking**: Real-time training progress updates |
| - 🔐 **OAuth Integration**: Secure authentication via Hugging Face login |
|
|
| ## How to Use |
|
|
| 1. **Login**: Click the "Sign in with Hugging Face" button |
| 2. **Configure**: Adjust training parameters (epochs, batch size, learning rate) |
| 3. **Train**: Click "Start Training" (requires GPU - upgrade Space to GPU tier) |
| 4. **Resume**: If training is interrupted, check "Resume from last checkpoint" and restart |
| 5. **Upload**: After training completes, click "Upload to Hugging Face Hub" |
|
|
| ### Persistence |
|
|
| This Space supports checkpoint persistence: |
| - Training checkpoints are saved every 500 steps |
| - If interrupted, you can resume from the last checkpoint |
| - For Docker deployment: Mount `/data` volume for full persistence |
| - On Spaces: Checkpoints persist within the same session and across rebuilds if using persistent storage tier |
|
|
| ## Requirements |
|
|
| - **Hardware**: GPU (T4, A10G, or better) strongly recommended |
| - **Account**: Hugging Face account with write permissions |
| - **Time**: Training takes several hours depending on configuration |
|
|
| ## About the Model |
|
|
| **CoDA (Code Diffusion with Autoregressive)** is a 1.7B parameter bidirectional diffusion model developed by Salesforce AI Research. Unlike traditional autoregressive models, CoDA uses discrete denoising for text generation. The Instruct version is pre-tuned for instruction following, making it ideal for fine-tuning on conversational data. |
|
|
| ### Model Configuration |
|
|
| ```json |
| { |
| "architectures": ["CoDALanguageModel"], |
| "hidden_size": 2048, |
| "num_hidden_layers": 28, |
| "num_attention_heads": 16, |
| "vocab_size": 151936, |
| "max_position_embeddings": 40960 |
| } |
| ``` |
|
|
| ## Dataset |
|
|
| The training uses the **baseten-admin/gpt-oss120b-generated-perfectblend** dataset: |
| - **Format**: Conversational data in ChatML format |
| - **Column**: `conversations` (list of role-content pairs) |
| - **Split**: Uses `train` split with 90/10 train/eval split |
|
|
| ## Training Details |
|
|
| - **Optimizer**: AdamW |
| - **Precision**: FP16 (on GPU) |
| - **Gradient Accumulation**: 4 steps |
| - **Gradient Checkpointing**: Enabled for memory efficiency |
| - **Max Sequence Length**: 2048 tokens |
|
|
| ## Citation |
|
|
| If you use this Space or the CoDA model, please cite: |
|
|
| ```bibtex |
| @article{coda2023, |
| title={CoDA: Bidirectional Code Diffusion}, |
| author={Salesforce AI Research}, |
| journal={arXiv preprint}, |
| year={2023} |
| } |
| ``` |
|
|
| ## License |
|
|
| Apache 2.0 |
|
|
