Instructions to use Kukedlc/NeuralExperiment-7b-MagicCoder-v5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Kukedlc/NeuralExperiment-7b-MagicCoder-v5 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Kukedlc/NeuralExperiment-7b-MagicCoder-v5")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Kukedlc/NeuralExperiment-7b-MagicCoder-v5")
model = AutoModelForCausalLM.from_pretrained("Kukedlc/NeuralExperiment-7b-MagicCoder-v5")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Kukedlc/NeuralExperiment-7b-MagicCoder-v5 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Kukedlc/NeuralExperiment-7b-MagicCoder-v5"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Kukedlc/NeuralExperiment-7b-MagicCoder-v5",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Kukedlc/NeuralExperiment-7b-MagicCoder-v5

SGLang

How to use Kukedlc/NeuralExperiment-7b-MagicCoder-v5 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Kukedlc/NeuralExperiment-7b-MagicCoder-v5" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Kukedlc/NeuralExperiment-7b-MagicCoder-v5",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Kukedlc/NeuralExperiment-7b-MagicCoder-v5" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Kukedlc/NeuralExperiment-7b-MagicCoder-v5",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Kukedlc/NeuralExperiment-7b-MagicCoder-v5 with Docker Model Runner:
```
docker model run hf.co/Kukedlc/NeuralExperiment-7b-MagicCoder-v5
```

How many hours of training?

by damerajee - opened Mar 5, 2024

Discussion

damerajee

Mar 5, 2024

Hey there i noticed you train the entire model on free kaggle GPU if you don't mind ,I wanted to know which you did you use and how many hours did you train it for ?,

Kukedlc

Owner Mar 5, 2024

Hi damerajee, I'm trained on the last 3 layers (29, 30, 31), not the entire model. It took about 20 hours with 2 GPUs. The goals of these projects are twofold. First, to be able to train 7b models for free (Kaggle) and to test the backward training notion, which is used in RNNs but not in LLMs.

damerajee

Mar 6, 2024

pretty cool thanks for this info

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment