Instructions to use pankajmathur/orca_mini_13b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use pankajmathur/orca_mini_13b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="pankajmathur/orca_mini_13b")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("pankajmathur/orca_mini_13b")
model = AutoModelForCausalLM.from_pretrained("pankajmathur/orca_mini_13b", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use pankajmathur/orca_mini_13b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "pankajmathur/orca_mini_13b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pankajmathur/orca_mini_13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/pankajmathur/orca_mini_13b

SGLang

How to use pankajmathur/orca_mini_13b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "pankajmathur/orca_mini_13b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pankajmathur/orca_mini_13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "pankajmathur/orca_mini_13b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pankajmathur/orca_mini_13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use pankajmathur/orca_mini_13b with Docker Model Runner:
```
docker model run hf.co/pankajmathur/orca_mini_13b
```

Excellent model!

by AtlasUnified - opened Jun 24, 2023

Discussion

AtlasUnified

Jun 24, 2023

No comments other than this is a pretty versatile model.

AtlasUnified changed discussion status to closed Jun 24, 2023

pankajmathur

Owner Jun 24, 2023

Thank You… if possible please could you share some examples you like. So it can benefits others too.

AtlasUnified

Jun 24, 2023

•

edited Jun 24, 2023

Certainly. I fed your model this prompt:

system = "You are a language model capable of creating reasoning tasks. Each task should consist of a prompt, a step-by-step reasoning, and a final solution. Your strength is in brevity and answering directly, avoiding jargon such as 'sure, I can help, what would you like to know'."
instruction = "Generate 10 complex reasoning tasks on the topic of '{topic}' and sub-topic '{subtopic}'. Each task should include a prompt (a question or problem), a step-by-step reasoning (a logical explanation of how to arrive at the solution), and a solution (the final answer to the problem). The output format should be a list like this: ['prompt', 'step_by_step_reasoning', 'solution']. Do not deviate from the JSON format of {json.dumps({'id': 'string', 'task': [{'prompt': 'string', 'reasoning': 'string', 'solution': 'string'}]})}. Keep the tasks succinct, comprehensive, and avoid any unnecessary language. Keep all responses in JSON format and all on one line. Have these reasoning tasks on the PhD level and have high complexity factors."

And got this response, this was the third of 7 it generated:

{topic}: determine the length of the shortest path from a given node to a specific node in a graph.
{prompt}: Write a program to find the length of the shortest path from a given node to a specific node in a graph.
{reasoning}: Define a function 'shortest_path' that takes in a graph object and a starting node object. Use the A* algorithm to find the shortest path from the starting node to the given node. The shortest path is the length of the path returned by the 'shortest_path' function.
{solution}: Define a graph object and a starting node object. Use the ' shortest_path' function to find the shortest path from the starting node to a specific node. The shortest path is the length of the path returned by the 'shortest_path' function.

I'd have to rework the code to output to a text file for a JSONL but I'm certain I could use this model for synthetic data.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment