Pārlūkot izejas kodu

Add jsonl support (#212)

Handled by default with same "json" type, the lib auto detects the precise type.
Angainor Development 3 gadi atpakaļ
vecāks
revīzija
c59d5672b0
1 mainītis faili ar 1 papildinājumiem un 1 dzēšanām
  1. 1 1
      finetune.py

+ 1 - 1
finetune.py

@@ -163,7 +163,7 @@ def train(
     )
     model = get_peft_model(model, config)
 
-    if data_path.endswith(".json"):  # todo: support jsonl
+    if data_path.endswith(".json") or data_path.endswith(".jsonl"):
         data = load_dataset("json", data_files=data_path)
     else:
         data = load_dataset(data_path)