-
Notifications
You must be signed in to change notification settings - Fork 338
-
I noticed that using popular AI tools to work with PKL leads to poor results:
llm-test $ cat docker-compose.yml | llm -x -m claude-3.7-sonnet -s "convert this YAML to PKL https://pkl-lang.org/" | tee test.pkl
module docker {
version = "3"
services {
alpine-curl {
image = "alpine/curl"
command = ["-fsSL", "http://manager:8080"]
networks = ["duser_default"]
}
}
networks {
duser_default {
external = true
}
}
}
llm-test $ pkl eval test.pkl
–– Pkl Error ––
Invalid token at position. Expected a class, typealias, method, or property.
1 | module docker {
^
at test (file:///Users/hendry/tmp/llm-test/test.pkl, line 1)
llm-test $ cat docker-compose.yml | llm -x -m gpt-4o-mini -s "convert this YAML to PKL https://pkl-lang.org/" | tee test.pkl
version: "1.0"
services:
- name: alpine-curl
image: "alpine/curl"
command: ["-fsSL", "http://manager:8080"]
networks:
- duser_default
networks:
- name: duser_default
external: true
llm-test $ pkl eval test.pkl
–– Pkl Error ––
Unexpected token `-`. Expected a type.
4 | - name: alpine-curl
^
at test (file:///Users/hendry/tmp/llm-test/test.pkl, line 4)
Do you have an MCP in mind or something to help models generate PKL code?
Beta Was this translation helpful? Give feedback.
All reactions
Replies: 4 comments 5 replies
-
To be honest, I haven't thought about this much, and I'm also a novice when it comes to LLMs.
Curious what thoughts other people have here.
Beta Was this translation helpful? Give feedback.
All reactions
-
I am not sure if there is a direct support to LLM inside PKL.
However, I made an AI agent framework powered by PKL.
Beta Was this translation helpful? Give feedback.
All reactions
-
LLMs doesn't know "anything" about pkl because "the world" doesn't know anything about pkl 😉.
Pkl is, compared to other languages, just not present. From where should LLM learn things?
Further, even Kotlin itself is "so new", that JetBrains decided actively to push training data for LLMs on the net (check their huggingface account).
As long as pkl stays like it is now, it will never©️ going to be useful in LLMs, I think.
But of course, if it's growth and more people adopt it, it is easier for LLMs to understand it.
Just because GitHub itself (their repositories with a permissive license) are useful training data.
So what can pkl do about it?
As mentioned, going a similar way as JetBrains and actively providing training data for pkl. Not saying that "the big foundation models" will definitely using it for training or fine tuning. But the changes is higher if there is something, right? 😁
You mention MCP. Yes, probably a MCP server would also help. But which tools or resources should it provide? Resources might be obvious, the super nice pkl documentation. But since it contains "only" code snippets, I would assume that this is too less for the LLM to map it to real world scenarios to provide useful code.
So it might help a bit, especially in making the documentation more "live" (cause you can go into an Q&A conversation with the LLM) but I think it won't help in generating pkl code.
Beta Was this translation helpful? Give feedback.
All reactions
-
Use pkl to generate the appropriate JSONSchema's, send to LLM, profit. It seems pretty easy using the existing libraries.
Beta Was this translation helpful? Give feedback.
All reactions
-
Isn't writing PKL the hard part we want to solve?
Beta Was this translation helpful? Give feedback.
All reactions
-
Pkl is probably just another tool for another problem you want to solve. So maybe it's not that we need AI support for writing pkl, but llm support for your underlying problem...? 👀
Beta Was this translation helpful? Give feedback.
All reactions
-
Umm, this is a bit meta. The underlying problem is reducing mistakes. The bottleneck is that it's hard to write pkl and validation processes / workflows to ensure compliance.
Beta Was this translation helpful? Give feedback.
All reactions
-
If it's hard to write pkl, maybe it's not the right tool for the right job?
Beta Was this translation helpful? Give feedback.
All reactions
-
It's hard to write in any language without MCP/LLM support nowadays. 😆
Beta Was this translation helpful? Give feedback.
All reactions
-
👎 1 -
👀 1