Skip to content

Navigation Menu

Sign in
Sign up

Generate YAML from Go struct - #60

Open
saswatamcode wants to merge 5 commits into
bwplotka:main from
saswatamcode:struct-yaml
Open

Generate YAML from Go struct #60
saswatamcode wants to merge 5 commits into
bwplotka:main from
saswatamcode:struct-yaml

Conversation

@saswatamcode

@saswatamcode saswatamcode commented Jul 12, 2021 •
edited
Loading

Copy link
Copy Markdown
Collaborator

This PR allows mdox to generate YAML from Go structs, using the following semantics(for now).
Thus, structs like below,

type Config struct {
	Version int `yaml:"version"`
	Validator []ValidatorConfig `yaml:"validators"`
	Ignore []IgnoreConfig `yaml:"ignore"`
}
type ValidatorConfig struct {
	Type string `yaml:"type"`
	Regex string `yaml:"regex"`
	Token string `yaml:"token"`
	r *regexp.Regexp
}
type IgnoreConfig struct {
	Url string `yaml:"url"`
	ID int `yaml:"id"`
	Token string `yaml:"token"`
}

Result in markdown like below,

```yaml mdox-gen-go-struct="main.go:Config"
version: 0
validators:
 - type: ""
 regex: ""
 token: ""
ignore:
 - url: ""
 id: 0
 token: ""

```yaml mdox-gen-go-struct="main.go:ValidatorConfig"
type: ""
regex: ""
token: ""

Dependencies added: jennifer, structtag

Few TODOs:

// TODO(saswatamcode): Add tests.
// TODO(saswatamcode): Check jennifer code for some safety.
// TODO(saswatamcode): Add mechanism for caching output from generated code.
// TODO(saswatamcode): Better errors.

Resolves #23.

@bwplotka bwplotka left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's some next level magic

AI programmer!

LGTM, just let's add tests 👍🏽

saswatamcode reacted with thumbs up emoji
Comment thread pkg/yamlgen/yamlgen.go
// TODO(saswatamcode): Currently takes file names, need to make it module based(something such as https://golang.org/pkg/cmd/go/internal/list/).

// GenGoCode generates Go code for yaml gen from structs in src file.
func GenGoCode(src []byte) (string, error) {

@bwplotka bwplotka Aug 3, 2021

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good for start, but I wonder if we can scope to just one desired structure, to limit imports. Maybe it's later optimization - so fine for now

saswatamcode reacted with thumbs up emoji

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add todo for that?

saswatamcode reacted with thumbs up emoji
Comment thread pkg/yamlgen/yamlgen.go
init = append(init, jen.Id("configs").Op(":=").Map(jen.String()).Interface().Values())

// Loop through declarations in file.
for _, decl := range f.Decls {

@bwplotka bwplotka Aug 3, 2021

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I would expect some recursion here (:

saswatamcode reacted with thumbs up emoji
Comment thread pkg/yamlgen/yamlgen.go Outdated
Comment on lines +112 to +114
generatedCode.Func().Id("main").Params().Block(
init...,
)

@bwplotka bwplotka Aug 3, 2021

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
generatedCode.Func().Id("main").Params().Block(
init...,
)
generatedCode.Func().Id("main").Params().Block(init...)

saswatamcode reacted with thumbs up emoji

@saswatamcode saswatamcode Aug 23, 2021

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

Comment thread pkg/yamlgen/yamlgen.go Outdated
cmd = exec.CommandContext(ctx, "go", "mod", "tidy")
cmd.Dir = tmpDir
if err := cmd.Run(); err != nil {
return nil, errors.Wrapf(err, "run %v", cmd)

@bwplotka bwplotka Aug 3, 2021

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return nil, errors.Wrapf(err, "run %v", cmd)
return nil, errors.Wrapf(err, "mod %v", cmd)

saswatamcode reacted with thumbs up emoji

@saswatamcode saswatamcode Aug 23, 2021

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

Comment thread pkg/yamlgen/yamlgen.go Outdated
cmd := exec.CommandContext(ctx, "go", "mod", "init", "structgen")
cmd.Dir = tmpDir
if err := cmd.Run(); err != nil {
return nil, errors.Wrapf(err, "run %v", cmd)

@bwplotka bwplotka Aug 3, 2021

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return nil, errors.Wrapf(err, "run %v", cmd)
return nil, errors.Wrapf(err, "mod init %v", cmd)

saswatamcode reacted with thumbs up emoji

@saswatamcode saswatamcode Aug 23, 2021

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

Comment thread pkg/yamlgen/yamlgen.go Outdated
return checkForOmitEmptyTagOptionRec(reflect.ValueOf(obj))
}

func checkForOmitEmptyTagOptionRec(v reflect.Value) error {

@bwplotka bwplotka Aug 3, 2021

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to check for emit empty if we fill all the structs? I think in this case we need to also set non default values for other things that slices... so maybe for next iteration

saswatamcode reacted with thumbs up emoji

@bwplotka bwplotka left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, let's add recursion and understand what map will work, what not. 👍🏽

saswatamcode reacted with thumbs up emoji
Comment thread pkg/yamlgen/yamlgen.go
// TODO(saswatamcode): Currently takes file names, need to make it module based(something such as https://golang.org/pkg/cmd/go/internal/list/).

// GenGoCode generates Go code for yaml gen from structs in src file.
func GenGoCode(src []byte) (string, error) {

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add todo for that?

saswatamcode reacted with thumbs up emoji
})

t.Run("struct with unexported field", func(t *testing.T) {
source := []byte("package main\n\nimport \"regexp\"\n\ntype ValidatorConfig struct {\n\tType string `yaml:\"type\"`\n\tRegex string `yaml:\"regex\"`\n\tToken string `yaml:\"token\"`\n\n\tr *regexp.Regexp\n}\n")

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would use ` and add + backtick + , something like that. Still easier than long string manually crafted.

Another idea - put all in files - that would do as well.

saswatamcode reacted with thumbs up emoji

Dentrax commented May 10, 2022

Copy link
Copy Markdown

Any update on this? @bwplotka

Copy link
Copy Markdown
Owner

I think @saswatamcode has some comments to be addressed, but quite close.

What do you need this for @Dentrax? Curious about your use case and if this PR fixes it (:

Dentrax commented May 11, 2022

Copy link
Copy Markdown

I've created a related issue thanos-io/thanos#4751 (comment) and closed due to inactivity. We want to add support for both comments and default values while generating YAML.

Dentrax commented Jan 26, 2023

Copy link
Copy Markdown

Kind ping, still looking for this feature! @saswatamcode @bwplotka

saswatamcode reacted with eyes emoji

bwplotka commented Jun 7, 2023

Copy link
Copy Markdown
Owner

cc @saswatamcode

saswatamcode reacted with eyes emoji 

Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>

Copy link
Copy Markdown
Collaborator Author

I'm trying to look into this, this week and bring it to a nice usable state, rebasing and correcting some of the old code. Plan to clean it up in the next few commits! 🙂

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bwplotka bwplotka bwplotka left review comments

At least 1 approving review is required to merge this pull request.

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add subcommand that will generate example YAML that will parsable from Go structs

AltStyle によって変換されたページ (->オリジナル) /