A flexible, language and frameworks agnostic tool that allows you to generate projects structure from templates based
on yaml
configuration (generate directories, files and execute commands) or use as library to build custom generator.
go install github.com/kozmod/progen@latest
make build
Use as lib
ⓘ
module github.com/some/custom_gen
go 1.22
require github.com/kozmod/progen v0.1.8
Name | Type | Default | Description |
---|---|---|---|
-f ⓘ ✱ |
string | progen.yml |
specify configuration file path |
-v ✱ |
bool | false |
verbose output |
-dr ⓘ ✱ |
bool | false |
dry run mode (to verbose output should be combine with -v ) |
-awd ⓘ |
string | . |
application working directory |
-printconf ⓘ |
bool | false |
output processed config |
-errtrace ⓘ ✱ |
bool | false |
output errors stack trace |
-pf ⓘ |
bool | true |
preprocessing files : load and process all files (all files actions ⓘ) as text/template before creating |
-tvar ⓘ ✱ |
[]string | [ ] |
text/template variables (override config variables tree) |
-missingkey ✱ |
[]string | error |
set missingkey text/template.Option execution option |
-skip ⓘ |
[]string | [ ] |
skip any action tag (regular expression) |
-gp ⓘ |
[]string | [ ] |
set of the action's groups to execution |
-version |
bool | false |
print version |
-help ✱ |
bool | false |
show flags |
✱ flags accessible in the lib
.
Key | Type | Optional | Description |
---|---|---|---|
settings | ✅ | progen settings section |
|
settings.httpⓘ | ✅ | http client configuration | |
settings.http.debug | bool | ✅ | http client DEBUG mode |
settings.http.base_url | string | ✅ | http client base URL |
settings.http.headers | map[string]string | ✅ | http client base request Headers |
settings.http.query_params | map[string]string | ✅ | http client base request Query Parameters |
settings.groupsⓘ | ✅ | groups of actions | |
settings.groups.name | string | ✅ | group's name |
settings.groups.actions | []string | ✅ | actions names |
settings.groups.manual | bool | ✅ | determines that the group starts automatically (default false ) |
dirs<unique_suffix> ⓘ |
[]string | ✅ | list of directories to create |
rm<unique_suffix> ⓘ |
[]string | ✅ | list for remove (files, dirs, all file in a dir) |
files<unique_suffix> ⓘ |
✅ | list file's path and data |
|
files.path | string | ❌ | save file path |
files.local | string | ❕ |
local file path to copy |
files.data | string | ❕ |
save file data |
files.get | ❕ |
struct describe GET request for getting file's data |
|
files.get.url | string | ❌ | request URL |
files.get.headers | map[string]string | ✅ | request Headers |
files.get.query_params | map[string]string | ✅ | request Query Parameters |
cmd<unique_suffix> ⓘ |
✅ | configuration command list | |
cmd.exec | string | ❌ | command to execution |
cmd.args | []slice | ✅ | list of command's arguments |
cmd.dir | string | ✅ | execution commands (cmd.exec ) directory |
fsⓘ | []string | ✅ | execute text/template.Option on the list of directories |
❕
only one must be specified in parent section
The cli executes commands and generate files and directories based on configuration file
## progen.yml
# list directories to creation
dirs:
- x/y
# list files to creation
files:
- path: x/some_file.txt
data: |
some data
# list commands to execution
cmd:
- touch second_file.txt
- tree
% progen -v
2023-02-05 14:11:47 INFO application working directory: /Users/user_1/GoProjects/service
2023-02-05 14:11:47 INFO configuration file: progen.yml
2023-02-05 14:11:47 INFO file process: x/some_file.txt
2023-02-05 14:11:47 INFO dir created: x/y
2023-02-05 14:11:47 INFO file saved: x/some_file.txt
2023-02-05 14:11:47 INFO execute [dir: .]: touch second_file.txt
2023-02-05 14:11:47 INFO execute [dir: .]: tree
out:
.
├── second_file.txt
└── x
├── some_file.txt
└── y
2 directories, 2 files
All actions execute in declared order. Base actions (dir
, files
,cmd
) could be configured
with <unique_suffix>
to separate action execution.
## progen.yml
dirs1:
- api/some_project/v1
cmd1:
- chmod -R 777 api
dirs2:
- api/some_project_2/v1
cmd2:
- chmod -R 777 api
% progen -v
2023-01-22 13:38:52 INFO application working direcotry: /Users/user_1/GoProjects/service
2023-01-22 13:38:52 INFO dir created: api/some_project/v1
2023-01-22 13:38:52 INFO execute [dir: .]: chmod -R 777 api
2023-01-22 13:38:52 INFO dir created: api/some_project_2/v1
2023-01-22 13:38:52 INFO execute [dir: .]: chmod -R 777 api
Configuration preprocessing uses text/template of golang's stdlib
.
Using templates could be useful to avoiding duplication in configuration file.
All text/template
variables must be declared as comments and can be used only to configure data of configuration
file (all ones skipping for file.data
section).
Configuration's yaml
tag tree also use as text/template
variables dictionary and can be use for avoiding duplication
in configuration file and files contents (files
section).
## progen.yml
## `text/template` variables declaration 👇
# {{$project_name := "SOME_PROJECT"}}
## unmapped section (not `dirs`, `files`, `cmd`, `http`) can be use as template variables
vars:
file_path: some/file/path
dirs:
- api/{{$project_name}}/v1 # used from `text/template` variables
- internal/{{.vars.file_path}} # used from `vars` section
- pkg/{{printf `%s-%s` $project_name `data`}}
files:
- path: internal/{{$project_name}}.txt
data: |
Project name:{{$project_name}}
- path: pkg/{{printf `%s-%s` $project_name `data`}}/some_file.txt
data: |
{{$project_name}}
cmd:
- "cat internal/{{$project_name}}.txt"
- exec: ls
dir: .
args: [ -l ]
- exec: tree
% progen -v
2023-01-22 13:03:58 INFO current working direcotry: /Users/user_1/GoProjects/service
2023-02-05 14:47:25 INFO application working directory: /Users/user_1/GoProjects/service
2023-02-05 14:47:25 INFO configuration file: progen.yaml
2023-02-05 14:47:25 INFO file process: internal/SOME_PROJECT.txt
2023-02-05 14:47:25 INFO file process: pkg/SOME_PROJECT-data/some_file.txt
2023-02-05 14:47:25 INFO dir created: api/SOME_PROJECT/v1
2023-02-05 14:47:25 INFO dir created: internal/some/file/path
2023-02-05 14:47:25 INFO dir created: pkg/SOME_PROJECT-data
2023-02-05 14:47:25 INFO file saved: internal/SOME_PROJECT.txt
2023-02-05 14:47:25 INFO file saved: pkg/SOME_PROJECT-data/some_file.txt
2023-02-05 14:47:25 INFO execute [dir: .]: cat internal/SOME_PROJECT.txt
out:
Project name:SOME_PROJECT
2023-02-05 14:47:25 INFO execute [dir: .]: ls -l
out:
total 0
drwxr-xr-x 3 19798572 646495703 96 Feb 5 14:47 api
drwxr-xr-x 4 19798572 646495703 128 Feb 5 14:47 internal
drwxr-xr-x 3 19798572 646495703 96 Feb 5 14:47 pkg
2023-02-05 14:47:25 INFO execute [dir: .]: tree
out:
.
├── api
│ └── SOME_PROJECT
│ └── v1
├── internal
│ ├── SOME_PROJECT.txt
│ └── some
│ └── file
│ └── path
├── pkg
│ └── SOME_PROJECT-data
│ └── some_file.txt
└── progen.yml
9 directories, 2 files
Function | args | Description |
---|---|---|
random |
||
random.Alpha |
length int |
Generates a random alphabetical (A-Z, a-z) string of a desired length. |
random.Num |
length int |
Generates a random numeric (0-9) string of a desired length. |
random.AlphaNum |
length int |
Generates a random alphanumeric (0-9, A-Z, a-z) string of a desired length. |
random.ASCII |
length int |
Generates a random string of a desired length, containing the set of printable characters from the 7-bit ASCII set. This includes space (’ ‘), but no other whitespace character. |
slice |
||
slice.New |
N any elements |
Create new slice from any numbers of elements ( { $element := slice.New "a" 1 "b" }} ) |
slice.Append |
slice, N any elements |
Add element to exists slice ( {{ $element := slice.Append $element "b"}} ) |
strings |
||
strings.Replace |
s, old, new string, n int | Replace returns a copy of the string s with old replaced by new (work the same as strings.Replace from stdlib ). |
Custom template's functions added as custom arguments to the template function map.
By default progen
try to find progen.yml
file for execution. -f
flag specify custom configuration file location:
progen -f custom_conf.yaml
Instead of specifying a config file, you can pass a single configuration file in the pipe the file in via STDIN
.
To pipe a progen.yml
from STDIN
:
progen - < progen.yml
or
cat progen.yml | progen -
If you use STDIN
the system ignores any -f
option.
Example (get progen.yml
from gitlab repository with replacing text/template
variables using -tvar
flag):
curl -H PRIVATE-TOKEN:token https://gitlab.some.com/api/v4/projects/13/repository/files/shared%2Fteplates%2Fsimple%2Fprogen.yml/raw\?ref\=feature/templates | progen -v -dr -tvar=.vars.GOPROXY=some_proxy -
To print a stack trace of the error which occurred during execution of the cli
,
use -errtrace
flag:
% progen -f ../not_exists_config.yml
2023-03-04 15:05:54 FATAL read config: config file:
github.com/kozmod/progen/internal/config.(*Reader).Read
/Users/some_user/projects/progen/internal/config/reader.go:39
- open ../not_exists_config.yml: no such file or directory
The -dr
flag uses to execute configuration in dry run mod. All action
will be executed without applying.
## progen.yml
# {{$project_name := "SOME_PROJECT"}}
dirs:
- api/{{ $project_name }}/v1 # apply template variables, but not create directories on 'dry run'
cmd:
- tree # not execute on 'dry run' mode
files:
- path: api/v1/some_file.txt # apply template variables and only printing the file's data
data: |
some file data data fot project: {{ $project_name }}
% progen -v -dr
2023-03-07 07:57:52 INFO application working directory: /Users/user_1/GoProjects/service
2023-03-07 07:57:52 INFO configuration file: progen.yml
2023-03-07 07:57:52 INFO file process: api/v1/some_file.txt
2023-03-07 07:57:52 INFO dir created: api/SOME_PROJECT/v1
2023-03-07 07:57:52 INFO execute [dir: .]: tree
2023-03-07 07:57:52 INFO save file: create dir [api/v1] to store file [%!s(func() string=0x136ecc0)]
2023-03-07 07:57:52 INFO file saved [path: api/v1/some_file.txt]:
some file data data fot project: SOME_PROJECT
2023-03-07 07:57:52 INFO execution time: 3.69506ms
The -awd
flag uses for setting application working directory.
All paths
declared in the config file are calculated considering the root directory.
To print the configuration file after processing as text/template,
use -printconf
flag:
## progen.yml
vars:
some_data: VARS_SOME_DATA
# {{- $var_1 := random.AlphaNum 15}}
# {{- $var_2 := "echo some_%s"}}
cmd:
- echo {{ $var_1 }}
- "{{ printf $var_2 `value` }}"
- echo {{ .vars.some_data }}
% progen -printconf
2023-03-04 14:57:43 INFO preprocessed config:
vars:
some_data: VARS_SOME_DATA
#
#
cmd:
- echo AHNsgyzVxRqeqLt
- "echo some_value"
- echo VARS_SOME_DATA
By default, all files loading to the memory and process as text/template before
saving to a file system.
To change this behavior, set -pf=false
.
% progen -v -dr -f progen.yml
2023-02-05 14:15:54 INFO application working directory: /Users/user_1/GoProjects/service
2023-02-05 14:15:54 INFO configuration file: progen.yml
2023-02-05 14:15:54 INFO file process: api/v1/some_file.txt
2023-02-05 14:15:54 INFO dir created: api/SOME_PROJECT/v1
2023-02-05 14:15:54 INFO execute cmd: chmod -R 777 api/v1
2023-02-05 14:15:54 INFO save file: create dir [api/v1] to store file [some_file.txt]
2023-02-05 14:15:54 INFO file saved [path: api/v1/some_file.txt]:
some file data data fot project: SOME_PROJECT
Any part of template variable tree can be overrides using -tvar
flag
## progen.yml
## `text/template` variables declaration 👇
# {{$project_name := "SOME_PROJECT"}}
## unmapped section (not `dirs`, `files`, `cmd`, `http`) can be use as template variables
vars:
file_path: some/file/path
file_path_2: some/file/path_2
dirs:
- api/{{$project_name}}/v1 # used from `text/template` variables
- internal/{{.vars.file_path}} # used from `vars` section
- internal/{{.vars.file_path_2}} # used overridden `vars` which set through args (-tvar=.vars.file_path 2=override path)
% progen -v -dr -tvar=.vars.file_path_2=overrided_path
2023-02-05 14:51:38 INFO application working directory: /Users/user_1/GoProjects/service
2023-02-05 14:51:38 INFO configuration file: progen.yml
2023-02-05 14:51:38 INFO dir created: api/SOME_PROJECT/v1
2023-02-05 14:51:38 INFO dir created: internal/some/file/path
2023-02-05 14:51:38 INFO dir created: internal/overrided_path
Set -skip
flag to skip any action
(only root actions: cmd
, files
, dirs
). Value of the flag is a regular
expression.
## progen.yml
dirs:
- api/v1
cmd:
- chmod -R 777 api/v1
dirs1:
- api/v2
cmd1:
- chmod -R 777 api/v2
dirs2:
- api/v3
cmd2:
- chmod -R 777 api/v3
% progen -v -dr -f progen.yml -skip=^dirs$ -skip=cmd.+
2023-02-05 14:18:11 INFO application working directory: /Users/user_1/GoProjects/service
2023-02-05 14:18:11 INFO configuration file: progen.yml
2023-02-05 14:18:11 INFO action will be skipped: [cmd1]
2023-02-05 14:18:11 INFO action will be skipped: [cmd2]
2023-02-05 14:18:11 INFO action will be skipped: [dirs]
2023-02-05 14:18:11 INFO execute cmd: chmod -R 777 api/v1
2023-02-05 14:18:11 INFO dir created: api/v2
2023-02-05 14:18:11 INFO dir created: api/v3
HTTP client configuration
## progen.yml
settings:
http:
debug: false
base_url: https://gitlab.repo_2.com/api/v4/projects/5/repository/files/
headers:
PRIVATE-TOKEN: glpat-SOME_TOKEN
query_params:
PARAM_1: Val_1
All actions execute in declaration order in the config file and can be union to groups.
All actions in manual
groups will be skipped during execution process.
settings:
groups:
- name: group1
actions: [ cmd, cmd_2 ]
manual: true
- name: group2
actions: [ cmd_2 ]
manual: true
cmd:
- echo CMD_1
cmd_2:
- echo CMD_2
cmd_3:
- echo CMD_3
cmd_4:
- echo CMD_4
% progen -v
2024-02-05 23:08:21 INFO application working directory: /Users/user_1/GoProjects/service
2024-02-05 23:08:21 INFO configuration file: progen.yml
2024-02-05 23:08:21 INFO manual actions will be skipped: [cmd, cmd_2]
2024-02-05 23:08:21 INFO execute [dir: .]: echo CMD_3
out:
CMD_3
2024-02-05 23:08:21 INFO execute [dir: .]: echo CMD_4
out:
CMD_4
2024-02-05 23:08:21 INFO execution time: 7.916615ms
Actions in manual
groups execute using gp
flag (all action execute only once independent on declaration's quantity
in different groups).
% progen -v -gp=group1 -gp=group2
2024-02-05 23:19:50 INFO application working directory: /Users/user_1/GoProjects/service
2024-02-05 23:19:50 INFO configuration file: progen.yml
2024-02-05 23:19:50 INFO groups will be execute: [group1, group2]
2024-02-05 23:19:50 INFO execute [dir: .]: echo CMD_1
out:
CMD_1
2024-02-05 23:19:50 INFO execute [dir: .]: echo CMD_2
out:
CMD_2
2024-02-05 23:19:50 INFO execution time: 7.192257ms
File's content can be declared in configuration file (files.data
tag) or
can be received from local (files.local
) or remote (files.get
) storage.
Any file's content uses as text/template
and configuration's yaml
tag tree applies as template variables.
## progen.yml
# settings of the cli
settings:
# common http client configuration
http:
debug: false
base_url: https://gitlab.repo_2.com/api/v4/projects/5/repository/files/
headers:
PRIVATE-TOKEN: glpat-SOME_TOKEN
# {{$project_name := "SOME_PROJECT"}}
# {{$gitlab_suffix := "/raw?ref=some_branch"}}
files:
- path: files/Readme.md
data: |
Project name: {{$project_name}}
- path: files/.gitignore
# copy file from location
local: some/dir/.gitignore.gotmpl
- path: files/.editorconfig
get:
url: "{{printf `%s%s` `.editorconfig` $gitlab_suffix}}"
- path: files/.gitlab-ci.yml
# GET file from remote storage
get:
# reset URL which set in http client configuration (http.base_url)
url: "https://some_file_server.com/files/.gitlab-ci.yml"
# reset headers of common http client configuration (http.headers)
headers:
some_header: header
query_params:
PARAM_1: Val_1
- path: files/Dockerfile
# GET file from remote storage (using common http client config)
get:
# reuse `base` URL of common http client config (http.base_url)
url: Dockerfile/raw?ref=feature/project_templates"
% progen -v
2023-02-05 14:47:25 INFO current working direcotry: /Users/user_1/GoProjects/service
2023-02-05 14:47:25 INFO configuration file: progen.yaml
2023-02-05 14:47:25 INFO file process: files/Readme.md
2023-02-05 14:47:25 INFO file process: files/.gitignore
2023-02-05 14:47:25 INFO file process: files/.editorconfig
2023-02-05 14:47:25 INFO file process: files/.gitlab-ci.yml
2023-02-05 14:47:25 INFO file process: files/Dockerfile
...
2023-02-05 14:47:25 INFO file saved: files/Readme.md
2023-02-05 14:47:25 INFO file saved: files/.gitignore
2023-02-05 14:47:25 INFO file saved: files/.editorconfig
2023-02-05 14:47:25 INFO file saved: files/.gitlab-ci.yml
2023-02-05 14:47:25 INFO file saved: files/Dockerfile
...
Execution commands process configured by specifying commands working directory and commands definition.
Default value of commands working directory (dir
tag) is .
.
Commands working directory calculate from the application working directory.
## progen.yml
cmd:
- exec: ls -l
args: [ - l ]
dir: .github/workflows
- exec: tree
args: [ -L, 1 ]
% progen -v
2023-02-02 22:18:20 INFO application working directory: /Users/user_1/GoProjects/progen
2023-02-02 22:18:20 INFO configuration read: progen.yml
2023-02-02 22:18:20 INFO execute [dir: .github/workflows]: ls -l
out:
total 16
-rw-r--r-- 1 19798572 646495703 762 Feb 1 09:15 release.yml
-rw-r--r-- 1 19798572 646495703 377 Jan 24 20:06 test.yml
2023-02-02 22:18:20 INFO execute [dir: .]: tree -L 1
out:
.
├── LICENSE
├── Makefile
├── Readme.md
├── go.mod
├── go.sum
├── internal
├── main.go
└── tmp
2 directories, 6 files
cmd
action maintains "short" declaration syntax
## progen.yml
cmd:
- pwd
- ls -a
% progen -v -dr
2023-02-15 17:56:58 INFO application working directory: /Users/user_1/GoProjects/progen
2023-02-15 17:56:58 INFO configuration file: short.yml
2023-02-15 17:56:58 INFO execute [dir: .]: pwd
2023-02-15 17:56:58 INFO execute [dir: .]: ls -a
fs
section configure execution text/template on a directories tree.
All files in the tree
processed as template
. Files and directories names also could be configured as templates.
## progen.yml
var_d: VAR_d
var_f: VAR_f
cmd:
- cp -a ../asserts/. ../out/
- exec: tree
dir: .
fs:
- test_dir
- test_dir_2
cmd_finish:
- exec: tree
dir: .
% progen -v -awd=out -f ../progen.yml
2023-02-12 14:01:45 INFO application working directory: /Users/user_1/GoProjects/progen
2023-02-12 14:01:45 INFO configuration file: ../progen.yml
2023-02-12 14:01:45 INFO execute [dir: .]: cp -a ../asserts/. ../out/
2023-02-12 14:01:45 INFO execute [dir: .]: tree
out:
.
├── test_dir
│ ├── file1
│ └── {{ .var_d }}
│ └── {{ .var_f }}
└── test_dir_2
├── file1
└── {{ .var_d }}
└── {{ .var_f }}
4 directories, 4 files
2023-02-12 14:01:45 INFO dir created: test_dir/VAR_d
2023-02-12 14:01:45 INFO file saved: test_dir/file1
2023-02-12 14:01:45 INFO file saved: test_dir/VAR_d/VAR_f
2023-02-12 14:01:45 INFO dir created: test_dir_2/VAR_d
2023-02-12 14:01:45 INFO file saved: test_dir_2/file1
2023-02-12 14:01:45 INFO file saved: test_dir_2/VAR_d/VAR_f
2023-02-12 14:01:45 INFO fs: remove: test_dir_2/{{ .var_d }}/{{ .var_f }}
2023-02-12 14:01:45 INFO fs: remove: test_dir_2/{{ .var_d }}
2023-02-12 14:01:45 INFO execute [dir: .]: tree
out:
.
├── test_dir
│ ├── VAR_d
│ │ └── VAR_f
│ └── file1
└── test_dir_2
├── VAR_d
│ └── VAR_f
└── file1
rm
use to remove files, directories or files inside a directory.
rm:
# remove the dir
- some_dir
# remove all files in the dir
- some_dir_2/*
# remove the file
- some_dir_3/file.txt
% progen -v
2024-02-09 22:50:51 INFO application working directory: /Users/user_1/GoProjects/progen
2023-02-12 14:01:45 INFO configuration file: progen.yml
2024-02-09 22:50:51 INFO rm: some_dir
2024-02-09 22:50:51 INFO rm all: some_dir_2/*
2024-02-09 22:50:51 INFO rm: some_dir_3/file.txt
2024-02-09 22:50:51 INFO execution time: 350.149µs
To use progen
for building custom generator based on go
language, imports pkg/core
package
and implements required algorithm:
package main
import (
"log"
"testing/fstest"
"github.com/kozmod/progen/pkg/core"
)
var fs = fstest.MapFS{
"1": {
Data: []byte("aaaa"),
},
}
func main() {
// Parse default config base on flags
c, err := core.ParseFlags()
if err != nil {
log.Fatal(err)
}
var e core.Engin
// add actions
e.AddActions(
// create files actions
core.FilesAction(
"create_file",
core.File{Path: "./xx/1", Data: []byte("file_1")},
core.File{Path: "./xx/2", Data: []byte("file_2")},
core.File{Path: "./xx/rm_1", Data: []byte("file_rm")},
).WithPriority(1),
// rm actions
core.RmAction(
"rm",
"./xx/rm_1",
).WithPriority(2),
)
e.AddActions(
// create file system action
core.FsSaveAction(
"fs_1",
core.TargetFs{
TargetDir: "./xx/fs",
Fs: fs, // any [io/fs.FS] (from local system, embed, etc.)
},
).WithPriority(3),
// cmd action
core.CmdAction(
"tree_1",
core.Cmd{
Cmd: "tree",
Dir: "./xx",
},
).WithPriority(4),
)
err = e.Run(c)
if err != nil {
log.Fatal(err)
}
}
% go build .
% ./tmp -v
2024-12-11 10:16:17 INFO action is going to be execute ('priopiry':'name')['1':'create_file','2':'rm','3':'fs_1','4':'tree_1']
2024-12-11 10:16:17 INFO file saved: xx/1
2024-12-11 10:16:17 INFO file saved: xx/2
2024-12-11 10:16:17 INFO file saved: xx/rm_1
2024-12-11 10:16:17 INFO rm: ./xx/rm_1
2024-12-11 10:16:17 INFO dir created: xx/fs
2024-12-11 10:16:17 INFO file saved: xx/fs/1
2024-12-11 10:16:17 INFO execute [dir: ./xx]: tree
out:
.
├── 1
├── 2
└── fs
└── 1
1 directory, 3 files
progen-example repository contains useful examples of usage cli