Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Commit 20ca435

Browse files
Merge pull request #59 from picostack/staging
v1.4.1
2 parents a56d4a1 + 40a642e commit 20ca435

File tree

5 files changed

+67
-205
lines changed

5 files changed

+67
-205
lines changed

‎README.md‎

Lines changed: 29 additions & 181 deletions
Original file line numberDiff line numberDiff line change
@@ -1,181 +1,29 @@
1-
# Pico
2-
3-
_The little git robot of automation!_
4-
5-
[![Build Status](https://travis-ci.org/picostack/pico.svg?branch=master)](https://travis-ci.org/picostack/pico)
6-
7-
Pico is a git-driven task runner to automate the application of configs.
8-
9-
## Overview
10-
11-
Pico is a little tool for implementing [Git-Ops][git-ops] in single-server environments. It's analogous to
12-
[kube-applier][kube-applier], [Terraform][terraform], [Ansible][ansible] but for automating lone servers that do not
13-
need cluster-level orchestration.
14-
15-
Instead, Pico aims to be extremely simple. You give it some Git repositories and tell it to run commands when those
16-
Git repositories receive commits and that's about it. It also provides a way of safely passing in credentials from
17-
[Hashicorp's Vault][vault].
18-
19-
## Install
20-
21-
### Linux
22-
23-
```sh
24-
curl -s https://raw.githubusercontent.com/picostack/pico/master/install.sh | bash
25-
```
26-
27-
Or via Docker:
28-
29-
```sh
30-
docker pull picostack/pico:v1
31-
```
32-
33-
See the docker section below and the image on [Docker Hub](https://hub.docker.com/r/picostack/pico).
34-
35-
### Everything Else
36-
37-
It's primarily a server side tool aimed at Linux servers, so there aren't any install scripts for other platforms. Most
38-
Windows/Mac usage is probably just local testing so just use `go get` for these use-cases.
39-
40-
## Usage
41-
42-
Currently, Pico has a single command: `run` and it takes a single parameter: a Git URL. This Git URL defines the
43-
"Config Repo" which contains Pico configuration files. These configuration files declare where Pico can find
44-
"Target Repos" which are the repos that contain all the stuff you want to automate. The reason Pico is designed
45-
this way instead of just using the target repos to define what Pico should do is 1. to consolidate Pico config
46-
into one place, 2. separate the config of the tools from the applications and 3. keep your target repos clean.
47-
48-
Pico also has a Docker image - see below for docker-specific information.
49-
50-
### Configuration
51-
52-
The precursor to Pico used JSON for configuration, this was fine for simple tasks but the ability to provide a
53-
little bit of logic and variables for repetitive configurations is very helpful. Inspired by [StackExchange's
54-
dnscontrol][dnscontrol], Pico uses JavaScript files as configuration. This provides a JSON-like environment with
55-
the added benefit of conditional logic.
56-
57-
Here's a simple example of a configuration that should exist in the Pico config repo that re-deploys a Docker
58-
Compose stack whenever it changes:
59-
60-
```js
61-
T({
62-
name: "my_app",
63-
url: "git@github.com:username/my-docker-compose-project",
64-
branch: "prod",
65-
up: ["docker-compose", "up", "-d"],
66-
down: ["docker-compose", "down"]
67-
});
68-
```
69-
70-
#### The `T` Function
71-
72-
The `T` function declares a "Target" which is essentially a Git repository. In this example, the repository
73-
`git@github.com:username/my-docker-compose-project` would contain a `docker-compose.yml` file for some application
74-
stack. Every time you make a change to this file and push it, Pico will pull the new version and run the command
75-
defined in the `up` attribute of the target, which is `docker-compose up -d`.
76-
77-
You can put as many target declarations as you want in the config file, and as many config files as you want in the
78-
config repo. You can also use variables to cut down on repeated things:
79-
80-
```js
81-
var GIT_HOST = "git@github.com:username/";
82-
T({
83-
name: "my_app",
84-
url: GIT_HOST + "my-docker-compose-project",
85-
up: ["docker-compose", "up", "-d"]
86-
});
87-
```
88-
89-
Or, if you have a ton of Docker Compose projects and they all live on the same Git host, why not declare a function that
90-
does all the hard work:
91-
92-
```js
93-
var GIT_HOST = "git@github.com:username/";
94-
95-
function Compose(name) {
96-
return {
97-
name: name,
98-
url: GIT_HOST + name,
99-
up: ["docker-compose", "up", "-d"]
100-
};
101-
}
102-
103-
T(Compose("homepage"));
104-
T(Compose("todo-app"));
105-
T(Compose("world-domination-scheme"));
106-
```
107-
108-
The object passed to the `T` function accepts the following keys:
109-
110-
- `name`: The name of the target
111-
- `url`: The Git URL (ssh or https)
112-
- `up`: The command to run on first-run and on changes
113-
- `down`: The command to run when the target is removed
114-
- `env`: Environment variables to pass to the target
115-
116-
#### The `E` Function
117-
118-
The only other function available in the configuration runtime is `E`, this declares an environment variable that will
119-
be passed to the `up` and `down` commands for all targets.
120-
121-
For example:
122-
123-
```js
124-
E("MOUNT_POINT", "/data");
125-
T({ name: "postgres", url: "...", up: "docker-compose", "up", "-d" });
126-
```
127-
128-
This would pass the environment variable `MOUNT_POINT=/data` to the `docker-compose` invocation. This is useful if you
129-
have a bunch of compose configs that all mount data to some path on the machine, you then use
130-
`${MOUNT_POINT}/postgres:/var/lib/postgres/data` as a volume declaration in your `docker-compose.yml`.
131-
132-
## Usage as a Docker Container
133-
134-
See the `docker-compose.yml` file for an example and read below for details.
135-
136-
You can run Pico as a Docker container. If you're using it to deploy Docker containers via compose, this makes the
137-
most sense. This is quite simple and is best done by writing a Docker Compose configuration for Pico in order to
138-
bootstrap your deployment.
139-
140-
The Pico image is built on the `docker/compose` image, since most use-cases will use Docker or Compose to deploy
141-
services. This means you must mount the Docker API socket into the container, just like Portainer or cAdvisor or any of
142-
the other Docker tools that also run inside a container.
143-
144-
The socket is located by default at `/var/run/docker.sock` and the `docker/compose` image expects this path too, so you
145-
just need to add a volume mount to your compose that specifies `/var/run/docker.sock:/var/run/docker.sock`.
146-
147-
Another minor detail you should know is that Pico exposes a `HOSTNAME` variable for the configuration script.
148-
However, when in a container, this hostname is a randomised string such as `b50fa67783ad`. This means, if your
149-
configuration performs checks such as `if (HOSTNAME === 'server031')`, this won't work. To resolve this, Pico will
150-
attempt to read the environment variable `HOSTNAME` and use that instead of using `/etc/hostname`.
151-
152-
This means, you can bootstrap a Pico deployment with only two variables:
153-
154-
```env
155-
VAULT_TOKEN=abcxyz
156-
HOSTNAME=server012
157-
```
158-
159-
### Docker Compose and `./` in Container Volume Mounts
160-
161-
Another caveat to running Pico in a container to execute `docker-compose` is the container filesystem will not
162-
match the host filesystem paths.
163-
164-
If you mount directories from your repository - a common strategy for versioning configuration - `./` will be expanded
165-
by Docker compose running inside the container, but this path may not be valid in the context of the Docker daemon,
166-
which will be running on the host.
167-
168-
The solution to this is both `DIRECTORY: "/cache"` and `/cache:/cache`: as long as the path used in the container also
169-
exists on the host, Docker compose will expand `./` to the same path as the host and everything will work fine.
170-
171-
This also means your config and target configurations will be persisted on the host's filesystem.
172-
173-
<!-- Links -->
174-
175-
[wadsworth]: https://i.imgur.com/RCYbkiq.png
176-
[git-ops]: https://www.weave.works/blog/gitops-operations-by-pull-request
177-
[kube-applier]: https://github.com/box/kube-applier
178-
[terraform]: https://terraform.io
179-
[ansible]: https://ansible.com
180-
[vault]: https://vaultproject.io
181-
[dnscontrol]: https://stackexchange.github.io/dnscontrol/
1+
<p align="center">
2+
<a aria-label="Pico logo" href="https://pico.sh">
3+
<img src="https://pico.sh/img/pico-wordmark-1000.png" width="420" />
4+
</a>
5+
</p>
6+
7+
<p align="center">
8+
<em>The little git robot of automation!</em>
9+
</p>
10+
11+
<p align="center">
12+
<img
13+
alt="GitHub Workflow Status"
14+
src="https://img.shields.io/github/workflow/status/picostack/pico/Test?style=for-the-badge"
15+
/>
16+
<img
17+
alt="License"
18+
src="https://img.shields.io/github/license/picostack/pico?style=for-the-badge"
19+
/>
20+
</p>
21+
22+
<p align="center">
23+
Pico is a Git-driven task runner built to facilitate GitOps and
24+
Infrastructure-as-Code while securely passing secrets to tasks.
25+
</p>
26+
27+
<p align="center">
28+
<a href="https://pico.sh">pico.sh</a>
29+
</p>

‎main.go‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,10 @@ package main
22

33
import (
44
"context"
5+
"log"
56
"os"
67
"os/signal"
8+
"runtime"
79
"time"
810

911
_ "github.com/joho/godotenv/autoload"
@@ -114,6 +116,19 @@ this repository has new commits, Pico will automatically reconfigure.`,
114116
},
115117
}
116118

119+
if os.Getenv("DEBUG") != "" {
120+
go func() {
121+
sigs := make(chan os.Signal, 1)
122+
signal.Notify(sigs, os.Interrupt)
123+
buf := make([]byte, 1<<20)
124+
for {
125+
<-sigs
126+
stacklen := runtime.Stack(buf, true)
127+
log.Printf("\nPrinting goroutine stack trace because `DEBUG` was set.\n%s\n", buf[:stacklen])
128+
}
129+
}()
130+
}
131+
117132
err := app.Run(os.Args)
118133
if err != nil {
119134
zap.L().Fatal("exit", zap.Error(err))

‎reconfigurer/git.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,9 @@ func (p *GitProvider) reconfigure(w watcher.Watcher) (err error) {
8888
state.Env["HOSTNAME"] = p.hostname
8989
}
9090

91+
zap.L().Debug("setting state for watcher",
92+
zap.Any("new_state", state))
93+
9194
return w.SetState(state)
9295
}
9396

‎service/service.go‎

Lines changed: 18 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,6 @@ import (
1010
"github.com/eapache/go-resiliency/retrier"
1111
"github.com/pkg/errors"
1212
"go.uber.org/zap"
13-
"golang.org/x/sync/errgroup"
1413
"gopkg.in/src-d/go-git.v4/plumbing/transport"
1514
"gopkg.in/src-d/go-git.v4/plumbing/transport/http"
1615
"gopkg.in/src-d/go-git.v4/plumbing/transport/ssh"
@@ -110,39 +109,34 @@ func Initialise(c Config) (app *App, err error) {
110109

111110
// Start launches the app and blocks until fatal error
112111
func (app *App) Start(ctx context.Context) error {
113-
g, ctx := errgroup.WithContext(ctx)
114-
115-
zap.L().Debug("starting service daemon")
116-
117-
// TODO: Replace this errgroup with a more resilient solution.
118-
// Not all of these tasks fail in the same way. Some don't fail at all.
119-
// This needs to be rewritten to be more considerate of different failure
120-
// states and potentially retry in some circumstances. Pico should be the
121-
// kind of service that barely goes down, only when absolutely necessary.
112+
errs := make(chan error)
122113

123114
ce := executor.NewCommandExecutor(app.secrets, app.config.PassEnvironment, app.config.VaultConfig, "GLOBAL_")
124-
g.Go(func()error {
115+
gofunc() {
125116
ce.Subscribe(app.bus)
126-
return nil
127-
})
117+
}()
128118

129-
// TODO: gw can fail when setting up the gitwatch instance, it should retry.
130119
gw := app.watcher.(*watcher.GitWatcher)
131-
g.Go(gw.Start)
120+
go func() {
121+
errs <- errors.Wrap(gw.Start(), "git watcher terminated fatally")
122+
}()
132123

133-
// TODO: reconfigurer can also fail when setting up gitwatch.
134-
g.Go(func() error {
135-
return app.reconfigurer.Configure(app.watcher)
136-
})
124+
go func() {
125+
errs <- errors.Wrap(app.reconfigurer.Configure(app.watcher), "git watcher terminated fatally")
126+
}()
137127

138128
if s, ok := app.secrets.(*vault.VaultSecrets); ok {
139-
g.Go(func() error {
140-
return retrier.New(retrier.ConstantBackoff(3, 100*time.Millisecond), nil).
141-
RunCtx(ctx, s.Renew)
142-
})
129+
go func() {
130+
errs <- errors.Wrap(retrier.New(retrier.ConstantBackoff(3, 100*time.Millisecond), nil).RunCtx(ctx, s.Renew), "git watcher terminated fatally")
131+
}()
143132
}
144133

145-
return g.Wait()
134+
select {
135+
case err := <-errs:
136+
return err
137+
case <-ctx.Done():
138+
return context.Canceled
139+
}
146140
}
147141

148142
func getAuthMethod(c Config, secretConfig map[string]string) (transport.AuthMethod, error) {

‎watcher/git.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,8 @@ func NewGitWatcher(
6262

6363
// Start runs the watcher loop and blocks until a fatal error occurs
6464
func (w *GitWatcher) Start() error {
65+
zap.L().Debug("git watcher initialising, waiting for first state to be set")
66+
6567
// wait for the first config event to set the initial state
6668
<-w.initialise
6769

0 commit comments

Comments
(0)

AltStyle によって変換されたページ (->オリジナル) /