Workceptor
Workceptor is a component of receptor that handles units of work.
work-command defines a type of work that can run on the node.
foo.yml
---
- node:
id: foo
- log-level:
level: Debug
- tcp-listener:
port: 2222
- control-service:
service: control
filename: /tmp/foo.sock
- work-command:
workType: echoint
command: bash
params: "-c \"for i in {1..5}; do echo $i; sleep 1; done\""
bar.yml
---
- node:
id: bar
- log-level:
level: Debug
- tcp-peer:
address: localhost:2222
- control-service:
service: control
- work-command:
worktype: echoint
command: bash
params: "-c \"for i in {1..10}; do echo $i; sleep 1; done\""
- work-command:
workType: echopayload
command: bash
params: "-c \"while read -r line; do echo ${line^^}; sleep 3; done\""
Configuring work commands
worktype User-defined name to give this work definition
command The executable that is invoked when running this work
params Command-line options passed to this executable
Local work
Start the work by connecting to the control-service and issuing a “work submit” command
$ receptorctl --socket /tmp/foo.sock work submit echoint --no-payload
Result: Job Started
Unit ID: t1BlAB18
Receptor started an instance of this work type, and labeled it with a unique “Unit ID”
Work results
Use the “Unit ID” to get work results
receptorctl --socket /tmp/foo.sock work results t1BlAB18
1
2
3
4
5
6
7
8
9
10
Remote work
Although connected to foo, by providing the “–node” option the work can be started on node bar.
The work type must be defined on the node it is intended to run on, e.g. bar must have a work-command called “echoint”, in this case.
$ receptorctl --socket /tmp/foo.sock work submit echoint --node bar --no-payload
Result: Job Started
Unit ID: 87Vwqb6A
Remote work submission ultimately results in two work units running at the same time; a local work unit and the remote work unit. These two units have their own Unit IDs. The local work unit’s goal is to monitor and stream results back from the running remote work unit.
Sequence of events for remote work submission
foo starts a local work unit of work type “remote”. This is a special work type that is built into receptor.
This work unit attempts to connect to bar’s control service and issue a “work submit echoint” command. From bar’s perspective, this is the exact same operation as if a user connected to bar directly and issued a work submit command. bar is not aware that foo is the one that issued the command.
Once submitted, foo will stream work results back to itself and store it on disk. It also periodically gets the
work statusof the work running on bar. Status includes information about the work state and the stdout size.foo continues streaming stdout results until the size stored on disk matches the StdoutSize reported in bar’s status.
Payload
in bar.yml
- work-command:
workType: echopayload
command: bash
params: "-c \"while read -r line; do echo ${line^^}; sleep 5; done\""
Here the bash command expects to read a line from stdin, echo the line in all uppercase letters, and sleep for 3 seconds.
Payloads can be passed into receptor using the “–payload” option.
$ echo -e "hi\ni am foo\nwhat is your name" | receptorctl --socket /tmp/foo.sock work submit echopayload --node bar --payload - -f
HI
I AM FOO
WHAT IS YOUR NAME
“–payload -” means the payload should be whatever the stdin is, which is piped in from the “echo -e …” command.
Note: “-f” instructs receptorctl to follow the work unit immediately, i.e. stream results to stdout. One could also use “work results” to stream the results.
Runtime Parameters
Work commands can be configured to allow parameters to be passed to commands when work is submitted:
- work-command:
workType: listcontents
command: ls
allowruntimeparams: true
The allowruntimeparams option will allow parameters to be passed to the work command by the
client submitting the work. The contents of a specific directory can be listed by passing the paths
to the receptor command as positional arguments immediately after the workType:
receptorctl --socket /tmp/foo.sock work submit --node bar --no-payload -f listcontents /root/ /bin/
/bin/:
bash
sh
/root/:
helloworld.sh
Passing options or flags to the work command needs to be done using the --param parameter to
extend the params work command setting. The --all flag can be passed to the work command this way:
receptorctl --socket /tmp/foo.sock work submit --node bar --no-payload -f --param params='--all' listcontents /root/
.
..
.bash_logout
.bash_profile
.bashrc
.cache
helloworld.sh
Work list
“work list” returns information about all work units that have ran on this receptor node. The following shows two work units, 12L8s8h2 and T0oN0CAp
$ receptorctl --socket /tmp/foo.sock work list
{'12L8s8h2': {'Detail': 'exit status 0',
'ExtraData': None,
'State': 2,
'StateName': 'Succeeded',
'StdoutSize': 21,
'WorkType': 'echoint'},
'T0oN0CAp': {'Detail': 'Running: PID 1700818',
'ExtraData': {'Expiration': '0001-01-01T00:00:00Z',
'LocalCancelled': False,
'LocalReleased': False,
'RemoteNode': 'bar',
'RemoteParams': {},
'RemoteStarted': True,
'RemoteUnitID': 'ATDzdViR',
'RemoteWorkType': 'echoint',
'TLSClient': ''},
'State': 1,
'StateName': 'Running',
'StdoutSize': 4,
'WorkType': 'remote'},
Notice that T0oN0CAp was a remote work submission, therefore its work type is “remote”. On bar there is a local unit ATDzdViR, with the “echoint” work type.
Work cancel
Cancel will stop any running work unit. Upon canceling a “remote” work unit, the local node will attempt to connect to the remote node’s control service and issue a work cancel. If the remote node is down, receptor will periodically attempt to connect to the remote node to do the cancellation.
Work release
Release will cancel the work and then delete files on disk associated with that work unit. For remote work submission, release will attempt to delete files both locally and on the remote machine. Like work cancel, the release can be pending if the remote node is down. In that situation, the local files will remain on disk until the remote node can be contacted.
Work force-release
It might be preferable to force a release, using the work force-release command. This will do a one-time attempt to connect to the remote node and issue a work release there. After this one attempt, it will then proceed to delete all local files associated with the work unit.
States
A unit of work can be in Pending, Running, Succeeded, or Failed state
For local work, transitioning from Pending to Running occurs the moment the command executable is started
For remote work, transitioning from Pending to Running occurs when the status reported from the remote node has a Running state.
Signed work
Remote work submissions can be digitally signed by the sender. The target node will verify the signature of the work command before starting the work unit.
A single pair of RSA public and private keys is created offline and distributed to the nodes. Distribute the public key (PKIX format) to any node that should receive work. Distribute the private key (PKCS1 format) to any node that needs authority to submit work.
The following commands can be used to create keys for signing work:
openssl genrsa -out signworkprivate.pem 2048
openssl rsa -in signworkprivate.pem -pubout -out signworkpublic.pem
in bar.yml
# PKIX
- work-verification:
publickey: /full/path/signworkpublic.pem
- work-command:
workType: echopayload
command: bash
params: "-c \"while read -r line; do echo ${line^^}; sleep 5; done\""
verifysignature: true
in foo.yml
# PKCS1
- work-signing:
privatekey: /full/path/signworkprivate.pem
tokenexpiration: 30m
Tokenexpiration determines how long a the signature is valid for. This expiration directly corresponds to the “expiresAt” field in the generated JSON web token. Valid units include “h” and “m”, e.g. 1h30m for one hour and 30 minutes.
Use the “–signwork” parameter to sign the work.
$ receptorctl --socket /tmp/foo.sock work submit echoint --node bar --no-payload --signwork
Units on disk
Netceptor, the main component of receptor that handles mesh connectivity and traffic, operates entirely in memory. That is, it does not store any state information on disk. However, Workceptor functionality is designed to be persistent across receptor restarts. Work units might be running commands that could take hours to complete, and as such needs to store some relevant information on disk in case the receptor process restarts.
By default receptor stores data under /tmp/receptor but can be changed by setting the datadir param under the node action in the config file.
For a given work unit, receptor will store files in {datadir}/{nodeID}/{unitID}/.
Here is the receptor directory tree after running work submit echopayload described in Payload.
$ tree /tmp/receptor
/tmp/receptor
├── bar
│ └── NImim5WA
│ ├── status
│ ├── status.lock
│ ├── stdin
│ └── stdout
└── foo
└── BsAjS4wi
├── status
├── status.lock
├── stdin
└── stdout
The main purpose of work unit BsAjS4wi on foo is to copy stdin, stdout, and status from NImim5WA on bar back to its own working directory.
stdin is a copy of the submitted payload. The contents of this file is the same on both the local (foo) and remote (bar) machines.
$ cat /tmp/receptor/bar/NImim5WA/stdin
hi
i am foo
what is your name
stdout contains the work unit results; the stdout of the command execution. It will also be the same on both the local node and remote node.
$ cat /tmp/receptor/bar/NImim5WA/stdout
HI
I AM FOO
WHAT IS YOUR NAME
status contains additional information related to the work unit. The contents of status are different on foo and bar.
$ cat /tmp/receptor/bar/NImim5WA/stdout
{
"State":2,
"Detail":"exit status 0",
"StdoutSize":30,
"WorkType":"echopayload",
"ExtraData":null
}
$ cat /tmp/receptor/foo/BsAjS4wi/stdout
{
"State":2,
"Detail":"exit status 0",
"StdoutSize":30,
"WorkType":"remote",
"ExtraData":{
"RemoteNode":"bar",
"RemoteWorkType":"echopayload",
"RemoteParams":{},
"RemoteUnitID":"NImim5WA",
"RemoteStarted":true,
"LocalCancelled":false,
"LocalReleased":false,
"TLSClient":"",
"Expiration":"0001-01-01T00:00:00Z"
}
}
The sequence of events during a work remote submission. Blue lines indicate moments when receptor writes files to disk.