feat: streaming matches to persistent connection backend
Hello,
Would you be interested in the ability to use stream outputs in reaction you can write matches into, so we save the overhead of spawning a binary each match?
This could be useful running reaction on a centralized syslog server, seeing lots of authentications. This server is not the host taking iptables/pf/whatever blacklisting actions, but instead it writes IP to ban into a message broker like redis, kafka, rabbitMQ, whatever.
It can already be done today by configuring streams.filters.actions.cmd to something like :
cmd: ['redis-cli', '-h', '1.2.3.4', '-a', 'mypasswordoncmdlinedontdothis', 'XADD', 'banstream', '*', 'ip', '<ip>']
but the overhead is costly with lots of match, especially when running reaction on an large datasource (like reading authentications logs from message broker with retention)
For an idea of a gain, I ran the following test :
- Setup :
- 4 streams : 3 matching, 1 not matching. Inspired by heavy-load.yml
- Each stream match 2 regexes, then send it to redis instance with authentication.
- 100010 lines of log per stream
- Tests :
-
Sending matches to redis with "cmd" :
1071.367u 22734.641s 28:03.19 1414.3% 2360+3258k 0+300077io 0pf+0w
-
Streaming results to locally persistent running redis-cli output :
38.419u 35.042s 0:35.32 207.9% 2458+3056k 0+300079io 0pf+0w
-
This is csh time output. We come from 28:03.19 running time with a load of 1414% ; to 00:35.32 with a load of 207%. Heh, spawning binaries cost a lot
I realize the test is somewhat overdone
Here is the current configuration syntax :
# Outputs are commands returning stdin you can use in write actions.
# This can ben used to get a persistent connection to p.e. a KV database you will write into,
# eliminating the overhead of executing a process each time action is trigged.
outputs:
redis:
start: ['redis-cli', '-h', 'redis.example.org', '-a', 'mypasswordoncmdlinedontdothis']
streams:
ssh:
cmd: ['tail', '-f', '/var/log/auth.log']
filters:
acceptedlogin:
regex:
- 'Accepted <method> for <login> from <ip> port <port>'
actions:
store2redis:
write:
output: redis
text: ['XADD', 'logins', '*', 'username', '<login>', 'method', '<method>', 'ip', '<ip>', 'port', '<port>']
See https://framagit.org/yoamarf/reaction/-/tree/persistent-stream-output?ref_type=heads for current running code.