In a multi-main setup, we have the following issue. The user's client
connects to main A and runs a test webhook, so main A starts listening
for a webhook call. A third-party service sends a request to the test
webhook URL. The request is forwarded by the load balancer to main B,
who is not listening for this webhook call. Therefore, the webhook call
is unhandled.
To start addressing this, cache test webhook registrations, using Redis
for queue mode and in-memory for regular mode. When the third-party
service sends a request to the test webhook URL, the request is
forwarded by the load balancer to main B, who fetches test webhooks from
the cache and, if it finds a match, executes the test webhook. This
should be transparent - test webhook behavior should remain the same as
so far.
Notes:
- Test webhook timeouts are not cached. A timeout is only relevant to
the process it was created in, so another process retrieving from Redis
a "foreign" timeout will be unable to act on it. A timeout also has
circular references, so `cache-manager-ioredis-yet` is unable to
serialize it.
- In a single-main scenario, the timeout remains in the single process
and is cleared on test webhook expiration, successful execution, and
manual cancellation - all as usual.
- In a multi-main scenario, we will need to have the process who
received the webhook call send a message to the process who created the
webhook directing this originating process to clear the timeout. This
will likely be implemented via execution lifecycle hooks and Redis
channel messages checking session ID. This implementation is out of
scope for this PR and will come next.
- Additional data in test webhooks is not cached. From what I can tell,
additional data is not needed for test webhooks to be executed.
Additional data also has circular references, so
`cache-manager-ioredis-yet` is unable to serialize it.
Follow-up to: #8155
also refactored the code to
1. stop passing around `scope === 'global'`, since this code can be used
only for changing globalRole.
2. leak less details when input validation fails.
## Review / Merge checklist
- [x] PR title and summary are descriptive
- [x] Tests included
## Summary
This PR continues refactoring webhooks code for better modularity.
Continued from #8069 to bring back `ActiveWebhooks`, but this time
actually handling active webhook calls in this class.
## Review / Merge checklist
- [x] PR title and summary are descriptive
This PR simplifies state in test webhooks so that it can be cached
easily. Caching this state will allow us to start using Redis for manual
webhooks, to support manual webhooks to work in multi-main setup.
- [x] Convert `workflowWebhooks` to a getter - no need to optimize for
deactivation
- [x] Remove array from value in `TestWebhooks.webhookUrls`
- [x] Consolidate `webhookUrls` and `registeredWebhooks`
## Summary
Fixes the issue that pinned data gets also used for production executions.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
- Move webhook, poller and trigger activation logs closer to activation
event
- Enrich response of `/debug/multi-main-setup`
- Ensure workflow updates broadcast activation state changes only if
state changed
- Fix bug on workflow activation after leadership change
- Ensure debug controller is not available in production
---------
Co-authored-by: Omar Ajoue <krynble@gmail.com>
## Summary
When a workflow gets started via a Test-Webhook the pinned data does get
ignored and the nodes executed anyway.
## Related tickets and issues
> Include links to **Linear ticket** or Github issue or Community forum
post. Important in order to close *automatically* and provide context to
reviewers.
## Review / Merge checklist
- [x] PR title and summary are descriptive. **Remember, the title
automatically goes into the changelog. Use `(no-changelog)` otherwise.**
([conventions](https://github.com/n8n-io/n8n/blob/master/.github/pull_request_title_conventions.md))
- [ ] [Docs updated](https://github.com/n8n-io/n8n-docs) or follow-up
ticket created.
- [X] Tests included.
> A bug is not considered fixed, unless a test is added to prevent it
from happening again.
> A feature is not complete without tests.
---------
Co-authored-by: Mutasem Aldmour <mutasem@n8n.io>
This change kept coming up in #6713, #7773, and #8135.
So this PR moves the existing code without actually changing anything,
to help get rid of some of the circular dependencies.
## Review / Merge checklist
- [x] PR title and summary are descriptive.
## Summary
Add `ShutdownService` and `OnShutdown` decorator for more unified way to
shutdown different components. Use this new way in the following
components:
- HTTP(S) server
- Pruning service
- Push connection
- License
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
## Summary
A circular dependency between `WorkflowService` and
`ActiveWorkflowRunner` is sometimes causing `this.activeWorkflowRunner`
to be `undefined` in `WorkflowService`.
Breaking this circular dependency should hopefully fix this issue.
## Related tickets and issues
#8122
## Review / Merge checklist
- [x] PR title and summary are descriptive
- [ ] Tests included
In case someone manually prunes their executions table before upgrading
to 1.x, `MigrateIntegerKeysToString` should gracefully handle that,
instead of crashing the application.
## Review / Merge checklist
- [x] PR title and summary are descriptive
In the case of a filesystem failure to rename the binary files as part
of the execution's cleanup process, the execution would fail to be saved
and would never finish. This catch prevents it.
## Summary
Whenever an execution is wrapping u to save the data, if it uses binary
data n8n will try to find possibly misallocated files and place them in
the right folder. If this process fails, the execution fails to finish.
Given the execution has already finished at this point, and we cannot
handle the binary data errors more gracefully, all we can do at this
point is log the message as it's a filesystem issue. The rest of the
execution saving process should remain as normal.
## Related tickets and issues
https://linear.app/n8n/issue/HELP-430
## Review / Merge checklist
- [ ] PR title and summary are descriptive. **Remember, the title
automatically goes into the changelog. Use `(no-changelog)` otherwise.**
([conventions](https://github.com/n8n-io/n8n/blob/master/.github/pull_request_title_conventions.md))
- [ ] [Docs updated](https://github.com/n8n-io/n8n-docs) or follow-up
ticket created.
- [ ] Tests included.
> A bug is not considered fixed, unless a test is added to prevent it
from happening again.
> A feature is not complete without tests.
---------
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
This reverts commit a895ee87fc (#8090)
Our telemetry backend is throwing 500s with the updated rudderstack sdk.
Until that is resolved, we need to downgrade.
## Review / Merge checklist
- [x] PR title and summary are descriptive
Remove duplication, improve readability, and expand tests for
`TestWebhooks.ts` - in anticipation for storing test webhooks in Redis.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
This helps remove some of the older versions of transient dependencies,
like axios 0.x and ioredis 4.x.
## Review / Merge checklist
- [x] PR title and summary are descriptive.
## Summary
We accidentally made some functions `async` in
https://github.com/n8n-io/n8n/pull/7846
This PR reverts that change.
## Review / Merge checklist
- [x] PR title and summary are descriptive.
`tsc-alias` doesn't seem to replace imports when using template strings
## Related tickets and issues
#8085
## Review / Merge checklist
- [x] PR title and summary are descriptive.
## Summary
This PR updates our backend sentry setup to remove integrations that
don't provide us any value. This also reduces the amount of PII that
gets sent to Sentry.
[Sample event](https://n8nio.sentry.io/issues/4725315362/)
## Related tickets
[ENG-95](https://linear.app/n8n/issue/ENG-95)
## Review / Merge checklist
- [x] PR title and summary are descriptive.
Add generic N8N_GRACEFUL_SHUTDOWN_TIMEOUT which controls how long n8n
process will wait for graceful exit before exitting forcefully. This
variables replaces the QUEUE_WORKER_TIMEOUT variable that was used for
worker process.
DEPRECATED: QUEUE_WORKER_TIMEOUT deprected
QUEUE_WORKER_TIMEOUT environment variable has been replaced with
N8N_GRACEFUL_SHUTDOWN_TIMEOUT.
If the process doesn't shutdown within a time limit, exit with error
code.
1. conceptually something timing out is an error.
2. on successful exit we close down the DB connection gracefully. On an
exit timeout we rather not do that, since it will wait for any active
connections to close and would possible block the exit.
## Summary
Hashicorp Vault prefers a `LIST` HTTP method to be used when fetching
secrets but not all environments will allow custom http methods through
WAFs. This PR adds `N8N_EXTERNAL_SECRETS_PREFER_GET` which when set to
`true` will use GET instead of LIST to fetch secrets.
## Review / Merge checklist
- [x] PR title and summary are descriptive. **Remember, the title
automatically goes into the changelog. Use `(no-changelog)` otherwise.**
([conventions](https://github.com/n8n-io/n8n/blob/master/.github/pull_request_title_conventions.md))
## Summary
Handle circular references in the public API for executions created
prior to the fix from #8030
## Related tickets
[PAY-1119](https://linear.app/n8n/issue/PAY-1119)
## Review / Merge checklist
- [x] PR title and summary are descriptive.
Refactor static workflow service classes into DI-compatible classes
Context: https://n8nio.slack.com/archives/C069HS026UF/p1702466571648889
Up next:
- Inject dependencies into workflow services
- Consolidate workflow controllers into one
- Make workflow controller injectable
- Inject dependencies into workflow controller
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
We added ID-less workflow reporting at #8031, which has already produced
multiple reports coming from internal, enough info to tackle [this
story](https://linear.app/n8n/issue/PAY-1147). To prevent an
overwhelming number of reports from cloud, this PR removes the reporting
for now.
