Iván Ovejero 22a5f5258d feat(core): Cache test webhook registrations (#8176) ... In a multi-main setup, we have the following issue. The user's client connects to main A and runs a test webhook, so main A starts listening for a webhook call. A third-party service sends a request to the test webhook URL. The request is forwarded by the load balancer to main B, who is not listening for this webhook call. Therefore, the webhook call is unhandled. To start addressing this, cache test webhook registrations, using Redis for queue mode and in-memory for regular mode. When the third-party service sends a request to the test webhook URL, the request is forwarded by the load balancer to main B, who fetches test webhooks from the cache and, if it finds a match, executes the test webhook. This should be transparent - test webhook behavior should remain the same as so far. Notes: - Test webhook timeouts are not cached. A timeout is only relevant to the process it was created in, so another process retrieving from Redis a "foreign" timeout will be unable to act on it. A timeout also has circular references, so `cache-manager-ioredis-yet` is unable to serialize it. - In a single-main scenario, the timeout remains in the single process and is cleared on test webhook expiration, successful execution, and manual cancellation - all as usual. - In a multi-main scenario, we will need to have the process who received the webhook call send a message to the process who created the webhook directing this originating process to clear the timeout. This will likely be implemented via execution lifecycle hooks and Redis channel messages checking session ID. This implementation is out of scope for this PR and will come next. - Additional data in test webhooks is not cached. From what I can tell, additional data is not needed for test webhooks to be executed. Additional data also has circular references, so `cache-manager-ioredis-yet` is unable to serialize it. Follow-up to: #8155		2024-01-03 16:58:33 +01:00
..
auth	refactor(core): Introduce password utility (no-changelog) (#7979)	2023-12-11 18:23:42 +01:00
collaboration	refactor(core): Continue moving typeorm operators to repositories (no-changelog) (#8186)	2024-01-02 17:53:24 +01:00
commands	refactor(core): Continue moving typeorm operators to repositories (no-changelog) (#8186)	2024-01-02 17:53:24 +01:00
config	refactor(core): Use Dependency Injection for all Controller classes (no-changelog) (#8146)	2023-12-27 11:50:43 +01:00
controllers	fix(core): Better input validation for the changeRole endpoint (#8189)	2024-01-03 09:33:35 +01:00
credentials	refactor(core): Continue moving typeorm operators to repositories (no-changelog) (#8186)	2024-01-02 17:53:24 +01:00
databases	refactor(core): Continue moving typeorm operators to repositories (no-changelog) (#8186)	2024-01-02 17:53:24 +01:00
decorators	refactor(core): Use Dependency Injection for all Controller classes (no-changelog) (#8146)	2023-12-27 11:50:43 +01:00
environments	refactor(core): Continue moving typeorm operators to repositories (no-changelog) (#8186)	2024-01-02 17:53:24 +01:00
errors	feat(core): Cache test webhook registrations (#8176)	2024-01-03 16:58:33 +01:00
eventbus	refactor(core): Continue moving typeorm operators to repositories (no-changelog) (#8186)	2024-01-02 17:53:24 +01:00
executionLifecycleHooks	fix: Stop binary data restoration from preventing execution from finishing (#8082)	2023-12-21 09:40:39 +00:00
executions	fix(core): Remove circular dependency in WorkflowService and ActiveWorkflowRunner (#8128)	2023-12-21 17:37:08 +01:00
ExternalSecrets	refactor(core): Use Dependency Injection for all Controller classes (no-changelog) (#8146)	2023-12-27 11:50:43 +01:00
helpers	fix(core): Add check that queue is defined and remove cyclic dependency (#7404)	2023-10-13 11:53:59 +02:00
Ldap	refactor(core): Switch plain errors in cli to ApplicationError (#7857)	2023-11-29 12:25:10 +01:00
license	refactor(core): Use Dependency Injection for all Controller classes (no-changelog) (#8146)	2023-12-27 11:50:43 +01:00
Mfa	ci(core): Reduce memory usage in tests (part-2) (no-changelog) (#7671)	2023-11-10 15:04:26 +01:00
middlewares	refactor(core): Switch plain errors in cli to ApplicationError (#7857)	2023-11-29 12:25:10 +01:00
permissions	feat(editor): Finalize workers view (#8052)	2023-12-20 17:49:14 +01:00
posthog	refactor: Run lintfix (no-changelog) (#7537)	2023-10-27 14:15:02 +02:00
PublicApi	refactor(core): Continue moving typeorm operators to repositories (no-changelog) (#8186)	2024-01-02 17:53:24 +01:00
push	feat(core): Unify application components shutdown (#8097)	2023-12-22 12:39:58 +02:00
security-audit	refactor(core): Move more typeorm operators to repositories (no-changelog) (#8143)	2023-12-22 16:20:30 +01:00
services	feat(core): Cache test webhook registrations (#8176)	2024-01-03 16:58:33 +01:00
shutdown	refactor(core): Use Dependency Injection for all Controller classes (no-changelog) (#8146)	2023-12-27 11:50:43 +01:00
sso	refactor(core): Use Dependency Injection for all Controller classes (no-changelog) (#8146)	2023-12-27 11:50:43 +01:00
telemetry	fix(core): Downgrade Rudderstack SDK (no-changelog) (#8107)	2023-12-20 11:59:44 +01:00
types	refactor: Move CLI commands types to own dir (no-changelog) (#6658)	2023-07-13 10:10:08 +02:00
UserManagement	refactor(core): Continue moving typeorm operators to repositories (no-changelog) (#8186)	2024-01-02 17:53:24 +01:00
workflows	feat(core): Cache test webhook registrations (#8176)	2024-01-03 16:58:33 +01:00
AbstractServer.ts	refactor(core): Extract ActiveWebhooks out of ActiveWorkflowRunner (no-changelog) (#8171)	2023-12-28 10:04:32 +01:00
ActivationErrors.service.ts	feat(core): Coordinate manual workflow activation and deactivation in multi-main scenario (#7643)	2023-11-17 15:58:50 +01:00
ActiveExecutions.ts	refactor(core): Don't use DB transactions on ExecutionRepository.createNewExecution (#8002)	2023-12-12 14:36:56 +01:00
ActiveWebhooks.ts	refactor(core): Extract ActiveWebhooks out of ActiveWorkflowRunner (no-changelog) (#8171)	2023-12-28 10:04:32 +01:00
ActiveWorkflowRunner.ts	refactor(core): Extract ActiveWebhooks out of ActiveWorkflowRunner (no-changelog) (#8171)	2023-12-28 10:04:32 +01:00
constants.ts	refactor(core): Introduce password utility (no-changelog) (#7979)	2023-12-11 18:23:42 +01:00
CrashJournal.ts	refactor(core): Make Logger a service (no-changelog) (#7494)	2023-10-25 16:35:22 +02:00
CredentialsHelper.ts	ci: Remove unnecessary async/await, enable await-thenable linting rule (no-changelog) (#8076)	2023-12-19 13:52:42 +01:00
CredentialsOverwrites.ts	refactor(core): Make Logger a service (no-changelog) (#7494)	2023-10-25 16:35:22 +02:00
CredentialTypes.ts	refactor(core): Switch plain errors in cli to ApplicationError (#7857)	2023-11-29 12:25:10 +01:00
curlconverter.d.ts	refactor: Clear unused ESLint directives from BE packages (no-changelog) (#6798)	2023-07-31 11:00:48 +02:00
CurlConverterHelper.ts	refactor: Clear unused ESLint directives from BE packages (no-changelog) (#6798)	2023-07-31 11:00:48 +02:00
Db.ts	refactor(core): Switch plain errors in cli to ApplicationError (#7857)	2023-11-29 12:25:10 +01:00
ErrorReporting.ts	refactor(core): Update backend sentry setup to reduce noise (no-changelog) (#8026)	2023-12-18 14:26:07 +01:00
ExpressionEvaluator.ts	fix(core): Fix expression evaluator filename typo (no-changelog) (#7960)	2023-12-07 17:50:49 +01:00
ExternalHooks.ts	refactor(core): Use Dependency Injection for all Controller classes (no-changelog) (#8146)	2023-12-27 11:50:43 +01:00
GenericHelpers.ts	fix(core): Better input validation for the changeRole endpoint (#8189)	2024-01-03 09:33:35 +01:00
index.ts	ci: Refactor cli tests to speed up CI (no-changelog) (#5718)	2023-03-17 17:24:05 +01:00
Interfaces.ts	feat(core): Cache test webhook registrations (#8176)	2024-01-03 16:58:33 +01:00
InternalHooks.ts	refactor(core): Use Dependency Injection for all Controller classes (no-changelog) (#8146)	2023-12-27 11:50:43 +01:00
jest.d.ts	feat(core): Security audit (#5034)	2023-01-05 13:28:40 +01:00
License.ts	feat(core): Unify application components shutdown (#8097)	2023-12-22 12:39:58 +02:00
LoadNodesAndCredentials.ts	refactor(core): Use Dependency Injection for all Controller classes (no-changelog) (#8146)	2023-12-27 11:50:43 +01:00
Logger.ts	ci: Stop disabling linting rules that are explicitly set to warn (no-changelog) (#7669)	2023-11-09 14:05:39 +01:00
NodeTypes.ts	refactor(core): Switch plain errors in cli to ApplicationError (#7857)	2023-11-29 12:25:10 +01:00
Queue.ts	refactor(core): Switch plain errors in cli to ApplicationError (#7857)	2023-11-29 12:25:10 +01:00
requests.ts	fix(core): Better input validation for the changeRole endpoint (#8189)	2024-01-03 09:33:35 +01:00
ResponseHelper.ts	feat(core): Add user.profile.beforeUpdate hook (#8144)	2024-01-02 10:15:12 +02:00
SecretsHelpers.ts	feat: External Secrets storage for credentials (#6477)	2023-08-25 10:33:46 +02:00
Server.ts	fix(core): Minor improvements to multi-main setup (no-changelog) (#8012)	2023-12-27 16:55:01 +01:00
sse-channel.d.ts	feat(core): Add support for WebSockets as an alternative to Server-Sent Events (#5443)	2023-02-10 15:02:47 +01:00
TestWebhooks.ts	feat(core): Cache test webhook registrations (#8176)	2024-01-03 16:58:33 +01:00
utils.ts	refactor(core): Simplify state in test webhooks (no-changelog) (#8155)	2023-12-28 09:28:12 +01:00
WaitingForms.ts	feat(n8n Form Trigger Node): Improvements (#7571)	2023-12-13 17:00:51 +02:00
WaitingWebhooks.ts	feat(n8n Form Trigger Node): Improvements (#7571)	2023-12-13 17:00:51 +02:00
WaitTracker.ts	refactor(core): Move typeorm operators from WaitTracker to ExecutionRepository (no-changelog) (#8163)	2023-12-28 19:22:09 +01:00
WebhookHelpers.ts	fix(core): Use pinned data only for manual mode (#8164)	2023-12-27 18:22:33 +01:00
WebhookServer.ts	feat(core): Unify application components shutdown (#8097)	2023-12-22 12:39:58 +02:00
WorkflowExecuteAdditionalData.ts	refactor(core): Move all base URLs to UrlService (no-changelog) (#8141)	2023-12-22 15:19:50 +01:00
WorkflowHelpers.ts	refactor(core): Move typeorm operators from various sources into repositories (no-changelog) (#8174)	2023-12-28 13:14:10 +01:00
WorkflowRunner.ts	fix(core): Use pinned data only for manual mode (#8164)	2023-12-27 18:22:33 +01:00
WorkflowRunnerProcess.ts	ci: Stop disabling linting rules that are explicitly set to warn (no-changelog) (#7669)	2023-11-09 14:05:39 +01:00