Story: https://linear.app/n8n/issue/PAY-1188
- Implement Redis hashes on the caching service, based on Micha's work
in #7747, adapted from `node-cache-manager-ioredis-yet`. Optimize
workflow ownership lookups and manual webhook lookups with Redis hashes.
- Simplify the caching service by removing all currently unused methods
and options: `enable`, `disable`, `getCache`, `keys`, `keyValues`,
`refreshFunctionEach`, `refreshFunctionMany`, `refreshTtl`, etc.
- Remove the flag `N8N_CACHE_ENABLED`. Currently some features on
`master` are broken with caching disabled, and test webhooks now rely
entirely on caching, for multi-main setup support. We originally
introduced this flag to protect against excessive memory usage, but
total cache usage is low enough that we decided to drop this setting.
Apparently this flag was also never documented.
- Overall caching service refactor: use generics, reduce branching, add
discriminants for cache kinds for better type safety, type caching
events, improve readability, remove outdated docs, etc. Also refactor
and expand caching service tests.
Follow-up to: https://github.com/n8n-io/n8n/pull/8176
---------
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
In a multi-main setup, we have the following issue. The user's client
connects to main A and runs a test webhook, so main A starts listening
for a webhook call. A third-party service sends a request to the test
webhook URL. The request is forwarded by the load balancer to main B,
who is not listening for this webhook call. Therefore, the webhook call
is unhandled.
To start addressing this, cache test webhook registrations, using Redis
for queue mode and in-memory for regular mode. When the third-party
service sends a request to the test webhook URL, the request is
forwarded by the load balancer to main B, who fetches test webhooks from
the cache and, if it finds a match, executes the test webhook. This
should be transparent - test webhook behavior should remain the same as
so far.
Notes:
- Test webhook timeouts are not cached. A timeout is only relevant to
the process it was created in, so another process retrieving from Redis
a "foreign" timeout will be unable to act on it. A timeout also has
circular references, so `cache-manager-ioredis-yet` is unable to
serialize it.
- In a single-main scenario, the timeout remains in the single process
and is cleared on test webhook expiration, successful execution, and
manual cancellation - all as usual.
- In a multi-main scenario, we will need to have the process who
received the webhook call send a message to the process who created the
webhook directing this originating process to clear the timeout. This
will likely be implemented via execution lifecycle hooks and Redis
channel messages checking session ID. This implementation is out of
scope for this PR and will come next.
- Additional data in test webhooks is not cached. From what I can tell,
additional data is not needed for test webhooks to be executed.
Additional data also has circular references, so
`cache-manager-ioredis-yet` is unable to serialize it.
Follow-up to: #8155
This PR simplifies state in test webhooks so that it can be cached
easily. Caching this state will allow us to start using Redis for manual
webhooks, to support manual webhooks to work in multi-main setup.
- [x] Convert `workflowWebhooks` to a getter - no need to optimize for
deactivation
- [x] Remove array from value in `TestWebhooks.webhookUrls`
- [x] Consolidate `webhookUrls` and `registeredWebhooks`
Remove duplication, improve readability, and expand tests for
`TestWebhooks.ts` - in anticipation for storing test webhooks in Redis.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Ensure all errors in `cli` are `ApplicationError` or children of it and
contain no variables in the message, to continue normalizing all the
errors we report to Sentry
Follow-up to: https://github.com/n8n-io/n8n/pull/7839
Ensure all errors in `cli` inherit from `ApplicationError` to continue
normalizing all the errors we report to Sentry
Follow-up to: https://github.com/n8n-io/n8n/pull/7820
Also,
1. Consistent CORS support ~on all three webhook types~ waiting webhooks never supported CORS. I'll fix that in another PR
2. [Fixes binary-data handling when request body is text, json, or xml](https://linear.app/n8n/issue/NODE-505/webhook-binary-data-handling-fails-for-textplain-files).
3. Reduced number of middleware that each request has to go through.
4. Removed the need to maintain webhook endpoints in the auth-exception list.
5. Skip all middlewares (apart from `compression`) on Webhook routes.
6. move `multipart/form-data` support out of individual nodes
7. upgrade `formidable`
8. fix the filenames on binary-data in webhooks nodes
9. add unit tests and integration tests for webhook request handling, and increase test coverage
* add typedi
* convert ActiveWorkflowRunner into an injectable service
* convert ExternalHooks into an injectable service
* convert InternalHooks into an injectable service
* convert LoadNodesAndCredentials into an injectable service
* convert NodeTypes and CredentialTypes into an injectable service
* convert ActiveExecutions into an injectable service
* convert WaitTracker into an injectable service
* convert Push into an injectable service
* convert ActiveWebhooks and TestWebhooks into an injectable services
* handle circular references, and log errors when a circular dependency is found
* ✨ Create rule `no-unneeded-backticks`
* 👕 Enable rule
* ⚡ Run rule on `cli`
* ⚡ Run rule on `core`
* ⚡ Run rule on `workflow`
* ⚡ Rule rule on `design-system`
* ⚡ Run rule on `node-dev`
* ⚡ Run rule on `editor-ui`
* ⚡ Run rule on `nodes-base`
* Improve code health
Fix TS typos in local variables
Fix CSS typos in local styles
Fix typos in comments
Fix typos in strings
* Fix order of n8n setup sections in CONTRIBUTING.md
* refactor saving
* refactor api layer to be stateless
* refactor header details
* set variable for menu height
* clean up scss
* clean up indentation
* clean up dropdown impl
* refactor no tags view
* split away header
* Fix tslint issues
* Refactor tag manager
* add tags to patch request
* clean up scss
* ⚡ Refactor types to entities
* fix issues
* update no workflow error
* clean up tagscontainer
* use getters instead of state
* remove imports
* use custom colors
* clean up tags container
* clean up dropdown
* clean up focusoncreate
* ⚡ Ignore mistaken ID in POST /workflows
* ⚡ Fix undefined tag ID in PATCH /workflows
* ⚡ Shorten response for POST /tags
* remove scss mixins
* clean up imports
* ⚡ Implement validation with class-validator
* address ivan's comments
* implement modals
* Fix lint issues
* fix disabling shortcuts
* fix focus issues
* fix focus issues
* fix focus issues with modal
* fix linting issues
* use dispatch
* use constants for modal keys
* fix focus
* fix lint issues
* remove unused prop
* add modal root
* fix lint issues
* remove unused methods
* fix shortcut
* remove max width
* ⚡ Fix duplicate entry error for pg and MySQL
* update rename messaging
* update order of buttons
* fix firefox overflow on windows
* fix dropdown height
* 🔨 refactor tag crud controllers
* 🧹 remove unused imports
* use variable for number of items
* fix dropdown spacing
* ⚡ Restore type to fix build
* ⚡ Fix post-refactor PATCH /workflows/:id
* ⚡ Fix PATCH /workflows/:id for zero tags
* ⚡ Fix usage count becoming stringified
* address max's comments
* fix filter spacing
* fix blur bug
* address most of ivan's comments
* address tags type concern
* remove defaults
* ⚡ return tag id as string
* 🔨 add hooks to tag CUD operations
* 🏎 simplify timestamp pruning
* remove blur event
* fix onblur bug
* ⚡ Fix fs import to fix build
* address max's comments
* implement responsive tag container
* fix lint issues
* update tag limits
* address ivan's comments
* remove rename, refactor header, implement new designs for save, remove responsive tag container
* update styling
* update styling
* implement responsive tag container
* implement header tags edit
* implement header tags edit
* fix lint issues
* implement expandable input
* minor fixes
* minor fixes
* use variable
* rename save as
* duplicate fixes
* minor edit fixes
* lint fixes
* style fixes
* hook up saving name
* hook up tags
* clean up impl
* fix dirty state bug
* update limit
* update notification messages
* on click outside
* fix minor bug with count
* lint fixes
* handle minor edge cases
* handle minor edge cases
* handle minor bugs; fix firefox dropdown issue
* Fix min width
* apply tags only after api success
* remove count fix
* clean up workflow tags impl, fix tags delete bug
* fix minor issue
* fix minor spacing issue
* disable wrap for ops
* fix viewport root; save on click in dropdown
* save button loading when saving name/tags
* implement max width on tags container
* implement cleaner create experience
* disable edit while updating
* codacy hex color
* refactor tags container
* fix clickability
* fix workflow open and count
* clean up structure
* fix up lint issues
* fix button size
* increase workflow name limit for larger screen
* tslint fixes
* disable responsiveness for workflow modal
* rename event
* change min width for tags
* clean up pr
* address max's comments on styles
* remove success toasts
* add hover mode to name
* minor fixes
* refactor name preview
* fix name input not to jiggle
* finish up name input
* Fix up add tags
* clean up param
* clean up scss
* fix resizing name
* fix resizing name
* fix resize bug
* clean up edit spacing
* ignore on esc
* fix input bug
* focus input on clear
* build
* fix up add tags clickablity
* remove scrollbars
* move into folders
* clean up multiple patch req
* remove padding top from edit
* update tags on enter
* build
* rollout blur on enter behavior
* rollout esc behavior
* fix tags bug when duplicating tags
* move key to reload tags
* update header spacing
* build
* update hex case
* refactor workflow title
* remove unusued prop
* keep focus on error, fix bug on error
* Fix bug with name / tags toggle on error
* fix connection push bug
* :spakles: Implement wait functionality
* 🐛 Do not delete waiting executions with prune
* ⚡ Improve SQLite migration to not lose execution data anymore
* ⚡ Make it possible to restart waiting execution via webhook
* ⚡ Add missing file
* 🐛 Some more merge fixes
* ⚡ Do not show error for Wait-Nodes if in time-mode
* ⚡ Make $executionId available in expressions
* 👕 Fix lint issue
* 👕 Fix lint issue
* 👕 Fix lint issue
* ⚡ Set the unlimited sleep time as a variable
* ⚡ Add also sleeping webhook path to config
* ⚡ Make it possible to retrieve restartUrl in workflow
* ⚡ Add authentication to Wait-Node in Webhook-Mode
* ⚡ Return 404 when trying to restart execution via webhook which does
not support it
* ✨ Make it possible to set absolute time on Wait-Node
* ⚡ Remove not needed imports
* ⚡ Fix description format
* ✨ Implement missing webhook features on Wait-Node
* ⚡ Display webhook variable in NodeWebhooks
* ⚡ Include also date in displayed sleep time
* ⚡ Make it possible to see sleep time on node
* ⚡ Make sure that no executions does get executed twice
* ⚡ Add comment
* ⚡ Further improvements
* ⚡ Make Wait-Node easier to use
* ✨ Add support for "notice" parameter type
* Fixing wait node to work with queue, improved logging and execution view
* Added support for mysql and pg
* ✨ Add support for webhook postfix path
* ✨ Make it possible to stop sleeping executions
* ⚡ Fix issue with webhook paths in not webhook mode
* ⚡ Remove not needed console.log
* ⚡ Update TODOs
* ⚡ Increase min time of workflow staying active to descrease possible issue
with overlap
* 👕 Fix lint issue
* 🐛 Fix issues with webhooks
* ⚡ Make error message clearer
* ⚡ Fix issue with missing execution ID in scaling mode
* Fixed execution list to correctly display waiting executins
* Feature: enable webhook wait workflows to continue after specified time
* Fixed linting
* ⚡ Improve waiting description text
* ⚡ Fix parameter display issue and rename
* ⚡ Remove comment
* ⚡ Do not display webhooks on Wait-Node
* Changed wording from restart to resume on wait node
* Fixed wording and inconsistent screen when changing resume modes
* Removed dots from the descriptions
* Changed docs url and renaming postfix to suffix
* Changed names from sleep to wait
* ⚡ Apply suggestions from ben
Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com>
* Some fixes by Ben
* ⚡ Remove console.logs
* ⚡ Fixes and improvements
Co-authored-by: Mutasem <mutdmour@gmail.com>
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com>
Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com>
Co-authored-by: Omar Ajoue <krynble@gmail.com>
* ✨ n8n start trigger node
* first declaration of WorkflowActivationMode
* implement first WorkflowActivationMode: 'init', 'create', 'update', 'activate'
* fix Server missing id
* add activation infos to triggers
* remove WorkflowActivationMode from webhook execution function
* add some missing activation and add manual activation
* clean up and fix some code
* fix UnhandledPromiseRejectionWarning: Error: Overwrite NodeExecuteFunctions.getExecuteTriggerFunctions.emit function!
* fix spaces
* use a better name for the node
* fix ident in package.json
* Contributions to lublak's PR #1287
* Fixed linting issues and change the way parameters are displayed
* ⚡ Fix name and minor improvements
Co-authored-by: lublak <lublak.de@gmail.com>
Co-authored-by: lublak <44057030+lublak@users.noreply.github.com>
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
* 🧹 clean up forgotten leftover
* 🎨 reset req.params
* 🐛 Bugfix and minor change on request parameters
* ⚡ Minor improvements on request parameters
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
* 🐛 Fix naming of events in AffinityTrigger Node
* 🚧 OPTIONS allow header response for production webhooks
* ✅ Implemented Allow header for test webhook OPTIONS response
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
* In packages/cli/src/Db.ts, conditional test of dbType replaced by a
switch;
* removeAll() function adapted to not cause an error using MySQL;
* Added the cross-env module in the "serve" script in the
packages/editor-ui/package.json file. This was done to ensure
compatibility between platforms when declaring environment variables.
Without it, Windows compilation would give an error, for example;
* .idea added to .gitignore (IntelliJ IDEA solutions);
