Since we do not store which executions produced binary data, for pruning
on S3 we need to query for binary data items for each execution in order
to delete them. To minimize requests to S3, allow the user to skip
pruning requests when setting TTL at bucket level.
This change ensures that things like `encryptionKey` and `instanceId`
are always available directly where they are needed, instead of passing
them around throughout the code.
Depends on https://github.com/n8n-io/n8n/pull/7220 | Story:
[PAY-840](https://linear.app/n8n/issue/PAY-840/introduce-object-store-service-and-manager-for-binary-data)
This PR introduces an object store service for Enterprise edition. Note
that the service is tested but currently unused - it will be integrated
soon as a binary data manager, and later for execution data.
`amazonaws.com` in the host is temporarily hardcoded until we integrate
the service and test against AWS, Cloudflare and Backblaze, in the next
PR.
This is ready for review - the PR it depends on is approved and waiting
for CI.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Depends on: #7092 | Story:
[PAY-768](https://linear.app/n8n/issue/PAY-768)
This PR:
- Generalizes the `IBinaryDataManager` interface.
- Adjusts `Filesystem.ts` to satisfy the interface.
- Sets up an S3 client stub to be filled in in the next PR.
- Turns `BinaryDataManager` into an injectable service.
- Adjusts the config schema and adds new validators.
Note that the PR looks large but all the main changes are in
`packages/core/src/binaryData`.
Out of scope:
- `BinaryDataManager` (now `BinaryDataService`) and `Filesystem.ts` (now
`fs.client.ts`) were slightly refactored for maintainability, but fully
overhauling them is **not** the focus of this PR, which is meant to
clear the way for the S3 implementation. Future improvements for these
two should include setting up a backwards-compatible dir structure that
makes it easier to locate binary data files to delete, removing
duplication, simplifying cloning methods, using integers for binary data
size instead of `prettyBytes()`, writing tests for existing binary data
logic, etc.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Based on #7065 | Story: https://linear.app/n8n/issue/PAY-771
n8n on filesystem mode marks binary data to delete on manual execution
deletion, on unsaved execution completion, and on every execution
pruning cycle. We later prune binary data in a separate cycle via these
marker files, based on the configured TTL. In the context of introducing
an S3 client to manage binary data, the filesystem mode's mark-and-prune
setup is too tightly coupled to the general binary data management
client interface.
This PR...
- Ensures the deletion of an execution causes the deletion of any binary
data associated to it. This does away with the need for binary data TTL
and simplifies the filesystem mode's mark-and-prune setup.
- Refactors all execution deletions (including pruning) to cause soft
deletions, hard-deletes soft-deleted executions based on the existing
pruning config, and adjusts execution endpoints to filter out
soft-deleted executions. This reduces DB load, and keeps binary data
around long enough for users to access it when building workflows with
unsaved executions.
- Moves all execution pruning work from an execution lifecycle hook to
`execution.repository.ts`. This keeps related logic in a single place.
- Removes all marking logic from the binary data manager. This
simplifies the interface that the S3 client will meet.
- Adds basic sanity-check tests to pruning logic and execution deletion.
Out of scope:
- Improving existing pruning logic.
- Improving existing execution repository logic.
- Adjusting dir structure for filesystem mode.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Github issue / Community forum post (link here to close automatically):
https://github.com/n8n-io/n8n/pull/6348
---------
Co-authored-by: Giulio Andreini <g.andreini@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
- For a saved execution, we write to disk binary data and metadata.
These two are only ever deleted via `POST /executions/delete`. No marker
file, so untouched by pruning.
- For an unsaved execution, we write to disk binary data, binary data
metadata, and a marker file at `/meta`. We later delete all three during
pruning.
- The third flow is legacy. Currently, if the execution is unsaved, we
actually store it in the DB while running the workflow and immediately
after the workflow is finished during the `onWorkflowPostExecute()` hook
we delete that execution, so the second flow applies. But formerly, we
did not store unsaved executions in the DB ("ephemeral executions") and
so we needed to write a marker file at `/persistMeta` so that, if the
ephemeral execution crashed after the step where binary data was stored,
we had a way to later delete its associated dangling binary data via a
second pruning cycle, and if the ephemeral execution succeeded, then we
immediately cleaned up the marker file at `/persistMeta` during the
`onWorkflowPostExecute()` hook.
This creation and cleanup at `/persistMeta` is still happening, but this
third flow no longer has a purpose, as we now store unsaved executions
in the DB and delete them immediately after. Hence the third flow can be
removed.
* fix(core): Make node execution order configurable, and backward-compatible
* ⚡ Also add new Merge-Node behaviour
* ⚡ Fix typo
* Fix lint issue
* update labels
* rename legacy to v0
* remove the unnecessary log
* default all new workflows to use v1 execution-order
* remove the controller changes
* clone default settings to avoid it getting modified
---------
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
* feat(core): Change node execution order (most top-left one first)
* ⚡ Fix issue with multi-output-nodes
* ⚡ Remove not needed meta-entry in test
* fix the e2e test
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* 🔧 Adjust base ESLint config
* 🔧 Adjust `lint` and `lintfix` in `nodes-base`
* 🔧 Include `test` and `utils` in `nodes-base`
* 📘 Convert JS tests to TS
* 👕 Apply lintfixes
* fix(core): Validate customData keys and values
Throws errors in manual mode and ignores and logs values in production
* fix: validate customData key characters
* refactor: review changes
* fix: logger not initialised for metadata tests
* fix: allow numbers for values
* wip: workflow execution filtering
* fix: import type failing to build
* fix: remove console.logs
* feat: execution metadata migrations
* fix(editor): Move global executions filter to its own component
* fix(editor): Using the same filter component in workflow level
* fix(editor): a small housekeeping
* checking workflowId in filter applied
* fix(editor): update filter after resolving merge conflicts
* fix(editor): unify empy filter status
* feat(editor): add datetime picker to filter
* feat(editor): add meta fields
* fix: fix button override in datepicker panel
* feat(editor): add filter metadata
* feat(core): add 'startedBefore' execution filter prop
* feat(core): add 'tags' execution query filter
* Revert "feat(core): add 'tags' execution query filter"
This reverts commit a7b968081c.
* feat(editor): add translations and tooltip and counting selected filter props
* fix(editor): fix label layouts
* fix(editor): update custom data docs link
* fix(editor): update custom data tooltip position
* fix(editor): update tooltip text
* refactor: Ignore metadata if not enabled by license
* fix(editor): Add paywall states to advanced execution filter
* refactor: Save custom data also for worker mode
* fix: Remove duplicate migration name from list
* fix(editor): Reducing filter complexity and add debounce to text inputs
* fix(editor): Remove unused import, add comment
* fix(editor): simplify event listener
* fix: Prevent error when there are running executions
* test(editor): Add advanced execution filter basic unit test
* test(editor): Add advanced execution filter state change unit test
* fix: Small lint issue
* feat: Add indices to speed up queries
* feat: add customData limits
* refactor: put metadata save in transaction
* chore: remove unneed comment
* test: add tests for execution metadata
* fix(editor): Fixes after merge conflict
* fix(editor): Remove unused import
* wordings and ui fixes
* fix(editor): type fixes
* feat: add code node autocompletions for customData
* fix: Prevent transaction issues and ambiguous ID in sql clauses
* fix(editor): Suppress requesting current executions if metadata is used in filter (#5739)
* fix(editor): Suppress requesting current executions if metadata is used in filter
* fix(editor): Fix arrows for select in popover
* refactor: Improve performance by correcting database indices
* fix: Lint issue
* test: Fix broken test
* fix: Broken test
* test: add call data check for saveExecutionMetadata test
---------
Co-authored-by: Valya Bullions <valya@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
* 📘 Clear all `@ts-ignore` comments from workflow package
* 👕 Default to error with package-level overrides
* refactor(core): clear all `@ts-ignore` comments from core package (#4473)
👕 Clear all `@ts-ignore` comments from core package
* ✏️ Update comment
* Improve code health
Fix TS typos in local variables
Fix CSS typos in local styles
Fix typos in comments
Fix typos in strings
* Fix order of n8n setup sections in CONTRIBUTING.md
* ✨ Introduce simple node versioning
* ⚡ Add example how to read version in node-code for custom logic
* 🐛 Fix setting of parameters
* 🐛 Fix another instance where it sets the wrong parameter
* ⚡ Remove unnecessary TOODs
* ⚡ Revert Set Node example changes
* ;rotating_light: Add test
* ✨ change FE to handle new object type
* 🚸 improve UX of handling invalid credentials
* 🚧 WIP
* 🎨 fix typescript issues
* 🐘 add migrations for all supported dbs
* ✏️ add description to migrations
* ⚡ add credential update on import
* ⚡ resolve after merge issues
* 👕 fix lint issues
* ⚡ check credentials on workflow create/update
* update interface
* 👕 fix ts issues
* ⚡ adaption to new credentials UI
* 🐛 intialize cache on BE for credentials check
* 🐛 fix undefined oldCredentials
* 🐛 fix deleting credential
* 🐛 fix check for undefined keys
* 🐛 fix disabling edit in execution
* 🎨 just show credential name on execution view
* ✏️ remove TODO
* ⚡ implement review suggestions
* ⚡ add cache to getCredentialsByType
* ⏪ use getter instead of cache
* ✏️ fix variable name typo
* 🐘 include waiting nodes to migrations
* 🐛 fix reverting migrations command
* ⚡ update typeorm command
* ✨ create db:revert command
* 👕 fix lint error
* ✨ Add optional authenticate method to credentials
* ⚡ Simplify code and add authentication support to MattermostApi
* 👕 Fix lint issue
* ⚡ Add support to own-mode
* 👕 Fix lint issue
* ✨ Add support for predefined auth types bearer and headerAuth
* ⚡ Make sure that DateTime Node always returns strings
* ⚡ Add support for moment types to If Node
* ⚡ Make it possible for HTTP Request Node to use all credential types
* ✨ Add basicAuth support
* Add a new dropcontact node
* ✨ First basic implementation of mainly JSON based nodes
* ✨ Add fixedCollection support, added value parameter and
expression support for value and property
* Improvements to #2389
* ⚡ Add credentials verification
* ⚡ Small improvement
* ⚡ set default time to 45 seconds
* ✨ Add support for preSend and postReceive methods
* ➕ Add lodash merge and set depedency to workflow
* 👕 Fix lint issue
* ⚡ Improvements
* ⚡ Improvements
* ⚡ Improvements
* ⚡ Improvements
* ⚡ Improvements
* 🐛 Set siren and language correctly
* ⚡ Add support for requestDefaults
* ⚡ Add support for baseURL to httpRequest
* ⚡ Move baseURL to correct location
* ✨ Add support for options loading
* 🐛 Fix error with fullAccess nodes
* ✨ Add credential test functionality
* 🐛 Fix issue with OAuth autentication and lint issue
* ⚡ Fix build issue
* 🐛 Fix issue that url got always overwritten to empty
* ✨ Add pagination support
* ⚡ Code fix required after merge
* ⚡ Remove not needed imports
* ⚡ Fix credential test
* ✨ Add expression support for request properties and $self
support on properties
* ⚡ Rename $self to $value
* 👕 Fix lint issue
* ⚡ Add example how to send data in path
* ✨ Make it possible to not sent in dot notation
* ✨ Add support for postReceive:rootProperty
* ⚡ Fix typo
* ✨ Add support for postReceive:set
* ⚡ Some fixes
* ⚡ Small improvement
* ;zap: Separate RoutingNode code
* ⚡ Simplify code and fix bug
* ⚡ Remove unused code
* ✨ Make it possible to define "request" and "requestProperty" on
options
* 👕 Fix lint issue
* ⚡ Change $credentials variables name
* ✨ Enable expressions and access to credentials in requestDefaults
* ⚡ Make parameter option loading use RoutingNode.makeRoutingRequest
* ✨ Allow requestOperations overwrite on LoadOptions
* ✨ Make it possible to access current node parameters in loadOptions
* ⚡ Rename parameters variable to make future proof
* ⚡ Make it possible to use offset-pagination with body
* ✨ Add support for queryAuth
* ⚡ Never return more items than requested
* ✨ Make it possible to overwrite requestOperations on parameter
and option level
* 👕 Fix lint issue
* ✨ Allow simplified auth also with regular nodes
* ✨ Add support for receiving binary data
* 🐛 Fix example node
* ⚡ Rename property "name" to "displayName" in loadOptions
* ⚡ Send data by default as "query" if nothing is set
* ⚡ Rename $self to $parent
* ⚡ Change to work with INodeExecutionData instead of IDataObject
* ⚡ Improve binaryData handling
* ⚡ Property design improvements
* ⚡ Fix property name
* 🚨 Add some tests
* ⚡ Add also test for request
* ⚡ Improve test and fix issues
* ⚡ Improvements to loadOptions
* ⚡ Normalize loadOptions with rest of code
* ⚡ Add info text
* ✨ Add support for $value in postReceive
* 🚨 Add tests for RoutingNode.runNode
* ⚡ Remove TODOs and make url property optional
* ⚡ Fix bug and lint issue
* 🐛 Fix bug that not the correct property got used
* 🚨 Add tests for CredentialsHelper.authenticate
* ⚡ Improve code and resolve expressions also everywhere for
loadOptions and credential test requests
* ✨ Make it possible to define multiple preSend and postReceive
actions
* ✨ Allow to define tests on credentials
* ⚡ Remove test data
* ⬆️ Update package-lock.json file
* ⚡ Remove old not longer used code
Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com>
Co-authored-by: Mutasem <mutdmour@gmail.com>
Co-authored-by: PaulineDropcontact <pauline@dropcontact.io>
Co-authored-by: ricardo <ricardoespinoza105@gmail.com>
* ✨ Add Webhook-Response-Node
* ⚡ Replace callback function with promise
* ✨ Add support for Bull and binary-data
* ✨ Add string response option
* ⚡ Remove some comments
* ✨ Make more generically possible & fix issue multi call in
queue mode
* ⚡ Fix startup and eslint issues
* ⚡ Improvements to webhook response node and functionality
* ⚡ Replace data with more generic type
* ⚡ Make statusMessage optional
* ⚡ Change parameter order
* ⚡ Move Response Code underneath options
* ⚡ Hide Response Code on Webhook node if mode responseNode got selected
* ⚡ Minor improvements
* ⚡ Add missing file and fix lint issue
* ⚡ Fix some node linting issues
* ⚡ Apply feedback
* ⚡ Minor improvements
