This change ensures that things like `encryptionKey` and `instanceId`
are always available directly where they are needed, instead of passing
them around throughout the code.
Depends on: #7092 | Story:
[PAY-768](https://linear.app/n8n/issue/PAY-768)
This PR:
- Generalizes the `IBinaryDataManager` interface.
- Adjusts `Filesystem.ts` to satisfy the interface.
- Sets up an S3 client stub to be filled in in the next PR.
- Turns `BinaryDataManager` into an injectable service.
- Adjusts the config schema and adds new validators.
Note that the PR looks large but all the main changes are in
`packages/core/src/binaryData`.
Out of scope:
- `BinaryDataManager` (now `BinaryDataService`) and `Filesystem.ts` (now
`fs.client.ts`) were slightly refactored for maintainability, but fully
overhauling them is **not** the focus of this PR, which is meant to
clear the way for the S3 implementation. Future improvements for these
two should include setting up a backwards-compatible dir structure that
makes it easier to locate binary data files to delete, removing
duplication, simplifying cloning methods, using integers for binary data
size instead of `prettyBytes()`, writing tests for existing binary data
logic, etc.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Based on #7065 | Story: https://linear.app/n8n/issue/PAY-771
n8n on filesystem mode marks binary data to delete on manual execution
deletion, on unsaved execution completion, and on every execution
pruning cycle. We later prune binary data in a separate cycle via these
marker files, based on the configured TTL. In the context of introducing
an S3 client to manage binary data, the filesystem mode's mark-and-prune
setup is too tightly coupled to the general binary data management
client interface.
This PR...
- Ensures the deletion of an execution causes the deletion of any binary
data associated to it. This does away with the need for binary data TTL
and simplifies the filesystem mode's mark-and-prune setup.
- Refactors all execution deletions (including pruning) to cause soft
deletions, hard-deletes soft-deleted executions based on the existing
pruning config, and adjusts execution endpoints to filter out
soft-deleted executions. This reduces DB load, and keeps binary data
around long enough for users to access it when building workflows with
unsaved executions.
- Moves all execution pruning work from an execution lifecycle hook to
`execution.repository.ts`. This keeps related logic in a single place.
- Removes all marking logic from the binary data manager. This
simplifies the interface that the S3 client will meet.
- Adds basic sanity-check tests to pruning logic and execution deletion.
Out of scope:
- Improving existing pruning logic.
- Improving existing execution repository logic.
- Adjusting dir structure for filesystem mode.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
- For a saved execution, we write to disk binary data and metadata.
These two are only ever deleted via `POST /executions/delete`. No marker
file, so untouched by pruning.
- For an unsaved execution, we write to disk binary data, binary data
metadata, and a marker file at `/meta`. We later delete all three during
pruning.
- The third flow is legacy. Currently, if the execution is unsaved, we
actually store it in the DB while running the workflow and immediately
after the workflow is finished during the `onWorkflowPostExecute()` hook
we delete that execution, so the second flow applies. But formerly, we
did not store unsaved executions in the DB ("ephemeral executions") and
so we needed to write a marker file at `/persistMeta` so that, if the
ephemeral execution crashed after the step where binary data was stored,
we had a way to later delete its associated dangling binary data via a
second pruning cycle, and if the ephemeral execution succeeded, then we
immediately cleaned up the marker file at `/persistMeta` during the
`onWorkflowPostExecute()` hook.
This creation and cleanup at `/persistMeta` is still happening, but this
third flow no longer has a purpose, as we now store unsaved executions
in the DB and delete them immediately after. Hence the third flow can be
removed.
fix(core): Reduce memory consumption on BinaryDataManager.init
When there are a few thousand binary data file to delete, the `deleteMarkedFiles` and `deleteMarkedPersistedFiles` methods need a lot of memory to process these files, irrespective of if these files have any data or not.
* use streams to upload files to google drive
* use streams to download files from google drive
* use resumable uploads api for google drive
* avoid dangling promises, and reduce memory usage in error logging
* upgrade ts-node
* move tslint and typescript to a single place
* source-map-support should be loaded in the `n8n` bin script, and not in core
* upgrade jest
* Support only node.js 14, 16, or 18
* ✨ change FE to handle new object type
* 🚸 improve UX of handling invalid credentials
* 🚧 WIP
* 🎨 fix typescript issues
* 🐘 add migrations for all supported dbs
* ✏️ add description to migrations
* ⚡ add credential update on import
* ⚡ resolve after merge issues
* 👕 fix lint issues
* ⚡ check credentials on workflow create/update
* update interface
* 👕 fix ts issues
* ⚡ adaption to new credentials UI
* 🐛 intialize cache on BE for credentials check
* 🐛 fix undefined oldCredentials
* 🐛 fix deleting credential
* 🐛 fix check for undefined keys
* 🐛 fix disabling edit in execution
* 🎨 just show credential name on execution view
* ✏️ remove TODO
* ⚡ implement review suggestions
* ⚡ add cache to getCredentialsByType
* ⏪ use getter instead of cache
* ✏️ fix variable name typo
* 🐘 include waiting nodes to migrations
* 🐛 fix reverting migrations command
* ⚡ update typeorm command
* ✨ create db:revert command
* 👕 fix lint error
* ✨ Add optional authenticate method to credentials
* ⚡ Simplify code and add authentication support to MattermostApi
* 👕 Fix lint issue
* ⚡ Add support to own-mode
* 👕 Fix lint issue
* ✨ Add support for predefined auth types bearer and headerAuth
* ⚡ Make sure that DateTime Node always returns strings
* ⚡ Add support for moment types to If Node
* ⚡ Make it possible for HTTP Request Node to use all credential types
* ✨ Add basicAuth support
* Add a new dropcontact node
* ✨ First basic implementation of mainly JSON based nodes
* ✨ Add fixedCollection support, added value parameter and
expression support for value and property
* Improvements to #2389
* ⚡ Add credentials verification
* ⚡ Small improvement
* ⚡ set default time to 45 seconds
* ✨ Add support for preSend and postReceive methods
* ➕ Add lodash merge and set depedency to workflow
* 👕 Fix lint issue
* ⚡ Improvements
* ⚡ Improvements
* ⚡ Improvements
* ⚡ Improvements
* ⚡ Improvements
* 🐛 Set siren and language correctly
* ⚡ Add support for requestDefaults
* ⚡ Add support for baseURL to httpRequest
* ⚡ Move baseURL to correct location
* ✨ Add support for options loading
* 🐛 Fix error with fullAccess nodes
* ✨ Add credential test functionality
* 🐛 Fix issue with OAuth autentication and lint issue
* ⚡ Fix build issue
* 🐛 Fix issue that url got always overwritten to empty
* ✨ Add pagination support
* ⚡ Code fix required after merge
* ⚡ Remove not needed imports
* ⚡ Fix credential test
* ✨ Add expression support for request properties and $self
support on properties
* ⚡ Rename $self to $value
* 👕 Fix lint issue
* ⚡ Add example how to send data in path
* ✨ Make it possible to not sent in dot notation
* ✨ Add support for postReceive:rootProperty
* ⚡ Fix typo
* ✨ Add support for postReceive:set
* ⚡ Some fixes
* ⚡ Small improvement
* ;zap: Separate RoutingNode code
* ⚡ Simplify code and fix bug
* ⚡ Remove unused code
* ✨ Make it possible to define "request" and "requestProperty" on
options
* 👕 Fix lint issue
* ⚡ Change $credentials variables name
* ✨ Enable expressions and access to credentials in requestDefaults
* ⚡ Make parameter option loading use RoutingNode.makeRoutingRequest
* ✨ Allow requestOperations overwrite on LoadOptions
* ✨ Make it possible to access current node parameters in loadOptions
* ⚡ Rename parameters variable to make future proof
* ⚡ Make it possible to use offset-pagination with body
* ✨ Add support for queryAuth
* ⚡ Never return more items than requested
* ✨ Make it possible to overwrite requestOperations on parameter
and option level
* 👕 Fix lint issue
* ✨ Allow simplified auth also with regular nodes
* ✨ Add support for receiving binary data
* 🐛 Fix example node
* ⚡ Rename property "name" to "displayName" in loadOptions
* ⚡ Send data by default as "query" if nothing is set
* ⚡ Rename $self to $parent
* ⚡ Change to work with INodeExecutionData instead of IDataObject
* ⚡ Improve binaryData handling
* ⚡ Property design improvements
* ⚡ Fix property name
* 🚨 Add some tests
* ⚡ Add also test for request
* ⚡ Improve test and fix issues
* ⚡ Improvements to loadOptions
* ⚡ Normalize loadOptions with rest of code
* ⚡ Add info text
* ✨ Add support for $value in postReceive
* 🚨 Add tests for RoutingNode.runNode
* ⚡ Remove TODOs and make url property optional
* ⚡ Fix bug and lint issue
* 🐛 Fix bug that not the correct property got used
* 🚨 Add tests for CredentialsHelper.authenticate
* ⚡ Improve code and resolve expressions also everywhere for
loadOptions and credential test requests
* ✨ Make it possible to define multiple preSend and postReceive
actions
* ✨ Allow to define tests on credentials
* ⚡ Remove test data
* ⬆️ Update package-lock.json file
* ⚡ Remove old not longer used code
Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com>
Co-authored-by: Mutasem <mutdmour@gmail.com>
Co-authored-by: PaulineDropcontact <pauline@dropcontact.io>
Co-authored-by: ricardo <ricardoespinoza105@gmail.com>
* ⚡ introduce versioned nodes
* Export versioned nodes for separate process run
* Add bse node for versioned nodes
* fix node name for versioned nodes
* extend node from nodeVersionedType
* improve nodes base and flow to FE
* revert lib es2019 to es2017
* include version in key to prevent duplicate key
* handle type versions on FE
* clean up
* cleanup nodes base
* add type versions in getNodeParameterOptions
* cleanup
* code review
* code review + add default version to node type description
* remove node default types from store
* 💄 cleanups
* Draft for migrated Mattermost node
* First version of Mattermost node versioned according to node standards
* Correcting deactivate operations name to match currently used one
* ✨ Create utility types
* ⚡ Simplify Mattermost types
* ⚡ Rename exports for consistency
* ⚡ Type channel properties
* ⚡ Type message properties
* ⚡ Type reaction properties
* ⚡ Type user properties
* ⚡ Add type import to router
* 🐛 Add missing key
* 🔨 Adjust typo in operation name
* 🔨 Inline exports for channel properties
* 🔨 Inline exports for message properties
* 🔨 Inline exports for reaction properties
* 🔨 Inline exports for user properties
* 🔨 Inline exports for load options
* 👕 Fix lint issue
* 🔨 Inline export for description
* 🔨 Rename descriptions for clarity
* 🔨 Refactor imports/exports for methods
* 🔨 Refactor latest version retrieval
* 🔥 Remove unneeded else clause
When the string literal union is exhausted, the resource key becomes never, so TS disallows wrong key usage.
* ✨ Add overloads to getNodeParameter
* ⚡ Improve overload
* 🔥 Remove superfluous INodeVersions type
* 🔨 Relocate pre-existing interface
* 🔥 Remove JSDoc arg descriptions
* ⚡ Minor reformatting in transport file
* ⚡ Fix API call function type
* Created first draft for Axios requests
* Working version of mattermost node with Axios
* Work in progress for replacing request library
* Improvements to request translations
* Fixed sending files via multipart / form-data
* Fixing translation from request to axios and loading node parameter options
* Improved typing for new http helper
* Added ignore any for specific lines for linting
* Fixed follow redirects changes on http request node and manual execution of previously existing workflow with older node versions
* Adding default headers according to body on httpRequest helper
* Spec error handling and fixed workflows with older node versions
* Showcase how to export errors in a standard format
* Merging master
* Refactored mattermost node to keep files in a uniform structure. Also fix bugs with merges
* Reverting changes to http request node
* Changed nullish comparison and removed repeated code from nodes
* Renamed queryString back to qs and simplified node output
* Simplified some comparisons
* Changed header names to be uc first
* Added default user agent to requests and patch http method support
* Fixed indentation, remove unnecessary file and console log
* Fixed mattermost node name
* Fixed lint issues
* Further fix linting issues
* Further fix lint issues
* Fixed http request helper's return type
Co-authored-by: ahsan-virani <ahsan.virani@gmail.com>
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* ⚡ Generalize unique entity name generation
* ⚡ Standardize variable names
* redo credentials
* revert some changes, replace got with was
* fix v-if order
* fix v-if order
* update linting
* update gulpfile
* update ssh display name
* update height
* update params
* update info tip sizes
* address design comments
* update google button disabled
* update icon size to 28px
* update design issues
* update info tab design
* address design comments
* update tab size
* update run data spacing
* address comments, update logo design
* fix spacing issues
* clean up store
* fix create new bug
* add loading state
* rename prop
* remove unused prop
* fix select bug
* remove label tag
* update word break
* build
* address design comments
* update font family of button
* update menu opacity
* update text
* update title
* address more comments
* update oauth messages
* add oauth validation
* hide disabled state
* update warning modal
* show button on text input
* clean up cred details
* add validation errors
* fix bug when deleting cred
* Frontend hack to display test button
* Created interfaces for testing and endpoint
* Testing slack node credentials working
* Adding test with node to endpoint for credential testing
* Fixed linting and test detectability
* Adding required for slack token
* Added google sheets credential testing
* update message
* Adding suggestions by Ivan and Mutasem
* Address comments
* keep blurred when focused
* update font weight of errors
* add oauth banner
* remove toast
* Fixed code bug and added telegram credential testing
* scroll to top on success
* clean up duplication
* Fixed telegram trigger node and added tests to typeform
* refactor modal
* add more validation support
* refactor info tab
* scroll to bottom on save, handle cred saving
* refactor save button
* save cred on valid
* save cred on valid
* scroll to top if has error
* add targets on input labels
* delete credentails input
* revert fe changes
* update validation logic
* clean interface
* test credentials
* update banner design
* show testing state
* update x position
* fix issues
* fix focus issues
* clean up validation behavior
* make error relative
* update banner component
* update error spacing
* don't close dialog
* rename button
* update how banners behave
* if has unsaved changes first
* move confirm message
* add success banner
* update time state
* disable transitions
* test on open
* clean up banner behavior
* update banner styling
* capitalize
* update error banner styling to handle long texts
* avoid unnessary content jostling
* add loading label
* show validation warnings when opening modal
* retest cred if not all props req
* update scroll to auto
* add error warning
* update color saturation
* set overflow to auto
* fix bug to get credentials when connected
* round down to minutes
* change tab name
* update casing oauth
* disable credential testing if it has expressions
* label same as title
* add more space between close and save
* remove check on making any changes
* hide close on confirm modals
* don't accept clicks outside dialog
* fix build issues
* undo test changes
* fix table scrollbar logs
* rename modals
* fix bug with same name
* refactor modal
* fix tslint issue
* refactor name
* update name behavior
* update monospace font
* remove comment
* refactor inputs
* refactor error handling
* reduce spacing changes
* fix doc url oauth1 oauth2
* build
* hide infotip if no inputs
* address most comments
* rename file
* fix menu alignment
* gst
* update types
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
Co-authored-by: Omar Ajoue <krynble@gmail.com>
* introduce binary data management
* cr
* add binary data changes to awsS3 node
* add binary data changes to Box node
* add binary data changes to CiscoWebex node
* add binary data changes to HumaniticAi node
* add binary data changes to Jira node
* add binary data changes to Line node
* add binary data changes to MicrosoftOneDrive node
* add binary data changes to MicrosoftOutlook node
* add binary data changes to Mindee node
* add binary data changes to NocoDB node
* add binary data changes to Pushbullet node
* add binary data changes to Pushover node
* add binary data changes to Raindrop node
* add binary data changes to S3 node
* add binary data changes to Salesforce node
* add binary data changes to Ssh node
* add binary data changes to TheHive node
* add binary data changes to Twist node
* add binary data changes to Twitter node
* remove changes not needed right now
* 🐛 Fix issue with multiple runs
* 🐛 Revert fix and add support for multiple inputs
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
