Depends on: #7092 | Story:
[PAY-768](https://linear.app/n8n/issue/PAY-768)
This PR:
- Generalizes the `IBinaryDataManager` interface.
- Adjusts `Filesystem.ts` to satisfy the interface.
- Sets up an S3 client stub to be filled in in the next PR.
- Turns `BinaryDataManager` into an injectable service.
- Adjusts the config schema and adds new validators.
Note that the PR looks large but all the main changes are in
`packages/core/src/binaryData`.
Out of scope:
- `BinaryDataManager` (now `BinaryDataService`) and `Filesystem.ts` (now
`fs.client.ts`) were slightly refactored for maintainability, but fully
overhauling them is **not** the focus of this PR, which is meant to
clear the way for the S3 implementation. Future improvements for these
two should include setting up a backwards-compatible dir structure that
makes it easier to locate binary data files to delete, removing
duplication, simplifying cloning methods, using integers for binary data
size instead of `prettyBytes()`, writing tests for existing binary data
logic, etc.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Based on #7065 | Story: https://linear.app/n8n/issue/PAY-771
n8n on filesystem mode marks binary data to delete on manual execution
deletion, on unsaved execution completion, and on every execution
pruning cycle. We later prune binary data in a separate cycle via these
marker files, based on the configured TTL. In the context of introducing
an S3 client to manage binary data, the filesystem mode's mark-and-prune
setup is too tightly coupled to the general binary data management
client interface.
This PR...
- Ensures the deletion of an execution causes the deletion of any binary
data associated to it. This does away with the need for binary data TTL
and simplifies the filesystem mode's mark-and-prune setup.
- Refactors all execution deletions (including pruning) to cause soft
deletions, hard-deletes soft-deleted executions based on the existing
pruning config, and adjusts execution endpoints to filter out
soft-deleted executions. This reduces DB load, and keeps binary data
around long enough for users to access it when building workflows with
unsaved executions.
- Moves all execution pruning work from an execution lifecycle hook to
`execution.repository.ts`. This keeps related logic in a single place.
- Removes all marking logic from the binary data manager. This
simplifies the interface that the S3 client will meet.
- Adds basic sanity-check tests to pruning logic and execution deletion.
Out of scope:
- Improving existing pruning logic.
- Improving existing execution repository logic.
- Adjusting dir structure for filesystem mode.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
- For a saved execution, we write to disk binary data and metadata.
These two are only ever deleted via `POST /executions/delete`. No marker
file, so untouched by pruning.
- For an unsaved execution, we write to disk binary data, binary data
metadata, and a marker file at `/meta`. We later delete all three during
pruning.
- The third flow is legacy. Currently, if the execution is unsaved, we
actually store it in the DB while running the workflow and immediately
after the workflow is finished during the `onWorkflowPostExecute()` hook
we delete that execution, so the second flow applies. But formerly, we
did not store unsaved executions in the DB ("ephemeral executions") and
so we needed to write a marker file at `/persistMeta` so that, if the
ephemeral execution crashed after the step where binary data was stored,
we had a way to later delete its associated dangling binary data via a
second pruning cycle, and if the ephemeral execution succeeded, then we
immediately cleaned up the marker file at `/persistMeta` during the
`onWorkflowPostExecute()` hook.
This creation and cleanup at `/persistMeta` is still happening, but this
third flow no longer has a purpose, as we now store unsaved executions
in the DB and delete them immediately after. Hence the third flow can be
removed.
Also,
1. Consistent CORS support ~on all three webhook types~ waiting webhooks never supported CORS. I'll fix that in another PR
2. [Fixes binary-data handling when request body is text, json, or xml](https://linear.app/n8n/issue/NODE-505/webhook-binary-data-handling-fails-for-textplain-files).
3. Reduced number of middleware that each request has to go through.
4. Removed the need to maintain webhook endpoints in the auth-exception list.
5. Skip all middlewares (apart from `compression`) on Webhook routes.
6. move `multipart/form-data` support out of individual nodes
7. upgrade `formidable`
8. fix the filenames on binary-data in webhooks nodes
9. add unit tests and integration tests for webhook request handling, and increase test coverage
* fix(core): Make node execution order configurable, and backward-compatible
* ⚡ Also add new Merge-Node behaviour
* ⚡ Fix typo
* Fix lint issue
* update labels
* rename legacy to v0
* remove the unnecessary log
* default all new workflows to use v1 execution-order
* remove the controller changes
* clone default settings to avoid it getting modified
---------
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
* First node set up.
* Progress: all Resources and Operations updated
* Upsates to all resources.
* Updated tooltip for Tweet > Search > Tweet fields.
* Upodate to resource locator items in User > Search.
* Added e.g. to placeholders and minor copy tweaks.
* Fixed Operations sorting.
* Added a couple of operations back.
* Removed 'Authorized API Call'.
* Remove authorization header when empty
* Import pkce
* Add OAuth2 with new grant type to Twitter
* Add pkce logic auto assign authorization code if pkce not defined
* Add pkce to ui and interfaces
* Fix scopes for Oauth2 twitter
* Deubg + pass it through header
* Add debug console, add airtable cred
* Remove all console.logs, make PKCE in th body only when it exists
* Remove invalid character ~
* Remove more console.logs
* remove body inside query
* Remove useless grantype check
* Hide oauth2 twitter waiting for overhaul
* Remove redundant header removal
* Remove more console.logs
* Add V2 twitter
* Add V1
* Fix description V1, V2
* Fix description for V1
* Add Oauth2 request
* Add user lookup
* Add search username by ID
* Search tweet feat
* Wip create tweet
* Generic function for returning ID
* Add like and retweet feat
* Add delete tweet feat
* Fix Location tweets
* Fix type
* Feat List add members
* Add scopes for dm and list
* Add direct message feature
* Improve response data
* Fix regex
* Add unit test to Twitter v2
* Fix unit test
* Remove console.logs
* Remove more console.logs
* Handle @ in username
* Minor copy tweaks.
* Add return all logic
* Add error for api permission error
* Update message api error
* Add error for date error
* Add notice for TwitterOAuth2 api link
* Fix display names location
* fix List RLC
* Fix like endpoint
* Fix error message check
* fix(core): Fix OAuth2 callback for grantType=clientCredentials
* Improve fix for callback
* update pnpm
* Fix iso time for end time
* sync oauth2Credential
* remove unused codeVerifer in Server.ts
* Add location and attachments notice
* Add notice to oauth1
* Improve notice for twitter
* moved credentials notice to TwitterOAuth1Api.credentials.ts
---------
Co-authored-by: agobrech <ael.gobrecht@gmail.com>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Co-authored-by: Marcus <marcus@n8n.io>
* Setup versionized node
* Fix node naming
* Set all possible actions
* Add Current Date operation
* Add timezone to current date
* feat add to date operator
* Change output field name to camel case
* Fix info box for luxons tip
* Feat subtract to date operation
* Feat format date operation
* Fix to node field for format date
* Feat rounding operation
* Feat get in between date operation
* Feat add extract date operation
* Add generic function for parsing date
* Remove moment methods from operations
* Change moment to luxon for the rest of the operations
* Fix Format date operation
* Fix format value
* Add timezone option for current date
* Add tests, improve workflow settings for testing, toString the results
* Change icon for V2
* Revert "Change icon for V2"
This reverts commit 46b59bea2e.
* Change workflow test name
* Fix ui bug for custom format
* Fix default value for format operation
* Fix info box for rounding operation
* Change default for units for between time operation
* Inprove fields and resort time units
* Fix extract week number
* Resolve issue with formating and timezones
* Fix field name and unit order
* ⚡ restored removed test case, sync v1 with curent master
* ⚡ parseDate update to support timestamps, tests
* Keep same field for substract and add time
* Update unit test
* Improve visibility, add iso to string option
* Update option naming
---------
Co-authored-by: Michael Kret <michael.k@radency.com>
* 🔧 Adjust base ESLint config
* 🔧 Adjust `lint` and `lintfix` in `nodes-base`
* 🔧 Include `test` and `utils` in `nodes-base`
* 📘 Convert JS tests to TS
* 👕 Apply lintfixes
* ⚡️test setup
* ⚡️fix 'testData' implicitly has an 'any' type.
* ✅ test github action file binary data reading
* ✅ checking for output binary equality
* ✅ writing files to different formats
* ✅ reading spreadsheet with different options
* ⚡️improve workflow file path replacement
* 🐛 fixing string.at() not supported in node 14.
* 🐛 trying to fix github action test error
* ⚡ fix for empty binary
* ⚡ switch for binary test
* ⚡️test helpers now return/compare json and binary (if not empty))
* ⚡ removed commented console log
---------
Co-authored-by: Michael Kret <michael.k@radency.com>
* 🧪 Add base for building unit testing within nodes
* Improve helper functions
* 🧪 If node test
* 🧪 Airtable node test
* 🧪 If node test improvements
* 🧪 Airtable node test improvements
* ♻️ cleanup node unit tests
* ♻️ refactor getting node result data to use helper method
* ⚡ removed unused variables
* ♻️ Helper to read json files
---------
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Michael Kret <michael.k@radency.com>
* add multi return
* add independently and transaction to query
* pgInsert normal and transaction
* independently for pgInsert
* normal, transaction and independently for pgUpdate
* cleanup
* implement it in other nodes
* multiple fixes
* add optional returning support
* clean up Postgres functions
* fix other postgres based dbs
* Added option to run queries as a transaction to Postgres
This commit allows users to configure Postgres, CrateDB, TimescaleDB and
QuestDB to run queries independently or as transactions as well as the
previous mode which is to execute multiple queries at once.
Previous behavior remains untouched so we only added new options.
* Standardize behavior across nodes that use postgres protocol
Also fixed unit tests.
* Added breaking change notice
* Added more information to breaking changes
* ⚡ Styling fixes
Co-authored-by: lublak <lublak.de@gmail.com>
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
* 538 add support for casting types in postgres
* Add typing to postgres insert nodes and removed unnecessary field from crate db
* Added placeholder and description for types in postgres
* Adding tests to insert and changes suggested by Ben
* ⚡ Minor improvement
Co-authored-by: mutdmour <mutdmour@gmail.com>
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
