This change ensures that things like `encryptionKey` and `instanceId`
are always available directly where they are needed, instead of passing
them around throughout the code.
This adds support for
1. custom delimiters
2. reading offsets to avoid having to read a large CSV all at once
3. excluding byte-order-mark
NODE-861
#7443
Story: [PAY-846](https://linear.app/n8n/issue/PAY-846) | Related:
https://github.com/n8n-io/n8n/pull/7225
For the S3 backend for external storage of binary data and execution
data, the `getAsStream` method in the binary data manager interface used
by FS and S3 will need to become async. This is a breaking change for
nodes-base.
Depends on: #7092 | Story:
[PAY-768](https://linear.app/n8n/issue/PAY-768)
This PR:
- Generalizes the `IBinaryDataManager` interface.
- Adjusts `Filesystem.ts` to satisfy the interface.
- Sets up an S3 client stub to be filled in in the next PR.
- Turns `BinaryDataManager` into an injectable service.
- Adjusts the config schema and adds new validators.
Note that the PR looks large but all the main changes are in
`packages/core/src/binaryData`.
Out of scope:
- `BinaryDataManager` (now `BinaryDataService`) and `Filesystem.ts` (now
`fs.client.ts`) were slightly refactored for maintainability, but fully
overhauling them is **not** the focus of this PR, which is meant to
clear the way for the S3 implementation. Future improvements for these
two should include setting up a backwards-compatible dir structure that
makes it easier to locate binary data files to delete, removing
duplication, simplifying cloning methods, using integers for binary data
size instead of `prettyBytes()`, writing tests for existing binary data
logic, etc.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Github issue / Community forum post (link here to close automatically):
https://github.com/n8n-io/n8n/pull/6348
---------
Co-authored-by: Giulio Andreini <g.andreini@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
Also,
1. Consistent CORS support ~on all three webhook types~ waiting webhooks never supported CORS. I'll fix that in another PR
2. [Fixes binary-data handling when request body is text, json, or xml](https://linear.app/n8n/issue/NODE-505/webhook-binary-data-handling-fails-for-textplain-files).
3. Reduced number of middleware that each request has to go through.
4. Removed the need to maintain webhook endpoints in the auth-exception list.
5. Skip all middlewares (apart from `compression`) on Webhook routes.
6. move `multipart/form-data` support out of individual nodes
7. upgrade `formidable`
8. fix the filenames on binary-data in webhooks nodes
9. add unit tests and integration tests for webhook request handling, and increase test coverage
* fix(Postgres Node): For select queries, empty result should be be replaced with `{"success":true}`
* ⚡ less checks
---------
Co-authored-by: Michael Kret <michael.k@radency.com>
* fix(core): Make node execution order configurable, and backward-compatible
* ⚡ Also add new Merge-Node behaviour
* ⚡ Fix typo
* Fix lint issue
* update labels
* rename legacy to v0
* remove the unnecessary log
* default all new workflows to use v1 execution-order
* remove the controller changes
* clone default settings to avoid it getting modified
---------
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
* rename sendinblue to brevo
* refactor sendinblue node files/folders to brevo
* update codex node, documentation urls and api baseUrls
* adding comments to original node names
* First node set up.
* Progress: all Resources and Operations updated
* Upsates to all resources.
* Updated tooltip for Tweet > Search > Tweet fields.
* Upodate to resource locator items in User > Search.
* Added e.g. to placeholders and minor copy tweaks.
* Fixed Operations sorting.
* Added a couple of operations back.
* Removed 'Authorized API Call'.
* Remove authorization header when empty
* Import pkce
* Add OAuth2 with new grant type to Twitter
* Add pkce logic auto assign authorization code if pkce not defined
* Add pkce to ui and interfaces
* Fix scopes for Oauth2 twitter
* Deubg + pass it through header
* Add debug console, add airtable cred
* Remove all console.logs, make PKCE in th body only when it exists
* Remove invalid character ~
* Remove more console.logs
* remove body inside query
* Remove useless grantype check
* Hide oauth2 twitter waiting for overhaul
* Remove redundant header removal
* Remove more console.logs
* Add V2 twitter
* Add V1
* Fix description V1, V2
* Fix description for V1
* Add Oauth2 request
* Add user lookup
* Add search username by ID
* Search tweet feat
* Wip create tweet
* Generic function for returning ID
* Add like and retweet feat
* Add delete tweet feat
* Fix Location tweets
* Fix type
* Feat List add members
* Add scopes for dm and list
* Add direct message feature
* Improve response data
* Fix regex
* Add unit test to Twitter v2
* Fix unit test
* Remove console.logs
* Remove more console.logs
* Handle @ in username
* Minor copy tweaks.
* Add return all logic
* Add error for api permission error
* Update message api error
* Add error for date error
* Add notice for TwitterOAuth2 api link
* Fix display names location
* fix List RLC
* Fix like endpoint
* Fix error message check
* fix(core): Fix OAuth2 callback for grantType=clientCredentials
* Improve fix for callback
* update pnpm
* Fix iso time for end time
* sync oauth2Credential
* remove unused codeVerifer in Server.ts
* Add location and attachments notice
* Add notice to oauth1
* Improve notice for twitter
* moved credentials notice to TwitterOAuth1Api.credentials.ts
---------
Co-authored-by: agobrech <ael.gobrecht@gmail.com>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Co-authored-by: Marcus <marcus@n8n.io>
* Draft setup
* ⚡ Implemented expression evaluation in Postgres node, minor SQL editor UI improvements, minor refacring
* ⚡ Added initial version of expression preview for SQL editor
* ⚡ Linking npm package for codemirror sql grammar instead of a local file
* ⚡ Moving expression editor wrapper elements to the component
* ⚡ Using expression preview in SQL editor
* Use SQL parser skipping whitespace
* ✨ Added support for custom skipped segments specification
* ✨ Fixing highlight problems with dots and expressions that resolve to zero
* 👕 Fixing linting error
* ✨ Added current item support
* ⚡ Added expression support to more nodes with sql editor
* ✨ Added expression support for other nodes
* ✨ Implemented different SQL dialect support
* 🐛 Fixing hard-coded parameter names for editors
* ✨ Fixing preview for nested queries, updating query when input data changes, adding keyboard shortcut to toggle comments
* ✨ Adding a custom automcomplete notice for different editors
* ⚡ Updating SQL autocomplete notice
* ✅ Added unit tests for SQL editor
* ⚡ Using latest grammar
* 🐛 Fixing code node editor rendering
* 💄 SQL preview dropdown matches editor width. Removing unnecessary css
* ⚡ Addressing PR review feedback
* 👌 Addressing PR review feedback pt2
* 👌 Added path alias for utils in nodes-base package
* 👌 Addressing more PR review feedback
* ✅ Adding tests for `getResolvables` utility function
* ⚡Fixing lodash imports
* 👌 Better focus handling, adding more plugins to the editor, other minor imrovements
* ⚡ Not showing SQL autocomplete suggestions inside expressions
* ⚡ Using npm package for sql grammar
* ⚡ Removing autocomplete notice, adding line highlight on syntax error
* 👌 Addressing code review feedback
---------
Co-authored-by: Milorad Filipovic <milorad@n8n.io>
* Remove authorization header when empty
* Import pkce
* Add OAuth2 with new grant type to Twitter
* Add pkce logic auto assign authorization code if pkce not defined
* Add pkce to ui and interfaces
* Fix scopes for Oauth2 twitter
* Deubg + pass it through header
* Add debug console, add airtable cred
* Remove all console.logs, make PKCE in th body only when it exists
* Remove invalid character ~
* Remove more console.logs
* remove body inside query
* Remove useless grantype check
* Hide oauth2 twitter waiting for overhaul
* Remove redundant header removal
* Remove more console.logs
* Add comment for code verifier
* Remove uneeded scopes
* Restore client id in callback
* Revert "Add OAuth2 with new grant type to Twitter"
This reverts commit 1c3b331aa1.
* Remove oauth2 from twitter
* Remove properties linked to oauth2
* Fix lodash imports
* remove redundant check
* remove redundant codeVerifier
* patch pkce-challenge to avoid generating `code_verifier` with `~`
* store `codeVerifier` on the DB like `csrfSecret`
* remove unrelated changes
---------
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* first commit for postgres migration
* (not working)
* sqlite migration
* quicksave
* fix tests
* fix pg test
* fix postgres
* fix variables import
* fix execution saving
* add user settings fix
* change migration to single lines
* patch preferences endpoint
* cleanup
* improve variable import
* cleanup unusued code
* Update packages/cli/src/PublicApi/v1/handlers/workflows/workflows.handler.ts
Co-authored-by: Omar Ajoue <krynble@gmail.com>
* address review notes
* fix var update/import
* refactor: Separate execution data to its own table (#6323)
* wip: Temporary migration process
* refactor: Create boilerplate repository methods for executions
* fix: Lint issues
* refactor: Added search endpoint to repository
* refactor: Make the execution list work again
* wip: Updating how we create and update executions everywhere
* fix: Lint issues and remove most of the direct access to execution model
* refactor: Remove includeWorkflowData flag and fix more tests
* fix: Lint issues
* fix: Fixed ordering of executions for FE, removed transaction when saving execution and removed unnecessary update
* refactor: Add comment about missing feature
* refactor: Refactor counting executions
* refactor: Add migration for other dbms and fix issues found
* refactor: Fix lint issues
* refactor: Remove unnecessary comment and auto inject repo to internal hooks
* refactor: remove type assertion
* fix: Fix broken tests
* fix: Remove unnecessary import
* Remove unnecessary toString() call
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* fix: Address comments after review
* refactor: Remove unused import
* fix: Lint issues
* fix: Add correct migration files
---------
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* remove null values from credential export
* fix: Fix an issue with queue mode where all running execution would be returned
* fix: Update n8n node to allow for workflow ids with letters
* set upstream on set branch
* remove typo
* add nodeAccess to credentials
* fix unsaved run check for undefined id
* fix(core): Rename version control feature to source control (#6480)
* rename versionControl to sourceControl
* fix source control tooltip wording
---------
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
* fix(editor): Pay 548 hide the set up version control button (#6485)
* feat(DebugHelper Node): Fix and include in main app (#6406)
* improve node a bit
* fixing continueOnFail() ton contain error in json
* improve pairedItem
* fix random data returning object results
* fix nanoId length typo
* update pnpm-lock file
---------
Co-authored-by: Marcus <marcus@n8n.io>
* fix(editor): Remove setup source control CTA button
* fix(editor): Remove setup source control CTA button
---------
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
* fix(editor): Update source control docs links (#6488)
* feat(DebugHelper Node): Fix and include in main app (#6406)
* improve node a bit
* fixing continueOnFail() ton contain error in json
* improve pairedItem
* fix random data returning object results
* fix nanoId length typo
* update pnpm-lock file
---------
Co-authored-by: Marcus <marcus@n8n.io>
* feat(editor): Replace root events with event bus events (no-changelog) (#6454)
* feat: replace root events with event bus events
* fix: prevent cypress from replacing global with globalThis in import path
* feat: remove emitter mixin
* fix: replace component events with event bus
* fix: fix linting issue
* fix: fix breaking expression switch
* chore: prettify ndv e2e suite code
* fix(editor): Update source control docs links
---------
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
* fix tag endpoint regex
---------
Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
Co-authored-by: Csaba Tuncsik <csaba@n8n.io>
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
