This change ensures that things like `encryptionKey` and `instanceId`
are always available directly where they are needed, instead of passing
them around throughout the code.
This adds support for
1. custom delimiters
2. reading offsets to avoid having to read a large CSV all at once
3. excluding byte-order-mark
NODE-861
#7443
Story: [PAY-846](https://linear.app/n8n/issue/PAY-846) | Related:
https://github.com/n8n-io/n8n/pull/7225
For the S3 backend for external storage of binary data and execution
data, the `getAsStream` method in the binary data manager interface used
by FS and S3 will need to become async. This is a breaking change for
nodes-base.
Depends on: #7092 | Story:
[PAY-768](https://linear.app/n8n/issue/PAY-768)
This PR:
- Generalizes the `IBinaryDataManager` interface.
- Adjusts `Filesystem.ts` to satisfy the interface.
- Sets up an S3 client stub to be filled in in the next PR.
- Turns `BinaryDataManager` into an injectable service.
- Adjusts the config schema and adds new validators.
Note that the PR looks large but all the main changes are in
`packages/core/src/binaryData`.
Out of scope:
- `BinaryDataManager` (now `BinaryDataService`) and `Filesystem.ts` (now
`fs.client.ts`) were slightly refactored for maintainability, but fully
overhauling them is **not** the focus of this PR, which is meant to
clear the way for the S3 implementation. Future improvements for these
two should include setting up a backwards-compatible dir structure that
makes it easier to locate binary data files to delete, removing
duplication, simplifying cloning methods, using integers for binary data
size instead of `prettyBytes()`, writing tests for existing binary data
logic, etc.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Github issue / Community forum post (link here to close automatically):
https://github.com/n8n-io/n8n/pull/6348
---------
Co-authored-by: Giulio Andreini <g.andreini@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
Also,
1. Consistent CORS support ~on all three webhook types~ waiting webhooks never supported CORS. I'll fix that in another PR
2. [Fixes binary-data handling when request body is text, json, or xml](https://linear.app/n8n/issue/NODE-505/webhook-binary-data-handling-fails-for-textplain-files).
3. Reduced number of middleware that each request has to go through.
4. Removed the need to maintain webhook endpoints in the auth-exception list.
5. Skip all middlewares (apart from `compression`) on Webhook routes.
6. move `multipart/form-data` support out of individual nodes
7. upgrade `formidable`
8. fix the filenames on binary-data in webhooks nodes
9. add unit tests and integration tests for webhook request handling, and increase test coverage
* fix(Postgres Node): For select queries, empty result should be be replaced with `{"success":true}`
* ⚡ less checks
---------
Co-authored-by: Michael Kret <michael.k@radency.com>
* fix(core): Make node execution order configurable, and backward-compatible
* ⚡ Also add new Merge-Node behaviour
* ⚡ Fix typo
* Fix lint issue
* update labels
* rename legacy to v0
* remove the unnecessary log
* default all new workflows to use v1 execution-order
* remove the controller changes
* clone default settings to avoid it getting modified
---------
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
* rename sendinblue to brevo
* refactor sendinblue node files/folders to brevo
* update codex node, documentation urls and api baseUrls
* adding comments to original node names
* First node set up.
* Progress: all Resources and Operations updated
* Upsates to all resources.
* Updated tooltip for Tweet > Search > Tweet fields.
* Upodate to resource locator items in User > Search.
* Added e.g. to placeholders and minor copy tweaks.
* Fixed Operations sorting.
* Added a couple of operations back.
* Removed 'Authorized API Call'.
* Remove authorization header when empty
* Import pkce
* Add OAuth2 with new grant type to Twitter
* Add pkce logic auto assign authorization code if pkce not defined
* Add pkce to ui and interfaces
* Fix scopes for Oauth2 twitter
* Deubg + pass it through header
* Add debug console, add airtable cred
* Remove all console.logs, make PKCE in th body only when it exists
* Remove invalid character ~
* Remove more console.logs
* remove body inside query
* Remove useless grantype check
* Hide oauth2 twitter waiting for overhaul
* Remove redundant header removal
* Remove more console.logs
* Add V2 twitter
* Add V1
* Fix description V1, V2
* Fix description for V1
* Add Oauth2 request
* Add user lookup
* Add search username by ID
* Search tweet feat
* Wip create tweet
* Generic function for returning ID
* Add like and retweet feat
* Add delete tweet feat
* Fix Location tweets
* Fix type
* Feat List add members
* Add scopes for dm and list
* Add direct message feature
* Improve response data
* Fix regex
* Add unit test to Twitter v2
* Fix unit test
* Remove console.logs
* Remove more console.logs
* Handle @ in username
* Minor copy tweaks.
* Add return all logic
* Add error for api permission error
* Update message api error
* Add error for date error
* Add notice for TwitterOAuth2 api link
* Fix display names location
* fix List RLC
* Fix like endpoint
* Fix error message check
* fix(core): Fix OAuth2 callback for grantType=clientCredentials
* Improve fix for callback
* update pnpm
* Fix iso time for end time
* sync oauth2Credential
* remove unused codeVerifer in Server.ts
* Add location and attachments notice
* Add notice to oauth1
* Improve notice for twitter
* moved credentials notice to TwitterOAuth1Api.credentials.ts
---------
Co-authored-by: agobrech <ael.gobrecht@gmail.com>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Co-authored-by: Marcus <marcus@n8n.io>
* Draft setup
* ⚡ Implemented expression evaluation in Postgres node, minor SQL editor UI improvements, minor refacring
* ⚡ Added initial version of expression preview for SQL editor
* ⚡ Linking npm package for codemirror sql grammar instead of a local file
* ⚡ Moving expression editor wrapper elements to the component
* ⚡ Using expression preview in SQL editor
* Use SQL parser skipping whitespace
* ✨ Added support for custom skipped segments specification
* ✨ Fixing highlight problems with dots and expressions that resolve to zero
* 👕 Fixing linting error
* ✨ Added current item support
* ⚡ Added expression support to more nodes with sql editor
* ✨ Added expression support for other nodes
* ✨ Implemented different SQL dialect support
* 🐛 Fixing hard-coded parameter names for editors
* ✨ Fixing preview for nested queries, updating query when input data changes, adding keyboard shortcut to toggle comments
* ✨ Adding a custom automcomplete notice for different editors
* ⚡ Updating SQL autocomplete notice
* ✅ Added unit tests for SQL editor
* ⚡ Using latest grammar
* 🐛 Fixing code node editor rendering
* 💄 SQL preview dropdown matches editor width. Removing unnecessary css
* ⚡ Addressing PR review feedback
* 👌 Addressing PR review feedback pt2
* 👌 Added path alias for utils in nodes-base package
* 👌 Addressing more PR review feedback
* ✅ Adding tests for `getResolvables` utility function
* ⚡Fixing lodash imports
* 👌 Better focus handling, adding more plugins to the editor, other minor imrovements
* ⚡ Not showing SQL autocomplete suggestions inside expressions
* ⚡ Using npm package for sql grammar
* ⚡ Removing autocomplete notice, adding line highlight on syntax error
* 👌 Addressing code review feedback
---------
Co-authored-by: Milorad Filipovic <milorad@n8n.io>
* Remove authorization header when empty
* Import pkce
* Add OAuth2 with new grant type to Twitter
* Add pkce logic auto assign authorization code if pkce not defined
* Add pkce to ui and interfaces
* Fix scopes for Oauth2 twitter
* Deubg + pass it through header
* Add debug console, add airtable cred
* Remove all console.logs, make PKCE in th body only when it exists
* Remove invalid character ~
* Remove more console.logs
* remove body inside query
* Remove useless grantype check
* Hide oauth2 twitter waiting for overhaul
* Remove redundant header removal
* Remove more console.logs
* Add comment for code verifier
* Remove uneeded scopes
* Restore client id in callback
* Revert "Add OAuth2 with new grant type to Twitter"
This reverts commit 1c3b331aa1.
* Remove oauth2 from twitter
* Remove properties linked to oauth2
* Fix lodash imports
* remove redundant check
* remove redundant codeVerifier
* patch pkce-challenge to avoid generating `code_verifier` with `~`
* store `codeVerifier` on the DB like `csrfSecret`
* remove unrelated changes
---------
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* first commit for postgres migration
* (not working)
* sqlite migration
* quicksave
* fix tests
* fix pg test
* fix postgres
* fix variables import
* fix execution saving
* add user settings fix
* change migration to single lines
* patch preferences endpoint
* cleanup
* improve variable import
* cleanup unusued code
* Update packages/cli/src/PublicApi/v1/handlers/workflows/workflows.handler.ts
Co-authored-by: Omar Ajoue <krynble@gmail.com>
* address review notes
* fix var update/import
* refactor: Separate execution data to its own table (#6323)
* wip: Temporary migration process
* refactor: Create boilerplate repository methods for executions
* fix: Lint issues
* refactor: Added search endpoint to repository
* refactor: Make the execution list work again
* wip: Updating how we create and update executions everywhere
* fix: Lint issues and remove most of the direct access to execution model
* refactor: Remove includeWorkflowData flag and fix more tests
* fix: Lint issues
* fix: Fixed ordering of executions for FE, removed transaction when saving execution and removed unnecessary update
* refactor: Add comment about missing feature
* refactor: Refactor counting executions
* refactor: Add migration for other dbms and fix issues found
* refactor: Fix lint issues
* refactor: Remove unnecessary comment and auto inject repo to internal hooks
* refactor: remove type assertion
* fix: Fix broken tests
* fix: Remove unnecessary import
* Remove unnecessary toString() call
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* fix: Address comments after review
* refactor: Remove unused import
* fix: Lint issues
* fix: Add correct migration files
---------
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* remove null values from credential export
* fix: Fix an issue with queue mode where all running execution would be returned
* fix: Update n8n node to allow for workflow ids with letters
* set upstream on set branch
* remove typo
* add nodeAccess to credentials
* fix unsaved run check for undefined id
* fix(core): Rename version control feature to source control (#6480)
* rename versionControl to sourceControl
* fix source control tooltip wording
---------
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
* fix(editor): Pay 548 hide the set up version control button (#6485)
* feat(DebugHelper Node): Fix and include in main app (#6406)
* improve node a bit
* fixing continueOnFail() ton contain error in json
* improve pairedItem
* fix random data returning object results
* fix nanoId length typo
* update pnpm-lock file
---------
Co-authored-by: Marcus <marcus@n8n.io>
* fix(editor): Remove setup source control CTA button
* fix(editor): Remove setup source control CTA button
---------
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
* fix(editor): Update source control docs links (#6488)
* feat(DebugHelper Node): Fix and include in main app (#6406)
* improve node a bit
* fixing continueOnFail() ton contain error in json
* improve pairedItem
* fix random data returning object results
* fix nanoId length typo
* update pnpm-lock file
---------
Co-authored-by: Marcus <marcus@n8n.io>
* feat(editor): Replace root events with event bus events (no-changelog) (#6454)
* feat: replace root events with event bus events
* fix: prevent cypress from replacing global with globalThis in import path
* feat: remove emitter mixin
* fix: replace component events with event bus
* fix: fix linting issue
* fix: fix breaking expression switch
* chore: prettify ndv e2e suite code
* fix(editor): Update source control docs links
---------
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
* fix tag endpoint regex
---------
Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
Co-authored-by: Csaba Tuncsik <csaba@n8n.io>
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
* Create new version for S3
* Update S3 to new aws s3 methods
* Switch from SAOP to Rest api
* Add multipart request
* Seperate stream into chunks and send the multipart
* Fix chunk into buffer
* Fix wrong sha256 mismatch
* Add abort multipart on error
* Complete multipart and list parts
* Change format to xml and add a minmum size of 5MB for each part
* Fix returned data for uploading a file
* Remove console.logs
* Seperate needed headers and multipart headers
* Throw error on aborting, remove console.logs
* Remove soap request from generic function
* Keep buffer
* Add unit test for V2
* fix upload file content body
* removed unused import
* Fix bug where the object was too smal and used only one part
* Fix naming for bucket name
* Fix issue with file name not returning data
* Add parent name
* Remove console.logs
* Add content type
* fix headears for other upload mode
---------
Co-authored-by: Marcus <marcus@n8n.io>
* Handle conversion to correct unix format
* Fix intervals, ranges for months
* fix regex to match 10, 11, 12
---------
Co-authored-by: Marcus <marcus@n8n.io>
* Boilerplate for PostgresTrigger
* Create trigger function as a query
* Add additional fields to customize trigger query
* Add customizable channel name && operation name
* Add concat () for function name
* Add hints and placeholders
* Add resource Locator to trigger postgres
* Add the ability for knowing trigger event
* Throw error for same function name
* Remove console.logs
* Remove schema from Chanel notifcation mode
* Add UUID and save trigger in workflow static data drop function
* Fix bug where wrongfully casted result in pgl
* Correctly drops the resources when manually executing the trigger
* Remove manual execution with special interaction
* Remove console.logs
* ♻️ Move related trigger functions to new file
* fix target using 'schema."tableName"' in quotes
To support targets with Uppercase table names
* Remove static Data and use node id for uuid
* Update deleting of the trigger and function
* Fix regex expression for channel name
* Change to drop cascade the trigger function
* Replace functions on restart if no name has been defined
* Parse payload result
* Improve handling with hyphens in names
* Remove duplicate code and clean up
* Add payload on delete
* Fix rlc
* fixing uppercase tableName
* fix multiple triggers/connections issues
* fixing rlc pgp.end() issues
* unify pgp init db method
* drop trigger only in createTrigger mode
---------
Co-authored-by: Marcus <marcus@n8n.io>
* ci: Start supporting Node.js 18
Node.js 18 became the active LTS on 2022-10-25, and Node.js 16 went into maintenance mode.
https://github.com/nodejs/Release#release-schedule
We should also slowly deprecate node 16 support, [as support for it is ends much earlier now, due to support for openssl 1.1.1 ending](https://nodejs.org/en/blog/announcements/nodejs16-eol).
* Remove hashing algorithms that are not available in newer node.js/openssl
- RSA-MD4
- RSA-MDC2
- md4
- md4WithRSAEncryption
- mdc2
- mdc2WithRSA
* in e2e tests, resolve `localhost` to ipv4 instead of ipv6
* Add later in workflow mode
* Add new operation
* Acknowledge message in next node
* Add response and emit for responsePromiseHook
* Remove double success message, close channel correctly
* Answser messages correctly
* Remove option from delete operation
* move operation name to camelCase
* Fix versioning
* To remove: add action item in v1
* Add notice for delete from queue
* Correctly only execute only the delete operation
* Refactor delete from queue operator and add return last items
---------
Co-authored-by: Marcus <marcus@n8n.io>
* Setup versionized node
* Fix node naming
* Set all possible actions
* Add Current Date operation
* Add timezone to current date
* feat add to date operator
* Change output field name to camel case
* Fix info box for luxons tip
* Feat subtract to date operation
* Feat format date operation
* Fix to node field for format date
* Feat rounding operation
* Feat get in between date operation
* Feat add extract date operation
* Add generic function for parsing date
* Remove moment methods from operations
* Change moment to luxon for the rest of the operations
* Fix Format date operation
* Fix format value
* Add timezone option for current date
* Add tests, improve workflow settings for testing, toString the results
* Change icon for V2
* Revert "Change icon for V2"
This reverts commit 46b59bea2e.
* Change workflow test name
* Fix ui bug for custom format
* Fix default value for format operation
* Fix info box for rounding operation
* Change default for units for between time operation
* Inprove fields and resort time units
* Fix extract week number
* Resolve issue with formating and timezones
* Fix field name and unit order
* ⚡ restored removed test case, sync v1 with curent master
* ⚡ parseDate update to support timestamps, tests
* Keep same field for substract and add time
* Update unit test
* Improve visibility, add iso to string option
* Update option naming
---------
Co-authored-by: Michael Kret <michael.k@radency.com>
* 🎨 Change current defualt ressource to contact
* Change display name for list membership
* 🎨 Change display name for ressource properties
* 🔥 Remove Name & ID in Display name
* 🐛 Fix simplify data where behavior was wrong
* 🎨 Move simplfy output to additional fields
* Change data type from string to number for IDs
* Add tooltip for contact ID
* 🎨 Change tooltip for list of Id's
* Change name of fields and reorgaized options
* 🥅 Add clearer errors for some specific cases
* 🎨 changed filters in Deal get all
* Deprecate form ressource
* ✨ add pipeline dropdown to auto fetch
* 🎨 Change direction's name
* 🎨 Change description of filter groups
* 🎨 Add validation and errors for maximum filters
* ✨ Add ressource locator to all ressources
* 🎨 add tooltip for timestamp when using expressions
* ✨add versioning
* 🐛 fixes small bugs and ui problems
* remove show options that was breaking custom prop
* ✨ add ownership to deals
* ✨ add notice for credential depreciation
* 🎨 Refactor names for RLC
* 🎨 Fixes to Deal
* ✨ Inprove delete confirmation message
* 🎨 Fixe titles of fields
* 🎨 Fixe correct ressource hint
* 🎨 Improve error when requesting non existing ressource
* ⚡️ Improve get deal
* ✨ Improve search by domain for companies
* ✨ Improve getting recently created deals
* 🎨 Improve versioning file structure
* 🥅 Improve error for unknown ids
* ✨ Add filters to recently search for companies
* ✨ Uniformize including properties for contact
* ✨ Improve get all operation for Companies
* 🎨 Change filters name
* 🎨 Remove useless descriptions
* 🎨Improve description
* ✨ Dynamically retrieve the correct operator for the properties
* 🥅 Improve error handeling
* 🥅 improve error when not finding the id
* ⚡️Add searchable feature to Deal and contacts
* 🚨 Fix linting issues
* 🚨 Fix linting issues for V1 Hubspot
* 🐛 Fix pairedItem for v2 hubspot
* 🚨 Fix linting in HubSpot Description
* 🚨 More linting fixes
* Fix error handling
* Update endpoint for deleting object
* Fix label for recent actions
* 🐛 fix listMemberships typo in V1 and V2
* docs: Stop assuming the user's gender in code comments (after merge)
c87262a312
* fixing V1 GenericFunctions.ts api query limit as number
* Update description for ticket
* Add type to RLC
* Revert "Add type to RLC"
This reverts commit 578dacfb97.
* Add type to RLC engagement
---------
Co-authored-by: Marcus <marcus@n8n.io>
* ✅ add Read Binary Files tests
* ⚡️updated ReadBinaryFiles test
* fix test result comparison
* fix test result comparison
---------
Co-authored-by: Michael Kret <michael.k@radency.com>
* Fix broken logic with type action
* Unit test base
* Add unit test using only nock
* Fix failing test
* remove console log
---------
Co-authored-by: Marcus <marcus@n8n.io>
* feat(editor): Add support for `loadOptionsDependsOn` to the Resource Locator component
* 🔥 Removing leftover log
* ✅ Added e2e tests for ResourceLocator component
