This adds support for
1. custom delimiters
2. reading offsets to avoid having to read a large CSV all at once
3. excluding byte-order-mark
NODE-861
#7443
Story: [PAY-846](https://linear.app/n8n/issue/PAY-846) | Related:
https://github.com/n8n-io/n8n/pull/7225
For the S3 backend for external storage of binary data and execution
data, the `getAsStream` method in the binary data manager interface used
by FS and S3 will need to become async. This is a breaking change for
nodes-base.
Depends on: #7092 | Story:
[PAY-768](https://linear.app/n8n/issue/PAY-768)
This PR:
- Generalizes the `IBinaryDataManager` interface.
- Adjusts `Filesystem.ts` to satisfy the interface.
- Sets up an S3 client stub to be filled in in the next PR.
- Turns `BinaryDataManager` into an injectable service.
- Adjusts the config schema and adds new validators.
Note that the PR looks large but all the main changes are in
`packages/core/src/binaryData`.
Out of scope:
- `BinaryDataManager` (now `BinaryDataService`) and `Filesystem.ts` (now
`fs.client.ts`) were slightly refactored for maintainability, but fully
overhauling them is **not** the focus of this PR, which is meant to
clear the way for the S3 implementation. Future improvements for these
two should include setting up a backwards-compatible dir structure that
makes it easier to locate binary data files to delete, removing
duplication, simplifying cloning methods, using integers for binary data
size instead of `prettyBytes()`, writing tests for existing binary data
logic, etc.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Github issue / Community forum post (link here to close automatically):
https://github.com/n8n-io/n8n/pull/6348
---------
Co-authored-by: Giulio Andreini <g.andreini@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
Also,
1. Consistent CORS support ~on all three webhook types~ waiting webhooks never supported CORS. I'll fix that in another PR
2. [Fixes binary-data handling when request body is text, json, or xml](https://linear.app/n8n/issue/NODE-505/webhook-binary-data-handling-fails-for-textplain-files).
3. Reduced number of middleware that each request has to go through.
4. Removed the need to maintain webhook endpoints in the auth-exception list.
5. Skip all middlewares (apart from `compression`) on Webhook routes.
6. move `multipart/form-data` support out of individual nodes
7. upgrade `formidable`
8. fix the filenames on binary-data in webhooks nodes
9. add unit tests and integration tests for webhook request handling, and increase test coverage
* fix(Postgres Node): For select queries, empty result should be be replaced with `{"success":true}`
* ⚡ less checks
---------
Co-authored-by: Michael Kret <michael.k@radency.com>
* fix(core): Make node execution order configurable, and backward-compatible
* ⚡ Also add new Merge-Node behaviour
* ⚡ Fix typo
* Fix lint issue
* update labels
* rename legacy to v0
* remove the unnecessary log
* default all new workflows to use v1 execution-order
* remove the controller changes
* clone default settings to avoid it getting modified
---------
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
* rename sendinblue to brevo
* refactor sendinblue node files/folders to brevo
* update codex node, documentation urls and api baseUrls
* adding comments to original node names
* First node set up.
* Progress: all Resources and Operations updated
* Upsates to all resources.
* Updated tooltip for Tweet > Search > Tweet fields.
* Upodate to resource locator items in User > Search.
* Added e.g. to placeholders and minor copy tweaks.
* Fixed Operations sorting.
* Added a couple of operations back.
* Removed 'Authorized API Call'.
* Remove authorization header when empty
* Import pkce
* Add OAuth2 with new grant type to Twitter
* Add pkce logic auto assign authorization code if pkce not defined
* Add pkce to ui and interfaces
* Fix scopes for Oauth2 twitter
* Deubg + pass it through header
* Add debug console, add airtable cred
* Remove all console.logs, make PKCE in th body only when it exists
* Remove invalid character ~
* Remove more console.logs
* remove body inside query
* Remove useless grantype check
* Hide oauth2 twitter waiting for overhaul
* Remove redundant header removal
* Remove more console.logs
* Add V2 twitter
* Add V1
* Fix description V1, V2
* Fix description for V1
* Add Oauth2 request
* Add user lookup
* Add search username by ID
* Search tweet feat
* Wip create tweet
* Generic function for returning ID
* Add like and retweet feat
* Add delete tweet feat
* Fix Location tweets
* Fix type
* Feat List add members
* Add scopes for dm and list
* Add direct message feature
* Improve response data
* Fix regex
* Add unit test to Twitter v2
* Fix unit test
* Remove console.logs
* Remove more console.logs
* Handle @ in username
* Minor copy tweaks.
* Add return all logic
* Add error for api permission error
* Update message api error
* Add error for date error
* Add notice for TwitterOAuth2 api link
* Fix display names location
* fix List RLC
* Fix like endpoint
* Fix error message check
* fix(core): Fix OAuth2 callback for grantType=clientCredentials
* Improve fix for callback
* update pnpm
* Fix iso time for end time
* sync oauth2Credential
* remove unused codeVerifer in Server.ts
* Add location and attachments notice
* Add notice to oauth1
* Improve notice for twitter
* moved credentials notice to TwitterOAuth1Api.credentials.ts
---------
Co-authored-by: agobrech <ael.gobrecht@gmail.com>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Co-authored-by: Marcus <marcus@n8n.io>
* Draft setup
* ⚡ Implemented expression evaluation in Postgres node, minor SQL editor UI improvements, minor refacring
* ⚡ Added initial version of expression preview for SQL editor
* ⚡ Linking npm package for codemirror sql grammar instead of a local file
* ⚡ Moving expression editor wrapper elements to the component
* ⚡ Using expression preview in SQL editor
* Use SQL parser skipping whitespace
* ✨ Added support for custom skipped segments specification
* ✨ Fixing highlight problems with dots and expressions that resolve to zero
* 👕 Fixing linting error
* ✨ Added current item support
* ⚡ Added expression support to more nodes with sql editor
* ✨ Added expression support for other nodes
* ✨ Implemented different SQL dialect support
* 🐛 Fixing hard-coded parameter names for editors
* ✨ Fixing preview for nested queries, updating query when input data changes, adding keyboard shortcut to toggle comments
* ✨ Adding a custom automcomplete notice for different editors
* ⚡ Updating SQL autocomplete notice
* ✅ Added unit tests for SQL editor
* ⚡ Using latest grammar
* 🐛 Fixing code node editor rendering
* 💄 SQL preview dropdown matches editor width. Removing unnecessary css
* ⚡ Addressing PR review feedback
* 👌 Addressing PR review feedback pt2
* 👌 Added path alias for utils in nodes-base package
* 👌 Addressing more PR review feedback
* ✅ Adding tests for `getResolvables` utility function
* ⚡Fixing lodash imports
* 👌 Better focus handling, adding more plugins to the editor, other minor imrovements
* ⚡ Not showing SQL autocomplete suggestions inside expressions
* ⚡ Using npm package for sql grammar
* ⚡ Removing autocomplete notice, adding line highlight on syntax error
* 👌 Addressing code review feedback
---------
Co-authored-by: Milorad Filipovic <milorad@n8n.io>
* Remove authorization header when empty
* Import pkce
* Add OAuth2 with new grant type to Twitter
* Add pkce logic auto assign authorization code if pkce not defined
* Add pkce to ui and interfaces
* Fix scopes for Oauth2 twitter
* Deubg + pass it through header
* Add debug console, add airtable cred
* Remove all console.logs, make PKCE in th body only when it exists
* Remove invalid character ~
* Remove more console.logs
* remove body inside query
* Remove useless grantype check
* Hide oauth2 twitter waiting for overhaul
* Remove redundant header removal
* Remove more console.logs
* Add comment for code verifier
* Remove uneeded scopes
* Restore client id in callback
* Revert "Add OAuth2 with new grant type to Twitter"
This reverts commit 1c3b331aa1.
* Remove oauth2 from twitter
* Remove properties linked to oauth2
* Fix lodash imports
* remove redundant check
* remove redundant codeVerifier
* patch pkce-challenge to avoid generating `code_verifier` with `~`
* store `codeVerifier` on the DB like `csrfSecret`
* remove unrelated changes
---------
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* first commit for postgres migration
* (not working)
* sqlite migration
* quicksave
* fix tests
* fix pg test
* fix postgres
* fix variables import
* fix execution saving
* add user settings fix
* change migration to single lines
* patch preferences endpoint
* cleanup
* improve variable import
* cleanup unusued code
* Update packages/cli/src/PublicApi/v1/handlers/workflows/workflows.handler.ts
Co-authored-by: Omar Ajoue <krynble@gmail.com>
* address review notes
* fix var update/import
* refactor: Separate execution data to its own table (#6323)
* wip: Temporary migration process
* refactor: Create boilerplate repository methods for executions
* fix: Lint issues
* refactor: Added search endpoint to repository
* refactor: Make the execution list work again
* wip: Updating how we create and update executions everywhere
* fix: Lint issues and remove most of the direct access to execution model
* refactor: Remove includeWorkflowData flag and fix more tests
* fix: Lint issues
* fix: Fixed ordering of executions for FE, removed transaction when saving execution and removed unnecessary update
* refactor: Add comment about missing feature
* refactor: Refactor counting executions
* refactor: Add migration for other dbms and fix issues found
* refactor: Fix lint issues
* refactor: Remove unnecessary comment and auto inject repo to internal hooks
* refactor: remove type assertion
* fix: Fix broken tests
* fix: Remove unnecessary import
* Remove unnecessary toString() call
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* fix: Address comments after review
* refactor: Remove unused import
* fix: Lint issues
* fix: Add correct migration files
---------
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* remove null values from credential export
* fix: Fix an issue with queue mode where all running execution would be returned
* fix: Update n8n node to allow for workflow ids with letters
* set upstream on set branch
* remove typo
* add nodeAccess to credentials
* fix unsaved run check for undefined id
* fix(core): Rename version control feature to source control (#6480)
* rename versionControl to sourceControl
* fix source control tooltip wording
---------
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
* fix(editor): Pay 548 hide the set up version control button (#6485)
* feat(DebugHelper Node): Fix and include in main app (#6406)
* improve node a bit
* fixing continueOnFail() ton contain error in json
* improve pairedItem
* fix random data returning object results
* fix nanoId length typo
* update pnpm-lock file
---------
Co-authored-by: Marcus <marcus@n8n.io>
* fix(editor): Remove setup source control CTA button
* fix(editor): Remove setup source control CTA button
---------
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
* fix(editor): Update source control docs links (#6488)
* feat(DebugHelper Node): Fix and include in main app (#6406)
* improve node a bit
* fixing continueOnFail() ton contain error in json
* improve pairedItem
* fix random data returning object results
* fix nanoId length typo
* update pnpm-lock file
---------
Co-authored-by: Marcus <marcus@n8n.io>
* feat(editor): Replace root events with event bus events (no-changelog) (#6454)
* feat: replace root events with event bus events
* fix: prevent cypress from replacing global with globalThis in import path
* feat: remove emitter mixin
* fix: replace component events with event bus
* fix: fix linting issue
* fix: fix breaking expression switch
* chore: prettify ndv e2e suite code
* fix(editor): Update source control docs links
---------
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
* fix tag endpoint regex
---------
Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
Co-authored-by: Csaba Tuncsik <csaba@n8n.io>
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
* Create new version for S3
* Update S3 to new aws s3 methods
* Switch from SAOP to Rest api
* Add multipart request
* Seperate stream into chunks and send the multipart
* Fix chunk into buffer
* Fix wrong sha256 mismatch
* Add abort multipart on error
* Complete multipart and list parts
* Change format to xml and add a minmum size of 5MB for each part
* Fix returned data for uploading a file
* Remove console.logs
* Seperate needed headers and multipart headers
* Throw error on aborting, remove console.logs
* Remove soap request from generic function
* Keep buffer
* Add unit test for V2
* fix upload file content body
* removed unused import
* Fix bug where the object was too smal and used only one part
* Fix naming for bucket name
* Fix issue with file name not returning data
* Add parent name
* Remove console.logs
* Add content type
* fix headears for other upload mode
---------
Co-authored-by: Marcus <marcus@n8n.io>
* Handle conversion to correct unix format
* Fix intervals, ranges for months
* fix regex to match 10, 11, 12
---------
Co-authored-by: Marcus <marcus@n8n.io>
* Boilerplate for PostgresTrigger
* Create trigger function as a query
* Add additional fields to customize trigger query
* Add customizable channel name && operation name
* Add concat () for function name
* Add hints and placeholders
* Add resource Locator to trigger postgres
* Add the ability for knowing trigger event
* Throw error for same function name
* Remove console.logs
* Remove schema from Chanel notifcation mode
* Add UUID and save trigger in workflow static data drop function
* Fix bug where wrongfully casted result in pgl
* Correctly drops the resources when manually executing the trigger
* Remove manual execution with special interaction
* Remove console.logs
* ♻️ Move related trigger functions to new file
* fix target using 'schema."tableName"' in quotes
To support targets with Uppercase table names
* Remove static Data and use node id for uuid
* Update deleting of the trigger and function
* Fix regex expression for channel name
* Change to drop cascade the trigger function
* Replace functions on restart if no name has been defined
* Parse payload result
* Improve handling with hyphens in names
* Remove duplicate code and clean up
* Add payload on delete
* Fix rlc
* fixing uppercase tableName
* fix multiple triggers/connections issues
* fixing rlc pgp.end() issues
* unify pgp init db method
* drop trigger only in createTrigger mode
---------
Co-authored-by: Marcus <marcus@n8n.io>
* ci: Start supporting Node.js 18
Node.js 18 became the active LTS on 2022-10-25, and Node.js 16 went into maintenance mode.
https://github.com/nodejs/Release#release-schedule
We should also slowly deprecate node 16 support, [as support for it is ends much earlier now, due to support for openssl 1.1.1 ending](https://nodejs.org/en/blog/announcements/nodejs16-eol).
* Remove hashing algorithms that are not available in newer node.js/openssl
- RSA-MD4
- RSA-MDC2
- md4
- md4WithRSAEncryption
- mdc2
- mdc2WithRSA
* in e2e tests, resolve `localhost` to ipv4 instead of ipv6
* Add later in workflow mode
* Add new operation
* Acknowledge message in next node
* Add response and emit for responsePromiseHook
* Remove double success message, close channel correctly
* Answser messages correctly
* Remove option from delete operation
* move operation name to camelCase
* Fix versioning
* To remove: add action item in v1
* Add notice for delete from queue
* Correctly only execute only the delete operation
* Refactor delete from queue operator and add return last items
---------
Co-authored-by: Marcus <marcus@n8n.io>
* Setup versionized node
* Fix node naming
* Set all possible actions
* Add Current Date operation
* Add timezone to current date
* feat add to date operator
* Change output field name to camel case
* Fix info box for luxons tip
* Feat subtract to date operation
* Feat format date operation
* Fix to node field for format date
* Feat rounding operation
* Feat get in between date operation
* Feat add extract date operation
* Add generic function for parsing date
* Remove moment methods from operations
* Change moment to luxon for the rest of the operations
* Fix Format date operation
* Fix format value
* Add timezone option for current date
* Add tests, improve workflow settings for testing, toString the results
* Change icon for V2
* Revert "Change icon for V2"
This reverts commit 46b59bea2e.
* Change workflow test name
* Fix ui bug for custom format
* Fix default value for format operation
* Fix info box for rounding operation
* Change default for units for between time operation
* Inprove fields and resort time units
* Fix extract week number
* Resolve issue with formating and timezones
* Fix field name and unit order
* ⚡ restored removed test case, sync v1 with curent master
* ⚡ parseDate update to support timestamps, tests
* Keep same field for substract and add time
* Update unit test
* Improve visibility, add iso to string option
* Update option naming
---------
Co-authored-by: Michael Kret <michael.k@radency.com>
* 🎨 Change current defualt ressource to contact
* Change display name for list membership
* 🎨 Change display name for ressource properties
* 🔥 Remove Name & ID in Display name
* 🐛 Fix simplify data where behavior was wrong
* 🎨 Move simplfy output to additional fields
* Change data type from string to number for IDs
* Add tooltip for contact ID
* 🎨 Change tooltip for list of Id's
* Change name of fields and reorgaized options
* 🥅 Add clearer errors for some specific cases
* 🎨 changed filters in Deal get all
* Deprecate form ressource
* ✨ add pipeline dropdown to auto fetch
* 🎨 Change direction's name
* 🎨 Change description of filter groups
* 🎨 Add validation and errors for maximum filters
* ✨ Add ressource locator to all ressources
* 🎨 add tooltip for timestamp when using expressions
* ✨add versioning
* 🐛 fixes small bugs and ui problems
* remove show options that was breaking custom prop
* ✨ add ownership to deals
* ✨ add notice for credential depreciation
* 🎨 Refactor names for RLC
* 🎨 Fixes to Deal
* ✨ Inprove delete confirmation message
* 🎨 Fixe titles of fields
* 🎨 Fixe correct ressource hint
* 🎨 Improve error when requesting non existing ressource
* ⚡️ Improve get deal
* ✨ Improve search by domain for companies
* ✨ Improve getting recently created deals
* 🎨 Improve versioning file structure
* 🥅 Improve error for unknown ids
* ✨ Add filters to recently search for companies
* ✨ Uniformize including properties for contact
* ✨ Improve get all operation for Companies
* 🎨 Change filters name
* 🎨 Remove useless descriptions
* 🎨Improve description
* ✨ Dynamically retrieve the correct operator for the properties
* 🥅 Improve error handeling
* 🥅 improve error when not finding the id
* ⚡️Add searchable feature to Deal and contacts
* 🚨 Fix linting issues
* 🚨 Fix linting issues for V1 Hubspot
* 🐛 Fix pairedItem for v2 hubspot
* 🚨 Fix linting in HubSpot Description
* 🚨 More linting fixes
* Fix error handling
* Update endpoint for deleting object
* Fix label for recent actions
* 🐛 fix listMemberships typo in V1 and V2
* docs: Stop assuming the user's gender in code comments (after merge)
c87262a312
* fixing V1 GenericFunctions.ts api query limit as number
* Update description for ticket
* Add type to RLC
* Revert "Add type to RLC"
This reverts commit 578dacfb97.
* Add type to RLC engagement
---------
Co-authored-by: Marcus <marcus@n8n.io>
* ✅ add Read Binary Files tests
* ⚡️updated ReadBinaryFiles test
* fix test result comparison
* fix test result comparison
---------
Co-authored-by: Michael Kret <michael.k@radency.com>
* Fix broken logic with type action
* Unit test base
* Add unit test using only nock
* Fix failing test
* remove console log
---------
Co-authored-by: Marcus <marcus@n8n.io>
