Story: [PAY-846](https://linear.app/n8n/issue/PAY-846) | Related:
https://github.com/n8n-io/n8n/pull/7225
For the S3 backend for external storage of binary data and execution
data, the `getAsStream` method in the binary data manager interface used
by FS and S3 will need to become async. This is a breaking change for
nodes-base.
Depends on: #7092 | Story:
[PAY-768](https://linear.app/n8n/issue/PAY-768)
This PR:
- Generalizes the `IBinaryDataManager` interface.
- Adjusts `Filesystem.ts` to satisfy the interface.
- Sets up an S3 client stub to be filled in in the next PR.
- Turns `BinaryDataManager` into an injectable service.
- Adjusts the config schema and adds new validators.
Note that the PR looks large but all the main changes are in
`packages/core/src/binaryData`.
Out of scope:
- `BinaryDataManager` (now `BinaryDataService`) and `Filesystem.ts` (now
`fs.client.ts`) were slightly refactored for maintainability, but fully
overhauling them is **not** the focus of this PR, which is meant to
clear the way for the S3 implementation. Future improvements for these
two should include setting up a backwards-compatible dir structure that
makes it easier to locate binary data files to delete, removing
duplication, simplifying cloning methods, using integers for binary data
size instead of `prettyBytes()`, writing tests for existing binary data
logic, etc.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* Draft setup
* ⚡ Implemented expression evaluation in Postgres node, minor SQL editor UI improvements, minor refacring
* ⚡ Added initial version of expression preview for SQL editor
* ⚡ Linking npm package for codemirror sql grammar instead of a local file
* ⚡ Moving expression editor wrapper elements to the component
* ⚡ Using expression preview in SQL editor
* Use SQL parser skipping whitespace
* ✨ Added support for custom skipped segments specification
* ✨ Fixing highlight problems with dots and expressions that resolve to zero
* 👕 Fixing linting error
* ✨ Added current item support
* ⚡ Added expression support to more nodes with sql editor
* ✨ Added expression support for other nodes
* ✨ Implemented different SQL dialect support
* 🐛 Fixing hard-coded parameter names for editors
* ✨ Fixing preview for nested queries, updating query when input data changes, adding keyboard shortcut to toggle comments
* ✨ Adding a custom automcomplete notice for different editors
* ⚡ Updating SQL autocomplete notice
* ✅ Added unit tests for SQL editor
* ⚡ Using latest grammar
* 🐛 Fixing code node editor rendering
* 💄 SQL preview dropdown matches editor width. Removing unnecessary css
* ⚡ Addressing PR review feedback
* 👌 Addressing PR review feedback pt2
* 👌 Added path alias for utils in nodes-base package
* 👌 Addressing more PR review feedback
* ✅ Adding tests for `getResolvables` utility function
* ⚡Fixing lodash imports
* 👌 Better focus handling, adding more plugins to the editor, other minor imrovements
* ⚡ Not showing SQL autocomplete suggestions inside expressions
* ⚡ Using npm package for sql grammar
* ⚡ Removing autocomplete notice, adding line highlight on syntax error
* 👌 Addressing code review feedback
---------
Co-authored-by: Milorad Filipovic <milorad@n8n.io>
* ✨ Change from binary buffer to binary streaming
* Remove console.logs
* Import Readable from the correct lib
* stream response
* parametersToKeyValue doesn't need to be async anymore
* Fix bodyParameter reduce method
* parametersToKeyValue doesn't need to be async anymore
* handle streaming responses
* send `Content-Length` and `Content-Type` on binary requests
* Add new helper function for binary data
* Add binary function to helpers interface
* Fix bug in error handler
* Fix issue with wrongfully assigned headers to body
* Fix test workflow
* Remove console.logs
* Remove unnecsessary type
* Remove concat dependency already imported in workflow package
* Update pnpm-lock file
* Small fixes, asyncronous error message
* reset the lockfile
* Remove buffer check and simplify error handling
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* refactor(core): Add more overloads for string-type node parameter (no-changelog)
continued from https://github.com/n8n-io/n8n/pull/4797
* include binaryPropertyOutput
* ✨ Create rule `no-unneeded-backticks`
* 👕 Enable rule
* ⚡ Run rule on `cli`
* ⚡ Run rule on `core`
* ⚡ Run rule on `workflow`
* ⚡ Rule rule on `design-system`
* ⚡ Run rule on `node-dev`
* ⚡ Run rule on `editor-ui`
* ⚡ Run rule on `nodes-base`
* ⚡ enabled array-type
* ⚡ await-thenable on
* ⚡ ban-types on
* ⚡ default-param-last on
* ⚡ dot-notation on
* ⚡ member-delimiter-style on
* ⚡ no-duplicate-imports on
* ⚡ no-empty-interface on
* ⚡ no-floating-promises on
* ⚡ no-for-in-array on
* ⚡ no-invalid-void-type on
* ⚡ no-loop-func on
* ⚡ no-shadow on
* ⚡ ban-ts-comment re enabled
* ⚡ @typescript-eslint/lines-between-class-members on
* address my own comment
* @typescript-eslint/return-await on
* @typescript-eslint/promise-function-async on
* @typescript-eslint/no-unnecessary-boolean-literal-compare on
* @typescript-eslint/no-unnecessary-type-assertion on
* prefer-const on
* @typescript-eslint/prefer-optional-chain on
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* SIGTERM/SIGINT should only be handled once
* move error-handling initialization to commands
* create a new `sleep` function in workflow utils
* detect crashes and report them to Sentry
* http node headers and query JSON parameter options added
* Query and Header added to option processing
* remove old set headers.
* ⚡ Small improvements
Co-authored-by: ricardo <ricardoespinoza105@gmail.com>
