Commit graph

2 commits

Author SHA1 Message Date
Iván Ovejero 6d6e2488c6
refactor(core): Generalize binary data manager interface (no-changelog) (#7164)
Depends on: #7092 | Story:
[PAY-768](https://linear.app/n8n/issue/PAY-768)

This PR: 
- Generalizes the `IBinaryDataManager` interface.
- Adjusts `Filesystem.ts` to satisfy the interface.
- Sets up an S3 client stub to be filled in in the next PR.
- Turns `BinaryDataManager` into an injectable service.
- Adjusts the config schema and adds new validators.

Note that the PR looks large but all the main changes are in
`packages/core/src/binaryData`.

Out of scope:
- `BinaryDataManager` (now `BinaryDataService`) and `Filesystem.ts` (now
`fs.client.ts`) were slightly refactored for maintainability, but fully
overhauling them is **not** the focus of this PR, which is meant to
clear the way for the S3 implementation. Future improvements for these
two should include setting up a backwards-compatible dir structure that
makes it easier to locate binary data files to delete, removing
duplication, simplifying cloning methods, using integers for binary data
size instead of `prettyBytes()`, writing tests for existing binary data
logic, etc.

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
2023-09-22 17:22:12 +02:00
agobrech 109442f38f
feat(AwsS3 Node): Small overhaul of the node with multipart uploading (#6017)
* Create new version for S3

* Update S3 to new aws s3 methods

* Switch from SAOP to Rest api

* Add multipart request

* Seperate stream into chunks and send the multipart

* Fix chunk into buffer

* Fix wrong sha256 mismatch

* Add abort multipart on error

* Complete multipart and list parts

* Change format to xml and add a minmum size of 5MB for each part

* Fix returned data for uploading a file

* Remove console.logs

* Seperate needed headers and multipart headers

* Throw error on aborting, remove console.logs

* Remove soap request from generic function

* Keep buffer

* Add unit test for V2

* fix upload file content body

* removed unused import

* Fix bug where the object was too smal and used only one part

* Fix naming for bucket name

* Fix issue with file name not returning data

* Add parent name

* Remove console.logs

* Add content type

* fix headears for other upload mode

---------

Co-authored-by: Marcus <marcus@n8n.io>
2023-06-15 13:19:22 +02:00