Based on #7065 | Story: https://linear.app/n8n/issue/PAY-771
n8n on filesystem mode marks binary data to delete on manual execution
deletion, on unsaved execution completion, and on every execution
pruning cycle. We later prune binary data in a separate cycle via these
marker files, based on the configured TTL. In the context of introducing
an S3 client to manage binary data, the filesystem mode's mark-and-prune
setup is too tightly coupled to the general binary data management
client interface.
This PR...
- Ensures the deletion of an execution causes the deletion of any binary
data associated to it. This does away with the need for binary data TTL
and simplifies the filesystem mode's mark-and-prune setup.
- Refactors all execution deletions (including pruning) to cause soft
deletions, hard-deletes soft-deleted executions based on the existing
pruning config, and adjusts execution endpoints to filter out
soft-deleted executions. This reduces DB load, and keeps binary data
around long enough for users to access it when building workflows with
unsaved executions.
- Moves all execution pruning work from an execution lifecycle hook to
`execution.repository.ts`. This keeps related logic in a single place.
- Removes all marking logic from the binary data manager. This
simplifies the interface that the S3 client will meet.
- Adds basic sanity-check tests to pruning logic and execution deletion.
Out of scope:
- Improving existing pruning logic.
- Improving existing execution repository logic.
- Adjusting dir structure for filesystem mode.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
- For a saved execution, we write to disk binary data and metadata.
These two are only ever deleted via `POST /executions/delete`. No marker
file, so untouched by pruning.
- For an unsaved execution, we write to disk binary data, binary data
metadata, and a marker file at `/meta`. We later delete all three during
pruning.
- The third flow is legacy. Currently, if the execution is unsaved, we
actually store it in the DB while running the workflow and immediately
after the workflow is finished during the `onWorkflowPostExecute()` hook
we delete that execution, so the second flow applies. But formerly, we
did not store unsaved executions in the DB ("ephemeral executions") and
so we needed to write a marker file at `/persistMeta` so that, if the
ephemeral execution crashed after the step where binary data was stored,
we had a way to later delete its associated dangling binary data via a
second pruning cycle, and if the ephemeral execution succeeded, then we
immediately cleaned up the marker file at `/persistMeta` during the
`onWorkflowPostExecute()` hook.
This creation and cleanup at `/persistMeta` is still happening, but this
third flow no longer has a purpose, as we now store unsaved executions
in the DB and delete them immediately after. Hence the third flow can be
removed.
Also,
1. Consistent CORS support ~on all three webhook types~ waiting webhooks never supported CORS. I'll fix that in another PR
2. [Fixes binary-data handling when request body is text, json, or xml](https://linear.app/n8n/issue/NODE-505/webhook-binary-data-handling-fails-for-textplain-files).
3. Reduced number of middleware that each request has to go through.
4. Removed the need to maintain webhook endpoints in the auth-exception list.
5. Skip all middlewares (apart from `compression`) on Webhook routes.
6. move `multipart/form-data` support out of individual nodes
7. upgrade `formidable`
8. fix the filenames on binary-data in webhooks nodes
9. add unit tests and integration tests for webhook request handling, and increase test coverage
* Setup versionized node
* Fix node naming
* Set all possible actions
* Add Current Date operation
* Add timezone to current date
* feat add to date operator
* Change output field name to camel case
* Fix info box for luxons tip
* Feat subtract to date operation
* Feat format date operation
* Fix to node field for format date
* Feat rounding operation
* Feat get in between date operation
* Feat add extract date operation
* Add generic function for parsing date
* Remove moment methods from operations
* Change moment to luxon for the rest of the operations
* Fix Format date operation
* Fix format value
* Add timezone option for current date
* Add tests, improve workflow settings for testing, toString the results
* Change icon for V2
* Revert "Change icon for V2"
This reverts commit 46b59bea2e.
* Change workflow test name
* Fix ui bug for custom format
* Fix default value for format operation
* Fix info box for rounding operation
* Change default for units for between time operation
* Inprove fields and resort time units
* Fix extract week number
* Resolve issue with formating and timezones
* Fix field name and unit order
* ⚡ restored removed test case, sync v1 with curent master
* ⚡ parseDate update to support timestamps, tests
* Keep same field for substract and add time
* Update unit test
* Improve visibility, add iso to string option
* Update option naming
---------
Co-authored-by: Michael Kret <michael.k@radency.com>
* 🔧 Adjust base ESLint config
* 🔧 Adjust `lint` and `lintfix` in `nodes-base`
* 🔧 Include `test` and `utils` in `nodes-base`
* 📘 Convert JS tests to TS
* 👕 Apply lintfixes
* ⚡️test setup
* ⚡️fix 'testData' implicitly has an 'any' type.
* ✅ test github action file binary data reading
* ✅ checking for output binary equality
* ✅ writing files to different formats
* ✅ reading spreadsheet with different options
* ⚡️improve workflow file path replacement
* 🐛 fixing string.at() not supported in node 14.
* 🐛 trying to fix github action test error
* ⚡ fix for empty binary
* ⚡ switch for binary test
* ⚡️test helpers now return/compare json and binary (if not empty))
* ⚡ removed commented console log
---------
Co-authored-by: Michael Kret <michael.k@radency.com>
* 🧪 Add base for building unit testing within nodes
* Improve helper functions
* 🧪 If node test
* 🧪 Airtable node test
* 🧪 If node test improvements
* 🧪 Airtable node test improvements
* ♻️ cleanup node unit tests
* ♻️ refactor getting node result data to use helper method
* ⚡ removed unused variables
* ♻️ Helper to read json files
---------
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Michael Kret <michael.k@radency.com>
