Iván Ovejero
6d6e2488c6
refactor(core): Generalize binary data manager interface (no-changelog) ( #7164 )
...
Depends on: #7092 | Story:
[PAY-768](https://linear.app/n8n/issue/PAY-768 )
This PR:
- Generalizes the `IBinaryDataManager` interface.
- Adjusts `Filesystem.ts` to satisfy the interface.
- Sets up an S3 client stub to be filled in in the next PR.
- Turns `BinaryDataManager` into an injectable service.
- Adjusts the config schema and adds new validators.
Note that the PR looks large but all the main changes are in
`packages/core/src/binaryData`.
Out of scope:
- `BinaryDataManager` (now `BinaryDataService`) and `Filesystem.ts` (now
`fs.client.ts`) were slightly refactored for maintainability, but fully
overhauling them is **not** the focus of this PR, which is meant to
clear the way for the S3 implementation. Future improvements for these
two should include setting up a backwards-compatible dir structure that
makes it easier to locate binary data files to delete, removing
duplication, simplifying cloning methods, using integers for binary data
size instead of `prettyBytes()`, writing tests for existing binary data
logic, etc.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
2023-09-22 17:22:12 +02:00
Iván Ovejero
cd08c8e4c6
refactor(core): Implement soft-deletions for executions ( #7092 )
...
Based on #7065 | Story: https://linear.app/n8n/issue/PAY-771
n8n on filesystem mode marks binary data to delete on manual execution
deletion, on unsaved execution completion, and on every execution
pruning cycle. We later prune binary data in a separate cycle via these
marker files, based on the configured TTL. In the context of introducing
an S3 client to manage binary data, the filesystem mode's mark-and-prune
setup is too tightly coupled to the general binary data management
client interface.
This PR...
- Ensures the deletion of an execution causes the deletion of any binary
data associated to it. This does away with the need for binary data TTL
and simplifies the filesystem mode's mark-and-prune setup.
- Refactors all execution deletions (including pruning) to cause soft
deletions, hard-deletes soft-deleted executions based on the existing
pruning config, and adjusts execution endpoints to filter out
soft-deleted executions. This reduces DB load, and keeps binary data
around long enough for users to access it when building workflows with
unsaved executions.
- Moves all execution pruning work from an execution lifecycle hook to
`execution.repository.ts`. This keeps related logic in a single place.
- Removes all marking logic from the binary data manager. This
simplifies the interface that the S3 client will meet.
- Adds basic sanity-check tests to pruning logic and execution deletion.
Out of scope:
- Improving existing pruning logic.
- Improving existing execution repository logic.
- Adjusting dir structure for filesystem mode.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
2023-09-20 15:21:42 +02:00
कारतोफ्फेलस्क्रिप्ट™
b895ba438a
refactor(core): Reduce boilterplate code in between tests 🧹 , and fix the tests in node.js 20 (no-changelog) ( #6654 )
...
refactor(core): Reduce boilterplate code in between tests
also cleaned up some imports, and fixed the tests in node.js 20
2023-07-13 10:14:48 +02:00
Iván Ovejero
8c008f5d22
refactor(core)!: Remove basic-auth, external-jwt-auth, and no-auth options ( #6362 )
...
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
2023-06-22 20:03:47 +02:00
Iván Ovejero
fe937a0ee3
ci: Forbid skipping tests (no-changelog) ( #6199 )
2023-05-24 09:45:37 +02:00
कारतोफ्फेलस्क्रिप्ट™
e046f656fe
feat(core): Remove all floating promises. Enforce @typescript-eslint/no-floating-promises ( #6281 )
2023-05-24 02:01:45 +02:00
Iván Ovejero
06fa6f1fb3
ci: Expand ESLint to tests in BE packages (no-changelog) ( #6147 )
...
* 🔧 Adjust base ESLint config
* 🔧 Adjust `lint` and `lintfix` in `nodes-base`
* 🔧 Include `test` and `utils` in `nodes-base`
* 📘 Convert JS tests to TS
* 👕 Apply lintfixes
2023-05-02 10:37:19 +02:00
Omar Ajoue
49d838f628
fix(core): Fix broken API permissions in public API ( #5978 )
2023-04-14 11:05:42 +02:00
कारतोफ्फेलस्क्रिप्ट™
6242cac53b
ci: Refactor cli tests to speed up CI (no-changelog) ( #5718 )
...
* ci: Refactor cli tests to speed up CI (no-changelog)
* upgrade jest to address memory leaks
2023-03-17 17:24:05 +01:00
कारतोफ्फेलस्क्रिप्ट™
845f0f9d20
refactor(core): Switch over all user-management routes to use decorators ( #5115 )
2023-01-27 11:19:47 +01:00
कारतोफ्फेलस्क्रिप्ट™
c8a146ba31
ci: Fix CI timeout issues (no-changelog) ( #5250 )
2023-01-26 11:44:50 +01:00
कारतोफ्फेलस्क्रिप्ट™
0a5ab560b1
refactor: Upgrade typeorm to 0.3.x ( #5151 )
2023-01-13 18:12:22 +01:00
Michael Auerswald
b67f803cbe
feat: Add global event bus ( #4860 )
...
* fix branch
* fix deserialize, add filewriter
* add catchAll eventGroup/Name
* adding simple Redis sender and receiver to eventbus
* remove native node threads
* improve eventbus
* refactor and simplify
* more refactoring and syslog client
* more refactor, improved endpoints and eventbus
* remove local broker and receivers from mvp
* destination de/serialization
* create MessageEventBusDestinationEntity
* db migrations, load destinations at startup
* add delete destination endpoint
* pnpm merge and circular import fix
* delete destination fix
* trigger log file shuffle after size reached
* add environment variables for eventbus
* reworking event messages
* serialize to thread fix
* some refactor and lint fixing
* add emit to eventbus
* cleanup and fix sending unsent
* quicksave frontend trial
* initial EventTree vue component
* basic log streaming settings in vue
* http request code merge
* create destination settings modals
* fix eventmessage options types
* credentials are loaded
* fix and clean up frontend code
* move request code to axios
* update lock file
* merge fix
* fix redis build
* move destination interfaces into workflow pkg
* revive sentry as destination
* migration fixes and frontend cleanup
* N8N-5777 / N8N-5789 N8N-5788
* N8N-5784
* N8N-5782 removed event levels
* N8N-5790 sentry destination cleanup
* N8N-5786 and refactoring
* N8N-5809 and refactor/cleanup
* UI fixes and anonymize renaming
* N8N-5837
* N8N-5834
* fix no-items UI issues
* remove card / settings label in modal
* N8N-5842 fix
* disable webhook auth for now and update ui
* change sidebar to tabs
* remove payload option
* extend audit events with more user data
* N8N-5853 and UI revert to sidebar
* remove redis destination
* N8N-5864 / N8N-5868 / N8N-5867 / N8N-5865
* ui and licensing fixes
* add node events and info bubbles to frontend
* ui wording changes
* frontend tests
* N8N-5896 and ee rename
* improves backend tests
* merge fix
* fix backend test
* make linter happy
* remove unnecessary cfg / limit actions to owners
* fix multiple sentry DSN and anon bug
* eslint fix
* more tests and fixes
* merge fix
* fix workflow audit events
* remove 'n8n.workflow.execution.error' event
* merge fix
* lint fix
* lint fix
* review fixes
* fix merge
* prettier fixes
* merge
* review changes
* use loggerproxy
* remove catch from internal hook promises
* fix tests
* lint fix
* include review PR changes
* review changes
* delete duplicate lines from a bad merge
* decouple log-streaming UI options from public API
* logstreaming -> log-streaming for consistency
* do not make unnecessary api calls when log streaming is disabled
* prevent sentryClient.close() from being called if init failed
* fix the e2e test for log-streaming
* review changes
* cleanup
* use `private` for one last private property
* do not use node prefix package names.. just yet
* remove unused import
* fix the tests
because there is a folder called `events`, tsc-alias is messing up all imports for native events module.
https://github.com/justkey007/tsc-alias/issues/152
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
2023-01-04 09:47:48 +01:00
कारतोफ्फेलस्क्रिप्ट™
ee28213538
refactor: Use string ids on Credentials, Workflows, Tags, and Executions DB entities ( #5041 )
2023-01-02 17:42:32 +01:00
कारतोफ्फेलस्क्रिप्ट™
698d96a617
refactor: Setup typescript project references across workflow, core, and cli ( #4519 )
...
* refactor: use consistent folder structure across workflow, core, and cli
* setup typescript project references across workflow, core, and cli
2022-11-09 15:25:00 +01:00
Aaron Delasy
49c85a1df8
fix: correct all the spelling typos ( #3960 )
...
* Improve code health
Fix TS typos in local variables
Fix CSS typos in local styles
Fix typos in comments
Fix typos in strings
* Fix order of n8n setup sections in CONTRIBUTING.md
2022-09-02 16:13:17 +02:00
Iván Ovejero
52015a6f03
fix(public-api): Fix executions pagination in Postgres and Mysql
...
🐛 Fix executions pagination
2022-08-11 12:34:32 -04:00
Iván Ovejero
620525ea85
fix(cli): Fix community nodes tests on Postgres and MySQL ( #3861 )
...
* 📘 Fix type
* ⚡ Adjust constants
* 🧪 Skip failing pagination fix
* 🧪 Make truncation sequential
2022-08-11 11:02:21 +02:00
Ricardo Espinoza
b9fe707cbd
fix(public-api): fix issue paginating executions
...
* 🐛 Fix pagination issue in /executions
* ⚡ Enable all executions tests
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
2022-08-08 16:15:56 -04:00
Ricardo Espinoza
c4b63fa999
test(API): Fix issues with Public API tests ( #3515 )
2022-06-17 22:20:21 -07:00
Iván Ovejero
b8e3bcc052
refactor(core): Post-release refactorings of Public API ( #3495 )
...
* ⚡ Post-release refactorings
* 🧪 Add `--forceExit`
* 🛠 typing refactor (#3486 )
* 🐛 Fix middleware arguments
* 👕 Fix lint
* ⚡ Restore commented out block
Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com>
2022-06-14 09:32:19 -07:00
Ricardo Espinoza
a18081d749
feat: Add n8n Public API ( #3064 )
...
* ✨ Inicial setup
* ⚡ Add authentication handler
* ⚡ Add GET /users route
* ⚡ Improvements
* 👕 Fix linting issues
* ⚡ Add GET /users/:identifier endpoint
* ⚡ Add POST /users endpoint
* ⚡ Add DELETE /users/:identifier endpoint
* ⚡ Return error using express native functions
* 👕 Fix linting issue
* ⚡ Possibility to add custom middleware
* ⚡ Refactor POST /users
* ⚡ Refactor DELETE /users
* ⚡ Improve cleaning function
* ⚡ Refactor GET /users and /users/:identifier
* ⚡ Add API spec to route
* ⚡ Add raw option to response helper
* 🐛 Fix issue adding custom middleware
* ⚡ Enable includeRole parameter in GET /users/:identifier
* ⚡ Fix linting issues after merge
* ⚡ Add missing config variable
* ⚡ General improvements
⚡ asasas
* ⚡ Add POST /users tests
* Debug public API tests
* Fix both sets of tests
* ⚡ Improvements
* ⚡ Load api versions dynamically
* ⚡ Add endpoints to UM to create/delete an API Key
* ⚡ Add index to apiKey column
* 👕 Fix linting issue
* ⚡ Clean open api spec
* ⚡ Improvements
* ⚡ Skip tests
* 🐛 Fix bug with test
* ⚡ Fix issue with the open api spec
* ⚡ Fix merge issue
* ⚡ Move token enpoints from /users to /me
* ⚡ Apply feedback to openapi.yml
* ⚡ Improvements to api-key endpoints
* 🐛 Fix test to suport API dynamic loading
* ⚡ Expose swagger ui in GET /{version}/docs
* ⚡ Allow to disable public api via env variable
* ⚡ Change handlers structure
* 🚧 WIP create credential, delete credential complete
* 🐛 fix route for creating api key
* ⚡ return api key of authenticated user
* ⚡ Expose public api activation to the settings
* ⬆️ Update package-lock.json file
* ⚡ Add execution resource
* ⚡ Fix linting issues
* 🛠 conditional public api endpoints excluding
* ⚡ ️ create credential complete
* ✨ Added n8n-card component. Added spacing utility classes.
* ♻️ Made use of n8n-card in existing components.
* ✨ Added api key setup view.
* ✨ Added api keys get/create/delete actions.
* ✨ Added public api permissions handling.
* ♻️ Temporarily disabling card tests.
* ♻️ Changed translations. Storing api key only in component.
* ✨ Added utilities storybook entry
* ♻️ Changed default value for generic copy input.
* 🧹 clean up createCredential
* ⚡ Add workflow resource to openapi spec
* 🐛 Fix naming with env variable
* ⚡ Allow multifile openapi spec
* ⚡ Add POST /workflows/:workflowId/activate
* fix up view, fix issues
* remove delete api key modal
* remove unused prop
* clean up store api
* remove getter
* remove unused dispatch
* fix component size to match
* use existing components
* match figma closely
* fix bug when um is disabled in sidebar
* set copy input color
* remove unused import
* ⚡ Remove css path
* ⚡ Add POST /workflows/:workflowId/desactivate
* ⚡ Add POST /workflows
* Revert "⚡ Remove css path"
a3d0a71719b70e29433e886e5164fb⚡ PUT /workflows/:workflowId
* ⚡ Refactor workflow endpoints
* ⚡ Refactor executions endpoints
* ⚡ Fix typo
* ✅ add credentials tests
* ✅ adjust users tests
* update text
* add try it out link
* ⚡ Add delete, getAll and get to the workflow resource
* address spacing comments
* ⚡ ️ apply correct structure
* ⚡ Add missing test to user resource and fix some issues
* ⚡ Add workflow tests
* ⚡ Add missing workflow tests and fix some issues
* ⚡ Executions tests
* ⚡ finish execution tests
* ⚡ Validate credentials data depending on type
* ⚡ ️ implement review comments
* 👕 fix lint issues
* ⚡ Add apiKey to sanatizeUser
* ⚡ Fix issues with spec and tests
* ⚡ Add new structure
* ⚡ Validate credentials type and properties
* ⚡ Make all endpoints except /users independent on UM
* ⚡ Add instance base path to swagger UI
* ⚡ Remove testing endpoints
* ⚡ Fix issue with openapi tags
* ⚡ Add endpoint GET /credentialTypes/:id/schema
* 🐛 Fix issue adding json middleware to public api
* ⚡ Add API playground path to FE
* ⚡ Add telemetry and external hooks
* 🐛 Fix issue with user tests
* ⚡ Move /credentialTypes under /credentials
* ⚡ Add test to GET /credentials/schema/:id
* 🛠 refactor schema naming
* ⚡ Add DB migrations
asas
* ✅ add tests for crd apiKey
* ✨ Added API View telemetry events.
* ⚡ Remove rsync from the building process as it is missing on alpine base image
* ⚡ add missing BE telemetry events
* 🐛 Fix credential tests
* ⚡ address outstanding feedback
* 🔨 Remove move:openapi script
* ⬆️ update dependency
* ⬆️ update package-lock.json
* 👕 Fix linting issue
* 🐛 Fix package.json issue
* 🐛 fix migrations and tests
* 🐛 fix typos + naming
* 🚧 WIP fixing tests
* ⚡ Add json schema validation
* ⚡ Add missing fields to node schema
* ⚡ Add limit max upper limit
* ⚡ Rename id paths
* 🐛 Fix tests
* Add package-lock.jsonto custom dockerfile
* ⬆️ Update package-lock.json
* 🐛 Fix issue with build
* ✏️ add beta label to api view
* 🔥 Remove user endpoints
* ⚡ Add schema examples to GET /credentials/schema/:id
* 🔥 Remove user endpoints tests
* 🐛 Fix tests
* 🎨 adapt points from design review
* 🔥 remove unnecessary text-align
* ⚡ ️ update UI
* 🐛 Fix issue with executions filter
* ⚡ Add tags filter to GET /workflows
* ⚡ Add missing error messages
* ✅ add and update public api tests
* ✅ add tests for owner activiating/deactivating non-owned wfs
* 🧪 add tests for filter for tags
* 🧪 add tests for more filter params
* 🐛 fix inclusion of tags
* 🛠 enhance readability
* ⚡ ️ small refactorings
* 💄 improving readability/naming
* ⚡ Set API latest version dinamically
* Add comments to toJsonSchema function
* ⚡ Fix issue
* ⚡ Make execution data usable
* ⚡ Fix validation issue
* ⚡ Rename data field and change parameter and options
* 🐛 Fix issue parameter "detailsFieldFormat" not resolving correctly
* Skip executions tests
* skip workflow failing test
* Rename details property to data
* ⚡ Add includeData parameter
* 🐛 Fix issue with openapi spec
* 🐛 Fix linting issue
* ⚡ Fix execution schema
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com>
Co-authored-by: Alex Grozav <alex@grozav.com>
Co-authored-by: Mutasem <mutdmour@gmail.com>
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
2022-06-08 20:53:12 +02:00
