This change expands on the command channel communication introduced
lately between the main instance(s) and the workers. The frontend gets a
new menu entry "Workers" which will, when opened, trigger a regular call
to getStatus from the workers. The workers then respond via their
response channel to the backend, which then pushes the status to the
frontend.
This introduces the use of ChartJS for metrics.
This feature is still in MVP state and thus disabled by default for the
moment.
https://linear.app/n8n/issue/PAY-933/set-up-leader-selection-for-multiple-main-instances
- [x] Set up new envs
- [x] Add config and license checks
- [x] Implement `MultiMainInstancePublisher`
- [x] Expand `RedisServicePubSubPublisher` to support
`MultiMainInstancePublisher`
- [x] Init `MultiMainInstancePublisher` on startup and destroy on
shutdown
- [x] Add to sandbox plans
- [x] Test manually
Note: This is only for setup - coordinating in reaction to leadership
changes will come in later PRs.
This change ensures that things like `encryptionKey` and `instanceId`
are always available directly where they are needed, instead of passing
them around throughout the code.
This fixes a bug in the pruning (soft-delete). The pruning was a bit too
aggressive, as it also pruned executions that weren't in an end state
yet. This only becomes an issue if there are long-running executions
(e.g. workflow with Wait node) or the prune parameters are set to keep
only a tiny number of executions.
Based on #7065 | Story: https://linear.app/n8n/issue/PAY-771
n8n on filesystem mode marks binary data to delete on manual execution
deletion, on unsaved execution completion, and on every execution
pruning cycle. We later prune binary data in a separate cycle via these
marker files, based on the configured TTL. In the context of introducing
an S3 client to manage binary data, the filesystem mode's mark-and-prune
setup is too tightly coupled to the general binary data management
client interface.
This PR...
- Ensures the deletion of an execution causes the deletion of any binary
data associated to it. This does away with the need for binary data TTL
and simplifies the filesystem mode's mark-and-prune setup.
- Refactors all execution deletions (including pruning) to cause soft
deletions, hard-deletes soft-deleted executions based on the existing
pruning config, and adjusts execution endpoints to filter out
soft-deleted executions. This reduces DB load, and keeps binary data
around long enough for users to access it when building workflows with
unsaved executions.
- Moves all execution pruning work from an execution lifecycle hook to
`execution.repository.ts`. This keeps related logic in a single place.
- Removes all marking logic from the binary data manager. This
simplifies the interface that the S3 client will meet.
- Adds basic sanity-check tests to pruning logic and execution deletion.
Out of scope:
- Improving existing pruning logic.
- Improving existing execution repository logic.
- Adjusting dir structure for filesystem mode.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* first commit for postgres migration
* (not working)
* sqlite migration
* quicksave
* fix tests
* fix pg test
* fix postgres
* fix variables import
* fix execution saving
* add user settings fix
* change migration to single lines
* patch preferences endpoint
* cleanup
* improve variable import
* cleanup unusued code
* Update packages/cli/src/PublicApi/v1/handlers/workflows/workflows.handler.ts
Co-authored-by: Omar Ajoue <krynble@gmail.com>
* address review notes
* fix var update/import
* refactor: Separate execution data to its own table (#6323)
* wip: Temporary migration process
* refactor: Create boilerplate repository methods for executions
* fix: Lint issues
* refactor: Added search endpoint to repository
* refactor: Make the execution list work again
* wip: Updating how we create and update executions everywhere
* fix: Lint issues and remove most of the direct access to execution model
* refactor: Remove includeWorkflowData flag and fix more tests
* fix: Lint issues
* fix: Fixed ordering of executions for FE, removed transaction when saving execution and removed unnecessary update
* refactor: Add comment about missing feature
* refactor: Refactor counting executions
* refactor: Add migration for other dbms and fix issues found
* refactor: Fix lint issues
* refactor: Remove unnecessary comment and auto inject repo to internal hooks
* refactor: remove type assertion
* fix: Fix broken tests
* fix: Remove unnecessary import
* Remove unnecessary toString() call
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* fix: Address comments after review
* refactor: Remove unused import
* fix: Lint issues
* fix: Add correct migration files
---------
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* remove null values from credential export
* fix: Fix an issue with queue mode where all running execution would be returned
* fix: Update n8n node to allow for workflow ids with letters
* set upstream on set branch
* remove typo
* add nodeAccess to credentials
* fix unsaved run check for undefined id
* fix(core): Rename version control feature to source control (#6480)
* rename versionControl to sourceControl
* fix source control tooltip wording
---------
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
* fix(editor): Pay 548 hide the set up version control button (#6485)
* feat(DebugHelper Node): Fix and include in main app (#6406)
* improve node a bit
* fixing continueOnFail() ton contain error in json
* improve pairedItem
* fix random data returning object results
* fix nanoId length typo
* update pnpm-lock file
---------
Co-authored-by: Marcus <marcus@n8n.io>
* fix(editor): Remove setup source control CTA button
* fix(editor): Remove setup source control CTA button
---------
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
* fix(editor): Update source control docs links (#6488)
* feat(DebugHelper Node): Fix and include in main app (#6406)
* improve node a bit
* fixing continueOnFail() ton contain error in json
* improve pairedItem
* fix random data returning object results
* fix nanoId length typo
* update pnpm-lock file
---------
Co-authored-by: Marcus <marcus@n8n.io>
* feat(editor): Replace root events with event bus events (no-changelog) (#6454)
* feat: replace root events with event bus events
* fix: prevent cypress from replacing global with globalThis in import path
* feat: remove emitter mixin
* fix: replace component events with event bus
* fix: fix linting issue
* fix: fix breaking expression switch
* chore: prettify ndv e2e suite code
* fix(editor): Update source control docs links
---------
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
* fix tag endpoint regex
---------
Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
Co-authored-by: Csaba Tuncsik <csaba@n8n.io>
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
* Add ExecutionsUsage component
* set $sidebar-expanded-width back to 200px
* add days using interpolation
* Rename PlanData type to CloudPlanData
* Rename Metadata type to PlanMetadata
* Make prop block in the update button
* Use variable in line-height
* Remove progressBarSection class
* fix trial expiration calculation
* mock expirationDate and fix issue with days left
* Remove unnecesary property from class .container
* inject component data via props
* Check for plan data during app mounting and keep data in the store
* Remove mounted hook
* redirect when upgrade plan is clicked
* Remove computed properties
* Remove instance property as it's not needed anymore
* Flatten plan object
* remove console.log
* Add all cloud types within its own namespace
* keep redirection inside component
* get computed properties back
* Improve polling logic
* Move cloudData to its own store
* Remove commented interfaces
* remove cloudPlan from user store
* fix imports
* update logic for userIsTrialing method
* centralize userIsTrialing method
* redirect to production change plan page always
* Call staging or production cloud api depending on base URL
* remove setting store form ExecutionUsage.vue
* fix linting issue
* Add trial group to PlanMetadata group
* Move helpers into the store
* make staging url check more specific
* make cloud state nullable
* fix linting issue
* swap mockup date for endpoint
* Make getCurrentPlan async
* asas
* Improvements
* small improvements
* chore: resolve conflicts
* make sure there is data before calculating trial expiration
* Fix issue with component not loading on first page load
* type safety improvements
* apply component ui feedback
* fix linting issue
* chore: clean up unnecessary change from merge conflict
* feat: Block api feature using licenses, show notice page for trial cloud users (#6187)
* rename planSpec to plan
* Remove instance property as it's not needed anymore
* Flatten plan object
* remove console.log
* feat: disable api using license
* feat: add api page
* chore: resolve conflicts
* chore: resolve conflicts
* feat: update and refactor a bit
* fix: update endpoints
* fix: update endpoints
* fix: use host
* feat: update copy
* fix linting issues
---------
Co-authored-by: ricardo <ricardoespinoza105@gmail.com>
* add pluralization to days left text
---------
Co-authored-by: Mutasem <mutdmour@gmail.com>
Co-authored-by: Mutasem Aldmour <4711238+mutdmour@users.noreply.github.com>
* WIP: Cypress parallel CI run test
* Trigger action on branch push
* Change build artifacts path
* Make sure to checkout the repo for testing job
* Use Cypress action for installing
* Lock cypress action userd version
* Skip node install step since we're using cypress node16 container
* Let Cypress handle pnpm install
* Use setup-node action for caching pnpm
* Set CYPRESS_CACHE_FOLDER
* Set CYPRESS_CACHE_FOLDER
* Manually cache pnpm store
* Dont fix pnpm version
* Use caching action also in testing job
* Zip packages dist before uploading the artifacts and change caching key
* Use absolute build paths for zipping job
* Use zip command in action
* Use tar for zipping packages
* Debuggin directory ls
* Debugging caching of modules
* Attempt to fix permissions issue
* Porivde Cypress executable via `CYPRESS_RUN_BINARY`
* Cache /github/home
* Adjust caching keys
* Debug: search for cypress exec
* Debugging: List dirs
* Use pnpm install action to install node_modules
* Do not log /home/runner
* Use node_modules/.bin Cypress binary
* Use absolute path to nodue modules
* Run Cypress via custom command
* Try with patched cypress action
* Revert logging
* Manually specify cypress config file
* Use absolute paths
* Fix cypress config name
* Debug print cypress config
* Remove debugging, increase to 4 containers
* Increase amount of containers
* Add env-version matrix
* Replace node14 with node18 in testing matrix
* Remove debugging and add node 14
* Use just node14
* Use cypress:base and remove browser req
* Give more general timeouts
* Try with node16
* Change cache directive position
* Replace zip artifact upload with cache
* Cache full packages not just dist
* Test with variable inputs
* Add commit info message
* Remove wrongly commited code
* Allow WF API dispatch
* Try Chrome browser again for comparison
* Include Monaco in the build
* Make e2e workflow re-usable
* Comment out invalid reusable workflow args
* Use electron and add node 14 run
* Fix env arg
* Provide custom ci-build-id
* Refactor remaining e2e workflow to use reusable action
* Remove single matrix directive
* Refactor ci-pull-req
* Make lint job dependant on test jobs
* Disable debugging job
* Make containers dynamic
* Cleanup & install git for linting action
* Use regular buntu image for PR linting
* Debugging failing tests
* Remove fixed spec name
* Debug e2e env var
* Do not use realkeypress which crashes electron runner
* Debugging
* chore: remove console
* chore: remove console
* test: remove node 14 tests
* test: replace test branch with master
* test: use tests in current branch
* test: use relative path
* chore: clean up
* test: only trigger on approval
* ci: update test PR
* ci: use curr branch
* ci: only run 14 on schedule, not for slack command
* ci: only run test on approval
* ci: clean up branch, rename step
* ci: rename steps
* ci: clean up cancel
* ci: clean up env var
* ci: set var
* ci: use chromef
* ci: use electron
* chore: add console log
* chore: add console log
* ci: update to string
* ci: set all env options
* test: build
* ci: fix step issue
* Fix failing tests & upgrade to Cypress 12
* Allow WF dispatch of e2e reusable
* Fix wrong naming in e2e-tests workflow
* Redeploy
* Fix tests
* Fix NDV tests and remove skipping of webhooks execution tests
* Fix clipboard read command
* Fix execution failing tests
* Reset before each 15 and 3
* Fix flaky tests
* Cleanup and log envs
* Test fixes
* Default owner spec fixes
* Get rid of CYPRESS_RUN_ENV
* Increase amount of containers, cleanup and add mock for credentials test call
* Cleanup & fix PR tests unit tests
* Wait for WF to loade in sharing spec
* Do linting and unit tests first
* Use frozen lockfile
* Revert back ci pull request jobs order
* Refine credential input selector and move cy.waitForLoad to correct position in 15-scheduler spec
* test: build
* Wait for WF execution instead of arbitraty timeout in WF execution spec, change order of jobs for ci pull request
* Fix flaky 3-default owner spec and wait for execution list to load in 20-workflow-executions
* Use setup node action
* Remove caching for lint/unit tests
* Experiment with parallel test & lint on ci
* Provide cache key dynamically
* Run e2e in parallel on pr
* Only run node14 e2e on daily schedule
* Make sure to generate generate new ci-build-id on re-runs
* Remove debugging prints
* Address PR comments
* Rename custom onBeforeUnload handler
* Make sure 19-execution spec waits for wf to load properly before import fixtures
---------
Co-authored-by: Mutasem <mutdmour@gmail.com>
* feat(core): Fix populating of node custom api call options
* lint fixes
* Adress PR comments
* Add e2e test and only inject custom API options for latest version
* Make sure to injectCustomApiCallOption for the latest version of node
* feat(cli): Move apiCallOption injection to LoadNodesAndCredentials and add e2e tests to check for custom nodes credentials
* Load nodes and credentials fixtures from a single place
* Console warning if credential is invalid during customApiOptions injection
* store n8n version string in a const and use that everywhere
* reduce code duplication between Server and WebhookServer
* unify redis checks
* fix linting
* feat: redact password field values in credentials
* feat: disable expanding password fields
* fix: redacting credentials without a valid type
This only seems to be a thing in testing?
* feat: Reduce initial memory spike at server startup (no-changelog)
This changes the frontend types generation to generate less garbage for the GC to collect.
* switch to stream pipelines for writing all the static files
and, move all static file generation before the server starts
* ✨ Make it possible to dynamically load node packages
* ⚡ Fix comment
* ✨ Make possible to dynamically install nodes from npm
* Created migration for sqlite regarding community nodes
* Saving to db whenever a package with nodes is installed
* Created endpoint to fetch installed packages
* WIP - uninstall package with nodes
* Fix lint issues
* Updating nodes via API
* Lint and improvement fixes
* Created community node helpers and removed packages taht do not contain nodes
* Check for package updates when fetching installed packages
* Blocked access to non-owner and preventing incorrect install of packages
* Added auto healing process
* Unit tests for helpers
* Finishing tests for helpers
* Improved unit tests, refactored more helpers and created integration tests for GET
* Implemented detection of missing packages on init and added warning to frontend settings
* Add check for banned packages and fix broken tests
* Create migrations for other db systems
* Updated with latest changes from master
* Fixed conflict errors
* Improved unit tests, refactored more helpers and created integration tests for GET
* Implemented detection of missing packages on init and added warning to frontend settings
* 🔥 Removing access check for the Settings sidebar item
* ✨ Added inital community nodes settings screen
* ⚡Added executionMode flag to settings
* ✨ Implemented N8N-callout component
* 💄Updating Callout component template propery names
* 💄 Updating Callout component styling.
* 💄Updating Callout component sizing and colors.
* ✔️ Updating Callout component test snapshots after styling changes
* ✨ Updating the `ActionBox` component so it supports callouts and conditional button rendering
* 💄 Removing duplicate callout theme validation in the `ActionBox` component. Adding a selection control for it in the storybook.
* ✨ Added warning message if instance is in the queue mode. Updated colors based on the new design.
* ⚡ Added a custom permission support to router
* 🔨 Implemented UM detection as a custom permission.
* 👌Updating route permission logic.
* ✨ Implemented installed community packages list in the settings view
* 👌 Updating settings routes rules and community nodes setting view.
* Allow installation of packages that failed to load
* 👌 Updating `ActionBox`, `CommuntyPackageCard` components and settings loading logic.
* 👌 Fixing community nodes loading state and sidebar icon spacing.
* ✨ Implemented loading skeletons for community package cards
* 👌 Handling errrors while loading installed package list. Updating spacing.
* 👌 Updating community nodes error messages.
* Added disable flag
* 🐛 Fixing a community nodes update detection bug when there are missing packages. (#3497)
* ✨ Added front-end support for community nodes feature flag
* ✨ Implemented community package installation modal dialog
* 💄 Community nodes installation modal updates: Moved links to constants and used them in translations, disabling inputs in loading state.
* ✨ Implemented community packages install flow
* Standardize error codes (#3501)
* Standardize error: 400 for request issues such as invalid package name and 500 for installation problems
* Fix http status code for when package is not found
* ✨ Implemented community package installation modal dialog
* 💄 Community nodes installation modal updates: Moved links to constants and used them in translations, disabling inputs in loading state.
* ✨ Implemented community packages install flow
* ✨ Updated error handling based on the response codes
* ✨ Implemented community package installation modal dialog
* ✨ Implemented community package uninstall flow.
* ✨ Finished update confirm modal UI
* 💄 Replaced community nodes tooltip image with the one exported from figma.
* ✨ Implemented community package update process
* ✨ Updating community nodes list after successful package update
* 🔒 Updating public API setting route to use new access rules. Updating express app definition in community nodes tests
* ✨ Implemented community package installation modal dialog
* 💄 Community nodes installation modal updates: Moved links to constants and used them in translations, disabling inputs in loading state.
* ✨ Implemented community packages install flow
* ✨ Updated error handling based on the response codes
* Change output for installation request
* Improve payload for update requests
* 👌 Updating community nodes install modal UI
* 👌 Updating community nodes confirm modal logic
* 👌 Refactoring community nodes confirm modal dialog
* 👌 Separating community nodes components loading states
* 💄 Updating community nodes install modal spacing.
* Fix behavior for installing already installed packages
* 💡 Commenting community nodes install process
* 🔥 Removing leftover commits of deleted Vue mutations
* ✨ Updated node list to identify community nodes and handle node name clash
* ✨ Implemented missing community node dialog.
* 💄 Updating n8n-tabs component to support tooltips
* ✨ Updating node details with community node details.
* 🔨 Using back-end response when updating community packages
* 👌 Updating tabs component and refactoring community nodes store mutations
* 👌 Adding community node flag to node type descriptions and using it to identify community nodes
* 👌 Hiding unnecessary elements from missing node details panel.
* 👌 Updating missing node type descriptions for custom and community nodes
* 👌 Updating community node package name detection logic
* 👌 Removing communityNode flag from node description
* ✨ Adding `force` flag to credentials fetching (#3527)
* ✨ Adding `force` flag to credentials fetching which can be used to skip check when loading credentials
* ✨ Forcing credentials loading when opening nodeView
* 👌 Minor updates to community nodes details panel
* tests for post endpoint
* duplicate comments
* Add Patch and Delete enpoints tests
* 🔒 Using `pageCategory`prop to assemble the list of settings routes instead of hard-coded array (#3562)
* 📈 Added front-end telemetry events for community nodes
* 📈 Updating community nodes telemetry events
* 💄 Updating community nodes settings UI elements based on product/design review
* 💄 Updating node view & node details view for community nodes based on product/design feedback
* 💄 Fixing community node text capitalisation
* ✨ Adding community node install error message under the package name input field
* Fixed and improved tests
* Fix lint issue
* feat: Migrated to npm release of riot-tmpl fork.
* 📈 Updating community nodes telemetry events based on the product review
* 💄 Updating community nodes UI based on the design feedback
* 🔀 Merging recent node draggable panels changes
* Implement self healing process
* Improve error messages for package name requirement and disk space
* 💄 Removing front-end error message override since appropriate response is available from the back-end
* Fix lint issues
* Fix installed node name
* 💄 Removed additional node name parsing
* 📈 Updating community nodes telemetry events
* Fix postgres migration for cascading nodes when package is removed
* Remove postman mock for banned packages
* 📈 Adding missing telemetry event for community node documentation click
* 🐛 Fixing community nodes UI bugs reported during the bug bash
* Fix issue with uninstalling packages not reflecting UI
* 🐛 Fixing a missing node type bug when trying to run a workflow.
* Improve error detection for installing packages
* 💄 Updating community nodes components styling and wording based on the product feedback
* Implement telemetry be events
* Add author name and email to packages
* Fix telemetry be events for community packages
* 📈 Updating front-end telemetry events with community nodes author data
* 💄 Updating credentials documentation link logic to handle community nodes credentials
* 🐛 Fixing draggable panels logic
* Fix duplicate wrong import
* 💄 Hiding community nodes credentials documentation links when they don't contain an absolute URL
* Fix issue with detection of missing packages
* 💄 Adding the `Docs` tab to community nodes
* 💄 Adding a failed loading indicator to community nodes list
* Prevent n8n from crashing on startup
* Refactor and improve code quality
* ⚡ Remove not needed depenedency
Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: Milorad Filipović <milorad@n8n.io>
Co-authored-by: Milorad FIlipović <miloradfilipovic19@gmail.com>
Co-authored-by: agobrech <ael.gobrecht@gmail.com>
Co-authored-by: Alex Grozav <alex@grozav.com>
