Story: https://linear.app/n8n/issue/PAY-1188
- Implement Redis hashes on the caching service, based on Micha's work
in #7747, adapted from `node-cache-manager-ioredis-yet`. Optimize
workflow ownership lookups and manual webhook lookups with Redis hashes.
- Simplify the caching service by removing all currently unused methods
and options: `enable`, `disable`, `getCache`, `keys`, `keyValues`,
`refreshFunctionEach`, `refreshFunctionMany`, `refreshTtl`, etc.
- Remove the flag `N8N_CACHE_ENABLED`. Currently some features on
`master` are broken with caching disabled, and test webhooks now rely
entirely on caching, for multi-main setup support. We originally
introduced this flag to protect against excessive memory usage, but
total cache usage is low enough that we decided to drop this setting.
Apparently this flag was also never documented.
- Overall caching service refactor: use generics, reduce branching, add
discriminants for cache kinds for better type safety, type caching
events, improve readability, remove outdated docs, etc. Also refactor
and expand caching service tests.
Follow-up to: https://github.com/n8n-io/n8n/pull/8176
---------
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Github issue / Community forum post (link here to close automatically):
---------
Co-authored-by: Giulio Andreini <andreini@netseven.it>
Co-authored-by: Marcus <marcus@n8n.io>
## Summary
This PR intends to add [Qdrant](https://qdrant.tech/) as a supported
vectorstore node to load and retrieve documents from in a workflow.
## Review / Merge checklist
- [x] PR title and summary are descriptive.
- [x] Node/credentials documentation to be updated in
https://github.com/n8n-io/n8n-docs/pull/1796.
---------
Co-authored-by: oleg <me@olegivaniv.com>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Since crypto-js was
[discontinued](1da3dabf93),
[we migrated all our backend encryption to native
crypto](https://github.com/n8n-io/n8n/pull/7556).
However I decided back then to not remove crypto-js just yet in
expressions, as I wanted to use `SubtleCrypto`. Unfortunately for that
to work, we'd need to make expressions async.
So, to get rid of `crypto-js`, I propose this interim solution.
## Related tickets and issues
N8N-7020
## Review / Merge checklist
- [x] PR title and summary are descriptive
- [x] Tests included
This reverts commit a895ee87fc (#8090)
Our telemetry backend is throwing 500s with the updated rudderstack sdk.
Until that is resolved, we need to downgrade.
## Review / Merge checklist
- [x] PR title and summary are descriptive
`@n8n/chat` was pulling in a lot of duplicate dependencies. This PR
updates all frontend packages to use an many of the same versions of
frontend devDependencies as possible.
## Review / Merge checklist
- [x] PR title and summary are descriptive
This helps remove some of the older versions of transient dependencies,
like axios 0.x and ioredis 4.x.
## Review / Merge checklist
- [x] PR title and summary are descriptive.
## Summary
We accidentally made some functions `async` in
https://github.com/n8n-io/n8n/pull/7846
This PR reverts that change.
## Review / Merge checklist
- [x] PR title and summary are descriptive.
## Summary
This PR updates our backend sentry setup to remove integrations that
don't provide us any value. This also reduces the amount of PII that
gets sent to Sentry.
[Sample event](https://n8nio.sentry.io/issues/4725315362/)
## Related tickets
[ENG-95](https://linear.app/n8n/issue/ENG-95)
## Review / Merge checklist
- [x] PR title and summary are descriptive.
## Summary
While backend development, this change explicitly logs the location and
index of a circular dependency, which helps getting rid of the circular
dependency.
## Summary
Handle circular references in the public API for executions created
prior to the fix from #8030
## Related tickets
[PAY-1119](https://linear.app/n8n/issue/PAY-1119)
## Review / Merge checklist
- [x] PR title and summary are descriptive.
## Summary
This PR removes `embeddingsTensorFlow` node and reverts our fix to
accommodate the Docker build in #7893 .
The node itself has been extracted as a [community node available from
npm](https://www.npmjs.com/package/n8n-nodes-tensorflow)
---------
Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>
- Remove Epub document loader override, use standard Langchain loader
- Use temporary buffer file for document loader processing and pass just
the path
- Remove replace `@gxl/epub-parser` library with `epub2`
Github issue / Community forum post (link here to close automatically):
---------
Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
extracted out of #7336
---------
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
Co-authored-by: Oleg Ivaniv <me@olegivaniv.com>
Co-authored-by: Alex Grozav <alex@grozav.com>
Once the packages are published to NPM, they don't have the
`pnpm-lock.yaml` to fix dependency versions. Which means that any
dependency with `^` gets auto-upgrade to the latest matching minor,
which can cause issues like
[this](https://github.com/node-cache-manager/node-cache-manager/issues/611).
This change expands on the command channel communication introduced
lately between the main instance(s) and the workers. The frontend gets a
new menu entry "Workers" which will, when opened, trigger a regular call
to getStatus from the workers. The workers then respond via their
response channel to the backend, which then pushes the status to the
frontend.
This introduces the use of ChartJS for metrics.
This feature is still in MVP state and thus disabled by default for the
moment.
Github issue / Community forum post (link here to close automatically):
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <netroy@users.noreply.github.com>
This change ensures that things like `encryptionKey` and `instanceId`
are always available directly where they are needed, instead of passing
them around throughout the code.
- Fix pinch-to-zoom
- Support command + scroll to zoom
- Improve accuracy of zooming (scroll more = zoom more)
- Zoom limits
- Zoom relative to mouse position
Depends on https://github.com/n8n-io/n8n/pull/7220 | Story:
[PAY-840](https://linear.app/n8n/issue/PAY-840/introduce-object-store-service-and-manager-for-binary-data)
This PR introduces an object store service for Enterprise edition. Note
that the service is tested but currently unused - it will be integrated
soon as a binary data manager, and later for execution data.
`amazonaws.com` in the host is temporarily hardcoded until we integrate
the service and test against AWS, Cloudflare and Backblaze, in the next
PR.
This is ready for review - the PR it depends on is approved and waiting
for CI.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
all commands sent between main instance and workers need to contain a
server id to prevent senders from reacting to their own messages,
causing loops
this PR makes sure all sent messages contain a sender id by default as
part of constructing a sending redis client.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Depends on: #7092 | Story:
[PAY-768](https://linear.app/n8n/issue/PAY-768)
This PR:
- Generalizes the `IBinaryDataManager` interface.
- Adjusts `Filesystem.ts` to satisfy the interface.
- Sets up an S3 client stub to be filled in in the next PR.
- Turns `BinaryDataManager` into an injectable service.
- Adjusts the config schema and adds new validators.
Note that the PR looks large but all the main changes are in
`packages/core/src/binaryData`.
Out of scope:
- `BinaryDataManager` (now `BinaryDataService`) and `Filesystem.ts` (now
`fs.client.ts`) were slightly refactored for maintainability, but fully
overhauling them is **not** the focus of this PR, which is meant to
clear the way for the S3 implementation. Future improvements for these
two should include setting up a backwards-compatible dir structure that
makes it easier to locate binary data files to delete, removing
duplication, simplifying cloning methods, using integers for binary data
size instead of `prettyBytes()`, writing tests for existing binary data
logic, etc.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Github issue / Community forum post (link here to close automatically):
---------
Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
This PR implements the updated license SDK so that worker and webhook
instances do not auto-renew licenses any more.
Instead, they receive a `reloadLicense` command via the Redis client
that will fetch the updated license after it was saved on the main
instance
This also contains some refactoring with moving redis sub and pub
clients into the event bus directly, to prevent cyclic dependency
issues.
* fix(editor): Disable telemetry in dev mode and in E2E tests
* fix(editor): add external js to rollup config
* fix(editor): using vite plugin to control external script loading
* fix(editor): update lock file version
* fix(editor): block telemetry host
Also,
1. Consistent CORS support ~on all three webhook types~ waiting webhooks never supported CORS. I'll fix that in another PR
2. [Fixes binary-data handling when request body is text, json, or xml](https://linear.app/n8n/issue/NODE-505/webhook-binary-data-handling-fails-for-textplain-files).
3. Reduced number of middleware that each request has to go through.
4. Removed the need to maintain webhook endpoints in the auth-exception list.
5. Skip all middlewares (apart from `compression`) on Webhook routes.
6. move `multipart/form-data` support out of individual nodes
7. upgrade `formidable`
8. fix the filenames on binary-data in webhooks nodes
9. add unit tests and integration tests for webhook request handling, and increase test coverage
* revert(core): Remove typeorm patches, but still enforce transactions on every migration
This reverts #6519
* always re-enable foreign keys, and explicitly rollback transaction
* feat(core): Change node execution order (most top-left one first)
* ⚡ Fix issue with multi-output-nodes
* ⚡ Remove not needed meta-entry in test
* fix the e2e test
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* Draft setup
* ⚡ Implemented expression evaluation in Postgres node, minor SQL editor UI improvements, minor refacring
* ⚡ Added initial version of expression preview for SQL editor
* ⚡ Linking npm package for codemirror sql grammar instead of a local file
* ⚡ Moving expression editor wrapper elements to the component
* ⚡ Using expression preview in SQL editor
* Use SQL parser skipping whitespace
* ✨ Added support for custom skipped segments specification
* ✨ Fixing highlight problems with dots and expressions that resolve to zero
* 👕 Fixing linting error
* ✨ Added current item support
* ⚡ Added expression support to more nodes with sql editor
* ✨ Added expression support for other nodes
* ✨ Implemented different SQL dialect support
* 🐛 Fixing hard-coded parameter names for editors
* ✨ Fixing preview for nested queries, updating query when input data changes, adding keyboard shortcut to toggle comments
* ✨ Adding a custom automcomplete notice for different editors
* ⚡ Updating SQL autocomplete notice
* ✅ Added unit tests for SQL editor
* ⚡ Using latest grammar
* 🐛 Fixing code node editor rendering
* 💄 SQL preview dropdown matches editor width. Removing unnecessary css
* ⚡ Addressing PR review feedback
* 👌 Addressing PR review feedback pt2
* 👌 Added path alias for utils in nodes-base package
* 👌 Addressing more PR review feedback
* ✅ Adding tests for `getResolvables` utility function
* ⚡Fixing lodash imports
* 👌 Better focus handling, adding more plugins to the editor, other minor imrovements
* ⚡ Not showing SQL autocomplete suggestions inside expressions
* ⚡ Using npm package for sql grammar
* ⚡ Removing autocomplete notice, adding line highlight on syntax error
* 👌 Addressing code review feedback
---------
Co-authored-by: Milorad Filipovic <milorad@n8n.io>
* Remove authorization header when empty
* Import pkce
* Add OAuth2 with new grant type to Twitter
* Add pkce logic auto assign authorization code if pkce not defined
* Add pkce to ui and interfaces
* Fix scopes for Oauth2 twitter
* Deubg + pass it through header
* Add debug console, add airtable cred
* Remove all console.logs, make PKCE in th body only when it exists
* Remove invalid character ~
* Remove more console.logs
* remove body inside query
* Remove useless grantype check
* Hide oauth2 twitter waiting for overhaul
* Remove redundant header removal
* Remove more console.logs
* Add comment for code verifier
* Remove uneeded scopes
* Restore client id in callback
* Revert "Add OAuth2 with new grant type to Twitter"
This reverts commit 1c3b331aa1.
* Remove oauth2 from twitter
* Remove properties linked to oauth2
* Fix lodash imports
* remove redundant check
* remove redundant codeVerifier
* patch pkce-challenge to avoid generating `code_verifier` with `~`
* store `codeVerifier` on the DB like `csrfSecret`
* remove unrelated changes
---------
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* first commit for postgres migration
* (not working)
* sqlite migration
* quicksave
* fix tests
* fix pg test
* fix postgres
* fix variables import
* fix execution saving
* add user settings fix
* change migration to single lines
* patch preferences endpoint
* cleanup
* improve variable import
* cleanup unusued code
* Update packages/cli/src/PublicApi/v1/handlers/workflows/workflows.handler.ts
Co-authored-by: Omar Ajoue <krynble@gmail.com>
* address review notes
* fix var update/import
* refactor: Separate execution data to its own table (#6323)
* wip: Temporary migration process
* refactor: Create boilerplate repository methods for executions
* fix: Lint issues
* refactor: Added search endpoint to repository
* refactor: Make the execution list work again
* wip: Updating how we create and update executions everywhere
* fix: Lint issues and remove most of the direct access to execution model
* refactor: Remove includeWorkflowData flag and fix more tests
* fix: Lint issues
* fix: Fixed ordering of executions for FE, removed transaction when saving execution and removed unnecessary update
* refactor: Add comment about missing feature
* refactor: Refactor counting executions
* refactor: Add migration for other dbms and fix issues found
* refactor: Fix lint issues
* refactor: Remove unnecessary comment and auto inject repo to internal hooks
* refactor: remove type assertion
* fix: Fix broken tests
* fix: Remove unnecessary import
* Remove unnecessary toString() call
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* fix: Address comments after review
* refactor: Remove unused import
* fix: Lint issues
* fix: Add correct migration files
---------
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* remove null values from credential export
* fix: Fix an issue with queue mode where all running execution would be returned
* fix: Update n8n node to allow for workflow ids with letters
* set upstream on set branch
* remove typo
* add nodeAccess to credentials
* fix unsaved run check for undefined id
* fix(core): Rename version control feature to source control (#6480)
* rename versionControl to sourceControl
* fix source control tooltip wording
---------
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
* fix(editor): Pay 548 hide the set up version control button (#6485)
* feat(DebugHelper Node): Fix and include in main app (#6406)
* improve node a bit
* fixing continueOnFail() ton contain error in json
* improve pairedItem
* fix random data returning object results
* fix nanoId length typo
* update pnpm-lock file
---------
Co-authored-by: Marcus <marcus@n8n.io>
* fix(editor): Remove setup source control CTA button
* fix(editor): Remove setup source control CTA button
---------
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
* fix(editor): Update source control docs links (#6488)
* feat(DebugHelper Node): Fix and include in main app (#6406)
* improve node a bit
* fixing continueOnFail() ton contain error in json
* improve pairedItem
* fix random data returning object results
* fix nanoId length typo
* update pnpm-lock file
---------
Co-authored-by: Marcus <marcus@n8n.io>
* feat(editor): Replace root events with event bus events (no-changelog) (#6454)
* feat: replace root events with event bus events
* fix: prevent cypress from replacing global with globalThis in import path
* feat: remove emitter mixin
* fix: replace component events with event bus
* fix: fix linting issue
* fix: fix breaking expression switch
* chore: prettify ndv e2e suite code
* fix(editor): Update source control docs links
---------
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
* fix tag endpoint regex
---------
Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
Co-authored-by: Csaba Tuncsik <csaba@n8n.io>
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
* test(editor): test n8n-select design system component
* test(editor): test version control settings happy path
* test(editor): test version control settings disconnect
* ci: Start supporting Node.js 18
Node.js 18 became the active LTS on 2022-10-25, and Node.js 16 went into maintenance mode.
https://github.com/nodejs/Release#release-schedule
We should also slowly deprecate node 16 support, [as support for it is ends much earlier now, due to support for openssl 1.1.1 ending](https://nodejs.org/en/blog/announcements/nodejs16-eol).
* Remove hashing algorithms that are not available in newer node.js/openssl
- RSA-MD4
- RSA-MDC2
- md4
- md4WithRSAEncryption
- mdc2
- mdc2WithRSA
* in e2e tests, resolve `localhost` to ipv4 instead of ipv6
* ⚡ Add user activation survey
* Fix typo
* Avoid showing the modal when there is a modal view
* Allow to redirect to specific execution
* Improve structure
* Handle errors when sharing feedback
* update withFeatureFlag function
* Fix linting issue
* Set user activation flag on workflowExecutionCompleted event
* Revert update user settings functionality
* Remove unnecessary changes
* fix linting issue
* account for new functionality in tests
* Small improvements
* keep once instace of the model open between tabs
* Add sorting to GET /executions
* type parameters for GET /executions
a
* Add constant for local store key
* Add execution mode filtering
* fix linting issue
* Do not override settings when setting isOnboarded true
* Add update user settings endpoint
* improvements
* revert changes to /GET executions
* Fix typo
* Add userActivated flag to user store
* Add E2E test
* Fix linting issue
* Update pnpm-lock
* Revert unnecessary change
* Centralize user's settings update
* Remove unused ref in userActivationSurvey modal
* Use aliased imports
* Use createEventBus function in component
* Fix tests
* fix(editor): Fix execution list item selection
* fix(editor): Delete only selected executions
* fix(editor): Fix clear selection
* fix(editor): Fix clear selection
* fix(editor): Fix clear selection
* feat(editor): Add select all existing executions checkbox
* fix(editor): Do not mark later loaded executions selected
* test(editor): Add execution list unit test
* fix(editor): Fix selection
* test(editor): update execution selection test
* fix(editor): Handle UI state when there is no execution
* fix(editor): Remove unnecessary logic
* test(editor): Add more execution list unit tests and fake data generation
* test(editor): Add more execution list unit tests
* test(editor): Simplifying test setup
* chore: update pnpm lock after resolving merge conflocts
* chore: fix package version
* fix: Improved executions deletion to prevent crashing and fixed removal of failed executions
* fix: Add comment to clarify why change was needed
* fix: fix executions list bug when selecting all and changing filter
* fix: fix execution lists running execution showing up on different workflow id
* fix(editor): Deleting an execution while all are selected
* fix(editor): Deleting an execution while all are selected
---------
Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: Alex Grozav <alex@grozav.com>
* WIP: Cypress parallel CI run test
* Trigger action on branch push
* Change build artifacts path
* Make sure to checkout the repo for testing job
* Use Cypress action for installing
* Lock cypress action userd version
* Skip node install step since we're using cypress node16 container
* Let Cypress handle pnpm install
* Use setup-node action for caching pnpm
* Set CYPRESS_CACHE_FOLDER
* Set CYPRESS_CACHE_FOLDER
* Manually cache pnpm store
* Dont fix pnpm version
* Use caching action also in testing job
* Zip packages dist before uploading the artifacts and change caching key
* Use absolute build paths for zipping job
* Use zip command in action
* Use tar for zipping packages
* Debuggin directory ls
* Debugging caching of modules
* Attempt to fix permissions issue
* Porivde Cypress executable via `CYPRESS_RUN_BINARY`
* Cache /github/home
* Adjust caching keys
* Debug: search for cypress exec
* Debugging: List dirs
* Use pnpm install action to install node_modules
* Do not log /home/runner
* Use node_modules/.bin Cypress binary
* Use absolute path to nodue modules
* Run Cypress via custom command
* Try with patched cypress action
* Revert logging
* Manually specify cypress config file
* Use absolute paths
* Fix cypress config name
* Debug print cypress config
* Remove debugging, increase to 4 containers
* Increase amount of containers
* Add env-version matrix
* Replace node14 with node18 in testing matrix
* Remove debugging and add node 14
* Use just node14
* Use cypress:base and remove browser req
* Give more general timeouts
* Try with node16
* Change cache directive position
* Replace zip artifact upload with cache
* Cache full packages not just dist
* Test with variable inputs
* Add commit info message
* Remove wrongly commited code
* Allow WF API dispatch
* Try Chrome browser again for comparison
* Include Monaco in the build
* Make e2e workflow re-usable
* Comment out invalid reusable workflow args
* Use electron and add node 14 run
* Fix env arg
* Provide custom ci-build-id
* Refactor remaining e2e workflow to use reusable action
* Remove single matrix directive
* Refactor ci-pull-req
* Make lint job dependant on test jobs
* Disable debugging job
* Make containers dynamic
* Cleanup & install git for linting action
* Use regular buntu image for PR linting
* Debugging failing tests
* Remove fixed spec name
* Debug e2e env var
* Do not use realkeypress which crashes electron runner
* Debugging
* chore: remove console
* chore: remove console
* test: remove node 14 tests
* test: replace test branch with master
* test: use tests in current branch
* test: use relative path
* chore: clean up
* test: only trigger on approval
* ci: update test PR
* ci: use curr branch
* ci: only run 14 on schedule, not for slack command
* ci: only run test on approval
* ci: clean up branch, rename step
* ci: rename steps
* ci: clean up cancel
* ci: clean up env var
* ci: set var
* ci: use chromef
* ci: use electron
* chore: add console log
* chore: add console log
* ci: update to string
* ci: set all env options
* test: build
* ci: fix step issue
* Fix failing tests & upgrade to Cypress 12
* Allow WF dispatch of e2e reusable
* Fix wrong naming in e2e-tests workflow
* Redeploy
* Fix tests
* Fix NDV tests and remove skipping of webhooks execution tests
* Fix clipboard read command
* Fix execution failing tests
* Reset before each 15 and 3
* Fix flaky tests
* Cleanup and log envs
* Test fixes
* Default owner spec fixes
* Get rid of CYPRESS_RUN_ENV
* Increase amount of containers, cleanup and add mock for credentials test call
* Cleanup & fix PR tests unit tests
* Wait for WF to loade in sharing spec
* Do linting and unit tests first
* Use frozen lockfile
* Revert back ci pull request jobs order
* Refine credential input selector and move cy.waitForLoad to correct position in 15-scheduler spec
* test: build
* Wait for WF execution instead of arbitraty timeout in WF execution spec, change order of jobs for ci pull request
* Fix flaky 3-default owner spec and wait for execution list to load in 20-workflow-executions
* Use setup node action
* Remove caching for lint/unit tests
* Experiment with parallel test & lint on ci
* Provide cache key dynamically
* Run e2e in parallel on pr
* Only run node14 e2e on daily schedule
* Make sure to generate generate new ci-build-id on re-runs
* Remove debugging prints
* Address PR comments
* Rename custom onBeforeUnload handler
* Make sure 19-execution spec waits for wf to load properly before import fixtures
---------
Co-authored-by: Mutasem <mutdmour@gmail.com>
* add typedi
* convert ActiveWorkflowRunner into an injectable service
* convert ExternalHooks into an injectable service
* convert InternalHooks into an injectable service
* convert LoadNodesAndCredentials into an injectable service
* convert NodeTypes and CredentialTypes into an injectable service
* convert ActiveExecutions into an injectable service
* convert WaitTracker into an injectable service
* convert Push into an injectable service
* convert ActiveWebhooks and TestWebhooks into an injectable services
* handle circular references, and log errors when a circular dependency is found
