This PR introduces the possibility of inviting new users with an `admin`
role and changing the role of already invited users.
Also using scoped permission checks where applicable instead of using
user role checks.
---------
Co-authored-by: Val <68596159+valya@users.noreply.github.com>
Co-authored-by: Alex Grozav <alex@grozav.com>
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
## Summary
Extend existing user types in the E2E database. Currently, we have only
owner and member but we need also admin
---------
Co-authored-by: Val <68596159+valya@users.noreply.github.com>
I have observed that the next hard deletion timeout is not scheduled if
the `hardDeleteOnPruningCycle` function throws when fetching the data
from the database. That is because the thrown error is not caught and
the `scheduleHardDeletion` method is not called.
This PR moves the call to `scheduleHardDeletion` into the
`scheduleHardDeletion` for better cohesion, and ensures that it is
called even if `hardDeleteOnPruningCycle` throws.
## Summary
Ensure `ownedBy` and `sharedWith` are present and uniform for
credentials and workflows.
Details in story: https://linear.app/n8n/issue/PAY-987
Ensure all errors in `cli` are `ApplicationError` or children of it and
contain no variables in the message, to continue normalizing all the
errors we report to Sentry
Follow-up to: https://github.com/n8n-io/n8n/pull/7839
extracted out of #7336
---------
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
Co-authored-by: Oleg Ivaniv <me@olegivaniv.com>
Co-authored-by: Alex Grozav <alex@grozav.com>
Ensure all errors in `cli` inherit from `ApplicationError` to continue
normalizing all the errors we report to Sentry
Follow-up to: https://github.com/n8n-io/n8n/pull/7820
This PR continues the effort of moving logic inside execution lifecycle
hooks into standalone testable functions, as a stepping stone to
refactoring the hooks themselves.
Keep reporting [path-related
errors](https://n8nio.sentry.io/issues/4649493725) in Sentry but
consolidate them in a single error group.
Also, add `options.extra` as `meta` so they remain visible in debug
logs:
```
2023-11-24T11:50:54.852Z | error | ReportableError: Something went wrong "{ test: 123, file: 'LoggerProxy.js', function: 'exports.error' }"
```
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
https://linear.app/n8n/issue/PAY-985
```
PATCH /users/:id/role
unauthenticated user
✓ should receive 401 (349 ms)
member
✓ should fail to demote owner to member (349 ms)
✓ should fail to demote owner to admin (359 ms)
✓ should fail to demote admin to member (381 ms)
✓ should fail to promote other member to owner (353 ms)
✓ should fail to promote other member to admin (377 ms)
✓ should fail to promote self to admin (354 ms)
✓ should fail to promote self to owner (371 ms)
admin
✓ should receive 400 on invalid payload (351 ms)
✓ should receive 404 on unknown target user (351 ms)
✓ should fail to demote owner to admin (349 ms)
✓ should fail to demote owner to member (347 ms)
✓ should fail to promote member to owner (384 ms)
✓ should fail to promote admin to owner (350 ms)
✓ should be able to demote admin to member (354 ms)
✓ should be able to demote self to member (350 ms)
✓ should be able to promote member to admin (349 ms)
owner
✓ should be able to promote member to admin (349 ms)
✓ should be able to demote admin to member (349 ms)
✓ should fail to demote self to admin (348 ms)
✓ should fail to demote self to member (354 ms)
```
This PR introduces the following changes:
- New Vue stores: `collaborationStore` and `pushConnectionStore`
- Front-end push connection handling overhaul: Keep only a singe
connection open and handle it from the new store
- Add user avatars in the editor header when there are multiple users
working on the same workflow
- Sending a heartbeat event to back-end service periodically to confirm
user is still active
- Back-end overhauls (authored by @tomi):
- Implementing a cleanup procedure that removes inactive users
- Refactoring collaboration service current implementation
---------
Co-authored-by: Tomi Turtiainen <10324676+tomi@users.noreply.github.com>
Validate first and last names before saving them to database. This
should prevent security issue with un-sanitized data that ends up in
emails.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
When we upgrade typeorm in #5151, we switched from no pooling to a
default pool-size of 10. This somehow significantly deteriorates the
performance of queries when the application is under load.
Followup to #7566 | Story: https://linear.app/n8n/issue/PAY-926
### Manual workflow activation and deactivation
In a multi-main scenario, if the user manually activates or deactivates
a workflow, the process (whether leader or follower) that handles the
PATCH request and updates its internal state should send a message into
the command channel, so that all other main processes update their
internal state accordingly:
- Add to `ActiveWorkflows` if activating
- Remove from `ActiveWorkflows` if deactivating
- Remove and re-add to `ActiveWorkflows` if the update did not change
activation status.
After updating their internal state, if activating or deactivating, the
recipient main processes should push a message to all connected
frontends so that these can update their stores and so reflect the value
in the UI.
### Workflow activation errors
On failure to activate a workflow, the main instance should record the
error in Redis - main instances should always pull activation errors
from Redis in a multi-main scenario.
### Leadership change
On leadership change...
- The old leader should stop pruning and the new leader should start
pruning.
- The old leader should remove trigger- and poller-based workflows and
the new leader should add them.
1. Reduce a lot of code duplication
2. Move more endpoints out of `Server.ts`
3. Move all query-param parsing and validation into a middleware to make
the route handlers simpler.
This PR:
- Creates `InvitationController`
- Moves `POST /users` to `POST /invitations` and move related test to
`invitations.api.tests`
- Moves `POST /users/:id` to `POST /invitations/:id/accept` and move
related test to `invitations.api.tests`
- Adjusts FE to use new endpoints
- Moves all the invitation logic to the `UserService`
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Github issue / Community forum post (link here to close automatically):
---------
Co-authored-by: Giulio Andreini <g.andreini@gmail.com>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
This change expands on the command channel communication introduced
lately between the main instance(s) and the workers. The frontend gets a
new menu entry "Workers" which will, when opened, trigger a regular call
to getStatus from the workers. The workers then respond via their
response channel to the backend, which then pushes the status to the
frontend.
This introduces the use of ChartJS for metrics.
This feature is still in MVP state and thus disabled by default for the
moment.
- Enable two-way communication with web sockets
- Enable sending push messages to specific users
- Add collaboration service for managing active users for workflow
Missing things:
- State is currently kept only in memory, making this not work in
multi-master setups
- Removing a user from active users in situations where they go inactive
or we miss the "workflow closed" message
- I think a timer based solution for this would cover most edge cases.
I.e. have FE ping every X minutes, BE removes the user unless they have
received a ping in Y minutes, where Y > X
- FE changes to be added later by @MiloradFilipovic
Github issue / Community forum post (link here to close automatically):
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Story: https://linear.app/n8n/issue/PAY-926
This PR coordinates workflow activation on instance startup and on
leadership change in multiple main scenario in the internal API. Part 3
on manual workflow activation and deactivation will be a separate PR.
### Part 1: Instance startup
In multi-main scenario, on starting an instance...
- [x] If the instance is the leader, it should add webhooks, triggers
and pollers.
- [x] If the instance is the follower, it should not add webhooks,
triggers or pollers.
- [x] Unit tests.
### Part 2: Leadership change
In multi-main scenario, if the main instance leader dies…
- [x] The new main instance leader must activate all trigger- and
poller-based workflows, excluding webhook-based workflows.
- [x] The old main instance leader must deactivate all trigger- and
poller-based workflows, excluding webhook-based workflows.
- [x] Unit tests.
To test, start two instances and check behavior on startup and
leadership change:
```
EXECUTIONS_MODE=queue N8N_LEADER_SELECTION_ENABLED=true N8N_LICENSE_TENANT_ID=... N8N_LICENSE_ACTIVATION_KEY=... N8N_LOG_LEVEL=debug npm run start
EXECUTIONS_MODE=queue N8N_LEADER_SELECTION_ENABLED=true N8N_LICENSE_TENANT_ID=... N8N_LICENSE_ACTIVATION_KEY=... N8N_LOG_LEVEL=debug N8N_PORT=5679 npm run start
```
This PR ensures `MultiMainInstancePublisher` is initialized before
checking if the instance is leader or follower. Followers skip license
init, license check, and pruning start and stop.
Github issue / Community forum post (link here to close automatically):
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <netroy@users.noreply.github.com>
To help debugging possible issues in startup and migrations, log the
executed migrations with log level 'info', instead of 'debug'.
Github issue / Community forum post (link here to close automatically):
Due to a change, during the credentials import command, the core's
Credential object is being called through its prototype. This caused the
Credential's cipher variable to not be set, thus no cipher service being
available during import. This fix catches this edge case and provides a
fix.
https://linear.app/n8n/issue/PAY-933/set-up-leader-selection-for-multiple-main-instances
- [x] Set up new envs
- [x] Add config and license checks
- [x] Implement `MultiMainInstancePublisher`
- [x] Expand `RedisServicePubSubPublisher` to support
`MultiMainInstancePublisher`
- [x] Init `MultiMainInstancePublisher` on startup and destroy on
shutdown
- [x] Add to sandbox plans
- [x] Test manually
Note: This is only for setup - coordinating in reaction to leadership
changes will come in later PRs.
Github issue / Community forum post (link here to close automatically):
---------
Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
This PR allows users to configure the settings to Bull, possibly
reducing the errors with `maxStalledCount` and other issues, that
usually happen either when a worker crashes or when the event loop is
super busy. Increasing the lease time and the `maxStalledCount` settings
might improve UX.
Github issue / Community forum post (link here to close automatically):
This PR converts the hard-deletion interval to a timeout:
- to prevent the interval from not being restored when hard deletion
throws, and
- to prevent a long-running hard deletion from leading to duplicate
deletions.
Since we do not store which executions produced binary data, for pruning
on S3 we need to query for binary data items for each execution in order
to delete them. To minimize requests to S3, allow the user to skip
pruning requests when setting TTL at bucket level.
This change ensures that things like `encryptionKey` and `instanceId`
are always available directly where they are needed, instead of passing
them around throughout the code.
This is related to an issue with how Bull handles stalled jobs, see
https://github.com/OptimalBits/bull/issues/1415 for reference.
CPU intensive workflows can in certain cases take a long while to finish
up, thereby blocking the thread and causing Bull queue to think the job
has stalled, even though it finished successfully. In these cases the
error handling could then overwrite the successful execution data with
the error message.
In a rare edge case an undefined queue could be returned - this should
not happen and now an error is thrown.
Also using the opportunity to remove a cyclic dependency from the Queue.
This fixes a bug in the pruning (soft-delete). The pruning was a bit too
aggressive, as it also pruned executions that weren't in an end state
yet. This only becomes an issue if there are long-running executions
(e.g. workflow with Wait node) or the prune parameters are set to keep
only a tiny number of executions.
This PR adds a message for queue mode which triggers an external secrets
provider reload inside the workers if the configuration has changed on
the main instance.
It also refactors some of the message handler code to remove cyclic
dependencies, as well as remove unnecessary duplicate redis clients
inside services (thanks to no more cyclic deps)
Depends on https://github.com/n8n-io/n8n/pull/7220 | Story:
[PAY-840](https://linear.app/n8n/issue/PAY-840/introduce-object-store-service-and-manager-for-binary-data)
This PR introduces an object store service for Enterprise edition. Note
that the service is tested but currently unused - it will be integrated
soon as a binary data manager, and later for execution data.
`amazonaws.com` in the host is temporarily hardcoded until we integrate
the service and test against AWS, Cloudflare and Backblaze, in the next
PR.
This is ready for review - the PR it depends on is approved and waiting
for CI.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
all commands sent between main instance and workers need to contain a
server id to prevent senders from reacting to their own messages,
causing loops
this PR makes sure all sent messages contain a sender id by default as
part of constructing a sending redis client.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Depends on: https://github.com/n8n-io/n8n/pull/7195 | Story:
[PAY-837](https://linear.app/n8n/issue/PAY-837/implement-object-store-manager-for-binary-data)
This PR includes `workflowId` in binary data writes so that the S3
manager can support this filepath structure
`/workflows/{workflowId}/executions/{executionId}/binaryData/{binaryFilename}`
to easily delete binary data for workflows. Also all binary data service
and manager methods that take `workflowId` and `executionId` are made
consistent in arg order.
Note: `workflowId` is included in filesystem mode for compatibility with
the common interface, but `workflowId` will remain unused by filesystem
mode until we decide to restructure how this mode stores data.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Story: [PAY-846](https://linear.app/n8n/issue/PAY-846) | Related:
https://github.com/n8n-io/n8n/pull/7225
For the S3 backend for external storage of binary data and execution
data, the `getAsStream` method in the binary data manager interface used
by FS and S3 will need to become async. This is a breaking change for
nodes-base.
Story: https://linear.app/n8n/issue/PAY-839
This is a longstanding bug, fixed now so that the S3 backend for binary
data can use execution IDs as part of the filename.
To reproduce:
1. Set up a workflow with a POST Webhook node that accepts binary data.
2. Activate the workflow and call it sending a binary file, e.g. `curl
-X POST -F "file=@/path/to/binary/file/test.jpg"
http://localhost:5678/webhook/uuid`
3. Check `~/.n8n/binaryData`. The binary data and metadata files will be
missing the execution ID, e.g. `11869055-83c4-4493-876a-9092c4708b9b`
instead of `39011869055-83c4-4493-876a-9092c4708b9b`.
Depends on: #7092 | Story:
[PAY-768](https://linear.app/n8n/issue/PAY-768)
This PR:
- Generalizes the `IBinaryDataManager` interface.
- Adjusts `Filesystem.ts` to satisfy the interface.
- Sets up an S3 client stub to be filled in in the next PR.
- Turns `BinaryDataManager` into an injectable service.
- Adjusts the config schema and adds new validators.
Note that the PR looks large but all the main changes are in
`packages/core/src/binaryData`.
Out of scope:
- `BinaryDataManager` (now `BinaryDataService`) and `Filesystem.ts` (now
`fs.client.ts`) were slightly refactored for maintainability, but fully
overhauling them is **not** the focus of this PR, which is meant to
clear the way for the S3 implementation. Future improvements for these
two should include setting up a backwards-compatible dir structure that
makes it easier to locate binary data files to delete, removing
duplication, simplifying cloning methods, using integers for binary data
size instead of `prettyBytes()`, writing tests for existing binary data
logic, etc.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Github issue / Community forum post (link here to close automatically):
---------
Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Based on #7065 | Story: https://linear.app/n8n/issue/PAY-771
n8n on filesystem mode marks binary data to delete on manual execution
deletion, on unsaved execution completion, and on every execution
pruning cycle. We later prune binary data in a separate cycle via these
marker files, based on the configured TTL. In the context of introducing
an S3 client to manage binary data, the filesystem mode's mark-and-prune
setup is too tightly coupled to the general binary data management
client interface.
This PR...
- Ensures the deletion of an execution causes the deletion of any binary
data associated to it. This does away with the need for binary data TTL
and simplifies the filesystem mode's mark-and-prune setup.
- Refactors all execution deletions (including pruning) to cause soft
deletions, hard-deletes soft-deleted executions based on the existing
pruning config, and adjusts execution endpoints to filter out
soft-deleted executions. This reduces DB load, and keeps binary data
around long enough for users to access it when building workflows with
unsaved executions.
- Moves all execution pruning work from an execution lifecycle hook to
`execution.repository.ts`. This keeps related logic in a single place.
- Removes all marking logic from the binary data manager. This
simplifies the interface that the S3 client will meet.
- Adds basic sanity-check tests to pruning logic and execution deletion.
Out of scope:
- Improving existing pruning logic.
- Improving existing execution repository logic.
- Adjusting dir structure for filesystem mode.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
This PR implements the updated license SDK so that worker and webhook
instances do not auto-renew licenses any more.
Instead, they receive a `reloadLicense` command via the Redis client
that will fetch the updated license after it was saved on the main
instance
This also contains some refactoring with moving redis sub and pub
clients into the event bus directly, to prevent cyclic dependency
issues.
PR adds a new field to the SourceControlPreferences as well as to the
POST parameters for the `source-control/preferences` and
`source-control/generate-key-pair` endpoints. Both now accept an
optional string parameter `keyGeneratorType` of `'ed25519' | 'rsa'`
Calling the `source-control/generate-key-pair` endpoint with the
parameter set, it will also update the stored preferences accordingly
(so that in the future new keys will use the same method)
By default ed25519 is being used. The default may be changed using a new
environment parameter:
`N8N_SOURCECONTROL_DEFAULT_SSH_KEY_TYPE` which can be `rsa` or `ed25519`
RSA keys are generated with a length of 4096 bytes.
# Motivation
In Queue mode, finished executions would cause the main instance to
always pull all execution data from the database, unflatten it and then
use it to send out event log events and telemetry events, as well as
required returns to Respond to Webhook nodes etc.
This could cause OOM errors when the data was large, since it had to be
fully unpacked and transformed on the main instance’s side, using up a
lot of memory (and time).
This PR attempts to limit this behaviour to only happen in those
required cases where the data has to be forwarded to some waiting
webhook, for example.
# Changes
Execution data is only required in cases, where the active execution has
a `postExecutePromise` attached to it. These usually forward the data to
some other endpoint (e.g. a listening webhook connection).
By adding a helper `getPostExecutePromiseCount()`, we can decide that in
cases where there is nothing listening at all, there is no reason to
pull the data on the main instance.
Previously, there would always be postExecutePromises because the
telemetry events were called. Now, these have been moved into the
workers, which have been given the various InternalHooks calls to their
hook function arrays, so they themselves issue these telemetry and event
calls.
This results in all event log messages to now be logged on the worker’s
event log, as well as the worker’s eventbus being the one to send out
the events to destinations. The main event log does…pretty much nothing.
We are not logging executions on the main event log any more, because
this would require all events to be replicated 1:1 from the workers to
the main instance(s) (this IS possible and implemented, see the worker’s
`replicateToRedisEventLogFunction` - but it is not enabled to reduce the
amount of traffic over redis).
Partial events in the main log could confuse the recovery process and
would result in, ironically, the recovery corrupting the execution data
by considering them crashed.
# Refactor
I have also used the opportunity to reduce duplicate code and move some
of the hook functionality into
`packages/cli/src/executionLifecycleHooks/shared/sharedHookFunctions.ts`
in preparation for a future full refactor of the hooks
This PR adds new endpoints to the REST API:
`/orchestration/worker/status` and `/orchestration/worker/id`
Currently these just trigger the return of status / ids from the workers
via the redis back channel, this still needs to be handled and passed
through to the frontend.
It also adds the eventbus to each worker, and triggers a reload of those
eventbus instances when the configuration changes on the main instances.
Until https://github.com/n8n-io/n8n/pull/7061 we had an edge case where
a manual unsaved workflow when run creates an orphan execution, i.e. a
saved execution not pointing to any workflow. This execution is only
ever visible to the instance owner (even if triggered by a member), and
is wrongly stored as unfinished and crashed. This PR enforces that the
DB disallows any such executions from making it into the DB.
This is needed also for the S3 client, which will include the
`workflowId` in the path-like filename.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
- For a saved execution, we write to disk binary data and metadata.
These two are only ever deleted via `POST /executions/delete`. No marker
file, so untouched by pruning.
- For an unsaved execution, we write to disk binary data, binary data
metadata, and a marker file at `/meta`. We later delete all three during
pruning.
- The third flow is legacy. Currently, if the execution is unsaved, we
actually store it in the DB while running the workflow and immediately
after the workflow is finished during the `onWorkflowPostExecute()` hook
we delete that execution, so the second flow applies. But formerly, we
did not store unsaved executions in the DB ("ephemeral executions") and
so we needed to write a marker file at `/persistMeta` so that, if the
ephemeral execution crashed after the step where binary data was stored,
we had a way to later delete its associated dangling binary data via a
second pruning cycle, and if the ephemeral execution succeeded, then we
immediately cleaned up the marker file at `/persistMeta` during the
`onWorkflowPostExecute()` hook.
This creation and cleanup at `/persistMeta` is still happening, but this
third flow no longer has a purpose, as we now store unsaved executions
in the DB and delete them immediately after. Hence the third flow can be
removed.
Github issue / Community forum post (link here to close automatically):
For the upcoming workflow history feature, we're creating the necessary
database tables.
Also changes the schema for Postgres so the versionId column is now
properly a UUID. The `using` statement prevents losing data, basically
converting the strings to UUIDs.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <netroy@users.noreply.github.com>
When ever we have migrations that use `.addColumn` or `.dropColumn`,
typeorm recreates tables for sqlite. so, we need to disable foreign key
enforcement for sqlite, or else data in some tables can get deleted
because of `ON DELETE CASCADE`
[This has happened in the
past](https://github.com/n8n-io/n8n/pull/6739), and we should really
come up with a way to prevent this from happening again.
---------
Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>
Co-authored-by: Oleg Ivaniv <me@olegivaniv.com>
Github issue / Community forum post (link here to close automatically):
This PR aims to address an issue where an Error workflow cannot be
started, either due to insufficient permissions or because its settings
prevent it from being called.
The way of addressing this is by creating a failed execution for the
appointed error workflow stating the error, as can be seen below.
This means the execution itself won't start, as it's prevented before
the execution beings, but we save a "stub" execution to show the error.
In scope:
- Consolidate `WorkflowService.getMany()`.
- Support non-entity field `ownedBy` for `select`.
- Support `tags` for `filter`.
- Move `addOwnerId` to `OwnershipService`.
- Remove unneeded check for `filter.id`.
- Simplify DTO validation for `filter` and `select`.
- Expand tests for `GET /workflows`.
Workflow list query DTOs:
```
filter → name, active, tags
select → id, name, active, tags, createdAt, updatedAt, versionId, ownedBy
```
Out of scope:
- Migrate `shared_workflow.roleId` and `shared_credential.roleId` to
string IDs.
- Refactor `WorkflowHelpers.getSharedWorkflowIds()`.
Issue: during startup, unfinished executions trigger a recovery process
that, under certain circumstances, can in itself crash the instance
(e.g. by running our of memory), resulting in an infinite recovery loop
This PR aims to change this behaviour by writing a flag file when the
recovery process starts, and removing it when it finishes. In the case
of a crash, this flag will persist and upon the next attempt, the
recovery will instead do the absolute minimal (marking executions as
'crashed'), without attempting any 'crashable' actions.
Changes in https://github.com/n8n-io/n8n/pull/6394 removed xml body parsing for all non-webhook routes. This broken SAML endpoints as they need the XML body parser to function correctly.
* fix handle empty keys in cache service
* add test
* add cache mock test
* add simpler mocking, and add tests for all the updated methods
* don't use RedisStore specifically in the mock
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* support redis cluster
* cleanup, fix config schema
* set default prefix to bull
* initial commit
* improve logging
* improve types and refactor
* list support and refactor
* fix redis service and tests
* add comment
* add redis and cache prefix
* use injection
* lint fix
* clean schema comments
* improve naming, tests, cluster client
* merge master
* cache returns unknown instead of T
* update cache service, tests and doc
* remove console.log
* VariablesService as DI, add caching, fix tests
* do not cache null or undefined values
* import fix
* more DI and remove collections
* fix merge
* lint fix
* rename to ~Cached
* fix test for CI
* fix ActiveWorkflowRunner test
Also,
1. Consistent CORS support ~on all three webhook types~ waiting webhooks never supported CORS. I'll fix that in another PR
2. [Fixes binary-data handling when request body is text, json, or xml](https://linear.app/n8n/issue/NODE-505/webhook-binary-data-handling-fails-for-textplain-files).
3. Reduced number of middleware that each request has to go through.
4. Removed the need to maintain webhook endpoints in the auth-exception list.
5. Skip all middlewares (apart from `compression`) on Webhook routes.
6. move `multipart/form-data` support out of individual nodes
7. upgrade `formidable`
8. fix the filenames on binary-data in webhooks nodes
9. add unit tests and integration tests for webhook request handling, and increase test coverage
* refactor: Set up ownership service
* refactor: Specify cache keys and values
* refactor: Replace util with service calls
* test: Mock service in tests
* refactor: Use dependency injection
* test: Write tests
* refactor: Apply feedback from Omar and Micha
* test: Fix tests
* test: Fix missing spot
* refactor: Return user entity from cache
* refactor: More dependency injection!
* db entities don't need an ID before they are inserted
* don't define constructors on entity classes, use repository.create instead
* use mixins to reduce duplicate code in db entity classes
* initial telemetry setup and adjusted pull return
* quicksave before merge
* feat: add conflicting workflow list to pull modal
* feat: update source control pull modal
* fix: fix linting issue
* feat: add Enter keydown event for submitting source control push modal (no-changelog)
feat: add Enter keydown event for submitting source control push modal
* quicksave
* user workflow table for export
* improve telemetry data
* pull api telemetry
* fix lint
* Copy tweaks.
* remove authorName and authorEmail and pick from user
* rename owners.json to workflow_owners.json
* ignore credential conflicts on pull
* feat: several push/pull flow changes and design update
* pull and push return same data format
* fix: add One last step toast for successful pull
* feat: add up to date pull toast
* fix: add proper Learn more link for push and pull modals
* do not await tracking being sent
* fix import
* fix await
* add more sourcecontrolfile status
* Minor copy tweak for "More info".
* Minor copy tweak for "More info".
* ignore variable_stub conflicts on pull
* ignore whitespace differences
* do not show remote workflows that are not yet created
* fix telemetry
* fix toast when pulling deleted wf
* lint fix
* refactor and make some imports dynamic
* fix variable edit validation
* fix telemetry response
* improve telemetry
* fix unintenional delete commit
* fix status unknown issue
* fix up to date toast
* do not export active state and reapply versionid
* use update instead of upsert
* fix: show all workflows when clicking push to git
* feat: update Up to date pull translation
* fix: update read only env checks
* do not update versionid of only active flag changes
* feat: prevent access to new workflow and templates import when read only env
* feat: send only active state and version if workflow state is not dirty
* fix: Detect when only active state has changed and prevent generation a new version ID
* feat: improve readonly env messages
* make getPreferences public
* fix telemetry issue
* fix: add partial workflow update based on dirty state when changing active state
* update unit tests
* fix: remove unsaved changes check in readOnlyEnv
* fix: disable push to git button when read onyl env
* fix: update readonly toast duration
* fix: fix pinning and title input in protected mode
* initial commit (NOT working)
* working push
* cleanup and implement pull
* fix getstatus
* update import to new method
* var and tag diffs are no conflicts
* only show pull conflict for workflows
* refactor and ignore faulty credentials
* add sanitycheck for missing git folder
* prefer fetch over pull and limit depth to 1
* back to pull...
* fix setting branch on initial connect
* fix test
* remove clean workfolder
* refactor: Remove some unnecessary code
* Fixed links to docs.
* fix getstatus query params
* lint fix
* dialog to show local and remote name on conflict
* only show remote name on conflict
* fix credential expression export
* fix: Broken test
* dont show toast on pull with empty var/tags and refactor
* apply frontend changes from old branch
* fix tag with same name import
* fix buttons shown for non instance owners
* prepare local storage key for removal
* refactor: Change wording on pushing and pulling
* refactor: Change menu item
* test: Fix broken test
* Update packages/cli/src/environments/sourceControl/types/sourceControlPushWorkFolder.ts
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
---------
Co-authored-by: Alex Grozav <alex@grozav.com>
Co-authored-by: Giulio Andreini <g.andreini@gmail.com>
Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* refactor(core): Move webhook DB access to repository (no-changelog)
* make sure `DataSource` is initialized before it's dependencies
at some point I hope to replace `DataSource` with a custom `DatabaseConnection` service class that can then disconnect and reconnect from DB without having to update all repositories.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* use jwt to reset password
* increase expiration time to 1d
* drop user id query string
* refactor
* use service instead of package in tests
* sqlite migration
* postgres migration
* mysql migration
* remove unused properties
* remove userId from FE
* fix test for users.api
* move migration to the common folder
* move type assertion to the jwt.service
* Add jwt secret as a readonly property
* use signData instead of sign in user.controller
* remove base class
* remove base class
* add tests
* ci: Fix test workflows (no-changelog)
We removed `pdf-parse` in #6640, so we need to get these test PDF files from the `test-workflows` repo instead ([which has been updated to include these files](0f6ef1c804))
* remove `\n` from ids and skipList text files
* ⚡ Implemented new grid row - banners
* ✨ Fixing node creator and executions sidebar position after layout update
* 💄 Added configurable round corners to the Callout component
* ⚡ Fixing mouse position detection and main tab bar position
* ⚡ Implemented basic banner component structure
* ⚡ Implemented banner state and dismiss logic
* ⚡ Fixing grid layout. Updating banners height state dynamically
* ⚡ Fix zoom to fit position, mouse position in demo mode and callout vertical alignment
* ⚡ Implementing proper trial banners logic
* 💄 Only showing execution usage data once the sidebar is fully expanded
* ✨ Implemented permanent/temporary dismiss logic for v1 flag
* ⚡ Minor refactoring of banner logic
* ⚡ Updating permanent dismiss logic to work with all banners
* 👕 Fixing linting errors
* ✔️ Updating Callout component test snapshots
* 💄 Tweaking zoom to fit position
* ✔️ Updating testing endpoints to use new store data
* ✅ Added banners unit tests
* ✔️ Fixing failing banner tests
* ✅ Added more banner tests
* ⚡ Updating banners dimensions on resize, removing leftover code
* ✔️ Removing store import from API file
* 👕 Fixing lint errors
* ⚡ Updating migration files
* ⚡ Using query parameters in migrations
* 👌 Addressing design review feedback
* ⚡ Updating upgrade plan button click
* ⚡ Updating the migrations syntax
* 👌 Updating permanent banner dismiss endpoint and back-end logic
* 👌 Refactoring trial banner component and ui store
* 👌 Addressing more points from code review
* 👌 Moving DOM logic from the store
* ✔️ Updated callout component snapshots
* 👌 Updating mysql migration file
* ✔️ Updating e2e test canvas coordinates after setting it's position to absolute
* 👌 Addressing back-end review feedback
* 👌 Improving typing around banners
* 👕 Fixing lint errors
fix(core): Reduce memory consumption on BinaryDataManager.init
When there are a few thousand binary data file to delete, the `deleteMarkedFiles` and `deleteMarkedPersistedFiles` methods need a lot of memory to process these files, irrespective of if these files have any data or not.
* feat(editor): Only show V1 banner to users who migrated
Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>
* Set the v1 banner dismissed flag in settings table create migration
Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>
---------
Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>
* fix(core): Make node execution order configurable, and backward-compatible
* ⚡ Also add new Merge-Node behaviour
* ⚡ Fix typo
* Fix lint issue
* update labels
* rename legacy to v0
* remove the unnecessary log
* default all new workflows to use v1 execution-order
* remove the controller changes
* clone default settings to avoid it getting modified
---------
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
* revert(core): Remove typeorm patches, but still enforce transactions on every migration
This reverts #6519
* always re-enable foreign keys, and explicitly rollback transaction
* Save manual executions by default
* Enable executions data pruning by default
* Keep a maximum of 10000 executions by default
Co-authored-by: Cornelius Suermann <cornelius@n8n.io>
* remove owner from exported workflow
* use owners file to export wf owners
* update sharedworkflow owners
* fix logic
* further update logic
* add updatetAt to local changes
* additional filter for cred export
* optimize query
* remove transactions and optimize query
* reduce array size and add updated at to tags status
* feat(core): Change node execution order (most top-left one first)
* ⚡ Fix issue with multi-output-nodes
* ⚡ Remove not needed meta-entry in test
* fix the e2e test
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* fix: Make migration for MySQL compatible with version 5.7
* fix: Correct syntax
* fix: Syntax for primary key creation
* fix: Remove unnecessary constraint statement
* fix: Name columns for sqlite migration to prevent issues
* ci: Always run DB tests with table-prefix and alt-schema
* add tablePrefix to migrations (mysql not fixed)
* fix character_set on mysql migrations
* fix mysql collation version difference
* cater for mariadb collations
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
* Remove authorization header when empty
* Import pkce
* Add OAuth2 with new grant type to Twitter
* Add pkce logic auto assign authorization code if pkce not defined
* Add pkce to ui and interfaces
* Fix scopes for Oauth2 twitter
* Deubg + pass it through header
* Add debug console, add airtable cred
* Remove all console.logs, make PKCE in th body only when it exists
* Remove invalid character ~
* Remove more console.logs
* remove body inside query
* Remove useless grantype check
* Hide oauth2 twitter waiting for overhaul
* Remove redundant header removal
* Remove more console.logs
* Add comment for code verifier
* Remove uneeded scopes
* Restore client id in callback
* Revert "Add OAuth2 with new grant type to Twitter"
This reverts commit 1c3b331aa1.
* Remove oauth2 from twitter
* Remove properties linked to oauth2
* Fix lodash imports
* remove redundant check
* remove redundant codeVerifier
* patch pkce-challenge to avoid generating `code_verifier` with `~`
* store `codeVerifier` on the DB like `csrfSecret`
* remove unrelated changes
---------
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* first commit for postgres migration
* (not working)
* sqlite migration
* quicksave
* fix tests
* fix pg test
* fix postgres
* fix variables import
* fix execution saving
* add user settings fix
* change migration to single lines
* patch preferences endpoint
* cleanup
* improve variable import
* cleanup unusued code
* Update packages/cli/src/PublicApi/v1/handlers/workflows/workflows.handler.ts
Co-authored-by: Omar Ajoue <krynble@gmail.com>
* address review notes
* fix var update/import
* refactor: Separate execution data to its own table (#6323)
* wip: Temporary migration process
* refactor: Create boilerplate repository methods for executions
* fix: Lint issues
* refactor: Added search endpoint to repository
* refactor: Make the execution list work again
* wip: Updating how we create and update executions everywhere
* fix: Lint issues and remove most of the direct access to execution model
* refactor: Remove includeWorkflowData flag and fix more tests
* fix: Lint issues
* fix: Fixed ordering of executions for FE, removed transaction when saving execution and removed unnecessary update
* refactor: Add comment about missing feature
* refactor: Refactor counting executions
* refactor: Add migration for other dbms and fix issues found
* refactor: Fix lint issues
* refactor: Remove unnecessary comment and auto inject repo to internal hooks
* refactor: remove type assertion
* fix: Fix broken tests
* fix: Remove unnecessary import
* Remove unnecessary toString() call
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* fix: Address comments after review
* refactor: Remove unused import
* fix: Lint issues
* fix: Add correct migration files
---------
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* remove null values from credential export
* fix: Fix an issue with queue mode where all running execution would be returned
* fix: Update n8n node to allow for workflow ids with letters
* set upstream on set branch
* remove typo
* add nodeAccess to credentials
* fix unsaved run check for undefined id
* fix(core): Rename version control feature to source control (#6480)
* rename versionControl to sourceControl
* fix source control tooltip wording
---------
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
* fix(editor): Pay 548 hide the set up version control button (#6485)
* feat(DebugHelper Node): Fix and include in main app (#6406)
* improve node a bit
* fixing continueOnFail() ton contain error in json
* improve pairedItem
* fix random data returning object results
* fix nanoId length typo
* update pnpm-lock file
---------
Co-authored-by: Marcus <marcus@n8n.io>
* fix(editor): Remove setup source control CTA button
* fix(editor): Remove setup source control CTA button
---------
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
* fix(editor): Update source control docs links (#6488)
* feat(DebugHelper Node): Fix and include in main app (#6406)
* improve node a bit
* fixing continueOnFail() ton contain error in json
* improve pairedItem
* fix random data returning object results
* fix nanoId length typo
* update pnpm-lock file
---------
Co-authored-by: Marcus <marcus@n8n.io>
* feat(editor): Replace root events with event bus events (no-changelog) (#6454)
* feat: replace root events with event bus events
* fix: prevent cypress from replacing global with globalThis in import path
* feat: remove emitter mixin
* fix: replace component events with event bus
* fix: fix linting issue
* fix: fix breaking expression switch
* chore: prettify ndv e2e suite code
* fix(editor): Update source control docs links
---------
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
* fix tag endpoint regex
---------
Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
Co-authored-by: Csaba Tuncsik <csaba@n8n.io>
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
* fix(core): Do not track errored workflow executions for automated executions
* fix test
* fix test
* fix test
* do not track 'Workflow execution count' event when all counts are 0
* fix test
* fix test
* fix test
* fix test
* fix test
* fix test
* fix test
* fix test
* fix test
* Add ExecutionsUsage component
* set $sidebar-expanded-width back to 200px
* add days using interpolation
* Rename PlanData type to CloudPlanData
* Rename Metadata type to PlanMetadata
* Make prop block in the update button
* Use variable in line-height
* Remove progressBarSection class
* fix trial expiration calculation
* mock expirationDate and fix issue with days left
* Remove unnecesary property from class .container
* inject component data via props
* Check for plan data during app mounting and keep data in the store
* Remove mounted hook
* redirect when upgrade plan is clicked
* Remove computed properties
* Remove instance property as it's not needed anymore
* Flatten plan object
* remove console.log
* Add all cloud types within its own namespace
* keep redirection inside component
* get computed properties back
* Improve polling logic
* Move cloudData to its own store
* Remove commented interfaces
* remove cloudPlan from user store
* fix imports
* update logic for userIsTrialing method
* centralize userIsTrialing method
* redirect to production change plan page always
* Call staging or production cloud api depending on base URL
* remove setting store form ExecutionUsage.vue
* fix linting issue
* Add trial group to PlanMetadata group
* Move helpers into the store
* make staging url check more specific
* make cloud state nullable
* fix linting issue
* swap mockup date for endpoint
* Make getCurrentPlan async
* asas
* Improvements
* small improvements
* chore: resolve conflicts
* make sure there is data before calculating trial expiration
* Fix issue with component not loading on first page load
* type safety improvements
* apply component ui feedback
* fix linting issue
* chore: clean up unnecessary change from merge conflict
* feat: Block api feature using licenses, show notice page for trial cloud users (#6187)
* rename planSpec to plan
* Remove instance property as it's not needed anymore
* Flatten plan object
* remove console.log
* feat: disable api using license
* feat: add api page
* chore: resolve conflicts
* chore: resolve conflicts
* feat: update and refactor a bit
* fix: update endpoints
* fix: update endpoints
* fix: use host
* feat: update copy
* fix linting issues
---------
Co-authored-by: ricardo <ricardoespinoza105@gmail.com>
* add pluralization to days left text
---------
Co-authored-by: Mutasem <mutdmour@gmail.com>
Co-authored-by: Mutasem Aldmour <4711238+mutdmour@users.noreply.github.com>
* fix: Prevent unnecessary error messages also for data loaded flag
* refactor: Search if data has been loaded before trying to save and fire other events
* fix broken test
* fix lint issue
* remove unnecesary Db re-initialization
this is from before we added `Db.init` in `WorkflowRunnerProcess`
* feat(core): Improved health check
* make health check not care about DB connections
* close DB connections, and shutdown the timer
* refactor: Add deprecation notice for MySQL and MariaDB
* Update packages/cli/src/commands/BaseCommand.ts
Co-authored-by: Cornelius Suermann <cornelius@n8n.io>
---------
Co-authored-by: Cornelius Suermann <cornelius@n8n.io>
* test: Add /setup intercept for `skipSetup` command (no-changelog)
* Drop all tables for e2e reset, intercept account setup request
* Fix linting issues
* Allow to skip setup account request intercept and linting fixes
* Make sure variables are loaded
* Use PATCH for enabling of e2e features
* Do not exclude migration table from truncation
* Add user sign-up intercept
add Base URL to redirects in acsHandler
I modified the redirections to include the base URL of the instance so that the redirects remain correct even if the instance is accessed from a subdirectory.
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
* Add userActivated migration
* Fix migration logic
* Remove duplication when retrieving the activated users
* Fix bug updating settings in mysql
* Make userSettings type conform with naming convention
* Disable naming convention rule only in IDatabaseCollections interface
* Fix down method in Postgres migration
* Reset '{}' to NULL when reversing migration
* ⚡ Add user activation survey
* Fix typo
* Avoid showing the modal when there is a modal view
* Allow to redirect to specific execution
* Improve structure
* Handle errors when sharing feedback
* update withFeatureFlag function
* Fix linting issue
* Set user activation flag on workflowExecutionCompleted event
* Revert update user settings functionality
* Remove unnecessary changes
* fix linting issue
* account for new functionality in tests
* Small improvements
* keep once instace of the model open between tabs
* Add sorting to GET /executions
* type parameters for GET /executions
a
* Add constant for local store key
* Add execution mode filtering
* fix linting issue
* Do not override settings when setting isOnboarded true
* Add update user settings endpoint
* improvements
* revert changes to /GET executions
* Fix typo
* Add userActivated flag to user store
* Add E2E test
* Fix linting issue
* Update pnpm-lock
* Revert unnecessary change
* Centralize user's settings update
* Remove unused ref in userActivationSurvey modal
* Use aliased imports
* Use createEventBus function in component
* Fix tests
If you install a community node with `polling: true`, activating a workflow with that node fails with an error: `WorkflowActivationError: There was a problem activating the workflow: "Could not get parameter "pollTimes"!"`.
You can test this by installing `n8n-nodes-rss-feed-trigger`, creating a workflow with the `RSS Trigger` node, and then trying to activate it. Activation will fail on `master`, but work as expected on this branch.
* wip: workflow execution filtering
* fix: import type failing to build
* fix: remove console.logs
* feat: execution metadata migrations
* fix(editor): Move global executions filter to its own component
* fix(editor): Using the same filter component in workflow level
* fix(editor): a small housekeeping
* checking workflowId in filter applied
* fix(editor): update filter after resolving merge conflicts
* fix(editor): unify empy filter status
* feat(editor): add datetime picker to filter
* feat(editor): add meta fields
* fix: fix button override in datepicker panel
* feat(editor): add filter metadata
* feat(core): add 'startedBefore' execution filter prop
* feat(core): add 'tags' execution query filter
* Revert "feat(core): add 'tags' execution query filter"
This reverts commit a7b968081c.
* feat(editor): add translations and tooltip and counting selected filter props
* fix(editor): fix label layouts
* fix(editor): update custom data docs link
* fix(editor): update custom data tooltip position
* fix(editor): update tooltip text
* refactor: Ignore metadata if not enabled by license
* fix(editor): Add paywall states to advanced execution filter
* refactor: Save custom data also for worker mode
* fix: Remove duplicate migration name from list
* fix(editor): Reducing filter complexity and add debounce to text inputs
* fix(editor): Remove unused import, add comment
* fix(editor): simplify event listener
* fix: Prevent error when there are running executions
* test(editor): Add advanced execution filter basic unit test
* test(editor): Add advanced execution filter state change unit test
* fix: Small lint issue
* feat: Add indices to speed up queries
* feat: add customData limits
* refactor: put metadata save in transaction
* chore: remove unneed comment
* test: add tests for execution metadata
* fix(editor): Fixes after merge conflict
* fix(editor): Remove unused import
* wordings and ui fixes
* fix(editor): type fixes
* feat: add code node autocompletions for customData
* fix: Prevent transaction issues and ambiguous ID in sql clauses
* fix(editor): Suppress requesting current executions if metadata is used in filter (#5739)
* fix(editor): Suppress requesting current executions if metadata is used in filter
* fix(editor): Fix arrows for select in popover
* refactor: Improve performance by correcting database indices
* fix: Lint issue
* test: Fix broken test
* fix: Broken test
* test: add call data check for saveExecutionMetadata test
---------
Co-authored-by: Valya Bullions <valya@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
* fix calling error workflows in main mode recovery
* cleanup
* remove WorkflowExecuteAdditionalData export from index
* revert refactor to fix test
* Update index.ts
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <netroy@users.noreply.github.com>
* fix(editor): Fix execution list item selection
* fix(editor): Delete only selected executions
* fix(editor): Fix clear selection
* fix(editor): Fix clear selection
* fix(editor): Fix clear selection
* feat(editor): Add select all existing executions checkbox
* fix(editor): Do not mark later loaded executions selected
* test(editor): Add execution list unit test
* fix(editor): Fix selection
* test(editor): update execution selection test
* fix(editor): Handle UI state when there is no execution
* fix(editor): Remove unnecessary logic
* test(editor): Add more execution list unit tests and fake data generation
* test(editor): Add more execution list unit tests
* test(editor): Simplifying test setup
* chore: update pnpm lock after resolving merge conflocts
* chore: fix package version
* fix: Improved executions deletion to prevent crashing and fixed removal of failed executions
* fix: Add comment to clarify why change was needed
* fix: fix executions list bug when selecting all and changing filter
* fix: fix execution lists running execution showing up on different workflow id
* fix(editor): Deleting an execution while all are selected
* fix(editor): Deleting an execution while all are selected
---------
Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: Alex Grozav <alex@grozav.com>
* use typedi for UserManagementMailer
* use typedi for SamlService
* fix typos
* use typedi for Queue
* use typedi for License
* convert some more code to use typedi
* move nodeTypes api to a controller class
* move tags api to a controller class
* move LDAP routes to a controller class
* move nodes routes to a controller class
* WIP: Cypress parallel CI run test
* Trigger action on branch push
* Change build artifacts path
* Make sure to checkout the repo for testing job
* Use Cypress action for installing
* Lock cypress action userd version
* Skip node install step since we're using cypress node16 container
* Let Cypress handle pnpm install
* Use setup-node action for caching pnpm
* Set CYPRESS_CACHE_FOLDER
* Set CYPRESS_CACHE_FOLDER
* Manually cache pnpm store
* Dont fix pnpm version
* Use caching action also in testing job
* Zip packages dist before uploading the artifacts and change caching key
* Use absolute build paths for zipping job
* Use zip command in action
* Use tar for zipping packages
* Debuggin directory ls
* Debugging caching of modules
* Attempt to fix permissions issue
* Porivde Cypress executable via `CYPRESS_RUN_BINARY`
* Cache /github/home
* Adjust caching keys
* Debug: search for cypress exec
* Debugging: List dirs
* Use pnpm install action to install node_modules
* Do not log /home/runner
* Use node_modules/.bin Cypress binary
* Use absolute path to nodue modules
* Run Cypress via custom command
* Try with patched cypress action
* Revert logging
* Manually specify cypress config file
* Use absolute paths
* Fix cypress config name
* Debug print cypress config
* Remove debugging, increase to 4 containers
* Increase amount of containers
* Add env-version matrix
* Replace node14 with node18 in testing matrix
* Remove debugging and add node 14
* Use just node14
* Use cypress:base and remove browser req
* Give more general timeouts
* Try with node16
* Change cache directive position
* Replace zip artifact upload with cache
* Cache full packages not just dist
* Test with variable inputs
* Add commit info message
* Remove wrongly commited code
* Allow WF API dispatch
* Try Chrome browser again for comparison
* Include Monaco in the build
* Make e2e workflow re-usable
* Comment out invalid reusable workflow args
* Use electron and add node 14 run
* Fix env arg
* Provide custom ci-build-id
* Refactor remaining e2e workflow to use reusable action
* Remove single matrix directive
* Refactor ci-pull-req
* Make lint job dependant on test jobs
* Disable debugging job
* Make containers dynamic
* Cleanup & install git for linting action
* Use regular buntu image for PR linting
* Debugging failing tests
* Remove fixed spec name
* Debug e2e env var
* Do not use realkeypress which crashes electron runner
* Debugging
* chore: remove console
* chore: remove console
* test: remove node 14 tests
* test: replace test branch with master
* test: use tests in current branch
* test: use relative path
* chore: clean up
* test: only trigger on approval
* ci: update test PR
* ci: use curr branch
* ci: only run 14 on schedule, not for slack command
* ci: only run test on approval
* ci: clean up branch, rename step
* ci: rename steps
* ci: clean up cancel
* ci: clean up env var
* ci: set var
* ci: use chromef
* ci: use electron
* chore: add console log
* chore: add console log
* ci: update to string
* ci: set all env options
* test: build
* ci: fix step issue
* Fix failing tests & upgrade to Cypress 12
* Allow WF dispatch of e2e reusable
* Fix wrong naming in e2e-tests workflow
* Redeploy
* Fix tests
* Fix NDV tests and remove skipping of webhooks execution tests
* Fix clipboard read command
* Fix execution failing tests
* Reset before each 15 and 3
* Fix flaky tests
* Cleanup and log envs
* Test fixes
* Default owner spec fixes
* Get rid of CYPRESS_RUN_ENV
* Increase amount of containers, cleanup and add mock for credentials test call
* Cleanup & fix PR tests unit tests
* Wait for WF to loade in sharing spec
* Do linting and unit tests first
* Use frozen lockfile
* Revert back ci pull request jobs order
* Refine credential input selector and move cy.waitForLoad to correct position in 15-scheduler spec
* test: build
* Wait for WF execution instead of arbitraty timeout in WF execution spec, change order of jobs for ci pull request
* Fix flaky 3-default owner spec and wait for execution list to load in 20-workflow-executions
* Use setup node action
* Remove caching for lint/unit tests
* Experiment with parallel test & lint on ci
* Provide cache key dynamically
* Run e2e in parallel on pr
* Only run node14 e2e on daily schedule
* Make sure to generate generate new ci-build-id on re-runs
* Remove debugging prints
* Address PR comments
* Rename custom onBeforeUnload handler
* Make sure 19-execution spec waits for wf to load properly before import fixtures
---------
Co-authored-by: Mutasem <mutdmour@gmail.com>
* fix(core): Execution pruning should delete query should use the `OR` operator
* fix(core): Prune executions in a chunk to avoid sqlite error "Expression tree is too large"
* reduce the memory usage during execution pruning
* add typedi
* convert ActiveWorkflowRunner into an injectable service
* convert ExternalHooks into an injectable service
* convert InternalHooks into an injectable service
* convert LoadNodesAndCredentials into an injectable service
* convert NodeTypes and CredentialTypes into an injectable service
* convert ActiveExecutions into an injectable service
* convert WaitTracker into an injectable service
* convert Push into an injectable service
* convert ActiveWebhooks and TestWebhooks into an injectable services
* handle circular references, and log errors when a circular dependency is found
* feat(editor): roll out schema view
* feat(editor): add posthog tracking
* refactor: use composables
* refactor: clean up console log
* refactor: clean up impl
* chore: clean up impl
* fix: fix demo var
* chore: add comment
* refactor: clean up
* chore: wrap error func
* refactor: clean up import
* refactor: make store
* feat: enable rudderstack usebeacon, move event to unload
* chore: clean up alert
* refactor: move tracking from hooks
* fix: reload flags on login
* fix: add func to setup
* fix: clear duplicate import
* chore: add console to tesT
* chore: add console to tesT
* fix: try reload
* chore: randomize instnace id for testing
* chore: randomize instnace id for testing
* chore: add console logs for testing
* chore: move random id to fe
* chore: use query param for testing
* feat: update PostHog api endpoint
* feat: update rs host
* feat: update rs host
* feat: update rs endpoints
* refactor: use api host for BE events as well
* refactor: refactor out posthog client
* feat: add feature flags to login
* feat: add feature flags to login
* feat: get feature flags to work
* feat: add created at to be events
* chore: add todos
* chore: clean up store
* chore: add created at to identify
* feat: add posthog config to settings
* feat: add bootstrapping
* chore: clean up
* chore: fix build
* fix: get dates to work
* fix: get posthog to recognize dates
* chore: refactor
* fix: update back to number
* fix: update key
* fix: get experiment evals to work
* feat: add posthog to signup router
* feat: add feature flags on sign up
* chore: clean up
* fix: fix import
* chore: clean up loading script
* feat: add timeout, fix: script loader
* fix: test timeout and get working on 8080
* refactor: move out posthog
* feat: add experiment tracking
* fix: clear tracked on reset
* fix: fix signup bug
* fix: handle errors when telmetry is disabled
* refactor: remove redundant await
* fix: add back posthog to telemetry
* test: fix test
* test: fix test
* test: add tests for posthog client
* lint: fix
* fix: fix issue with slow decide endpoint
* lint: fix
* lint: fix
* lint: fix
* lint: fix
* chore: address PR feedback
* chore: address PR feedback
* feat: add onboarding experiment
* Prune execution data when more than cofnfigured limit
* use stricter typings
* use `pruneDataMaxCount`
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* adds ExecutionEvents view modal to ExecutionList
* fix time rendering and remove wf column
* checks for unfinished executions and fails them
* prevent re-setting stoppedAt for execution
* some cleanup / manually create rundata after crash
* quicksave
* remove Threads lib, log worker rewrite
* cleanup comment
* fix sentry destination return value
* test for tests...
* run tests with single worker
* fix tests
* remove console log
* add endpoint for execution data recovery
* lint cleanup and some refactoring
* fix accidental recursion
* remove cyclic imports
* add rundata recovery to Workflowrunner
* remove comments
* cleanup and refactor
* adds a status field to executions
* setExecutionStatus on queued worker
* fix onWorkflowPostExecute
* set waiting from worker
* get crashed status into frontend
* remove comment
* merge fix
* cleanup
* catch empty rundata in recovery
* refactor IExecutionsSummary and inject nodeExecution Errors
* reduce default event log size to 10mb from 100mb
* add per node execution status
* lint fix
* merge and lint fix
* phrasing change
* improve preview rendering and messaging
* remove debug
* Improve partial rundata recovery
* fix labels
* fix line through
* send manual rundata to ui at crash
* some type and msg push fixes
* improve recovered item rendering in preview
* update workflowStatistics on recover
* merge fix
* review fixes
* merge fix
* notify eventbus when ui is back up
* add a small timeout to make sure the UI is back up
* increase reconnect timeout to 30s
* adjust recover timeout and ui connection lost msg
* do not stop execution in editor after x reconnects
* add executionRecovered push event
* fix recovered connection not green
* remove reconnect toast and merge existing rundata
* merge editor and recovered data for own mode
* feat: update PostHog api endpoint
* feat: update rs host
* feat: update rs host
* feat: update rs endpoints
* refactor: use api host for BE events as well
This fixes the following issues:
* After a community node is installed, we were not calling `postProcessLoaders`, which was causing a bunch of unexpected behaviors, and sometimes even crashes. This was only happening in the session where the package was installed. After a crash, the restarted service was working without these issues.
* After a community node is installed, the icon for the nodes and credentials were missing in the UI, as we were creating one icons route per installed package at startup, and this did not handle newly installed packages. restarting the service fixes this issue as well.
Fixes https://community.n8n.io/t/showing-weird-count-on-community-nodes/23035
* Remove check by safe list to skip list
* add an option to skip using statistics events
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* fix(editor): Prevent creation of input connections for nodes without input
* WIP: Workflow checks service and controller
* fix: Created SQLite migration to remove broken connections
* Cleanup & add mysql/posgres migrations
* Linter fixes
* Unify the migration scripts
* Escape migration workflow_entity
* Wrap the migration in try/catch and do not parse nodes and connection if mysql/postgres
* Do migration changes also fro mysql
* refactor: Wrap only the necessary call in try catch block
---------
Co-authored-by: Omar Ajoue <krynble@gmail.com>
* refactor: Hide prompt for desktop
* feat: add email field to personalization modal
* fix: update survey interfaces
* chore: enable personalization survey email key display condition
* feat: add users page upsell for desktop client
* feat: disable UM on desktop where possible
* refactor: Have a single function to decide whether UM is enabled
* feat: update community nodes upsell link
---------
Co-authored-by: Alex Grozav <alex@grozav.com>
Co-authored-by: krynble <omar@n8n.io>
Co-authored-by: freyamade <freya@n8n.io>
`PRAGMA foreign_keys` is [a no-op within a sqlite transaction](https://www.sqlite.org/pragma.html#pragma_foreign_keys).
This causes cascade deletes to happen when we create a new user table, and drop the old user table.
* feat(core): Fix populating of node custom api call options
* lint fixes
* Adress PR comments
* Add e2e test and only inject custom API options for latest version
* Make sure to injectCustomApiCallOption for the latest version of node
* feat(cli): Move apiCallOption injection to LoadNodesAndCredentials and add e2e tests to check for custom nodes credentials
* Load nodes and credentials fixtures from a single place
* Console warning if credential is invalid during customApiOptions injection
* 🔨 - Remove `shared` key from execution save data
* 👕 - Using import type where needed
* remove console.log
* 🔨 - Create new clean workflowData instead of removing shared
If IWorkflowBase changes in future, TS will error out here ensuring it's kept up to date
* 🔨 - use lodash.pick for less verbosity
* 🔨 - fix lodash imports
* feat(core): Fix populating of node custom api call options
* lint fixes
* Adress PR comments
* Add e2e test and only inject custom API options for latest version
* Make sure to injectCustomApiCallOption for the latest version of node
* create prometheus metrics from events
* feat(core): Add more Prometheus metrics (experimental) (#5187)
* refactor(core): Add Prometheus labels to relevant metrics
* feat(core): Add more Prometheus metrics (experimental)
* add 'v' prefix to value of version label
Co-authored-by: Cornelius Suermann <cornelius@n8n.io>
* always each DB migrations in a transaction
* `VACUUM` isn't allowed inside transactions.
* `PRAGMA foreign_keys` are automatically toggled before and after every down migration
* adds ExecutionEvents view modal to ExecutionList
* fix time rendering and remove wf column
* checks for unfinished executions and fails them
* prevent re-setting stoppedAt for execution
* removing UI changes but keeping eventbus fixes
* remove comment
* fix: Stop OOM crashed in Execution Data pruning
Currently while pruning execution data, we are loading all the data in memory. For instances where there are thousands of executions, this causes the container to run out of memory.
Since ids is all we need, we should only query for ids.
* query for Executions only when ids are actually needed for pruning binary data
in default mode the binary data is in the database, and will get pruned along with the executions.
* feat: update n8n-users-list to no longer use preset list of actions
* feat: prepared users settings for invite links feature
* refactor: Return invite link URLs when inviting users (#5079)
* refactor: Return invite link URLs when inviting users
* test: Refactor and add tests to mailer
* feat: Add FE inviteAcceptUrl integration (#5085)
* feat: update n8n-users-list to no longer use preset list of actions
* feat: prepared users settings for invite links feature
* feat: add integration with new inviteAcceptUrl changes
* feat: Add inviteAcceptUrl to user list for pending users
Co-authored-by: Alex Grozav <alex@grozav.com>
* fix conflicts
* fix lint issue
* test: Make sure inviteAcceptUrl is defined
* feat: update smtp setup suggestion
* feat: add invite link summary when inviting multiple users
* refactor: Add telemetry flag for when email is sent
* fix: add email_sent correctly to telemetry event
* feat: move SMTP info-tip to invite modal
Co-authored-by: Omar Ajoue <krynble@gmail.com>
* store n8n version string in a const and use that everywhere
* reduce code duplication between Server and WebhookServer
* unify redis checks
* fix linting
since QueryBuilder api doesn't use entity field transforms, we should remove the usage of QueryBuilder wherever there is a filter on a transformed column.
* refactor: Load swagger and openapi dependencies conditionally
* disable public api in tests to reduce heal usage
* update the link and text in SettingsApiView when swagger ui is disabled
* ✨ Create rule `no-unneeded-backticks`
* 👕 Enable rule
* ⚡ Run rule on `cli`
* ⚡ Run rule on `core`
* ⚡ Run rule on `workflow`
* ⚡ Rule rule on `design-system`
* ⚡ Run rule on `node-dev`
* ⚡ Run rule on `editor-ui`
* ⚡ Run rule on `nodes-base`
* fix: Prevent workflows with only manual trigger from being activated
* fix: Fix workflow id when sharing from workflows list
* fix: Update sharing modal translations
* fix: Allow sharees to disable workflows and fix issue with unique key when removing a user
* refactor: Improve error messages and change logging level to be less verbose
* fix: Broken user removal transfer issue
* feat: Implement workflow sharing BE telemetry
* chore: temporarily add sharing env vars
* feat: Implement BE telemetry for workflow sharing
* fix: Prevent issues with possibly missing workflow id
* feat: Replace WorkflowSharing flag references (no-changelog) (#4918)
* ci: Block all external network calls in tests (no-changelog) (#4930)
* setup nock to prevent tests from making any external requests
* mock all calls to posthog sdk
* feat: Replace WorkflowSharing flag references (no-changelog)
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <netroy@users.noreply.github.com>
* refactor: Remove temporary feature flag for workflow sharing
* refactor: add sharing_role to both manual and node executions
* refactor: Allow changing name, position and disabled of read only nodes
* feat: Overhaul dynamic translations for local and cloud (#4943)
* feat: Overhaul dynamic translations for local and cloud
* fix: remove type casting
* chore: remove unused translations
* fix: fix workflow sharing translation
* test: Fix broken test
* refactor: remove unnecessary import
* refactor: Minor code improvements
* refactor: rename dynamicTranslations to contextBasedTranslationKeys
* fix: fix type imports
* refactor: Consolidate sharing feature check
* feat: update cred sharing unavailable translations
* feat: update upgrade message when user management not available
* fix: rename plan names to Pro and Power
* feat: update translations to no longer contain plan names
* wip: subworkflow permissions
* feat: add workflowsFromSameOwner caller policy
* feat: Fix subworkflow permissions
* shared entites should check for role when deleting users
* refactor: remove circular dependency
* role filter shouldn't be an array
* fixed role issue
* fix: Corrected behavior when removing users
* feat: show instance owner credential sharing message only if isnt sharee
* feat: update workflow caller policy caller ids labels
* feat: update upgrade plan links to contain instance ids
* fix: show check errors below creds message only to owner
* fix(editor): Hide usage page on cloud
* fix: update credential validation error message for sharee
* fix(core): Remove duplicate import
* fix(editor): Extending deployment types
* feat: Overhaul contextual translations (#4992)
feat: update how contextual translations work
* refactor: improve messageing for subworkflow permissions
* test: Fix issue with user deletion and transfer
* fix: Explicitly throw error message so it can be displayed in UI
Co-authored-by: Alex Grozav <alex@grozav.com>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <netroy@users.noreply.github.com>
Co-authored-by: freyamade <freya@n8n.io>
Co-authored-by: Csaba Tuncsik <csaba@n8n.io>
* delete unused code
* fix: Do not set the `Content-Disposition` header when viewing binary files
* remove the duplicate styles.
these already exist in BinaryDataDisplayEmbed.vue
* 🔨 - getSharedWorkflowIds returns string[]
* 🔨 - update the sharedWorkflow function in public api
* 🔨 - update existing code to handle new data type
* simplify code
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* feat: redact password field values in credentials
* feat: disable expanding password fields
* fix: redacting credentials without a valid type
This only seems to be a thing in testing?
fix: Ensure parent directory exists before copying over the icon to generated static directory.
This fixes the issue of icons not showing up for custom nodes that use a package-name with a `/` in them.
* test(editor): Fix default-owner and credentials saving e2e specs
* test(editor): fix webhook node test
* test(editor): add cy command for browser permissions
* test(editor): add cy command for reading clipboard
* Fix 3-default-owner spec
* Resolve review comments
* Merge spec
* Fix http node and expression editor modal specs
* Add optional param to credentials modal saving action to wait for the test endpoint
* Improve sidebar items clicking and increase credentials saving timeout
* Rename http e2e spec to fix ordering
* Fix pasting and copying of nodes e2e spec
* Make sure to only access error.cause if it exists
* Wait longer for the keyboard press
* Make sure to focus the body when typing
* Try type delay
* Use meta key based on the running platform
* Fix flaky workflowTagElements getter
Co-authored-by: Csaba Tuncsik <csaba@n8n.io>
* feat: Reduce initial memory spike at server startup (no-changelog)
This changes the frontend types generation to generate less garbage for the GC to collect.
* switch to stream pipelines for writing all the static files
and, move all static file generation before the server starts
* fix: Avoid hashing workflows in the listing page
* stop returning full nodes data on the listings page when sharing is enabled
* fix the relations array for workflow listing
* add a comment explaining the hash skipping hack
* Mark binary data to be deleted when pruning executions
* eslint
* make pruneExecutionData async
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* add tests
* ci: Setup cypress tasks for resetting DB, and setting up an owner
* add test tests to check for settings
* add more tests
* clean up
* rename tag
* update test id
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* use user-folder override consistently everywhere, including for the `.cache` folder
* use consistent config for e2e tesing, skipping config loading from env and config files
* simplify all the cypress commands, and run all e2e tests on master
fix: invalid credentials should throw 401 and not 500
I missed to add these in https://github.com/n8n-io/n8n/pull/4691
I've checked not that there are no other cases of http errors that aren't using error classes. This was the last one.
* feat(editor): extract credentials view into reusable layout components for workflows view
* feat(editor): add workflow card and start work on empty state
* feat: add hoverable card and finish workflows empty state
* fix: undo workflows response interface changes
* chore: fix linting issues.
* fix: remove enterprise sharing env schema
* fix(editor): fix workflows resource view when sharing is enabled
* fix: change owner tag design and order
* feat: add personalization survey on workflows page
* fix: update component snapshots
* feat: refactored workflow card to use workflow-activator properly
* fix: fix workflow activator and proptypes
* fix: hide owner tag for workflow card until sharing is available
* fix: fixed ownedBy and sharedWith appearing for workflows list
* feat: update tags component design
* refactor: change resource filter select to n8n-user-select
* fix: made telemetry messages reusable
* chore: remove unused import
* refactor: fix component name casing
* refactor: use Vue.set to make workflow property reactive
* feat: add support for clicking on tags for filtering
* chore: fix tags linting issues
* fix: fix resources list layout when title words are very long
* refactor: add active and inactive status text to workflow activator
* fix: fix credentials and workflows sorting when name contains leading whitespace
* fix: remove wrongfully added style tag
* feat: add translations and storybook examples for truncated tags
* fix: remove enterprise sharing env from schema
* refactor: fix workflows module and workflows field store naming conflict
* feat: add workflow share button and open dummy modal
* feat: add workflow sharing modal (in progress)
* feat: add message when sharing disabled
* feat: add sharing messages based on flags
* feat: add workflow sharing api integration and readonly state handling
* fix: change how foreign credentials are handled
* refactor: migrate newly added workflow sharing store methods to pinia
* fix: update foreign credentials handler and add executable prop to node-settings
* fix: fix credentials display issue caused by addCredentials override
* fix: fix various issues when sharing from empty state
* fix: update node duplication credentials
* fix: revert defautl values for sharing env
* feat: hide share button behind feature flag
* chore: add env variable for sharing feature (testing only)
* fix: change enterprise-edition component casing
