Commit graph

154 commits

Author SHA1 Message Date
Iván Ovejero 9ac8825a67
refactor(core): Move error execution creation to execution service (no-changelog) (#8006)
Continue breaking down legacy helpers.

Note: `getUserById` is unused.
2023-12-21 14:15:46 +01:00
कारतोफ्फेलस्क्रिप्ट™ a63d94f28c
refactor(core): Move license endpoints to a decorated controller class (no-changelog) (#8074) 2023-12-19 12:13:19 +01:00
Iván Ovejero d0e44d450f
feat(core): Add multi-main setup debug endpoint (no-changelog) (#7991)
## Summary
Provide details about your pull request and what it adds, fixes, or
changes. Photos and videos are recommended.
Adi's idea here to help diagnose:
https://n8nio.slack.com/archives/C069KJBJ8HE/p1702300349277609?thread_ts=1702299930.728029&cid=C069KJBJ8HE
...

#### How to test the change:
1. ...


## Issues fixed
Include links to Github issue or Community forum post or **Linear
ticket**:
> Important in order to close automatically and provide context to
reviewers

...


## Review / Merge checklist
- [ ] PR title and summary are descriptive. **Remember, the title
automatically goes into the changelog. Use `(no-changelog)` otherwise.**
([conventions](https://github.com/n8n-io/n8n/blob/master/.github/pull_request_title_conventions.md))
- [ ] [Docs updated](https://github.com/n8n-io/n8n-docs) or follow-up
ticket created.
- [ ] Tests included.
> A bug is not considered fixed, unless a test is added to prevent it
from happening again. A feature is not complete without tests.
  >
> *(internal)* You can use Slack commands to trigger [e2e
tests](https://www.notion.so/n8n/How-to-use-Test-Instances-d65f49dfc51f441ea44367fb6f67eb0a?pvs=4#a39f9e5ba64a48b58a71d81c837e8227)
or [deploy test
instance](https://www.notion.so/n8n/How-to-use-Test-Instances-d65f49dfc51f441ea44367fb6f67eb0a?pvs=4#f6a177d32bde4b57ae2da0b8e454bfce)
or [deploy early access version on
Cloud](https://www.notion.so/n8n/Cloudbot-3dbe779836004972b7057bc989526998?pvs=4#fef2d36ab02247e1a0f65a74f6fb534e).
2023-12-12 15:18:32 +01:00
Iván Ovejero c378f60a25
refactor(core): Introduce password utility (no-changelog) (#7979)
## Summary
Provide details about your pull request and what it adds, fixes, or
changes. Photos and videos are recommended.
Continue breaking down `UserManagementHelper.ts`
...

#### How to test the change:
1. ...


## Issues fixed
Include links to Github issue or Community forum post or **Linear
ticket**:
> Important in order to close automatically and provide context to
reviewers

...


## Review / Merge checklist
- [ ] PR title and summary are descriptive. **Remember, the title
automatically goes into the changelog. Use `(no-changelog)` otherwise.**
([conventions](https://github.com/n8n-io/n8n/blob/master/.github/pull_request_title_conventions.md))
- [ ] [Docs updated](https://github.com/n8n-io/n8n-docs) or follow-up
ticket created.
- [ ] Tests included.
> A bug is not considered fixed, unless a test is added to prevent it
from happening again. A feature is not complete without tests.
  >
> *(internal)* You can use Slack commands to trigger [e2e
tests](https://www.notion.so/n8n/How-to-use-Test-Instances-d65f49dfc51f441ea44367fb6f67eb0a?pvs=4#a39f9e5ba64a48b58a71d81c837e8227)
or [deploy test
instance](https://www.notion.so/n8n/How-to-use-Test-Instances-d65f49dfc51f441ea44367fb6f67eb0a?pvs=4#f6a177d32bde4b57ae2da0b8e454bfce)
or [deploy early access version on
Cloud](https://www.notion.so/n8n/Cloudbot-3dbe779836004972b7057bc989526998?pvs=4#fef2d36ab02247e1a0f65a74f6fb534e).
2023-12-11 18:23:42 +01:00
Iván Ovejero 29e7a98f3e
test(core): Use license mocker in RBAC tests (no-changelog) (#7912)
## Summary
Provide details about your pull request and what it adds, fixes, or
changes. Photos and videos are recommended.

...

#### How to test the change:
1. ...


## Issues fixed
Include links to Github issue or Community forum post or **Linear
ticket**:
> Important in order to close automatically and provide context to
reviewers

...


## Review / Merge checklist
- [ ] PR title and summary are descriptive. **Remember, the title
automatically goes into the changelog. Use `(no-changelog)` otherwise.**
([conventions](https://github.com/n8n-io/n8n/blob/master/.github/pull_request_title_conventions.md))
- [ ] [Docs updated](https://github.com/n8n-io/n8n-docs) or follow-up
ticket created.
- [ ] Tests included.
> A bug is not considered fixed, unless a test is added to prevent it
from happening again. A feature is not complete without tests.
  >
> *(internal)* You can use Slack commands to trigger [e2e
tests](https://www.notion.so/n8n/How-to-use-Test-Instances-d65f49dfc51f441ea44367fb6f67eb0a?pvs=4#a39f9e5ba64a48b58a71d81c837e8227)
or [deploy test
instance](https://www.notion.so/n8n/How-to-use-Test-Instances-d65f49dfc51f441ea44367fb6f67eb0a?pvs=4#f6a177d32bde4b57ae2da0b8e454bfce)
or [deploy early access version on
Cloud](https://www.notion.so/n8n/Cloudbot-3dbe779836004972b7057bc989526998?pvs=4#fef2d36ab02247e1a0f65a74f6fb534e).
2023-12-04 13:56:48 +01:00
Val 5f4a9524ec
refactor(core): Add central license mock for integration tests (no-changelog) (#7871)
Github issue / Community forum post (link here to close automatically):
2023-11-30 09:23:09 +01:00
Val 1cb92ffe16
feat: Replace owner checks with scope checks (no-changelog) (#7846)
Github issue / Community forum post (link here to close automatically):
2023-11-29 14:48:36 +00:00
Iván Ovejero 476806ebb0
feat(core): Allow admin creation (#7837)
https://linear.app/n8n/issue/PAY-1038
2023-11-29 13:55:41 +01:00
Iván Ovejero 2356fb0f0c
feat(core): Set up endpoint for all existing roles with license flag (#7834)
https://linear.app/n8n/issue/PAY-1034/create-endpoint-to-list-all-existing-roles
2023-11-28 14:16:47 +01:00
Val 5acb7b94c0
refactor: Refactor variables controller into a RestController (no-changelog) (#7822)
Github issue / Community forum post (link here to close automatically):
2023-11-27 12:17:09 +00:00
Iván Ovejero 75a5807c72
perf(core): Make user controller tests faster (no-changelog) (#7819)
Before: `17.949 s`
After: `3.886 s`

Followup to:
https://github.com/n8n-io/n8n/pull/7797#discussion_r1404148034
2023-11-27 11:56:06 +01:00
Iván Ovejero 7a86d36068
feat(core): Allow user role modification (#7797)
https://linear.app/n8n/issue/PAY-985

```
PATCH /users/:id/role
  unauthenticated user
    ✓ should receive 401 (349 ms)
  member
    ✓ should fail to demote owner to member (349 ms)
    ✓ should fail to demote owner to admin (359 ms)
    ✓ should fail to demote admin to member (381 ms)
    ✓ should fail to promote other member to owner (353 ms)
    ✓ should fail to promote other member to admin (377 ms)
    ✓ should fail to promote self to admin (354 ms)
    ✓ should fail to promote self to owner (371 ms)
  admin
    ✓ should receive 400 on invalid payload (351 ms)
    ✓ should receive 404 on unknown target user (351 ms)
    ✓ should fail to demote owner to admin (349 ms)
    ✓ should fail to demote owner to member (347 ms)
    ✓ should fail to promote member to owner (384 ms)
    ✓ should fail to promote admin to owner (350 ms)
    ✓ should be able to demote admin to member (354 ms)
    ✓ should be able to demote self to member (350 ms)
    ✓ should be able to promote member to admin (349 ms)
  owner
    ✓ should be able to promote member to admin (349 ms)
    ✓ should be able to demote admin to member (349 ms)
    ✓ should fail to demote self to admin (348 ms)
    ✓ should fail to demote self to member (354 ms)
```
2023-11-24 11:40:08 +01:00
Iván Ovejero 4c4082503c
feat(core): Coordinate manual workflow activation and deactivation in multi-main scenario (#7643)
Followup to #7566 | Story: https://linear.app/n8n/issue/PAY-926

### Manual workflow activation and deactivation

In a multi-main scenario, if the user manually activates or deactivates
a workflow, the process (whether leader or follower) that handles the
PATCH request and updates its internal state should send a message into
the command channel, so that all other main processes update their
internal state accordingly:

- Add to `ActiveWorkflows` if activating
- Remove from `ActiveWorkflows` if deactivating
- Remove and re-add to `ActiveWorkflows` if the update did not change
activation status.

After updating their internal state, if activating or deactivating, the
recipient main processes should push a message to all connected
frontends so that these can update their stores and so reflect the value
in the UI.

### Workflow activation errors

On failure to activate a workflow, the main instance should record the
error in Redis - main instances should always pull activation errors
from Redis in a multi-main scenario.

### Leadership change

On leadership change...

- The old leader should stop pruning and the new leader should start
pruning.
- The old leader should remove trigger- and poller-based workflows and
the new leader should add them.
2023-11-17 15:58:50 +01:00
Ricardo Espinoza 8e0ae3cf8c
refactor: Extract Invitation routes to InvitationController (no-changelog) (#7726)
This PR:

- Creates `InvitationController`
- Moves `POST /users` to `POST /invitations` and move related test to
`invitations.api.tests`
- Moves `POST /users/:id` to `POST /invitations/:id/accept` and move
related test to `invitations.api.tests`
- Adjusts FE to use new endpoints
- Moves all the invitation logic to the `UserService`

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
2023-11-16 12:39:43 -05:00
कारतोफ्फेलस्क्रिप्ट™ 000e76e3b4
ci(core): Reduce memory usage in tests (part-2) (no-changelog) (#7671)
This also gets rid of `Db.collection`, which was another source of
circular dependencies.
2023-11-10 15:04:26 +01:00
कारतोफ्फेलस्क्रिप्ट™ 5edf722209
ci: Stop disabling linting rules that are explicitly set to warn (no-changelog) (#7669) 2023-11-09 14:05:39 +01:00
कारतोफ्फेलस्क्रिप्ट™ 0346b211a7
ci(core): Reduce memory usage in tests (part-1) (no-changelog) (#7654) 2023-11-08 16:29:39 +01:00
कारतोफ्फेलस्क्रिप्ट™ 60314248f4
fix(core): Make password-reset urls valid only for single-use (#7622) 2023-11-07 15:35:43 +01:00
Iván Ovejero c857e42677
feat(core): Coordinate workflow activation in multiple main scenario in internal API (#7566)
Story: https://linear.app/n8n/issue/PAY-926

This PR coordinates workflow activation on instance startup and on
leadership change in multiple main scenario in the internal API. Part 3
on manual workflow activation and deactivation will be a separate PR.

### Part 1: Instance startup

In multi-main scenario, on starting an instance...
- [x] If the instance is the leader, it should add webhooks, triggers
and pollers.
- [x] If the instance is the follower, it should not add webhooks,
triggers or pollers.
- [x] Unit tests.

### Part 2: Leadership change 

In multi-main scenario, if the main instance leader dies…

- [x] The new main instance leader must activate all trigger- and
poller-based workflows, excluding webhook-based workflows.
- [x] The old main instance leader must deactivate all trigger- and
poller-based workflows, excluding webhook-based workflows.
- [x] Unit tests.

To test, start two instances and check behavior on startup and
leadership change:

```
EXECUTIONS_MODE=queue N8N_LEADER_SELECTION_ENABLED=true N8N_LICENSE_TENANT_ID=... N8N_LICENSE_ACTIVATION_KEY=... N8N_LOG_LEVEL=debug npm run start

EXECUTIONS_MODE=queue N8N_LEADER_SELECTION_ENABLED=true N8N_LICENSE_TENANT_ID=... N8N_LICENSE_ACTIVATION_KEY=... N8N_LOG_LEVEL=debug N8N_PORT=5679 npm run start
```
2023-11-07 13:48:48 +01:00
Michael Auerswald b350568505
fix(core): Fix data decryption on credentials import (#7560)
Due to a change, during the credentials import command, the core's
Credential object is being called through its prototype. This caused the
Credential's cipher variable to not be set, thus no cipher service being
available during import. This fix catches this edge case and provides a
fix.
2023-10-31 13:15:09 +01:00
Iván Ovejero 62c096710f
refactor: Run lintfix (no-changelog) (#7537)
- Fix autofixable violations
- Remove unused directives
- Allow for PascalCased variables - needed for dynamically imported or
assigned classes, decorators, routers, etc.
2023-10-27 14:15:02 +02:00
कारतोफ्फेलस्क्रिप्ट™ 05586a900d
refactor(core): Make Logger a service (no-changelog) (#7494) 2023-10-25 16:35:22 +02:00
Val 93cfabbeac
fix(core): Fix workflow activation with history and workflow history for EE (no-changelog) (#7508)
Github issue / Community forum post (link here to close automatically):
2023-10-25 11:07:11 +01:00
कारतोफ्फेलस्क्रिप्ट™ b6de910cbe
refactor(core): Abstract away InstanceSettings and encryptionKey into injectable services (no-changelog) (#7471)
This change ensures that things like `encryptionKey` and `instanceId`
are always available directly where they are needed, instead of passing
them around throughout the code.
2023-10-23 13:39:35 +02:00
कारतोफ्फेलस्क्रिप्ट™ 55c6a1b0d3
fix(core): Do not return inviteAcceptUrl in response if email was sent (#7465) 2023-10-19 13:58:06 +02:00
कारतोफ्फेलस्क्रिप्ट™ c5ee06cc61
refactor(core): Refactor nodes loading (no-changelog) (#7283)
fixes PAY-605
2023-10-09 16:09:23 +02:00
Iván Ovejero 34bda535e6
refactor(core): Create controller for binary data (no-changelog) (#7363)
This PR adds a controller for binary data + integration tests.
2023-10-06 16:21:13 +02:00
Iván Ovejero 1a661e6d00
feat(core): Integrate object store as binary data manager (#7253)
Depends on: #7225 | Story:
[PAY-848](https://linear.app/n8n/issue/PAY-848)

This PR integrates the object store service as a new binary data manager
for Enterprise.
2023-10-05 15:25:17 +02:00
कारतोफ्फेलस्क्रिप्ट™ 169175080a
ci: Fix WorkflowHistoryManager tests (no-changelog) (#7356)
[DB Tests](https://github.com/n8n-io/n8n/actions/runs/6418058186)
2023-10-05 13:14:57 +02:00
Tomi Turtiainen 1b4848afcb
fix(core): Fix pruning of non-finished executions (#7333)
This fixes a bug in the pruning (soft-delete). The pruning was a bit too
aggressive, as it also pruned executions that weren't in an end state
yet. This only becomes an issue if there are long-running executions
(e.g. workflow with Wait node) or the prune parameters are set to keep
only a tiny number of executions.
2023-10-04 16:32:05 +03:00
Val 0adc533719
feat: Workflow History pruning and prune time settings (#7343)
Github issue / Community forum post (link here to close automatically):
2023-10-04 13:57:21 +01:00
Val 0083a9e45d
feat(core): Initial workflow history API (#7234)
Github issue / Community forum post (link here to close automatically):
2023-09-27 15:22:39 +01:00
Iván Ovejero 6d6e2488c6
refactor(core): Generalize binary data manager interface (no-changelog) (#7164)
Depends on: #7092 | Story:
[PAY-768](https://linear.app/n8n/issue/PAY-768)

This PR: 
- Generalizes the `IBinaryDataManager` interface.
- Adjusts `Filesystem.ts` to satisfy the interface.
- Sets up an S3 client stub to be filled in in the next PR.
- Turns `BinaryDataManager` into an injectable service.
- Adjusts the config schema and adds new validators.

Note that the PR looks large but all the main changes are in
`packages/core/src/binaryData`.

Out of scope:
- `BinaryDataManager` (now `BinaryDataService`) and `Filesystem.ts` (now
`fs.client.ts`) were slightly refactored for maintainability, but fully
overhauling them is **not** the focus of this PR, which is meant to
clear the way for the S3 implementation. Future improvements for these
two should include setting up a backwards-compatible dir structure that
makes it easier to locate binary data files to delete, removing
duplication, simplifying cloning methods, using integers for binary data
size instead of `prettyBytes()`, writing tests for existing binary data
logic, etc.

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
2023-09-22 17:22:12 +02:00
Iván Ovejero cd08c8e4c6
refactor(core): Implement soft-deletions for executions (#7092)
Based on #7065 | Story: https://linear.app/n8n/issue/PAY-771

n8n on filesystem mode marks binary data to delete on manual execution
deletion, on unsaved execution completion, and on every execution
pruning cycle. We later prune binary data in a separate cycle via these
marker files, based on the configured TTL. In the context of introducing
an S3 client to manage binary data, the filesystem mode's mark-and-prune
setup is too tightly coupled to the general binary data management
client interface.

This PR...
- Ensures the deletion of an execution causes the deletion of any binary
data associated to it. This does away with the need for binary data TTL
and simplifies the filesystem mode's mark-and-prune setup.
- Refactors all execution deletions (including pruning) to cause soft
deletions, hard-deletes soft-deleted executions based on the existing
pruning config, and adjusts execution endpoints to filter out
soft-deleted executions. This reduces DB load, and keeps binary data
around long enough for users to access it when building workflows with
unsaved executions.
- Moves all execution pruning work from an execution lifecycle hook to
`execution.repository.ts`. This keeps related logic in a single place.
- Removes all marking logic from the binary data manager. This
simplifies the interface that the S3 client will meet.
- Adds basic sanity-check tests to pruning logic and execution deletion.

Out of scope:

- Improving existing pruning logic.
- Improving existing execution repository logic.
- Adjusting dir structure for filesystem mode.

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
2023-09-20 15:21:42 +02:00
कारतोफ्फेलस्क्रिप्ट™ 1d1a022def
feat(core): Add an option to enable WAL mode for SQLite (#7118)
https://www.sqlite.org/wal.html
2023-09-07 15:58:48 +02:00
कारतोफ्फेलस्क्रिप्ट™ 2f647974f4
refactor: Use actual nodes/credentials in cli and core tests (no-changelog) (#7115) 2023-09-06 10:23:47 +02:00
कारतोफ्फेलस्क्रिप्ट™ 36f246929e
ci: Fix linting (no-changelog) (#7110) 2023-09-05 14:41:56 +02:00
Michael Auerswald 011ee2e04b
fix(core): Split event bus controller into community and ee (#7107) 2023-09-05 13:32:09 +02:00
कारतोफ्फेलस्क्रिप्ट™ 6aa7b93473
refactor(core): Deprecate prepareOutputData (no-changelog) (#7091) 2023-09-05 12:59:02 +02:00
Iván Ovejero 51093f649d
refactor: Move community package logic to service (no-changelog) (#6973) 2023-09-01 15:13:19 +02:00
कारतोफ्फेलस्क्रिप्ट™ 281c0ae829
ci: Define explicit charset for mysql/mariadb test database (no-changelog) (#7053)
[DB
Tests](https://github.com/n8n-io/n8n/actions/runs/6035513299/job/16375993427)
2023-08-31 11:57:40 +02:00
Iván Ovejero b716241b42
feat(core): Add filtering, selection and pagination to users (#6994)
https://linear.app/n8n/issue/PAY-646
2023-08-28 16:13:17 +02:00
Iván Ovejero 87cf1d9c1b
refactor(core): Make controller constructors consistent (no-changelog) (#7015) 2023-08-25 13:23:22 +02:00
Alex Grozav ed927d34b2
feat: External Secrets storage for credentials (#6477)
Github issue / Community forum post (link here to close automatically):

---------

Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
Co-authored-by: Valya Bullions <valya@n8n.io>
Co-authored-by: Csaba Tuncsik <csaba@n8n.io>
Co-authored-by: Giulio Andreini <g.andreini@gmail.com>
Co-authored-by: Omar Ajoue <krynble@gmail.com>
2023-08-25 10:33:46 +02:00
Ricardo Espinoza 2b7ba6fdf1
feat(core): Add MFA (#4767)
https://linear.app/n8n/issue/ADO-947/sync-branch-with-master-and-fix-fe-e2e-tets

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
2023-08-23 22:59:16 -04:00
Iván Ovejero 96a9de68a0
refactor(core): Move all user DB access to UserRepository (#6910)
Prep for https://linear.app/n8n/issue/PAY-646
2023-08-22 15:58:05 +02:00
Iván Ovejero 2cfa6d344e
refactor: Consolidate WorkflowService.getMany() (no-changelog) (#6892)
In scope:

- Consolidate `WorkflowService.getMany()`.
- Support non-entity field `ownedBy` for `select`.
- Support `tags` for `filter`.
- Move `addOwnerId` to `OwnershipService`.
- Remove unneeded check for `filter.id`.
- Simplify DTO validation for `filter` and `select`.
- Expand tests for `GET /workflows`.

Workflow list query DTOs:

```
filter → name, active, tags
select → id, name, active, tags, createdAt, updatedAt, versionId, ownedBy
```

Out of scope:

- Migrate `shared_workflow.roleId` and `shared_credential.roleId` to
string IDs.
- Refactor `WorkflowHelpers.getSharedWorkflowIds()`.
2023-08-22 13:19:37 +02:00
कारतोफ्फेलस्क्रिप्ट™ 10c15874b2
fix(core): Restore body parsing for all content-types for non webhook routes (no-changelog)(#6911)
Changes in https://github.com/n8n-io/n8n/pull/6394 removed xml body parsing for all non-webhook routes. This broken SAML endpoints as they need the XML body parser to function correctly.
2023-08-11 09:18:33 +02:00
Iván Ovejero 11440bfd3c
refactor(core): Move tag collection into repository (no-changelog) (#6860)
* refactor(core): Move tag collection into repository

* Fix tests

* Address feedback

* Fix missing spot
2023-08-08 14:08:56 +02:00
Michael Auerswald adcf5a96e8
feat(core): Add metrics option to cache (#6846)
* add metrics to cache

* use events for metrics

* pr comments / broken test

* lint fix

* update the test

* improve tests

* Update packages/cli/src/config/schema.ts

* disable flaky test

* lint fix

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <netroy@users.noreply.github.com>
2023-08-04 20:51:07 +02:00