Commit graph

321 commits

Author SHA1 Message Date
Iván Ovejero 29e7a98f3e
test(core): Use license mocker in RBAC tests (no-changelog) (#7912)
## Summary
Provide details about your pull request and what it adds, fixes, or
changes. Photos and videos are recommended.

...

#### How to test the change:
1. ...


## Issues fixed
Include links to Github issue or Community forum post or **Linear
ticket**:
> Important in order to close automatically and provide context to
reviewers

...


## Review / Merge checklist
- [ ] PR title and summary are descriptive. **Remember, the title
automatically goes into the changelog. Use `(no-changelog)` otherwise.**
([conventions](https://github.com/n8n-io/n8n/blob/master/.github/pull_request_title_conventions.md))
- [ ] [Docs updated](https://github.com/n8n-io/n8n-docs) or follow-up
ticket created.
- [ ] Tests included.
> A bug is not considered fixed, unless a test is added to prevent it
from happening again. A feature is not complete without tests.
  >
> *(internal)* You can use Slack commands to trigger [e2e
tests](https://www.notion.so/n8n/How-to-use-Test-Instances-d65f49dfc51f441ea44367fb6f67eb0a?pvs=4#a39f9e5ba64a48b58a71d81c837e8227)
or [deploy test
instance](https://www.notion.so/n8n/How-to-use-Test-Instances-d65f49dfc51f441ea44367fb6f67eb0a?pvs=4#f6a177d32bde4b57ae2da0b8e454bfce)
or [deploy early access version on
Cloud](https://www.notion.so/n8n/Cloudbot-3dbe779836004972b7057bc989526998?pvs=4#fef2d36ab02247e1a0f65a74f6fb534e).
2023-12-04 13:56:48 +01:00
Val 5f4a9524ec
refactor(core): Add central license mock for integration tests (no-changelog) (#7871)
Github issue / Community forum post (link here to close automatically):
2023-11-30 09:23:09 +01:00
Val cd474f1562
feat: Allow owner to share workflows/credentials they don't own (no-changelog) (#7869)
Github issue / Community forum post (link here to close automatically):
2023-11-29 16:32:27 +00:00
Val 1cb92ffe16
feat: Replace owner checks with scope checks (no-changelog) (#7846)
Github issue / Community forum post (link here to close automatically):
2023-11-29 14:48:36 +00:00
Iván Ovejero 476806ebb0
feat(core): Allow admin creation (#7837)
https://linear.app/n8n/issue/PAY-1038
2023-11-29 13:55:41 +01:00
Val e282ea242d
fix: Return scopes on owner setup endpoint (no-changelog) (#7860)
Github issue / Community forum post (link here to close automatically):
2023-11-29 11:33:32 +00:00
Iván Ovejero c08c5cc37b
refactor(core): Switch plain errors in cli to ApplicationError (#7857)
Ensure all errors in `cli` are `ApplicationError` or children of it and
contain no variables in the message, to continue normalizing all the
errors we report to Sentry

Follow-up to: https://github.com/n8n-io/n8n/pull/7839
2023-11-29 12:25:10 +01:00
Iván Ovejero 2356fb0f0c
feat(core): Set up endpoint for all existing roles with license flag (#7834)
https://linear.app/n8n/issue/PAY-1034/create-endpoint-to-list-all-existing-roles
2023-11-28 14:16:47 +01:00
Iván Ovejero 1c6178759c
refactor(core): Reorganize error hierarchy in cli package (no-changelog) (#7839)
Ensure all errors in `cli` inherit from `ApplicationError` to continue
normalizing all the errors we report to Sentry

Follow-up to: https://github.com/n8n-io/n8n/pull/7820
2023-11-28 10:19:27 +01:00
Iván Ovejero 9b87a596ca
fix(core): Ensure member and admin cannot be promoted to owner (#7830)
https://linear.app/n8n/issue/PAY-985/add-user-role-modification-endpoint#comment-62355f6b
2023-11-27 17:35:58 +01:00
Omar Ajoue 847f6ac771
fix(core): Prevent error messages due to statistics about data loading (#7824)
Statistics collection about the first time a workflow loads data simply
attempts an insert to db, and if it fails, we just ignore.

This was causing this query to fire against production workflows
multiple times, and since we want to insert only and detect whether the
insertion failed, performing a select first provides gains both in terms
of performance, as it's usually faster than trying an insertion as well
as preventing unnecessary noise in logs.

Github issue / Community forum post (link here to close automatically):

https://community.n8n.io/t/duplicate-key-value-violates-unique-constraint-workflow-statistics-pkey-still-happening/29283
https://github.com/n8n-io/n8n/issues/7256
https://community.n8n.io/t/error-log-arriving-in-postgres/30191
https://github.com/n8n-io/n8n/issues/7256

https://community.n8n.io/t/cant-launch-webhooks-unable-to-find-data-of-execution/31867

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
2023-11-27 15:43:48 +01:00
Val 5acb7b94c0
refactor: Refactor variables controller into a RestController (no-changelog) (#7822)
Github issue / Community forum post (link here to close automatically):
2023-11-27 12:17:09 +00:00
Iván Ovejero 7b8532d3a3
refactor(core): Move execution progress saving to standalone utility (no-changelog) (#7770)
This PR continues the effort of moving logic inside execution lifecycle
hooks into standalone testable functions, as a stepping stone to
refactoring the hooks themselves.
2023-11-27 13:10:43 +01:00
Iván Ovejero 75a5807c72
perf(core): Make user controller tests faster (no-changelog) (#7819)
Before: `17.949 s`
After: `3.886 s`

Followup to:
https://github.com/n8n-io/n8n/pull/7797#discussion_r1404148034
2023-11-27 11:56:06 +01:00
कारतोफ्फेलस्क्रिप्ट™ 1b60cfb8f1
ci: Fix new user patching endpoint tests (no-changelog) (#7816)
user ids are uuids, and in the future we should add proper input
validation to prevent invalid user ids reaching the DB like this.
2023-11-27 09:35:09 +01:00
Iván Ovejero 7a86d36068
feat(core): Allow user role modification (#7797)
https://linear.app/n8n/issue/PAY-985

```
PATCH /users/:id/role
  unauthenticated user
    ✓ should receive 401 (349 ms)
  member
    ✓ should fail to demote owner to member (349 ms)
    ✓ should fail to demote owner to admin (359 ms)
    ✓ should fail to demote admin to member (381 ms)
    ✓ should fail to promote other member to owner (353 ms)
    ✓ should fail to promote other member to admin (377 ms)
    ✓ should fail to promote self to admin (354 ms)
    ✓ should fail to promote self to owner (371 ms)
  admin
    ✓ should receive 400 on invalid payload (351 ms)
    ✓ should receive 404 on unknown target user (351 ms)
    ✓ should fail to demote owner to admin (349 ms)
    ✓ should fail to demote owner to member (347 ms)
    ✓ should fail to promote member to owner (384 ms)
    ✓ should fail to promote admin to owner (350 ms)
    ✓ should be able to demote admin to member (354 ms)
    ✓ should be able to demote self to member (350 ms)
    ✓ should be able to promote member to admin (349 ms)
  owner
    ✓ should be able to promote member to admin (349 ms)
    ✓ should be able to demote admin to member (349 ms)
    ✓ should fail to demote self to admin (348 ms)
    ✓ should fail to demote self to member (354 ms)
```
2023-11-24 11:40:08 +01:00
Milorad FIlipović 77bc8ecd4b
feat(editor): Show avatars for users currently working on the same workflow (#7763)
This PR introduces the following changes:
- New Vue stores: `collaborationStore` and `pushConnectionStore`
- Front-end push connection handling overhaul: Keep only a singe
connection open and handle it from the new store
- Add user avatars in the editor header when there are multiple users
working on the same workflow
- Sending a heartbeat event to back-end service periodically to confirm
user is still active

- Back-end overhauls (authored by @tomi):
  - Implementing a cleanup procedure that removes inactive users
  - Refactoring collaboration service current implementation

---------

Co-authored-by: Tomi Turtiainen <10324676+tomi@users.noreply.github.com>
2023-11-23 10:14:34 +01:00
Michael Kret 99a9ea497a
feat(core): Add Support for custom CORS origins for webhooks (#7455)
node-850
https://community.n8n.io/t/add-ability-to-set-cors-allow-list-in-n8n-webhooks/7610
https://community.n8n.io/t/configure-cors-pre-flight-request-option-method-in-the-roadmap/32189

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
2023-11-22 17:49:56 +01:00
Milorad FIlipović 2064f7f251
fix(editor): Validate user info before submiting (#7608)
Validate first and last names before saving them to database. This
should prevent security issue with un-sanitized data that ends up in
emails.

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
2023-11-22 12:14:44 +01:00
Iván Ovejero 3459eb6c2f
refactor(core): Include execution progress in save settings (no-changelog) (#7769) 2023-11-21 17:33:44 +01:00
Iván Ovejero 4c4082503c
feat(core): Coordinate manual workflow activation and deactivation in multi-main scenario (#7643)
Followup to #7566 | Story: https://linear.app/n8n/issue/PAY-926

### Manual workflow activation and deactivation

In a multi-main scenario, if the user manually activates or deactivates
a workflow, the process (whether leader or follower) that handles the
PATCH request and updates its internal state should send a message into
the command channel, so that all other main processes update their
internal state accordingly:

- Add to `ActiveWorkflows` if activating
- Remove from `ActiveWorkflows` if deactivating
- Remove and re-add to `ActiveWorkflows` if the update did not change
activation status.

After updating their internal state, if activating or deactivating, the
recipient main processes should push a message to all connected
frontends so that these can update their stores and so reflect the value
in the UI.

### Workflow activation errors

On failure to activate a workflow, the main instance should record the
error in Redis - main instances should always pull activation errors
from Redis in a multi-main scenario.

### Leadership change

On leadership change...

- The old leader should stop pruning and the new leader should start
pruning.
- The old leader should remove trigger- and poller-based workflows and
the new leader should add them.
2023-11-17 15:58:50 +01:00
Iván Ovejero b4ebb1a28d
fix(core): Account for non-ASCII chars in filename on binary data download (#7742)
https://n8nio.sentry.io/issues/4641538638
2023-11-17 10:07:44 +01:00
Ricardo Espinoza 8e0ae3cf8c
refactor: Extract Invitation routes to InvitationController (no-changelog) (#7726)
This PR:

- Creates `InvitationController`
- Moves `POST /users` to `POST /invitations` and move related test to
`invitations.api.tests`
- Moves `POST /users/:id` to `POST /invitations/:id/accept` and move
related test to `invitations.api.tests`
- Adjusts FE to use new endpoints
- Moves all the invitation logic to the `UserService`

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
2023-11-16 12:39:43 -05:00
Val d39bb2540f
feat: Add scopes to /login endpoint (no-changelog) (#7718)
Github issue / Community forum post (link here to close automatically):
2023-11-16 11:11:55 +00:00
Ricardo Espinoza 4020c14d59
refactor: Use POST /users to re-invite users (no-changelog) (#7714) 2023-11-15 06:40:57 -05:00
कारतोफ्फेलस्क्रिप्ट™ 3460eb5eeb
fix(core): Initialize JWT Secret before it's used anywhere (#7707)
HELP-394
2023-11-15 12:17:18 +01:00
Iván Ovejero b2ca050031
perf(core): Lazyload security audit reporters (#7696)
Also converting to service.

Followup to https://github.com/n8n-io/n8n/pull/7663
2023-11-13 11:50:43 +01:00
Ricardo Espinoza 0ddafd2b82
test: Unify users.controller.test and users.api.test (no-changelog) (#7658)
Groundwork to be able to safely refactor and move the invitation logic
to the UserService.

Fixes ADO-1358

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
2023-11-10 09:41:58 -05:00
कारतोफ्फेलस्क्रिप्ट™ 000e76e3b4
ci(core): Reduce memory usage in tests (part-2) (no-changelog) (#7671)
This also gets rid of `Db.collection`, which was another source of
circular dependencies.
2023-11-10 15:04:26 +01:00
कारतोफ्फेलस्क्रिप्ट™ 5edf722209
ci: Stop disabling linting rules that are explicitly set to warn (no-changelog) (#7669) 2023-11-09 14:05:39 +01:00
कारतोफ्फेलस्क्रिप्ट™ 0346b211a7
ci(core): Reduce memory usage in tests (part-1) (no-changelog) (#7654) 2023-11-08 16:29:39 +01:00
Iván Ovejero 48c068f97b
fix(core): Comply with custom default for workflow saving settings (#7634)
https://linear.app/n8n/issue/PAY-982
2023-11-07 16:26:55 +01:00
Tomi Turtiainen ac877014ed
feat(core): Initial support for two-way communication over websockets (#7570)
- Enable two-way communication with web sockets
- Enable sending push messages to specific users
- Add collaboration service for managing active users for workflow

Missing things:
- State is currently kept only in memory, making this not work in
multi-master setups
- Removing a user from active users in situations where they go inactive
or we miss the "workflow closed" message
- I think a timer based solution for this would cover most edge cases.
I.e. have FE ping every X minutes, BE removes the user unless they have
received a ping in Y minutes, where Y > X
- FE changes to be added later by @MiloradFilipovic 

Github issue / Community forum post (link here to close automatically):

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
2023-11-07 17:26:45 +02:00
कारतोफ्फेलस्क्रिप्ट™ f0fc5b16d3
ci(core): Load config schema after process.env has been overwritten (no-changelog) (#7550) 2023-11-07 15:58:28 +01:00
कारतोफ्फेलस्क्रिप्ट™ 60314248f4
fix(core): Make password-reset urls valid only for single-use (#7622) 2023-11-07 15:35:43 +01:00
Iván Ovejero c857e42677
feat(core): Coordinate workflow activation in multiple main scenario in internal API (#7566)
Story: https://linear.app/n8n/issue/PAY-926

This PR coordinates workflow activation on instance startup and on
leadership change in multiple main scenario in the internal API. Part 3
on manual workflow activation and deactivation will be a separate PR.

### Part 1: Instance startup

In multi-main scenario, on starting an instance...
- [x] If the instance is the leader, it should add webhooks, triggers
and pollers.
- [x] If the instance is the follower, it should not add webhooks,
triggers or pollers.
- [x] Unit tests.

### Part 2: Leadership change 

In multi-main scenario, if the main instance leader dies…

- [x] The new main instance leader must activate all trigger- and
poller-based workflows, excluding webhook-based workflows.
- [x] The old main instance leader must deactivate all trigger- and
poller-based workflows, excluding webhook-based workflows.
- [x] Unit tests.

To test, start two instances and check behavior on startup and
leadership change:

```
EXECUTIONS_MODE=queue N8N_LEADER_SELECTION_ENABLED=true N8N_LICENSE_TENANT_ID=... N8N_LICENSE_ACTIVATION_KEY=... N8N_LOG_LEVEL=debug npm run start

EXECUTIONS_MODE=queue N8N_LEADER_SELECTION_ENABLED=true N8N_LICENSE_TENANT_ID=... N8N_LICENSE_ACTIVATION_KEY=... N8N_LOG_LEVEL=debug N8N_PORT=5679 npm run start
```
2023-11-07 13:48:48 +01:00
कारतोफ्फेलस्क्रिप्ट™ acec9bad71
refactor(core): Convert OAuth1/OAuth2 routes to decorated controller classes (no-changelog) (#5973) 2023-11-03 17:20:54 +01:00
Iván Ovejero 1c77d6597f
refactor(core): Ensure only leader handles licensing in multi-main scenario (#7558)
https://linear.app/n8n/issue/PAY-953/ensure-only-main-instance-leader-handles-licensing
2023-11-02 14:16:22 +01:00
Iván Ovejero 921d213ae5
refactor(core): Create pruning service (no-changelog) (#7564)
https://linear.app/n8n/issue/PAY-954/ensure-only-main-instance-leader-handles-pruning
2023-11-02 12:24:25 +01:00
Jon 479f90231d
fix(core): Fix issue that prevents owner logging in when using ldap (#7408)
This PR prioritises the internal email account over LDAP for the Owner.

---------

Co-authored-by: ricardo <ricardoespinoza105@gmail.com>
2023-11-01 23:02:49 -04:00
Omar Ajoue 437c95e84e
fix(core): Permission check for subworkflow properly checking for workflow settings (#7576)
The `sharing` related code is legacy that was not removed. Subworkflow
execution should check workflow settings alone, and this is now
reflected in the code.

Github issue / Community forum post (link here to close automatically):

https://community.n8n.io/t/bug-when-using-the-execute-workflow-node-when-workflow-is-shared/32207

---------

Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
2023-11-01 18:31:34 +01:00
Michael Auerswald b350568505
fix(core): Fix data decryption on credentials import (#7560)
Due to a change, during the credentials import command, the core's
Credential object is being called through its prototype. This caused the
Credential's cipher variable to not be set, thus no cipher service being
available during import. This fix catches this edge case and provides a
fix.
2023-10-31 13:15:09 +01:00
Iván Ovejero 442c73e63b
feat(core): Set up leader selection for multiple main instances (#7527)
https://linear.app/n8n/issue/PAY-933/set-up-leader-selection-for-multiple-main-instances

- [x] Set up new envs
- [x] Add config and license checks
- [x] Implement `MultiMainInstancePublisher`
- [x] Expand `RedisServicePubSubPublisher` to support
`MultiMainInstancePublisher`
- [x] Init `MultiMainInstancePublisher` on startup and destroy on
shutdown
- [x] Add to sandbox plans
- [x] Test manually

Note: This is only for setup - coordinating in reaction to leadership
changes will come in later PRs.
2023-10-30 16:22:32 +01:00
कारतोफ्फेलस्क्रिप्ट™ 35bb42c1b9
refactor(core): Avoid passing around static state like default timezone (no-changelog) (#7221) 2023-10-27 14:17:52 +02:00
Iván Ovejero 62c096710f
refactor: Run lintfix (no-changelog) (#7537)
- Fix autofixable violations
- Remove unused directives
- Allow for PascalCased variables - needed for dynamically imported or
assigned classes, decorators, routers, etc.
2023-10-27 14:15:02 +02:00
Iván Ovejero ae8c7a635e
refactor(core)!: Remove webhook deregistration at startup and shutdown (#7515)
https://linear.app/n8n/issue/PAY-932/deprecate-flag-to-skip-webhook-deregistration-on-shutdown
2023-10-26 14:37:54 +02:00
Iván Ovejero 5477e3fb45
refactor(core): Move execution save settings into lifecycle function (no-changelog) (#7370)
Move the handling of execution save settings into a tested lifecycle
function as discussed with Omar
2023-10-26 14:35:38 +02:00
कारतोफ्फेलस्क्रिप्ट™ 05586a900d
refactor(core): Make Logger a service (no-changelog) (#7494) 2023-10-25 16:35:22 +02:00
Val 93cfabbeac
fix(core): Fix workflow activation with history and workflow history for EE (no-changelog) (#7508)
Github issue / Community forum post (link here to close automatically):
2023-10-25 11:07:11 +01:00
Iván Ovejero 671c95760b
refactor(core): Make executions pruning more resilient (#7480)
This PR converts the hard-deletion interval to a timeout:
- to prevent the interval from not being restored when hard deletion
throws, and
- to prevent a long-running hard deletion from leading to duplicate
deletions.
2023-10-24 16:16:45 +02:00