mudhorn/n8n
1
0
Fork 0
mirror of https://github.com/n8n-io/n8n.git synced 2024-12-25 12:44:07 -08:00
Code Issues Actions Packages Projects Releases Wiki Activity
9392 commits 1140 branches 1610 tags 412 MiB
Commit graph

115 commits

Author SHA1 Message Date
कारतोफ्फेलस्क्रिप्ट™ c5ee06cc61
refactor(core): Refactor nodes loading (no-changelog) (#7283) 
fixes PAY-605
 2023-10-09 16:09:23 +02:00
Michael Auerswald afa683a06f
refactor(core): Have one orchestration service per instance type (#7303) 
webhook instances will not listen to either worker or event log messages
on the Redis pub/sub channel
 2023-10-06 13:58:11 +02:00
Iván Ovejero 1a661e6d00
feat(core): Integrate object store as binary data manager (#7253) 
Depends on: #7225 | Story:
[PAY-848](https://linear.app/n8n/issue/PAY-848)

This PR integrates the object store service as a new binary data manager
for Enterprise.
 2023-10-05 15:25:17 +02:00
Tomi Turtiainen 1b4848afcb
fix(core): Fix pruning of non-finished executions (#7333) 
This fixes a bug in the pruning (soft-delete). The pruning was a bit too
aggressive, as it also pruned executions that weren't in an end state
yet. This only becomes an issue if there are long-running executions
(e.g. workflow with Wait node) or the prune parameters are set to keep
only a tiny number of executions.
 2023-10-04 16:32:05 +03:00
Val 0adc533719
feat: Workflow History pruning and prune time settings (#7343) 
Github issue / Community forum post (link here to close automatically):
 2023-10-04 13:57:21 +01:00
Iván Ovejero 1691223789
fix(core): Account for itemless case on restoring binary data ID (#7305) 
https://linear.app/n8n/issue/PAY-862
 2023-10-02 09:13:55 +02:00
Omar Ajoue ec0379378e
feat(core): Add plan name to telemetry (no-changelog) (#7296) 
Github issue / Community forum post (link here to close automatically):
 2023-09-29 15:41:32 +02:00
Michael Auerswald 53a7502d20
feat(core): Add secrets provider reload and refactor (#7277) 
This PR adds a message for queue mode which triggers an external secrets
provider reload inside the workers if the configuration has changed on
the main instance.

It also refactors some of the message handler code to remove cyclic
dependencies, as well as remove unnecessary duplicate redis clients
inside services (thanks to no more cyclic deps)
 2023-09-28 12:57:35 +02:00
Michael Auerswald 6d7fe95c58
fix(core): Do not set ttl globally in test (no-changelog) (#7271) 2023-09-27 12:03:15 +02:00
Michael Auerswald 4b014286cf
fix(core): Make senderId required for all command messages (#7252) 
all commands sent between main instance and workers need to contain a
server id to prevent senders from reacting to their own messages,
causing loops

this PR makes sure all sent messages contain a sender id by default as
part of constructing a sending redis client.

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
 2023-09-26 13:58:06 +02:00
Iván Ovejero 77d6e3fc07
refactor(core): Include workflow ID in binary data writes (no-changelog) (#7220) 
Depends on: https://github.com/n8n-io/n8n/pull/7195 | Story:
[PAY-837](https://linear.app/n8n/issue/PAY-837/implement-object-store-manager-for-binary-data)

This PR includes `workflowId` in binary data writes so that the S3
manager can support this filepath structure
`/workflows/{workflowId}/executions/{executionId}/binaryData/{binaryFilename}`
to easily delete binary data for workflows. Also all binary data service
and manager methods that take `workflowId` and `executionId` are made
consistent in arg order.

Note: `workflowId` is included in filesystem mode for compatibility with
the common interface, but `workflowId` will remain unused by filesystem
mode until we decide to restructure how this mode stores data.

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
 2023-09-25 18:04:52 +02:00
Iván Ovejero 33991e92d0
fix(core): Fix missing execution ID in webhook-based workflow producing binary data (#7244) 
Story: https://linear.app/n8n/issue/PAY-839

This is a longstanding bug, fixed now so that the S3 backend for binary
data can use execution IDs as part of the filename.

To reproduce:

1. Set up a workflow with a POST Webhook node that accepts binary data.
2. Activate the workflow and call it sending a binary file, e.g. `curl
-X POST -F "file=@/path/to/binary/file/test.jpg"
http://localhost:5678/webhook/uuid`
3. Check `~/.n8n/binaryData`. The binary data and metadata files will be
missing the execution ID, e.g. `11869055-83c4-4493-876a-9092c4708b9b`
instead of `39011869055-83c4-4493-876a-9092c4708b9b`.
 2023-09-25 12:30:28 +02:00
Iván Ovejero cd08c8e4c6
refactor(core): Implement soft-deletions for executions (#7092) 
Based on #7065 | Story: https://linear.app/n8n/issue/PAY-771

n8n on filesystem mode marks binary data to delete on manual execution
deletion, on unsaved execution completion, and on every execution
pruning cycle. We later prune binary data in a separate cycle via these
marker files, based on the configured TTL. In the context of introducing
an S3 client to manage binary data, the filesystem mode's mark-and-prune
setup is too tightly coupled to the general binary data management
client interface.

This PR...
- Ensures the deletion of an execution causes the deletion of any binary
data associated to it. This does away with the need for binary data TTL
and simplifies the filesystem mode's mark-and-prune setup.
- Refactors all execution deletions (including pruning) to cause soft
deletions, hard-deletes soft-deleted executions based on the existing
pruning config, and adjusts execution endpoints to filter out
soft-deleted executions. This reduces DB load, and keeps binary data
around long enough for users to access it when building workflows with
unsaved executions.
- Moves all execution pruning work from an execution lifecycle hook to
`execution.repository.ts`. This keeps related logic in a single place.
- Removes all marking logic from the binary data manager. This
simplifies the interface that the S3 client will meet.
- Adds basic sanity-check tests to pruning logic and execution deletion.

Out of scope:

- Improving existing pruning logic.
- Improving existing execution repository logic.
- Adjusting dir structure for filesystem mode.

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
 2023-09-20 15:21:42 +02:00
Cornelius Suermann 050ba706d3
fix: Attempt license renewal when n8n starts (no-changelog) (#7204) 
This change is needed so that the newly introduced floating licenses
will be attempted to be claimed when n8n starts up.
 2023-09-19 12:10:23 +02:00
Michael Auerswald 9f797b96d8
feat(core): Add command to trigger license refresh on workers (#7184) 
This PR implements the updated license SDK so that worker and webhook
instances do not auto-renew licenses any more.

Instead, they receive a `reloadLicense` command via the Redis client
that will fetch the updated license after it was saved on the main
instance

This also contains some refactoring with moving redis sub and pub
clients into the event bus directly, to prevent cyclic dependency
issues.
 2023-09-17 11:05:54 +02:00
Michael Auerswald fdac2c8572
feat(core): Add rsa option to ssh key generation (#7154) 
PR adds a new field to the SourceControlPreferences as well as to the
POST parameters for the `source-control/preferences` and
`source-control/generate-key-pair` endpoints. Both now accept an
optional string parameter `keyGeneratorType` of `'ed25519' | 'rsa'`

Calling the `source-control/generate-key-pair` endpoint with the
parameter set, it will also update the stored preferences accordingly
(so that in the future new keys will use the same method)

By default ed25519 is being used. The default may be changed using a new
environment parameter:

`N8N_SOURCECONTROL_DEFAULT_SSH_KEY_TYPE` which can be `rsa` or `ed25519`

RSA keys are generated with a length of 4096 bytes.
 2023-09-14 11:34:51 +02:00
Michael Auerswald 7b49cf2a2c
feat(core): Add commands to workers to respond with current state (#7029) 
This PR adds new endpoints to the REST API:
`/orchestration/worker/status` and `/orchestration/worker/id`

Currently these just trigger the return of status / ids from the workers
via the redis back channel, this still needs to be handled and passed
through to the frontend.

It also adds the eventbus to each worker, and triggers a reload of those
eventbus instances when the configuration changes on the main instances.
 2023-09-07 14:44:19 +02:00
Iván Ovejero b7320f5322
refactor(core): Set up ExecutionMetadata service (no-changelog) (#7103) 
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
 2023-09-05 09:13:30 +02:00
Iván Ovejero 442b910ffb
refactor(core): Consolidate CredentialsService.getMany() (no-changelog) (#7028) 
Consolidate `CredentialsService.getMany()` in preparation for adding
list query middleware to `GET /credentials`.
 2023-09-04 10:37:16 +02:00
Iván Ovejero 51093f649d
refactor: Move community package logic to service (no-changelog) (#6973) 2023-09-01 15:13:19 +02:00
Iván Ovejero b716241b42
feat(core): Add filtering, selection and pagination to users (#6994) 
https://linear.app/n8n/issue/PAY-646
 2023-08-28 16:13:17 +02:00
Iván Ovejero 87cf1d9c1b
refactor(core): Make controller constructors consistent (no-changelog) (#7015) 2023-08-25 13:23:22 +02:00
Alex Grozav ed927d34b2
feat: External Secrets storage for credentials (#6477) 
Github issue / Community forum post (link here to close automatically):

---------

Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
Co-authored-by: Valya Bullions <valya@n8n.io>
Co-authored-by: Csaba Tuncsik <csaba@n8n.io>
Co-authored-by: Giulio Andreini <g.andreini@gmail.com>
Co-authored-by: Omar Ajoue <krynble@gmail.com>
 2023-08-25 10:33:46 +02:00
Iván Ovejero 96a9de68a0
refactor(core): Move all user DB access to UserRepository (#6910) 
Prep for https://linear.app/n8n/issue/PAY-646
 2023-08-22 15:58:05 +02:00
Iván Ovejero 2cfa6d344e
refactor: Consolidate WorkflowService.getMany() (no-changelog) (#6892) 
In scope:

- Consolidate `WorkflowService.getMany()`.
- Support non-entity field `ownedBy` for `select`.
- Support `tags` for `filter`.
- Move `addOwnerId` to `OwnershipService`.
- Remove unneeded check for `filter.id`.
- Simplify DTO validation for `filter` and `select`.
- Expand tests for `GET /workflows`.

Workflow list query DTOs:

```
filter → name, active, tags
select → id, name, active, tags, createdAt, updatedAt, versionId, ownedBy
```

Out of scope:

- Migrate `shared_workflow.roleId` and `shared_credential.roleId` to
string IDs.
- Refactor `WorkflowHelpers.getSharedWorkflowIds()`.
 2023-08-22 13:19:37 +02:00
Iván Ovejero dd233bdf7d
test: Fix license tests when tenantId is not default (no-changelog) (#6977) 
License tests fail if the env has a non-default `tenantId`:

<details>
<summary>Stack trace</summary>

```
nt -- license

> n8n@1.3.0 test /Users/ivov/Development/n8n/packages/cli
> pnpm test:sqlite "--" "license"


> n8n@1.3.0 test:sqlite /Users/ivov/Development/n8n/packages/cli
> N8N_LOG_LEVEL=silent DB_TYPE=sqlite jest "--" "license"

 FAIL  test/unit/License.test.ts
  License
    ✕ initializes license manager (8 ms)
    ✓ attempts to activate license with provided key
    ✓ renews license (1 ms)
    ✓ check if feature is enabled (2 ms)
    ✓ check if sharing feature is enabled (1 ms)
    ✓ check fetching entitlements
    ✓ check fetching feature values (1 ms)
    ✓ check management jwt
    ✓ getMainPlan() returns the right entitlement (3 ms)
    ✓ getMainPlan() returns undefined if there is no main plan

  ● License › initializes license manager

    expect(jest.fn()).toHaveBeenCalledWith(...expected)

    - Expected
    + Received

      Object {
        "autoRenewEnabled": true,
        "autoRenewOffset": 259200,
    -   "deviceFingerprint": Any<Function>,
    -   "loadCertStr": Any<Function>,
    -   "logger": Anything,
    +   "deviceFingerprint": [Function deviceFingerprint],
    +   "loadCertStr": [Function loadCertStr],
    +   "logger": Logger {
    +     "logger": DerivedLogger {
    +       "_events": Object {
    +         "data": [Function ondata],
    +         "end": [Function bound onceWrapper],
    +         "prefinish": [Function prefinish],
    +       },
    +       "_eventsCount": 3,
    +       "_maxListeners": undefined,
    +       "_readableState": ReadableState {
    +         "autoDestroy": false,
    +         "awaitDrain": 0,
    +         "buffer": BufferList {
    +           "head": null,
    +           "length": 0,
    +           "tail": null,
    +         },
    +         "decoder": null,
    +         "defaultEncoding": "utf8",
    +         "destroyed": false,
    +         "emitClose": true,
    +         "emittedReadable": false,
    +         "encoding": null,
    +         "endEmitted": false,
    +         "ended": false,
    +         "flowing": true,
    +         "highWaterMark": 16,
    +         "length": 0,
    +         "needReadable": true,
    +         "objectMode": true,
    +         "paused": false,
    +         "pipes": Console {
    +           "__winstonerror": [Function bound transportEvent],
    +           "__winstonwarn": [Function bound transportEvent],
    +           "_events": Object {
    +             "close": [Function bound onceWrapper],
    +             "drain": [Function pipeOnDrainFunctionResult],
    +             "error": Array [
    +               [Function onerror],
    +               [Function bound transportEvent],
    +             ],
    +             "finish": [Function bound onceWrapper],
    +             "unpipe": Array [
    +               [Function bound onceWrapper],
    +               [Function onunpipe],
    +             ],
    +             "warn": [Function bound transportEvent],
    +           },
    +           "_eventsCount": 6,
    +           "_maxListeners": 30,
    +           "_writableState": WritableState {
    +             "autoDestroy": false,
    +             "bufferProcessing": false,
    +             "bufferedRequest": null,
    +             "bufferedRequestCount": 0,
    +             "corked": 0,
    +             "corkedRequestsFree": CorkedRequest {
    +               "entry": null,
    +               "finish": [Function anonymous],
    +               "next": null,
    +             },
    +             "decodeStrings": true,
    +             "defaultEncoding": "utf8",
    +             "destroyed": false,
    +             "emitClose": true,
    +             "ended": false,
    +             "ending": false,
    +             "errorEmitted": false,
    +             "finalCalled": false,
    +             "finished": false,
    +             "highWaterMark": 16,
    +             "lastBufferedRequest": null,
    +             "length": 0,
    +             "needDrain": false,
    +             "objectMode": true,
    +             "onwrite": [Function anonymous],
    +             "pendingcb": 0,
    +             "prefinished": false,
    +             "sync": true,
    +             "writecb": null,
    +             "writelen": 0,
    +             "writing": false,
    +           },
    +           "consoleWarnLevels": Object {},
    +           "eol": "
    + ",
    +           "format": Printf {
    +             "template": [Function anonymous],
    +           },
    +           "handleExceptions": undefined,
    +           "handleRejections": undefined,
    +           "level": undefined,
    +           "levels": Object {
    +             "debug": 5,
    +             "error": 0,
    +             "http": 3,
    +             "info": 2,
    +             "silly": 6,
    +             "verbose": 4,
    +             "warn": 1,
    +           },
    +           "name": "console",
    +           "parent": [Circular],
    +           "silent": undefined,
    +           "stderrLevels": Object {},
    +           "writable": true,
    +           Symbol(kCapture): false,
    +         },
    +         "pipesCount": 1,
    +         "readableListening": false,
    +         "reading": false,
    +         "readingMore": false,
    +         "resumeScheduled": true,
    +         "sync": false,
    +       },
    +       "_transformState": Object {
    +         "afterTransform": [Function bound afterTransform],
    +         "needTransform": false,
    +         "transforming": false,
    +         "writecb": null,
    +         "writechunk": null,
    +         "writeencoding": null,
    +       },
    +       "_writableState": WritableState {
    +         "autoDestroy": false,
    +         "bufferProcessing": false,
    +         "bufferedRequest": null,
    +         "bufferedRequestCount": 0,
    +         "corked": 0,
    +         "corkedRequestsFree": CorkedRequest {
    +           "entry": null,
    +           "finish": [Function anonymous],
    +           "next": null,
    +         },
    +         "decodeStrings": true,
    +         "defaultEncoding": "utf8",
    +         "destroyed": false,
    +         "emitClose": true,
    +         "ended": false,
    +         "ending": false,
    +         "errorEmitted": false,
    +         "finalCalled": false,
    +         "finished": false,
    +         "highWaterMark": 16,
    +         "lastBufferedRequest": null,
    +         "length": 0,
    +         "needDrain": false,
    +         "objectMode": true,
    +         "onwrite": [Function anonymous],
    +         "pendingcb": 0,
    +         "prefinished": false,
    +         "sync": true,
    +         "writecb": null,
    +         "writelen": 0,
    +         "writing": false,
    +       },
    +       "allowHalfOpen": true,
    +       "defaultMeta": null,
    +       "exceptions": ExceptionHandler {
    +         "handlers": Map {},
    +         "logger": [Circular],
    +       },
    +       "exitOnError": true,
    +       "format": Format {
    +         "options": Object {},
    +       },
    +       "level": "silent",
    +       "levels": Object {
    +         "debug": 5,
    +         "error": 0,
    +         "http": 3,
    +         "info": 2,
    +         "silly": 6,
    +         "verbose": 4,
    +         "warn": 1,
    +       },
    +       "profilers": Object {},
    +       "readable": true,
    +       "rejections": RejectionHandler {
    +         "handlers": Map {},
    +         "logger": [Circular],
    +       },
    +       "silent": true,
    +       "writable": true,
    +       Symbol(kCapture): false,
    +     },
    +   },
        "productIdentifier": "n8n-1.3.0",
    -   "saveCertStr": Any<Function>,
    +   "saveCertStr": [Function saveCertStr],
        "server": "https://server.com/v1",
    -   "tenantId": 1,
    +   "tenantId": 1001,
      },

    Number of calls: 1

      29 |
      30 | 	test('initializes license manager', async () => {
    > 31 | 		expect(LicenseManager).toHaveBeenCalledWith({
         | 		                       ^
      32 | 			autoRenewEnabled: true,
      33 | 			autoRenewOffset: MOCK_RENEW_OFFSET,
      34 | 			deviceFingerprint: expect.any(Function),

      at Object.<anonymous> (test/unit/License.test.ts:31:26)


 RUNS  test/integration/license.api.test.ts

Test Suites: 1 failed, 1 of 2 total
Tests:       1 failed, 9 passed, 10 total
Snapshots:   0 total
Time:        2 s, estimated 4 s
████████████████████████████████████████^C ELIFECYCLE  Command failed.
 ELIFECYCLE  Test failed. See above for more details.
```

</details>
 2023-08-21 10:29:21 +02:00
कारतोफ्फेलस्क्रिप्ट™ a82107fb05
fix(Respond to Webhook Node): Return headers in response (#6921) 2023-08-14 12:38:17 +02:00
Michael Kret 7f0db60f15
refactor(core): Abstract away duplication when finding first pinned trigger (no-changelog) (#5927) 2023-08-10 15:06:16 +03:00
Iván Ovejero dceff675ec
perf(core): Add filtering and pagination to GET /workflows (#6845) 
* Initial setup

* Specify max paginated items

* Simplify

* Add tests

* Add more tests

* Add migrations

* Add top-level property

* Add field selection

* Cleanup

* Rename `total` to `count`

* More cleanup

* Move query logic into `WorkflowRepository`

* Create `AbstractRepository`

* Cleanup

* Fix name

* Remove leftover comments

* Replace reference

* Add default for `rawSkip`

* Remove unneeded typing

* Switch to `class-validator`

* Simplify

* Simplify

* Type as optional

* Make typing more accurate

* Fix lint

* Use `getOwnPropertyNames`

* Use DSL

* Set schema at repo level

* Cleanup

* Remove comment

* Refactor repository methods to middleware

* Add middleware tests

* Remove old test files

* Remove generic experiment

* Reuse `reportError`

* Remove unused type

* Cleanup

* Improve wording

* Reduce diff

* Add missing mw

* Use `Container.get`

* Adjust lint rule

* Reorganize into subdir

* Remove unused directive

* Remove nodes

* Silly mistake

* Validate take

* refactor(core): Adjust index handling in new migrations DSL (no-changelog) (#6876)

* refactor(core): Adjust index handling in new migrations DSL (no-changelog)

* Account for custom index name

* Also for dropping

* Fix `select` issue with `relations`

* Tighten validation

* Ensure `ownerId` is not added when specifying `select`
 2023-08-09 12:30:02 +02:00
Michael Auerswald adcf5a96e8
feat(core): Add metrics option to cache (#6846) 
* add metrics to cache

* use events for metrics

* pr comments / broken test

* lint fix

* update the test

* improve tests

* Update packages/cli/src/config/schema.ts

* disable flaky test

* lint fix

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <netroy@users.noreply.github.com>
 2023-08-04 20:51:07 +02:00
Michael Auerswald fdfc6c5a92
fix(core): Handle empty keys in cache service (no-changelog) (#6854) 
* fix handle empty keys in cache service

* add test

* add cache mock test

* add simpler mocking, and add tests for all the updated methods

* don't use RedisStore specifically in the mock

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
 2023-08-04 19:44:41 +02:00
Iván Ovejero 0511458d41
perf(core): Cache webhooks (#6825) 
* refactor: Initial setup

* Refactor for clarity

* Comments to clarify

* More replacements

* Simplify with `fullPath`

* Fix tests

* Implement remaining methods

* chore: Fix misresolved conflicts

* Simplify syntax

* Reduce diff

* Minor cleanup

* Fix lint

* Inject dependency

* Improve typings

* Remove unused method

* Restore method

* Add comment

* Rename in test

* Restore comments

* Clean up dynamic webhook handling

* Clean up tests

* Remove redundant `cache` prefix

* fix: Correct `uniquePath` for dynamic webhooks
 2023-08-04 11:52:45 +02:00
Iván Ovejero e4f041815a
perf(core): Cache roles (#6803) 
* refactor: Create `RoleService`

* refactor: Refactor to use service

* refactor: Move `getUserRoleForWorkflow`

* refactor: Clear out old `RoleService`

* refactor: Consolidate utils into service

* refactor: Remove unused methods

* test: Add tests

* refactor: Remove redundant return types

* refactor: Missing utility

* chore: Remove commented out bit

* refactor: Make `Db.collections.Repository` inaccessible

* chore: Cleanup

* feat: Prepopulate cache

* chore: Remove logging

* fix: Account for tests where roles are undefined

* fix: Restore `prettier.prettierPath`

* test: Account for cache enabled and disabled

* fix: Restore `Role` in `Db.collections`

* refactor: Simplify by removing `orFail`

* refactor: Rename for clarity

* refactor: Use `cacheKey` for readability

* refactor: Validate role before creation

* refacator: Remove redundant `cache` prefix

* ci: Lint fix

* test: Fix e2e
 2023-08-03 08:58:36 +02:00
Michael Auerswald 659ca26fe7
fix(core): Change VariablesService to DI and use caching (#6827) 
* support redis cluster

* cleanup, fix config schema

* set default prefix to bull

* initial commit

* improve logging

* improve types and refactor

* list support and refactor

* fix redis service and tests

* add comment

* add redis and cache prefix

* use injection

* lint fix

* clean schema comments

* improve naming, tests, cluster client

* merge master

* cache returns unknown instead of T

* update cache service, tests and doc

* remove console.log

* VariablesService as DI, add caching, fix tests

* do not cache null or undefined values

* import fix

* more DI and remove collections

* fix merge

* lint fix

* rename to ~Cached

* fix test for CI

* fix ActiveWorkflowRunner test
 2023-08-02 14:51:09 +02:00
Michael Auerswald 3cad60e918
feat(core): Make Redis available for backend communication (#6719) 
* support redis cluster

* cleanup, fix config schema

* set default prefix to bull

* initial commit

* improve logging

* improve types and refactor

* list support and refactor

* fix redis service and tests

* add comment

* add redis and cache prefix

* use injection

* lint fix

* clean schema comments

* improve naming, tests, cluster client

* merge master

* cache returns unknown instead of T

* update cache service, tests and doc

* remove console.log

* do not cache null or undefined values

* fix merge

* lint fix
 2023-08-02 12:51:25 +02:00
कारतोफ्फेलस्क्रिप्ट™ 31d8f478ee
refactor(core): Parse Webhook request bodies on-demand (#6394) 
Also,
1. Consistent CORS support ~on all three webhook types~ waiting webhooks never supported CORS. I'll fix that in another PR
2. [Fixes binary-data handling when request body is text, json, or xml](https://linear.app/n8n/issue/NODE-505/webhook-binary-data-handling-fails-for-textplain-files).
3. Reduced number of middleware that each request has to go through.
4. Removed the need to maintain webhook endpoints in the auth-exception list.
5. Skip all middlewares (apart from `compression`) on Webhook routes. 
6. move `multipart/form-data` support out of individual nodes
7. upgrade `formidable`
8. fix the filenames on binary-data in webhooks nodes
9. add unit tests and integration tests for webhook request handling, and increase test coverage
 2023-08-01 17:32:30 +02:00
Iván Ovejero ffae8edce3
refactor(core): Cache workflow ownership (#6738) 
* refactor: Set up ownership service

* refactor: Specify cache keys and values

* refactor: Replace util with service calls

* test: Mock service in tests

* refactor: Use dependency injection

* test: Write tests

* refactor: Apply feedback from Omar and Micha

* test: Fix tests

* test: Fix missing spot

* refactor: Return user entity from cache

* refactor: More dependency injection!
 2023-07-31 11:37:09 +02:00
Michael Auerswald fc7aa8bd66
feat: Environments release using source control (#6653) 
* initial telemetry setup and adjusted pull return

* quicksave before merge

* feat: add conflicting workflow list to pull modal

* feat: update source control pull modal

* fix: fix linting issue

* feat: add Enter keydown event for submitting source control push modal (no-changelog)

feat: add Enter keydown event for submitting source control push modal

* quicksave

* user workflow table for export

* improve telemetry data

* pull api telemetry

* fix lint

* Copy tweaks.

* remove authorName and authorEmail and pick from user

* rename owners.json to workflow_owners.json

* ignore credential conflicts on pull

* feat: several push/pull flow changes and design update

* pull and push return same data format

* fix: add One last step toast for successful pull

* feat: add up to date pull toast

* fix: add proper Learn more link for push and pull modals

* do not await tracking being sent

* fix import

* fix await

* add more sourcecontrolfile status

* Minor copy tweak for "More info".

* Minor copy tweak for "More info".

* ignore variable_stub conflicts on pull

* ignore whitespace differences

* do not show remote workflows that are not yet created

* fix telemetry

* fix toast when pulling deleted wf

* lint fix

* refactor and make some imports dynamic

* fix variable edit validation

* fix telemetry response

* improve telemetry

* fix unintenional delete commit

* fix status unknown issue

* fix up to date toast

* do not export active state and reapply versionid

* use update instead of upsert

* fix: show all workflows when clicking push to git

* feat: update Up to date pull translation

* fix: update read only env checks

* do not update versionid of only active flag changes

* feat: prevent access to new workflow and templates import when read only env

* feat: send only active state and version if workflow state is not dirty

* fix: Detect when only active state has changed and prevent generation a new version ID

* feat: improve readonly env messages

* make getPreferences public

* fix telemetry issue

* fix: add partial workflow update based on dirty state when changing active state

* update unit tests

* fix: remove unsaved changes check in readOnlyEnv

* fix: disable push to git button when read onyl env

* fix: update readonly toast duration

* fix: fix pinning and title input in protected mode

* initial commit (NOT working)

* working push

* cleanup and implement pull

* fix getstatus

* update import to new method

* var and tag diffs are no conflicts

* only show pull conflict for workflows

* refactor and ignore faulty credentials

* add sanitycheck for missing git folder

* prefer fetch over pull and limit depth to 1

* back to pull...

* fix setting branch on initial connect

* fix test

* remove clean workfolder

* refactor: Remove some unnecessary code

* Fixed links to docs.

* fix getstatus query params

* lint fix

* dialog to show local and remote name on conflict

* only show remote name on conflict

* fix credential expression export

* fix: Broken test

* dont show toast on pull with empty var/tags and refactor

* apply frontend changes from old branch

* fix tag with same name import

* fix buttons shown for non instance owners

* prepare local storage key for removal

* refactor: Change wording on pushing and pulling

* refactor: Change menu item

* test: Fix broken test

* Update packages/cli/src/environments/sourceControl/types/sourceControlPushWorkFolder.ts

Co-authored-by: Iván Ovejero <ivov.src@gmail.com>

---------

Co-authored-by: Alex Grozav <alex@grozav.com>
Co-authored-by: Giulio Andreini <g.andreini@gmail.com>
Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
 2023-07-26 09:25:01 +02:00
Iván Ovejero bcfc5e717b
refactor(core): Move webhook DB access to repository (no-changelog) (#6706) 
* refactor(core): Move webhook DB access to repository (no-changelog)

* make sure `DataSource` is initialized before it's dependencies

at some point I hope to replace `DataSource` with a custom `DatabaseConnection` service class that can then disconnect and reconnect from DB without having to update all repositories.

---------

Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
 2023-07-25 18:17:34 +02:00
Michael Auerswald c0d2bac94d
feat(core): Add cache service (#6729) 
* add cache service

* PR adjustments

* switch to maxSize for memory cache
 2023-07-25 11:56:38 +02:00
Ricardo Espinoza 89f44021b9
fix(core): Use JWT as reset password token (#6714) 
* use jwt to reset password

* increase expiration time to 1d

* drop user id query string

* refactor

* use service instead of package in tests

* sqlite migration

* postgres migration

* mysql migration

* remove unused properties

* remove userId from FE

* fix test for users.api

* move migration to the common folder

* move type assertion to the jwt.service

* Add jwt secret as a readonly property

* use signData instead of sign in user.controller

* remove base class

* remove base class

* add tests
 2023-07-24 17:40:17 -04:00
Iván Ovejero 667c15d0df
fix(core): Filter out workflows that failed to activate on startup (#6676) 
* fix(core): Deactivate on init workflow that should not be retried

* fix(core): Filter out workflows with activation errors
 2023-07-18 15:57:14 +02:00
कारतोफ्फेलस्क्रिप्ट™ f4a18ba87d
refactor(core): Refactor WorkflowStatistics code (no-changelog) (#6617) 
refactor(core): Refactor WorkflowStatistics code
 2023-07-18 11:28:24 +02:00
कारतोफ्फेलस्क्रिप्ट™ b895ba438a
refactor(core): Reduce boilterplate code in between tests 🧹, and fix the tests in node.js 20 (no-changelog) (#6654) 
refactor(core): Reduce boilterplate code in between tests

also cleaned up some imports, and fixed the tests in node.js 20
 2023-07-13 10:14:48 +02:00
OlegIvaniv e5620ab1e4
feat(API): Implement users account quota guards (#6434) 
* feat(cli): Implement users account quota guards

Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>

* Remove comment

Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>

* Address PR comments

- Getting `usersQuota` from `Settings` repo
- Revert `isUserManagementEnabled` helper
- Fix FE listing of users

Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>

* Refactor isWithinUserQuota getter and fix tests

Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>

* Revert testDb.ts changes

Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>

* Cleanup & improve types

Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>

* Fix duplicated method

* Fix failing test

* Remove `isUserManagementEnabled` completely

Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>

* Check for globalRole.name to determine if user is owner

Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>

* Fix unit tests

Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>

* Set isInstanceOwnerSetUp in specs

* Fix SettingsUserView UM

Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>

* refactor: License typings suggestions for users quota guards (#6636)

refactor: License typings suggestions

* Update packages/cli/src/Ldap/helpers.ts

Co-authored-by: Iván Ovejero <ivov.src@gmail.com>

* Update packages/cli/test/integration/shared/utils.ts

Co-authored-by: Iván Ovejero <ivov.src@gmail.com>

* Address PR comments

Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>

* Use 403 for all user quota related errors

Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>

---------

Signed-off-by: Oleg Ivaniv <me@olegivaniv.com>
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
 2023-07-12 14:11:46 +02:00
कारतोफ्फेलस्क्रिप्ट™ abe7f71627 refactor: Remove pre-setup prompt on owner setup (#6495) 2023-06-22 21:11:38 +02:00
Iván Ovejero 8c008f5d22 refactor(core)!: Remove basic-auth, external-jwt-auth, and no-auth options (#6362) 
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
 2023-06-22 20:03:47 +02:00
Michael Auerswald c3ba0123ad
feat: Migrate integer primary keys to nanoids (#6345) 
* first commit for postgres migration

* (not working)

* sqlite migration

* quicksave

* fix tests

* fix pg test

* fix postgres

* fix variables import

* fix execution saving

* add user settings fix

* change migration to single lines

* patch preferences endpoint

* cleanup

* improve variable import

* cleanup unusued code

* Update packages/cli/src/PublicApi/v1/handlers/workflows/workflows.handler.ts

Co-authored-by: Omar Ajoue <krynble@gmail.com>

* address review notes

* fix var update/import

* refactor: Separate execution data to its own table (#6323)

* wip: Temporary migration process

* refactor: Create boilerplate repository methods for executions

* fix: Lint issues

* refactor: Added search endpoint to repository

* refactor: Make the execution list work again

* wip: Updating how we create and update executions everywhere

* fix: Lint issues and remove most of the direct access to execution model

* refactor: Remove includeWorkflowData flag and fix more tests

* fix: Lint issues

* fix: Fixed ordering of executions for FE, removed transaction when saving execution and removed unnecessary update

* refactor: Add comment about missing feature

* refactor: Refactor counting executions

* refactor: Add migration for other dbms and fix issues found

* refactor: Fix lint issues

* refactor: Remove unnecessary comment and auto inject repo to internal hooks

* refactor: remove type assertion

* fix: Fix broken tests

* fix: Remove unnecessary import

* Remove unnecessary toString() call

Co-authored-by: Iván Ovejero <ivov.src@gmail.com>

* fix: Address comments after review

* refactor: Remove unused import

* fix: Lint issues

* fix: Add correct migration files

---------

Co-authored-by: Iván Ovejero <ivov.src@gmail.com>

* remove null values from credential export

* fix: Fix an issue with queue mode where all running execution would be returned

* fix: Update n8n node to allow for workflow ids with letters

* set upstream on set branch

* remove typo

* add nodeAccess to credentials

* fix unsaved run check for undefined id

* fix(core): Rename version control feature to source control (#6480)

* rename versionControl to sourceControl

* fix source control tooltip wording

---------

Co-authored-by: Romain Minaud <romain.minaud@gmail.com>

* fix(editor): Pay 548 hide the set up version control button (#6485)

* feat(DebugHelper Node): Fix and include in main app (#6406)

* improve node a bit

* fixing continueOnFail() ton contain error in json

* improve pairedItem

* fix random data returning object results

* fix nanoId length typo

* update pnpm-lock file

---------

Co-authored-by: Marcus <marcus@n8n.io>

* fix(editor): Remove setup source control CTA button

* fix(editor): Remove setup source control CTA button

---------

Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>

* fix(editor): Update source control docs links (#6488)

* feat(DebugHelper Node): Fix and include in main app (#6406)

* improve node a bit

* fixing continueOnFail() ton contain error in json

* improve pairedItem

* fix random data returning object results

* fix nanoId length typo

* update pnpm-lock file

---------

Co-authored-by: Marcus <marcus@n8n.io>

* feat(editor): Replace root events with event bus events (no-changelog) (#6454)

* feat: replace root events with event bus events

* fix: prevent cypress from replacing global with globalThis in import path

* feat: remove emitter mixin

* fix: replace component events with event bus

* fix: fix linting issue

* fix: fix breaking expression switch

* chore: prettify ndv e2e suite code

* fix(editor): Update source control docs links

---------

Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>

* fix tag endpoint regex

---------

Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
Co-authored-by: Csaba Tuncsik <csaba@n8n.io>
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
 2023-06-20 19:13:18 +02:00
Csaba Tuncsik 1b321416c0
feat: Version control mvp (#6271) 
* implement basic git service

* cleanup connected prop

* add skeleton of git functions

* initial import/export setup

* split out export service

* refactor and improve export

* begin import

* more commands and basic import

* clean up imports with transactions

* work folder import functions

* reintroduce versionid

* add missing import to pull workfolder

* add get-status endpoint

* add cleanup to disconnect

* add initRepo options

* add more checks and cleanup

* minor cleanup

* refactor prefs

* fix server.ts

* fix sending deleted files

* rename files to ee

* add variable override and fix critical cred import bug

* fix mkdir race condition

* make initRepo default to true

* fix front back integration

* improve connect flow

* add comment to generated ssh key

* fix(editor): use useToast composable

* fix buttons size

* commenting out repo init for now

* fix(editor): update UI logic

* fix(editor): remove console.log

* fix(editor): remove unused ref

* adjust endpoints for improved UI

* fix(editor): add push and pull buttons

* keep or not ssh key

* switching file name to id

* fix(editor): add success messages, fix save button

* fixed faulty diff preventing pull

* fix build

* fix(editor): adding loader to VC components

* removing duplicate exports

* improve conflict finding on push pull

* manage pull conflict

* alternate push pull

* fix pull confirmation

* fix rm and credential export/import

* switch to alternative pull implementation

* fix initial commit

* fix(editor): subscribing to VC store action to refresh lists

* fix(editor): wrap VC store actions with try

* feat: add fine-grained file selection for push action

* fix: close modal after successful push

* fix(editor): VC preferences validation

* fix confirm

* fix: update endpoint to /get-status

* feat: update pull modal override changes message

* fix missing wf error

* undo

* removing connect endpoint

* fix(editor): add button titles

* fix(editor): cleaning up store action

* add version-control/set-read-only protection

* fix(editor): adding set branch readonly

* fix(editor): remove Push button if branch set to readonly

* fix(editor): fix some styles

* fix(editor): remove duplicate and delete actions in WF list when branch is readonly

* fix: load status before opening selective push modal

* fix(editor): extend readonly logic

* add cleanup after failed initRepo

* fix deleted files crashing get-status

* fix n8n-checkbox in staging dialog

* fix(editor): fix loading

* fix(editor): resize buttons

* fix(editor): fix translation

* fix(editor): fix copy text size

* fix(editor): fix copy text size

* fix(editor): add disconnection confirmation

* fix(editor): add disconnection confirmation

* fix(editor): set large buttons

* add public api Pull endpoint

* feat: add refresh ssh key

* return prefs when new keys are generated

* fix(editor): adding readOnly mode to main header

* fix(editor): adding readOnly mode to workflow settings

* improve credential owner import

* add middleware to endpoints

* improve public api error/doc

* do not create branch if one already exists

* update wordings for connect toasts

* fix(editor): updating and separating readonly modes

* fix(editor): fix readonly mode in WF list

* fix(editor): disable elements dragging on canvas in readonly mode (WIP: not working when NodeView page is loaded first)

* fix(editor): fix canvas draggables in readonly env

* fix(editor): remove unused variables

* fix(editor): hide actions in node connections when readonly

* fix(editor): hide actions in node connections when readonly

* fix(editor): disable Save button when readonly

* fix(editor): disable Save settings if no branch is selected

* fix(editor): lint fix

* fix(editor): update snapshots

* fix(editor): replace Loading... text

* fix(editor): reset Loading... text

* reset branchname on disconnect

* fix(editor): adding some translations

* fix(editor): fix unit test

* fix(editor): fix loading

* fix(editor): set settings saved message

* fix(editor): update connection flag

* fix branchName not returning after connect

* temporary (but still breaking) fix for postgres

* fix(editor): adding tooltip to Push/Pull buttons when they're collapsed

* fix(editor): enabled activator in readonly mode

* fix test

* fix(editor): disabling new item addition for workflows in readonly mode

* fix(editor): modify Pull/Push button tooltips

* do not commit empty variables file

---------

Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
Co-authored-by: Alex Grozav <alex@grozav.com>
 2023-05-31 15:01:57 +02:00
Cornelius Suermann 0e4c615d0d
fix(core): Do not track errored workflow executions for automated executions (no-changelog) (#6322) 
* fix(core): Do not track errored workflow executions for automated executions

* fix test

* fix test

* fix test

* do not track 'Workflow execution count' event when all counts are 0

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test
 2023-05-28 12:08:46 +02:00