diff --git a/.ai/lessons.md b/.ai/lessons.md index ea671e124f..71bb8c263d 100644 --- a/.ai/lessons.md +++ b/.ai/lessons.md @@ -7,16 +7,20 @@ - Test serialized worker tasks with the real process driver. An in-process test does not prove that captured models or nested closures survive serialization. - Use the same regression test before and after the production change so the result shows the behavior difference. - Several dev instances can run from one checkout (see `./scripts/dev urls`). After you add a migration, run it on each running instance of that checkout, or those instances fail on the new code. -- The checkout is shared with other sessions. When the full suite fails, rerun each failing file alone and compare with a clean `git archive HEAD` copy before you connect a failure to your change. +- The checkout is shared with other sessions. When the full suite fails, rerun each failing file alone and compare with a clean `git archive HEAD` copy before you connect a failure to your change. The dev instance of the checkout runs its live files, so a live repro also runs the uncommitted edits of other sessions: run `git diff` on the code under test before you trust the result. - Run seeders and other artisan commands that write SSH keys in a dev container as `www-data` (`docker exec -u www-data`). Root-owned files in `storage/app/ssh` make every SSH call from the web process fail. +- Files written on a server with `tee` get the server umask (027/077 on hardened hosts). Set the mode explicitly (`chmod 644`) for files that a non-root container user reads through a read-only bind mount. - `sshd` keeps the login shell of an open multiplexed SSH connection. Call `SshMultiplexingHelper::removeMuxFile()` before a live test of SSH or login-shell behavior. - Remote command batches run in `bash -se` (`sh -se` only when the host has no bash). A command that can fail without being an error (such as `cat` of an optional file) stops the batch unless it ends with `|| true`. `Process::fake()` cannot show this, so run such a command once in a real `bash -se` on the target host. - SSH retries replay the full command batch. Only exit code 255 is an SSH transport failure; other exit codes come from the remote command, so do not retry them. Keep batches safe to run twice (for example `docker rm -f || true` before `docker run --name `). +- Do not put `pkill -f "playwright run-server"` in the same shell command as other work; the pattern matches that shell and kills it. Run it as its own command. - Redirect browser test output to a file (`> /tmp/x.log 2>&1`); piping it (`| tail`) hangs because the Playwright server keeps the pipe open. - Browser `click()`/`assertSee()` wait forever when nothing matches, so a bad selector hangs the run instead of failing it. `click('Text')` can match a hidden element first (use `button:visible:has-text("Text")`); `:text-is()` on an element with child elements and `assertSee()` text with double quotes never match. Prefer plain CSS selectors. - A new worktree has no `vendor/` or `public/build/`. Run `composer install --ignore-platform-req=ext-zip` on the host and copy `public/build` from the main checkout before page-rendering tests; otherwise they fail with a Vite manifest error. - Never run `php artisan test` inside a dev container (`./scripts/dev exec` or `docker exec coolify-dev-*`). The container sets `DB_CONNECTION=pgsql` as a real environment variable, which wins over `phpunit.xml`, so `RefreshDatabase` runs `migrate:fresh` on the dev database and wipes it. Run tests on the host only. +- The helper image runs `docker compose` for deployments and for the Coolify upgrade itself. Before you bump its Docker CLI or Compose, test them against the oldest supported Docker Engine (for example a `docker:-dind` container); newer clients can reject data from older daemons. - Do not symlink `vendor` into another worktree. Composer resolves the base path from the real `vendor` path, so the tests load the app classes of the other checkout. Run `composer install --ignore-platform-req=ext-zip` in the worktree. +- To assert that the unsaved-changes bar shows, check `el.classList.contains('is-dirty')`. Playwright treats the hidden bar (`opacity-0`) as visible, and its class attribute always contains the text `is-dirty` from Tailwind variants. - Do not call `sleep()` in a browser test. The app server runs in the test process, so `sleep()` stops it; use `$page->wait()`. - In tests, turning on `is_metrics_enabled` (and other Sentinel settings) with `save()`/`update()` runs the settings hook, which starts Sentinel over a real SSH connection on the sync queue. Use `saveQuietly()` or `Queue::fake()`. - Call `visit()` directly in each `tests/v4/Browser` test body; Pest does not mark a test that only uses helper-wrapped `visit()` as a browser test, so it fails with `sendText() on null`. @@ -118,3 +122,9 @@ ## Cached and stored state - `StandaloneDocker`/`SwarmDocker` `server` uses `Server::findCached()`, a per-process identity map that is flushed only when the `servers` row updates. A `ServerSetting` save does not flush it, so pass the current `Server` instance (`setRelation('server', $server)`) when generated output depends on just-saved settings. - Application deployments use the stored `custom_labels` when they exist. When a server-level setting changes the generated labels, regenerate the stored labels of read-only (`is_container_label_readonly_enabled`) applications, or a redeploy keeps the old labels. + +## Keep browser form restore away from app state +- Firefox session restore writes saved input values back and fires a trusted `input` event. Add `autocomplete="off"` to inputs that only mirror client state (for example an Alpine-bound color picker), or their handlers run without a user action. + +## Session writes are last-writer-wins +- Laravel saves the whole session payload at the end of every request, and pages run concurrent `wire:poll` requests. A value that one request changes can be overwritten by a request that started earlier. `KeepConcurrentTeamSwitch` protects `currentTeam`; protect any new session value that changes mid-session the same way, or keep it out of the session. diff --git a/.gitignore b/.gitignore index 3b583795bf..7f342543fe 100644 --- a/.gitignore +++ b/.gitignore @@ -54,3 +54,6 @@ ref # Multi-instance local Coolify env files (scripts/dev-instances) .dev-instances/ + +# Service parser template snapshots: generated locally by scripts/parser-snapshot-diff, not committed +/tests/Fixtures/parser/expected/templates/ diff --git a/AGENTS.md b/AGENTS.md index d87aeea253..3016656c89 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -66,6 +66,22 @@ npm run dev # vite dev server npm run build # production build ``` +## Testing the Service Parser + +`tests/Feature/ServiceParserReferenceTest.php` is a golden test for both compose parsers (`serviceParser` with `updateCompose`, and `applicationParser`) and the proxy label and magic-variable helpers. It parses `tests/Fixtures/parser/service-reference.yaml` and `application-reference.yaml` (one service per parser feature, with setting variations and error fixtures) and every template in `templates/compose`, then compares the output with `tests/Fixtures/parser/expected/`. Only the reference expected files are committed. Template expected files are gitignored, so CI checks only that each template parses. + +```bash +./scripts/parser-snapshot-diff save # before a parser change: save the output of all cases (~90 s) +./scripts/parser-snapshot-diff baseline # after the change: one line per changed case, full diff in storage/framework/testing/parser-snapshot-diff.diff +php artisan test --compact tests/Feature/ServiceParserReferenceTest.php # before you commit +UPDATE_PARSER_SNAPSHOTS=1 php artisan test --compact tests/Feature/ServiceParserReferenceTest.php # accept an intended change, then review git diff tests/Fixtures/parser/ +``` + +- A refactor must give "No differences". For an intended change, check every changed line in the diff file. +- When you add a parser feature, add a small service that uses it to `service-reference.yaml` or `application-reference.yaml` and regenerate the expected files. +- `./scripts/parser-snapshot-diff ` compares with a commit instead of a saved baseline. +- It covers only the latest compose parsing version. Older parsing versions are not covered. + ## Browser Tests (Pest Browser Plugin) Uses `pestphp/pest-plugin-browser` (Playwright). Browser tests go in `tests/v4/Browser/`. diff --git a/app/Actions/Database/StartClickhouse.php b/app/Actions/Database/StartClickhouse.php index d053142d2c..6463b70cdf 100644 --- a/app/Actions/Database/StartClickhouse.php +++ b/app/Actions/Database/StartClickhouse.php @@ -192,7 +192,7 @@ class StartClickhouse $environment_variables->push("CLICKHOUSE_DB={$this->database->clickhouse_db}"); } - add_coolify_default_environment_variables($this->database, $environment_variables, $environment_variables); + $this->addCoolifyEnvironmentVariables($environment_variables); return $environment_variables->all(); } diff --git a/app/Actions/Database/StartDragonfly.php b/app/Actions/Database/StartDragonfly.php index 16ad09c95d..72f66a64ce 100644 --- a/app/Actions/Database/StartDragonfly.php +++ b/app/Actions/Database/StartDragonfly.php @@ -269,6 +269,8 @@ class StartDragonfly $environment_variables->push("REDIS_PASSWORD={$this->database->dragonfly_password}"); } + $this->addCoolifyEnvironmentVariables($environment_variables); + return $environment_variables->all(); } diff --git a/app/Actions/Database/StartKeydb.php b/app/Actions/Database/StartKeydb.php index fa86576d5e..75827b4bac 100644 --- a/app/Actions/Database/StartKeydb.php +++ b/app/Actions/Database/StartKeydb.php @@ -269,7 +269,7 @@ class StartKeydb $environment_variables->push("REDIS_PASSWORD={$this->database->keydb_password}"); } - add_coolify_default_environment_variables($this->database, $environment_variables, $environment_variables); + $this->addCoolifyEnvironmentVariables($environment_variables); return $environment_variables->all(); } diff --git a/app/Actions/Database/StartMariadb.php b/app/Actions/Database/StartMariadb.php index bf5739a86b..5440ed8611 100644 --- a/app/Actions/Database/StartMariadb.php +++ b/app/Actions/Database/StartMariadb.php @@ -268,7 +268,7 @@ class StartMariadb $environment_variables->push("MARIADB_PASSWORD={$this->database->mariadb_password}"); } - add_coolify_default_environment_variables($this->database, $environment_variables, $environment_variables); + $this->addCoolifyEnvironmentVariables($environment_variables); return $environment_variables->all(); } @@ -282,5 +282,7 @@ class StartMariadb $content = $this->database->mariadb_conf; $content_base64 = base64_encode($content); $this->commands[] = "echo '{$content_base64}' | base64 -d | tee $this->configuration_dir/{$filename} > /dev/null"; + // The container user must read the read-only mount; a strict server umask leaves the file unreadable. + $this->commands[] = "chmod 644 $this->configuration_dir/{$filename}"; } } diff --git a/app/Actions/Database/StartMongodb.php b/app/Actions/Database/StartMongodb.php index 0afb0c4a6f..49cee8ea3f 100644 --- a/app/Actions/Database/StartMongodb.php +++ b/app/Actions/Database/StartMongodb.php @@ -325,7 +325,7 @@ class StartMongodb $environment_variables->push("MONGO_INITDB_DATABASE={$this->database->mongo_initdb_database}"); } - add_coolify_default_environment_variables($this->database, $environment_variables, $environment_variables); + $this->addCoolifyEnvironmentVariables($environment_variables); return $environment_variables->all(); } @@ -372,6 +372,8 @@ class StartMongodb $content = $this->database->mongo_conf; $content_base64 = base64_encode($content); $this->commands[] = "echo '{$content_base64}' | base64 -d | tee $this->configuration_dir/{$filename} > /dev/null"; + // The container user must read the read-only mount; a strict server umask leaves the file unreadable. + $this->commands[] = "chmod 644 $this->configuration_dir/{$filename}"; } private function add_default_database() diff --git a/app/Actions/Database/StartMysql.php b/app/Actions/Database/StartMysql.php index ea723aada7..6fb288b7b0 100644 --- a/app/Actions/Database/StartMysql.php +++ b/app/Actions/Database/StartMysql.php @@ -280,7 +280,7 @@ class StartMysql $environment_variables->push("MYSQL_PASSWORD={$this->database->mysql_password}"); } - add_coolify_default_environment_variables($this->database, $environment_variables, $environment_variables); + $this->addCoolifyEnvironmentVariables($environment_variables); return $environment_variables->all(); } @@ -294,5 +294,7 @@ class StartMysql $content = $this->database->mysql_conf; $content_base64 = base64_encode($content); $this->commands[] = "echo '{$content_base64}' | base64 -d | tee $this->configuration_dir/{$filename} > /dev/null"; + // The container user must read the read-only mount; a strict server umask leaves the file unreadable. + $this->commands[] = "chmod 644 $this->configuration_dir/{$filename}"; } } diff --git a/app/Actions/Database/StartPostgresql.php b/app/Actions/Database/StartPostgresql.php index c0949e51b8..05f9be02f9 100644 --- a/app/Actions/Database/StartPostgresql.php +++ b/app/Actions/Database/StartPostgresql.php @@ -291,7 +291,7 @@ class StartPostgresql $environment_variables->push("POSTGRES_DB={$this->database->postgres_db}"); } - add_coolify_default_environment_variables($this->database, $environment_variables, $environment_variables); + $this->addCoolifyEnvironmentVariables($environment_variables); return $environment_variables->all(); } @@ -342,5 +342,7 @@ class StartPostgresql } $content_base64 = base64_encode($content); $this->commands[] = "echo '{$content_base64}' | base64 -d | tee $config_file_path > /dev/null"; + // The container user must read the read-only mount; a strict server umask leaves the file unreadable. + $this->commands[] = "chmod 644 $config_file_path"; } } diff --git a/app/Actions/Database/StartRedis.php b/app/Actions/Database/StartRedis.php index c12381aca8..3d447d91da 100644 --- a/app/Actions/Database/StartRedis.php +++ b/app/Actions/Database/StartRedis.php @@ -284,7 +284,7 @@ class StartRedis } } - add_coolify_default_environment_variables($this->database, $environment_variables, $environment_variables); + $this->addCoolifyEnvironmentVariables($environment_variables); return $environment_variables->all(); } diff --git a/app/Actions/Database/StartSqlite.php b/app/Actions/Database/StartSqlite.php index 9289cdf5a0..b85d4e73e3 100644 --- a/app/Actions/Database/StartSqlite.php +++ b/app/Actions/Database/StartSqlite.php @@ -153,7 +153,7 @@ class StartSqlite $environment_variables->push("SQLITE_DATABASES={$this->database->sqlite_databases}"); } - add_coolify_default_environment_variables($this->database, $environment_variables, $environment_variables); + $this->addCoolifyEnvironmentVariables($environment_variables); return $environment_variables->all(); } diff --git a/app/Actions/Server/CleanupDocker.php b/app/Actions/Server/CleanupDocker.php index c49c057060..ff4df515f4 100644 --- a/app/Actions/Server/CleanupDocker.php +++ b/app/Actions/Server/CleanupDocker.php @@ -131,10 +131,7 @@ class CleanupDocker implements ShouldBeUnique } } $helperImageVersion = getHelperVersion(); - $helperImage = coolifyHelperImage(); - $helperImageWithVersion = "$helperImage:$helperImageVersion"; - $helperImageWithoutPrefix = 'coollabsio/coolify-helper'; - $helperImageWithoutPrefixVersion = "coollabsio/coolify-helper:$helperImageVersion"; + $helperImageWithVersion = coolifyHelperImage().":$helperImageVersion"; $cleanupLog = []; @@ -163,8 +160,7 @@ class CleanupDocker implements ShouldBeUnique 'docker builder prune -af', // The prune fails on a stopped builder, so start it first. railpackBuilderHelperCommand(railpackBuildxMetadataVolume($server), $helperImageWithVersion, 'docker buildx inspect --bootstrap coolify-railpack >/dev/null 2>&1; docker buildx prune --builder coolify-railpack -af'), - "docker images --filter before=$helperImageWithVersion --filter reference=$helperImage | grep $helperImage | awk '{print $3}' | xargs -r docker rmi -f", - "docker images --filter before=$helperImageWithoutPrefixVersion --filter reference=$helperImageWithoutPrefix | grep $helperImageWithoutPrefix | awk '{print $3}' | xargs -r docker rmi -f", + $this->buildCoolifyImageCleanupCommand($helperImageVersion), ]; if ($deleteUnusedVolumes) { @@ -219,6 +215,22 @@ class CleanupDocker implements ShouldBeUnique "awk '{print $1}' | xargs -r docker rm"; } + /** + * Build a command that removes unused images of the Coolify helper, realtime and sentinel, + * from any registry. They have the coolify.managed=true label, so the generic image prune + * skips them. The current helper version stays, and docker rmi (without -f) keeps images + * that a container uses, for example the running sentinel. + */ + private function buildCoolifyImageCleanupCommand(string $helperImageVersion): string + { + $escapedHelperVersion = preg_replace('/([.\\\\+*?\[\]^$(){}|])/', '\\\\$1', $helperImageVersion); + + return "docker images --format '{{.Repository}}:{{.Tag}}' | ". + "grep -E '(^|/)coollabsio/(coolify-helper|coolify-realtime|sentinel):' | ". + "grep -v -E ':$|(^|/)coollabsio/coolify-helper:{$escapedHelperVersion}$' | ". + 'xargs -r docker rmi 2>/dev/null || true'; + } + /** * Build a docker image prune command that excludes application image repositories. * @@ -254,17 +266,18 @@ class CleanupDocker implements ShouldBeUnique // - Don't have coolify.managed=true label // Images in use by containers will fail silently with docker rmi // Pattern matches both uuid:tag and uuid_servicename:tag (Docker Compose with build) + // Images pulled by digest have no tag and are not dangling, so they are listed as repo@digest $grepCommands = "grep -v ''"; // Add application repo exclusion if there are applications if ($applicationImageRepos->isNotEmpty()) { - $grepCommands .= " | grep -v -E '^({$appExcludePatterns})[_:].+'"; + $grepCommands .= " | grep -v -E '^({$appExcludePatterns})[_:@].+'"; } // Add infrastructure image exclusion (matches any registry prefix) $grepCommands .= " | grep -v -E '{$infraExcludePattern}'"; - $commands[] = "docker images --format '{{.Repository}}:{{.Tag}}' | ". + $commands[] = "docker images --digests --format '{{if eq .Tag \"\"}}{{.Repository}}@{{.Digest}}{{else}}{{.Repository}}:{{.Tag}}{{end}}' | ". $grepCommands.' | '. "xargs -r -I {} sh -c 'docker inspect --format \"{{index .Config.Labels \\\"coolify.managed\\\"}}\" \"{}\" 2>/dev/null | grep -q true || docker rmi \"{}\" 2>/dev/null' || true"; diff --git a/app/Actions/Server/StartSentinel.php b/app/Actions/Server/StartSentinel.php index 3dcc2650ac..b888892cea 100644 --- a/app/Actions/Server/StartSentinel.php +++ b/app/Actions/Server/StartSentinel.php @@ -148,7 +148,9 @@ class StartSentinel ? '-v '.escapeshellarg("{$trafficLogDirectory}:{$trafficLogDirectory}:ro").' ' : ''; $network = $server->isLocalhost() ? ' --network coolify' : ''; - $dockerCommand = "docker run -d$network $dockerEnvironments --name coolify-sentinel --restart unless-stopped -v /var/run/docker.sock:/var/run/docker.sock -v $mountDir:/app/db {$trafficMount}--pid host --health-cmd \"curl --fail http://127.0.0.1:8888/api/health || exit 1\" --health-start-period 120s --health-interval 10s --health-retries 3 --add-host=host.docker.internal:host-gateway --label $dockerLabels $image"; + // A userns-remap daemon refuses --pid host unless the container opts out of remapping. + // On other daemons --userns host has no effect. https://docs.docker.com/engine/security/userns-remap/ + $dockerCommand = "docker run -d$network $dockerEnvironments --name coolify-sentinel --restart unless-stopped -v /var/run/docker.sock:/var/run/docker.sock -v $mountDir:/app/db {$trafficMount}--userns host --pid host --health-cmd \"curl --fail http://127.0.0.1:8888/api/health || exit 1\" --health-start-period 120s --health-interval 10s --health-retries 3 --add-host=host.docker.internal:host-gateway --label $dockerLabels $image"; $server->sentinelHeartbeat(isReset: true); $server->forceFill(['sentinel_waiting_since' => now()])->save(); diff --git a/app/Actions/Service/DeployServiceApplication.php b/app/Actions/Service/DeployServiceApplication.php index 4db4834d4c..2a475cd462 100644 --- a/app/Actions/Service/DeployServiceApplication.php +++ b/app/Actions/Service/DeployServiceApplication.php @@ -47,24 +47,30 @@ class DeployServiceApplication $commands->push("docker network inspect {$safeProjectName} >/dev/null 2>&1 || docker network create --attachable {$safeProjectName}"); } + $connectToPredefinedNetwork = (bool) data_get($service, 'connect_to_docker_network'); $upCommand = "docker compose --project-directory {$safeWorkdir} -f {$safeComposeFile} --project-name {$safeProjectName} up -d --no-deps"; if ($forceRebuild) { $upCommand .= ' --build'; } + if ($connectToPredefinedNetwork) { + $upCommand .= ' --no-start'; + } // No stdin for Compose prompts: they would read the next script lines (see StartService). $upCommand .= " {$safeComposeServiceName} < /dev/null"; $commands->push('echo Starting service container.'); $commands->push($upCommand); - $commands->push("docker network connect {$safeProjectName} coolify-proxy >/dev/null 2>&1 || true"); - - if (data_get($service, 'connect_to_docker_network')) { - $network = escapeshellarg($service->destination->network); - $containerName = escapeshellarg("{$composeServiceName}-{$service->uuid}"); - $networkAlias = escapeshellarg("{$composeServiceName}-{$service->uuid}"); - $commands->push("docker network connect --alias {$networkAlias} {$network} {$containerName} >/dev/null 2>&1 || true"); + if ($connectToPredefinedNetwork) { + // Connect to the predefined network before the container starts (see StartService). + $serviceConfig = data_get(parseDockerComposeYaml($service->docker_compose ?? ''), "services.{$composeServiceName}"); + if ($command = StartService::predefinedNetworkConnectCommand($service, $composeServiceName, $serviceConfig)) { + $commands->push($command); + } + $commands->push('docker start '.escapeshellarg("{$composeServiceName}-{$service->uuid}").' >/dev/null'); } + $commands->push("docker network connect {$safeProjectName} coolify-proxy >/dev/null 2>&1 || true"); + return remote_process($commands->toArray(), $service->server, type_uuid: $service->uuid, callEventOnFinish: 'ServiceStartFinished', callEventData: $service->id, queue: deployment_queue()); } } diff --git a/app/Actions/Service/StartService.php b/app/Actions/Service/StartService.php index a5812ced35..b8c60b3762 100644 --- a/app/Actions/Service/StartService.php +++ b/app/Actions/Service/StartService.php @@ -55,17 +55,22 @@ class StartService $commands[] = "docker network inspect $service->uuid >/dev/null 2>&1 || docker network create --attachable $service->uuid"; } $commands[] = 'echo Starting service.'; - $commands[] = "docker compose --project-directory {$workdir} -f {$workdir}/docker-compose.yml --project-name {$service->uuid} up -d --remove-orphans --force-recreate --build < /dev/null"; - $commands[] = "docker network connect $service->uuid coolify-proxy >/dev/null 2>&1 || true"; + $compose = "docker compose --project-directory {$workdir} -f {$workdir}/docker-compose.yml --project-name {$service->uuid}"; if (data_get($service, 'connect_to_docker_network')) { - $compose = data_get($service, 'docker_compose', []); - $safeNetwork = escapeshellarg($service->destination->network); - $serviceNames = data_get(parseDockerComposeYaml($compose), 'services', []); - foreach ($serviceNames as $serviceName => $serviceConfig) { - $containerName = escapeshellarg("{$serviceName}-{$service->uuid}"); - $commands[] = "docker network connect --alias {$containerName} {$safeNetwork} {$containerName} >/dev/null 2>&1 || true"; + // Connect the created containers to the predefined network before they start. A container + // that needs this network during a service_healthy wait never gets it after `up -d`. + $commands[] = "{$compose} up -d --remove-orphans --force-recreate --build --no-start < /dev/null"; + $composeServices = data_get(parseDockerComposeYaml(data_get($service, 'docker_compose', '')), 'services', []); + foreach ($composeServices as $serviceName => $serviceConfig) { + if ($command = self::predefinedNetworkConnectCommand($service, $serviceName, $serviceConfig)) { + $commands[] = $command; + } } + $commands[] = "{$compose} start < /dev/null"; + } else { + $commands[] = "{$compose} up -d --remove-orphans --force-recreate --build < /dev/null"; } + $commands[] = "docker network connect $service->uuid coolify-proxy >/dev/null 2>&1 || true"; $commands = array_merge($commands, $this->logDrainNetworkConnectCommands($service)); if ($service->is_maintenance_enabled) { // Domains can change with the compose file, so the maintenance routes follow them. @@ -79,6 +84,24 @@ class StartService return remote_process($commands, $service->server, type_uuid: $service->uuid, callEventOnFinish: 'ServiceStartFinished', callEventData: $service->id, queue: deployment_queue()); } + /** + * Connects a container to the predefined network of the destination, unless it is already + * connected (declared in the compose file, or a running container that was not recreated). + * Containers with a network_mode cannot join other networks and are skipped. + */ + public static function predefinedNetworkConnectCommand(Service $service, string $serviceName, mixed $serviceConfig): ?string + { + if (data_get($serviceConfig, 'network_mode') !== null) { + return null; + } + $network = escapeshellarg($service->destination->network); + $containerName = escapeshellarg("{$serviceName}-{$service->uuid}"); + $warning = escapeshellarg("Warning: Could not connect {$serviceName} to the {$service->destination->network} network."); + $connectedNetworks = "docker container inspect --format '{{range \$name, \$_ := .NetworkSettings.Networks}}{{println \$name}}{{end}}' {$containerName}"; + + return "{$connectedNetworks} | grep -qxF {$network} || docker network connect --alias {$containerName} {$network} {$containerName} || echo {$warning}"; + } + /** * Shows the volume warnings of the last parse (for example an external volume that the * service does not use yet) in the start log. diff --git a/app/Console/Kernel.php b/app/Console/Kernel.php index a88ba64a81..7f3e5fb8bf 100644 --- a/app/Console/Kernel.php +++ b/app/Console/Kernel.php @@ -25,6 +25,8 @@ use Illuminate\Foundation\Console\Kernel as ConsoleKernel; class Kernel extends ConsoleKernel { + private const CDN_JITTER_SECONDS = 600; + private Schedule $scheduleInstance; private InstanceSettings $settings; @@ -91,8 +93,8 @@ class Kernel extends ConsoleKernel $this->scheduleInstance->command('horizon:snapshot')->everyFiveMinutes(); $this->scheduleInstance->command('cleanup:unreachable-servers')->daily()->onOneServer(); - $this->scheduleInstance->job(new PullTemplatesFromCDN)->cron($this->updateCheckFrequency)->timezone($this->instanceTimezone)->onOneServer(); - $this->scheduleInstance->job(new PullChangelog)->cron($this->updateCheckFrequency)->timezone($this->instanceTimezone)->onOneServer(); + $this->scheduleInstance->job($this->withCdnJitter(new PullTemplatesFromCDN))->cron($this->updateCheckFrequency)->timezone($this->instanceTimezone)->onOneServer(); + $this->scheduleInstance->job($this->withCdnJitter(new PullChangelog))->cron($this->updateCheckFrequency)->timezone($this->instanceTimezone)->onOneServer(); $this->scheduleInstance->job(new CleanupInstanceStuffsJob)->everyTwoMinutes()->onOneServer(); $this->scheduleUpdates(); @@ -117,7 +119,7 @@ class Kernel extends ConsoleKernel private function pullImages(): void { - $this->scheduleInstance->job(new CheckHelperImageJob) + $this->scheduleInstance->job($this->withCdnJitter(new CheckHelperImageJob)) ->cron($this->updateCheckFrequency) ->timezone($this->instanceTimezone) ->onOneServer(); @@ -162,20 +164,29 @@ class Kernel extends ConsoleKernel private function scheduleUpdates(): void { - $this->scheduleInstance->job(new CheckForUpdatesJob) + $this->scheduleInstance->job($this->withCdnJitter(new CheckForUpdatesJob)) ->cron($this->updateCheckFrequency) ->timezone($this->instanceTimezone) ->onOneServer(); if ($this->settings->is_auto_update_enabled) { $autoUpdateFrequency = $this->settings->auto_update_frequency; - $this->scheduleInstance->job(new UpdateCoolifyJob) + $this->scheduleInstance->job($this->withCdnJitter(new UpdateCoolifyJob)) ->cron($autoUpdateFrequency) ->timezone($this->instanceTimezone) ->onOneServer(); } } + /** + * All instances share the same default cron, so without a random delay they all request the CDN + * at the same second and Cloudflare rejects the burst (error 1200). + */ + private function withCdnJitter(object $job): object + { + return $job->delay(random_int(0, self::CDN_JITTER_SECONDS)); + } + protected function commands(): void { $this->load(__DIR__.'/Commands'); diff --git a/app/Events/DockerCleanupDone.php b/app/Events/DockerCleanupDone.php index 2bd7fee198..f229c66451 100644 --- a/app/Events/DockerCleanupDone.php +++ b/app/Events/DockerCleanupDone.php @@ -21,4 +21,18 @@ class DockerCleanupDone implements ShouldBroadcast new PrivateChannel('team.'.$this->execution->server->team->id), ]; } + + /** + * Keep the payload under Reverb's max message size: the cleanup log can be large. + * + * @return array{executionId: int, serverId: int, status: string} + */ + public function broadcastWith(): array + { + return [ + 'executionId' => $this->execution->id, + 'serverId' => $this->execution->server_id, + 'status' => $this->execution->status, + ]; + } } diff --git a/app/Http/Controllers/Api/ApplicationsController.php b/app/Http/Controllers/Api/ApplicationsController.php index 4096ef94e5..2904a61806 100644 --- a/app/Http/Controllers/Api/ApplicationsController.php +++ b/app/Http/Controllers/Api/ApplicationsController.php @@ -421,7 +421,7 @@ class ApplicationsController extends Controller ], ), ], - 'watch_paths' => ['type' => 'string', 'description' => 'The watch paths.'], + 'watch_paths' => ['type' => 'string', 'description' => 'Watch path patterns, one per line (newline-separated). Commas do not separate patterns.'], 'use_build_server' => ['type' => 'boolean', 'nullable' => true, 'description' => 'Use build server.'], 'use_build_secrets' => ['type' => 'boolean', 'default' => false, 'description' => 'Use Docker Build Secrets for build-time environment variables.'], 'is_git_submodules_enabled' => ['type' => 'boolean', 'description' => 'Clone Git submodules.'], @@ -622,7 +622,7 @@ class ApplicationsController extends Controller ], ), ], - 'watch_paths' => ['type' => 'string', 'description' => 'The watch paths.'], + 'watch_paths' => ['type' => 'string', 'description' => 'Watch path patterns, one per line (newline-separated). Commas do not separate patterns.'], 'use_build_server' => ['type' => 'boolean', 'nullable' => true, 'description' => 'Use build server.'], 'use_build_secrets' => ['type' => 'boolean', 'default' => false, 'description' => 'Use Docker Build Secrets for build-time environment variables.'], 'is_git_submodules_enabled' => ['type' => 'boolean', 'description' => 'Clone Git submodules.'], @@ -823,7 +823,7 @@ class ApplicationsController extends Controller ], ), ], - 'watch_paths' => ['type' => 'string', 'description' => 'The watch paths.'], + 'watch_paths' => ['type' => 'string', 'description' => 'Watch path patterns, one per line (newline-separated). Commas do not separate patterns.'], 'use_build_server' => ['type' => 'boolean', 'nullable' => true, 'description' => 'Use build server.'], 'use_build_secrets' => ['type' => 'boolean', 'default' => false, 'description' => 'Use Docker Build Secrets for build-time environment variables.'], 'is_git_submodules_enabled' => ['type' => 'boolean', 'description' => 'Clone Git submodules.'], @@ -1276,6 +1276,9 @@ class ApplicationsController extends Controller $this->authorize('create', Application::class); + // removeUnnecessaryFieldsFromRequest() strips force_domain_override before the docker_compose_domains checks run. + $forceDomainOverride = $request->boolean('force_domain_override'); + if ($request->instant_deploy) { abort_unless($request->user()->tokenCan('deploy') || $request->user()->tokenCan('root'), 403, 'Missing required permissions: deploy'); } @@ -1531,7 +1534,7 @@ class ApplicationsController extends Controller }); $duplicates = $urls->duplicates()->unique()->values(); - if ($duplicates->isNotEmpty() && ! $request->boolean('force_domain_override')) { + if ($duplicates->isNotEmpty() && ! $forceDomainOverride) { $errors[] = 'The current request contains conflicting URLs: '.implode(', ', $duplicates->toArray()).' Use force_domain_override=true to proceed.'; } @@ -1552,7 +1555,7 @@ class ApplicationsController extends Controller ], 422); } - if ($result['hasConflicts'] && ! $request->boolean('force_domain_override')) { + if ($result['hasConflicts'] && ! $forceDomainOverride) { return response()->json([ 'message' => 'Domain conflicts detected. Use force_domain_override=true to proceed.', 'conflicts' => $result['conflicts'], @@ -1577,7 +1580,7 @@ class ApplicationsController extends Controller $application->domain_port_overrides = $domainPortOverrides; } $gitRepository = $application->git_repository; - $httpsRepository = scpStyleGitUrlToHttps($gitRepository); + $httpsRepository = sshGitUrlToHttps($gitRepository); if (is_string($httpsRepository)) { $gitRepository = $httpsRepository; $application->git_repository = $httpsRepository; @@ -1799,7 +1802,7 @@ class ApplicationsController extends Controller }); $duplicates = $urls->duplicates()->unique()->values(); - if ($duplicates->isNotEmpty() && ! $request->boolean('force_domain_override')) { + if ($duplicates->isNotEmpty() && ! $forceDomainOverride) { $errors[] = 'The current request contains conflicting URLs: '.implode(', ', $duplicates->toArray()).' Use force_domain_override=true to proceed. '; } @@ -1820,7 +1823,7 @@ class ApplicationsController extends Controller ], 422); } - if ($result['hasConflicts'] && ! $request->boolean('force_domain_override')) { + if ($result['hasConflicts'] && ! $forceDomainOverride) { return response()->json([ 'message' => 'Domain conflicts detected. Use force_domain_override=true to proceed.', 'conflicts' => $result['conflicts'], @@ -2032,7 +2035,7 @@ class ApplicationsController extends Controller }); $duplicates = $urls->duplicates()->unique()->values(); - if ($duplicates->isNotEmpty() && ! $request->boolean('force_domain_override')) { + if ($duplicates->isNotEmpty() && ! $forceDomainOverride) { $errors[] = 'The current request contains conflicting URLs: '.implode(', ', $duplicates->toArray()).' Use force_domain_override=true to proceed.'; } @@ -2053,7 +2056,7 @@ class ApplicationsController extends Controller ], 422); } - if ($result['hasConflicts'] && ! $request->boolean('force_domain_override')) { + if ($result['hasConflicts'] && ! $forceDomainOverride) { return response()->json([ 'message' => 'Domain conflicts detected. Use force_domain_override=true to proceed.', 'conflicts' => $result['conflicts'], @@ -3093,14 +3096,10 @@ class ApplicationsController extends Controller schema: new OA\Schema( type: 'object', properties: [ - 'project_uuid' => ['type' => 'string', 'description' => 'The project UUID.'], - 'server_uuid' => ['type' => 'string', 'description' => 'The server UUID.'], - 'environment_name' => ['type' => 'string', 'description' => 'The environment name.'], 'github_app_uuid' => ['type' => 'string', 'description' => 'The Github App UUID.'], 'git_repository' => ['type' => 'string', 'description' => 'The git repository URL.'], 'git_branch' => ['type' => 'string', 'description' => 'The git branch.'], 'ports_exposes' => ['type' => 'string', 'description' => 'The ports to expose.'], - 'destination_uuid' => ['type' => 'string', 'description' => 'The destination UUID.'], 'build_pack' => ['type' => 'string', 'enum' => ['nixpacks', 'railpack', 'static', 'dockerfile', 'dockercompose'], 'description' => 'The build pack type.'], 'name' => ['type' => 'string', 'description' => 'The application name.'], 'description' => ['type' => 'string', 'description' => 'The application description.'], @@ -3168,7 +3167,7 @@ class ApplicationsController extends Controller ], ), ], - 'watch_paths' => ['type' => 'string', 'description' => 'The watch paths.'], + 'watch_paths' => ['type' => 'string', 'description' => 'Watch path patterns, one per line (newline-separated). Commas do not separate patterns.'], 'use_build_server' => ['type' => 'boolean', 'nullable' => true, 'description' => 'Use build server.'], 'use_build_secrets' => ['type' => 'boolean', 'description' => 'Use Docker Build Secrets for build-time environment variables.'], 'http_basic_auth_hash_algorithm' => ['type' => 'string', 'enum' => ['bcrypt', 'argon2id'], 'description' => 'Hash algorithm for the HTTP Basic Authentication password. Argon2id needs the Caddy proxy, version 2.11 or newer.'], @@ -3301,10 +3300,12 @@ class ApplicationsController extends Controller $dnsCleanup = app(ManagedDnsRecordCleanup::class); $previousDnsHostnames = $dnsCleanup->hostnamesOf($application); $server = $application->destination->server; - $allowedFields = ['name', 'description', 'is_static', 'is_spa', 'is_auto_deploy_enabled', 'is_force_https_enabled', 'is_preview_deployments_enabled', 'domains', 'noindex_domains', 'git_repository', 'git_branch', 'git_commit_sha', 'docker_registry_image_name', 'docker_registry_image_tag', 'build_pack', 'static_image', 'install_command', 'build_command', 'start_command', 'ports_exposes', 'ports_mappings', 'custom_network_aliases', 'base_directory', 'publish_directory', 'health_check_enabled', 'health_check_type', 'health_check_command', 'health_check_path', 'health_check_port', 'health_check_host', 'health_check_method', 'health_check_return_code', 'health_check_scheme', 'health_check_response_text', 'health_check_interval', 'health_check_timeout', 'health_check_retries', 'health_check_start_period', 'limits_memory', 'limits_memory_swap', 'limits_memory_swappiness', 'limits_memory_reservation', 'limits_cpus', 'limits_cpuset', 'limits_cpu_shares', 'custom_labels', 'custom_docker_run_options', 'post_deployment_command', 'post_deployment_command_container', 'pre_deployment_command', 'pre_deployment_command_container', 'watch_paths', 'manual_webhook_secret_github', 'manual_webhook_secret_gitlab', 'manual_webhook_secret_bitbucket', 'manual_webhook_secret_gitea', 'dockerfile_location', 'dockerfile_target_build', 'docker_compose_location', 'docker_compose_custom_start_command', 'docker_compose_custom_build_command', 'docker_compose_domains', 'redirect', 'instant_deploy', 'use_build_server', 'use_build_secrets', 'custom_nginx_configuration', 'is_http_basic_auth_enabled', 'http_basic_auth_username', 'http_basic_auth_password', 'http_basic_auth_hash_algorithm', 'http_basic_auth_bcrypt_cost', 'http_basic_auth_argon2id_memory_cost', 'http_basic_auth_argon2id_time_cost', 'connect_to_docker_network', 'force_domain_override', 'is_container_label_escape_enabled', 'is_preserve_repository_enabled', 'preview_url_template', 'max_restart_count', ...self::APPLICATION_SETTING_FIELDS]; + $allowedFields = ['name', 'description', 'is_static', 'is_spa', 'is_auto_deploy_enabled', 'is_force_https_enabled', 'is_preview_deployments_enabled', 'domains', 'noindex_domains', 'git_repository', 'git_branch', 'git_commit_sha', 'docker_registry_image_name', 'docker_registry_image_tag', 'build_pack', 'static_image', 'install_command', 'build_command', 'start_command', 'ports_exposes', 'ports_mappings', 'custom_network_aliases', 'base_directory', 'publish_directory', 'health_check_enabled', 'health_check_type', 'health_check_command', 'health_check_path', 'health_check_port', 'health_check_host', 'health_check_method', 'health_check_return_code', 'health_check_scheme', 'health_check_response_text', 'health_check_interval', 'health_check_timeout', 'health_check_retries', 'health_check_start_period', 'limits_memory', 'limits_memory_swap', 'limits_memory_swappiness', 'limits_memory_reservation', 'limits_cpus', 'limits_cpuset', 'limits_cpu_shares', 'custom_labels', 'custom_docker_run_options', 'post_deployment_command', 'post_deployment_command_container', 'pre_deployment_command', 'pre_deployment_command_container', 'watch_paths', 'manual_webhook_secret_github', 'manual_webhook_secret_gitlab', 'manual_webhook_secret_bitbucket', 'manual_webhook_secret_gitea', 'dockerfile_location', 'dockerfile_target_build', 'docker_compose_location', 'docker_compose_custom_start_command', 'docker_compose_custom_build_command', 'docker_compose_domains', 'redirect', 'instant_deploy', 'use_build_server', 'use_build_secrets', 'custom_nginx_configuration', 'is_http_basic_auth_enabled', 'http_basic_auth_username', 'http_basic_auth_password', 'http_basic_auth_hash_algorithm', 'http_basic_auth_bcrypt_cost', 'http_basic_auth_argon2id_memory_cost', 'http_basic_auth_argon2id_time_cost', 'connect_to_docker_network', 'force_domain_override', 'is_container_label_escape_enabled', 'is_preserve_repository_enabled', 'preview_url_template', 'max_restart_count', 'github_app_uuid', 'dockerfile', ...self::APPLICATION_SETTING_FIELDS]; $validationRules = [ 'name' => 'string|max:255', + 'github_app_uuid' => 'string', + 'dockerfile' => 'string', 'description' => 'string|nullable', 'watch_paths' => 'string|nullable', 'docker_compose_domains' => 'array|nullable', @@ -3364,6 +3365,28 @@ class ApplicationsController extends Controller 'custom_nginx_configuration' => $customNginxConfiguration, ]); } + if ($request->has('dockerfile')) { + if (blank($application->dockerfile)) { + return response()->json([ + 'message' => 'Validation failed.', + 'errors' => [ + 'dockerfile' => 'The dockerfile can only be updated for applications created from a Dockerfile.', + ], + ], 422); + } + $dockerfile = is_string($request->dockerfile) && isBase64Encoded($request->dockerfile) ? base64_decode($request->dockerfile) : false; + if ($dockerfile === false || blank($dockerfile) || mb_detect_encoding($dockerfile, 'UTF-8', true) === false) { + return response()->json([ + 'message' => 'Validation failed.', + 'errors' => [ + 'dockerfile' => 'The dockerfile should be base64 encoded.', + ], + ], 422); + } + $request->merge([ + 'dockerfile' => $dockerfile, + ]); + } $return = $this->validateDataApplications($request, $server); if ($return instanceof JsonResponse) { return $return; @@ -3383,6 +3406,24 @@ class ApplicationsController extends Controller ], 422); } + $githubApp = null; + if ($request->has('github_app_uuid')) { + if ($application->build_pack === 'dockerimage' || filled($application->dockerfile) || $application->deploymentType() === 'deploy_key') { + return response()->json([ + 'message' => 'Validation failed.', + 'errors' => [ + 'github_app_uuid' => 'The github_app_uuid can only be changed for applications deployed from a GitHub App or public repository.', + ], + ], 422); + } + $githubApp = GithubApp::where('uuid', $request->github_app_uuid) + ->where(fn ($query) => $query->where('team_id', $teamId)->orWhere('is_system_wide', true)) + ->first(); + if (! $githubApp?->isConnected()) { + return response()->json(['message' => 'Github App not found.'], 404); + } + } + $applicationSettings = $this->applicationSettingsFromRequest($request); if ($prefixValidation = $this->containerNamePrefixValidationResponse($applicationSettings, $application->destination->server, $application)) { return $prefixValidation; @@ -3642,6 +3683,12 @@ class ApplicationsController extends Controller $requestHasNoindexDomains = $request->has('noindex_domains'); data_forget($data, 'noindex_domains'); $application->fill($data); + if ($githubApp && ($application->source_id !== $githubApp->id || $application->source_type !== $githubApp->getMorphClass())) { + $application->source_id = $githubApp->id; + $application->source_type = $githubApp->getMorphClass(); + // The next deployment fetches the repository ID from the new source. + $application->repository_project_id = null; + } if ($requestHasNoindexDomains) { // Must run after fqdn is filled: flags are kept only for domains the app still has. $application->setNoindexDomains($request->input('noindex_domains') ?? []); @@ -3650,6 +3697,9 @@ class ApplicationsController extends Controller $application->custom_labels = str(implode('|coolify|', generateLabelsApplication($application)))->replace('|coolify|', "\n"); } $application->withoutAuditLogging(fn () => $application->save()); + if ($application->wasChanged('dockerfile')) { + $application->parseHealthcheckFromDockerfile($application->dockerfile); + } $dnsCleanup->queueReleaseOfRemovedHostnames($application, $previousDnsHostnames, (int) $teamId); auditLog('api.application.updated', [ diff --git a/app/Http/Controllers/Api/DatabasesController.php b/app/Http/Controllers/Api/DatabasesController.php index fd7c53e90a..634f20720d 100644 --- a/app/Http/Controllers/Api/DatabasesController.php +++ b/app/Http/Controllers/Api/DatabasesController.php @@ -556,6 +556,7 @@ class DatabasesController extends Controller 'limits_cpus' => ['type' => 'string', 'description' => 'CPU limit of the database'], 'limits_cpuset' => ['type' => 'string', 'description' => 'CPU set of the database'], 'limits_cpu_shares' => ['type' => 'integer', 'description' => 'CPU shares of the database'], + 'instant_deploy' => ['type' => 'boolean', 'description' => 'Restart the database after the update. Requires the deploy ability.'], 'postgres_user' => ['type' => 'string', 'description' => 'PostgreSQL user'], 'postgres_password' => ['type' => 'string', 'description' => 'PostgreSQL password'], 'postgres_db' => ['type' => 'string', 'description' => 'PostgreSQL database'], @@ -644,6 +645,7 @@ class DatabasesController extends Controller 'limits_cpus' => 'string', 'limits_cpuset' => 'string|nullable', 'limits_cpu_shares' => 'numeric', + 'instant_deploy' => 'boolean', 'redact_all_env_values_in_logs' => 'boolean', ]); @@ -654,6 +656,7 @@ class DatabasesController extends Controller ], 422); } $uuid = $request->uuid; + $instantDeploy = $request->boolean('instant_deploy'); $redactAllEnvValuesInLogs = $request->has('redact_all_env_values_in_logs') ? $request->boolean('redact_all_env_values_in_logs') : null; removeUnnecessaryFieldsFromRequest($request); $database = queryDatabaseByUuidWithinTeam($uuid, $teamId); @@ -663,6 +666,11 @@ class DatabasesController extends Controller $this->authorize('update', $database); + if ($instantDeploy) { + abort_unless($request->user()->tokenCan('deploy') || $request->user()->tokenCan('root'), 403, 'Missing required permissions: deploy'); + $this->authorize('manage', $database); + } + if ($redactAllEnvValuesInLogs !== null) { $this->authorize('manageEnvironment', $database); } @@ -890,6 +898,17 @@ class DatabasesController extends Controller 'errors' => $errors, ], 422); } + $restartReservation = null; + if ($instantDeploy) { + $prerequisiteError = StartDatabase::prerequisiteError($database); + if ($prerequisiteError !== null) { + return response()->json(['message' => $prerequisiteError], 422); + } + $restartReservation = StartDatabase::reserveOperation($database); + if ($restartReservation === null) { + return response()->json(['message' => ResourceStartActivity::DATABASE_OPERATION_IN_PROGRESS_MESSAGE], 409); + } + } $whatToDoWithDatabaseProxy = null; if ($request->is_public === false && $database->is_public === true) { $whatToDoWithDatabaseProxy = 'stop'; @@ -903,12 +922,22 @@ class DatabasesController extends Controller $database->redact_all_env_values_in_logs = $redactAllEnvValuesInLogs; } $database->update($request->only($allowedFields)); + // The Redis password is stored in the REDIS_PASSWORD variable, not in a column. + if ($database instanceof StandaloneRedis && filled($request->redis_password)) { + $database->runtime_environment_variables()->updateOrCreate( + ['key' => 'REDIS_PASSWORD'], + ['value' => $request->redis_password, 'resourceable_id' => $database->id] + ); + } if ($whatToDoWithDatabaseProxy === 'start') { StartDatabaseProxy::dispatch($database); } elseif ($whatToDoWithDatabaseProxy === 'stop') { StopDatabaseProxy::dispatch($database); } + if ($restartReservation !== null) { + StartDatabase::dispatchReserved(RestartDatabase::class, $database, $restartReservation); + } auditLog('api.database.updated', [ 'team_id' => $teamId, diff --git a/app/Http/Controllers/Api/ServicesController.php b/app/Http/Controllers/Api/ServicesController.php index c858c1bc8c..0a86606518 100644 --- a/app/Http/Controllers/Api/ServicesController.php +++ b/app/Http/Controllers/Api/ServicesController.php @@ -210,6 +210,7 @@ class ServicesController extends Controller $application->fqdn = $containerUrls; $application->save(); + updateCompose($application); }); if (! empty($errors)) { @@ -308,6 +309,7 @@ class ServicesController extends Controller 'destination_uuid' => ['type' => 'string', 'description' => 'Destination UUID. Required if server has multiple destinations.'], 'instant_deploy' => ['type' => 'boolean', 'default' => false, 'description' => 'Start the service immediately after creation.'], 'docker_compose_raw' => ['type' => 'string', 'description' => 'The base64 encoded Docker Compose content.'], + 'connect_to_docker_network' => ['type' => 'boolean', 'default' => false, 'description' => 'Connect the service to the predefined docker network.'], 'urls' => [ 'type' => 'array', 'description' => 'Array of URLs to be applied to containers of a service.', @@ -389,7 +391,7 @@ class ServicesController extends Controller )] public function create_service(Request $request) { - $allowedFields = ['type', 'name', 'description', 'project_uuid', 'environment_name', 'environment_uuid', 'server_uuid', 'destination_uuid', 'instant_deploy', 'docker_compose_raw', 'urls', 'force_domain_override', 'is_container_label_escape_enabled', 'redact_all_env_values_in_logs', 'tags']; + $allowedFields = ['type', 'name', 'description', 'project_uuid', 'environment_name', 'environment_uuid', 'server_uuid', 'destination_uuid', 'instant_deploy', 'docker_compose_raw', 'connect_to_docker_network', 'urls', 'force_domain_override', 'is_container_label_escape_enabled', 'redact_all_env_values_in_logs', 'tags']; $teamId = getTeamIdFromToken(); if (is_null($teamId)) { @@ -417,6 +419,7 @@ class ServicesController extends Controller 'name' => 'string|max:255', 'description' => 'string|nullable', 'instant_deploy' => 'boolean', + 'connect_to_docker_network' => 'boolean', 'urls' => 'array|nullable', 'urls.*' => 'array:name,url', 'urls.*.name' => 'string|required', @@ -546,7 +549,7 @@ class ServicesController extends Controller 'destination_id' => $destination->id, 'destination_type' => $destination->getMorphClass(), ]; - if (in_array($oneClickServiceName, NEEDS_TO_CONNECT_TO_PREDEFINED_NETWORK)) { + if (in_array($oneClickServiceName, NEEDS_TO_CONNECT_TO_PREDEFINED_NETWORK) || $request->boolean('connect_to_docker_network')) { data_set($servicePayload, 'connect_to_docker_network', true); } $service = new Service($servicePayload); diff --git a/app/Http/Controllers/Webhook/Bitbucket.php b/app/Http/Controllers/Webhook/Bitbucket.php index 46e4b50362..3138d168a3 100644 --- a/app/Http/Controllers/Webhook/Bitbucket.php +++ b/app/Http/Controllers/Webhook/Bitbucket.php @@ -85,7 +85,11 @@ class Bitbucket extends Controller $failure_key = $this->manualWebhookFailureRateLimitKey($request, 'bitbucket', $full_name, $branch); // A redelivery of the same signed payload is one guess. $failure_attempt = $this->manualWebhookSignedPayloadAttempt($request, $x_bitbucket_token); - $applications = $this->manualWebhookApplications(Application::query()->where('git_branch', $branch), $full_name); + $applications = Application::query(); + if ($x_bitbucket_event !== 'pullrequest:rejected' && $x_bitbucket_event !== 'pullrequest:fulfilled') { + $applications->where('git_branch', $branch); + } + $applications = $this->manualWebhookApplications($applications, $full_name); if ($applications->isEmpty()) { return $this->unauthenticatedManualWebhookResponse($failure_key, $failure_attempt); } diff --git a/app/Http/Controllers/Webhook/Gitea.php b/app/Http/Controllers/Webhook/Gitea.php index 6a9c8dd624..05ceebeabd 100644 --- a/app/Http/Controllers/Webhook/Gitea.php +++ b/app/Http/Controllers/Webhook/Gitea.php @@ -85,7 +85,10 @@ class Gitea extends Controller } } if ($x_gitea_event === 'pull_request') { - $applications = $this->manualWebhookApplications($applications->where('git_branch', $base_branch), $full_name); + if ($action !== 'closed') { + $applications->where('git_branch', $base_branch); + } + $applications = $this->manualWebhookApplications($applications, $full_name); if ($applications->isEmpty()) { return $this->unauthenticatedManualWebhookResponse($failure_key, $failure_attempt); } diff --git a/app/Http/Controllers/Webhook/Github.php b/app/Http/Controllers/Webhook/Github.php index 64a357ea49..e615abd3c6 100644 --- a/app/Http/Controllers/Webhook/Github.php +++ b/app/Http/Controllers/Webhook/Github.php @@ -72,7 +72,6 @@ class Github extends Controller 'before_sha' => $before_sha, 'after_sha' => $after_sha, 'commit_sha' => $commit_sha, - 'author_association' => $author_association, 'is_fork_pull_request' => $is_fork_pull_request, ] = $this->readPullRequestPayload($payload); $full_name = data_get($payload, 'repository.full_name'); @@ -241,7 +240,6 @@ class Github extends Controller beforeSha: $before_sha, afterSha: $after_sha, commitSha: $commit_sha, - authorAssociation: $author_association, fullName: $full_name, isForkPullRequest: $is_fork_pull_request, ); @@ -402,7 +400,12 @@ class Github extends Controller // Just pong return response('pong'); } - $github_app = GithubApp::where('app_id', $x_github_hook_installation_target_id)->first(); + // One GitHub App can be installed on several accounts, each added as its own source. + $installation_id = data_get($payload, 'installation.id'); + $github_app = is_numeric($installation_id) + ? GithubApp::where('app_id', $x_github_hook_installation_target_id)->where('installation_id', (int) $installation_id)->first() + : null; + $github_app ??= GithubApp::where('app_id', $x_github_hook_installation_target_id)->first(); if (is_null($github_app)) { return response('Nothing to do. No GitHub App found.'); } @@ -462,7 +465,6 @@ class Github extends Controller 'before_sha' => $before_sha, 'after_sha' => $after_sha, 'commit_sha' => $commit_sha, - 'author_association' => $author_association, 'is_fork_pull_request' => $is_fork_pull_request, ] = $this->readPullRequestPayload($payload); $full_name = $this->manualWebhookRepositoryFullName(data_get($payload, 'repository.full_name')); @@ -487,6 +489,9 @@ class Github extends Controller if ($applications->isEmpty()) { return response("Nothing to do. No applications found with branch '$branch'."); } + if ($applications->contains(fn (Application $application): bool => filled($application->watch_paths))) { + $changed_files = $this->githubPushComparedFiles($github_app, $payload) ?? $changed_files; + } } if ($x_github_event === 'pull_request') { if ($action !== 'closed') { @@ -601,7 +606,6 @@ class Github extends Controller beforeSha: $before_sha, afterSha: $after_sha, commitSha: $commit_sha, - authorAssociation: $author_association, fullName: $full_name, isForkPullRequest: $is_fork_pull_request, ); @@ -623,10 +627,39 @@ class Github extends Controller } } + /** + * Files that the push changes between its before and after commits, from + * the GitHub compare API. + * + * The file lists of the pushed commits are not the net change: a merge + * commit names the files that it changes relative to its first parent, + * also when the push does not change them. Returns null for a new branch, + * a missing repository name or an API failure; the caller then uses the + * file lists of the pushed commits. + * + * @return Collection|null + * + * @throws InvalidWebhookPayloadException When the before or after value is not a commit SHA. + */ + private function githubPushComparedFiles(GithubApp $githubApp, mixed $payload): ?Collection + { + $before = $this->webhookCommitSha($payload, 'before'); + $after = $this->webhookCommitSha($payload, 'after'); + $fullName = $this->manualWebhookRepositoryFullName(data_get($payload, 'repository.full_name')); + if ($before === null || $after === null || $fullName === null || preg_match('/\A0+\z/', $before) === 1) { + return null; + } + + [$owner, $repo] = explode('/', $fullName, 2); + $files = getGithubCommitRangeFiles($githubApp, $owner, $repo, $before, $after); + + return $files === null ? null : collect($files); + } + /** * Read and validate the pull_request payload fields that the handlers use. * - * @return array{action: string, pull_request_id: int, pull_request_html_url: string, pull_request_title: ?string, branch: ?string, base_branch: ?string, before_sha: ?string, after_sha: ?string, commit_sha: string, author_association: ?string, is_fork_pull_request: bool} + * @return array{action: string, pull_request_id: int, pull_request_html_url: string, pull_request_title: ?string, branch: ?string, base_branch: ?string, before_sha: ?string, after_sha: ?string, commit_sha: string, is_fork_pull_request: bool} * * @throws InvalidWebhookPayloadException When a field has a wrong type or format. */ @@ -644,7 +677,6 @@ class Github extends Controller 'before_sha' => $this->webhookCommitSha($payload, 'before'), 'after_sha' => $this->webhookCommitSha($payload, 'after') ?? $head_sha, 'commit_sha' => $head_sha ?? 'HEAD', - 'author_association' => $this->webhookPayloadString($payload, 'pull_request.author_association'), 'is_fork_pull_request' => $this->isForkPullRequest($payload), ]; } @@ -652,13 +684,15 @@ class Github extends Controller /** * Determine whether a pull_request webhook payload originates from a fork. * - * GitHub's `author_association` is not a reliable trust signal (it grants - * CONTRIBUTOR to anyone who has merely opened an issue/PR before), so fork - * detection is gated on whether the PR crosses repository boundaries. + * A same-repository PR branch can only be pushed by users with write access, + * so this is the trust signal for preview deployments. GitHub's + * `author_association` is not reliable for this: users with access through a + * team or a private organization membership arrive as CONTRIBUTOR or NONE. * * The repository id comparison is the canonical signal; the `head.repo.fork` * flag and a case-insensitive full_name comparison are fallbacks for payloads - * where the ids are unavailable (e.g. a deleted head repository). + * where the ids are unavailable. A PR whose head repository is unknown (e.g. + * a deleted fork) is treated as a fork. * * @throws InvalidWebhookPayloadException When a repository id is not a positive integer. */ @@ -682,7 +716,7 @@ class Github extends Controller return Str::lower($headRepoFullName) !== Str::lower($baseRepoFullName); } - return false; + return true; } public function redirect(Request $request) diff --git a/app/Http/Controllers/Webhook/Gitlab.php b/app/Http/Controllers/Webhook/Gitlab.php index 893e8a7bb3..892408976d 100644 --- a/app/Http/Controllers/Webhook/Gitlab.php +++ b/app/Http/Controllers/Webhook/Gitlab.php @@ -237,7 +237,10 @@ class Gitlab extends Controller 'skip_deploy' => $skip_deploy_pr, ] = $this->readMergeRequestPayload($payload); - $applications = $applications->where('git_branch', $base_branch)->get(); + if (! in_array($action, ['closed', 'close', 'merge'])) { + $applications->where('git_branch', $base_branch); + } + $applications = $applications->get(); foreach ($applications as $application) { if (! $application->destination->server->isFunctional()) { @@ -446,7 +449,10 @@ class Gitlab extends Controller } } if ($x_gitlab_event === 'merge_request') { - $applications = $this->manualWebhookApplications($applications->where('git_branch', $base_branch), $full_name); + if (! in_array($action, ['closed', 'close', 'merge'])) { + $applications->where('git_branch', $base_branch); + } + $applications = $this->manualWebhookApplications($applications, $full_name); if ($applications->isEmpty()) { return $this->unauthenticatedManualWebhookResponse($failure_key, $failure_attempt); } diff --git a/app/Http/Kernel.php b/app/Http/Kernel.php index 2e6c36b722..48710f88fe 100644 --- a/app/Http/Kernel.php +++ b/app/Http/Kernel.php @@ -16,6 +16,7 @@ use App\Http\Middleware\EncryptCookies; use App\Http\Middleware\EnsureMcpEnabled; use App\Http\Middleware\EnsureTeamMcpEnabled; use App\Http\Middleware\EnsureTokenBelongsToCurrentTeamMember; +use App\Http\Middleware\KeepConcurrentTeamSwitch; use App\Http\Middleware\PreventRequestsDuringMaintenance; use App\Http\Middleware\RedirectIfAuthenticated; use App\Http\Middleware\RequirePassword; @@ -73,6 +74,7 @@ class Kernel extends HttpKernel EncryptCookies::class, AddQueuedCookiesToResponse::class, StartSession::class, + KeepConcurrentTeamSwitch::class, ShareErrorsFromSession::class, VerifyCsrfToken::class, SubstituteBindings::class, diff --git a/app/Http/Middleware/ApiIpAllowed.php b/app/Http/Middleware/ApiIpAllowed.php index da2d009576..ede32567d0 100644 --- a/app/Http/Middleware/ApiIpAllowed.php +++ b/app/Http/Middleware/ApiIpAllowed.php @@ -14,13 +14,9 @@ class ApiIpAllowed return $next($request); } - $allowedIps = instanceSettings()->allowed_ips; - if (! $allowedIps) { - return $next($request); - } - - // Check for special case: 0.0.0.0 means allow all - if (trim($allowedIps) === '0.0.0.0') { + $allowedIps = trim((string) instanceSettings()->allowed_ips); + // Empty or 0.0.0.0 means allow all + if ($allowedIps === '' || $allowedIps === '0.0.0.0') { return $next($request); } diff --git a/app/Http/Middleware/KeepConcurrentTeamSwitch.php b/app/Http/Middleware/KeepConcurrentTeamSwitch.php new file mode 100644 index 0000000000..7f9a796ade --- /dev/null +++ b/app/Http/Middleware/KeepConcurrentTeamSwitch.php @@ -0,0 +1,37 @@ +session()->get('currentTeam'), 'id'); + + $response = $next($request); + + $session = $request->session(); + if (data_get($session->get('currentTeam'), 'id') !== $teamIdAtStart) { + return $response; + } + + $storedSession = clone $session; + $storedSession->start(); + $storedTeam = $storedSession->get('currentTeam'); + if ($storedTeam && data_get($storedTeam, 'id') !== $teamIdAtStart) { + $session->put('currentTeam', $storedTeam); + } + + return $response; + } +} diff --git a/app/Jobs/ApplicationDeploymentJob.php b/app/Jobs/ApplicationDeploymentJob.php index 9287e2030d..9fa8621fb1 100644 --- a/app/Jobs/ApplicationDeploymentJob.php +++ b/app/Jobs/ApplicationDeploymentJob.php @@ -260,6 +260,8 @@ class ApplicationDeploymentJob implements ShouldBeEncrypted, ShouldQueue $source = data_get($this->application, 'source'); if ($source) { $this->source = $source->getMorphClass()::where('id', $this->application->source->id)->first(); + // Share one source instance so the job reuses its in-memory GitHub installation token. + $this->application->setRelation('source', $this->source); } $this->server = Server::find($this->application_deployment_queue->server_id); $this->timeout = $this->server->settings->dynamic_timeout; @@ -975,18 +977,35 @@ class ApplicationDeploymentJob implements ShouldBeEncrypted, ShouldQueue return $composeFile; } + /** + * Pulls the registry images of the Compose services before the current deployment stops. Images that + * this deployment builds are not pulled: a pull fails for an image that is only local, and a pull of + * an image that is also in a registry replaces the new build with the old registry image. + */ private function pull_docker_compose_images(): void { + if ($this->docker_compose_custom_build_command && ! $this->use_build_server) { + $this->application_deployment_queue->addLogEntry('Skipping the image pull before stopping the current deployment, because the custom build command can build images outside of Docker Compose.'); + + return; + } + + $services = $this->registry_image_compose_services(); + if ($services->isEmpty()) { + return; + } + $this->application_deployment_queue->addLogEntry('Pulling image-based services before stopping the current deployment.'); + $services = $services->map(fn (string $service) => escapeshellarg($service))->implode(' '); if ($this->use_build_server) { $this->write_deployment_configurations(); $this->server = $this->mainServer; $workdir = $this->application->workdir(); - $command = "{$this->coolify_variables} docker compose --env-file {$workdir}/.env --project-name {$this->application->uuid} --project-directory {$workdir} -f {$workdir}{$this->docker_compose_location} pull --ignore-buildable"; + $command = "{$this->coolify_variables} docker compose --env-file {$workdir}/.env --project-name {$this->application->uuid} --project-directory {$workdir} -f {$workdir}{$this->docker_compose_location} pull --ignore-buildable {$services}"; } else { $workdir = $this->workdir; - $command = executeInDocker($this->deployment_uuid, "{$this->coolify_variables} docker compose --env-file {$workdir}/.env --project-name {$this->application->uuid} --project-directory {$workdir} -f {$workdir}{$this->docker_compose_location} pull --ignore-buildable"); + $command = executeInDocker($this->deployment_uuid, "{$this->coolify_variables} docker compose --env-file {$workdir}/.env --project-name {$this->application->uuid} --project-directory {$workdir} -f {$workdir}{$this->docker_compose_location} pull --ignore-buildable {$services}"); } $this->execute_remote_command([ @@ -995,6 +1014,26 @@ class ApplicationDeploymentJob implements ShouldBeEncrypted, ShouldQueue ]); } + /** + * Returns the names of the Compose services that get their image from a registry. A service with a + * `build` section, or a service that uses the image of such a service, gets its image from the build. + * + * @return Collection + */ + private function registry_image_compose_services(): Collection + { + $services = collect(data_get(Yaml::parse($this->generated_docker_compose), 'services', [])); + $builtImages = $services->filter(fn ($service) => filled(data_get($service, 'build')))->pluck('image')->filter(); + + return $services + ->filter(fn ($service) => filled(data_get($service, 'image')) + && blank(data_get($service, 'build')) + && ! $builtImages->contains(data_get($service, 'image'))) + ->keys() + ->map(fn ($service) => (string) $service) + ->values(); + } + /** * Starts the Compose containers. Content files are written first, because `docker compose up` * creates a missing bind source as an empty directory. @@ -1711,7 +1750,7 @@ class ApplicationDeploymentJob implements ShouldBeEncrypted, ShouldQueue $ports = $this->application->main_port(); $coolify_envs = $this->generate_coolify_env_variables(); $coolify_envs->each(function ($item, $key) use ($envs) { - $envs->push($key.'='.$item); + $envs->push($key.'='.escapeComposeEnvFileValue($item)); }); if ($this->pull_request_id === 0) { // Generate SERVICE_ variables first for dockercompose @@ -3189,13 +3228,12 @@ class ApplicationDeploymentJob implements ShouldBeEncrypted, ShouldQueue private function clone_repository() { - $importCommands = $this->generate_git_import_commands(); $this->application_deployment_queue->addLogEntry("\n----------------------------------------"); $this->application_deployment_queue->addLogEntry("Importing {$this->customRepository}:{$this->application->git_branch} (commit sha {$this->commit}) to {$this->basedir}."); if ($this->pull_request_id !== 0) { $this->application_deployment_queue->addLogEntry("Checking out tag pull/{$this->pull_request_id}/head."); } - $this->execute_remote_command(...$this->gitCommandDefinitions($importCommands)); + $this->import_repository(); $this->create_workdir(); $this->execute_remote_command( [ @@ -3213,6 +3251,37 @@ class ApplicationDeploymentJob implements ShouldBeEncrypted, ShouldQueue } } + /** + * SSH exits with 255 when the connection drops, which can happen during a long clone. Git exits with + * 128 for its own errors, so only 255 is retried. The interrupted clone can still run in the helper + * container, so stop it and remove the partial checkout before cloning again. + */ + private function import_repository(): void + { + $maxAttempts = config('constants.ssh.max_retries'); + + for ($attempt = 1; ; $attempt++) { + try { + $this->execute_remote_command(...$this->gitCommandDefinitions($this->generate_git_import_commands())); + + return; + } catch (DeploymentException $e) { + if ($e->getCode() !== 255 || $attempt >= $maxAttempts) { + throw $e; + } + + $delay = $this->calculateRetryDelay($attempt - 1); + $this->application_deployment_queue->addLogEntry('Lost the SSH connection while importing the repository. Retrying in '.$delay.'s (attempt '.($attempt + 1)."/{$maxAttempts}).", 'stderr'); + Sleep::for($delay)->seconds(); + $this->execute_remote_command([ + executeInDocker($this->deployment_uuid, 'pkill git; rm -rf '.escapeshellarg($this->basedir)), + 'hidden' => true, + 'ignore_errors' => true, + ]); + } + } + } + private function generate_git_import_commands() { ['commands' => $commands, 'branch' => $this->branch, 'fullRepoUrl' => $this->fullRepoUrl] = $this->application->generateGitImportCommands( @@ -3910,7 +3979,10 @@ COPY ./nginx.conf /etc/nginx/conf.d/default.conf"); } } - add_coolify_default_environment_variables($this->application, $coolify_envs, $this->application->environment_variables_preview); + // Runtime only, so a project or server change does not break the Docker build cache + if (! $forBuildTime) { + $coolify_envs = $coolify_envs->merge(coolifyDefaultEnvironmentVariables($this->application, $this->application->environment_variables_preview->pluck('key'), $this->mainServer)); + } } else { // Only add SOURCE_COMMIT for runtime OR when explicitly enabled for build-time @@ -3954,7 +4026,10 @@ COPY ./nginx.conf /etc/nginx/conf.d/default.conf"); } } - add_coolify_default_environment_variables($this->application, $coolify_envs, $this->application->environment_variables); + // Runtime only, so a project or server change does not break the Docker build cache + if (! $forBuildTime) { + $coolify_envs = $coolify_envs->merge(coolifyDefaultEnvironmentVariables($this->application, $this->application->environment_variables->pluck('key'), $this->mainServer)); + } } diff --git a/app/Jobs/CheckHelperImageJob.php b/app/Jobs/CheckHelperImageJob.php index 6d76da8eb7..56118a238a 100644 --- a/app/Jobs/CheckHelperImageJob.php +++ b/app/Jobs/CheckHelperImageJob.php @@ -9,6 +9,7 @@ use Illuminate\Foundation\Bus\Dispatchable; use Illuminate\Queue\InteractsWithQueue; use Illuminate\Queue\SerializesModels; use Illuminate\Support\Facades\Http; +use Illuminate\Support\Facades\Log; class CheckHelperImageJob implements ShouldBeEncrypted, ShouldQueue { @@ -20,20 +21,18 @@ class CheckHelperImageJob implements ShouldBeEncrypted, ShouldQueue public function handle(): void { - try { - $response = Http::retry(3, 1000)->get(config('constants.coolify.versions_url')); - if ($response->successful()) { - $versions = $response->json(); - $settings = instanceSettings(); - $latest_version = data_get($versions, 'coolify.helper.version'); - $current_version = $settings->helper_version; - if (version_compare($latest_version, $current_version, '>')) { - $settings->update(['helper_version' => $latest_version]); - } - } - } catch (\Throwable $e) { - send_internal_notification('CheckHelperImageJob failed with: '.$e->getMessage()); - throw $e; + $response = Http::retry(3, 1000, throw: false)->get(config('constants.coolify.versions_url')); + if ($response->failed()) { + // The CDN can be briefly unavailable. The next scheduled run tries again, so do not notify. + Log::warning('CheckHelperImageJob could not fetch versions', ['status' => $response->status()]); + + return; + } + + $latest_version = data_get($response->json(), 'coolify.helper.version'); + $settings = instanceSettings(); + if (version_compare($latest_version, $settings->helper_version, '>')) { + $settings->update(['helper_version' => $latest_version]); } } } diff --git a/app/Jobs/CleanupStaleMultiplexedConnections.php b/app/Jobs/CleanupStaleMultiplexedConnections.php index 69e151d24f..3162ae9c41 100644 --- a/app/Jobs/CleanupStaleMultiplexedConnections.php +++ b/app/Jobs/CleanupStaleMultiplexedConnections.php @@ -28,8 +28,14 @@ class CleanupStaleMultiplexedConnections implements ShouldQueue /** * Kill backgrounded ssh master processes that lost the ControlPath socket * race. Such processes are not masters, so ControlPersist never reaps them - * and they leak memory until the container restarts. A legitimate master - * always owns its socket file; an orphan has none. + * and they hold an authenticated session until the container restarts. + * + * Every `ssh -fN` for a server carries the same ControlPath, so the socket + * file existing says nothing about which process owns it. Ownership is read + * from the socket itself (`ssh -O check` reports the master pid); every + * other process for that ControlPath is an orphan. When the socket file is + * missing, no process owns it. When it exists but does not answer, the + * owner is unknown and nothing is reaped. * * Processes younger than the minimum age are skipped: a freshly forked * master creates its socket a few milliseconds after starting, so a young @@ -39,6 +45,7 @@ class CleanupStaleMultiplexedConnections implements ShouldQueue { $muxDir = storage_path('app/ssh/mux'); $minAge = (int) config('constants.ssh.mux_orphan_min_age'); + $socketOwners = []; foreach ($this->listProcesses() as $process) { // Backgrounded ssh master: current `ssh -fN` or legacy `ssh -fNM`. @@ -51,20 +58,44 @@ class CleanupStaleMultiplexedConnections implements ShouldQueue continue; } - if ($process['etimes'] >= $minAge - && ! file_exists($pathMatch[1]) - && ! SshMultiplexingHelper::isMuxProcessRetiring($process['pid'], $pathMatch[1])) { + $muxSocket = $pathMatch[1]; + if ($process['etimes'] < $minAge + || SshMultiplexingHelper::isMuxProcessRetiring($process['pid'], $muxSocket)) { + continue; + } + + $socketOwners[$muxSocket] ??= $this->muxSocketOwner($muxSocket); + if ($socketOwners[$muxSocket] !== false && $socketOwners[$muxSocket] !== $process['pid']) { $this->reapOrphan('ssh', $process); } } } + /** + * Pid of the master that owns a mux socket. Returns null when the socket + * file does not exist, and false when it exists but its owner is unknown. + */ + private function muxSocketOwner(string $muxSocket): string|false|null + { + if (! file_exists($muxSocket)) { + return null; + } + + $check = Process::run('timeout 10 ssh -O check -o ControlPath='.escapeshellarg($muxSocket).' localhost'); + + return preg_match('/pid=(\d+)/', $check->output().$check->errorOutput(), $matches) + ? $matches[1] + : false; + } + /** * Kill orphaned `cloudflared access ssh` proxy processes. Each is spawned * as the SSH ProxyCommand transport for a Cloudflare Tunnel server and must * die with its parent ssh. When that ssh is killed or orphaned (e.g. a lost * mux master), the cloudflared process can leak and accumulate. A legitimate - * proxy always has a live ssh parent; one without is safe to reap. + * proxy always has a live ssh at the other end of its stdin pipe. Its parent + * is not enough: `ssh -f` forks after connecting, so the proxy of a live + * master is reparented away from ssh. * * Processes younger than the minimum age are skipped so a proxy whose parent * ssh is still starting up, or a transient `ssh -O check` proxy mid-exit, is @@ -84,22 +115,49 @@ class CleanupStaleMultiplexedConnections implements ShouldQueue } foreach ($processes as $process) { - // `cloudflared access ssh`, never the `cloudflared tunnel` daemon. - if (! str_contains($process['args'], 'cloudflared access ssh')) { + // `cloudflared access ssh`, never the `cloudflared tunnel` daemon, nor + // an ssh whose ProxyCommand option contains the same words. + if (isset($sshPids[$process['pid']]) || ! str_contains($process['args'], 'cloudflared access ssh')) { continue; } - // Orphaned when no live ssh process is its parent. - if ($process['etimes'] >= $minAge && ! isset($sshPids[$process['ppid']])) { + if ($process['etimes'] >= $minAge + && ! isset($sshPids[$process['ppid']]) + && ! $this->stdinPipeHeldByAny($process['pid'], array_keys($sshPids))) { $this->reapOrphan('cloudflared', $process); } } } /** - * Reap a detected orphan process. When orphan reaping is disabled (the - * default), the orphan is only logged — a dry-run mode that lets operators - * verify what would be killed before enabling it for real. + * Whether one of the given processes holds the other end of a process's + * stdin pipe. An unreadable stdin counts as held, so it is never reaped. + * + * @param list $holderPids + */ + private function stdinPipeHeldByAny(string $pid, array $holderPids): bool + { + $stdin = @readlink("/proc/{$pid}/fd/0"); + if ($stdin === false || ! str_starts_with($stdin, 'pipe:')) { + return true; + } + + foreach ($holderPids as $holderPid) { + foreach (glob("/proc/{$holderPid}/fd/*") ?: [] as $fd) { + if (@readlink($fd) === $stdin) { + return true; + } + } + } + + return false; + } + + /** + * Reap a detected orphan process. When orphan reaping is disabled + * (SSH_MUX_ORPHAN_REAP_ENABLED=false, the default), the orphan is only + * logged — a dry-run mode that lets operators verify what would be killed + * before enabling it for real. * * @param array{pid: string, ppid: string, etimes: int, args: string} $process */ @@ -115,7 +173,10 @@ class CleanupStaleMultiplexedConnections implements ShouldQueue return; } - Process::run('kill '.escapeshellarg($process['pid'])); + // SIGKILL ssh: a former master unlinks its ControlPath on SIGTERM, even + // when a newer master owns that path now. + $signal = $kind === 'ssh' ? 'kill -9 ' : 'kill '; + Process::run($signal.escapeshellarg($process['pid'])); Log::info("Killed orphaned {$kind} process", [ 'pid' => $process['pid'], 'etimes' => $process['etimes'], @@ -195,8 +256,9 @@ class CleanupStaleMultiplexedConnections implements ShouldQueue /** * Close and delete a stale mux socket file. When orphan reaping is disabled - * (the default), the file is only logged — a dry-run mode that lets operators - * verify what would be removed before enabling it for real. + * (SSH_MUX_ORPHAN_REAP_ENABLED=false, the default), the file is only logged — + * a dry-run mode that lets operators verify what would be removed before + * enabling it for real. */ private function removeMultiplexFile(string $muxFile, string $reason): void { diff --git a/app/Jobs/DockerCleanupJob.php b/app/Jobs/DockerCleanupJob.php index ed430a600c..146f751b23 100644 --- a/app/Jobs/DockerCleanupJob.php +++ b/app/Jobs/DockerCleanupJob.php @@ -41,6 +41,8 @@ class DockerCleanupJob implements ShouldBeEncrypted, ShouldQueue public ?DockerCleanupExecution $execution_log = null; + public const INTERRUPTED_MESSAGE = 'Docker cleanup was interrupted: its worker stopped before the cleanup finished.'; + /** * Shares the per-server lock with queued CleanupDocker runs, so two cleanups never run on * one server at once. A scheduled run that finds the lock held marks its occurrence @@ -232,6 +234,40 @@ class DockerCleanupJob implements ShouldBeEncrypted, ShouldQueue return $jobUuid ? 'docker-cleanup-execution:'.$jobUuid : null; } + /** + * Fails running executions that are older than the server lock. The worker kills a cleanup at the + * job timeout, which is shorter than the lock, so such a run lost its worker before failed() ran + * (for example a killed container) and would otherwise stay running. + */ + public static function failInterruptedExecutions(): void + { + $executions = DockerCleanupExecution::query() + ->with('server.team') + ->where('status', 'running') + ->whereNull('finished_at') + ->where('created_at', '<', now()->subSeconds(CleanupDocker::overlapLockExpiresAfter())) + ->get(); + + foreach ($executions as $execution) { + $updated = DockerCleanupExecution::query() + ->whereKey($execution->id) + ->where('status', 'running') + ->whereNull('finished_at') + ->update([ + 'status' => 'failed', + 'message' => self::INTERRUPTED_MESSAGE, + 'finished_at' => Carbon::now()->toImmutable(), + ]); + + if ($updated === 0) { + continue; + } + + event(new DockerCleanupDone($execution->refresh())); + $execution->server?->team?->notify(new DockerCleanupFailed($execution->server, 'Docker cleanup job failed with the following error: '.self::INTERRUPTED_MESSAGE)); + } + } + public function failed(?\Throwable $exception): void { if ($this->occurrenceUuid) { diff --git a/app/Jobs/ProcessGithubPullRequestWebhook.php b/app/Jobs/ProcessGithubPullRequestWebhook.php index 362ad0a2d8..1602b576a2 100644 --- a/app/Jobs/ProcessGithubPullRequestWebhook.php +++ b/app/Jobs/ProcessGithubPullRequestWebhook.php @@ -14,6 +14,7 @@ use Illuminate\Contracts\Queue\ShouldQueue; use Illuminate\Foundation\Bus\Dispatchable; use Illuminate\Queue\InteractsWithQueue; use Illuminate\Queue\SerializesModels; +use Illuminate\Support\Facades\Log; use Throwable; class ProcessGithubPullRequestWebhook implements ShouldBeEncrypted, ShouldQueue @@ -37,7 +38,6 @@ class ProcessGithubPullRequestWebhook implements ShouldBeEncrypted, ShouldQueue public ?string $beforeSha, public ?string $afterSha, public string $commitSha, - public ?string $authorAssociation, public string $fullName, public bool $isForkPullRequest = false, ) { @@ -109,39 +109,46 @@ class ProcessGithubPullRequestWebhook implements ShouldBeEncrypted, ShouldQueue return; } - // Check if PR deployments from public contributors are restricted - if (! $application->settings->is_pr_deployments_public_enabled) { - // Fork PRs carry untrusted code from a repository outside our control. - // GitHub's author_association cannot be trusted to gate these (it grants - // CONTRIBUTOR to anyone who has merely opened an issue/PR before), so fork - // PRs are never deployed automatically when public previews are off. - if ($this->isForkPullRequest) { - return; - } + // Fork PRs carry untrusted code, so they deploy only when public previews are on. + // Same-repository branches can only be pushed by users with write access. + if ($this->isForkPullRequest && ! $application->settings->is_pr_deployments_public_enabled) { + Log::info('Skipped GitHub pull request preview deployment from a fork because public PR deployments are disabled.', [ + 'application_uuid' => $application->uuid, + 'pull_request_id' => $this->pullRequestId, + 'action' => $this->action, + ]); - // Same-repo (non-fork) branch PRs require push access to the base repo, - // so only trusted associations are allowed to trigger a deployment. - $trustedAssociations = ['OWNER', 'MEMBER', 'COLLABORATOR']; - if (! in_array($this->authorAssociation, $trustedAssociations)) { - return; - } + return; } - // Get changed files for watch path filtering - $changed_files = collect(); + // Get changed files for watch path filtering (null means the full list is unknown) + $changed_files = []; if ($this->action === 'synchronize' && $this->beforeSha && $this->afterSha) { // For synchronize events, get files changed between before and after commits - $changed_files = collect(getGithubCommitRangeFiles($githubApp, $owner, $repo, $this->beforeSha, $this->afterSha)); + $changed_files = getGithubCommitRangeFiles($githubApp, $owner, $repo, $this->beforeSha, $this->afterSha); } elseif ($this->action === 'opened' || $this->action === 'reopened') { // For opened/reopened events, get all files in the PR - $changed_files = collect(getGithubPullRequestFiles($githubApp, $owner, $repo, $this->pullRequestId)); + $changed_files = getGithubPullRequestFiles($githubApp, $owner, $repo, $this->pullRequestId); } // Apply watch path filtering - $is_watch_path_triggered = $application->isWatchPathsTriggered($changed_files); - if (! $is_watch_path_triggered && ! blank($application->watch_paths)) { - return; + if (! blank($application->watch_paths)) { + $logContext = [ + 'application_uuid' => $application->uuid, + 'pull_request_id' => $this->pullRequestId, + 'action' => $this->action, + ]; + + if (is_null($changed_files)) { + Log::warning('Could not get the full list of changed files from GitHub. Deploying the pull request preview without the watch paths check.', $logContext); + } elseif (! $application->isWatchPathsTriggered(collect($changed_files))) { + Log::info('Skipped GitHub pull request preview deployment because no changed file matches the watch paths.', $logContext + [ + 'changed_files_count' => count($changed_files), + ]); + + return; + } } // Create ApplicationPreview if not exists diff --git a/app/Jobs/PullTemplatesFromCDN.php b/app/Jobs/PullTemplatesFromCDN.php index ec09285e1e..2125581f50 100644 --- a/app/Jobs/PullTemplatesFromCDN.php +++ b/app/Jobs/PullTemplatesFromCDN.php @@ -44,13 +44,11 @@ class PullTemplatesFromCDN implements ShouldBeEncrypted, ShouldQueue 'status' => $response->status(), 'body' => str($response->body())->limit(500)->toString(), ]); - send_internal_notification('PullTemplatesAndVersions failed with: '.$response->status().' '.$response->body()); } } catch (\Throwable $e) { Log::error('PullTemplatesFromCDN exception', [ 'message' => $e->getMessage(), ]); - send_internal_notification('PullTemplatesAndVersions failed with: '.$e->getMessage()); } } } diff --git a/app/Jobs/ScheduledJobManager.php b/app/Jobs/ScheduledJobManager.php index c218a53652..7e137dadaa 100644 --- a/app/Jobs/ScheduledJobManager.php +++ b/app/Jobs/ScheduledJobManager.php @@ -191,6 +191,7 @@ class ScheduledJobManager implements ShouldQueue // Process Docker cleanups - don't let failures stop the job manager try { if ($this->includes('docker-cleanups')) { + DockerCleanupJob::failInterruptedExecutions(); $this->processDockerCleanups(); } } catch (\Throwable $e) { diff --git a/app/Jobs/ScheduledTaskJob.php b/app/Jobs/ScheduledTaskJob.php index 56c5bb7821..765e6f89d2 100644 --- a/app/Jobs/ScheduledTaskJob.php +++ b/app/Jobs/ScheduledTaskJob.php @@ -85,6 +85,17 @@ class ScheduledTaskJob implements ShouldBeEncrypted, ShouldQueue $this->executionUuid = new_public_id(); } + /** + * The lock outlives the worker timeout, so a run killed at the timeout still blocks a new run + * while its remote command can still be running. + */ + public function middleware(): array + { + $expireAfter = ($this->task->timeout ?? 300) + 300; + + return [ScheduledJobDeliveryService::withoutOverlapping('scheduled-task-'.$this->task->id, $this->occurrenceUuid)->expireAfter($expireAfter)->dontRelease()]; + } + private function initializeExecutionContext(): void { $this->task->loadMissing([ diff --git a/app/Jobs/UpdateCoolifyJob.php b/app/Jobs/UpdateCoolifyJob.php index f0e43cbc0a..e2310f0bee 100644 --- a/app/Jobs/UpdateCoolifyJob.php +++ b/app/Jobs/UpdateCoolifyJob.php @@ -44,8 +44,7 @@ class UpdateCoolifyJob implements ShouldBeEncrypted, ShouldQueue Log::info('Starting Coolify update process...'); UpdateCoolify::run(false); // false means it's not a manual update - $settings->update(['new_version_available' => false]); - Log::info('Coolify update completed successfully.'); + Log::info('Coolify upgrade script started. Result: /data/coolify/source/.upgrade-status, log: /data/coolify/source/upgrade-*.log.'); } catch (\Throwable $e) { Log::error('UpdateCoolifyJob failed: '.$e->getMessage()); // Consider implementing a notification to administrators diff --git a/app/Jobs/VolumeBackupJob.php b/app/Jobs/VolumeBackupJob.php index 9c36f67e53..6439942e1a 100644 --- a/app/Jobs/VolumeBackupJob.php +++ b/app/Jobs/VolumeBackupJob.php @@ -99,8 +99,7 @@ class VolumeBackupJob implements ShouldBeEncrypted, ShouldQueue ? 'docker volume inspect '.escapeshellarg($source).' >/dev/null' : 'test -d '.escapeshellarg($source); - $compressorCommand = BackupCompression::compressorCommand($compressionCpuPercentage); - $archiveScript = "compressor=\$({$compressorCommand}); tar -I \"\$compressor\" -cf - -C /volume ."; + $archiveScript = self::archiveScript($compressionCpuPercentage); if ($streamToS3) { $this->execution->update(['local_storage_deleted' => true]); $archiveCommand = $this->streamToS3Command($destinations->first(), $archiveScript, $backupLocation, $source, $containerName, $image); @@ -303,6 +302,17 @@ class VolumeBackupJob implements ShouldBeEncrypted, ShouldQueue )); } + /** + * GNU tar exits with 1 when a file changes while it is read (for example, an app that writes to the volume + * during a live backup). The archive is still valid, so only exit codes above 1 fail the backup. + */ + public static function archiveScript(int $compressionCpuPercentage): string + { + $compressorCommand = BackupCompression::compressorCommand($compressionCpuPercentage); + + return "compressor=\$({$compressorCommand}); { tar -I \"\$compressor\" -cf - -C /volume . || { status=\$?; [ \$status -eq 1 ] || exit \$status; }; }"; + } + /** * Redact the key and secret of every S3 destination of the backup from text that is stored or sent in notifications. */ @@ -323,12 +333,15 @@ class VolumeBackupJob implements ShouldBeEncrypted, ShouldQueue } /** + * Containers can exit and be removed (e.g. `--rm` backup helpers) between `docker ps` and + * `docker inspect`. Skip those, but still fail when an existing container cannot be inspected. + * * @return array */ private function containersUsingVolume(string $source, Server $server): array { $output = instant_remote_process( - ["containers=\$(docker ps -q) || exit 1; for container in \$containers; do paused=\$(docker inspect --format '{{.State.Paused}}' \"\$container\") || exit 1; [ \"\$paused\" = true ] && continue; mounts=\$(docker inspect --format '{{range .Mounts}}{{println .Source}}{{println .Name}}{{end}}' \"\$container\") || exit 1; if printf '%s\\n' \"\$mounts\" | grep -Fqx -- ".escapeshellarg($source).'; then echo "$container"; fi; done'], + ["containers=\$(docker ps -q) || exit 1; for container in \$containers; do mounts=\$(docker inspect --format '{{if not .State.Paused}}{{range .Mounts}}{{println .Source}}{{println .Name}}{{end}}{{end}}' \"\$container\" 2>/dev/null) || { remaining=\$(docker ps -aq --filter \"id=\$container\") || exit 1; [ -z \"\$remaining\" ] && continue; docker inspect \"\$container\" >/dev/null; exit 1; }; if printf '%s\\n' \"\$mounts\" | grep -Fqx -- ".escapeshellarg($source).'; then echo "$container"; fi; done'], $server, disableMultiplexing: true, ); @@ -449,9 +462,13 @@ class VolumeBackupJob implements ShouldBeEncrypted, ShouldQueue ->implode(' '); $resolveOptions = $resolveOptions === '' ? '' : ' '.$resolveOptions; $destination = 'temporary/'.$s3->bucket.$backupLocation; + // `mc pipe --quiet` panics in recent mc releases, so keep its progress bar out of the error output instead. + // Each progress frame starts with "\r", so printing the last field per line keeps only the final frame and real errors. + $mcPipeLog = '/tmp/mc-pipe.log'; $streamScript = 'set -o pipefail; mc alias set'.$resolveOptions.' temporary ' .escapeshellarg($s3->endpoint).' '.escapeshellarg($s3->key).' '.escapeshellarg($s3->secret) - .' >/dev/null && ('.$archiveScript.' | mc pipe --quiet'.$resolveOptions.' '.escapeshellarg($destination).' >/dev/null)' + .' >/dev/null && ('.$archiveScript.' | mc pipe'.$resolveOptions.' '.escapeshellarg($destination).' >/dev/null 2>'.$mcPipeLog + .' || { status=$?; awk -F '.escapeshellarg('\r').' '.escapeshellarg('{ print $NF }').' '.$mcPipeLog.' >&2; exit $status; })' .' && mc stat --json'.$resolveOptions.' '.escapeshellarg($destination) .' | sed -n '.escapeshellarg('s/.*"size":\([0-9][0-9]*\).*/\1/p'); diff --git a/app/Listeners/CloudflareTunnelChangedNotification.php b/app/Listeners/CloudflareTunnelChangedNotification.php index 34e713f954..3cc44c2f82 100644 --- a/app/Listeners/CloudflareTunnelChangedNotification.php +++ b/app/Listeners/CloudflareTunnelChangedNotification.php @@ -26,7 +26,7 @@ class CloudflareTunnelChangedNotification for ($i = 1; $i <= $attempts; $i++) { \Log::debug("Cloudflare health check attempt {$i}/{$attempts}", ['server_id' => $server_id]); - $result = instant_remote_process_with_timeout(['docker inspect coolify-cloudflared | jq -e ".[0].State.Health.Status == \"healthy\""'], $this->server, false, 10); + $result = instant_remote_process_with_timeout(['docker inspect coolify-cloudflared | jq -e ".[0].State.Health.Status == \"healthy\""'], $this->server, false); if (blank($result)) { \Log::debug("Cloudflare Tunnels container not found on attempt {$i}", ['server_id' => $server_id]); diff --git a/app/Livewire/Dashboard/ServerMetricsChart.php b/app/Livewire/Dashboard/ServerMetricsChart.php index fce9460064..83748052ce 100644 --- a/app/Livewire/Dashboard/ServerMetricsChart.php +++ b/app/Livewire/Dashboard/ServerMetricsChart.php @@ -4,14 +4,23 @@ namespace App\Livewire\Dashboard; use App\Models\Server; use Illuminate\Contracts\View\View; +use Livewire\Attributes\Isolate; use Livewire\Component; +/** + * Each chart sends its own request, so one slow server does not delay the charts of the other servers. + */ +#[Isolate] class ServerMetricsChart extends Component { public Server $server; public function loadData(): void { + if (! $this->server->isFunctional()) { + return; + } + try { $this->dispatch("dashboard-server-metrics-{$this->server->uuid}", [ 'cpu' => $this->server->getCpuMetrics(10), diff --git a/app/Livewire/Project/New/GithubPrivateRepositoryDeployKey.php b/app/Livewire/Project/New/GithubPrivateRepositoryDeployKey.php index d55972e97d..5a777d7a59 100644 --- a/app/Livewire/Project/New/GithubPrivateRepositoryDeployKey.php +++ b/app/Livewire/Project/New/GithubPrivateRepositoryDeployKey.php @@ -248,6 +248,15 @@ class GithubPrivateRepositoryDeployKey extends Component return; } + // Keep ssh:// URLs as-is so a custom SSH port is preserved for deployments. + if (str_starts_with($this->repository_url, 'ssh://')) { + $this->git_host = parse_url($this->repository_url, PHP_URL_HOST); + $this->git_repository = $this->repository_url; + $this->git_source = 'other'; + + return; + } + $this->repository_url_parsed = Url::fromString($this->repository_url); $this->git_host = $this->repository_url_parsed->getHost(); $this->git_repository = $this->repository_url_parsed->getSegment(1).'/'.$this->repository_url_parsed->getSegment(2); diff --git a/app/Livewire/Project/New/PublicGitRepository.php b/app/Livewire/Project/New/PublicGitRepository.php index c14c9dcd37..a898204fc2 100644 --- a/app/Livewire/Project/New/PublicGitRepository.php +++ b/app/Livewire/Project/New/PublicGitRepository.php @@ -139,7 +139,7 @@ class PublicGitRepository extends Component throw new \RuntimeException('Invalid repository URL: '.$validator->errors()->first('repository_url')); } - $httpsRepositoryUrl = scpStyleGitUrlToHttps($this->repository_url); + $httpsRepositoryUrl = sshGitUrlToHttps($this->repository_url); if (is_string($httpsRepositoryUrl)) { $this->repository_url = $httpsRepositoryUrl; } diff --git a/app/Livewire/Project/New/Select.php b/app/Livewire/Project/New/Select.php index 9986bfa1f5..2549cf5239 100644 --- a/app/Livewire/Project/New/Select.php +++ b/app/Livewire/Project/New/Select.php @@ -118,6 +118,7 @@ class Select extends Component 'name' => str($serviceKey)->headline(), 'docsSlug' => str($serviceKey)->lower()->value(), 'templateLastUpdated' => $templateLastUpdatedMap[$serviceKey] ?? null, + 'website' => $this->serviceWebsiteUrl(data_get($service, 'documentation')), ] + service_logo_urls(data_get($service, 'logo')) + (array) $service; })->all(); @@ -288,6 +289,29 @@ class Select extends Component } } + /** + * Derive a website link from the documentation URL. Shared hosts keep the + * path that identifies the project, so the link does not open the host root. + */ + private function serviceWebsiteUrl(?string $documentation): ?string + { + $url = parse_url((string) $documentation); + if (! in_array($url['scheme'] ?? null, ['http', 'https'], true) || blank($url['host'] ?? null)) { + return null; + } + + $projectPathSegments = [ + 'github.com' => 2, + 'gitlab.com' => 2, + 'codeberg.org' => 2, + 'hub.docker.com' => 3, + ][str($url['host'])->lower()->chopStart('www.')->value()] ?? 0; + + $projectPath = str($url['path'] ?? '')->explode('/')->filter()->take($projectPathSegments)->implode('/'); + + return rtrim("{$url['scheme']}://{$url['host']}/{$projectPath}", '/'); + } + private function serviceTemplatesLastUpdated(): ?string { $fetchedAt = get_service_templates_fetched_at(); diff --git a/app/Livewire/Project/Shared/EnvironmentVariable/All.php b/app/Livewire/Project/Shared/EnvironmentVariable/All.php index daad018b2b..736075dc7c 100644 --- a/app/Livewire/Project/Shared/EnvironmentVariable/All.php +++ b/app/Livewire/Project/Shared/EnvironmentVariable/All.php @@ -57,6 +57,8 @@ class All extends Component public bool $use_build_secrets = false; + public bool $include_coolify_default_variables = true; + public bool $redact_all_env_values_in_logs = true; /** @@ -111,6 +113,8 @@ class All extends Component { $this->is_env_sorting_enabled = data_get($this->resource, 'settings.is_env_sorting_enabled', false); $this->use_build_secrets = data_get($this->resource, 'settings.use_build_secrets', false); + $coolifyDefaultVariablesOwner = $this->resource instanceof Application ? $this->resource->settings : $this->resource; + $this->include_coolify_default_variables = data_get($coolifyDefaultVariablesOwner, 'include_coolify_default_variables') !== false; if ($this->hasLogRedactionSetting()) { $this->redact_all_env_values_in_logs = $this->resource->redactsAllEnvValuesInLogs(); } @@ -201,6 +205,25 @@ class All extends Component } } + public function saveCoolifyDefaultVariables() + { + try { + $this->authorize('manageEnvironment', $this->resource); + + if ($this->resource instanceof Application) { + $this->resource->settings->include_coolify_default_variables = $this->include_coolify_default_variables; + $this->saveApplicationSettingsWithAudit($this->resource); + } else { + $this->resource->include_coolify_default_variables = $this->include_coolify_default_variables; + $this->resource->save(); + } + $this->dispatch('success', 'Environment variable settings updated.'); + $this->dispatch('configurationChanged'); + } catch (\Throwable $e) { + return handleError($e, $this); + } + } + public function getEnvironmentVariablesProperty() { if (! $this->readyToLoad) { diff --git a/app/Livewire/Project/Shared/ScheduledTask/Executions.php b/app/Livewire/Project/Shared/ScheduledTask/Executions.php index 76e0735ff4..f7e5ca1d1f 100644 --- a/app/Livewire/Project/Shared/ScheduledTask/Executions.php +++ b/app/Livewire/Project/Shared/ScheduledTask/Executions.php @@ -6,6 +6,7 @@ use App\Models\ScheduledTask; use App\Traits\ListensToTeamChannel; use Illuminate\Support\Collection; use Illuminate\Support\Facades\Auth; +use Livewire\Attributes\Computed; use Livewire\Attributes\Locked; use Livewire\Component; @@ -13,9 +14,6 @@ class Executions extends Component { use ListensToTeamChannel; - #[Locked] - public ScheduledTask $task; - #[Locked] public int $taskId; @@ -48,11 +46,11 @@ class Executions extends Component { try { $this->taskId = $taskId; - $this->task = ScheduledTask::where('team_id', Auth::user()->currentTeam()->id)->findOrFail($taskId); - $this->executions = $this->task->executions()->take(20)->get(); - $this->serverTimezone = data_get($this->task, 'application.destination.server.settings.server_timezone'); + $task = ScheduledTask::where('team_id', Auth::user()->currentTeam()->id)->findOrFail($taskId); + $this->executions = $task->executions()->take(20)->get(); + $this->serverTimezone = data_get($task, 'application.destination.server.settings.server_timezone'); if (! $this->serverTimezone) { - $this->serverTimezone = data_get($this->task, 'service.destination.server.settings.server_timezone'); + $this->serverTimezone = data_get($task, 'service.destination.server.settings.server_timezone'); } if (! $this->serverTimezone) { $this->serverTimezone = 'UTC'; @@ -62,8 +60,22 @@ class Executions extends Component } } + /** + * Load the task on each request instead of storing it as a model property. A stored model fails + * with a 404 when the poll runs after the task is deleted and before the page navigates away. + */ + #[Computed] + public function task(): ?ScheduledTask + { + return ScheduledTask::find($this->taskId); + } + public function refreshExecutions(): void { + if (! $this->task) { + return; + } + $this->executions = $this->task->executions()->take(20)->get(); if ($this->selectedKey) { $this->selectedExecution = $this->task->executions()->find($this->selectedKey); @@ -84,7 +96,7 @@ class Executions extends Component return; } $this->selectedKey = $key; - $this->selectedExecution = $this->task->executions()->find($key); + $this->selectedExecution = $this->task?->executions()->find($key); $this->currentPage = 1; // Start polling if task is running @@ -157,7 +169,9 @@ class Executions extends Component */ public function canReadOutput(): bool { - return auth()->user()?->can('update', $this->task->application ?? $this->task->service) ?? false; + $resource = $this->task?->application ?? $this->task?->service; + + return $resource && auth()->user()?->can('update', $resource); } public function hasMoreLogs() diff --git a/app/Livewire/Project/Shared/ScheduledTask/Show.php b/app/Livewire/Project/Shared/ScheduledTask/Show.php index 397ecf4710..5f5689b5de 100644 --- a/app/Livewire/Project/Shared/ScheduledTask/Show.php +++ b/app/Livewire/Project/Shared/ScheduledTask/Show.php @@ -147,6 +147,9 @@ class Show extends Component auditLog('ui.scheduled_task.deleted', $this->taskAuditContext()); $this->dispatch('success', 'Scheduled task deleted.'); $this->dispatch('scheduledTaskDeleted')->to(All::class); + + // Without true, the confirmation modal refreshes this deleted task. + return true; } catch (\Exception $e) { return handleError($e); } diff --git a/app/Livewire/Project/Shared/Webhooks.php b/app/Livewire/Project/Shared/Webhooks.php index 172d91fc5e..4a21654838 100644 --- a/app/Livewire/Project/Shared/Webhooks.php +++ b/app/Livewire/Project/Shared/Webhooks.php @@ -2,11 +2,14 @@ namespace App\Livewire\Project\Shared; +use App\Models\Application; +use App\Models\GitlabApp; use App\Rules\ManualWebhookSecret; use Illuminate\Foundation\Auth\Access\AuthorizesRequests; use Illuminate\Support\Str; use Illuminate\Validation\ValidationException; use Livewire\Component; +use Spatie\Url\Url; // Refactored ✅ class Webhooks extends Component @@ -25,6 +28,12 @@ class Webhooks extends Component public ?string $giteaManualWebhook; + /** + * Source-level endpoint for GitLab App sources. GitLab has no app-level webhooks, + * so users add this URL to each GitLab project themselves. + */ + public ?string $gitlabAppWebhook = null; + public ?string $githubManualWebhookSecret = null; public ?string $gitlabManualWebhookSecret = null; @@ -33,6 +42,14 @@ class Webhooks extends Component public ?string $giteaManualWebhookSecret = null; + /** + * Stored secret per component property. The unsaved bar compares the fields + * with these values, because Livewire syncs unsaved field edits with every request. + * + * @var array + */ + public array $savedSecrets = []; + /** * Maps each manual webhook provider to its component property and model attribute. * @@ -54,12 +71,17 @@ class Webhooks extends Component $this->gitlabManualWebhookSecret = data_get($this->resource, 'manual_webhook_secret_gitlab'); $this->bitbucketManualWebhookSecret = data_get($this->resource, 'manual_webhook_secret_bitbucket'); $this->giteaManualWebhookSecret = data_get($this->resource, 'manual_webhook_secret_gitea'); + $this->syncSavedSecrets(); } $this->githubManualWebhook = generateGitManualWebhook($this->resource, 'github'); $this->gitlabManualWebhook = generateGitManualWebhook($this->resource, 'gitlab'); $this->bitbucketManualWebhook = generateGitManualWebhook($this->resource, 'bitbucket'); $this->giteaManualWebhook = generateGitManualWebhook($this->resource, 'gitea'); + + if ($this->resource instanceof Application && $this->resource->source instanceof GitlabApp) { + $this->gitlabAppWebhook = Url::fromString(base_url()).'/webhooks/source/gitlab/events'; + } } public function canViewSecrets(): bool @@ -78,6 +100,7 @@ class Webhooks extends Component 'manual_webhook_secret_bitbucket' => $this->bitbucketManualWebhookSecret, 'manual_webhook_secret_gitea' => $this->giteaManualWebhookSecret, ]); + $this->syncSavedSecrets(); $this->dispatch('success', 'Webhook secrets saved.'); } catch (ValidationException $e) { throw $e; @@ -87,10 +110,10 @@ class Webhooks extends Component } /** - * Return a random secret for the provider's field. The browser fills the field - * as an unsaved change, and the user saves it with the form. + * Generate a random secret for the provider and save it immediately. + * Other unsaved secret edits in the form are kept as they are. */ - public function generateSecret(string $provider): ?string + public function generateSecret(string $provider): void { try { $this->authorize('update', $this->resource); @@ -99,11 +122,21 @@ class Webhooks extends Component throw new \InvalidArgumentException('Unknown webhook provider.'); } - return Str::random(40); + $field = self::SECRET_FIELDS[$provider]; + $secret = Str::random(40); + $this->resource->update([$field['attribute'] => $secret]); + $this->{$field['property']} = $secret; + $this->savedSecrets[$field['property']] = $secret; + $this->dispatch('success', "New {$field['label']} generated and saved."); } catch (\Exception $e) { handleError($e, $this); + } + } - return null; + private function syncSavedSecrets(): void + { + foreach (self::SECRET_FIELDS as $field) { + $this->savedSecrets[$field['property']] = data_get($this->resource, $field['attribute']); } } diff --git a/app/Livewire/Security/ApiTokens.php b/app/Livewire/Security/ApiTokens.php index bd7189bd24..0a255370e5 100644 --- a/app/Livewire/Security/ApiTokens.php +++ b/app/Livewire/Security/ApiTokens.php @@ -2,7 +2,6 @@ namespace App\Livewire\Security; -use App\Models\InstanceSettings; use Illuminate\Foundation\Auth\Access\AuthorizesRequests; use Illuminate\Validation\Rule; use Laravel\Sanctum\PersonalAccessToken; @@ -35,10 +34,6 @@ class ApiTokens extends Component 365 => '1 year', ]; - public $isApiEnabled; - - public $isMcpEnabled; - #[Locked] public bool $canUseRootPermissions = false; @@ -58,9 +53,6 @@ class ApiTokens extends Component public function mount() { - $settings = InstanceSettings::get(); - $this->isApiEnabled = $settings->is_api_enabled; - $this->isMcpEnabled = $settings->is_mcp_server_enabled; $this->canUseRootPermissions = auth()->user()->can('useRootPermissions', PersonalAccessToken::class); $this->canUseWritePermissions = auth()->user()->can('useWritePermissions', PersonalAccessToken::class); $this->canUseDeployPermissions = auth()->user()->can('useDeployPermissions', PersonalAccessToken::class); diff --git a/app/Livewire/Settings/Index.php b/app/Livewire/Settings/Index.php index 75826f2d4e..cd047b1f43 100644 --- a/app/Livewire/Settings/Index.php +++ b/app/Livewire/Settings/Index.php @@ -201,6 +201,31 @@ class Index extends Component } } + /** + * Look up the public IPs of this host again and save them. + */ + public function detectPublicIps() + { + try { + $this->authorize('update', $this->settings); + + $detected = detect_public_ips(); + if ($detected === []) { + $this->dispatch('error', 'Could not detect a public IP address.'); + + return; + } + + $this->settings->fill($detected); + $this->saveAndAudit(); + $this->public_ipv4 = $this->settings->public_ipv4; + $this->public_ipv6 = $this->settings->public_ipv6; + $this->dispatch('success', 'Detected '.implode(' and ', $detected).'.'); + } catch (\Throwable $e) { + return handleError($e, $this); + } + } + public function buildHelperImage() { try { diff --git a/app/Livewire/Settings/Updates.php b/app/Livewire/Settings/Updates.php index 10b3180640..aba7521dd9 100644 --- a/app/Livewire/Settings/Updates.php +++ b/app/Livewire/Settings/Updates.php @@ -130,7 +130,7 @@ class Updates extends Component if ($this->is_auto_update_enabled && ! validate_cron_expression($this->auto_update_frequency)) { $this->dispatch('error', 'Invalid Cron / Human expression for Auto Update Frequency.'); if (empty($this->auto_update_frequency)) { - $this->auto_update_frequency = '0 0 * * *'; + $this->auto_update_frequency = InstanceSettings::randomAutoUpdateFrequency(); } return; diff --git a/app/Models/Application.php b/app/Models/Application.php index fc9085752c..a2dc1d6f62 100644 --- a/app/Models/Application.php +++ b/app/Models/Application.php @@ -790,7 +790,7 @@ class Application extends BaseModel return "{$this->source->html_url}/{$this->git_repository}/tree/{$this->git_branch}{$base_dir}"; } - $httpsRepository = $this->httpsUrlFromScpStyleGitRepository(); + $httpsRepository = $this->httpsUrlFromSshGitRepository(); if (is_string($httpsRepository)) { if (str($this->git_repository)->contains('bitbucket')) { return "{$httpsRepository}/src/{$this->git_branch}{$base_dir}"; @@ -811,7 +811,7 @@ class Application extends BaseModel if (! is_null($this->source?->html_url) && ! is_null($this->git_repository) && ! is_null($this->git_branch)) { return "{$this->source->html_url}/{$this->git_repository}/settings/hooks"; } - $httpsRepository = $this->httpsUrlFromScpStyleGitRepository(); + $httpsRepository = $this->httpsUrlFromSshGitRepository(); if (is_string($httpsRepository)) { return "{$httpsRepository}/settings/hooks"; } @@ -828,7 +828,7 @@ class Application extends BaseModel if (! is_null($this->source?->html_url) && ! is_null($this->git_repository) && ! is_null($this->git_branch)) { return "{$this->source->html_url}/{$this->git_repository}/commits/{$this->git_branch}"; } - $httpsRepository = $this->httpsUrlFromScpStyleGitRepository(); + $httpsRepository = $this->httpsUrlFromSshGitRepository(); if (is_string($httpsRepository)) { return "{$httpsRepository}/commits/{$this->git_branch}"; } @@ -849,11 +849,9 @@ class Application extends BaseModel } $git_repository = $this->git_repository; - $httpsRepository = scpStyleGitUrlToHttps($git_repository); + $httpsRepository = sshGitUrlToHttps($git_repository); if (is_string($httpsRepository)) { $git_repository = $httpsRepository; - } elseif (str($this->git_repository)->startsWith('ssh://')) { - $git_repository = 'https://'.parse_url($git_repository, PHP_URL_HOST).parse_url($git_repository, PHP_URL_PATH); } if (! filter_var($git_repository, FILTER_VALIDATE_URL) || ! in_array(parse_url($git_repository, PHP_URL_SCHEME), ['http', 'https'], true)) { @@ -868,9 +866,9 @@ class Application extends BaseModel return $url->__toString(); } - private function httpsUrlFromScpStyleGitRepository(): ?string + private function httpsUrlFromSshGitRepository(): ?string { - $httpsRepository = scpStyleGitUrlToHttps($this->git_repository); + $httpsRepository = sshGitUrlToHttps($this->git_repository); if (! is_string($httpsRepository)) { return null; @@ -1032,11 +1030,11 @@ class Application extends BaseModel $additional_servers_status = $this->additional_servers->pluck('pivot.status'); foreach ($additional_servers_status as $status) { $server_status = str($status)->before(':')->value(); - $server_health = str($status)->after(':')->value() ?? 'unhealthy'; + $server_health = str($status)->contains(':') ? str($status)->after(':')->value() : null; if ($main_server_status !== $server_status) { $complex_status = 'degraded'; } - if ($main_server_health !== $server_health) { + if ($server_health !== null && $main_server_health !== $server_health) { $complex_health = 'unhealthy'; } } @@ -2193,7 +2191,7 @@ class Application extends BaseModel continue; } if ($source->startsWith('.')) { - $source = str($workdir.$source->after('.')); + $source = replaceLocalSource($source, str($workdir)); } $mkdirCommand = rawComposeBindMkdirCommand($source->value()); if ($mkdirCommand !== null) { diff --git a/app/Models/ApplicationPreview.php b/app/Models/ApplicationPreview.php index 4a0f6d78f0..17b791d733 100644 --- a/app/Models/ApplicationPreview.php +++ b/app/Models/ApplicationPreview.php @@ -54,10 +54,13 @@ class ApplicationPreview extends BaseModel if (data_get($preview, 'application.build_pack') === 'dockercompose') { $composeFile = $application->parse(pull_request_id: $preview->pull_request_id); - foreach ($preview->generatedComposeVolumeNames($application, collect(data_get($composeFile, 'volumes', []))) as $volumeName) { + $volumeNames = $preview->generatedComposeVolumeNames($application, collect(data_get($composeFile, 'volumes', []))); + foreach ($volumeNames as $volumeName) { // Docker does not remove a volume that a container still uses, also with -f. instant_remote_process(['docker volume rm -f '.escapeshellarg($volumeName)], $server, false); } + // The compose parser stores preview volumes on the application, not on the preview + $application->persistentStorages()->whereIn('name', $volumeNames)->delete(); $preview->removeComposePreviewNetwork($application, $server); } else { // Regular application volume cleanup diff --git a/app/Models/ApplicationSetting.php b/app/Models/ApplicationSetting.php index 41249a5201..abe317530f 100644 --- a/app/Models/ApplicationSetting.php +++ b/app/Models/ApplicationSetting.php @@ -36,6 +36,7 @@ use OpenApi\Attributes as OA; 'custom_container_name_prefix' => ['type' => 'string', 'nullable' => true], 'is_container_label_escape_enabled' => ['type' => 'boolean'], 'is_env_sorting_enabled' => ['type' => 'boolean'], + 'include_coolify_default_variables' => ['type' => 'boolean'], 'is_container_label_readonly_enabled' => ['type' => 'boolean'], 'is_preserve_repository_enabled' => ['type' => 'boolean'], 'disable_build_cache' => ['type' => 'boolean'], @@ -88,6 +89,7 @@ class ApplicationSetting extends Model 'is_stripprefix_enabled' => 'boolean', 'connect_to_docker_network' => 'boolean', 'is_env_sorting_enabled' => 'boolean', + 'include_coolify_default_variables' => 'boolean', 'redact_all_env_values_in_logs' => 'boolean', 'disable_build_cache' => 'boolean', ]; @@ -120,6 +122,7 @@ class ApplicationSetting extends Model 'custom_container_name_prefix', 'is_container_label_escape_enabled', 'is_env_sorting_enabled', + 'include_coolify_default_variables', 'is_container_label_readonly_enabled', 'is_preserve_repository_enabled', 'disable_build_cache', diff --git a/app/Models/GithubApp.php b/app/Models/GithubApp.php index d0964f53a8..49e32e0b85 100644 --- a/app/Models/GithubApp.php +++ b/app/Models/GithubApp.php @@ -57,6 +57,34 @@ class GithubApp extends BaseModel 'webhook_secret', ]; + /** + * Installation token kept in memory, so one job does not request a new token for each GitHub call. + * + * @var array{credentials: string, token: string, expires_at: int}|null + */ + private ?array $installationTokenInMemory = null; + + public function installationToken(): string + { + $credentials = implode('|', [$this->api_url, $this->app_id, $this->installation_id, $this->private_key_id]); + $cached = $this->installationTokenInMemory; + + if ($cached && $cached['credentials'] === $credentials && $cached['expires_at'] > now()->getTimestamp()) { + return $cached['token']; + } + + $token = generateGithubToken($this, 'installation'); + + // GitHub installation tokens expire after one hour; renew them well before that. + $this->installationTokenInMemory = [ + 'credentials' => $credentials, + 'token' => $token, + 'expires_at' => now()->addMinutes(50)->getTimestamp(), + ]; + + return $token; + } + protected static function booted(): void { static::deleting(function (GithubApp $github_app) { diff --git a/app/Models/InstanceSettings.php b/app/Models/InstanceSettings.php index 09b9971c6d..8726a9f0f6 100644 --- a/app/Models/InstanceSettings.php +++ b/app/Models/InstanceSettings.php @@ -109,6 +109,12 @@ class InstanceSettings extends Model protected static function booted(): void { + static::creating(function ($settings) { + if (blank($settings->getAttributes()['auto_update_frequency'] ?? null)) { + $settings->auto_update_frequency = self::randomAutoUpdateFrequency(); + } + }); + static::created(function () { Once::flush(); }); @@ -126,6 +132,14 @@ class InstanceSettings extends Model }); } + /** + * Daily auto update at a random minute after midnight, so instances do not all pull images from the registry in the same second. + */ + public static function randomAutoUpdateFrequency(): string + { + return random_int(0, 59).' 0 * * *'; + } + public function isPasswordRegistrationAllowed(): bool { if (! $this->is_registration_enabled) { diff --git a/app/Models/Server.php b/app/Models/Server.php index 7fa3c12f0a..c1cebd7287 100644 --- a/app/Models/Server.php +++ b/app/Models/Server.php @@ -11,6 +11,7 @@ use App\Enums\ApplicationDeploymentStatus; use App\Enums\ProxyTypes; use App\Enums\ServerRole; use App\Events\ServerReachabilityChanged; +use App\Helpers\SshMultiplexingHelper; use App\Helpers\SslHelper; use App\Jobs\CheckAndStartSentinelJob; use App\Jobs\CheckTraefikVersionForServerJob; @@ -189,6 +190,9 @@ class Server extends BaseModel static::saved(function ($server) { if ($server->wasChanged('private_key_id') || $server->privateKey?->isDirty()) { refresh_server_connection($server->privateKey); + } elseif ($server->wasChanged(['user', 'ip', 'port'])) { + // An open multiplexed connection keeps the old user and host until it is closed. + SshMultiplexingHelper::removeMuxFile($server); } }); static::created(function ($server) { @@ -2060,25 +2064,34 @@ $siteAddress { public function validateOS(): bool|Stringable { $os_release = instant_remote_process(['cat /etc/os-release'], $this); - $releaseLines = collect(explode("\n", $os_release)); - $collectedData = collect([]); - foreach ($releaseLines as $line) { - $item = str($line)->trim(); - $collectedData->put($item->before('=')->value(), $item->after('=')->lower()->replace('"', '')->value()); - } - $ID = data_get($collectedData, 'ID'); - // $ID_LIKE = data_get($collectedData, 'ID_LIKE'); - // $VERSION_ID = data_get($collectedData, 'VERSION_ID'); - $supported = collect(SUPPORTED_OS)->filter(function ($supportedOs) use ($ID) { - if (str($supportedOs)->contains($ID)) { - return str($ID); + + return self::supportedOsFromOsRelease($os_release); + } + + /** + * Match /etc/os-release against SUPPORTED_OS: ID first, then each ID_LIKE token (e.g. Manjaro: ID=manjaro, ID_LIKE=arch). + */ + public static function supportedOsFromOsRelease(?string $osRelease): bool|Stringable + { + $collectedData = collect(explode("\n", (string) $osRelease)) + ->mapWithKeys(function ($line) { + $item = str($line)->trim(); + + return [$item->before('=')->value() => $item->after('=')->lower()->replace(['"', "'"], '')->value()]; + }); + + $candidates = collect([data_get($collectedData, 'ID')]) + ->merge(explode(' ', (string) data_get($collectedData, 'ID_LIKE'))) + ->filter(); + + foreach ($candidates as $candidate) { + $supported = collect(SUPPORTED_OS)->filter(fn ($supportedOs) => str($supportedOs)->contains($candidate)); + if ($supported->count() === 1) { + return str($supported->first()); } - }); - if ($supported->count() === 1) { - return str($supported->first()); - } else { - return false; } + + return false; } public function gatherServerMetadata(): ?array diff --git a/app/Models/Service.php b/app/Models/Service.php index 1f413491bd..2ebff2ad9c 100644 --- a/app/Models/Service.php +++ b/app/Models/Service.php @@ -71,6 +71,7 @@ class Service extends BaseModel protected $appends = ['server_status', 'status']; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'redact_all_env_values_in_logs' => 'boolean', ]; @@ -753,7 +754,7 @@ class Service extends BaseModel ]); } $fields->put('Supabase', $data->toArray()); - case $image->contains('minio'): + case $image->contains(['minio', 'pgsty/silo']): $data = collect([]); $console_url = $this->environment_variables()->where('key', 'MINIO_BROWSER_REDIRECT_URL')->first(); $s3_api_url = $this->environment_variables()->where('key', 'MINIO_SERVER_URL')->first(); diff --git a/app/Models/StandaloneClickhouse.php b/app/Models/StandaloneClickhouse.php index 147dab9832..eeaa59c97c 100644 --- a/app/Models/StandaloneClickhouse.php +++ b/app/Models/StandaloneClickhouse.php @@ -74,6 +74,7 @@ class StandaloneClickhouse extends BaseModel ]; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Models/StandaloneDragonfly.php b/app/Models/StandaloneDragonfly.php index 7de46038c6..23d7131244 100644 --- a/app/Models/StandaloneDragonfly.php +++ b/app/Models/StandaloneDragonfly.php @@ -70,6 +70,7 @@ class StandaloneDragonfly extends BaseModel ]; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Models/StandaloneKeydb.php b/app/Models/StandaloneKeydb.php index 3257a9c108..d2aa105bd3 100644 --- a/app/Models/StandaloneKeydb.php +++ b/app/Models/StandaloneKeydb.php @@ -71,6 +71,7 @@ class StandaloneKeydb extends BaseModel ]; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Models/StandaloneMariadb.php b/app/Models/StandaloneMariadb.php index ea3bc39ce4..be5c464e0d 100644 --- a/app/Models/StandaloneMariadb.php +++ b/app/Models/StandaloneMariadb.php @@ -73,6 +73,7 @@ class StandaloneMariadb extends BaseModel ]; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Models/StandaloneMongodb.php b/app/Models/StandaloneMongodb.php index 41bf67678b..208b5e917a 100644 --- a/app/Models/StandaloneMongodb.php +++ b/app/Models/StandaloneMongodb.php @@ -72,6 +72,7 @@ class StandaloneMongodb extends BaseModel ]; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Models/StandaloneMysql.php b/app/Models/StandaloneMysql.php index 2593bafa32..68a73ae573 100644 --- a/app/Models/StandaloneMysql.php +++ b/app/Models/StandaloneMysql.php @@ -74,6 +74,7 @@ class StandaloneMysql extends BaseModel ]; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Models/StandalonePostgresql.php b/app/Models/StandalonePostgresql.php index 9e6f66c958..7829ab0d0e 100644 --- a/app/Models/StandalonePostgresql.php +++ b/app/Models/StandalonePostgresql.php @@ -76,6 +76,7 @@ class StandalonePostgresql extends BaseModel ]; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Models/StandaloneRedis.php b/app/Models/StandaloneRedis.php index 7fe86c9974..d3c8004ae3 100644 --- a/app/Models/StandaloneRedis.php +++ b/app/Models/StandaloneRedis.php @@ -72,6 +72,7 @@ class StandaloneRedis extends BaseModel ]; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Models/StandaloneSqlite.php b/app/Models/StandaloneSqlite.php index e3490c1031..2f6993865c 100644 --- a/app/Models/StandaloneSqlite.php +++ b/app/Models/StandaloneSqlite.php @@ -60,6 +60,7 @@ class StandaloneSqlite extends BaseModel protected $appends = ['database_type', 'server_status']; protected $casts = [ + 'include_coolify_default_variables' => 'boolean', 'health_check_enabled' => 'boolean', 'health_check_interval' => 'integer', 'health_check_timeout' => 'integer', diff --git a/app/Rules/ValidGitRepositoryUrl.php b/app/Rules/ValidGitRepositoryUrl.php index 9a04409046..8b2d484e54 100644 --- a/app/Rules/ValidGitRepositoryUrl.php +++ b/app/Rules/ValidGitRepositoryUrl.php @@ -29,7 +29,7 @@ class ValidGitRepositoryUrl implements ValidationRule // Brackets are only allowed around an IPv6 host, so check the rest of the URL without it. $valueWithoutIpv6Host = preg_replace_callback( - '/^((?:https?|git):\/\/|[a-zA-Z0-9._-]+@)\[([0-9A-Fa-f:.]+)\]/', + '/^((?:https?|git):\/\/|(?:ssh:\/\/)?[a-zA-Z0-9._-]+@)\[([0-9A-Fa-f:.]+)\]/', fn (array $matches): string => filter_var($matches[2], FILTER_VALIDATE_IP, FILTER_FLAG_IPV6) !== false ? $matches[1].'ipv6-host' : $matches[0], $value, ); @@ -96,6 +96,19 @@ class ValidGitRepositoryUrl implements ValidationRule if ($scp === null || preg_match('/^(\[[0-9A-Fa-f:.]+\]|[a-zA-Z0-9.-]+)$/', $scp['host']) !== 1 || preg_match('/^[a-zA-Z0-9\-_\/.~]+$/', $scp['path']) !== 1) { $fail('The :attribute is not a valid SSH repository URL.'); + return; + } + } elseif (str_starts_with($value, 'ssh://')) { + if (! $this->allowSSH) { + $fail('SSH URLs are not allowed.'); + + return; + } + + // Validate ssh:// URL format (ssh://user@host[:port]/user/repo.git) + if (! preg_match('/^ssh:\/\/[a-zA-Z0-9._-]+@(\[[0-9A-Fa-f:.]+\]|[a-zA-Z0-9.-]+)(:[0-9]{1,5})?\/[a-zA-Z0-9\-_\/.~]+$/', $value)) { + $fail('The :attribute is not a valid SSH repository URL.'); + return; } } elseif (str_starts_with($value, 'http://') || str_starts_with($value, 'https://')) { @@ -160,7 +173,7 @@ class ValidGitRepositoryUrl implements ValidationRule return; } } else { - $fail('The :attribute must start with https://, http://, git://, or be an SSH URL (user@host:path).'); + $fail('The :attribute must start with https://, http://, git://, ssh://, or be an SSH URL (user@host:path).'); return; } diff --git a/app/Services/ScheduledJobDeliveryService.php b/app/Services/ScheduledJobDeliveryService.php index 0d5ddd633a..b2dc7446ea 100644 --- a/app/Services/ScheduledJobDeliveryService.php +++ b/app/Services/ScheduledJobDeliveryService.php @@ -627,8 +627,8 @@ class ScheduledJobDeliveryService return false; } - // An open previous occurrence usually means the previous run is still running. Backup jobs - // then skip this one without running it (see withoutOverlapping()). + // An open previous occurrence usually means the previous run is still running. Backup and + // task jobs then skip this one without running it (see withoutOverlapping()). $previousOpen = ScheduledJobDelivery::query() ->where('schedule_key', $occurrence->schedule_key) ->where('scheduled_for', '<', $occurrence->scheduled_for) diff --git a/app/Services/ServerTransfer/ServerTransferExporter.php b/app/Services/ServerTransfer/ServerTransferExporter.php index b93c9f9ccc..66895c4209 100644 --- a/app/Services/ServerTransfer/ServerTransferExporter.php +++ b/app/Services/ServerTransfer/ServerTransferExporter.php @@ -957,6 +957,11 @@ class ServerTransferExporter } } + // Not fillable + if (in_array('include_coolify_default_variables', $columns, true)) { + $attrs['include_coolify_default_variables'] = $database->include_coolify_default_variables; + } + $destinationKey = $database->destination_type.':'.$database->destination_id; return [ @@ -1014,8 +1019,8 @@ class ServerTransferExporter $attrs[$field] = $service->{$field}; } - // Hidden compose fields and maintenance mode (not fillable) - foreach (['docker_compose', 'docker_compose_raw', 'is_maintenance_enabled', 'maintenance_page'] as $hidden) { + // Hidden compose fields, maintenance mode and the Coolify default variables switch (not fillable) + foreach (['docker_compose', 'docker_compose_raw', 'is_maintenance_enabled', 'maintenance_page', 'include_coolify_default_variables'] as $hidden) { if (in_array($hidden, $columns, true)) { $attrs[$hidden] = $service->{$hidden}; } diff --git a/app/Services/ServerTransfer/ServerTransferImporter.php b/app/Services/ServerTransfer/ServerTransferImporter.php index 80cf44efd3..053d043d0d 100644 --- a/app/Services/ServerTransfer/ServerTransferImporter.php +++ b/app/Services/ServerTransfer/ServerTransferImporter.php @@ -1384,8 +1384,8 @@ class ServerTransferImporter $attributes['destination_type'] = $destination->getMorphClass(); $service = new Service($attributes); - // Maintenance mode is not fillable. - $service->forceFill(collect($attributes)->only(['is_maintenance_enabled', 'maintenance_page'])->all()); + // Maintenance mode and the Coolify default variables switch are not fillable. + $service->forceFill(collect($attributes)->only(['is_maintenance_enabled', 'maintenance_page', 'include_coolify_default_variables'])->all()); $service->save(); $this->importEnvVars(data_get($payload, 'environment_variables', []), $service, false); diff --git a/app/Traits/ExecuteRemoteCommand.php b/app/Traits/ExecuteRemoteCommand.php index 86ed88e9b8..21912fb82f 100644 --- a/app/Traits/ExecuteRemoteCommand.php +++ b/app/Traits/ExecuteRemoteCommand.php @@ -248,7 +248,22 @@ trait ExecuteRemoteCommand $error = $this->redact_sensitive_info($error); - return "Command execution failed (exit code {$exitCode}): {$commandText}\nError: {$error}"; + return "Command execution failed (exit code {$exitCode}): {$commandText}\nError: {$error}".$this->dockerNetworkGatewayHint($error); + } + + /** + * Docker Engine 27.0 to 28.0.0 reports an IPv6 network gateway with its prefix length (fd00::1/64) until the + * daemon restarts, and newer Docker CLI and Compose versions fail to parse it. + * + * @see https://github.com/moby/moby/pull/49520 + */ + private function dockerNetworkGatewayHint(string $error): string + { + if (! preg_match('~ParseAddr\("[0-9A-Fa-f:.]+/\d+"\)~', $error)) { + return ''; + } + + return "\nDocker Engine on this server reports a Docker network gateway with a prefix length, which is a Docker Engine bug fixed in 28.0.1. Run 'systemctl restart docker' on the server (this restarts all containers) or upgrade Docker Engine, then deploy again."; } /** diff --git a/app/Traits/ExecutesDatabaseStartCommands.php b/app/Traits/ExecutesDatabaseStartCommands.php index d61cfb3146..dadc4dd68c 100644 --- a/app/Traits/ExecutesDatabaseStartCommands.php +++ b/app/Traits/ExecutesDatabaseStartCommands.php @@ -27,4 +27,23 @@ trait ExecutesDatabaseStartCommands { return $environmentVariables->contains(fn ($variable) => str($variable)->before('=')->value() === $key); } + + /** + * Adds the Coolify variables that applications and services also get. A key that the user + * defines wins. + * + * @param Collection $environmentVariables + */ + private function addCoolifyEnvironmentVariables(Collection $environmentVariables): void + { + $definedKeys = $environmentVariables->map(fn ($variable) => str($variable)->before('=')->value()); + $coolifyVariables = collect([ + 'COOLIFY_RESOURCE_UUID' => $this->database->uuid, + 'COOLIFY_CONTAINER_NAME' => $this->database->uuid, + ])->except($definedKeys->all())->merge(coolifyDefaultEnvironmentVariables($this->database, $definedKeys)); + + foreach ($coolifyVariables as $key => $value) { + $environmentVariables->push($key.'='.escapeDollarSign($value)); + } + } } diff --git a/bootstrap/helpers/constants.php b/bootstrap/helpers/constants.php index 4ab7d74698..543901932f 100644 --- a/bootstrap/helpers/constants.php +++ b/bootstrap/helpers/constants.php @@ -81,6 +81,8 @@ const SPECIFIC_SERVICES = [ 'minio/minio', 'ghcr.io/coollabsio/minio', 'coollabsio/minio', + 'pgsty/silo', + 'docker.io/pgsty/silo', 'svhd/logto', 'dxflrs/garage', ]; diff --git a/bootstrap/helpers/docker.php b/bootstrap/helpers/docker.php index a8f8ebee5a..33dd277d03 100644 --- a/bootstrap/helpers/docker.php +++ b/bootstrap/helpers/docker.php @@ -7,6 +7,7 @@ use App\Models\Server; use App\Models\Service; use App\Models\ServiceApplication; use App\Models\ServiceDatabase; +use App\Support\DomainPortOverrides; use App\Support\ValidationPatterns; use Illuminate\Database\Eloquent\Model; use Illuminate\Support\Collection; @@ -675,7 +676,7 @@ function generateServiceSpecificFqdns(ServiceApplication|Application $resource) $variables = collect($environment_variables); $payload = collect([]); switch ($type) { - case $type?->contains('minio'): + case $type?->contains(['minio', 'pgsty/silo']): $MINIO_BROWSER_REDIRECT_URL = $variables->where('key', 'MINIO_BROWSER_REDIRECT_URL')->first(); $MINIO_SERVER_URL = $variables->where('key', 'MINIO_SERVER_URL')->first(); @@ -765,9 +766,12 @@ function isNoindexDomain(string $domain, ?Collection $noindex_domains): bool return false; } + // Flags are stored without a port, so compare portless: https://app.example.com:3000 matches https://app.example.com. + $normalize = fn (string $url) => DomainPortOverrides::withoutPort(ValidationPatterns::normalizeApplicationDomainUrl($url)); + return $noindex_domains - ->map(fn (string $noindex_domain) => ValidationPatterns::normalizeApplicationDomainUrl($noindex_domain)) - ->contains(ValidationPatterns::normalizeApplicationDomainUrl($domain)); + ->map($normalize) + ->contains($normalize($domain)); } /** diff --git a/bootstrap/helpers/domains.php b/bootstrap/helpers/domains.php index 8be6d9a77a..704c90988f 100644 --- a/bootstrap/helpers/domains.php +++ b/bootstrap/helpers/domains.php @@ -3,6 +3,7 @@ use App\Models\Application; use App\Models\Service; use App\Models\ServiceApplication; +use App\Models\ServiceDatabase; use Illuminate\Support\Collection; /** @@ -591,6 +592,47 @@ function findServiceApplicationForEnvName(Service $resource, string $envServiceN return $resource->applications()->where('name', $resolved)->first(); } +/** + * Check if a service copies a SERVICE_FQDN_ / SERVICE_URL_ declaration of the application named in it. + * The copy (same name, port and value; URL or FQDN) must not generate a domain or overwrite the variable (#11924). + * Declarations with another port or path share the host on purpose (for example Appwrite). + * + * @param array|Collection $services Parsed compose services + */ +function isCopiedServiceUrlDeclaration( + Service $resource, + array|Collection $services, + ServiceApplication|ServiceDatabase $current, + string $key, +): bool { + $owner = findServiceApplicationForEnvName($resource, parseServiceEnvironmentVariable($key)['service_name']); + if (! $owner || $owner->name === $current->name) { + return false; + } + + // Same declaration sources as serviceParser: environment and build arguments. + $declarationsOf = fn (string $serviceName) => convertToKeyValueCollection( + collect(data_get($services, [$serviceName, 'environment'], [])) + ->merge(data_get($services, [$serviceName, 'build', 'args'], [])) + ); + $ownerDeclarations = $declarationsOf($owner->name); + $currentDeclarations = $declarationsOf($current->name); + + if (! $currentDeclarations->has($key)) { + return false; + } + + // SERVICE_URL_X_3000 and SERVICE_FQDN_X_3000 write the same variable pair, so either one counts. + $twinKey = str_starts_with($key, 'SERVICE_URL_') + ? 'SERVICE_FQDN_'.substr($key, strlen('SERVICE_URL_')) + : 'SERVICE_URL_'.substr($key, strlen('SERVICE_FQDN_')); + $value = (string) $currentDeclarations->get($key); + + return collect([$key, $twinKey])->contains( + fn (string $ownerKey) => $ownerDeclarations->has($ownerKey) && (string) $ownerDeclarations->get($ownerKey) === $value + ); +} + /** * Put/update a service domain entry under the original compose service name, * removing legacy twin keys that resolve unambiguously to the same name. diff --git a/bootstrap/helpers/github.php b/bootstrap/helpers/github.php index d66ffd88b8..edcbca88f6 100644 --- a/bootstrap/helpers/github.php +++ b/bootstrap/helpers/github.php @@ -4,6 +4,8 @@ use App\Models\GithubApp; use App\Models\PrivateKey; use Carbon\Carbon; use Carbon\CarbonImmutable; +use Illuminate\Http\Client\ConnectionException; +use Illuminate\Http\Client\RequestException; use Illuminate\Support\Facades\Cache; use Illuminate\Support\Facades\Http; use Illuminate\Support\Str; @@ -195,13 +197,20 @@ function generateGithubToken(GithubApp $source, string $type) return match ($type) { 'jwt' => $jwt, 'installation' => (function () use ($source, $jwt) { + // GitHub sometimes answers with a 5xx page or drops the connection; a new token request is harmless. $response = Http::GitSource($source->api_url)->withHeaders([ 'Authorization' => "Bearer $jwt", 'Accept' => 'application/vnd.github.machine-man-preview+json', - ])->post("{$source->api_url}/app/installations/{$source->installation_id}/access_tokens"); + ])->retry( + 3, + fn (int $attempt) => $attempt * 1000, + fn (Throwable $exception) => $exception instanceof ConnectionException + || ($exception instanceof RequestException && $exception->response->serverError()), + throw: false, + )->post("{$source->api_url}/app/installations/{$source->installation_id}/access_tokens"); if (! $response->successful()) { - $error = data_get($response->json(), 'message', 'no error message found'); + $error = data_get($response->json(), 'message', "HTTP {$response->status()}"); if ($error === 'Not Found') { $error = 'Repository not found. Is it moved or deleted?'; } @@ -216,7 +225,7 @@ function generateGithubToken(GithubApp $source, string $type) function generateGithubInstallationToken(GithubApp $source) { - return generateGithubToken($source, 'installation'); + return $source->installationToken(); } function generateGithubJwt(GithubApp $source) @@ -397,28 +406,34 @@ function loadRepositoryByPage(GithubApp $source, string $token, int $page) 'repositories' => $json['repositories'], ]; } -function getGithubCommitRangeFiles(?GithubApp $source, string $owner, string $repo, string $beforeSha, string $afterSha): array +/** + * Returns null when the full list of changed files is unknown (API error or truncated list). + */ +function getGithubCommitRangeFiles(?GithubApp $source, string $owner, string $repo, string $beforeSha, string $afterSha): ?array { try { if (! $source) { // Manual webhooks don't have GitHub App authentication - // Return empty array so watch paths are ignored (current behavior) return []; } $endpoint = "/repos/{$owner}/{$repo}/compare/{$beforeSha}...{$afterSha}"; - $response = githubApi($source, $endpoint, 'get', null, false); - - if (! $response) { - return []; - } - + $response = githubApi($source, $endpoint); $files = collect(data_get($response, 'data.files', [])); - return $files->pluck('filename')->filter()->values()->toArray(); - } catch (Exception $e) { + // The compare API returns at most 300 files and cannot paginate them. + if ($files->count() >= 300) { + return null; + } - return []; + // A renamed file also changes its previous path. + return $files->flatMap(fn (mixed $file): array => [data_get($file, 'filename'), data_get($file, 'previous_filename')]) + ->filter(fn (mixed $filename): bool => is_string($filename) && $filename !== '') + ->unique() + ->values() + ->toArray(); + } catch (Exception $e) { + return null; } } @@ -444,27 +459,34 @@ function getGithubCommitMessage(?GithubApp $source, string $owner, string $repo, } } -function getGithubPullRequestFiles(?GithubApp $source, string $owner, string $repo, int $pullRequestId): array +/** + * Returns null when the full list of changed files is unknown (API error or truncated list). + */ +function getGithubPullRequestFiles(?GithubApp $source, string $owner, string $repo, int $pullRequestId): ?array { try { if (! $source) { // Manual webhooks don't have GitHub App authentication - // Return empty array so watch paths are ignored (current behavior) return []; } - $endpoint = "/repos/{$owner}/{$repo}/pulls/{$pullRequestId}/files"; - $response = githubApi($source, $endpoint, 'get', null, false); + // GitHub returns at most 100 files per page and 3000 files per pull request. + $perPage = 100; + $maxPages = 30; + $files = collect(); - if (! $response) { - return []; + for ($page = 1; $page <= $maxPages; $page++) { + $endpoint = "/repos/{$owner}/{$repo}/pulls/{$pullRequestId}/files?per_page={$perPage}&page={$page}"; + $pageFiles = collect(data_get(githubApi($source, $endpoint), 'data', [])); + $files = $files->merge($pageFiles->pluck('filename')); + + if ($pageFiles->count() < $perPage) { + return $files->filter()->values()->toArray(); + } } - $files = collect(data_get($response, 'data', [])); - - return $files->pluck('filename')->filter()->values()->toArray(); + return null; } catch (Exception $e) { - - return []; + return null; } } diff --git a/bootstrap/helpers/parsers.php b/bootstrap/helpers/parsers.php index 01a5b04a66..aa9639f737 100644 --- a/bootstrap/helpers/parsers.php +++ b/bootstrap/helpers/parsers.php @@ -2031,7 +2031,9 @@ function applicationParser(Application $resource, int $pull_request_id = 0, ?int $coolifyEnvironments->put('COOLIFY_URL', $fqdns->map(fn ($fqdn) => getFqdnWithoutPort($fqdn))->implode(',')); $coolifyEnvironments->put('COOLIFY_FQDN', $fqdns->map(fn ($fqdn) => getHostWithoutPort($fqdn))->implode(',')); } - add_coolify_default_environment_variables($resource, $coolifyEnvironments, $resource->environment_variables); + $userDefinedKeys = $resource->environment_variables->pluck('key')->merge($environment->keys()); + coolifyDefaultEnvironmentVariables($resource, $userDefinedKeys) + ->each(fn ($value, $key) => $coolifyEnvironments->put($key, escapeDollarSign($value))); if ($environment->count() > 0) { $environment = $environment->filter(function ($value, $key) { return ! str($key)->startsWith('SERVICE_FQDN_'); @@ -2547,6 +2549,10 @@ function serviceParser(Service $resource): Collection $port = $parsed['port']; $fqdnFor = $parsed['service_name']; + if (isCopiedServiceUrlDeclaration($resource, $services, $savedService, $key->value())) { + continue; + } + // Only ServiceApplication has fqdn column, ServiceDatabase does not $isServiceApplication = $savedService instanceof ServiceApplication; @@ -2667,8 +2673,8 @@ function serviceParser(Service $resource): Collection // Also check if a port-suffixed version exists (e.g., SERVICE_FQDN_UMAMI_3000) $portSuffixedExists = $resource->environment_variables() ->where('key', 'LIKE', $key->value().'_%') - ->whereRaw('key ~ ?', ['^'.$key->value().'_[0-9]+$']) - ->exists(); + ->pluck('key') + ->contains(fn ($envKey) => preg_match('/^'.preg_quote($key->value(), '/').'_[0-9]+$/', $envKey) === 1); $serviceExists = findServiceApplicationForEnvName($resource, (string) $fqdnFor); // Check if FQDN already has a port set (contains ':' after the domain) $fqdnHasPort = $serviceExists && str($serviceExists->fqdn)->contains(':') && str($serviceExists->fqdn)->afterLast(':')->isMatch('/^\d+$/'); @@ -2712,8 +2718,8 @@ function serviceParser(Service $resource): Collection // Also check if a port-suffixed version exists (e.g., SERVICE_URL_DASHBOARD_6791) $portSuffixedExists = $resource->environment_variables() ->where('key', 'LIKE', $key->value().'_%') - ->whereRaw('key ~ ?', ['^'.$key->value().'_[0-9]+$']) - ->exists(); + ->pluck('key') + ->contains(fn ($envKey) => preg_match('/^'.preg_quote($key->value(), '/').'_[0-9]+$/', $envKey) === 1); $serviceExists = findServiceApplicationForEnvName($resource, (string) $urlFor); // Check if FQDN already has a port set (contains ':' after the domain) $fqdnHasPort = $serviceExists && str($serviceExists->fqdn)->contains(':') && str($serviceExists->fqdn)->afterLast(':')->isMatch('/^\d+$/'); @@ -3333,7 +3339,9 @@ function serviceParser(Service $resource): Collection $coolifyEnvironments->put('COOLIFY_FQDN', $fqdns->map(fn ($fqdn) => getHostWithoutPort($fqdn))->implode(',')); $coolifyEnvironments->put('COOLIFY_URL', $fqdns->map(fn ($fqdn) => getFqdnWithoutPort($fqdn))->implode(',')); } - add_coolify_default_environment_variables($resource, $coolifyEnvironments, $resource->environment_variables); + $userDefinedKeys = $resource->environment_variables->pluck('key')->merge($environment->keys()); + coolifyDefaultEnvironmentVariables($resource, $userDefinedKeys) + ->each(fn ($value, $key) => $coolifyEnvironments->put($key, escapeDollarSign($value))); if ($environment->count() > 0) { $environment = $environment->filter(function ($value, $key) { return ! str($key)->startsWith('SERVICE_FQDN_'); diff --git a/bootstrap/helpers/services.php b/bootstrap/helpers/services.php index 66154ce7e8..667b9b18e3 100644 --- a/bootstrap/helpers/services.php +++ b/bootstrap/helpers/services.php @@ -409,8 +409,16 @@ function updateCompose(ServiceApplication|ServiceDatabase $resource) // Those belong to other services and will be updated when THOSE services are updated } - // Remove duplicates - $templateVariableNames = array_unique($templateVariableNames); + // Remove duplicates and copies of another application's declaration + $templateVariableNames = array_filter( + array_unique($templateVariableNames), + fn ($variableName) => ! isCopiedServiceUrlDeclaration( + $resource->service, + collect(data_get($dockerCompose, 'services', [])), + $resource, + $variableName, + ), + ); // Extract unique service names to process (preserving the original case from template) // This allows us to create both URL and FQDN pairs regardless of which one is in the template diff --git a/bootstrap/helpers/shared.php b/bootstrap/helpers/shared.php index 537b7a4d59..89f408fae8 100644 --- a/bootstrap/helpers/shared.php +++ b/bootstrap/helpers/shared.php @@ -935,6 +935,15 @@ function s3_image_url(?int $storageId, ?string $path, int $version): ?string return null; } + // S3-compatible endpoints (e.g. Cloudflare R2) require authenticated requests for + // every object, including plain GETs, so an unauthenticated direct link only works + // when a public CDN has been configured in front of the bucket. Without one, callers + // must fall back to the app's own authenticated proxy route. + $cdnUrl = instanceSettings()->image_cdn_url; + if (blank($cdnUrl)) { + return null; + } + $storage = S3Storage::query() ->whereKey($storageId) ->whereTeamId(0) @@ -945,9 +954,7 @@ function s3_image_url(?int $storageId, ?string $path, int $version): ?string return null; } - $baseUrl = instanceSettings()->image_cdn_url ?: $storage->awsUrl(); - - return rtrim($baseUrl, '/').'/'.ltrim($path, '/').'?v='.$version; + return rtrim($cdnUrl, '/').'/'.ltrim($path, '/').'?v='.$version; } /** @@ -1517,7 +1524,25 @@ function sslip(Server $server) } $ip = $server->ip === 'host.docker.internal' ? base_ip() : $server->ip; - return 'http://'.sslipHostLabel($ip).'.sslip.io'; + return 'http://'.sslipHostLabel(resolveHostToIp($ip)).'.sslip.io'; +} + +/** + * Resolve a hostname to its IPv4 (or else IPv6) address. sslip.io only answers for names that + * contain an IP address. Returns the input unchanged if it is already an IP or does not resolve. + */ +function resolveHostToIp(string $host): string +{ + if (filter_var(normalizeIpAddress($host), FILTER_VALIDATE_IP) !== false) { + return $host; + } + $ipv4 = gethostbyname($host); + if (filter_var($ipv4, FILTER_VALIDATE_IP, FILTER_FLAG_IPV4) !== false) { + return $ipv4; + } + $ipv6 = data_get(@dns_get_record($host, DNS_AAAA), '0.ipv6'); + + return is_string($ipv6) && $ipv6 !== '' ? $ipv6 : $host; } function service_templates_cache_key(): string @@ -2683,44 +2708,53 @@ function deduplicateAllowlist(array $entries): array return $result; } -function get_public_ips() +/** + * Detect the instance public IPs with ifconfig.io. Only empty settings are filled, so an address saved in Settings is kept. + */ +function get_public_ips(): void { try { - [$first, $second] = Process::concurrently(function (Pool $pool) { - $pool->path(__DIR__)->command('curl -4s https://ifconfig.io'); - $pool->path(__DIR__)->command('curl -6s https://ifconfig.io'); - }); - $ipv4 = $first->output(); - if ($ipv4) { - $ipv4 = trim($ipv4); - $validate_ipv4 = filter_var($ipv4, FILTER_VALIDATE_IP, FILTER_FLAG_IPV4); - if ($validate_ipv4 == false) { - echo "Invalid ipv4: $ipv4\n"; - - return; - } - InstanceSettings::get()->update(['public_ipv4' => $ipv4]); + $settings = InstanceSettings::get(); + $columns = array_values(array_filter(['public_ipv4', 'public_ipv6'], fn (string $column) => blank($settings->$column))); + if ($columns === []) { + return; } - } catch (Exception $e) { - echo "Error: {$e->getMessage()}\n"; - } - try { - $ipv6 = $second->output(); - if ($ipv6) { - $ipv6 = trim($ipv6); - $validate_ipv6 = filter_var($ipv6, FILTER_VALIDATE_IP, FILTER_FLAG_IPV6); - if ($validate_ipv6 == false) { - echo "Invalid ipv6: $ipv6\n"; - return; - } - InstanceSettings::get()->update(['public_ipv6' => $ipv6]); + $detected = detect_public_ips($columns); + if ($detected !== []) { + $settings->update($detected); } } catch (Throwable $e) { echo "Error: {$e->getMessage()}\n"; } } +/** + * Look up the public IPs of this host with ifconfig.io, without saving them. + * + * @param list<'public_ipv4'|'public_ipv6'> $columns + * @return array<'public_ipv4'|'public_ipv6', string> valid addresses only + */ +function detect_public_ips(array $columns = ['public_ipv4', 'public_ipv6']): array +{ + $versions = array_intersect_key(['public_ipv4' => 4, 'public_ipv6' => 6], array_flip($columns)); + $results = Process::concurrently(function (Pool $pool) use ($versions) { + foreach ($versions as $column => $version) { + $pool->as($column)->path(__DIR__)->command("curl -{$version}s https://ifconfig.io"); + } + }); + + $detected = []; + foreach ($versions as $column => $version) { + $ip = trim($results[$column]->output()); + if (filter_var($ip, FILTER_VALIDATE_IP, $version === 4 ? FILTER_FLAG_IPV4 : FILTER_FLAG_IPV6) !== false) { + $detected[$column] = $ip; + } + } + + return $detected; +} + function isAnyDeploymentInprogress(bool $showAll = false) { $runningJobs = ApplicationDeploymentQueue::where('horizon_job_worker', gethostname())->where('status', ApplicationDeploymentStatus::IN_PROGRESS->value)->get(); @@ -3624,19 +3658,8 @@ function parseDockerComposeFile(Service|Application $resource, bool $isNew = fal $parsedServiceVariables->put('COOLIFY_RESOURCE_UUID', "{$resource->uuid}"); $parsedServiceVariables->put('COOLIFY_CONTAINER_NAME', "$serviceName-{$resource->uuid}"); - // TODO: move this in a shared function - if (! $parsedServiceVariables->has('COOLIFY_APP_NAME')) { - $parsedServiceVariables->put('COOLIFY_APP_NAME', "\"{$resource->name}\""); - } - if (! $parsedServiceVariables->has('COOLIFY_SERVER_IP')) { - $parsedServiceVariables->put('COOLIFY_SERVER_IP', "\"{$resource->destination->server->ip}\""); - } - if (! $parsedServiceVariables->has('COOLIFY_ENVIRONMENT_NAME')) { - $parsedServiceVariables->put('COOLIFY_ENVIRONMENT_NAME', "\"{$resource->environment->name}\""); - } - if (! $parsedServiceVariables->has('COOLIFY_PROJECT_NAME')) { - $parsedServiceVariables->put('COOLIFY_PROJECT_NAME', "\"{$resource->project()->name}\""); - } + coolifyDefaultEnvironmentVariables($resource, $parsedServiceVariables->keys()) + ->each(fn ($value, $key) => $parsedServiceVariables->put($key, escapeDollarSign($value))); $parsedServiceVariables = $parsedServiceVariables->map(function ($value, $key) use ($envs_from_coolify) { if (! str($value)->startsWith('$')) { @@ -4484,56 +4507,32 @@ function isAssociativeArray($array) } /** - * This method adds the default environment variables to the resource. - * - COOLIFY_APP_NAME - * - COOLIFY_PROJECT_NAME - * - COOLIFY_SERVER_IP - * - COOLIFY_ENVIRONMENT_NAME + * The variables that tell a container where it runs. Values are raw: each caller escapes them + * for its output (compose file, .env file, or build arguments). Keys in $definedKeys are + * skipped, so a variable that the user defines wins. Pass $server when the container runs on + * another server than the main destination (an application on an additional server). * - * Theses variables are added in place to the $where_to_add array. + * @param iterable $definedKeys + * @return Collection */ -function add_coolify_default_environment_variables(StandaloneRedis|StandalonePostgresql|StandaloneMongodb|StandaloneMysql|StandaloneMariadb|StandaloneKeydb|StandaloneDragonfly|StandaloneClickhouse|StandaloneSqlite|Application|Service $resource, Collection &$where_to_add, ?Collection $where_to_check = null) +function coolifyDefaultEnvironmentVariables(StandaloneRedis|StandalonePostgresql|StandaloneMongodb|StandaloneMysql|StandaloneMariadb|StandaloneKeydb|StandaloneDragonfly|StandaloneClickhouse|StandaloneSqlite|Application|Service $resource, iterable $definedKeys = [], ?Server $server = null): Collection { - // Currently disabled - return; - if ($resource instanceof Service) { - $ip = $resource->server->ip; - } else { - $ip = $resource->destination->server->ip; - } - if (isAssociativeArray($where_to_add)) { - $isAssociativeArray = true; - } else { - $isAssociativeArray = false; - } - if ($where_to_check != null && $where_to_check->where('key', 'COOLIFY_APP_NAME')->isEmpty()) { - if ($isAssociativeArray) { - $where_to_add->put('COOLIFY_APP_NAME', "\"{$resource->name}\""); - } else { - $where_to_add->push("COOLIFY_APP_NAME=\"{$resource->name}\""); - } - } - if ($where_to_check != null && $where_to_check->where('key', 'COOLIFY_SERVER_IP')->isEmpty()) { - if ($isAssociativeArray) { - $where_to_add->put('COOLIFY_SERVER_IP', "\"{$ip}\""); - } else { - $where_to_add->push("COOLIFY_SERVER_IP=\"{$ip}\""); - } - } - if ($where_to_check != null && $where_to_check->where('key', 'COOLIFY_ENVIRONMENT_NAME')->isEmpty()) { - if ($isAssociativeArray) { - $where_to_add->put('COOLIFY_ENVIRONMENT_NAME', "\"{$resource->environment->name}\""); - } else { - $where_to_add->push("COOLIFY_ENVIRONMENT_NAME=\"{$resource->environment->name}\""); - } - } - if ($where_to_check != null && $where_to_check->where('key', 'COOLIFY_PROJECT_NAME')->isEmpty()) { - if ($isAssociativeArray) { - $where_to_add->put('COOLIFY_PROJECT_NAME', "\"{$resource->project()->name}\""); - } else { - $where_to_add->push("COOLIFY_PROJECT_NAME=\"{$resource->project()->name}\""); - } + // A model created in this request has no column default loaded yet (null), which means on. + $isEnabled = $resource instanceof Application + ? $resource->settings?->include_coolify_default_variables + : $resource->include_coolify_default_variables; + if ($isEnabled === false) { + return collect(); } + + $server ??= $resource instanceof Service ? $resource->server : $resource->destination?->server; + + return collect([ + 'COOLIFY_APP_NAME' => $resource->name, + 'COOLIFY_PROJECT_NAME' => $resource->project()?->name, + 'COOLIFY_ENVIRONMENT_NAME' => $resource->environment?->name, + 'COOLIFY_SERVER_IP' => $server?->ip, + ])->whereNotNull()->map(fn ($value) => (string) $value)->except(collect($definedKeys)->all()); } function convertToKeyValueCollection($environment) @@ -4858,8 +4857,15 @@ function parseScpStyleGitUrl(?string $gitRepository): ?array ]; } -function scpStyleGitUrlToHttps(?string $gitRepository): ?string +function sshGitUrlToHttps(?string $gitRepository): ?string { + if (is_string($gitRepository) && str_starts_with($gitRepository, 'ssh://')) { + $host = parse_url($gitRepository, PHP_URL_HOST); + $path = ltrim((string) parse_url($gitRepository, PHP_URL_PATH), '/'); + + return filled($host) && $path !== '' ? "https://{$host}/{$path}" : null; + } + $parts = parseScpStyleGitUrl($gitRepository); if ($parts === null) { @@ -4875,8 +4881,8 @@ function gitRepositorySlug(?string $gitRepository): string return ''; } - if (($scp = parseScpStyleGitUrl($gitRepository)) !== null) { - $gitRepository = $scp['path']; + if (($httpsRepository = sshGitUrlToHttps($gitRepository)) !== null) { + $gitRepository = (string) parse_url($httpsRepository, PHP_URL_PATH); } elseif (str($gitRepository)->startsWith('http') || str($gitRepository)->contains('github.com')) { $gitRepository = str($gitRepository)->replace('https://', '')->replace('http://', '')->replace('github.com/', ''); } diff --git a/database/migrations/2026_10_15_213357_spread_default_auto_update_frequency.php b/database/migrations/2026_10_15_213357_spread_default_auto_update_frequency.php new file mode 100644 index 0000000000..9648ab50ec --- /dev/null +++ b/database/migrations/2026_10_15_213357_spread_default_auto_update_frequency.php @@ -0,0 +1,24 @@ +where('auto_update_frequency', '0 0 * * *') + ->update(['auto_update_frequency' => InstanceSettings::randomAutoUpdateFrequency()]); + } + + public function down(): void + { + // + } +}; diff --git a/database/migrations/2026_10_18_103517_add_include_coolify_default_variables_to_resources.php b/database/migrations/2026_10_18_103517_add_include_coolify_default_variables_to_resources.php new file mode 100644 index 0000000000..5059e44005 --- /dev/null +++ b/database/migrations/2026_10_18_103517_add_include_coolify_default_variables_to_resources.php @@ -0,0 +1,40 @@ +boolean('include_coolify_default_variables')->default(true); + }); + } + } + + public function down(): void + { + foreach (self::TABLES as $tableName) { + Schema::table($tableName, function (Blueprint $table) { + $table->dropColumn('include_coolify_default_variables'); + }); + } + } +}; diff --git a/docker/development/Dockerfile b/docker/development/Dockerfile index aa178765ad..06e779d4ae 100644 --- a/docker/development/Dockerfile +++ b/docker/development/Dockerfile @@ -70,6 +70,7 @@ RUN apk add --no-cache \ git-lfs \ jq \ lsof \ + procps \ vim # Install PHP extensions diff --git a/docker/production/Dockerfile b/docker/production/Dockerfile index 9ddff6476d..5060ef6b13 100644 --- a/docker/production/Dockerfile +++ b/docker/production/Dockerfile @@ -135,6 +135,7 @@ RUN --mount=type=cache,target=/var/cache/apk \ git-lfs \ jq \ lsof \ + procps \ vim # Install PHP extensions @@ -186,6 +187,8 @@ COPY --chown=www-data:www-data resources/views ./resources/views COPY --chown=www-data:www-data artisan artisan COPY --chown=www-data:www-data openapi.yaml ./openapi.yaml COPY --chown=www-data:www-data changelogs/ ./changelogs/ +# Version data for Traefik, Sentinel, and helper checks until CheckForUpdatesJob refreshes it +COPY --chown=www-data:www-data versions.json ./versions.json RUN composer dump-autoload diff --git a/docs/server-transfer-full-coverage.md b/docs/server-transfer-full-coverage.md index 447798cb90..af6636b8f0 100644 --- a/docs/server-transfer-full-coverage.md +++ b/docs/server-transfer-full-coverage.md @@ -93,7 +93,7 @@ When you add a model or a setting to the transfer, add it to this file too. | ➕ | Limits | memory, swap, swappiness, reservation, CPUs, cpuset, CPU shares | | ➕ | Advanced | custom labels, custom docker run options, pre/post deployment command + container, manual webhook secrets | | ✅ | Maintenance | maintenance mode on + custom maintenance page | -| ➕ | All application settings that are not the default | git submodules, LFS, shallow clone, auto deploy off, force HTTPS, debug, preview deployments on, PR deployments public, log drain, GPU (driver, count, device IDs, options), include timestamps, swarm-only worker nodes, build server, consistent container name, gzip off, strip prefix off, connect to Docker network, custom internal name, container label escape off, env sorting, read-only labels, preserve repository, disable build cache, use build secrets, stop grace period 30, inject build args off, include source commit, Docker images to keep 5, custom container name prefix, redact env values in logs, Traefik certificate resolver | +| ➕ | All application settings that are not the default | git submodules, LFS, shallow clone, auto deploy off, force HTTPS, debug, preview deployments on, PR deployments public, log drain, GPU (driver, count, device IDs, options), include timestamps, swarm-only worker nodes, build server, consistent container name, gzip off, strip prefix off, connect to Docker network, custom internal name, container label escape off, env sorting, read-only labels, preserve repository, disable build cache, use build secrets, stop grace period 30, inject build args off, include source commit, Docker images to keep 5, custom container name prefix, redact env values in logs, Coolify default variables off, Traefik certificate resolver | | ➕ | Environment variables | normal, multiline, literal, shown once, build time only, runtime only, required, with comment, preview-only, and references to `{{team.*}}`, `{{project.*}}`, `{{environment.*}}`, `{{server.*}}` | | ✅/➕ | Persistent volume | the seed has one; add a host-path bind mount and preview suffix on | | ➕ | File mount + directory mount | content, chown, chmod | @@ -116,7 +116,7 @@ When you add a model or a setting to the transfer, add it to this file too. | ➕ | KeyDB, Dragonfly, ClickHouse | defaults are enough, plus the common settings below | | ➕ | SQLite | database files list, volume link | -Common settings on at least one database: custom image, public port + public on + public port timeout, limits, custom docker run options, health check values, log drain, include timestamps, redact env values, environment variables, persistent volume + volume backup, tags, secret manager link (if the type supports it). +Common settings on at least one database: custom image, public port + public on + public port timeout, limits, custom docker run options, health check values, log drain, include timestamps, redact env values, Coolify default variables off, environment variables, persistent volume + volume backup, tags, secret manager link (if the type supports it). Database backups (✅ one exists): add S3 on, dump all, databases list, local and S3 retention (amount, days, max storage), timeout, disable local backup, missing-backup days. @@ -128,7 +128,7 @@ Database backups (✅ one exists): add S3 on, dump all, databases list, local an | ➕ | Custom compose service with an app and a database part | | | ➕ | Service app settings | domain, gzip off, strip prefix off, force HTTPS, exclude from status, redirect, port overrides, log drain, custom image, Traefik resolver | | ➕ | Service database settings | public port + public on, custom type | -| ➕ | Service level | connect to network, label escape off, redact env values, maintenance mode + page, environment variables, scheduled task, tags, volume backup on a service volume | +| ➕ | Service level | connect to network, label escape off, redact env values, Coolify default variables off, maintenance mode + page, environment variables, scheduled task, tags, volume backup on a service volume | ## 9. After the transfer, check diff --git a/openapi.json b/openapi.json index 075213afee..805871904f 100644 --- a/openapi.json +++ b/openapi.json @@ -846,7 +846,7 @@ }, "watch_paths": { "type": "string", - "description": "The watch paths." + "description": "Watch path patterns, one per line (newline-separated). Commas do not separate patterns." }, "use_build_server": { "type": "boolean", @@ -1476,7 +1476,7 @@ }, "watch_paths": { "type": "string", - "description": "The watch paths." + "description": "Watch path patterns, one per line (newline-separated). Commas do not separate patterns." }, "use_build_server": { "type": "boolean", @@ -2106,7 +2106,7 @@ }, "watch_paths": { "type": "string", - "description": "The watch paths." + "description": "Watch path patterns, one per line (newline-separated). Commas do not separate patterns." }, "use_build_server": { "type": "boolean", @@ -3586,18 +3586,6 @@ "application\/json": { "schema": { "properties": { - "project_uuid": { - "type": "string", - "description": "The project UUID." - }, - "server_uuid": { - "type": "string", - "description": "The server UUID." - }, - "environment_name": { - "type": "string", - "description": "The environment name." - }, "github_app_uuid": { "type": "string", "description": "The Github App UUID." @@ -3614,10 +3602,6 @@ "type": "string", "description": "The ports to expose." }, - "destination_uuid": { - "type": "string", - "description": "The destination UUID." - }, "build_pack": { "type": "string", "enum": [ @@ -3887,7 +3871,7 @@ }, "watch_paths": { "type": "string", - "description": "The watch paths." + "description": "Watch path patterns, one per line (newline-separated). Commas do not separate patterns." }, "use_build_server": { "type": "boolean", @@ -7302,6 +7286,10 @@ "type": "integer", "description": "CPU shares of the database" }, + "instant_deploy": { + "type": "boolean", + "description": "Restart the database after the update. Requires the deploy ability." + }, "postgres_user": { "type": "string", "description": "PostgreSQL user" @@ -19905,6 +19893,11 @@ "type": "string", "description": "The base64 encoded Docker Compose content." }, + "connect_to_docker_network": { + "type": "boolean", + "default": false, + "description": "Connect the service to the predefined docker network." + }, "urls": { "type": "array", "description": "Array of URLs to be applied to containers of a service.", diff --git a/openapi.yaml b/openapi.yaml index c91d927233..c4975a0564 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -542,7 +542,7 @@ paths: items: { properties: { name: { type: string, description: 'The service name as defined in docker-compose.' }, domain: { type: string, description: 'Comma-separated list of URLs (e.g. "https://app.coolify.io,https://app2.coolify.io")' }, redirect: { type: string, nullable: true, description: 'Per-service www/non-www redirect for this compose service.', enum: [www, non-www, both] } }, type: object } watch_paths: type: string - description: 'The watch paths.' + description: 'Watch path patterns, one per line (newline-separated). Commas do not separate patterns.' use_build_server: type: boolean nullable: true @@ -958,7 +958,7 @@ paths: items: { properties: { name: { type: string, description: 'The service name as defined in docker-compose.' }, domain: { type: string, description: 'Comma-separated list of URLs (e.g. "https://app.coolify.io,https://app2.coolify.io")' }, redirect: { type: string, nullable: true, description: 'Per-service www/non-www redirect for this compose service.', enum: [www, non-www, both] } }, type: object } watch_paths: type: string - description: 'The watch paths.' + description: 'Watch path patterns, one per line (newline-separated). Commas do not separate patterns.' use_build_server: type: boolean nullable: true @@ -1374,7 +1374,7 @@ paths: items: { properties: { name: { type: string, description: 'The service name as defined in docker-compose.' }, domain: { type: string, description: 'Comma-separated list of URLs (e.g. "https://app.coolify.io,https://app2.coolify.io")' }, redirect: { type: string, nullable: true, description: 'Per-service www/non-www redirect for this compose service.', enum: [www, non-www, both] } }, type: object } watch_paths: type: string - description: 'The watch paths.' + description: 'Watch path patterns, one per line (newline-separated). Commas do not separate patterns.' use_build_server: type: boolean nullable: true @@ -2369,15 +2369,6 @@ paths: application/json: schema: properties: - project_uuid: - type: string - description: 'The project UUID.' - server_uuid: - type: string - description: 'The server UUID.' - environment_name: - type: string - description: 'The environment name.' github_app_uuid: type: string description: 'The Github App UUID.' @@ -2390,9 +2381,6 @@ paths: ports_exposes: type: string description: 'The ports to expose.' - destination_uuid: - type: string - description: 'The destination UUID.' build_pack: type: string enum: [nixpacks, railpack, static, dockerfile, dockercompose] @@ -2572,7 +2560,7 @@ paths: items: { properties: { name: { type: string, description: 'The service name as defined in docker-compose.' }, domain: { type: string, description: 'Comma-separated list of URLs (e.g. "https://app.coolify.io,https://app2.coolify.io")' }, redirect: { type: string, nullable: true, description: 'Per-service www/non-www redirect for this compose service.', enum: [www, non-www, both] } }, type: object } watch_paths: type: string - description: 'The watch paths.' + description: 'Watch path patterns, one per line (newline-separated). Commas do not separate patterns.' use_build_server: type: boolean nullable: true @@ -4715,6 +4703,9 @@ paths: limits_cpu_shares: type: integer description: 'CPU shares of the database' + instant_deploy: + type: boolean + description: 'Restart the database after the update. Requires the deploy ability.' postgres_user: type: string description: 'PostgreSQL user' @@ -12614,6 +12605,10 @@ paths: docker_compose_raw: type: string description: 'The base64 encoded Docker Compose content.' + connect_to_docker_network: + type: boolean + default: false + description: 'Connect the service to the predefined docker network.' urls: type: array description: 'Array of URLs to be applied to containers of a service.' diff --git a/other/nightly/install.sh b/other/nightly/install.sh index 5b5d74f1b1..ebd309a021 100755 --- a/other/nightly/install.sh +++ b/other/nightly/install.sh @@ -7,7 +7,7 @@ ## ROOT_USER_PASSWORD - Predefined root user password ## DOCKER_ADDRESS_POOL_BASE - Custom Docker address pool base (default: 10.0.0.0/8) ## DOCKER_ADDRESS_POOL_SIZE - Custom Docker address pool size (default: 24) -## DOCKER_POOL_FORCE_OVERRIDE - Force override Docker address pool configuration (default: false) +## DOCKER_POOL_FORCE_OVERRIDE - Force override Docker address pool configuration, also when Docker must restart with running containers (default: false) ## AUTOUPDATE - Set to "false" to disable auto-updates ## REGISTRY_URL - Custom registry URL for Docker images (default: docker.io) @@ -31,14 +31,21 @@ fi # Resource directories hold data whose owner and mode the resource images or a non-root SSH user need. # Set their owner and mode only when they are created, never on an upgrade (#11945). -for COOLIFY_DIRECTORY in /data/coolify /data/coolify/{applications,databases,backups,services,proxy,proxy/dynamic,sentinel}; do - if [ ! -d "$COOLIFY_DIRECTORY" ]; then - mkdir -p "$COOLIFY_DIRECTORY" - chown 9999:root "$COOLIFY_DIRECTORY" - chmod 700 "$COOLIFY_DIRECTORY" +create_data_directory() { + if [ ! -d "$1" ]; then + mkdir -p "$1" + chown 9999:root "$1" + chmod 700 "$1" fi -done -mkdir -p /data/coolify/source /data/coolify/images /data/coolify/ssh/{keys,mux} +} + +# Run only after the checks and downloads succeed, so a failed run does not change the data directories (#11973). +create_coolify_directories() { + for COOLIFY_DIRECTORY in /data/coolify/{applications,databases,backups,services,proxy,proxy/dynamic,sentinel}; do + create_data_directory "$COOLIFY_DIRECTORY" + done + mkdir -p /data/coolify/images /data/coolify/ssh/{keys,mux} +} # Coolify's own directories, used by the coolify container (UID 9999) set_coolify_directory_permissions() { @@ -46,7 +53,9 @@ set_coolify_directory_permissions() { chmod -R 700 /data/coolify/{source,ssh,images} } -set_coolify_directory_permissions +# The installation log and the downloaded files go to /data/coolify/source, so only it is created before the checks. +create_data_directory /data/coolify +mkdir -p /data/coolify/source INSTALLATION_LOG_WITH_DATE="/data/coolify/source/installation-${DATE}.log" @@ -828,131 +837,39 @@ step_done "${DOCKER_SERVER_VERSION:+v$DOCKER_SERVER_VERSION}" step_start "Configuring Docker daemon" "log rotation · address pools" -echo " - Network pool configuration: ${DOCKER_ADDRESS_POOL_BASE}/${DOCKER_ADDRESS_POOL_SIZE}" -echo " - To override existing configuration: DOCKER_POOL_FORCE_OVERRIDE=true" - -mkdir -p /etc/docker - -# Backup original daemon.json if it exists -if [ -f /etc/docker/daemon.json ]; then - cp /etc/docker/daemon.json /etc/docker/daemon.json.original-"$DATE" -fi - -# Create coolify configuration with or without address pools based on whether they were explicitly provided +# Changing daemon.json needs a Docker restart, which stops every running container. +# Coolify sets log rotation and the network pool only when no pool is configured or DOCKER_POOL_FORCE_OVERRIDE=true, +# keeps all other keys, and does not restart a Docker daemon with running containers unless DOCKER_POOL_FORCE_OVERRIDE=true. if [ "$DOCKER_POOL_FORCE_OVERRIDE" = true ] || [ "$EXISTING_POOL_CONFIGURED" = false ]; then - # First check if the configuration would actually change anything - if [ -f /etc/docker/daemon.json ]; then - CURRENT_POOL_BASE=$(jq -r '.["default-address-pools"][0].base' /etc/docker/daemon.json 2>/dev/null) - CURRENT_POOL_SIZE=$(jq -r '.["default-address-pools"][0].size' /etc/docker/daemon.json 2>/dev/null) - - if [ "$CURRENT_POOL_BASE" = "$DOCKER_ADDRESS_POOL_BASE" ] && [ "$CURRENT_POOL_SIZE" = "$DOCKER_ADDRESS_POOL_SIZE" ]; then - echo " - Network pool configuration unchanged, skipping update" - NEED_MERGE=false + mkdir -p /etc/docker + CURRENT_DAEMON_CONFIG='{}' + if [ -s /etc/docker/daemon.json ] && ! CURRENT_DAEMON_CONFIG=$(jq -S . /etc/docker/daemon.json 2>/dev/null); then + CURRENT_DAEMON_CONFIG="" + fi + if [ -z "$CURRENT_DAEMON_CONFIG" ]; then + warn "Docker daemon configuration not changed: /etc/docker/daemon.json is not valid JSON." + else + NEW_DAEMON_CONFIG=$(jq -S --arg base "$DOCKER_ADDRESS_POOL_BASE" --argjson size "$DOCKER_ADDRESS_POOL_SIZE" \ + '. * {"log-driver": "json-file", "log-opts": {"max-size": "10m", "max-file": "3"}, "default-address-pools": [{"base": $base, "size": $size}]}' <<<"$CURRENT_DAEMON_CONFIG") + RUNNING_CONTAINERS=$({ docker ps -q 2>/dev/null || true; } | wc -l) + if [ "$NEW_DAEMON_CONFIG" = "$CURRENT_DAEMON_CONFIG" ]; then + echo " - Configuration is up to date" + elif [ "$RUNNING_CONTAINERS" -gt 0 ] && [ "$DOCKER_POOL_FORCE_OVERRIDE" != true ]; then + warn "Docker daemon configuration not changed: log rotation and the ${DOCKER_ADDRESS_POOL_BASE} (size ${DOCKER_ADDRESS_POOL_SIZE}) network pool need a Docker restart, which stops ${RUNNING_CONTAINERS} running container(s) and can change the docker0 bridge IP. To apply it, run the installer again with DOCKER_POOL_FORCE_OVERRIDE=true." + UI_STEP_DETAIL="skipped · containers running" else - # If force override is enabled or no existing configuration exists, - # create a new configuration with the specified address pools - echo " - Creating new Docker configuration with network pool: ${DOCKER_ADDRESS_POOL_BASE}/${DOCKER_ADDRESS_POOL_SIZE}" - cat >/etc/docker/daemon.json </etc/docker/daemon.json </dev/null 2>&1; then - echo " - Log configuration is up to date" - NEED_MERGE=false - else - # Create a configuration without address pools to preserve existing ones - cat >/etc/docker/daemon.json.coolify </etc/docker/daemon.json </etc/docker/daemon.json + echo " - Network pool configuration: ${DOCKER_ADDRESS_POOL_BASE} (size ${DOCKER_ADDRESS_POOL_SIZE})" echo " - Configuration updated - restarting Docker daemon..." restart_docker_service - else - echo " - Configuration is up to date" fi - else - echo " - Configuration is up to date" fi else - if [ "$NEED_MERGE" = true ]; then - echo " - Configuration updated - restarting Docker daemon..." - restart_docker_service - else - echo " - Configuration is up to date" - fi + echo " - Using the existing network pool ${DOCKER_ADDRESS_POOL_BASE} (size ${DOCKER_ADDRESS_POOL_SIZE})" + echo " - To override it: DOCKER_POOL_FORCE_OVERRIDE=true" fi step_done @@ -1070,6 +987,7 @@ step_done step_start "Checking SSH key for localhost access" SSH_KEY_DETAIL="existing key" +create_coolify_directories if [ ! -f ~/.ssh/authorized_keys ]; then mkdir -p ~/.ssh chmod 700 ~/.ssh diff --git a/other/nightly/upgrade.sh b/other/nightly/upgrade.sh index ce8fdb6589..4d594cb9e9 100644 --- a/other/nightly/upgrade.sh +++ b/other/nightly/upgrade.sh @@ -38,6 +38,65 @@ write_status() { echo "${step}|${message}|$(date -Iseconds)" > "$STATUS_FILE" } +# Print the first gateway of a network that Docker reports with a prefix length (fd00::1/64). +# Docker Engine 27.0 to 28.0.0 does this for IPv6 until the daemon restarts (https://github.com/moby/moby/pull/49520). +# The Docker CLI and Compose in the helper image fail on such a network with a ParseAddr error. +network_gateway_with_prefix() { + docker network inspect "$1" --format '{{range .IPAM.Config}}{{.Gateway}} {{end}}' 2>&1 | grep -oE '[0-9A-Fa-f:.]+/[0-9]+' | head -n1 +} + +# Make sure the coolify network exists and the helper image can read it. +# It is created when starting Coolify with docker compose +ensure_coolify_network() { + log "Checking Docker network 'coolify'..." + if ! docker network inspect coolify >/dev/null 2>&1; then + log "Network 'coolify' does not exist, creating..." + if ! docker network create --attachable --ipv6 coolify 2>/dev/null; then + log "Failed to create network with IPv6, trying without IPv6..." + docker network create --attachable coolify 2>/dev/null + log "Network 'coolify' created without IPv6" + elif [ -n "$(network_gateway_with_prefix coolify)" ]; then + # No container uses the new network yet, so it is safe to recreate it. + log "Docker reports the IPv6 gateway of network 'coolify' with a prefix length, recreating it without IPv6..." + docker network rm coolify >/dev/null 2>&1 + docker network create --attachable coolify 2>/dev/null + log "Network 'coolify' created without IPv6" + else + log "Network 'coolify' created with IPv6 support" + fi + return 0 + fi + + log "Network 'coolify' already exists" + local gateway + gateway=$(network_gateway_with_prefix coolify) + if [ -n "$gateway" ]; then + local message="Docker Engine $(docker version --format '{{.Server.Version}}' 2>/dev/null) reports the gateway of the 'coolify' network as ${gateway}, which Coolify cannot read (Docker Engine bug, fixed in 28.0.1). Run 'systemctl restart docker' (this restarts all containers) or upgrade Docker Engine, then upgrade Coolify again." + log "ERROR: ${message}" + write_status "error" "$message" + echo " ERROR: ${message}" + return 1 + fi +} + +# Pull an image. Retry with backoff, because the registry can rate limit many instances that pull at the same time. +pull_image() { + local image="$1" + local attempt=1 + local max_attempts=5 + local delay=15 + until docker pull "$image" >>"$LOGFILE" 2>&1; do + if [ "$attempt" -ge "$max_attempts" ]; then + return 1 + fi + log "Pull of $image failed (attempt $attempt/$max_attempts). Retrying in ${delay}s..." + echo " Pull failed (attempt $attempt/$max_attempts). Retrying in ${delay}s..." + sleep "$((delay + RANDOM % 10))" + attempt=$((attempt + 1)) + delay=$((delay * 2)) + done +} + echo "" echo "==========================================" echo " Coolify Upgrade - ${DATE}" @@ -159,21 +218,7 @@ fi log "Environment variables check complete" echo " Done." -# Make sure coolify network exists -# It is created when starting Coolify with docker compose -log "Checking Docker network 'coolify'..." -if ! docker network inspect coolify >/dev/null 2>&1; then - log "Network 'coolify' does not exist, creating..." - if ! docker network create --attachable --ipv6 coolify 2>/dev/null; then - log "Failed to create network with IPv6, trying without IPv6..." - docker network create --attachable coolify 2>/dev/null - log "Network 'coolify' created without IPv6" - else - log "Network 'coolify' created with IPv6 support" - fi -else - log "Network 'coolify' already exists" -fi +ensure_coolify_network || exit 1 mkdir -p /data/coolify/images/{avatars,project-icons} chown -R 9999:root /data/coolify/images @@ -205,7 +250,7 @@ echo " This may take a few minutes depending on your connection." HELPER_IMAGE="${REGISTRY_URL:-docker.io}/coollabsio/coolify-helper:${LATEST_HELPER_VERSION}" echo " - Pulling $HELPER_IMAGE..." log "Pulling image: $HELPER_IMAGE" -if docker pull "$HELPER_IMAGE" >>"$LOGFILE" 2>&1; then +if pull_image "$HELPER_IMAGE"; then log "Successfully pulled $HELPER_IMAGE" else log "ERROR: Failed to pull $HELPER_IMAGE" @@ -220,7 +265,7 @@ for IMAGE in $IMAGES; do if [ -n "$IMAGE" ]; then echo " - Pulling $IMAGE..." log "Pulling image: $IMAGE" - if docker pull "$IMAGE" >>"$LOGFILE" 2>&1; then + if pull_image "$IMAGE"; then log "Successfully pulled $IMAGE" else log "ERROR: Failed to pull $IMAGE" @@ -298,8 +343,16 @@ nohup bash -c " fi log 'Running docker compose up...' - docker run -v /data/coolify/source:/data/coolify/source -v /var/run/docker.sock:/var/run/docker.sock \${DOCKER_CONFIG_MOUNT} --rm \${REGISTRY_URL:-docker.io}/coollabsio/coolify-helper:\${LATEST_HELPER_VERSION} bash -c \"LATEST_IMAGE=\${LATEST_IMAGE} docker compose --env-file /data/coolify/source/.env \${COMPOSE_FILES} up -d --remove-orphans --wait --wait-timeout 60\" >>\"\$LOGFILE\" 2>&1 - log 'Docker compose up completed' + if docker run -v /data/coolify/source:/data/coolify/source -v /var/run/docker.sock:/var/run/docker.sock \${DOCKER_CONFIG_MOUNT} --rm \${REGISTRY_URL:-docker.io}/coollabsio/coolify-helper:\${LATEST_HELPER_VERSION} bash -c \"LATEST_IMAGE=\${LATEST_IMAGE} docker compose --env-file /data/coolify/source/.env \${COMPOSE_FILES} up -d --remove-orphans --wait --wait-timeout 60\" >>\"\$LOGFILE\" 2>&1; then + log 'Docker compose up completed' + elif [ \"\$(docker inspect --format '{{.State.Running}}' coolify 2>/dev/null)\" = 'true' ]; then + # A slow first start (for example long migrations) can pass the wait timeout. Coolify is still starting. + log 'Docker compose up did not finish waiting, but the coolify container is running' + else + log 'ERROR: Docker compose up failed and the coolify container is not running' + write_status 'error' \"Coolify did not start. Check the upgrade log: \$LOGFILE\" + exit 1 + fi # Final log entry echo '' >>\"\$LOGFILE\" diff --git a/package-lock.json b/package-lock.json index fd2834a7b9..4c99d1fa38 100644 --- a/package-lock.json +++ b/package-lock.json @@ -684,6 +684,72 @@ "node": ">=14.0.0" } }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/@emnapi/core": { + "version": "1.11.1", + "dev": true, + "inBundle": true, + "license": "MIT", + "optional": true, + "dependencies": { + "@emnapi/wasi-threads": "1.2.2", + "tslib": "^2.4.0" + } + }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/@emnapi/runtime": { + "version": "1.11.1", + "dev": true, + "inBundle": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/@emnapi/wasi-threads": { + "version": "1.2.2", + "dev": true, + "inBundle": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/@napi-rs/wasm-runtime": { + "version": "1.1.4", + "dev": true, + "inBundle": true, + "license": "MIT", + "optional": true, + "dependencies": { + "@tybys/wasm-util": "^0.10.1" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + }, + "peerDependencies": { + "@emnapi/core": "^1.7.1", + "@emnapi/runtime": "^1.7.1" + } + }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/@tybys/wasm-util": { + "version": "0.10.2", + "dev": true, + "inBundle": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@tailwindcss/oxide-wasm32-wasi/node_modules/tslib": { + "version": "2.8.1", + "dev": true, + "inBundle": true, + "license": "0BSD", + "optional": true + }, "node_modules/@tailwindcss/oxide-win32-arm64-msvc": { "version": "4.3.3", "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-win32-arm64-msvc/-/oxide-win32-arm64-msvc-4.3.3.tgz", @@ -1479,9 +1545,9 @@ } }, "node_modules/source-map-js": { - "version": "1.2.1", - "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz", - "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==", + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.2.tgz", + "integrity": "sha512-KGj/8Y43x35aZVDtt+J4mK1hoLGHULMYfSkODJNQjNDC3oW1PqPoxMwo0pLUsWM/UEGzON/NxeHywEfNXNP3Vw==", "dev": true, "license": "BSD-3-Clause", "engines": { @@ -1568,14 +1634,6 @@ "url": "https://github.com/sponsors/SuperchupuDev" } }, - "node_modules/tslib": { - "version": "2.8.1", - "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", - "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", - "dev": true, - "license": "0BSD", - "optional": true - }, "node_modules/tw-animate-css": { "version": "1.4.0", "resolved": "https://registry.npmjs.org/tw-animate-css/-/tw-animate-css-1.4.0.tgz", diff --git a/resources/css/app.css b/resources/css/app.css index c46f1ddf06..d9a36f40da 100644 --- a/resources/css/app.css +++ b/resources/css/app.css @@ -3022,13 +3022,14 @@ input[type="search"]::-webkit-search-results-decoration { width: 100%; } -.data-table-row.service-backup-table-grid { - background: var(--coollabs-base); - border-bottom: 1px solid var(--coollabs-fill); +.data-table-header.service-backup-table-grid { + border-bottom: 0; } -.data-table-row.service-backup-table-grid:last-child { - border-bottom: 0; +/* Rows are reordered with CSS order, so dividers must not depend on DOM position. */ +.data-table-row.service-backup-table-grid { + background: var(--coollabs-base); + border-top: 1px solid var(--coollabs-fill); } .data-table-row.service-backup-table-grid:hover { @@ -3279,9 +3280,9 @@ input[type="search"]::-webkit-search-results-decoration { } .backup-executions-table-grid { - grid-template-columns: 6.5rem minmax(7rem, 0.8fr) minmax(14rem, 1.5fr) 8.5rem 5rem minmax(8rem, 1fr) 5rem; + grid-template-columns: 6.5rem minmax(7rem, 0.8fr) minmax(14rem, 1.5fr) 10rem 5rem minmax(8rem, 1fr) 5rem; gap: 0.75rem; - min-width: 66rem; + min-width: 67.5rem; } .backup-executions-table-scroll { @@ -3290,7 +3291,7 @@ input[type="search"]::-webkit-search-results-decoration { /* Match the grid width so the log output spans the full scrolled row. */ .backup-execution-row { - min-width: 66rem; + min-width: 67.5rem; } .backup-execution-row:last-child > .data-table-row { @@ -3298,8 +3299,8 @@ input[type="search"]::-webkit-search-results-decoration { } .volume-backup-executions-grid { - grid-template-columns: 7.5rem minmax(14rem, 1fr) 8.5rem 7rem 9rem minmax(9rem, 0.7fr); - min-width: 50rem; + grid-template-columns: 7.5rem minmax(14rem, 1fr) 11rem 7rem 9rem minmax(9rem, 0.7fr); + min-width: 52.5rem; } .clone-destinations-table-grid { @@ -3345,7 +3346,7 @@ input[type="search"]::-webkit-search-results-decoration { } .backup-executions-table-grid { - grid-template-columns: 7.5rem minmax(9rem, 0.8fr) minmax(14rem, 1.5fr) 8.5rem minmax(8rem, 1fr) 5rem; + grid-template-columns: 7.5rem minmax(9rem, 0.8fr) minmax(14rem, 1.5fr) 10rem minmax(8rem, 1fr) 5rem; } .backup-executions-table-grid > :nth-child(5) { diff --git a/resources/views/components/server/status-summary.blade.php b/resources/views/components/server/status-summary.blade.php index ea2306397e..854c2525f8 100644 --- a/resources/views/components/server/status-summary.blade.php +++ b/resources/views/components/server/status-summary.blade.php @@ -47,8 +47,8 @@ -