Upgrading kloud
Open, Unbreak Now!Public
Actions

Assigned To

Authored By

	keithzg
	Sep 15 2024, 6:36 PM

Description

It's going alright, but going worse now...

Related Objects
Search...

Status	Assigned	Task
		Restricted Maniphest Task
Open	keithzg	T189 Upgrading kloud
		Restricted Maniphest Task
Open	keithzg	T202 Make kloud a bit more robust

Event Timeline

keithzg created this task.Sep 15 2024, 6:36 PM

Current error:

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

[Mon Sep 16 00:35:57.249808 2024] [php:error] [pid 649033] [client 184.70.164.246:45258] PHP Fatal error:  Uncaught Doctrine\\DBAL\\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [2002] Connection refused in /mnt/kloud_storage_200gb/owncloud/lib/private/DB/Connection.php:147\nStack trace:\n#0 /mnt/kloud_storage_200gb/owncloud/3rdparty/doctrine/dbal/src/Connection.php(1531): OC\\DB\\Connection->connect()\n#1 /mnt/kloud_storage_200gb/owncloud/3rdparty/doctrine/dbal/src/Connection.php(1029): Doctrine\\DBAL\\Connection->getWrappedConnection()\n#2 /mnt/kloud_storage_200gb/owncloud/lib/private/DB/Connection.php(267): Doctrine\\DBAL\\Connection->executeQuery()\n#3 /mnt/kloud_storage_200gb/owncloud/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php(345): OC\\DB\\Connection->executeQuery()\n#4 /mnt/kloud_storage_200gb/owncloud/lib/private/DB/QueryBuilder/QueryBuilder.php(280): Doctrine\\DBAL\\Query\\QueryBuilder->execute()\n#5 /mnt/kloud_storage_200gb/owncloud/lib/private/AppConfig.php(421): OC\\DB\\QueryBuilder\\QueryBuilder->execute()\n#6 /mnt/kloud_storage_200gb/owncloud/lib/private/AppConfig.php(187): OC\\AppConfig->loadConfigValues()\n#7 /mnt/kloud_storage_200gb/owncloud/lib/private/AppConfig.php(377): OC\\AppConfig->getApps()\n#8 /mnt/kloud_storage_200gb/owncloud/lib/private/legacy/OC_App.php(803): OC\\AppConfig->getValues()\n#9 /mnt/kloud_storage_200gb/owncloud/lib/private/Server.php(737): OC_App::getAppVersions()\n#10 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(171): OC\\Server->OC\\{closure}()\n#11 /mnt/kloud_storage_200gb/owncloud/3rdparty/pimple/pimple/src/Pimple/Container.php(122): OC\\AppFramework\\Utility\\SimpleContainer->OC\\AppFramework\\Utility\\{closure}()\n#12 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(138): Pimple\\Container->offsetGet()\n#13 /mnt/kloud_storage_200gb/owncloud/lib/private/ServerContainer.php(171): OC\\AppFramework\\Utility\\SimpleContainer->query()\n#14 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(65): OC\\ServerContainer->query()\n#15 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(193): OC\\AppFramework\\Utility\\SimpleContainer->get()\n#16 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(171): OC\\AppFramework\\Utility\\SimpleContainer->OC\\AppFramework\\Utility\\{closure}()\n#17 /mnt/kloud_storage_200gb/owncloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\\AppFramework\\Utility\\SimpleContainer->OC\\AppFramework\\Utility\\{closure}()\n#18 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(138): Pimple\\Container->offsetGet()\n#19 /mnt/kloud_storage_200gb/owncloud/lib/private/ServerContainer.php(171): OC\\AppFramework\\Utility\\SimpleContainer->query()\n#20 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(65): OC\\ServerContainer->query()\n#21 /mnt/kloud_storage_200gb/owncloud/lib/private/Server.php(1131): OC\\AppFramework\\Utility\\SimpleContainer->get()\n#22 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(171): OC\\Server->OC\\{closure}()\n#23 /mnt/kloud_storage_200gb/owncloud/3rdparty/pimple/pimple/src/Pimple/Container.php(122): OC\\AppFramework\\Utility\\SimpleContainer->OC\\AppFramework\\Utility\\{closure}()\n#24 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(138): Pimple\\Container->offsetGet()\n#25 /mnt/kloud_storage_200gb/owncloud/lib/private/ServerContainer.php(171): OC\\AppFramework\\Utility\\SimpleContainer->query()\n#26 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(65): OC\\ServerContainer->query()\n#27 /mnt/kloud_storage_200gb/owncloud/lib/private/Server.php(2091): OC\\AppFramework\\Utility\\SimpleContainer->get()\n#28 /mnt/kloud_storage_200gb/owncloud/lib/private/Files/View.php(108): OC\\Server->getLockingProvider()\n#29 /mnt/kloud_storage_200gb/owncloud/lib/private/Server.php(471): OC\\Files\\View->__construct()\n#30 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(171): OC\\Server->OC\\{closure}()\n#31 /mnt/kloud_storage_200gb/owncloud/3rdparty/pimple/pimple/src/Pimple/Container.php(122): OC\\AppFramework\\Utility\\SimpleContainer->OC\\AppFramework\\Utility\\{closure}()\n#32 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(138): Pimple\\Container->offsetGet()\n#33 /mnt/kloud_storage_200gb/owncloud/lib/private/ServerContainer.php(171): OC\\AppFramework\\Utility\\SimpleContainer->query()\n#34 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(65): OC\\ServerContainer->query()\n#35 /mnt/kloud_storage_200gb/owncloud/lib/private/Server.php(1493): OC\\AppFramework\\Utility\\SimpleContainer->get()\n#36 /mnt/kloud_storage_200gb/owncloud/lib/base.php(635): OC\\Server->boot()\n#37 /mnt/kloud_storage_200gb/owncloud/lib/base.php(1196): OC::init()\n#38 /mnt/kloud_storage_200gb/owncloud/ocs/v1.php(31): require_once('...')\n#39 /mnt/kloud_storage_200gb/owncloud/ocs/v2.php(23): require_once('...')\n#40 {main}\n  thrown in /mnt/kloud_storage_200gb/owncloud/lib/private/DB/Connection.php on line 147

ah, well

root@kloud:/var/www/owncloud# service mariadb status
× mariadb.service - MariaDB 10.6.18 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/mariadb.service.d
             └─migrated-from-my.cnf-settings.conf
     Active: failed (Result: oom-kill) since Mon 2024-09-16 00:30:11 UTC; 11min ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 1044 ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=killed, signal=KILL)
   Main PID: 1044 (code=killed, signal=KILL)
     Status: "Taking your SQL requests now..."
        CPU: 1min 9.381s

Sep 15 02:00:57 kloud systemd[1]: /etc/systemd/system/mariadb.service.d/migrated-from-my.cnf-settings.conf:7: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Sep 15 02:00:57 kloud systemd[1]: /etc/systemd/system/mariadb.service.d/migrated-from-my.cnf-settings.conf:8: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Sep 15 02:00:58 kloud systemd[1]: /etc/systemd/system/mariadb.service.d/migrated-from-my.cnf-settings.conf:7: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Sep 15 02:00:58 kloud systemd[1]: /etc/systemd/system/mariadb.service.d/migrated-from-my.cnf-settings.conf:8: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Sep 15 02:00:58 kloud systemd[1]: /etc/systemd/system/mariadb.service.d/migrated-from-my.cnf-settings.conf:7: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Sep 15 02:00:58 kloud systemd[1]: /etc/systemd/system/mariadb.service.d/migrated-from-my.cnf-settings.conf:8: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Sep 16 00:30:11 kloud systemd[1]: mariadb.service: A process of this unit has been killed by the OOM killer.
Sep 16 00:30:11 kloud systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
Sep 16 00:30:11 kloud systemd[1]: mariadb.service: Failed with result 'oom-kill'.
Sep 16 00:30:11 kloud systemd[1]: mariadb.service: Consumed 1min 9.381s CPU time.

Got Nextcloud to 27.1.11, latest is 29.0.7 with 30 in release candidates.

The recurring OOM killing of the database server isn't the only issue, alas. As reported to me by Val in Discord, https://kloud.keithzg.ca/pymusicleague/spotify_track_availability.php isn't working right:

<@230569237150826496> your availability tracker seems to not be working anymore. It keeps giving me responses saying songs I have access to are unavailable anywhere including Ireland

keithzg changed the visibility from "All Users" to "Public (No Login Required)".Sep 15 2024, 10:11 PM

Hmmmm the immediate problem was I didn't have the tekore module installed, but when I install it then I get another error:

Traceback (most recent call last):
  File "/var/www/html/pymusicleague/./check_track_availability.py", line 36, in <module>
    track_id = track.asbuiltin()['id']
  File "/usr/local/lib/python3.10/dist-packages/pydantic/main.py", line 853, in __getattr__
    raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')
AttributeError: 'FullTrack' object has no attribute 'asbuiltin'

Maybe Tekore has changed their classes? Clearly I need to either update my Python logic accordingly, or just rewrite the checks in pure PHP myself.

Yeah, Tekore 4.6.1 worked but earlier 5.x versions didn't, I mean cheers to that project for actually taking semantic versioning seriously?

Current status:

Current version is 28.0.10.

Update to Nextcloud 29.0.7 available. (channel: "stable")

Currently:

Current version is 29.0.7.

Update to Nextcloud 30.0.0 available. (channel: "stable")

Yeah naw that's good, lets not jump right onto a .0.0 revision.

Probably time then to do another OS upgrade . . .

I suppose I should perhaps clean up the Nextcloud warnings first.

Okay, first that jumps out at me because it seems quite doable is:

One or more mimetype migrations are available. Occasionally new mimetypes are added to better handle certain file types. Migrating the mimetypes take a long time on larger instances so this is not done automatically during upgrades. Use the command occ maintenance:repair --include-expensive to perform the migrations.

That went fine other than a warning about phone numbers that's also in the warnings,

Your installation has no default phone region set. This is required to validate phone numbers in the profile settings without a country code. To allow numbers without a country code, please add "default_phone_region" with the respective ISO 3166-1 code of the region to your config file. For more details see the documentation ↗.

Funny to cite Wikipedia as "the documentation" but that's trivial enough, this fixed it:

config/config.php

'default_phone_region' => '1',

sudo -u www-data php -f occ db:add-missing-indices fixed the most verbose warning, unsurprisingly.

Your webserver is not set up to serve .js.map files. Without these files, JavaScript Source Maps won't function properly, making it more challenging to troubleshoot and debug any issues that may arise.

But actually, it seems fine:

$ curl -I https://kloud.keithzg.ca/owncloud/apps/settings/js/map-test.js.map
HTTP/1.1 200 OK
Date: Thu, 19 Sep 2024 20:57:46 GMT
Server: Apache/2.4.52 (Ubuntu)
Referrer-Policy: no-referrer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Robots-Tag: noindex, nofollow
X-XSS-Protection: 1; mode=block
Last-Modified: Thu, 12 Sep 2024 12:29:14 GMT
ETag: "5a-621eb41215a80"
Accept-Ranges: bytes
Content-Length: 90
Vary: Accept-Encoding
Content-Type: text/javascript

So maybe the problem lies more in

Could not check for JavaScript support via any of your trusted_domains nor overwrite.cli.url. This may be the result of a server-side DNS mismatch or outbound firewall rule. Please check manually if your webserver serves .mjs files using the JavaScript MIME type. To allow this check to run you have to make sure that your webserver can connect to itself. Therefor it must be able to resolve and connect to at least one its trusted_domains or the overwrite.cli.url.

I thiiiink trusted_domains is right, even though I'm using different array syntax, as is overwrite-cli-url.

'overwritewebroot' => '/owncloud',
...
  'trusted_domains' => 
  array (
    0 => 'kloud.keithzg.ca',
  ),
...
'overwrite.cli.url' => 'https://kloud.keithzg.ca/owncloud',

Nothing in the actual Apache log, so lets check the Nextcloud log, ahhh great it's blobs of JSON, okay parsed it, it says in part:

{"reqId":"b5Jj4cf0UQlxbaWZdxRA","level":0,"time":"2024-09-19T21:16:41+00:00","remoteAddr":"198.48.134.113","user":"keithzg","app":"settings","method":"GET","url":"/owncloud/index.php/settings/ajax/checksetup","message":"Can not connect to local server for running setup checks","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36","version":"29.0.7.1","exception":{"Exception":"GuzzleHttp\\Exception\\RequestException","Message":"cURL error 3:  (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for /owncloud/.well-known/webfinger","Code":0,

I am . . . not sure if that's the expected URL but it kinda looks right to me other than being a URI rather than a URL? Ah, that seems to be it; the error is pointing towards a no longer existing doc page (hmm if that's still true I should submit a patch) but https://curl.se/libcurl/c/libcurl-errors.html says for error 3:

The URL was not properly formatted.

But I don't see how with my current settings I would do anything differently that would in turn correct that. Hmm.

Maybe https://help.nextcloud.com/t/how-to-make-apache-serve-js-map/202468/10 is the key:

It seems something doesn’t work correct when using subfolders, at least in my case some requests are made to domain/nextcloud/, while others are made to domain/nextcloud/nextcloud/
...
i ended up adding a redirect rule to my .htaccess file:
Redirect “/nextcloud/nextcloud” “/nextcloud”
I’m certain there are better ways to fix this but it’s well enough for my personal needs.

keithzg added a subtask: Restricted Maniphest Task.Sep 20 2024, 12:33 AM

keithzg closed subtask Restricted Maniphest Task as Resolved.

keithzg mentioned this in T202: Make kloud a bit more robust.Nov 12 2024, 6:36 PM

Well, Redirect “/nextcloud/nextcloud” “/nextcloud” gave me /var/www/owncloud/.htaccess: Redirect to non-URL, and RewriteRule “/owncloud/owncloud” “/owncloud” has changed nothing (I didn't bother yet looking up the syntax to make sure I'm doing that right though, I have a headache anyways!).

I'm gonna go and update Nextcloud anyways! At least to the latest point release for 29 (going from 29.0.7 to 29.0.8) but I'm tempted to just go to 30.

The current stable News app needs at least PHP 8.2 itself, so upgrading the server itself to 24.04! Fingers crossed it doesn't go too wrong!

Huh, once I had finally remembered enough times to get past all the "hey, wanna replace this config file with an updated default one?" prompts I'd always say no to, including the final one to remove obsoleted packages, it booted back up into Ubuntu Server 24.04 just fine (well, after switching over the php8.1 mod enabled in the webserver to 24.04's php8.3).

I bumped up the upload file limit in the shared custom PHP config, odd that it was getting away without such a setting before honestly.

Ironically I didn't upgrade to Nextcloud v30, which would have been the less drastic overhaul of the software stack here! There's some point releases out now though so I'm less worried about it than I would've been, but I also don't know of anything new in v30 that I care about.

Upgrading kloudOpen, Unbreak Now!PublicActions

Description

Related ObjectsSearch...

Event Timeline

Upgrading kloud
Open, Unbreak Now!Public
Actions

Related Objects
Search...