It's going alright, but going worse now...
Description
Description
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Restricted Maniphest Task | ||||
| Open | keithzg | T189 Upgrading kloud | ||
| Restricted Maniphest Task | ||||
| Open | keithzg | T202 Make kloud a bit more robust |
Event Timeline
Comment Actions
Current error:
Internal Server Error The server encountered an internal error and was unable to complete your request. Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report. More details can be found in the server log.
[Mon Sep 16 00:35:57.249808 2024] [php:error] [pid 649033] [client 184.70.164.246:45258] PHP Fatal error: Uncaught Doctrine\\DBAL\\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [2002] Connection refused in /mnt/kloud_storage_200gb/owncloud/lib/private/DB/Connection.php:147\nStack trace:\n#0 /mnt/kloud_storage_200gb/owncloud/3rdparty/doctrine/dbal/src/Connection.php(1531): OC\\DB\\Connection->connect()\n#1 /mnt/kloud_storage_200gb/owncloud/3rdparty/doctrine/dbal/src/Connection.php(1029): Doctrine\\DBAL\\Connection->getWrappedConnection()\n#2 /mnt/kloud_storage_200gb/owncloud/lib/private/DB/Connection.php(267): Doctrine\\DBAL\\Connection->executeQuery()\n#3 /mnt/kloud_storage_200gb/owncloud/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php(345): OC\\DB\\Connection->executeQuery()\n#4 /mnt/kloud_storage_200gb/owncloud/lib/private/DB/QueryBuilder/QueryBuilder.php(280): Doctrine\\DBAL\\Query\\QueryBuilder->execute()\n#5 /mnt/kloud_storage_200gb/owncloud/lib/private/AppConfig.php(421): OC\\DB\\QueryBuilder\\QueryBuilder->execute()\n#6 /mnt/kloud_storage_200gb/owncloud/lib/private/AppConfig.php(187): OC\\AppConfig->loadConfigValues()\n#7 /mnt/kloud_storage_200gb/owncloud/lib/private/AppConfig.php(377): OC\\AppConfig->getApps()\n#8 /mnt/kloud_storage_200gb/owncloud/lib/private/legacy/OC_App.php(803): OC\\AppConfig->getValues()\n#9 /mnt/kloud_storage_200gb/owncloud/lib/private/Server.php(737): OC_App::getAppVersions()\n#10 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(171): OC\\Server->OC\\{closure}()\n#11 /mnt/kloud_storage_200gb/owncloud/3rdparty/pimple/pimple/src/Pimple/Container.php(122): OC\\AppFramework\\Utility\\SimpleContainer->OC\\AppFramework\\Utility\\{closure}()\n#12 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(138): Pimple\\Container->offsetGet()\n#13 /mnt/kloud_storage_200gb/owncloud/lib/private/ServerContainer.php(171): OC\\AppFramework\\Utility\\SimpleContainer->query()\n#14 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(65): OC\\ServerContainer->query()\n#15 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(193): OC\\AppFramework\\Utility\\SimpleContainer->get()\n#16 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(171): OC\\AppFramework\\Utility\\SimpleContainer->OC\\AppFramework\\Utility\\{closure}()\n#17 /mnt/kloud_storage_200gb/owncloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\\AppFramework\\Utility\\SimpleContainer->OC\\AppFramework\\Utility\\{closure}()\n#18 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(138): Pimple\\Container->offsetGet()\n#19 /mnt/kloud_storage_200gb/owncloud/lib/private/ServerContainer.php(171): OC\\AppFramework\\Utility\\SimpleContainer->query()\n#20 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(65): OC\\ServerContainer->query()\n#21 /mnt/kloud_storage_200gb/owncloud/lib/private/Server.php(1131): OC\\AppFramework\\Utility\\SimpleContainer->get()\n#22 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(171): OC\\Server->OC\\{closure}()\n#23 /mnt/kloud_storage_200gb/owncloud/3rdparty/pimple/pimple/src/Pimple/Container.php(122): OC\\AppFramework\\Utility\\SimpleContainer->OC\\AppFramework\\Utility\\{closure}()\n#24 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(138): Pimple\\Container->offsetGet()\n#25 /mnt/kloud_storage_200gb/owncloud/lib/private/ServerContainer.php(171): OC\\AppFramework\\Utility\\SimpleContainer->query()\n#26 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(65): OC\\ServerContainer->query()\n#27 /mnt/kloud_storage_200gb/owncloud/lib/private/Server.php(2091): OC\\AppFramework\\Utility\\SimpleContainer->get()\n#28 /mnt/kloud_storage_200gb/owncloud/lib/private/Files/View.php(108): OC\\Server->getLockingProvider()\n#29 /mnt/kloud_storage_200gb/owncloud/lib/private/Server.php(471): OC\\Files\\View->__construct()\n#30 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(171): OC\\Server->OC\\{closure}()\n#31 /mnt/kloud_storage_200gb/owncloud/3rdparty/pimple/pimple/src/Pimple/Container.php(122): OC\\AppFramework\\Utility\\SimpleContainer->OC\\AppFramework\\Utility\\{closure}()\n#32 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(138): Pimple\\Container->offsetGet()\n#33 /mnt/kloud_storage_200gb/owncloud/lib/private/ServerContainer.php(171): OC\\AppFramework\\Utility\\SimpleContainer->query()\n#34 /mnt/kloud_storage_200gb/owncloud/lib/private/AppFramework/Utility/SimpleContainer.php(65): OC\\ServerContainer->query()\n#35 /mnt/kloud_storage_200gb/owncloud/lib/private/Server.php(1493): OC\\AppFramework\\Utility\\SimpleContainer->get()\n#36 /mnt/kloud_storage_200gb/owncloud/lib/base.php(635): OC\\Server->boot()\n#37 /mnt/kloud_storage_200gb/owncloud/lib/base.php(1196): OC::init()\n#38 /mnt/kloud_storage_200gb/owncloud/ocs/v1.php(31): require_once('...')\n#39 /mnt/kloud_storage_200gb/owncloud/ocs/v2.php(23): require_once('...')\n#40 {main}\n thrown in /mnt/kloud_storage_200gb/owncloud/lib/private/DB/Connection.php on line 147Comment Actions
ah, well
root@kloud:/var/www/owncloud# service mariadb status
× mariadb.service - MariaDB 10.6.18 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: failed (Result: oom-kill) since Mon 2024-09-16 00:30:11 UTC; 11min ago
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Process: 1044 ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=killed, signal=KILL)
Main PID: 1044 (code=killed, signal=KILL)
Status: "Taking your SQL requests now..."
CPU: 1min 9.381s
Sep 15 02:00:57 kloud systemd[1]: /etc/systemd/system/mariadb.service.d/migrated-from-my.cnf-settings.conf:7: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Sep 15 02:00:57 kloud systemd[1]: /etc/systemd/system/mariadb.service.d/migrated-from-my.cnf-settings.conf:8: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Sep 15 02:00:58 kloud systemd[1]: /etc/systemd/system/mariadb.service.d/migrated-from-my.cnf-settings.conf:7: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Sep 15 02:00:58 kloud systemd[1]: /etc/systemd/system/mariadb.service.d/migrated-from-my.cnf-settings.conf:8: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Sep 15 02:00:58 kloud systemd[1]: /etc/systemd/system/mariadb.service.d/migrated-from-my.cnf-settings.conf:7: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Sep 15 02:00:58 kloud systemd[1]: /etc/systemd/system/mariadb.service.d/migrated-from-my.cnf-settings.conf:8: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
Sep 16 00:30:11 kloud systemd[1]: mariadb.service: A process of this unit has been killed by the OOM killer.
Sep 16 00:30:11 kloud systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
Sep 16 00:30:11 kloud systemd[1]: mariadb.service: Failed with result 'oom-kill'.
Sep 16 00:30:11 kloud systemd[1]: mariadb.service: Consumed 1min 9.381s CPU time.Comment Actions
The recurring OOM killing of the database server isn't the only issue, alas. As reported to me by Val in Discord, https://kloud.keithzg.ca/pymusicleague/spotify_track_availability.php isn't working right:
<@230569237150826496> your availability tracker seems to not be working anymore. It keeps giving me responses saying songs I have access to are unavailable anywhere including Ireland
Comment Actions
Hmmmm the immediate problem was I didn't have the tekore module installed, but when I install it then I get another error:
Traceback (most recent call last):
File "/var/www/html/pymusicleague/./check_track_availability.py", line 36, in <module>
track_id = track.asbuiltin()['id']
File "/usr/local/lib/python3.10/dist-packages/pydantic/main.py", line 853, in __getattr__
raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')
AttributeError: 'FullTrack' object has no attribute 'asbuiltin'Maybe Tekore has changed their classes? Clearly I need to either update my Python logic accordingly, or just rewrite the checks in pure PHP myself.
Comment Actions
Yeah, Tekore 4.6.1 worked but earlier 5.x versions didn't, I mean cheers to that project for actually taking semantic versioning seriously?
Comment Actions
Current status:
Current version is 28.0.10. Update to Nextcloud 29.0.7 available. (channel: "stable")
Comment Actions
Currently:
Current version is 29.0.7. Update to Nextcloud 30.0.0 available. (channel: "stable")
Yeah naw that's good, lets not jump right onto a .0.0 revision.
Probably time then to do another OS upgrade . . .
Comment Actions
I suppose I should perhaps clean up the Nextcloud warnings first.
Okay, first that jumps out at me because it seems quite doable is:
- One or more mimetype migrations are available. Occasionally new mimetypes are added to better handle certain file types. Migrating the mimetypes take a long time on larger instances so this is not done automatically during upgrades. Use the command occ maintenance:repair --include-expensive to perform the migrations.
That went fine other than a warning about phone numbers that's also in the warnings,
- Your installation has no default phone region set. This is required to validate phone numbers in the profile settings without a country code. To allow numbers without a country code, please add "default_phone_region" with the respective ISO 3166-1 code of the region to your config file. For more details see the documentation ↗.
Funny to cite Wikipedia as "the documentation" but that's trivial enough, this fixed it:
config/config.php
'default_phone_region' => '1',
sudo -u www-data php -f occ db:add-missing-indices fixed the most verbose warning, unsurprisingly.
- Your webserver is not set up to serve .js.map files. Without these files, JavaScript Source Maps won't function properly, making it more challenging to troubleshoot and debug any issues that may arise.
But actually, it seems fine:
$ curl -I https://kloud.keithzg.ca/owncloud/apps/settings/js/map-test.js.map HTTP/1.1 200 OK Date: Thu, 19 Sep 2024 20:57:46 GMT Server: Apache/2.4.52 (Ubuntu) Referrer-Policy: no-referrer X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN X-Permitted-Cross-Domain-Policies: none X-Robots-Tag: noindex, nofollow X-XSS-Protection: 1; mode=block Last-Modified: Thu, 12 Sep 2024 12:29:14 GMT ETag: "5a-621eb41215a80" Accept-Ranges: bytes Content-Length: 90 Vary: Accept-Encoding Content-Type: text/javascript
So maybe the problem lies more in
- Could not check for JavaScript support via any of your trusted_domains nor overwrite.cli.url. This may be the result of a server-side DNS mismatch or outbound firewall rule. Please check manually if your webserver serves .mjs files using the JavaScript MIME type. To allow this check to run you have to make sure that your webserver can connect to itself. Therefor it must be able to resolve and connect to at least one its trusted_domains or the overwrite.cli.url.
I thiiiink trusted_domains is right, even though I'm using different array syntax, as is overwrite-cli-url.
'overwritewebroot' => '/owncloud', ... 'trusted_domains' => array ( 0 => 'kloud.keithzg.ca', ), ... 'overwrite.cli.url' => 'https://kloud.keithzg.ca/owncloud',
Nothing in the actual Apache log, so lets check the Nextcloud log, ahhh great it's blobs of JSON, okay parsed it, it says in part:
{"reqId":"b5Jj4cf0UQlxbaWZdxRA","level":0,"time":"2024-09-19T21:16:41+00:00","remoteAddr":"198.48.134.113","user":"keithzg","app":"settings","method":"GET","url":"/owncloud/index.php/settings/ajax/checksetup","message":"Can not connect to local server for running setup checks","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36","version":"29.0.7.1","exception":{"Exception":"GuzzleHttp\\Exception\\RequestException","Message":"cURL error 3: (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for /owncloud/.well-known/webfinger","Code":0,
I am . . . not sure if that's the expected URL but it kinda looks right to me other than being a URI rather than a URL? Ah, that seems to be it; the error is pointing towards a no longer existing doc page (hmm if that's still true I should submit a patch) but https://curl.se/libcurl/c/libcurl-errors.html says for error 3:
The URL was not properly formatted.
But I don't see how with my current settings I would do anything differently that would in turn correct that. Hmm.
Maybe https://help.nextcloud.com/t/how-to-make-apache-serve-js-map/202468/10 is the key:
It seems something doesn’t work correct when using subfolders, at least in my case some requests are made to domain/nextcloud/, while others are made to domain/nextcloud/nextcloud/
...
i ended up adding a redirect rule to my .htaccess file:Redirect “/nextcloud/nextcloud” “/nextcloud”I’m certain there are better ways to fix this but it’s well enough for my personal needs.