Disk became read-only after some heavy I/O

[askfongjojo]

The VM was newly created on rack3 after upgrading to R17. There was no sled reboot involved during the lifetime of the VM based on the sled uptimes.

[ 4700.636735] nvme0n1: I/O Cmd(0x2) @ LBA 131209936, 200 blocks, I/O Error (sct 0x3 / sc 0x71) 
[ 4700.639133] I/O error, dev nvme0n1, sector 131209936 op 0x0:(READ) flags 0x80700 phys_seg 25 prio class 0
[ 4730.850358] I/O error, dev nvme0n1, sector 167431176 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0
[ 4730.850361] I/O error, dev nvme0n1, sector 4217392 op 0x1:(WRITE) flags 0x9800 phys_seg 7 prio class 2
[ 4730.850383] I/O error, dev nvme0n1, sector 10630288 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[ 4730.850402] Buffer I/O error on device nvme0n1p1, logical block 1066386
[ 4730.850444] I/O error, dev nvme0n1, sector 84999240 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[ 4730.850453] Buffer I/O error on device nvme0n1p1, logical block 10362505
[ 4730.852930] I/O error, dev nvme0n1, sector 167432224 op 0x1:(WRITE) flags 0x0 phys_seg 92 prio class 0
[ 4730.852936] I/O error, dev nvme0n1, sector 167432976 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0
[ 4730.852943] I/O error, dev nvme0n1, sector 167434032 op 0x1:(WRITE) flags 0x0 phys_seg 128 prio class 0
[ 4730.855458] Aborting journal on device nvme0n1p1-8.
[ 4730.857765] I/O error, dev nvme0n1, sector 167435144 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0
[ 4730.857768] I/O error, dev nvme0n1, sector 167436216 op 0x1:(WRITE) flags 0x0 phys_seg 128 prio class 0
[ 4730.857773] I/O error, dev nvme0n1, sector 167437352 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0
[ 4730.861980] EXT4-fs error (device nvme0n1p1): ext4_journal_check_start:84: comm rs:main Q:Reg: Detected aborted journal
[ 4730.862702] EXT4-fs error (device nvme0n1p1): ext4_journal_check_start:84: comm kworker/u32:1: Detected aborted journal
[ 4730.887051] Buffer I/O error on dev nvme0n1p1, logical block 262144, lost sync page write
[ 4730.887414] EXT4-fs (nvme0n1p1): ext4_do_writepages: jbd2_start: 992 pages, ino 76523; err -5
[ 4730.889200] JBD2: I/O error when updating journal superblock for nvme0n1p1-8.
[ 4730.892216] Buffer I/O error on device nvme0n1p1, logical block 36366
[ 4730.893367] Buffer I/O error on device nvme0n1p1, logical block 1066394
[ 4730.895082] Buffer I/O error on device nvme0n1p1, logical block 1066378
[ 4730.896794] Buffer I/O error on device nvme0n1p1, logical block 1066404
[ 4730.898476] Buffer I/O error on device nvme0n1p1, logical block 1066386
[ 4730.898548] Buffer I/O error on device nvme0n1p1, logical block 1066393
[ 4730.898554] Buffer I/O error on device nvme0n1p1, logical block 1066394
[ 4730.898567] Buffer I/O error on device nvme0n1p1, logical block 1066396
[ 4730.898775] EXT4-fs (nvme0n1p1): ext4_do_writepages: jbd2_start: 9223372036854775739 pages, ino 76523; err -5
[ 4730.914225] Buffer I/O error on dev nvme0n1p1, logical block 0, lost sync page write
[ 4730.916662] EXT4-fs (nvme0n1p1): I/O error while writing superblock
[ 4730.916717] EXT4-fs (nvme0n1p1): previous I/O error to superblock detected
[ 4730.918436] EXT4-fs (nvme0n1p1): Remounting filesystem read-only
[ 4730.920204] Buffer I/O error on dev nvme0n1p1, logical block 0, lost sync page write
[ 4730.920210] EXT4-fs (nvme0n1p1): I/O error while writing superblock
[ 4730.920212] EXT4-fs (nvme0n1p1): Remounting filesystem read-only
[ 4736.516349] EXT4-fs (nvme0n1p16): shut down requested (2)
[ 4736.519492] Aborting journal on device nvme0n1p16-8.
[ 4736.522331] Buffer I/O error on dev nvme0n1p16, logical block 65536, lost sync page write
[ 4736.524773] JBD2: I/O error when updating journal superblock for nvme0n1p16-8.
 4736.516349] EXT4-fs (nvme0n1p16): shut down requested (2)
[ 4736.519492] Aborting journal on device nvme0n1p16-8.
[ 4736.522331] Buffer I/O error on dev nvme0n1p16, logical block 65536, lost sync page write
[ 4736.524773] JBD2: I/O error when updating journal superblock for nvme0n1p16-8.

Related VM/VMM/disk info:

root@oxz_switch1:~# omdb db instance info e8353858-3eef-45e9-89a5-40c0c67a7051

== INSTANCE ====================================================================
                        ID: e8353858-3eef-45e9-89a5-40c0c67a7051
                project ID: af8f74e9-0dcd-4d65-977a-ef871e2e3ee3
                      name: test
               description: 
                created at: 2025-11-13 20:36:53.356195 UTC
          last modified at: 2025-11-13 20:36:53.356195 UTC

== CONFIGURATION ===============================================================
                     vCPUs: 16
                    memory: 32 GiB
                  hostname: test
                 boot disk: Some(79720eb5-2775-4164-b002-d437f4c29009)
              auto-restart:
                  InstanceAutoRestart {
                      policy: None,
                      cooldown: None,
                  }

== RUNTIME STATE ===============================================================
               nexus state: Vmm
(i)     external API state: Running
            intended state: running
           last updated at: 2025-11-13T20:36:53.356195Z (generation 3)
       needs reincarnation: false
             karmic status: saṃsāra (reincarnation enabled)
      last reincarnated at: None
             active VMM ID: Some(7bfb54cc-743c-4c50-874a-ff863176c10b)
             target VMM ID: None
              migration ID: None
              updater lock: UNLOCKED at generation: 1

== ACTIVE VMM ==================================================================
                        ID: 7bfb54cc-743c-4c50-874a-ff863176c10b
               instance ID: e8353858-3eef-45e9-89a5-40c0c67a7051
                created at: 2025-11-13 20:37:03.590631 UTC
                     state: running
                updated at: 2025-11-13T20:37:13.913481Z (generation 4)
          propolis address: fd00:1122:3344:10f::1:4e2:12400
                   sled ID: 06535bc5-f5de-4cfd-99f6-097f806530a8
              CPU platform: AmdMilan

== ATTACHED DISKS ==============================================================
# ID                                   SIZE    STATE    NAME              
0 79720eb5-2775-4164-b002-d437f4c29009 100 GiB attached test-noble-7d59a7 

root@oxz_switch1:~# omdb db disks info 79720eb5-2775-4164-b002-d437f4c29009

HOST_SERIAL DISK_NAME         INSTANCE_NAME PROPOLIS_ZONE                                            VOLUME_ID                            DISK_STATE IMPORT_ADDRESS 
BRM42220079 test-noble-7d59a7 test          oxz_propolis-server_7bfb54cc-743c-4c50-874a-ff863176c10b 0991860d-b9bc-491c-a7d2-01e8acd7200a attached   -              
HOST_SERIAL REGION                               DATASET                              PHYSICAL_DISK                        
BRM42220078 75162594-b694-4706-a4d6-56a65cb69f9a 06233bfe-a857-4819-aefe-212af9eeb90f 436120fa-5b4e-46fa-8b62-9210a11e2ca0 
BRM44220016 1f776e93-edea-420f-9a7f-6b2b278c99d6 0d796c52-37ca-490d-b42f-dcc22fe5fd6b 9dc677ed-b50c-4064-b0c3-205ef44cb14a 
BRM42220042 85200152-d742-47b2-a146-d2074f4fe98e bf99d4f8-edf1-4de5-98d4-8e6a24965005 40f02df7-0d8d-415c-afa0-736dfb406fe8 
VCR from volume ID 0991860d-b9bc-491c-a7d2-01e8acd7200a
ID                                   BS  SUB_VOLUMES READ_ONLY_PARENT 
79720eb5-2775-4164-b002-d437f4c29009 512 1           false            

SUB VOLUME 0
    ID                                   BS  BPE    EC   GEN READ_ONLY 
    79720eb5-2775-4164-b002-d437f4c29009 512 131072 1600 2   false     
    [fd00:1122:3344:11f::5]:19008
    [fd00:1122:3344:114::c]:19013
    [fd00:1122:3344:119::8]:19018

root@oxz_switch1:~# pilot host exec -c 'uptime' 3 15 16 26
 3  BRM42220079        ok: 23:20:54    up 1 day(s), 21:41,  0 users,  load average: 1.12, 1.67, 2.41
15  BRM44220016        ok: 23:20:41    up 1 day(s), 21:42,  0 users,  load average: 0.96, 1.00, 1.54
16  BRM42220078        ok: 23:20:41    up 1 day(s), 21:42,  0 users,  load average: 4.53, 5.20, 6.40
26  BRM42220042        ok: 23:20:41    up 1 day(s), 21:43,  0 users,  load average: 3.05, 3.43, 2.67

[askfongjojo]

Propolis and crucible logs are in /staff/core/crucible-1844/.

[leftwo]

Looking through the propolis logs, I don't see any obvious signs of distress from Crucible.

The only thing that seems out of ordinary is:

2025-11-13 21:52:02.212Z WARN propolis-server/4600 (vm_state_driver) on oxz_propolis-server_7bfb54cc-743c-4c50-874a-ff863176c10b: flush check fired despite having jobs; resetting it
2025-11-13 21:52:03.214Z WARN propolis-server/4600 (vm_state_driver) on oxz_propolis-server_7bfb54cc-743c-4c50-874a-ff863176c10b: flush check fired despite having jobs; resetting it
2025-11-13 21:52:04.215Z WARN propolis-server/4600 (vm_state_driver) on oxz_propolis-server_7bfb54cc-743c-4c50-874a-ff863176c10b: flush check fired despite having jobs; resetting it
2025-11-13 21:52:05.430Z WARN propolis-server/4600 (vm_state_driver) on oxz_propolis-server_7bfb54cc-743c-4c50-874a-ff863176c10b: flush check fired despite having jobs; resetting it
2025-11-13 21:52:06.217Z WARN propolis-server/4600 (vm_state_driver) on oxz_propolis-server_7bfb54cc-743c-4c50-874a-ff863176c10b: flush check fired despite having jobs; resetting it
2025-11-13 21:52:07.681Z WARN propolis-server/4600 (vm_state_driver) on oxz_propolis-server_7bfb54cc-743c-4c50-874a-ff863176c10b: flush check fired despite having jobs; resetting it

Which continue for a bit to this point:

2025-11-13 21:57:17.683Z WARN propolis-server/4600 (vm_state_driver) on oxz_propolis-server_7bfb54cc-743c-4c50-874a-ff863176c10b: flush check fired despite having jobs; resetting it
2025-11-13 21:57:20.700Z WARN propolis-server/4600 (vm_state_driver) on oxz_propolis-server_7bfb54cc-743c-4c50-874a-ff863176c10b: flush check fired despite having jobs; resetting it
2025-11-13 21:57:21.656Z WARN propolis-server/4600 (vm_state_driver) on oxz_propolis-server_7bfb54cc-743c-4c50-874a-ff863176c10b: flush check fired despite having jobs; resetting it

[leftwo]

(From @lgfa29 in chat) To reproduce steps are,

Create a new instance with:

16 vCPUs
32 GiB RAM
noble image
100 GiB disk size

SSH into the instance and install Docker

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh ./get-docker.sh
sudo usermod -aG docker $USER
newgrp docker

Clone the terraform-provider-oxide repo and switch to the luiz-tmp branch.

git clone https://github.com/oxidecomputer/terraform-provider-oxide.git
cd terraform-provider-oxide
git checkout luiz-tmp

Start the test containers.

cd acctest
docker compose up

Wait for the message Attaching to omicron-dev-1 and, from another terminal, SSH into the instance and exec into the container.

docker exec -it $(docker ps --format '{{.ID}}') /bin/bash

From the container, compile and run omicron-dev.

source env.sh && cargo xtask omicron-dev run-all

Stop omicron-dev, switch branch and run it again.

git stash
git checkout rel/v17/rc3
cargo xtask omicron-dev run-all

Repeat the last two steps of switching between branches multiple times.

git checkout -
cargo xtask omicron-dev run-all