Jul 06 19:11:37.559151 ip-10-0-143-192 systemd[1]: Starting Kubernetes Kubelet... Jul 06 19:11:39.075194 ip-10-0-143-192 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:11:39.075194 ip-10-0-143-192 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 19:11:39.075194 ip-10-0-143-192 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:11:39.075194 ip-10-0-143-192 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 19:11:39.075194 ip-10-0-143-192 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:11:39.075194 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.072884 2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078343 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078356 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078360 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078363 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078366 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078369 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078372 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078375 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078378 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078381 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078384 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078387 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078389 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078392 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078394 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078398 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078400 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078403 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:39.080697 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078409 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078412 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078414 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078417 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078419 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078422 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078425 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078429 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078432 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078434 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078437 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078441 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078443 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078446 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078449 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078452 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078454 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078457 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078460 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078463 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:39.081381 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078465 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078468 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078470 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078473 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078475 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078477 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078480 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078483 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078485 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078488 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078490 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078492 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078495 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078498 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078501 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078503 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078506 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078509 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078511 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078514 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:39.179889 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078516 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:39.149932 ip-10-0-143-192 systemd[1]: Started Kubernetes Kubelet. Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078519 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078521 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078526 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078529 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078532 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078535 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078538 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078540 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078545 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078548 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078551 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078554 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078557 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078559 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078562 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078564 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078567 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078569 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:39.181753 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078571 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078574 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078577 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078580 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078582 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078584 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078588 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078591 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078593 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078596 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078598 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078601 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078603 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078606 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078609 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078611 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078614 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078616 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078620 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078623 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:39.183679 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078625 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.078628 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083324 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083340 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083344 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083349 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083353 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083358 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083366 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083372 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083377 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083382 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083386 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083392 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083397 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083401 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083407 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083411 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083415 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:39.184389 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083420 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083423 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083428 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083433 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083438 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083442 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083447 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083451 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083454 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083460 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083464 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083468 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083471 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083476 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083479 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083484 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083488 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083492 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083495 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083499 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:39.185485 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083503 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083507 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083510 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083514 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083518 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083522 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083525 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083529 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083533 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083537 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083540 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083544 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083548 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083552 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083556 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083560 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083564 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083569 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083573 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083577 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:39.186150 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083580 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083584 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083587 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083591 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083595 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083599 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083603 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083607 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083610 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083614 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083619 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083623 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083627 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083631 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083635 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083639 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083643 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083647 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083664 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083669 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:39.186867 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083673 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083677 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083681 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083685 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083689 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083692 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083697 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083701 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083705 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083711 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083717 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083721 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083730 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083734 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083740 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083744 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083749 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083753 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083757 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:39.187596 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083760 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083764 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.083770 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084372 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084379 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084384 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084388 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084392 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084396 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084400 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084404 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084410 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084415 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084419 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084422 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084426 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084430 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084434 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084438 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084443 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:39.188683 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084447 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084451 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084454 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084459 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084463 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084469 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084472 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084476 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084480 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084483 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084487 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084491 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084495 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084499 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084505 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084509 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084514 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084518 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084522 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084526 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:39.189366 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084529 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084533 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084537 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084541 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084545 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084549 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084552 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084556 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084560 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084563 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084569 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084573 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084577 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084581 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084584 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084588 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084592 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084597 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084602 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084605 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:39.190094 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084609 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084612 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084616 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084620 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084624 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084630 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084637 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084642 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084646 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084666 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084672 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084676 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084680 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084684 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084688 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084692 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084697 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084701 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084704 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:39.191202 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084708 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084711 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084715 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084719 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084722 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084726 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084731 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084734 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084738 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084741 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084747 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084750 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084754 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084758 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084762 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084765 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084769 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084773 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084777 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084782 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:39.191915 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084786 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084790 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.084794 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084882 2581 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084892 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084901 2581 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084907 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084913 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084918 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084924 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084930 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084934 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084938 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084944 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084949 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084953 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084958 2581 flags.go:64] FLAG: --cgroup-root="" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084963 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084967 2581 flags.go:64] FLAG: --client-ca-file="" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084972 2581 flags.go:64] FLAG: --cloud-provider="external" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084977 2581 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084983 2581 flags.go:64] FLAG: --cluster-domain="" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084989 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 19:11:39.192625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084993 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.084998 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085003 2581 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085014 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085019 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085024 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085028 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085033 2581 flags.go:64] FLAG: --contention-profiling="false" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085039 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085043 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085048 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085052 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085058 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085063 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085067 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085071 2581 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085076 2581 flags.go:64] FLAG: --enable-server="true" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085081 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085088 2581 flags.go:64] FLAG: --event-burst="100" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085093 2581 flags.go:64] FLAG: --event-qps="50" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085097 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085102 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085106 2581 flags.go:64] FLAG: --eviction-hard="" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085111 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 19:11:39.194747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085116 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085120 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085125 2581 flags.go:64] FLAG: --eviction-soft="" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085129 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085133 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085139 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085143 2581 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085149 2581 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085154 2581 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085158 2581 flags.go:64] FLAG: --feature-gates="" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085163 2581 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085168 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085173 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085177 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085182 2581 flags.go:64] FLAG: --healthz-port="10248" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085186 2581 flags.go:64] FLAG: --help="false" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085192 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-143-192.ec2.internal" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085197 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085202 2581 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085206 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085212 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085217 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085221 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 19:11:39.195819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085226 2581 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085230 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085234 2581 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085238 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085243 2581 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085247 2581 flags.go:64] FLAG: --kube-reserved="" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085251 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085256 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085260 2581 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085264 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085268 2581 flags.go:64] FLAG: --lock-file="" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085273 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085277 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085281 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085289 2581 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085295 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085301 2581 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085304 2581 flags.go:64] FLAG: --logging-format="text" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085308 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085313 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085317 2581 flags.go:64] FLAG: --manifest-url="" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085321 2581 flags.go:64] FLAG: --manifest-url-header="" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085327 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085332 2581 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085337 2581 flags.go:64] FLAG: --max-pods="110" Jul 06 19:11:39.197937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085344 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085349 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085353 2581 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085357 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085361 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085366 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085370 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085382 2581 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085386 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085390 2581 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085395 2581 flags.go:64] FLAG: --pod-cidr="" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085399 2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085407 2581 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085411 2581 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085415 2581 flags.go:64] FLAG: --pods-per-core="0" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085420 2581 flags.go:64] FLAG: --port="10250" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085424 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085429 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-081d811d2ed99b5e2" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085433 2581 flags.go:64] FLAG: --qos-reserved="" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085438 2581 flags.go:64] FLAG: --read-only-port="10255" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085443 2581 flags.go:64] FLAG: --register-node="true" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085447 2581 flags.go:64] FLAG: --register-with-taints="" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085452 2581 flags.go:64] FLAG: --registry-burst="10" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085459 2581 flags.go:64] FLAG: --registry-qps="5" Jul 06 19:11:39.198998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085464 2581 flags.go:64] FLAG: --reserved-cpus="" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085468 2581 flags.go:64] FLAG: --reserved-memory="" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085473 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085478 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085482 2581 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085487 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085491 2581 flags.go:64] FLAG: --runonce="false" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085495 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085500 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085506 2581 flags.go:64] FLAG: --seccomp-default="false" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085510 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085514 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085519 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085524 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085529 2581 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085533 2581 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085538 2581 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085542 2581 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085546 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085551 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085555 2581 flags.go:64] FLAG: --system-cgroups="" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085560 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085567 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085572 2581 flags.go:64] FLAG: --tls-cert-file="" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085576 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 19:11:39.201115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085582 2581 flags.go:64] FLAG: --tls-min-version="" Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085586 2581 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085590 2581 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085594 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085599 2581 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085603 2581 flags.go:64] FLAG: --v="2" Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085610 2581 flags.go:64] FLAG: --version="false" Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085616 2581 flags.go:64] FLAG: --vmodule="" Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085622 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.085626 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085808 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085814 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085819 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085823 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085826 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085831 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085836 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085840 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085844 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085848 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085852 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085856 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:39.202169 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085860 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085863 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085867 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085872 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085876 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085879 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085882 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085886 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085891 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085895 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085899 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085903 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085906 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085910 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085914 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085919 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085924 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085928 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085932 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085936 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:39.203027 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085940 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085943 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085947 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085951 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085957 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085962 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085967 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085972 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085976 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085981 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085985 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085989 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085993 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.085998 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086004 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086009 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086013 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086017 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086021 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:39.203758 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086025 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086029 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086033 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086037 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086040 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086044 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086048 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086052 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086055 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086061 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086065 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086069 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086074 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086080 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086085 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086090 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086094 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086099 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086102 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086108 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:39.204443 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086111 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086115 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086119 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086122 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086126 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086130 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086134 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086138 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086142 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086146 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086150 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086154 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086158 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086162 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086166 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086169 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086173 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086177 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086181 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:39.206623 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086184 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:39.207331 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086189 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:39.207331 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086195 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:39.207331 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086198 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:39.207331 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086202 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:39.207331 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086206 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:39.207331 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086211 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:39.207331 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086215 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:39.207331 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.086219 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:39.207331 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.086225 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:11:39.207331 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.093345 2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 19:11:39.207331 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.093361 2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 19:11:39.207331 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.094032 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 19:11:39.207331 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.094042 2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 19:11:39.207331 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094102 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:39.207331 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094107 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094111 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094114 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094118 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094121 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094124 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094127 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094129 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094132 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094135 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094137 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094140 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094143 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094145 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094148 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094150 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094153 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094156 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094158 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094161 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:39.208410 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094163 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094166 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094168 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094172 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094175 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094177 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094180 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094183 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094186 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094189 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094191 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094194 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094197 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094200 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094202 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094205 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094208 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094210 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094213 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094215 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:39.294369 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094218 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094221 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094223 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094226 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094228 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094231 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094234 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094237 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094239 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094242 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094245 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094248 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094251 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094253 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094256 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094258 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094269 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094272 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094275 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:39.295999 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094278 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094282 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094284 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094287 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094290 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094293 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094296 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094300 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094304 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094307 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094310 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094314 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094317 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094320 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094324 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094327 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094330 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094332 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094335 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:39.296707 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094338 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094344 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094347 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094350 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094352 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094355 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094358 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094361 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094364 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094366 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094369 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094372 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094375 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094378 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094381 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094383 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094386 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094388 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094391 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:39.298100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094393 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:39.298823 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.094398 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:11:39.298823 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094516 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:39.298823 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094520 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:39.298823 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094523 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:39.298823 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094526 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:39.298823 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094529 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:39.298823 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094531 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:39.298823 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094534 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:39.298823 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094536 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:39.298823 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094539 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:39.298823 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094541 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:39.298823 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094544 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:39.298823 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094546 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:39.298823 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094549 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094553 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094556 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094559 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094562 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094564 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094567 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094569 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094572 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094575 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094577 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094581 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094583 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094586 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094589 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094591 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094594 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094596 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094599 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094602 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:39.300220 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094604 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094608 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094611 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094613 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094616 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094619 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094621 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094624 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094626 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094629 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094631 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094634 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094636 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094640 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094642 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094645 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094647 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094687 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094692 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094695 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:39.300941 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094698 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094701 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094704 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094706 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094710 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094714 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094716 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094720 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094724 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094727 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094730 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094732 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094735 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094739 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094742 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094744 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094746 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094749 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094752 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:39.302355 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094755 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094759 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094770 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094773 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094775 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094778 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094782 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094784 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094787 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094789 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094792 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094795 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094797 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094800 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094802 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094805 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094808 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094810 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094813 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094816 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:39.303123 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094818 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:39.304479 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094821 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:39.304479 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094824 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:39.304479 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094828 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:39.304479 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094831 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:39.304479 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094834 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:39.304479 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094838 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:39.304479 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:39.094840 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:39.304479 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.094845 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:11:39.304479 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.094951 2581 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 19:11:39.304479 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.096963 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 19:11:39.304479 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.098644 2581 server.go:1019] "Starting client certificate rotation" Jul 06 19:11:39.304479 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.098753 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:11:39.304479 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.098798 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:11:39.305190 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.101408 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:11:39.305190 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.102551 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:11:39.305190 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.114211 2581 log.go:25] "Validated CRI v1 runtime API" Jul 06 19:11:39.305190 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.116877 2581 log.go:25] "Validated CRI v1 image API" Jul 06 19:11:39.305190 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.117603 2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 19:11:39.305190 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.120239 2581 fs.go:135] Filesystem UUIDs: map[24d53055-0ff4-4301-a1ed-b54e480c33b2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bb923efb-1233-421e-84ac-336536a8481a:/dev/nvme0n1p3] Jul 06 19:11:39.305190 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.120258 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.127227 2581 manager.go:216] Machine: {Timestamp:2026-07-06 19:11:39.125634743 +0000 UTC m=+0.341272278 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099626 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23e2e58bd06b82e8bcdbb61b541292 SystemUUID:ec23e2e5-8bd0-6b82-e8bc-dbb61b541292 BootID:3eb27f77-300b-4bc2-92a1-b27fc9eb452a Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4f:16:7a:98:a5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4f:16:7a:98:a5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:d5:9b:21:31:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.127330 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.127410 2581 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.131143 2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.131165 2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-192.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.131902 2581 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.131913 2581 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.131920 2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.132579 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.133958 2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.136723 2581 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.136836 2581 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.137715 2581 kubelet.go:486] "Attempting to sync node with API server" Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.137725 2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.137739 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.137749 2581 kubelet.go:398] "Adding apiserver pod source" Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.137758 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 19:11:39.307301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.138101 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.138125 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.140804 2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.143000 2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.143038 2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.144560 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.144576 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.144584 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.144592 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.144604 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.144613 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.144624 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.144631 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.144641 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.144648 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.144673 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.144682 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.145421 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.145432 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.149098 2581 server.go:1268] "Started kubelet" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.149173 2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.149287 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.149357 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.150884 2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.151416 2581 server.go:310] "Adding debug handlers to kubelet server" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.154008 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-192.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.154017 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-192.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.154661 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.155892 2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.157722 2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.157741 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.157795 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.158686 2581 factory.go:55] Registering systemd factory Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.158723 2581 factory.go:223] Registration of the systemd container factory successfully Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.158866 2581 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.158877 2581 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.158947 2581 factory.go:153] Registering CRI-O factory Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.158961 2581 factory.go:223] Registration of the crio container factory successfully Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.158999 2581 reconstruct.go:97] "Volume reconstruction finished" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.159007 2581 reconciler.go:29] "Reconciler: start to sync state" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.159077 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.159105 2581 factory.go:103] Registering Raw factory Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.159120 2581 manager.go:1196] Started watching for new ooms in manager Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.159483 2581 manager.go:349] Starting recovery of all containers Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.159812 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 19:11:39.308000 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.161266 2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-192.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.161907 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.161379 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-192.ec2.internal.18bfc91be79bfcb5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-192.ec2.internal,UID:ip-10-0-143-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-192.ec2.internal,},FirstTimestamp:2026-07-06 19:11:39.149065397 +0000 UTC m=+0.364702919,LastTimestamp:2026-07-06 19:11:39.149065397 +0000 UTC m=+0.364702919,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-192.ec2.internal,}" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.171001 2581 manager.go:354] Recovery completed Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.176264 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.178376 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.178409 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.178423 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.178929 2581 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.178941 2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.178956 2581 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.178974 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.179755 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.179782 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.179796 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.180172 2581 policy_none.go:49] "None policy: Start" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.180185 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.180196 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:39.181098 2581 policy_none.go:47] "Start" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.181186 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-192.ec2.internal.18bfc91be95b7717 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-192.ec2.internal,UID:ip-10-0-143-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-192.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-192.ec2.internal,},FirstTimestamp:2026-07-06 19:11:39.178391319 +0000 UTC m=+0.394028841,LastTimestamp:2026-07-06 19:11:39.178391319 +0000 UTC m=+0.394028841,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-192.ec2.internal,}" Jul 06 19:11:39.309396 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.190236 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-192.ec2.internal.18bfc91be95bd50f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-192.ec2.internal,UID:ip-10-0-143-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-192.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-192.ec2.internal,},FirstTimestamp:2026-07-06 19:11:39.178415375 +0000 UTC m=+0.394052896,LastTimestamp:2026-07-06 19:11:39.178415375 +0000 UTC m=+0.394052896,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-192.ec2.internal,}" Jul 06 19:11:39.310845 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.202359 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-192.ec2.internal.18bfc91be95c088e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-192.ec2.internal,UID:ip-10-0-143-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-192.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-192.ec2.internal,},FirstTimestamp:2026-07-06 19:11:39.178428558 +0000 UTC m=+0.394066081,LastTimestamp:2026-07-06 19:11:39.178428558 +0000 UTC m=+0.394066081,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-192.ec2.internal,}" Jul 06 19:11:39.310845 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.204786 2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-192.ec2.internal.18bfc91be95b7717\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-192.ec2.internal.18bfc91be95b7717 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-192.ec2.internal,UID:ip-10-0-143-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-192.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-192.ec2.internal,},FirstTimestamp:2026-07-06 19:11:39.178391319 +0000 UTC m=+0.394028841,LastTimestamp:2026-07-06 19:11:39.179771146 +0000 UTC m=+0.395408677,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-192.ec2.internal,}" Jul 06 19:11:39.310845 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.214675 2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-192.ec2.internal.18bfc91be95bd50f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-192.ec2.internal.18bfc91be95bd50f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-192.ec2.internal,UID:ip-10-0-143-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-192.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-192.ec2.internal,},FirstTimestamp:2026-07-06 19:11:39.178415375 +0000 UTC m=+0.394052896,LastTimestamp:2026-07-06 19:11:39.179788491 +0000 UTC m=+0.395426013,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-192.ec2.internal,}" Jul 06 19:11:39.310845 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.227266 2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-192.ec2.internal.18bfc91be95c088e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-192.ec2.internal.18bfc91be95c088e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-192.ec2.internal,UID:ip-10-0-143-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-192.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-192.ec2.internal,},FirstTimestamp:2026-07-06 19:11:39.178428558 +0000 UTC m=+0.394066081,LastTimestamp:2026-07-06 19:11:39.179801432 +0000 UTC m=+0.395438953,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-192.ec2.internal,}" Jul 06 19:11:39.310845 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.258362 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Jul 06 19:11:39.403752 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.358697 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Jul 06 19:11:39.403752 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.371947 2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-192.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms" Jul 06 19:11:39.506511 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.459260 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Jul 06 19:11:39.616912 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.559592 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Jul 06 19:11:39.723911 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.660251 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Jul 06 19:11:39.831467 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.760665 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Jul 06 19:11:39.831467 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.781971 2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-192.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms" Jul 06 19:11:39.927629 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.861262 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Jul 06 19:11:39.961774 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:39.961743 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Jul 06 19:11:40.025893 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:40.025865 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 19:11:40.043064 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.042894 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:11:40.062162 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:40.062142 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Jul 06 19:11:40.062247 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.062240 2581 manager.go:337] "Starting Device Plugin manager" Jul 06 19:11:40.062279 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:40.062269 2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 19:11:40.062279 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.062278 2581 server.go:87] "Starting device plugin registration server" Jul 06 19:11:40.062553 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.062542 2581 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 19:11:40.062613 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.062553 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 19:11:40.063199 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.063176 2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 19:11:40.063298 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.063276 2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 19:11:40.063298 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.063285 2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 19:11:40.063378 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:40.063320 2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 19:11:40.063378 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:40.063371 2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-192.ec2.internal\" not found" Jul 06 19:11:40.066519 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:40.066453 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-192.ec2.internal.18bfc91c1e2a711b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-192.ec2.internal,UID:ip-10-0-143-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-143-192.ec2.internal,},FirstTimestamp:2026-07-06 19:11:40.064370971 +0000 UTC m=+1.280008478,LastTimestamp:2026-07-06 19:11:40.064370971 +0000 UTC m=+1.280008478,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-192.ec2.internal,}" Jul 06 19:11:40.074845 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.074827 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c4d75" Jul 06 19:11:40.084882 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.084863 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c4d75" Jul 06 19:11:40.101965 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.098234 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 19:11:40.139866 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.139822 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 19:11:40.141175 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.141156 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 19:11:40.141261 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.141194 2581 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 19:11:40.141261 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.141216 2581 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 19:11:40.141261 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:40.141250 2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 19:11:40.156841 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.156814 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:11:40.162752 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.162738 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:40.163528 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.163514 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:40.163596 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.163547 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:40.163596 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.163562 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:40.163596 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.163583 2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.163859 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.163847 2581 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-192.ec2.internal" not found Jul 06 19:11:40.175563 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.175543 2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.175563 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:40.175564 2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-192.ec2.internal\": node \"ip-10-0-143-192.ec2.internal\" not found" Jul 06 19:11:40.193595 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:40.193535 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Jul 06 19:11:40.218513 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.218491 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:40.219318 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.219304 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:40.219393 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.219332 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:40.219393 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.219346 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:40.242311 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.242287 2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal"] Jul 06 19:11:40.242384 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.242354 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:40.243147 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.243132 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:40.243223 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.243158 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:40.243223 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.243170 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:40.244524 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.244513 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:40.244630 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.244618 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.244689 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.244648 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:40.245196 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.245180 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:40.245275 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.245207 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:40.245275 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.245220 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:40.245275 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.245242 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:40.245275 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.245260 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:40.245275 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.245271 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:40.246357 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.246344 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.246405 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.246373 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:40.248928 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.248914 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:40.248988 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.248938 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:40.248988 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.248947 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:40.262956 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.262935 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/05e5ff3e6ffb5a5e73115081d86cdae7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal\" (UID: \"05e5ff3e6ffb5a5e73115081d86cdae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.263035 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.262959 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e5ff3e6ffb5a5e73115081d86cdae7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal\" (UID: \"05e5ff3e6ffb5a5e73115081d86cdae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.263035 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.262975 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b77e81dd26018ebe7401261e83c3a4d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-192.ec2.internal\" (UID: \"1b77e81dd26018ebe7401261e83c3a4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.272352 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:40.272333 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-192.ec2.internal\" not found" node="ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.276731 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:40.276715 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-192.ec2.internal\" not found" node="ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.294109 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:40.294093 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Jul 06 19:11:40.356730 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.356706 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:11:40.363704 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.363681 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/05e5ff3e6ffb5a5e73115081d86cdae7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal\" (UID: \"05e5ff3e6ffb5a5e73115081d86cdae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.363815 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.363708 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e5ff3e6ffb5a5e73115081d86cdae7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal\" (UID: \"05e5ff3e6ffb5a5e73115081d86cdae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.363815 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.363725 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b77e81dd26018ebe7401261e83c3a4d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-192.ec2.internal\" (UID: \"1b77e81dd26018ebe7401261e83c3a4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.363815 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.363759 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b77e81dd26018ebe7401261e83c3a4d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-192.ec2.internal\" (UID: \"1b77e81dd26018ebe7401261e83c3a4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.363815 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.363783 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/05e5ff3e6ffb5a5e73115081d86cdae7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal\" (UID: \"05e5ff3e6ffb5a5e73115081d86cdae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.363815 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.363783 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e5ff3e6ffb5a5e73115081d86cdae7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal\" (UID: \"05e5ff3e6ffb5a5e73115081d86cdae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.394480 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:40.394454 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Jul 06 19:11:40.495250 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:40.495170 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-192.ec2.internal\" not found" Jul 06 19:11:40.565771 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.565746 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:11:40.575974 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.575955 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.580916 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.580416 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.658068 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.658031 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.678698 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.678677 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:11:40.680328 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.680314 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" Jul 06 19:11:40.700136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:40.700122 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:11:41.087635 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.087578 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 19:06:40 +0000 UTC" deadline="2028-02-22 13:50:35.705860614 +0000 UTC" Jul 06 19:11:41.087635 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.087628 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14298h38m54.618236457s" Jul 06 19:11:41.138796 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.138770 2581 apiserver.go:52] "Watching apiserver" Jul 06 19:11:41.148012 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.147995 2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 19:11:41.149228 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.149210 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-d9l66","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal","openshift-multus/network-metrics-daemon-sr7tr","openshift-network-diagnostics/network-check-target-894sw","openshift-network-operator/iptables-alerter-c2kdq","openshift-ovn-kubernetes/ovnkube-node-c6c7d","kube-system/global-pull-secret-syncer-sk86p","kube-system/konnectivity-agent-zlhst","kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2","openshift-cluster-node-tuning-operator/tuned-lvx7n","openshift-dns/node-resolver-g6qxq","openshift-multus/multus-additional-cni-plugins-fszg8","openshift-multus/multus-z5nvg"] Jul 06 19:11:41.150522 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.150510 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d9l66" Jul 06 19:11:41.151638 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.151625 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:41.151716 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.151697 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:11:41.152740 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.152719 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:11:41.152824 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.152793 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-894sw" podUID="793ee5b4-63f0-4814-b790-8a492924c6d6" Jul 06 19:11:41.152968 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.152956 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 19:11:41.153011 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.152982 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9nznv\"" Jul 06 19:11:41.153184 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.153169 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 19:11:41.153455 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.153444 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 19:11:41.153866 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.153855 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c2kdq" Jul 06 19:11:41.155129 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.155111 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.156021 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.156005 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:11:41.156273 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.156260 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 19:11:41.156273 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.156269 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 19:11:41.156409 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.156328 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:41.156409 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.156386 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk86p" podUID="fee777de-2d7a-4470-8a1d-b64c9fad8390" Jul 06 19:11:41.156496 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.156425 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xxs5g\"" Jul 06 19:11:41.157328 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.157315 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 19:11:41.157517 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.157506 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zlhst" Jul 06 19:11:41.157748 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.157729 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 19:11:41.157862 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.157767 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 19:11:41.157971 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.157958 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 19:11:41.158079 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.158067 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 19:11:41.158123 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.158093 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 19:11:41.158525 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.158508 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8cb24\"" Jul 06 19:11:41.158632 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.158619 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 19:11:41.158797 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.158783 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.160165 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.160149 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.160416 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.160401 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 19:11:41.160501 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.160436 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cxhq2\"" Jul 06 19:11:41.161130 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.161117 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 19:11:41.161277 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.161265 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 19:11:41.161457 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.161447 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 19:11:41.162591 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.162524 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g6qxq" Jul 06 19:11:41.163073 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.162696 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 19:11:41.163073 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.162948 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fxlq4\"" Jul 06 19:11:41.163289 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.163112 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vt98f\"" Jul 06 19:11:41.163289 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.163238 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:11:41.164062 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.163788 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 19:11:41.165233 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.165214 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 19:11:41.165891 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.165875 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.165970 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.165930 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 19:11:41.166813 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.166799 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zvgzt\"" Jul 06 19:11:41.167057 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.167042 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.168106 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168090 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cba62fed-78f0-409f-8cc1-744bfbc23165-ovnkube-config\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.168176 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168115 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-socket-dir\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.168176 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168131 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-kubernetes\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.168176 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168151 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9-host\") pod \"node-ca-d9l66\" (UID: \"b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9\") " pod="openshift-image-registry/node-ca-d9l66" Jul 06 19:11:41.168277 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168180 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-run-ovn\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.168277 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168216 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-var-lib-openvswitch\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.168277 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168253 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-etc-openvswitch\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.168394 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168285 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-sysconfig\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.168394 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168317 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.168394 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168345 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-registration-dir\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.168394 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168373 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret\") pod \"global-pull-secret-syncer-sk86p\" (UID: \"fee777de-2d7a-4470-8a1d-b64c9fad8390\") " pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:41.168512 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168406 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9-serviceca\") pod \"node-ca-d9l66\" (UID: \"b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9\") " pod="openshift-image-registry/node-ca-d9l66" Jul 06 19:11:41.168512 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168429 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-cni-netd\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.168512 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168452 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cba62fed-78f0-409f-8cc1-744bfbc23165-ovn-node-metrics-cert\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.168512 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168478 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cba62fed-78f0-409f-8cc1-744bfbc23165-ovnkube-script-lib\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.168512 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168501 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs\") pod \"network-metrics-daemon-sr7tr\" (UID: \"8d718bbb-07dd-481b-81a1-19b7b28c34c1\") " pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:41.168734 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168523 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv7mm\" (UniqueName: \"kubernetes.io/projected/8d718bbb-07dd-481b-81a1-19b7b28c34c1-kube-api-access-nv7mm\") pod \"network-metrics-daemon-sr7tr\" (UID: \"8d718bbb-07dd-481b-81a1-19b7b28c34c1\") " pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:41.168734 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168545 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-cni-bin\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.168734 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168567 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-device-dir\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.168734 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168588 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f7lc\" (UniqueName: \"kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc\") pod \"network-check-target-894sw\" (UID: \"793ee5b4-63f0-4814-b790-8a492924c6d6\") " pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:11:41.168734 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168605 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-host\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.168734 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168640 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1b502af-fe31-489d-a623-e70d82db0390-etc-tuned\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.168954 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168744 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkd6m\" (UniqueName: \"kubernetes.io/projected/cba62fed-78f0-409f-8cc1-744bfbc23165-kube-api-access-mkd6m\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.168954 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168768 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-systemd-units\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.168954 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168790 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-slash\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.168954 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168811 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-run-ovn-kubernetes\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.168954 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168835 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-etc-selinux\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.168954 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168882 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-lib-modules\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.168954 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168910 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.168954 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168938 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h799x\" (UniqueName: \"kubernetes.io/projected/a1b502af-fe31-489d-a623-e70d82db0390-kube-api-access-h799x\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.169343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168963 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-sys-fs\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.169343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.168996 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg998\" (UniqueName: \"kubernetes.io/projected/62552869-4a5c-40e3-bb1a-3b7acec6ff06-kube-api-access-hg998\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.169343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169020 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cba62fed-78f0-409f-8cc1-744bfbc23165-env-overrides\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.169343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169044 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fee777de-2d7a-4470-8a1d-b64c9fad8390-kubelet-config\") pod \"global-pull-secret-syncer-sk86p\" (UID: \"fee777de-2d7a-4470-8a1d-b64c9fad8390\") " pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:41.169343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169068 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-sysctl-d\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.169343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169099 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-var-lib-kubelet\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.169343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169122 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 19:11:41.169343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169131 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3d885e22-9b77-4736-a3a4-a5652effa44e-agent-certs\") pod \"konnectivity-agent-zlhst\" (UID: \"3d885e22-9b77-4736-a3a4-a5652effa44e\") " pod="kube-system/konnectivity-agent-zlhst" Jul 06 19:11:41.169343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169154 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1b502af-fe31-489d-a623-e70d82db0390-tmp\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.169343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169180 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/400d2c16-5a1f-4a9d-915c-4e69c3833f5a-host-slash\") pod \"iptables-alerter-c2kdq\" (UID: \"400d2c16-5a1f-4a9d-915c-4e69c3833f5a\") " pod="openshift-network-operator/iptables-alerter-c2kdq" Jul 06 19:11:41.169343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169213 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-kubelet\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.169343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169227 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-run-systemd\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.169343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169244 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3d885e22-9b77-4736-a3a4-a5652effa44e-konnectivity-ca\") pod \"konnectivity-agent-zlhst\" (UID: \"3d885e22-9b77-4736-a3a4-a5652effa44e\") " pod="kube-system/konnectivity-agent-zlhst" Jul 06 19:11:41.169343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169278 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-run\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.169343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169311 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fee777de-2d7a-4470-8a1d-b64c9fad8390-dbus\") pod \"global-pull-secret-syncer-sk86p\" (UID: \"fee777de-2d7a-4470-8a1d-b64c9fad8390\") " pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:41.169343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169343 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-systemd\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.169938 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169374 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-sys\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.169938 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169397 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/400d2c16-5a1f-4a9d-915c-4e69c3833f5a-iptables-alerter-script\") pod \"iptables-alerter-c2kdq\" (UID: \"400d2c16-5a1f-4a9d-915c-4e69c3833f5a\") " pod="openshift-network-operator/iptables-alerter-c2kdq" Jul 06 19:11:41.169938 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169420 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-run-netns\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.169938 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169444 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-run-openvswitch\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.169938 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169467 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-modprobe-d\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.169938 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169495 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-sysctl-conf\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.169938 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169523 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nqfg\" (UniqueName: \"kubernetes.io/projected/b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9-kube-api-access-7nqfg\") pod \"node-ca-d9l66\" (UID: \"b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9\") " pod="openshift-image-registry/node-ca-d9l66" Jul 06 19:11:41.169938 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169544 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-node-log\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.169938 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169566 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hjgk\" (UniqueName: \"kubernetes.io/projected/400d2c16-5a1f-4a9d-915c-4e69c3833f5a-kube-api-access-6hjgk\") pod \"iptables-alerter-c2kdq\" (UID: \"400d2c16-5a1f-4a9d-915c-4e69c3833f5a\") " pod="openshift-network-operator/iptables-alerter-c2kdq" Jul 06 19:11:41.169938 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.169586 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-log-socket\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.170817 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.170800 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 19:11:41.170901 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.170828 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 19:11:41.170901 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.170891 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-56pnx\"" Jul 06 19:11:41.171021 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.170934 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6vfsn\"" Jul 06 19:11:41.171097 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.171083 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 19:11:41.171201 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.171118 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 19:11:41.171201 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.171177 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 19:11:41.178825 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.178809 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:11:41.210445 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.210425 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-89dkz" Jul 06 19:11:41.221548 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.221531 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-89dkz" Jul 06 19:11:41.260286 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.260259 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 19:11:41.269802 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.269781 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cba62fed-78f0-409f-8cc1-744bfbc23165-env-overrides\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.269938 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.269807 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fee777de-2d7a-4470-8a1d-b64c9fad8390-kubelet-config\") pod \"global-pull-secret-syncer-sk86p\" (UID: \"fee777de-2d7a-4470-8a1d-b64c9fad8390\") " pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:41.269938 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.269825 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-sysctl-d\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.269938 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.269848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-var-lib-kubelet\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.269938 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.269889 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3d885e22-9b77-4736-a3a4-a5652effa44e-agent-certs\") pod \"konnectivity-agent-zlhst\" (UID: \"3d885e22-9b77-4736-a3a4-a5652effa44e\") " pod="kube-system/konnectivity-agent-zlhst" Jul 06 19:11:41.269938 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.269911 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fee777de-2d7a-4470-8a1d-b64c9fad8390-kubelet-config\") pod \"global-pull-secret-syncer-sk86p\" (UID: \"fee777de-2d7a-4470-8a1d-b64c9fad8390\") " pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:41.269938 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.269922 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1b502af-fe31-489d-a623-e70d82db0390-tmp\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.269937 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-var-lib-kubelet\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.269946 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/400d2c16-5a1f-4a9d-915c-4e69c3833f5a-host-slash\") pod \"iptables-alerter-c2kdq\" (UID: \"400d2c16-5a1f-4a9d-915c-4e69c3833f5a\") " pod="openshift-network-operator/iptables-alerter-c2kdq" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.269973 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/050dc6de-99c6-4fb3-8252-28cb84d52771-cnibin\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.269983 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-sysctl-d\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.269998 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq6xj\" (UniqueName: \"kubernetes.io/projected/0e43ffe8-6267-4b64-94c9-8b6ce791a0b2-kube-api-access-rq6xj\") pod \"node-resolver-g6qxq\" (UID: \"0e43ffe8-6267-4b64-94c9-8b6ce791a0b2\") " pod="openshift-dns/node-resolver-g6qxq" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270033 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-kubelet\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270032 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/400d2c16-5a1f-4a9d-915c-4e69c3833f5a-host-slash\") pod \"iptables-alerter-c2kdq\" (UID: \"400d2c16-5a1f-4a9d-915c-4e69c3833f5a\") " pod="openshift-network-operator/iptables-alerter-c2kdq" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270058 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-run-systemd\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270093 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-kubelet\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270103 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-run-systemd\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270119 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3d885e22-9b77-4736-a3a4-a5652effa44e-konnectivity-ca\") pod \"konnectivity-agent-zlhst\" (UID: \"3d885e22-9b77-4736-a3a4-a5652effa44e\") " pod="kube-system/konnectivity-agent-zlhst" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270141 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-run\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270156 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-run-netns\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270177 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fee777de-2d7a-4470-8a1d-b64c9fad8390-dbus\") pod \"global-pull-secret-syncer-sk86p\" (UID: \"fee777de-2d7a-4470-8a1d-b64c9fad8390\") " pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270200 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-systemd\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270221 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-sys\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270241 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/400d2c16-5a1f-4a9d-915c-4e69c3833f5a-iptables-alerter-script\") pod \"iptables-alerter-c2kdq\" (UID: \"400d2c16-5a1f-4a9d-915c-4e69c3833f5a\") " pod="openshift-network-operator/iptables-alerter-c2kdq" Jul 06 19:11:41.270248 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270221 2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 19:11:41.271111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270269 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-run\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.271111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270299 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cba62fed-78f0-409f-8cc1-744bfbc23165-env-overrides\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.271111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270267 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/050dc6de-99c6-4fb3-8252-28cb84d52771-os-release\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.271111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270339 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00a50c45-a6c2-442d-b432-f2a776790bbd-multus-daemon-config\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.271111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270346 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-systemd\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.271111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270382 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-sys\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.271111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270382 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fee777de-2d7a-4470-8a1d-b64c9fad8390-dbus\") pod \"global-pull-secret-syncer-sk86p\" (UID: \"fee777de-2d7a-4470-8a1d-b64c9fad8390\") " pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:41.271111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270414 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-run-netns\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.271111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270439 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-run-openvswitch\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.271111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270480 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-run-netns\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.271111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270505 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-modprobe-d\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.271111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270515 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-run-openvswitch\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.271111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270561 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-sysctl-conf\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.271111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270583 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nqfg\" (UniqueName: \"kubernetes.io/projected/b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9-kube-api-access-7nqfg\") pod \"node-ca-d9l66\" (UID: \"b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9\") " pod="openshift-image-registry/node-ca-d9l66" Jul 06 19:11:41.271111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270605 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-os-release\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.271111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270628 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/050dc6de-99c6-4fb3-8252-28cb84d52771-cni-binary-copy\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.271111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270674 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3d885e22-9b77-4736-a3a4-a5652effa44e-konnectivity-ca\") pod \"konnectivity-agent-zlhst\" (UID: \"3d885e22-9b77-4736-a3a4-a5652effa44e\") " pod="kube-system/konnectivity-agent-zlhst" Jul 06 19:11:41.271982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270706 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-sysctl-conf\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.271982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270727 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-node-log\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.271982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270679 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-node-log\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.271982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270798 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hjgk\" (UniqueName: \"kubernetes.io/projected/400d2c16-5a1f-4a9d-915c-4e69c3833f5a-kube-api-access-6hjgk\") pod \"iptables-alerter-c2kdq\" (UID: \"400d2c16-5a1f-4a9d-915c-4e69c3833f5a\") " pod="openshift-network-operator/iptables-alerter-c2kdq" Jul 06 19:11:41.271982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270848 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/050dc6de-99c6-4fb3-8252-28cb84d52771-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.271982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270876 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-multus-conf-dir\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.271982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270901 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e43ffe8-6267-4b64-94c9-8b6ce791a0b2-tmp-dir\") pod \"node-resolver-g6qxq\" (UID: \"0e43ffe8-6267-4b64-94c9-8b6ce791a0b2\") " pod="openshift-dns/node-resolver-g6qxq" Jul 06 19:11:41.271982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270911 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/400d2c16-5a1f-4a9d-915c-4e69c3833f5a-iptables-alerter-script\") pod \"iptables-alerter-c2kdq\" (UID: \"400d2c16-5a1f-4a9d-915c-4e69c3833f5a\") " pod="openshift-network-operator/iptables-alerter-c2kdq" Jul 06 19:11:41.271982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270926 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-log-socket\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.271982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270949 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cba62fed-78f0-409f-8cc1-744bfbc23165-ovnkube-config\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.271982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270964 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-modprobe-d\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.271982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.270983 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-socket-dir\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.271982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271007 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-log-socket\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.271982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271011 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-kubernetes\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.271982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271087 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9-host\") pod \"node-ca-d9l66\" (UID: \"b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9\") " pod="openshift-image-registry/node-ca-d9l66" Jul 06 19:11:41.271982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271115 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-hostroot\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.271982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271120 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-socket-dir\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.272852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271162 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9-host\") pod \"node-ca-d9l66\" (UID: \"b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9\") " pod="openshift-image-registry/node-ca-d9l66" Jul 06 19:11:41.272852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271165 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e43ffe8-6267-4b64-94c9-8b6ce791a0b2-hosts-file\") pod \"node-resolver-g6qxq\" (UID: \"0e43ffe8-6267-4b64-94c9-8b6ce791a0b2\") " pod="openshift-dns/node-resolver-g6qxq" Jul 06 19:11:41.272852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271180 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-kubernetes\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.272852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271217 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-run-ovn\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.272852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271234 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-var-lib-openvswitch\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.272852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271268 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-etc-openvswitch\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.272852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271301 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-run-ovn\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.272852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271308 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-sysconfig\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.272852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271346 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-var-lib-openvswitch\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.272852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271351 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/050dc6de-99c6-4fb3-8252-28cb84d52771-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.272852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271376 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-cnibin\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.272852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271390 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-etc-openvswitch\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.272852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271399 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cba62fed-78f0-409f-8cc1-744bfbc23165-ovnkube-config\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.272852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271403 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00a50c45-a6c2-442d-b432-f2a776790bbd-cni-binary-copy\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.272852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271450 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-var-lib-cni-bin\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.272852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271461 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-etc-sysconfig\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.272852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271479 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.273709 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271503 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-registration-dir\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.273709 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271524 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret\") pod \"global-pull-secret-syncer-sk86p\" (UID: \"fee777de-2d7a-4470-8a1d-b64c9fad8390\") " pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:41.273709 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271553 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.273709 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271550 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9-serviceca\") pod \"node-ca-d9l66\" (UID: \"b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9\") " pod="openshift-image-registry/node-ca-d9l66" Jul 06 19:11:41.273709 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271589 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-multus-cni-dir\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.273709 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271609 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-var-lib-kubelet\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.273709 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271613 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-registration-dir\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.273709 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.271632 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf4rw\" (UniqueName: \"kubernetes.io/projected/00a50c45-a6c2-442d-b432-f2a776790bbd-kube-api-access-nf4rw\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.273709 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.271754 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:41.273709 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.271861 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret podName:fee777de-2d7a-4470-8a1d-b64c9fad8390 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:41.771838056 +0000 UTC m=+2.987475583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret") pod "global-pull-secret-syncer-sk86p" (UID: "fee777de-2d7a-4470-8a1d-b64c9fad8390") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:41.273709 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272083 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9-serviceca\") pod \"node-ca-d9l66\" (UID: \"b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9\") " pod="openshift-image-registry/node-ca-d9l66" Jul 06 19:11:41.273709 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272086 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-cni-netd\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.273709 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272134 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-cni-netd\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.273709 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272157 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cba62fed-78f0-409f-8cc1-744bfbc23165-ovn-node-metrics-cert\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.273709 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272210 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cba62fed-78f0-409f-8cc1-744bfbc23165-ovnkube-script-lib\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.273709 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272233 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs\") pod \"network-metrics-daemon-sr7tr\" (UID: \"8d718bbb-07dd-481b-81a1-19b7b28c34c1\") " pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:41.273709 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272263 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv7mm\" (UniqueName: \"kubernetes.io/projected/8d718bbb-07dd-481b-81a1-19b7b28c34c1-kube-api-access-nv7mm\") pod \"network-metrics-daemon-sr7tr\" (UID: \"8d718bbb-07dd-481b-81a1-19b7b28c34c1\") " pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:41.274605 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272295 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/050dc6de-99c6-4fb3-8252-28cb84d52771-system-cni-dir\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.274605 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272354 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-system-cni-dir\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.274605 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.272394 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:41.274605 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272380 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-cni-bin\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.274605 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272440 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-device-dir\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.274605 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.272462 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs podName:8d718bbb-07dd-481b-81a1-19b7b28c34c1 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:41.77244115 +0000 UTC m=+2.988078668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs") pod "network-metrics-daemon-sr7tr" (UID: "8d718bbb-07dd-481b-81a1-19b7b28c34c1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:41.274605 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272498 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f7lc\" (UniqueName: \"kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc\") pod \"network-check-target-894sw\" (UID: \"793ee5b4-63f0-4814-b790-8a492924c6d6\") " pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:11:41.274605 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272541 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-device-dir\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.274605 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272548 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-host\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.274605 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1b502af-fe31-489d-a623-e70d82db0390-etc-tuned\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.274605 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272599 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/050dc6de-99c6-4fb3-8252-28cb84d52771-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.274605 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272605 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-cni-bin\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.274605 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272629 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-multus-socket-dir-parent\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.274605 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272676 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkd6m\" (UniqueName: \"kubernetes.io/projected/cba62fed-78f0-409f-8cc1-744bfbc23165-kube-api-access-mkd6m\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.274605 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272704 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-systemd-units\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.274605 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272726 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-slash\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.274605 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272764 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-run-ovn-kubernetes\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.275499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272767 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-host\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.275499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272790 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-etc-selinux\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.275499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272886 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-etc-selinux\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.275499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272892 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-slash\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.275499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272935 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-run-ovn-kubernetes\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.275499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272948 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-systemd-units\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.275499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.272994 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-lib-modules\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.275499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.273034 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7g5d\" (UniqueName: \"kubernetes.io/projected/050dc6de-99c6-4fb3-8252-28cb84d52771-kube-api-access-s7g5d\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.275499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.273072 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-var-lib-cni-multus\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.275499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.273098 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-run-multus-certs\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.275499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.273117 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1b502af-fe31-489d-a623-e70d82db0390-lib-modules\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.275499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.273127 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.275499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.273153 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h799x\" (UniqueName: \"kubernetes.io/projected/a1b502af-fe31-489d-a623-e70d82db0390-kube-api-access-h799x\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.275499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.273174 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-run-k8s-cni-cncf-io\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.275499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.273182 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cba62fed-78f0-409f-8cc1-744bfbc23165-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.275499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.273196 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-sys-fs\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.275499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.273222 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg998\" (UniqueName: \"kubernetes.io/projected/62552869-4a5c-40e3-bb1a-3b7acec6ff06-kube-api-access-hg998\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.276336 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.273245 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-etc-kubernetes\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.276336 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.273340 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62552869-4a5c-40e3-bb1a-3b7acec6ff06-sys-fs\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.276336 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.273490 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cba62fed-78f0-409f-8cc1-744bfbc23165-ovnkube-script-lib\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.276336 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.274121 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3d885e22-9b77-4736-a3a4-a5652effa44e-agent-certs\") pod \"konnectivity-agent-zlhst\" (UID: \"3d885e22-9b77-4736-a3a4-a5652effa44e\") " pod="kube-system/konnectivity-agent-zlhst" Jul 06 19:11:41.276336 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.274412 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1b502af-fe31-489d-a623-e70d82db0390-tmp\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.276336 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.274847 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cba62fed-78f0-409f-8cc1-744bfbc23165-ovn-node-metrics-cert\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.276336 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.275036 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a1b502af-fe31-489d-a623-e70d82db0390-etc-tuned\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.281573 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.281555 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:11:41.281713 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.281582 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:11:41.281713 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.281591 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8f7lc for pod openshift-network-diagnostics/network-check-target-894sw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:41.281713 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.281640 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc podName:793ee5b4-63f0-4814-b790-8a492924c6d6 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:41.781628652 +0000 UTC m=+2.997266160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8f7lc" (UniqueName: "kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc") pod "network-check-target-894sw" (UID: "793ee5b4-63f0-4814-b790-8a492924c6d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:41.282963 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.282946 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hjgk\" (UniqueName: \"kubernetes.io/projected/400d2c16-5a1f-4a9d-915c-4e69c3833f5a-kube-api-access-6hjgk\") pod \"iptables-alerter-c2kdq\" (UID: \"400d2c16-5a1f-4a9d-915c-4e69c3833f5a\") " pod="openshift-network-operator/iptables-alerter-c2kdq" Jul 06 19:11:41.283388 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.283374 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv7mm\" (UniqueName: \"kubernetes.io/projected/8d718bbb-07dd-481b-81a1-19b7b28c34c1-kube-api-access-nv7mm\") pod \"network-metrics-daemon-sr7tr\" (UID: \"8d718bbb-07dd-481b-81a1-19b7b28c34c1\") " pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:41.287270 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.287251 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg998\" (UniqueName: \"kubernetes.io/projected/62552869-4a5c-40e3-bb1a-3b7acec6ff06-kube-api-access-hg998\") pod \"aws-ebs-csi-driver-node-n5br2\" (UID: \"62552869-4a5c-40e3-bb1a-3b7acec6ff06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.287363 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.287291 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nqfg\" (UniqueName: \"kubernetes.io/projected/b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9-kube-api-access-7nqfg\") pod \"node-ca-d9l66\" (UID: \"b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9\") " pod="openshift-image-registry/node-ca-d9l66" Jul 06 19:11:41.287737 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.287717 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h799x\" (UniqueName: \"kubernetes.io/projected/a1b502af-fe31-489d-a623-e70d82db0390-kube-api-access-h799x\") pod \"tuned-lvx7n\" (UID: \"a1b502af-fe31-489d-a623-e70d82db0390\") " pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.287993 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.287967 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkd6m\" (UniqueName: \"kubernetes.io/projected/cba62fed-78f0-409f-8cc1-744bfbc23165-kube-api-access-mkd6m\") pod \"ovnkube-node-c6c7d\" (UID: \"cba62fed-78f0-409f-8cc1-744bfbc23165\") " pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.359984 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:41.359948 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b77e81dd26018ebe7401261e83c3a4d.slice/crio-6a040cafa748d60c40070c94e3d7f74124142c572ffd2271f7645c4333109acb WatchSource:0}: Error finding container 6a040cafa748d60c40070c94e3d7f74124142c572ffd2271f7645c4333109acb: Status 404 returned error can't find the container with id 6a040cafa748d60c40070c94e3d7f74124142c572ffd2271f7645c4333109acb Jul 06 19:11:41.360440 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:41.360417 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e5ff3e6ffb5a5e73115081d86cdae7.slice/crio-7f237bd36cf7cdd4f06a6b9984e45a28eaf0df87e91108551c7b9d8b0b3b665a WatchSource:0}: Error finding container 7f237bd36cf7cdd4f06a6b9984e45a28eaf0df87e91108551c7b9d8b0b3b665a: Status 404 returned error can't find the container with id 7f237bd36cf7cdd4f06a6b9984e45a28eaf0df87e91108551c7b9d8b0b3b665a Jul 06 19:11:41.365940 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.365926 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:11:41.374452 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.374431 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/050dc6de-99c6-4fb3-8252-28cb84d52771-cnibin\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.374523 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.374458 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq6xj\" (UniqueName: \"kubernetes.io/projected/0e43ffe8-6267-4b64-94c9-8b6ce791a0b2-kube-api-access-rq6xj\") pod \"node-resolver-g6qxq\" (UID: \"0e43ffe8-6267-4b64-94c9-8b6ce791a0b2\") " pod="openshift-dns/node-resolver-g6qxq" Jul 06 19:11:41.374523 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.374476 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-run-netns\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.374523 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.374493 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/050dc6de-99c6-4fb3-8252-28cb84d52771-os-release\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.374619 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.374555 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/050dc6de-99c6-4fb3-8252-28cb84d52771-cnibin\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.374619 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.374559 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-run-netns\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.374701 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.374686 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00a50c45-a6c2-442d-b432-f2a776790bbd-multus-daemon-config\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.374737 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.374722 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-os-release\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.374776 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.374749 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/050dc6de-99c6-4fb3-8252-28cb84d52771-cni-binary-copy\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.374827 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.374785 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/050dc6de-99c6-4fb3-8252-28cb84d52771-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.374827 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.374724 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/050dc6de-99c6-4fb3-8252-28cb84d52771-os-release\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.374827 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.374806 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-os-release\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.374827 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.374808 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-multus-conf-dir\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.375017 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.374865 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-multus-conf-dir\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.375017 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.374879 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e43ffe8-6267-4b64-94c9-8b6ce791a0b2-tmp-dir\") pod \"node-resolver-g6qxq\" (UID: \"0e43ffe8-6267-4b64-94c9-8b6ce791a0b2\") " pod="openshift-dns/node-resolver-g6qxq" Jul 06 19:11:41.375017 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.374954 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-hostroot\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.375157 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375013 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-hostroot\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.375157 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375076 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e43ffe8-6267-4b64-94c9-8b6ce791a0b2-hosts-file\") pod \"node-resolver-g6qxq\" (UID: \"0e43ffe8-6267-4b64-94c9-8b6ce791a0b2\") " pod="openshift-dns/node-resolver-g6qxq" Jul 06 19:11:41.375157 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375135 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e43ffe8-6267-4b64-94c9-8b6ce791a0b2-hosts-file\") pod \"node-resolver-g6qxq\" (UID: \"0e43ffe8-6267-4b64-94c9-8b6ce791a0b2\") " pod="openshift-dns/node-resolver-g6qxq" Jul 06 19:11:41.375304 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375175 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/050dc6de-99c6-4fb3-8252-28cb84d52771-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.375304 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375201 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-cnibin\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.375304 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375226 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00a50c45-a6c2-442d-b432-f2a776790bbd-cni-binary-copy\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.375304 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375259 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-var-lib-cni-bin\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.375304 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375296 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-multus-cni-dir\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.375532 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375304 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/00a50c45-a6c2-442d-b432-f2a776790bbd-multus-daemon-config\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.375532 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375319 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-var-lib-kubelet\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.375532 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375342 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e43ffe8-6267-4b64-94c9-8b6ce791a0b2-tmp-dir\") pod \"node-resolver-g6qxq\" (UID: \"0e43ffe8-6267-4b64-94c9-8b6ce791a0b2\") " pod="openshift-dns/node-resolver-g6qxq" Jul 06 19:11:41.375532 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375345 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4rw\" (UniqueName: \"kubernetes.io/projected/00a50c45-a6c2-442d-b432-f2a776790bbd-kube-api-access-nf4rw\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.375532 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375352 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/050dc6de-99c6-4fb3-8252-28cb84d52771-cni-binary-copy\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.375532 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375391 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-var-lib-cni-bin\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.375532 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375352 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/050dc6de-99c6-4fb3-8252-28cb84d52771-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.375532 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375404 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/050dc6de-99c6-4fb3-8252-28cb84d52771-system-cni-dir\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.375532 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375393 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-var-lib-kubelet\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.375532 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375391 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-cnibin\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.375532 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375438 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-system-cni-dir\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.375532 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375445 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-multus-cni-dir\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.375532 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375468 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/050dc6de-99c6-4fb3-8252-28cb84d52771-system-cni-dir\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.375532 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375482 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/050dc6de-99c6-4fb3-8252-28cb84d52771-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.375532 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375493 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-system-cni-dir\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.375532 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375508 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-multus-socket-dir-parent\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.376136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375550 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7g5d\" (UniqueName: \"kubernetes.io/projected/050dc6de-99c6-4fb3-8252-28cb84d52771-kube-api-access-s7g5d\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.376136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-var-lib-cni-multus\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.376136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375596 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-run-multus-certs\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.376136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375637 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-run-k8s-cni-cncf-io\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.376136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375676 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-var-lib-cni-multus\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.376136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375691 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-etc-kubernetes\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.376136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375708 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/050dc6de-99c6-4fb3-8252-28cb84d52771-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.376136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375724 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-run-k8s-cni-cncf-io\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.376136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375762 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-etc-kubernetes\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.376136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375770 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-host-run-multus-certs\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.376136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375847 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00a50c45-a6c2-442d-b432-f2a776790bbd-cni-binary-copy\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.376136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375863 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/00a50c45-a6c2-442d-b432-f2a776790bbd-multus-socket-dir-parent\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.376136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.375889 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/050dc6de-99c6-4fb3-8252-28cb84d52771-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.383196 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.383177 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq6xj\" (UniqueName: \"kubernetes.io/projected/0e43ffe8-6267-4b64-94c9-8b6ce791a0b2-kube-api-access-rq6xj\") pod \"node-resolver-g6qxq\" (UID: \"0e43ffe8-6267-4b64-94c9-8b6ce791a0b2\") " pod="openshift-dns/node-resolver-g6qxq" Jul 06 19:11:41.383821 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.383806 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7g5d\" (UniqueName: \"kubernetes.io/projected/050dc6de-99c6-4fb3-8252-28cb84d52771-kube-api-access-s7g5d\") pod \"multus-additional-cni-plugins-fszg8\" (UID: \"050dc6de-99c6-4fb3-8252-28cb84d52771\") " pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.383867 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.383846 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf4rw\" (UniqueName: \"kubernetes.io/projected/00a50c45-a6c2-442d-b432-f2a776790bbd-kube-api-access-nf4rw\") pod \"multus-z5nvg\" (UID: \"00a50c45-a6c2-442d-b432-f2a776790bbd\") " pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.487585 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.487533 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d9l66" Jul 06 19:11:41.493523 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.493504 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z5nvg" Jul 06 19:11:41.496107 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:41.496080 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9cd29e9_e9ac_4a2e_90f2_8e410ed0ede9.slice/crio-2d3c650b203512d7870a0ffcd647d52da67ac2b3ad504159c04bd8c5281fc6a6 WatchSource:0}: Error finding container 2d3c650b203512d7870a0ffcd647d52da67ac2b3ad504159c04bd8c5281fc6a6: Status 404 returned error can't find the container with id 2d3c650b203512d7870a0ffcd647d52da67ac2b3ad504159c04bd8c5281fc6a6 Jul 06 19:11:41.501545 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:41.501527 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a50c45_a6c2_442d_b432_f2a776790bbd.slice/crio-8d87b061f7157830ddc7f340f9fb287d107ae3f0695a6564413a825af7a9f4dd WatchSource:0}: Error finding container 8d87b061f7157830ddc7f340f9fb287d107ae3f0695a6564413a825af7a9f4dd: Status 404 returned error can't find the container with id 8d87b061f7157830ddc7f340f9fb287d107ae3f0695a6564413a825af7a9f4dd Jul 06 19:11:41.508180 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.508160 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c2kdq" Jul 06 19:11:41.515968 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:41.515948 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400d2c16_5a1f_4a9d_915c_4e69c3833f5a.slice/crio-486839741f923659ffeea6302c4a16811a13371d4465d74090830ad77851d0f0 WatchSource:0}: Error finding container 486839741f923659ffeea6302c4a16811a13371d4465d74090830ad77851d0f0: Status 404 returned error can't find the container with id 486839741f923659ffeea6302c4a16811a13371d4465d74090830ad77851d0f0 Jul 06 19:11:41.521835 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.521818 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:11:41.528318 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:41.528299 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba62fed_78f0_409f_8cc1_744bfbc23165.slice/crio-df0e8c47595e59cf7a386fc93f8c0fd51dcf288c4ed470461f8880f261fbe027 WatchSource:0}: Error finding container df0e8c47595e59cf7a386fc93f8c0fd51dcf288c4ed470461f8880f261fbe027: Status 404 returned error can't find the container with id df0e8c47595e59cf7a386fc93f8c0fd51dcf288c4ed470461f8880f261fbe027 Jul 06 19:11:41.536602 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.536585 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zlhst" Jul 06 19:11:41.543802 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:41.543779 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d885e22_9b77_4736_a3a4_a5652effa44e.slice/crio-d87c1f3848afd43614c25919a7933d4d57fdf5a1fb813bcff3b13474e08fbff8 WatchSource:0}: Error finding container d87c1f3848afd43614c25919a7933d4d57fdf5a1fb813bcff3b13474e08fbff8: Status 404 returned error can't find the container with id d87c1f3848afd43614c25919a7933d4d57fdf5a1fb813bcff3b13474e08fbff8 Jul 06 19:11:41.561576 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.561554 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" Jul 06 19:11:41.567333 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.567317 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" Jul 06 19:11:41.568715 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:41.568694 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62552869_4a5c_40e3_bb1a_3b7acec6ff06.slice/crio-a867a49f931ccb5bf1cab92a2ff110b283cc300ac9afe193665df7e11a8d25db WatchSource:0}: Error finding container a867a49f931ccb5bf1cab92a2ff110b283cc300ac9afe193665df7e11a8d25db: Status 404 returned error can't find the container with id a867a49f931ccb5bf1cab92a2ff110b283cc300ac9afe193665df7e11a8d25db Jul 06 19:11:41.574140 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:41.574108 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b502af_fe31_489d_a623_e70d82db0390.slice/crio-6662502ef17780c8c171fe5a8eb81af004876b39c9901162de3af20f7a762255 WatchSource:0}: Error finding container 6662502ef17780c8c171fe5a8eb81af004876b39c9901162de3af20f7a762255: Status 404 returned error can't find the container with id 6662502ef17780c8c171fe5a8eb81af004876b39c9901162de3af20f7a762255 Jul 06 19:11:41.592091 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.592071 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g6qxq" Jul 06 19:11:41.596333 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.596316 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fszg8" Jul 06 19:11:41.600255 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:41.600209 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e43ffe8_6267_4b64_94c9_8b6ce791a0b2.slice/crio-f30f4390fb20b6cfd6aae34efe18b7ef291553db96d382761f692bd38ef1d650 WatchSource:0}: Error finding container f30f4390fb20b6cfd6aae34efe18b7ef291553db96d382761f692bd38ef1d650: Status 404 returned error can't find the container with id f30f4390fb20b6cfd6aae34efe18b7ef291553db96d382761f692bd38ef1d650 Jul 06 19:11:41.603920 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:11:41.603897 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050dc6de_99c6_4fb3_8252_28cb84d52771.slice/crio-3ffe7537e5d1b24614ee089245a7b905ff388db6ade1f3027aea61f8d0eceba8 WatchSource:0}: Error finding container 3ffe7537e5d1b24614ee089245a7b905ff388db6ade1f3027aea61f8d0eceba8: Status 404 returned error can't find the container with id 3ffe7537e5d1b24614ee089245a7b905ff388db6ade1f3027aea61f8d0eceba8 Jul 06 19:11:41.778746 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.778708 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs\") pod \"network-metrics-daemon-sr7tr\" (UID: \"8d718bbb-07dd-481b-81a1-19b7b28c34c1\") " pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:41.778917 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.778792 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret\") pod \"global-pull-secret-syncer-sk86p\" (UID: \"fee777de-2d7a-4470-8a1d-b64c9fad8390\") " pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:41.778917 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.778859 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:41.778917 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.778869 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:41.778917 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.778919 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret podName:fee777de-2d7a-4470-8a1d-b64c9fad8390 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:42.778906988 +0000 UTC m=+3.994544495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret") pod "global-pull-secret-syncer-sk86p" (UID: "fee777de-2d7a-4470-8a1d-b64c9fad8390") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:41.779050 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.778936 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs podName:8d718bbb-07dd-481b-81a1-19b7b28c34c1 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:42.778926052 +0000 UTC m=+3.994563560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs") pod "network-metrics-daemon-sr7tr" (UID: "8d718bbb-07dd-481b-81a1-19b7b28c34c1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:41.880051 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:41.879963 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f7lc\" (UniqueName: \"kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc\") pod \"network-check-target-894sw\" (UID: \"793ee5b4-63f0-4814-b790-8a492924c6d6\") " pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:11:41.880182 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.880092 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:11:41.880182 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.880107 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:11:41.880182 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.880116 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8f7lc for pod openshift-network-diagnostics/network-check-target-894sw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:41.880182 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:41.880166 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc podName:793ee5b4-63f0-4814-b790-8a492924c6d6 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:42.880152603 +0000 UTC m=+4.095790127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8f7lc" (UniqueName: "kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc") pod "network-check-target-894sw" (UID: "793ee5b4-63f0-4814-b790-8a492924c6d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:42.137198 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:42.137121 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:11:42.150707 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:42.150634 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" event={"ID":"a1b502af-fe31-489d-a623-e70d82db0390","Type":"ContainerStarted","Data":"6662502ef17780c8c171fe5a8eb81af004876b39c9901162de3af20f7a762255"} Jul 06 19:11:42.154390 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:42.154322 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" event={"ID":"62552869-4a5c-40e3-bb1a-3b7acec6ff06","Type":"ContainerStarted","Data":"a867a49f931ccb5bf1cab92a2ff110b283cc300ac9afe193665df7e11a8d25db"} Jul 06 19:11:42.156566 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:42.156195 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zlhst" event={"ID":"3d885e22-9b77-4736-a3a4-a5652effa44e","Type":"ContainerStarted","Data":"d87c1f3848afd43614c25919a7933d4d57fdf5a1fb813bcff3b13474e08fbff8"} Jul 06 19:11:42.161275 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:42.161249 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" event={"ID":"cba62fed-78f0-409f-8cc1-744bfbc23165","Type":"ContainerStarted","Data":"df0e8c47595e59cf7a386fc93f8c0fd51dcf288c4ed470461f8880f261fbe027"} Jul 06 19:11:42.170883 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:42.170824 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c2kdq" event={"ID":"400d2c16-5a1f-4a9d-915c-4e69c3833f5a","Type":"ContainerStarted","Data":"486839741f923659ffeea6302c4a16811a13371d4465d74090830ad77851d0f0"} Jul 06 19:11:42.172994 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:42.172970 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z5nvg" event={"ID":"00a50c45-a6c2-442d-b432-f2a776790bbd","Type":"ContainerStarted","Data":"8d87b061f7157830ddc7f340f9fb287d107ae3f0695a6564413a825af7a9f4dd"} Jul 06 19:11:42.178852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:42.178827 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fszg8" event={"ID":"050dc6de-99c6-4fb3-8252-28cb84d52771","Type":"ContainerStarted","Data":"3ffe7537e5d1b24614ee089245a7b905ff388db6ade1f3027aea61f8d0eceba8"} Jul 06 19:11:42.190098 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:42.190071 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g6qxq" event={"ID":"0e43ffe8-6267-4b64-94c9-8b6ce791a0b2","Type":"ContainerStarted","Data":"f30f4390fb20b6cfd6aae34efe18b7ef291553db96d382761f692bd38ef1d650"} Jul 06 19:11:42.199059 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:42.199033 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d9l66" event={"ID":"b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9","Type":"ContainerStarted","Data":"2d3c650b203512d7870a0ffcd647d52da67ac2b3ad504159c04bd8c5281fc6a6"} Jul 06 19:11:42.207678 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:42.205608 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" event={"ID":"1b77e81dd26018ebe7401261e83c3a4d","Type":"ContainerStarted","Data":"6a040cafa748d60c40070c94e3d7f74124142c572ffd2271f7645c4333109acb"} Jul 06 19:11:42.209075 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:42.209023 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" event={"ID":"05e5ff3e6ffb5a5e73115081d86cdae7","Type":"ContainerStarted","Data":"7f237bd36cf7cdd4f06a6b9984e45a28eaf0df87e91108551c7b9d8b0b3b665a"} Jul 06 19:11:42.223534 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:42.223434 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:06:41 +0000 UTC" deadline="2028-04-13 20:38:39.523731622 +0000 UTC" Jul 06 19:11:42.223534 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:42.223488 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15529h26m57.300248008s" Jul 06 19:11:42.787726 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:42.787670 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret\") pod \"global-pull-secret-syncer-sk86p\" (UID: \"fee777de-2d7a-4470-8a1d-b64c9fad8390\") " pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:42.787726 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:42.787723 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs\") pod \"network-metrics-daemon-sr7tr\" (UID: \"8d718bbb-07dd-481b-81a1-19b7b28c34c1\") " pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:42.787973 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:42.787883 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:42.787973 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:42.787940 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs podName:8d718bbb-07dd-481b-81a1-19b7b28c34c1 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:44.78792288 +0000 UTC m=+6.003560391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs") pod "network-metrics-daemon-sr7tr" (UID: "8d718bbb-07dd-481b-81a1-19b7b28c34c1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:42.788336 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:42.788317 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:42.788401 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:42.788372 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret podName:fee777de-2d7a-4470-8a1d-b64c9fad8390 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:44.788357628 +0000 UTC m=+6.003995141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret") pod "global-pull-secret-syncer-sk86p" (UID: "fee777de-2d7a-4470-8a1d-b64c9fad8390") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:42.888805 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:42.888195 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f7lc\" (UniqueName: \"kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc\") pod \"network-check-target-894sw\" (UID: \"793ee5b4-63f0-4814-b790-8a492924c6d6\") " pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:11:42.888805 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:42.888350 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:11:42.888805 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:42.888370 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:11:42.888805 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:42.888383 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8f7lc for pod openshift-network-diagnostics/network-check-target-894sw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:42.888805 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:42.888434 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc podName:793ee5b4-63f0-4814-b790-8a492924c6d6 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:44.888416932 +0000 UTC m=+6.104054443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8f7lc" (UniqueName: "kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc") pod "network-check-target-894sw" (UID: "793ee5b4-63f0-4814-b790-8a492924c6d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:43.143037 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:43.142225 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:43.143037 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:43.142368 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:11:43.143037 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:43.142781 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:11:43.143037 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:43.142871 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-894sw" podUID="793ee5b4-63f0-4814-b790-8a492924c6d6" Jul 06 19:11:43.143037 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:43.142926 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:43.143037 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:43.142992 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk86p" podUID="fee777de-2d7a-4470-8a1d-b64c9fad8390" Jul 06 19:11:43.224671 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:43.224618 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:06:41 +0000 UTC" deadline="2028-03-03 21:31:16.036750531 +0000 UTC" Jul 06 19:11:43.224671 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:43.224669 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14546h19m32.81208693s" Jul 06 19:11:44.806014 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:44.805903 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret\") pod \"global-pull-secret-syncer-sk86p\" (UID: \"fee777de-2d7a-4470-8a1d-b64c9fad8390\") " pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:44.806014 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:44.805960 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs\") pod \"network-metrics-daemon-sr7tr\" (UID: \"8d718bbb-07dd-481b-81a1-19b7b28c34c1\") " pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:44.806467 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:44.806110 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:44.806467 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:44.806164 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs podName:8d718bbb-07dd-481b-81a1-19b7b28c34c1 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:48.806146634 +0000 UTC m=+10.021784146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs") pod "network-metrics-daemon-sr7tr" (UID: "8d718bbb-07dd-481b-81a1-19b7b28c34c1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:44.806467 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:44.806378 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:44.806467 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:44.806441 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret podName:fee777de-2d7a-4470-8a1d-b64c9fad8390 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:48.806423763 +0000 UTC m=+10.022061276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret") pod "global-pull-secret-syncer-sk86p" (UID: "fee777de-2d7a-4470-8a1d-b64c9fad8390") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:44.906840 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:44.906802 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f7lc\" (UniqueName: \"kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc\") pod \"network-check-target-894sw\" (UID: \"793ee5b4-63f0-4814-b790-8a492924c6d6\") " pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:11:44.907080 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:44.907027 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:11:44.907080 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:44.907047 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:11:44.907080 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:44.907059 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8f7lc for pod openshift-network-diagnostics/network-check-target-894sw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:44.907245 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:44.907119 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc podName:793ee5b4-63f0-4814-b790-8a492924c6d6 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:48.907100136 +0000 UTC m=+10.122737644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8f7lc" (UniqueName: "kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc") pod "network-check-target-894sw" (UID: "793ee5b4-63f0-4814-b790-8a492924c6d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:45.142675 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:45.142402 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:45.142675 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:45.142540 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk86p" podUID="fee777de-2d7a-4470-8a1d-b64c9fad8390" Jul 06 19:11:45.142675 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:45.142612 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:45.142918 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:45.142719 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:11:45.142918 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:45.142769 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:11:45.142918 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:45.142832 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-894sw" podUID="793ee5b4-63f0-4814-b790-8a492924c6d6" Jul 06 19:11:47.141817 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:47.141774 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:47.142270 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:47.141828 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:11:47.142270 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:47.141774 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:47.142270 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:47.141981 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:11:47.142270 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:47.142070 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk86p" podUID="fee777de-2d7a-4470-8a1d-b64c9fad8390" Jul 06 19:11:47.142270 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:47.142137 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-894sw" podUID="793ee5b4-63f0-4814-b790-8a492924c6d6" Jul 06 19:11:48.837967 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:48.837914 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret\") pod \"global-pull-secret-syncer-sk86p\" (UID: \"fee777de-2d7a-4470-8a1d-b64c9fad8390\") " pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:48.837967 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:48.837968 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs\") pod \"network-metrics-daemon-sr7tr\" (UID: \"8d718bbb-07dd-481b-81a1-19b7b28c34c1\") " pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:48.838490 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:48.838114 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:48.838490 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:48.838178 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs podName:8d718bbb-07dd-481b-81a1-19b7b28c34c1 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:56.838158502 +0000 UTC m=+18.053796034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs") pod "network-metrics-daemon-sr7tr" (UID: "8d718bbb-07dd-481b-81a1-19b7b28c34c1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:48.838628 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:48.838572 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:48.838710 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:48.838640 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret podName:fee777de-2d7a-4470-8a1d-b64c9fad8390 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:56.838620991 +0000 UTC m=+18.054258502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret") pod "global-pull-secret-syncer-sk86p" (UID: "fee777de-2d7a-4470-8a1d-b64c9fad8390") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:48.938969 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:48.938927 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f7lc\" (UniqueName: \"kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc\") pod \"network-check-target-894sw\" (UID: \"793ee5b4-63f0-4814-b790-8a492924c6d6\") " pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:11:48.939150 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:48.939105 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:11:48.939150 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:48.939127 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:11:48.939150 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:48.939140 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8f7lc for pod openshift-network-diagnostics/network-check-target-894sw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:48.939324 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:48.939205 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc podName:793ee5b4-63f0-4814-b790-8a492924c6d6 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:56.939186925 +0000 UTC m=+18.154824433 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8f7lc" (UniqueName: "kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc") pod "network-check-target-894sw" (UID: "793ee5b4-63f0-4814-b790-8a492924c6d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:49.142301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:49.142220 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:49.142301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:49.142239 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:49.142301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:49.142243 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:11:49.142560 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:49.142348 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk86p" podUID="fee777de-2d7a-4470-8a1d-b64c9fad8390" Jul 06 19:11:49.142560 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:49.142474 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:11:49.142560 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:49.142549 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-894sw" podUID="793ee5b4-63f0-4814-b790-8a492924c6d6" Jul 06 19:11:51.141721 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:51.141688 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:51.142197 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:51.141688 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:51.142197 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:51.141798 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk86p" podUID="fee777de-2d7a-4470-8a1d-b64c9fad8390" Jul 06 19:11:51.142197 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:51.141906 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:11:51.142197 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:51.141688 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:11:51.142197 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:51.142019 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-894sw" podUID="793ee5b4-63f0-4814-b790-8a492924c6d6" Jul 06 19:11:53.141475 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:53.141438 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:11:53.141985 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:53.141438 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:53.141985 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:53.141576 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-894sw" podUID="793ee5b4-63f0-4814-b790-8a492924c6d6" Jul 06 19:11:53.141985 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:53.141632 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk86p" podUID="fee777de-2d7a-4470-8a1d-b64c9fad8390" Jul 06 19:11:53.141985 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:53.141443 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:53.141985 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:53.141759 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:11:55.141621 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:55.141579 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:11:55.142106 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:55.141584 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:55.142106 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:55.141584 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:55.142106 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:55.141732 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-894sw" podUID="793ee5b4-63f0-4814-b790-8a492924c6d6" Jul 06 19:11:55.142106 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:55.141828 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk86p" podUID="fee777de-2d7a-4470-8a1d-b64c9fad8390" Jul 06 19:11:55.142106 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:55.141923 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:11:56.896520 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:56.896437 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret\") pod \"global-pull-secret-syncer-sk86p\" (UID: \"fee777de-2d7a-4470-8a1d-b64c9fad8390\") " pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:56.896520 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:56.896477 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs\") pod \"network-metrics-daemon-sr7tr\" (UID: \"8d718bbb-07dd-481b-81a1-19b7b28c34c1\") " pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:56.896942 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:56.896584 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:56.896942 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:56.896594 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:56.896942 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:56.896648 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret podName:fee777de-2d7a-4470-8a1d-b64c9fad8390 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:12.896632906 +0000 UTC m=+34.112270417 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret") pod "global-pull-secret-syncer-sk86p" (UID: "fee777de-2d7a-4470-8a1d-b64c9fad8390") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:56.896942 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:56.896682 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs podName:8d718bbb-07dd-481b-81a1-19b7b28c34c1 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:12.896674117 +0000 UTC m=+34.112311624 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs") pod "network-metrics-daemon-sr7tr" (UID: "8d718bbb-07dd-481b-81a1-19b7b28c34c1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:56.997367 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:56.997325 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f7lc\" (UniqueName: \"kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc\") pod \"network-check-target-894sw\" (UID: \"793ee5b4-63f0-4814-b790-8a492924c6d6\") " pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:11:56.997547 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:56.997524 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:11:56.997603 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:56.997554 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:11:56.997603 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:56.997569 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8f7lc for pod openshift-network-diagnostics/network-check-target-894sw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:56.997712 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:56.997635 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc podName:793ee5b4-63f0-4814-b790-8a492924c6d6 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:12.997615127 +0000 UTC m=+34.213252645 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8f7lc" (UniqueName: "kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc") pod "network-check-target-894sw" (UID: "793ee5b4-63f0-4814-b790-8a492924c6d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:57.141806 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:57.141771 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:57.141806 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:57.141801 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:57.142029 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:57.141907 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk86p" podUID="fee777de-2d7a-4470-8a1d-b64c9fad8390" Jul 06 19:11:57.142029 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:57.141936 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:11:57.142127 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:57.142036 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:11:57.142195 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:57.142122 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-894sw" podUID="793ee5b4-63f0-4814-b790-8a492924c6d6" Jul 06 19:11:59.142169 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:59.141955 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:11:59.142569 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:59.142022 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:11:59.142569 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:59.142257 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:11:59.142569 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:59.142346 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-894sw" podUID="793ee5b4-63f0-4814-b790-8a492924c6d6" Jul 06 19:11:59.142569 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:11:59.142043 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:11:59.142569 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:11:59.142416 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk86p" podUID="fee777de-2d7a-4470-8a1d-b64c9fad8390" Jul 06 19:12:01.142373 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:01.142338 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:12:01.142793 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:01.142377 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:12:01.142793 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:01.142390 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:12:01.142793 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:01.142471 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-894sw" podUID="793ee5b4-63f0-4814-b790-8a492924c6d6" Jul 06 19:12:01.142793 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:01.142543 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:12:01.142793 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:01.142637 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk86p" podUID="fee777de-2d7a-4470-8a1d-b64c9fad8390" Jul 06 19:12:01.261442 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:01.261406 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" event={"ID":"1b77e81dd26018ebe7401261e83c3a4d","Type":"ContainerStarted","Data":"60bf1731b52cebab10ba62f39bded12b9f4ddbdcca3718e495ab715ceaee9fb7"} Jul 06 19:12:01.277487 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:01.277439 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-192.ec2.internal" podStartSLOduration=21.277402636 podStartE2EDuration="21.277402636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:01.27693028 +0000 UTC m=+22.492567810" watchObservedRunningTime="2026-07-06 19:12:01.277402636 +0000 UTC m=+22.493040167" Jul 06 19:12:02.268481 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.268130 2581 generic.go:356] "Generic (PLEG): container finished" podID="05e5ff3e6ffb5a5e73115081d86cdae7" containerID="fd88cb3838f778e33371a170314f9203e7a7e80688dc5cefd056db7ae0cc64ea" exitCode=0 Jul 06 19:12:02.269329 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.268226 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" event={"ID":"05e5ff3e6ffb5a5e73115081d86cdae7","Type":"ContainerDied","Data":"fd88cb3838f778e33371a170314f9203e7a7e80688dc5cefd056db7ae0cc64ea"} Jul 06 19:12:02.270500 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.270470 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" event={"ID":"a1b502af-fe31-489d-a623-e70d82db0390","Type":"ContainerStarted","Data":"8890aa26fcc588bafc93edfa33fd7c7e6d99596728a4dc0780b6688f96e034f1"} Jul 06 19:12:02.272240 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.272214 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" event={"ID":"62552869-4a5c-40e3-bb1a-3b7acec6ff06","Type":"ContainerStarted","Data":"c7925a089c95caa317b5bd73c0d08d4b2fe6bdcba438d5dc255051118418957e"} Jul 06 19:12:02.273781 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.273757 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zlhst" event={"ID":"3d885e22-9b77-4736-a3a4-a5652effa44e","Type":"ContainerStarted","Data":"ed95ebf052f8c13fdee0fc2fc40e78e63f1546a91c30993b7f9a8a35dd1d3287"} Jul 06 19:12:02.277256 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.277232 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" event={"ID":"cba62fed-78f0-409f-8cc1-744bfbc23165","Type":"ContainerStarted","Data":"73eeb3563424dfd6d9374b4c378674e7677674ea38c44a9591611825ace6922d"} Jul 06 19:12:02.277336 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.277304 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" event={"ID":"cba62fed-78f0-409f-8cc1-744bfbc23165","Type":"ContainerStarted","Data":"36300083de7ffb569e49b0f5af29a227ebf683cdc24dfe3c002e5463c114a059"} Jul 06 19:12:02.277389 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.277352 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" event={"ID":"cba62fed-78f0-409f-8cc1-744bfbc23165","Type":"ContainerStarted","Data":"4f73c70dcf6a02683fc863f232398ed2db33062d5f6ee8877474e99515521180"} Jul 06 19:12:02.277447 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.277399 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" event={"ID":"cba62fed-78f0-409f-8cc1-744bfbc23165","Type":"ContainerStarted","Data":"5553e6163347aec5f78eec779b70f8c7f62e335318cc083c6ad7a97c9cf15add"} Jul 06 19:12:02.277447 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.277444 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" event={"ID":"cba62fed-78f0-409f-8cc1-744bfbc23165","Type":"ContainerStarted","Data":"d68f9da926284906542dad3865dbed41c8519afd781807bfaf00d4f924d2ef85"} Jul 06 19:12:02.277539 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.277490 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" event={"ID":"cba62fed-78f0-409f-8cc1-744bfbc23165","Type":"ContainerStarted","Data":"de04f3e6b5ae0a00b0991b8baffb7c1de99fb3db2eee06a473b6ef12e4fddf56"} Jul 06 19:12:02.278701 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.278679 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z5nvg" event={"ID":"00a50c45-a6c2-442d-b432-f2a776790bbd","Type":"ContainerStarted","Data":"6055f91c6b368c108ba511ece8239358255a37ccaabf422342ca45bec5f11c4e"} Jul 06 19:12:02.280059 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.280034 2581 generic.go:356] "Generic (PLEG): container finished" podID="050dc6de-99c6-4fb3-8252-28cb84d52771" containerID="d9b5040af8779b60b89eec448e3d17290f29cf578c01836023ecdbacd7edf219" exitCode=0 Jul 06 19:12:02.280168 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.280107 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fszg8" event={"ID":"050dc6de-99c6-4fb3-8252-28cb84d52771","Type":"ContainerDied","Data":"d9b5040af8779b60b89eec448e3d17290f29cf578c01836023ecdbacd7edf219"} Jul 06 19:12:02.281811 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.281783 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g6qxq" event={"ID":"0e43ffe8-6267-4b64-94c9-8b6ce791a0b2","Type":"ContainerStarted","Data":"3050a7146cc2eb1895797f3045511f9802d01d03f510d6b0fece000e66e33c01"} Jul 06 19:12:02.283333 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.283308 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d9l66" event={"ID":"b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9","Type":"ContainerStarted","Data":"17944ab31fd7f99e12bc8670c064b38225f1bae47968afae09ab54baee30a0e6"} Jul 06 19:12:02.306576 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.306528 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d9l66" podStartSLOduration=7.285488187 podStartE2EDuration="22.306508586s" totalImagesPullingTime="15.021020399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:02.305991827 +0000 UTC m=+23.521629358" watchObservedRunningTime="2026-07-06 19:12:02.306508586 +0000 UTC m=+23.522146118" Jul 06 19:12:02.323347 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.323256 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lvx7n" podStartSLOduration=2.784447527 podStartE2EDuration="22.323237144s" totalImagesPullingTime="19.538789617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:02.323039588 +0000 UTC m=+23.538677119" watchObservedRunningTime="2026-07-06 19:12:02.323237144 +0000 UTC m=+23.538874675" Jul 06 19:12:02.366357 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.366309 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z5nvg" podStartSLOduration=2.588352658 podStartE2EDuration="22.366289688s" totalImagesPullingTime="19.77793703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:02.365501392 +0000 UTC m=+23.581138922" watchObservedRunningTime="2026-07-06 19:12:02.366289688 +0000 UTC m=+23.581927223" Jul 06 19:12:02.397973 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.397884 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zlhst" podStartSLOduration=2.840994213 podStartE2EDuration="22.397864367s" totalImagesPullingTime="19.556870154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:02.381445738 +0000 UTC m=+23.597083269" watchObservedRunningTime="2026-07-06 19:12:02.397864367 +0000 UTC m=+23.613501897" Jul 06 19:12:02.398318 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:02.398287 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g6qxq" podStartSLOduration=2.889313555 podStartE2EDuration="22.398277312s" totalImagesPullingTime="19.508963757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:02.397816862 +0000 UTC m=+23.613454393" watchObservedRunningTime="2026-07-06 19:12:02.398277312 +0000 UTC m=+23.613914842" Jul 06 19:12:03.142347 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:03.142279 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:12:03.142347 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:03.142297 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:12:03.142347 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:03.142289 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:12:03.142571 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:03.142502 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk86p" podUID="fee777de-2d7a-4470-8a1d-b64c9fad8390" Jul 06 19:12:03.142571 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:03.142542 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-894sw" podUID="793ee5b4-63f0-4814-b790-8a492924c6d6" Jul 06 19:12:03.142704 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:03.142621 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:12:03.149809 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:03.149769 2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 19:12:03.287785 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:03.287694 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" event={"ID":"62552869-4a5c-40e3-bb1a-3b7acec6ff06","Type":"ContainerStarted","Data":"2f52faf2482b0fc194628f7340c61262d59d9f672c8dd6190ada4c6e4bac127c"} Jul 06 19:12:03.289300 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:03.289275 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c2kdq" event={"ID":"400d2c16-5a1f-4a9d-915c-4e69c3833f5a","Type":"ContainerStarted","Data":"d5544cb91e2fba6900d9d3560d2b85a0dcf293dc907db46b91043c352ed146b5"} Jul 06 19:12:03.291119 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:03.291083 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" event={"ID":"05e5ff3e6ffb5a5e73115081d86cdae7","Type":"ContainerStarted","Data":"4ca3a5a51ffa56eef1ceef32114d5dd328647fff4800727bf9069b771adcc2fe"} Jul 06 19:12:03.309571 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:03.309526 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c2kdq" podStartSLOduration=3.715326735 podStartE2EDuration="23.309512153s" totalImagesPullingTime="19.594185418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:03.309031399 +0000 UTC m=+24.524668929" watchObservedRunningTime="2026-07-06 19:12:03.309512153 +0000 UTC m=+24.525149682" Jul 06 19:12:03.330436 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:03.330394 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-192.ec2.internal" podStartSLOduration=23.330374325 podStartE2EDuration="23.330374325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:03.330101417 +0000 UTC m=+24.545738952" watchObservedRunningTime="2026-07-06 19:12:03.330374325 +0000 UTC m=+24.546011858" Jul 06 19:12:03.554127 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:03.554047 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zlhst" Jul 06 19:12:03.554813 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:03.554791 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zlhst" Jul 06 19:12:04.076314 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:04.076180 2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T19:12:03.149781512Z","UUID":"a7fa33a1-7e54-4f37-867a-eecd7b702ebf","Handler":null,"Name":"","Endpoint":""} Jul 06 19:12:04.077833 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:04.077811 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 19:12:04.077833 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:04.077844 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 19:12:05.142078 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:05.142038 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:12:05.142431 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:05.142049 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:12:05.142431 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:05.142158 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk86p" podUID="fee777de-2d7a-4470-8a1d-b64c9fad8390" Jul 06 19:12:05.142431 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:05.142232 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-894sw" podUID="793ee5b4-63f0-4814-b790-8a492924c6d6" Jul 06 19:12:05.142431 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:05.142064 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:12:05.142431 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:05.142321 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:12:05.296967 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:05.296935 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" event={"ID":"62552869-4a5c-40e3-bb1a-3b7acec6ff06","Type":"ContainerStarted","Data":"9e6b618a0164ce17df53345b0ebaf4af8a81e52cf815869285d671a0175ea8b1"} Jul 06 19:12:05.300165 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:05.300137 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" event={"ID":"cba62fed-78f0-409f-8cc1-744bfbc23165","Type":"ContainerStarted","Data":"28986b9b2084a398f4607773303f79a46da1103bc5a9a443449f7538a9775cc5"} Jul 06 19:12:05.300266 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:05.300204 2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 19:12:05.394916 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:05.394825 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5br2" podStartSLOduration=3.181633349 podStartE2EDuration="25.394810472s" totalImagesPullingTime="22.213177123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:05.394709914 +0000 UTC m=+26.610347451" watchObservedRunningTime="2026-07-06 19:12:05.394810472 +0000 UTC m=+26.610448001" Jul 06 19:12:07.142309 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:07.142278 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:12:07.143285 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:07.142353 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:12:07.143285 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:07.142472 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:12:07.143285 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:07.142497 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:12:07.143285 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:07.142594 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk86p" podUID="fee777de-2d7a-4470-8a1d-b64c9fad8390" Jul 06 19:12:07.143285 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:07.142691 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-894sw" podUID="793ee5b4-63f0-4814-b790-8a492924c6d6" Jul 06 19:12:07.310063 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:07.310021 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" event={"ID":"cba62fed-78f0-409f-8cc1-744bfbc23165","Type":"ContainerStarted","Data":"82b1a8edebbbeb943db30522112f5041bed851dd9172b526048e36587848cedd"} Jul 06 19:12:07.310384 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:07.310361 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:12:07.310521 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:07.310437 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:12:07.327993 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:07.327818 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:12:07.390019 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:07.389970 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" podStartSLOduration=7.397129262 podStartE2EDuration="27.389955015s" totalImagesPullingTime="19.992825753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:07.349817 +0000 UTC m=+28.565454533" watchObservedRunningTime="2026-07-06 19:12:07.389955015 +0000 UTC m=+28.605592544" Jul 06 19:12:08.136542 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:08.136507 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zlhst" Jul 06 19:12:08.136747 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:08.136712 2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 19:12:08.137381 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:08.137306 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zlhst" Jul 06 19:12:08.312460 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:08.312430 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:12:08.328331 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:08.328292 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:12:08.637922 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:08.637854 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-894sw"] Jul 06 19:12:08.638161 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:08.637988 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:12:08.638161 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:08.638092 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-894sw" podUID="793ee5b4-63f0-4814-b790-8a492924c6d6" Jul 06 19:12:08.639028 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:08.638991 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sk86p"] Jul 06 19:12:08.639197 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:08.639105 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:12:08.639281 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:08.639198 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk86p" podUID="fee777de-2d7a-4470-8a1d-b64c9fad8390" Jul 06 19:12:08.639855 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:08.639828 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sr7tr"] Jul 06 19:12:08.639954 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:08.639944 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:12:08.640093 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:08.640055 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:12:10.142931 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:10.142904 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:12:10.143271 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:10.142980 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-894sw" podUID="793ee5b4-63f0-4814-b790-8a492924c6d6" Jul 06 19:12:10.143271 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:10.143071 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:12:10.143271 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:10.143201 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:12:10.143271 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:10.143252 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:12:10.143413 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:10.143335 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk86p" podUID="fee777de-2d7a-4470-8a1d-b64c9fad8390" Jul 06 19:12:10.318316 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:10.318277 2581 generic.go:356] "Generic (PLEG): container finished" podID="050dc6de-99c6-4fb3-8252-28cb84d52771" containerID="1d74c219bfc23fe42cd74b4757f883d4d7a6915c2d89a6949ac079ce0226dec0" exitCode=0 Jul 06 19:12:10.318460 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:10.318354 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fszg8" event={"ID":"050dc6de-99c6-4fb3-8252-28cb84d52771","Type":"ContainerDied","Data":"1d74c219bfc23fe42cd74b4757f883d4d7a6915c2d89a6949ac079ce0226dec0"} Jul 06 19:12:12.141794 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:12.141592 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:12:12.142204 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:12.141592 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:12:12.142204 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:12.141881 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:12:12.142204 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:12.141595 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:12:12.142204 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:12.141942 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-894sw" podUID="793ee5b4-63f0-4814-b790-8a492924c6d6" Jul 06 19:12:12.142204 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:12.142012 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sk86p" podUID="fee777de-2d7a-4470-8a1d-b64c9fad8390" Jul 06 19:12:12.324492 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:12.324457 2581 generic.go:356] "Generic (PLEG): container finished" podID="050dc6de-99c6-4fb3-8252-28cb84d52771" containerID="3cdc09409fc52a8dcb319bc31577291c6ac07ab32edb91d8bbd7b39b34ecbbac" exitCode=0 Jul 06 19:12:12.324627 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:12.324519 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fszg8" event={"ID":"050dc6de-99c6-4fb3-8252-28cb84d52771","Type":"ContainerDied","Data":"3cdc09409fc52a8dcb319bc31577291c6ac07ab32edb91d8bbd7b39b34ecbbac"} Jul 06 19:12:12.917074 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:12.917041 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret\") pod \"global-pull-secret-syncer-sk86p\" (UID: \"fee777de-2d7a-4470-8a1d-b64c9fad8390\") " pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:12:12.917074 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:12.917076 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs\") pod \"network-metrics-daemon-sr7tr\" (UID: \"8d718bbb-07dd-481b-81a1-19b7b28c34c1\") " pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:12:12.917275 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:12.917184 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:12:12.917275 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:12.917186 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:12:12.917275 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:12.917235 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs podName:8d718bbb-07dd-481b-81a1-19b7b28c34c1 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:44.91722246 +0000 UTC m=+66.132859968 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs") pod "network-metrics-daemon-sr7tr" (UID: "8d718bbb-07dd-481b-81a1-19b7b28c34c1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:12:12.917275 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:12.917246 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret podName:fee777de-2d7a-4470-8a1d-b64c9fad8390 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:44.917241005 +0000 UTC m=+66.132878512 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret") pod "global-pull-secret-syncer-sk86p" (UID: "fee777de-2d7a-4470-8a1d-b64c9fad8390") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:12:13.017456 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:13.017413 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f7lc\" (UniqueName: \"kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc\") pod \"network-check-target-894sw\" (UID: \"793ee5b4-63f0-4814-b790-8a492924c6d6\") " pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:12:13.017612 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:13.017577 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:12:13.017612 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:13.017599 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:12:13.017612 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:13.017609 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8f7lc for pod openshift-network-diagnostics/network-check-target-894sw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:12:13.017792 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:13.017680 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc podName:793ee5b4-63f0-4814-b790-8a492924c6d6 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:45.017647846 +0000 UTC m=+66.233285354 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8f7lc" (UniqueName: "kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc") pod "network-check-target-894sw" (UID: "793ee5b4-63f0-4814-b790-8a492924c6d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:12:14.101406 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.101381 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-192.ec2.internal" event="NodeReady" Jul 06 19:12:14.101979 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.101510 2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 19:12:14.141473 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.141449 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:12:14.141473 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.141463 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:12:14.141722 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.141455 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:12:14.145056 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.145038 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:12:14.145906 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.145891 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r97mf\"" Jul 06 19:12:14.146723 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.146705 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:12:14.146960 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.146946 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:12:14.147041 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.147024 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:12:14.152171 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.152154 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-422bs\"" Jul 06 19:12:14.237979 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.237952 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6xknf"] Jul 06 19:12:14.253582 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.253561 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4rhqw"] Jul 06 19:12:14.253708 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.253691 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6xknf" Jul 06 19:12:14.261035 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.261014 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 19:12:14.261140 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.261049 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 19:12:14.261195 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.261172 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zsvz6\"" Jul 06 19:12:14.262031 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.262016 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6xknf"] Jul 06 19:12:14.262108 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.262090 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:14.270193 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.270177 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 19:12:14.271945 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.271932 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 19:12:14.272154 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.272142 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 19:12:14.276316 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.276302 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c8xcq\"" Jul 06 19:12:14.288770 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.288749 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4rhqw"] Jul 06 19:12:14.327758 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.327734 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:14.327921 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.327774 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c87bed4-66aa-420d-98e1-8071ede06a4b-tmp-dir\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:14.327921 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.327811 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c87bed4-66aa-420d-98e1-8071ede06a4b-config-volume\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:14.327921 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.327834 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxjnw\" (UniqueName: \"kubernetes.io/projected/218097a1-0eb0-4a54-af89-ba9b3d5c4586-kube-api-access-bxjnw\") pod \"ingress-canary-6xknf\" (UID: \"218097a1-0eb0-4a54-af89-ba9b3d5c4586\") " pod="openshift-ingress-canary/ingress-canary-6xknf" Jul 06 19:12:14.327921 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.327883 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmwvl\" (UniqueName: \"kubernetes.io/projected/9c87bed4-66aa-420d-98e1-8071ede06a4b-kube-api-access-cmwvl\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:14.327921 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.327899 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/218097a1-0eb0-4a54-af89-ba9b3d5c4586-cert\") pod \"ingress-canary-6xknf\" (UID: \"218097a1-0eb0-4a54-af89-ba9b3d5c4586\") " pod="openshift-ingress-canary/ingress-canary-6xknf" Jul 06 19:12:14.331416 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.331397 2581 generic.go:356] "Generic (PLEG): container finished" podID="050dc6de-99c6-4fb3-8252-28cb84d52771" containerID="faa18afb81fe0e0e4525992957b8d73d660d96c4c2ab55cc3b57406b7c0aa0af" exitCode=0 Jul 06 19:12:14.331512 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.331431 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fszg8" event={"ID":"050dc6de-99c6-4fb3-8252-28cb84d52771","Type":"ContainerDied","Data":"faa18afb81fe0e0e4525992957b8d73d660d96c4c2ab55cc3b57406b7c0aa0af"} Jul 06 19:12:14.428852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.428813 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:14.429011 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.428880 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c87bed4-66aa-420d-98e1-8071ede06a4b-tmp-dir\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:14.429011 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.428918 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c87bed4-66aa-420d-98e1-8071ede06a4b-config-volume\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:14.429011 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.428936 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxjnw\" (UniqueName: \"kubernetes.io/projected/218097a1-0eb0-4a54-af89-ba9b3d5c4586-kube-api-access-bxjnw\") pod \"ingress-canary-6xknf\" (UID: \"218097a1-0eb0-4a54-af89-ba9b3d5c4586\") " pod="openshift-ingress-canary/ingress-canary-6xknf" Jul 06 19:12:14.429175 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:14.429005 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:14.429175 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.429070 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmwvl\" (UniqueName: \"kubernetes.io/projected/9c87bed4-66aa-420d-98e1-8071ede06a4b-kube-api-access-cmwvl\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:14.429175 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:14.429084 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls podName:9c87bed4-66aa-420d-98e1-8071ede06a4b nodeName:}" failed. No retries permitted until 2026-07-06 19:12:14.92906214 +0000 UTC m=+36.144699651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls") pod "dns-default-4rhqw" (UID: "9c87bed4-66aa-420d-98e1-8071ede06a4b") : secret "dns-default-metrics-tls" not found Jul 06 19:12:14.429175 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.429121 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/218097a1-0eb0-4a54-af89-ba9b3d5c4586-cert\") pod \"ingress-canary-6xknf\" (UID: \"218097a1-0eb0-4a54-af89-ba9b3d5c4586\") " pod="openshift-ingress-canary/ingress-canary-6xknf" Jul 06 19:12:14.429397 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.429249 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c87bed4-66aa-420d-98e1-8071ede06a4b-tmp-dir\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:14.430345 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.429878 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c87bed4-66aa-420d-98e1-8071ede06a4b-config-volume\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:14.433992 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.433971 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/218097a1-0eb0-4a54-af89-ba9b3d5c4586-cert\") pod \"ingress-canary-6xknf\" (UID: \"218097a1-0eb0-4a54-af89-ba9b3d5c4586\") " pod="openshift-ingress-canary/ingress-canary-6xknf" Jul 06 19:12:14.440507 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.440487 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxjnw\" (UniqueName: \"kubernetes.io/projected/218097a1-0eb0-4a54-af89-ba9b3d5c4586-kube-api-access-bxjnw\") pod \"ingress-canary-6xknf\" (UID: \"218097a1-0eb0-4a54-af89-ba9b3d5c4586\") " pod="openshift-ingress-canary/ingress-canary-6xknf" Jul 06 19:12:14.446382 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.446360 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmwvl\" (UniqueName: \"kubernetes.io/projected/9c87bed4-66aa-420d-98e1-8071ede06a4b-kube-api-access-cmwvl\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:14.562013 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.561980 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6xknf" Jul 06 19:12:14.760935 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.760491 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6xknf"] Jul 06 19:12:14.768072 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:12:14.768039 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218097a1_0eb0_4a54_af89_ba9b3d5c4586.slice/crio-76e548d3224ad1726800267327700d58ea6f25aacc329608d9e1cc12d540d104 WatchSource:0}: Error finding container 76e548d3224ad1726800267327700d58ea6f25aacc329608d9e1cc12d540d104: Status 404 returned error can't find the container with id 76e548d3224ad1726800267327700d58ea6f25aacc329608d9e1cc12d540d104 Jul 06 19:12:14.932482 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:14.932443 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:14.932675 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:14.932593 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:14.932747 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:14.932691 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls podName:9c87bed4-66aa-420d-98e1-8071ede06a4b nodeName:}" failed. No retries permitted until 2026-07-06 19:12:15.93266755 +0000 UTC m=+37.148305076 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls") pod "dns-default-4rhqw" (UID: "9c87bed4-66aa-420d-98e1-8071ede06a4b") : secret "dns-default-metrics-tls" not found Jul 06 19:12:15.335909 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:15.335870 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6xknf" event={"ID":"218097a1-0eb0-4a54-af89-ba9b3d5c4586","Type":"ContainerStarted","Data":"76e548d3224ad1726800267327700d58ea6f25aacc329608d9e1cc12d540d104"} Jul 06 19:12:15.939093 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:15.939053 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:15.939321 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:15.939243 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:15.939384 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:15.939321 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls podName:9c87bed4-66aa-420d-98e1-8071ede06a4b nodeName:}" failed. No retries permitted until 2026-07-06 19:12:17.939300412 +0000 UTC m=+39.154937923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls") pod "dns-default-4rhqw" (UID: "9c87bed4-66aa-420d-98e1-8071ede06a4b") : secret "dns-default-metrics-tls" not found Jul 06 19:12:17.954118 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:17.954028 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:17.954495 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:17.954195 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:17.954495 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:17.954283 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls podName:9c87bed4-66aa-420d-98e1-8071ede06a4b nodeName:}" failed. No retries permitted until 2026-07-06 19:12:21.95426099 +0000 UTC m=+43.169898512 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls") pod "dns-default-4rhqw" (UID: "9c87bed4-66aa-420d-98e1-8071ede06a4b") : secret "dns-default-metrics-tls" not found Jul 06 19:12:21.353613 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:21.353574 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6xknf" event={"ID":"218097a1-0eb0-4a54-af89-ba9b3d5c4586","Type":"ContainerStarted","Data":"10a640fdab90ab6f17f6f523e52d267626ac24ac823a0662aba0e79b7c4553e6"} Jul 06 19:12:21.356236 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:21.356205 2581 generic.go:356] "Generic (PLEG): container finished" podID="050dc6de-99c6-4fb3-8252-28cb84d52771" containerID="1f642d895f48e71f1587e589ca44f39f85a2a66721f72ee806519b5383ca327f" exitCode=0 Jul 06 19:12:21.356382 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:21.356251 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fszg8" event={"ID":"050dc6de-99c6-4fb3-8252-28cb84d52771","Type":"ContainerDied","Data":"1f642d895f48e71f1587e589ca44f39f85a2a66721f72ee806519b5383ca327f"} Jul 06 19:12:21.369723 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:21.369675 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6xknf" podStartSLOduration=1.667407577 podStartE2EDuration="7.369644089s" totalImagesPullingTime="5.702236512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:21.369549632 +0000 UTC m=+42.585187163" watchObservedRunningTime="2026-07-06 19:12:21.369644089 +0000 UTC m=+42.585281619" Jul 06 19:12:21.983094 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:21.983058 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:21.983274 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:21.983231 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:21.983315 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:21.983299 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls podName:9c87bed4-66aa-420d-98e1-8071ede06a4b nodeName:}" failed. No retries permitted until 2026-07-06 19:12:29.983283813 +0000 UTC m=+51.198921325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls") pod "dns-default-4rhqw" (UID: "9c87bed4-66aa-420d-98e1-8071ede06a4b") : secret "dns-default-metrics-tls" not found Jul 06 19:12:22.361593 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:22.361505 2581 generic.go:356] "Generic (PLEG): container finished" podID="050dc6de-99c6-4fb3-8252-28cb84d52771" containerID="633bad04e0d9bc72091194200b247f85ccb41b5b8e94eff5aa789773c3504107" exitCode=0 Jul 06 19:12:22.361593 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:22.361580 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fszg8" event={"ID":"050dc6de-99c6-4fb3-8252-28cb84d52771","Type":"ContainerDied","Data":"633bad04e0d9bc72091194200b247f85ccb41b5b8e94eff5aa789773c3504107"} Jul 06 19:12:23.367449 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:23.367404 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fszg8" event={"ID":"050dc6de-99c6-4fb3-8252-28cb84d52771","Type":"ContainerStarted","Data":"a5e25b2ed70b57a70c202ded9dc4881aa074976a6c038d43cd15c7447df8af9a"} Jul 06 19:12:23.395962 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:23.395898 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fszg8" podStartSLOduration=7.775203926 podStartE2EDuration="43.395881562s" totalImagesPullingTime="35.620677636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:23.394301158 +0000 UTC m=+44.609938689" watchObservedRunningTime="2026-07-06 19:12:23.395881562 +0000 UTC m=+44.611519165" Jul 06 19:12:30.042443 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:30.042404 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:30.042973 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:30.042559 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:30.042973 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:30.042635 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls podName:9c87bed4-66aa-420d-98e1-8071ede06a4b nodeName:}" failed. No retries permitted until 2026-07-06 19:12:46.042619009 +0000 UTC m=+67.258256516 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls") pod "dns-default-4rhqw" (UID: "9c87bed4-66aa-420d-98e1-8071ede06a4b") : secret "dns-default-metrics-tls" not found Jul 06 19:12:40.329288 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:40.329259 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c6c7d" Jul 06 19:12:44.956318 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:44.956264 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret\") pod \"global-pull-secret-syncer-sk86p\" (UID: \"fee777de-2d7a-4470-8a1d-b64c9fad8390\") " pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:12:44.956318 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:44.956324 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs\") pod \"network-metrics-daemon-sr7tr\" (UID: \"8d718bbb-07dd-481b-81a1-19b7b28c34c1\") " pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:12:44.959418 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:44.959395 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:12:44.959531 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:44.959439 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:12:44.967181 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:44.967158 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 19:12:44.967257 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:44.967244 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs podName:8d718bbb-07dd-481b-81a1-19b7b28c34c1 nodeName:}" failed. No retries permitted until 2026-07-06 19:13:48.967222658 +0000 UTC m=+130.182860188 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs") pod "network-metrics-daemon-sr7tr" (UID: "8d718bbb-07dd-481b-81a1-19b7b28c34c1") : secret "metrics-daemon-secret" not found Jul 06 19:12:44.970108 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:44.970089 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fee777de-2d7a-4470-8a1d-b64c9fad8390-original-pull-secret\") pod \"global-pull-secret-syncer-sk86p\" (UID: \"fee777de-2d7a-4470-8a1d-b64c9fad8390\") " pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:12:45.056722 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:45.056687 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sk86p" Jul 06 19:12:45.056887 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:45.056789 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f7lc\" (UniqueName: \"kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc\") pod \"network-check-target-894sw\" (UID: \"793ee5b4-63f0-4814-b790-8a492924c6d6\") " pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:12:45.062472 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:45.062454 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:12:45.070089 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:45.070066 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:12:45.080234 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:45.080205 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f7lc\" (UniqueName: \"kubernetes.io/projected/793ee5b4-63f0-4814-b790-8a492924c6d6-kube-api-access-8f7lc\") pod \"network-check-target-894sw\" (UID: \"793ee5b4-63f0-4814-b790-8a492924c6d6\") " pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:12:45.187074 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:45.187044 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sk86p"] Jul 06 19:12:45.190025 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:12:45.189994 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee777de_2d7a_4470_8a1d_b64c9fad8390.slice/crio-d5425750ba87d5c1fa39d2c92741aae3841d78730de5e49eae8d61d02f0b3b1f WatchSource:0}: Error finding container d5425750ba87d5c1fa39d2c92741aae3841d78730de5e49eae8d61d02f0b3b1f: Status 404 returned error can't find the container with id d5425750ba87d5c1fa39d2c92741aae3841d78730de5e49eae8d61d02f0b3b1f Jul 06 19:12:45.354429 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:45.354345 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-422bs\"" Jul 06 19:12:45.361762 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:45.361737 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:12:45.419549 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:45.419515 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sk86p" event={"ID":"fee777de-2d7a-4470-8a1d-b64c9fad8390","Type":"ContainerStarted","Data":"d5425750ba87d5c1fa39d2c92741aae3841d78730de5e49eae8d61d02f0b3b1f"} Jul 06 19:12:45.506289 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:45.506263 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-894sw"] Jul 06 19:12:45.508518 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:12:45.508496 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793ee5b4_63f0_4814_b790_8a492924c6d6.slice/crio-baca05fb918e362d9562363802a8ba50cbd140fb99beb04605dd0e23d26cd63c WatchSource:0}: Error finding container baca05fb918e362d9562363802a8ba50cbd140fb99beb04605dd0e23d26cd63c: Status 404 returned error can't find the container with id baca05fb918e362d9562363802a8ba50cbd140fb99beb04605dd0e23d26cd63c Jul 06 19:12:46.061772 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:46.061739 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:12:46.062178 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:46.061858 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:46.062178 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:12:46.061913 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls podName:9c87bed4-66aa-420d-98e1-8071ede06a4b nodeName:}" failed. No retries permitted until 2026-07-06 19:13:18.061897504 +0000 UTC m=+99.277535012 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls") pod "dns-default-4rhqw" (UID: "9c87bed4-66aa-420d-98e1-8071ede06a4b") : secret "dns-default-metrics-tls" not found Jul 06 19:12:46.423932 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:46.423875 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-894sw" event={"ID":"793ee5b4-63f0-4814-b790-8a492924c6d6","Type":"ContainerStarted","Data":"baca05fb918e362d9562363802a8ba50cbd140fb99beb04605dd0e23d26cd63c"} Jul 06 19:12:51.441604 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:51.441559 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sk86p" event={"ID":"fee777de-2d7a-4470-8a1d-b64c9fad8390","Type":"ContainerStarted","Data":"0fa25968f383ddcd49d2ad0a14286e7f965bb979460e2b1194257af3de4a67d5"} Jul 06 19:12:51.442993 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:51.442963 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-894sw" event={"ID":"793ee5b4-63f0-4814-b790-8a492924c6d6","Type":"ContainerStarted","Data":"6f409c2e3f6d8f0cb747ab6d2bdcf02d5a1e09e1212b6c81c9916cbb58fd2c2f"} Jul 06 19:12:51.443100 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:51.443059 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:12:51.461824 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:51.461785 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sk86p" podStartSLOduration=66.20885362 podStartE2EDuration="1m11.461770249s" totalImagesPullingTime="5.252916629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:51.461131604 +0000 UTC m=+72.676769135" watchObservedRunningTime="2026-07-06 19:12:51.461770249 +0000 UTC m=+72.677407778" Jul 06 19:12:51.479193 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:12:51.479147 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-894sw" podStartSLOduration=66.544379409 podStartE2EDuration="1m11.47913137s" totalImagesPullingTime="4.934751961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:51.478721281 +0000 UTC m=+72.694358808" watchObservedRunningTime="2026-07-06 19:12:51.47913137 +0000 UTC m=+72.694768899" Jul 06 19:13:18.043641 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.043190 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr"] Jul 06 19:13:18.045991 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.045971 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" Jul 06 19:13:18.050821 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.050798 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 19:13:18.052407 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.052271 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 19:13:18.052407 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.052295 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 19:13:18.052407 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.052352 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 06 19:13:18.070400 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.070364 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c95q\" (UniqueName: \"kubernetes.io/projected/6bbd37d5-18c6-46a9-8865-a71990ed07e6-kube-api-access-9c95q\") pod \"klusterlet-addon-workmgr-6f78bcc85b-wsqwr\" (UID: \"6bbd37d5-18c6-46a9-8865-a71990ed07e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" Jul 06 19:13:18.070574 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.070424 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:13:18.070574 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.070487 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bbd37d5-18c6-46a9-8865-a71990ed07e6-tmp\") pod \"klusterlet-addon-workmgr-6f78bcc85b-wsqwr\" (UID: \"6bbd37d5-18c6-46a9-8865-a71990ed07e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" Jul 06 19:13:18.070574 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.070517 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6bbd37d5-18c6-46a9-8865-a71990ed07e6-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f78bcc85b-wsqwr\" (UID: \"6bbd37d5-18c6-46a9-8865-a71990ed07e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" Jul 06 19:13:18.070574 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:13:18.070528 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:13:18.070729 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:13:18.070620 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls podName:9c87bed4-66aa-420d-98e1-8071ede06a4b nodeName:}" failed. No retries permitted until 2026-07-06 19:14:22.070606356 +0000 UTC m=+163.286243864 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls") pod "dns-default-4rhqw" (UID: "9c87bed4-66aa-420d-98e1-8071ede06a4b") : secret "dns-default-metrics-tls" not found Jul 06 19:13:18.075897 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.075869 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr"] Jul 06 19:13:18.171179 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.171147 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bbd37d5-18c6-46a9-8865-a71990ed07e6-tmp\") pod \"klusterlet-addon-workmgr-6f78bcc85b-wsqwr\" (UID: \"6bbd37d5-18c6-46a9-8865-a71990ed07e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" Jul 06 19:13:18.171319 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.171184 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6bbd37d5-18c6-46a9-8865-a71990ed07e6-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f78bcc85b-wsqwr\" (UID: \"6bbd37d5-18c6-46a9-8865-a71990ed07e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" Jul 06 19:13:18.171319 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.171225 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c95q\" (UniqueName: \"kubernetes.io/projected/6bbd37d5-18c6-46a9-8865-a71990ed07e6-kube-api-access-9c95q\") pod \"klusterlet-addon-workmgr-6f78bcc85b-wsqwr\" (UID: \"6bbd37d5-18c6-46a9-8865-a71990ed07e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" Jul 06 19:13:18.171589 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.171569 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6bbd37d5-18c6-46a9-8865-a71990ed07e6-tmp\") pod \"klusterlet-addon-workmgr-6f78bcc85b-wsqwr\" (UID: \"6bbd37d5-18c6-46a9-8865-a71990ed07e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" Jul 06 19:13:18.173631 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.173610 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6bbd37d5-18c6-46a9-8865-a71990ed07e6-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f78bcc85b-wsqwr\" (UID: \"6bbd37d5-18c6-46a9-8865-a71990ed07e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" Jul 06 19:13:18.190874 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.190840 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c95q\" (UniqueName: \"kubernetes.io/projected/6bbd37d5-18c6-46a9-8865-a71990ed07e6-kube-api-access-9c95q\") pod \"klusterlet-addon-workmgr-6f78bcc85b-wsqwr\" (UID: \"6bbd37d5-18c6-46a9-8865-a71990ed07e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" Jul 06 19:13:18.355305 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.355209 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" Jul 06 19:13:18.490068 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.490040 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr"] Jul 06 19:13:18.494424 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:13:18.494398 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbd37d5_18c6_46a9_8865_a71990ed07e6.slice/crio-0fb8de903a2c67b2ccd64c5759bb453916a949ff13d6ee970ebc0a7455bd2ddf WatchSource:0}: Error finding container 0fb8de903a2c67b2ccd64c5759bb453916a949ff13d6ee970ebc0a7455bd2ddf: Status 404 returned error can't find the container with id 0fb8de903a2c67b2ccd64c5759bb453916a949ff13d6ee970ebc0a7455bd2ddf Jul 06 19:13:18.508281 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:18.508248 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" event={"ID":"6bbd37d5-18c6-46a9-8865-a71990ed07e6","Type":"ContainerStarted","Data":"0fb8de903a2c67b2ccd64c5759bb453916a949ff13d6ee970ebc0a7455bd2ddf"} Jul 06 19:13:22.448043 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:22.448013 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-894sw" Jul 06 19:13:44.569771 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:44.569734 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" event={"ID":"6bbd37d5-18c6-46a9-8865-a71990ed07e6","Type":"ContainerStarted","Data":"8f2fb00e43c1c8c6fe0769762ea4190781e0e9d549102b05d7cf31b51bcfe2ea"} Jul 06 19:13:44.570190 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:44.569991 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" Jul 06 19:13:44.571553 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:44.571534 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" Jul 06 19:13:44.599900 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:44.599854 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" podStartSLOduration=1.027647174 podStartE2EDuration="26.599841105s" totalImagesPullingTime="25.572193931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:13:44.599587179 +0000 UTC m=+125.815224709" watchObservedRunningTime="2026-07-06 19:13:44.599841105 +0000 UTC m=+125.815478683" Jul 06 19:13:48.976481 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:13:48.976444 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs\") pod \"network-metrics-daemon-sr7tr\" (UID: \"8d718bbb-07dd-481b-81a1-19b7b28c34c1\") " pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:13:48.977016 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:13:48.976599 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 19:13:48.977016 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:13:48.976725 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs podName:8d718bbb-07dd-481b-81a1-19b7b28c34c1 nodeName:}" failed. No retries permitted until 2026-07-06 19:15:50.976701995 +0000 UTC m=+252.192339517 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs") pod "network-metrics-daemon-sr7tr" (UID: "8d718bbb-07dd-481b-81a1-19b7b28c34c1") : secret "metrics-daemon-secret" not found Jul 06 19:14:17.162148 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:14:17.162106 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:14:17.269700 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:14:17.269644 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4rhqw" podUID="9c87bed4-66aa-420d-98e1-8071ede06a4b" Jul 06 19:14:17.647085 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:14:17.647015 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4rhqw" Jul 06 19:14:22.086428 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:14:22.086383 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:14:22.086897 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:14:22.086553 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:14:22.086897 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:14:22.086648 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls podName:9c87bed4-66aa-420d-98e1-8071ede06a4b nodeName:}" failed. No retries permitted until 2026-07-06 19:16:24.086628522 +0000 UTC m=+285.302266030 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls") pod "dns-default-4rhqw" (UID: "9c87bed4-66aa-420d-98e1-8071ede06a4b") : secret "dns-default-metrics-tls" not found Jul 06 19:14:32.141718 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:14:32.141614 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:15:44.571065 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:44.571010 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" podUID="6bbd37d5-18c6-46a9-8865-a71990ed07e6" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused" Jul 06 19:15:44.851175 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:44.851087 2581 generic.go:356] "Generic (PLEG): container finished" podID="6bbd37d5-18c6-46a9-8865-a71990ed07e6" containerID="8f2fb00e43c1c8c6fe0769762ea4190781e0e9d549102b05d7cf31b51bcfe2ea" exitCode=1 Jul 06 19:15:44.851175 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:44.851122 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" event={"ID":"6bbd37d5-18c6-46a9-8865-a71990ed07e6","Type":"ContainerDied","Data":"8f2fb00e43c1c8c6fe0769762ea4190781e0e9d549102b05d7cf31b51bcfe2ea"} Jul 06 19:15:44.851457 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:44.851442 2581 scope.go:117] "RemoveContainer" containerID="8f2fb00e43c1c8c6fe0769762ea4190781e0e9d549102b05d7cf31b51bcfe2ea" Jul 06 19:15:45.855215 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:45.855182 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" event={"ID":"6bbd37d5-18c6-46a9-8865-a71990ed07e6","Type":"ContainerStarted","Data":"71cf0a5dbcc5dbbce5c5f19e46d0c4b007ebf6e1ec5262d10605115b356872f1"} Jul 06 19:15:45.855615 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:45.855465 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" Jul 06 19:15:45.855952 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:45.855937 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f78bcc85b-wsqwr" Jul 06 19:15:50.982820 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:50.982772 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs\") pod \"network-metrics-daemon-sr7tr\" (UID: \"8d718bbb-07dd-481b-81a1-19b7b28c34c1\") " pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:15:50.983305 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:50.982918 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 19:15:50.983305 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:50.982979 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs podName:8d718bbb-07dd-481b-81a1-19b7b28c34c1 nodeName:}" failed. No retries permitted until 2026-07-06 19:17:52.982964165 +0000 UTC m=+374.198601673 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs") pod "network-metrics-daemon-sr7tr" (UID: "8d718bbb-07dd-481b-81a1-19b7b28c34c1") : secret "metrics-daemon-secret" not found Jul 06 19:15:56.542734 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.542705 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75b5544c69-jf4f5"] Jul 06 19:15:56.547155 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.547119 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.549669 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.549629 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 19:15:56.549805 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.549684 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 19:15:56.550050 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.550035 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 19:15:56.550118 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.550102 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lwjk9\"" Jul 06 19:15:56.556701 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.556683 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 19:15:56.566467 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.566446 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75b5544c69-jf4f5"] Jul 06 19:15:56.623522 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.623490 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ae9e185-f263-472c-8ca9-03ed1f4f689c-image-registry-private-configuration\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.623522 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.623525 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ae9e185-f263-472c-8ca9-03ed1f4f689c-ca-trust-extracted\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.623775 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.623546 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ae9e185-f263-472c-8ca9-03ed1f4f689c-installation-pull-secrets\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.623775 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.623562 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-bound-sa-token\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.623775 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.623579 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-certificates\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.623775 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.623595 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ae9e185-f263-472c-8ca9-03ed1f4f689c-trusted-ca\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.623775 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.623635 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7qnw\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-kube-api-access-z7qnw\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.623775 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.623709 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.646925 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.646888 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-ftqzp"] Jul 06 19:15:56.649533 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.649516 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5df75489c9-898zs"] Jul 06 19:15:56.649687 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.649668 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.652080 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.652060 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:56.655091 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.655065 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 19:15:56.655265 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.655251 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 19:15:56.655340 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.655303 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 06 19:15:56.655399 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.655378 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 19:15:56.655600 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.655582 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 06 19:15:56.655600 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.655581 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 06 19:15:56.655784 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.655630 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ncvb2\"" Jul 06 19:15:56.655784 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.655664 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 06 19:15:56.655882 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.655866 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-khbc9\"" Jul 06 19:15:56.656521 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.656505 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 19:15:56.657360 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.657342 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 06 19:15:56.657446 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.657362 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 06 19:15:56.665057 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.665040 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 06 19:15:56.670400 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.670380 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-ftqzp"] Jul 06 19:15:56.695855 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.695822 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5df75489c9-898zs"] Jul 06 19:15:56.724729 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.724700 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.724872 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.724740 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-serving-cert\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.724872 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.724825 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.724872 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:56.724858 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:15:56.724872 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:56.724870 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b5544c69-jf4f5: secret "image-registry-tls" not found Jul 06 19:15:56.724997 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.724856 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ae9e185-f263-472c-8ca9-03ed1f4f689c-ca-trust-extracted\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.724997 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:56.724923 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls podName:2ae9e185-f263-472c-8ca9-03ed1f4f689c nodeName:}" failed. No retries permitted until 2026-07-06 19:15:57.224904877 +0000 UTC m=+258.440542403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls") pod "image-registry-75b5544c69-jf4f5" (UID: "2ae9e185-f263-472c-8ca9-03ed1f4f689c") : secret "image-registry-tls" not found Jul 06 19:15:56.724997 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.724942 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7qnw\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-kube-api-access-z7qnw\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.724997 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.724960 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-default-certificate\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:56.724997 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.724986 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ae9e185-f263-472c-8ca9-03ed1f4f689c-installation-pull-secrets\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.725194 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.725011 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.725194 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.725069 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:56.725194 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.725111 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-snapshots\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.725194 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.725134 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tgnm\" (UniqueName: \"kubernetes.io/projected/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-kube-api-access-4tgnm\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.725194 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.725163 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ae9e185-f263-472c-8ca9-03ed1f4f689c-ca-trust-extracted\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.725194 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.725188 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ae9e185-f263-472c-8ca9-03ed1f4f689c-trusted-ca\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.725451 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.725213 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ae9e185-f263-472c-8ca9-03ed1f4f689c-image-registry-private-configuration\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.725451 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.725245 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-certificates\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.725451 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.725263 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-bound-sa-token\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.725451 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.725277 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-tmp\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.725451 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.725297 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-stats-auth\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:56.725451 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.725320 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:56.725451 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.725411 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9wsr\" (UniqueName: \"kubernetes.io/projected/51a04452-ffcd-45fe-b844-519e1c90426b-kube-api-access-b9wsr\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:56.725835 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.725814 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-certificates\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.726079 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.726063 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ae9e185-f263-472c-8ca9-03ed1f4f689c-trusted-ca\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.727698 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.727678 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ae9e185-f263-472c-8ca9-03ed1f4f689c-installation-pull-secrets\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.727816 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.727767 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ae9e185-f263-472c-8ca9-03ed1f4f689c-image-registry-private-configuration\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.740202 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.740175 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7qnw\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-kube-api-access-z7qnw\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.745190 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.745171 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg"] Jul 06 19:15:56.746184 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.746165 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-bound-sa-token\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:56.748994 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.748976 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-ntfk2"] Jul 06 19:15:56.749164 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.749133 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg" Jul 06 19:15:56.751866 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.751850 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:15:56.757179 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.757160 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 06 19:15:56.757430 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.757414 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 06 19:15:56.757643 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.757627 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 06 19:15:56.757701 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.757631 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:15:56.759937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.759920 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:15:56.760095 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.759960 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 06 19:15:56.760095 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.759970 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 06 19:15:56.760379 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.760362 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-r5sv8\"" Jul 06 19:15:56.760702 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.760688 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-84d5h\"" Jul 06 19:15:56.761543 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.761526 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 06 19:15:56.768260 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.768242 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 06 19:15:56.787142 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.787121 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg"] Jul 06 19:15:56.787818 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.787802 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-ntfk2"] Jul 06 19:15:56.826627 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.826552 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-tmp\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.826627 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.826589 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-stats-auth\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:56.826627 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.826610 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:56.826892 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.826636 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x7kk\" (UniqueName: \"kubernetes.io/projected/20ac7de1-b189-4c87-b6a0-9b45a480d3f0-kube-api-access-2x7kk\") pod \"console-operator-68b757fb67-ntfk2\" (UID: \"20ac7de1-b189-4c87-b6a0-9b45a480d3f0\") " pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:15:56.826892 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.826676 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9wsr\" (UniqueName: \"kubernetes.io/projected/51a04452-ffcd-45fe-b844-519e1c90426b-kube-api-access-b9wsr\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:56.826892 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.826706 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d3ee22af-f46c-445f-97dd-e25e4206aa7c-config\") pod \"service-ca-operator-5d46d9ffd4-wsgdg\" (UID: \"d3ee22af-f46c-445f-97dd-e25e4206aa7c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg" Jul 06 19:15:56.826892 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.826744 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-serving-cert\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.826892 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:56.826745 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:15:56.826892 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.826776 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20ac7de1-b189-4c87-b6a0-9b45a480d3f0-serving-cert\") pod \"console-operator-68b757fb67-ntfk2\" (UID: \"20ac7de1-b189-4c87-b6a0-9b45a480d3f0\") " pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:15:56.826892 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:56.826827 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs podName:51a04452-ffcd-45fe-b844-519e1c90426b nodeName:}" failed. No retries permitted until 2026-07-06 19:15:57.326803953 +0000 UTC m=+258.542441472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs") pod "router-default-5df75489c9-898zs" (UID: "51a04452-ffcd-45fe-b844-519e1c90426b") : secret "router-metrics-certs-default" not found Jul 06 19:15:56.827228 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.826943 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.827228 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.826980 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-default-certificate\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:56.827228 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.827006 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d3ee22af-f46c-445f-97dd-e25e4206aa7c-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-wsgdg\" (UID: \"d3ee22af-f46c-445f-97dd-e25e4206aa7c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg" Jul 06 19:15:56.827228 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.827013 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-tmp\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.827228 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.827033 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20ac7de1-b189-4c87-b6a0-9b45a480d3f0-trusted-ca\") pod \"console-operator-68b757fb67-ntfk2\" (UID: \"20ac7de1-b189-4c87-b6a0-9b45a480d3f0\") " pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:15:56.827228 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.827067 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.827228 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.827146 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:56.827228 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.827174 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-snapshots\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.827228 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.827206 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tgnm\" (UniqueName: \"kubernetes.io/projected/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-kube-api-access-4tgnm\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.827700 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.827232 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjrdb\" (UniqueName: \"kubernetes.io/projected/d3ee22af-f46c-445f-97dd-e25e4206aa7c-kube-api-access-gjrdb\") pod \"service-ca-operator-5d46d9ffd4-wsgdg\" (UID: \"d3ee22af-f46c-445f-97dd-e25e4206aa7c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg" Jul 06 19:15:56.827700 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.827262 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20ac7de1-b189-4c87-b6a0-9b45a480d3f0-config\") pod \"console-operator-68b757fb67-ntfk2\" (UID: \"20ac7de1-b189-4c87-b6a0-9b45a480d3f0\") " pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:15:56.827700 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:56.827333 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle podName:51a04452-ffcd-45fe-b844-519e1c90426b nodeName:}" failed. No retries permitted until 2026-07-06 19:15:57.327315668 +0000 UTC m=+258.542953196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle") pod "router-default-5df75489c9-898zs" (UID: "51a04452-ffcd-45fe-b844-519e1c90426b") : configmap references non-existent config key: service-ca.crt Jul 06 19:15:56.827700 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.827695 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.827892 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.827824 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-snapshots\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.828030 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.828010 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.829587 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.829559 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-serving-cert\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.829715 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.829692 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-stats-auth\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:56.829845 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.829780 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-default-certificate\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:56.839573 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.839552 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9wsr\" (UniqueName: \"kubernetes.io/projected/51a04452-ffcd-45fe-b844-519e1c90426b-kube-api-access-b9wsr\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:56.839717 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.839700 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tgnm\" (UniqueName: \"kubernetes.io/projected/a65794b0-70cc-4b6f-bc1c-cd741bfd9de6-kube-api-access-4tgnm\") pod \"insights-operator-5c847dcfc4-ftqzp\" (UID: \"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6\") " pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:56.927969 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.927930 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x7kk\" (UniqueName: \"kubernetes.io/projected/20ac7de1-b189-4c87-b6a0-9b45a480d3f0-kube-api-access-2x7kk\") pod \"console-operator-68b757fb67-ntfk2\" (UID: \"20ac7de1-b189-4c87-b6a0-9b45a480d3f0\") " pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:15:56.927969 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.927973 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d3ee22af-f46c-445f-97dd-e25e4206aa7c-config\") pod \"service-ca-operator-5d46d9ffd4-wsgdg\" (UID: \"d3ee22af-f46c-445f-97dd-e25e4206aa7c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg" Jul 06 19:15:56.928181 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.928073 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20ac7de1-b189-4c87-b6a0-9b45a480d3f0-serving-cert\") pod \"console-operator-68b757fb67-ntfk2\" (UID: \"20ac7de1-b189-4c87-b6a0-9b45a480d3f0\") " pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:15:56.928181 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.928116 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d3ee22af-f46c-445f-97dd-e25e4206aa7c-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-wsgdg\" (UID: \"d3ee22af-f46c-445f-97dd-e25e4206aa7c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg" Jul 06 19:15:56.928181 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.928132 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20ac7de1-b189-4c87-b6a0-9b45a480d3f0-trusted-ca\") pod \"console-operator-68b757fb67-ntfk2\" (UID: \"20ac7de1-b189-4c87-b6a0-9b45a480d3f0\") " pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:15:56.928181 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.928166 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjrdb\" (UniqueName: \"kubernetes.io/projected/d3ee22af-f46c-445f-97dd-e25e4206aa7c-kube-api-access-gjrdb\") pod \"service-ca-operator-5d46d9ffd4-wsgdg\" (UID: \"d3ee22af-f46c-445f-97dd-e25e4206aa7c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg" Jul 06 19:15:56.928378 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.928286 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20ac7de1-b189-4c87-b6a0-9b45a480d3f0-config\") pod \"console-operator-68b757fb67-ntfk2\" (UID: \"20ac7de1-b189-4c87-b6a0-9b45a480d3f0\") " pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:15:56.928721 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.928690 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d3ee22af-f46c-445f-97dd-e25e4206aa7c-config\") pod \"service-ca-operator-5d46d9ffd4-wsgdg\" (UID: \"d3ee22af-f46c-445f-97dd-e25e4206aa7c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg" Jul 06 19:15:56.928918 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.928898 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20ac7de1-b189-4c87-b6a0-9b45a480d3f0-config\") pod \"console-operator-68b757fb67-ntfk2\" (UID: \"20ac7de1-b189-4c87-b6a0-9b45a480d3f0\") " pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:15:56.929197 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.929172 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20ac7de1-b189-4c87-b6a0-9b45a480d3f0-trusted-ca\") pod \"console-operator-68b757fb67-ntfk2\" (UID: \"20ac7de1-b189-4c87-b6a0-9b45a480d3f0\") " pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:15:56.930406 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.930379 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d3ee22af-f46c-445f-97dd-e25e4206aa7c-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-wsgdg\" (UID: \"d3ee22af-f46c-445f-97dd-e25e4206aa7c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg" Jul 06 19:15:56.930504 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.930476 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20ac7de1-b189-4c87-b6a0-9b45a480d3f0-serving-cert\") pod \"console-operator-68b757fb67-ntfk2\" (UID: \"20ac7de1-b189-4c87-b6a0-9b45a480d3f0\") " pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:15:56.938933 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.938908 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjrdb\" (UniqueName: \"kubernetes.io/projected/d3ee22af-f46c-445f-97dd-e25e4206aa7c-kube-api-access-gjrdb\") pod \"service-ca-operator-5d46d9ffd4-wsgdg\" (UID: \"d3ee22af-f46c-445f-97dd-e25e4206aa7c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg" Jul 06 19:15:56.939135 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.939119 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x7kk\" (UniqueName: \"kubernetes.io/projected/20ac7de1-b189-4c87-b6a0-9b45a480d3f0-kube-api-access-2x7kk\") pod \"console-operator-68b757fb67-ntfk2\" (UID: \"20ac7de1-b189-4c87-b6a0-9b45a480d3f0\") " pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:15:56.960086 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:56.960067 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" Jul 06 19:15:57.059093 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:57.059064 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg" Jul 06 19:15:57.065172 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:57.065139 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:15:57.087334 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:57.087264 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-ftqzp"] Jul 06 19:15:57.090384 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:15:57.090354 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65794b0_70cc_4b6f_bc1c_cd741bfd9de6.slice/crio-053ea99256d06930a416d5fae926ad1886083745b71aef555bff100b0ebcde34 WatchSource:0}: Error finding container 053ea99256d06930a416d5fae926ad1886083745b71aef555bff100b0ebcde34: Status 404 returned error can't find the container with id 053ea99256d06930a416d5fae926ad1886083745b71aef555bff100b0ebcde34 Jul 06 19:15:57.193222 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:57.193190 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg"] Jul 06 19:15:57.196244 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:15:57.196220 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ee22af_f46c_445f_97dd_e25e4206aa7c.slice/crio-9eccc2df8aba982ff0775cd5c207ea5001419dbd97dfbbdce7581fe378a46207 WatchSource:0}: Error finding container 9eccc2df8aba982ff0775cd5c207ea5001419dbd97dfbbdce7581fe378a46207: Status 404 returned error can't find the container with id 9eccc2df8aba982ff0775cd5c207ea5001419dbd97dfbbdce7581fe378a46207 Jul 06 19:15:57.208434 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:57.208411 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-ntfk2"] Jul 06 19:15:57.211126 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:15:57.211103 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ac7de1_b189_4c87_b6a0_9b45a480d3f0.slice/crio-37d220a62d72951959f61ff4da7b616251ef4bfbea4e62c2202caa434e1804b0 WatchSource:0}: Error finding container 37d220a62d72951959f61ff4da7b616251ef4bfbea4e62c2202caa434e1804b0: Status 404 returned error can't find the container with id 37d220a62d72951959f61ff4da7b616251ef4bfbea4e62c2202caa434e1804b0 Jul 06 19:15:57.231965 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:57.231937 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:57.232137 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:57.232118 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:15:57.232206 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:57.232140 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b5544c69-jf4f5: secret "image-registry-tls" not found Jul 06 19:15:57.232206 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:57.232192 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls podName:2ae9e185-f263-472c-8ca9-03ed1f4f689c nodeName:}" failed. No retries permitted until 2026-07-06 19:15:58.232176108 +0000 UTC m=+259.447813616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls") pod "image-registry-75b5544c69-jf4f5" (UID: "2ae9e185-f263-472c-8ca9-03ed1f4f689c") : secret "image-registry-tls" not found Jul 06 19:15:57.332561 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:57.332534 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:57.332749 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:57.332702 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:15:57.332823 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:57.332702 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:57.332823 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:57.332767 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs podName:51a04452-ffcd-45fe-b844-519e1c90426b nodeName:}" failed. No retries permitted until 2026-07-06 19:15:58.332748652 +0000 UTC m=+259.548386165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs") pod "router-default-5df75489c9-898zs" (UID: "51a04452-ffcd-45fe-b844-519e1c90426b") : secret "router-metrics-certs-default" not found Jul 06 19:15:57.332934 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:57.332838 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle podName:51a04452-ffcd-45fe-b844-519e1c90426b nodeName:}" failed. No retries permitted until 2026-07-06 19:15:58.332819609 +0000 UTC m=+259.548457117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle") pod "router-default-5df75489c9-898zs" (UID: "51a04452-ffcd-45fe-b844-519e1c90426b") : configmap references non-existent config key: service-ca.crt Jul 06 19:15:57.887848 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:57.887809 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" event={"ID":"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6","Type":"ContainerStarted","Data":"053ea99256d06930a416d5fae926ad1886083745b71aef555bff100b0ebcde34"} Jul 06 19:15:57.888991 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:57.888958 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" event={"ID":"20ac7de1-b189-4c87-b6a0-9b45a480d3f0","Type":"ContainerStarted","Data":"37d220a62d72951959f61ff4da7b616251ef4bfbea4e62c2202caa434e1804b0"} Jul 06 19:15:57.890139 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:57.890115 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg" event={"ID":"d3ee22af-f46c-445f-97dd-e25e4206aa7c","Type":"ContainerStarted","Data":"9eccc2df8aba982ff0775cd5c207ea5001419dbd97dfbbdce7581fe378a46207"} Jul 06 19:15:58.238971 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:58.238920 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:15:58.239154 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:58.239097 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:15:58.239154 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:58.239120 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b5544c69-jf4f5: secret "image-registry-tls" not found Jul 06 19:15:58.239288 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:58.239180 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls podName:2ae9e185-f263-472c-8ca9-03ed1f4f689c nodeName:}" failed. No retries permitted until 2026-07-06 19:16:00.239163032 +0000 UTC m=+261.454800549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls") pod "image-registry-75b5544c69-jf4f5" (UID: "2ae9e185-f263-472c-8ca9-03ed1f4f689c") : secret "image-registry-tls" not found Jul 06 19:15:58.339844 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:58.339815 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:58.340002 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:15:58.339868 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:15:58.340002 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:58.339979 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:15:58.340002 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:58.339996 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle podName:51a04452-ffcd-45fe-b844-519e1c90426b nodeName:}" failed. No retries permitted until 2026-07-06 19:16:00.339976821 +0000 UTC m=+261.555614329 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle") pod "router-default-5df75489c9-898zs" (UID: "51a04452-ffcd-45fe-b844-519e1c90426b") : configmap references non-existent config key: service-ca.crt Jul 06 19:15:58.340132 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:15:58.340028 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs podName:51a04452-ffcd-45fe-b844-519e1c90426b nodeName:}" failed. No retries permitted until 2026-07-06 19:16:00.340013087 +0000 UTC m=+261.555650598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs") pod "router-default-5df75489c9-898zs" (UID: "51a04452-ffcd-45fe-b844-519e1c90426b") : secret "router-metrics-certs-default" not found Jul 06 19:16:00.256936 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:00.256724 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:16:00.256936 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:00.256915 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:16:00.256936 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:00.256936 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b5544c69-jf4f5: secret "image-registry-tls" not found Jul 06 19:16:00.257460 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:00.257017 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls podName:2ae9e185-f263-472c-8ca9-03ed1f4f689c nodeName:}" failed. No retries permitted until 2026-07-06 19:16:04.256994923 +0000 UTC m=+265.472632441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls") pod "image-registry-75b5544c69-jf4f5" (UID: "2ae9e185-f263-472c-8ca9-03ed1f4f689c") : secret "image-registry-tls" not found Jul 06 19:16:00.357849 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:00.357813 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:16:00.358027 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:00.357874 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:16:00.358027 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:00.358013 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle podName:51a04452-ffcd-45fe-b844-519e1c90426b nodeName:}" failed. No retries permitted until 2026-07-06 19:16:04.357991831 +0000 UTC m=+265.573629343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle") pod "router-default-5df75489c9-898zs" (UID: "51a04452-ffcd-45fe-b844-519e1c90426b") : configmap references non-existent config key: service-ca.crt Jul 06 19:16:00.358127 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:00.358046 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:16:00.358127 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:00.358105 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs podName:51a04452-ffcd-45fe-b844-519e1c90426b nodeName:}" failed. No retries permitted until 2026-07-06 19:16:04.358091197 +0000 UTC m=+265.573728705 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs") pod "router-default-5df75489c9-898zs" (UID: "51a04452-ffcd-45fe-b844-519e1c90426b") : secret "router-metrics-certs-default" not found Jul 06 19:16:00.900587 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:00.900509 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/0.log" Jul 06 19:16:00.900587 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:00.900549 2581 generic.go:356] "Generic (PLEG): container finished" podID="20ac7de1-b189-4c87-b6a0-9b45a480d3f0" containerID="ef97a7cfa0cbf64de67e621c89d9448a524454159d7dfdf02e00a9faf5811122" exitCode=255 Jul 06 19:16:00.900834 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:00.900617 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" event={"ID":"20ac7de1-b189-4c87-b6a0-9b45a480d3f0","Type":"ContainerDied","Data":"ef97a7cfa0cbf64de67e621c89d9448a524454159d7dfdf02e00a9faf5811122"} Jul 06 19:16:00.900933 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:00.900916 2581 scope.go:117] "RemoveContainer" containerID="ef97a7cfa0cbf64de67e621c89d9448a524454159d7dfdf02e00a9faf5811122" Jul 06 19:16:00.902239 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:00.902212 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg" event={"ID":"d3ee22af-f46c-445f-97dd-e25e4206aa7c","Type":"ContainerStarted","Data":"3e9161bdc407a5aa416f420dae5988c1d1151551f16a28089d9dd42ac9d66ea7"} Jul 06 19:16:00.903735 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:00.903697 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" event={"ID":"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6","Type":"ContainerStarted","Data":"76e2b372c15073c9aa30bc784cd65c4687e0d740c2b3eb0a2821375768ed6efa"} Jul 06 19:16:00.959423 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:00.959106 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" podStartSLOduration=1.398163455 podStartE2EDuration="4.95908595s" totalImagesPullingTime="3.560922495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:00.941237702 +0000 UTC m=+262.156875233" watchObservedRunningTime="2026-07-06 19:16:00.95908595 +0000 UTC m=+262.174723481" Jul 06 19:16:00.959423 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:00.959301 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg" podStartSLOduration=1.496792592 podStartE2EDuration="4.959295186s" totalImagesPullingTime="3.462502594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:00.958702687 +0000 UTC m=+262.174340216" watchObservedRunningTime="2026-07-06 19:16:00.959295186 +0000 UTC m=+262.174932717" Jul 06 19:16:01.909983 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:01.909953 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:16:01.910591 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:01.910569 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/0.log" Jul 06 19:16:01.910804 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:01.910767 2581 generic.go:356] "Generic (PLEG): container finished" podID="20ac7de1-b189-4c87-b6a0-9b45a480d3f0" containerID="0f3a0ad5b0254668b6a1ff982182230648b2bce6c2e2847ca9b9261b47ce243e" exitCode=255 Jul 06 19:16:01.910925 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:01.910841 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" event={"ID":"20ac7de1-b189-4c87-b6a0-9b45a480d3f0","Type":"ContainerDied","Data":"0f3a0ad5b0254668b6a1ff982182230648b2bce6c2e2847ca9b9261b47ce243e"} Jul 06 19:16:01.910991 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:01.910927 2581 scope.go:117] "RemoveContainer" containerID="ef97a7cfa0cbf64de67e621c89d9448a524454159d7dfdf02e00a9faf5811122" Jul 06 19:16:01.911978 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:01.911923 2581 scope.go:117] "RemoveContainer" containerID="0f3a0ad5b0254668b6a1ff982182230648b2bce6c2e2847ca9b9261b47ce243e" Jul 06 19:16:01.912165 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:01.912143 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-ntfk2_openshift-console-operator(20ac7de1-b189-4c87-b6a0-9b45a480d3f0)\"" pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" podUID="20ac7de1-b189-4c87-b6a0-9b45a480d3f0" Jul 06 19:16:02.455353 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.455323 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6djr7"] Jul 06 19:16:02.459526 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.459508 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:02.461997 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.461972 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 19:16:02.462346 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.462333 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 19:16:02.463136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.463120 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2n6g9\"" Jul 06 19:16:02.471409 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.471387 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6djr7"] Jul 06 19:16:02.578047 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.578010 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c099894f-48eb-4431-acca-5464590389f8-data-volume\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:02.578222 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.578121 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:02.578222 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.578153 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:02.578222 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.578198 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg95p\" (UniqueName: \"kubernetes.io/projected/c099894f-48eb-4431-acca-5464590389f8-kube-api-access-wg95p\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:02.578334 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.578252 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c099894f-48eb-4431-acca-5464590389f8-crio-socket\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:02.678951 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.678913 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:02.679163 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.678953 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:02.679163 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.678980 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg95p\" (UniqueName: \"kubernetes.io/projected/c099894f-48eb-4431-acca-5464590389f8-kube-api-access-wg95p\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:02.679163 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.679023 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c099894f-48eb-4431-acca-5464590389f8-crio-socket\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:02.679163 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:02.679041 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:16:02.679163 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:02.679099 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:02.679163 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:02.679117 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm podName:c099894f-48eb-4431-acca-5464590389f8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:03.179090174 +0000 UTC m=+264.394727690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6djr7" (UID: "c099894f-48eb-4431-acca-5464590389f8") : configmap "kube-rbac-proxy" not found Jul 06 19:16:02.679163 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.679141 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c099894f-48eb-4431-acca-5464590389f8-data-volume\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:02.679163 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:02.679151 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls podName:c099894f-48eb-4431-acca-5464590389f8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:03.179136269 +0000 UTC m=+264.394773791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6djr7" (UID: "c099894f-48eb-4431-acca-5464590389f8") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:02.679163 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.679151 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c099894f-48eb-4431-acca-5464590389f8-crio-socket\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:02.679466 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.679447 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c099894f-48eb-4431-acca-5464590389f8-data-volume\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:02.687930 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.687899 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg95p\" (UniqueName: \"kubernetes.io/projected/c099894f-48eb-4431-acca-5464590389f8-kube-api-access-wg95p\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:02.915749 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.915719 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:16:02.916129 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:02.916040 2581 scope.go:117] "RemoveContainer" containerID="0f3a0ad5b0254668b6a1ff982182230648b2bce6c2e2847ca9b9261b47ce243e" Jul 06 19:16:02.916200 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:02.916182 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-ntfk2_openshift-console-operator(20ac7de1-b189-4c87-b6a0-9b45a480d3f0)\"" pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" podUID="20ac7de1-b189-4c87-b6a0-9b45a480d3f0" Jul 06 19:16:03.183036 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:03.182935 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:03.183036 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:03.182979 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:03.183248 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:03.183053 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:16:03.183248 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:03.183089 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:03.183248 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:03.183120 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm podName:c099894f-48eb-4431-acca-5464590389f8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:04.183104188 +0000 UTC m=+265.398741696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6djr7" (UID: "c099894f-48eb-4431-acca-5464590389f8") : configmap "kube-rbac-proxy" not found Jul 06 19:16:03.183248 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:03.183140 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls podName:c099894f-48eb-4431-acca-5464590389f8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:04.183125494 +0000 UTC m=+265.398763008 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6djr7" (UID: "c099894f-48eb-4431-acca-5464590389f8") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:04.192161 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:04.192127 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:04.192161 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:04.192162 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:04.192605 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:04.192241 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:16:04.192605 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:04.192301 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm podName:c099894f-48eb-4431-acca-5464590389f8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:06.192286087 +0000 UTC m=+267.407923595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6djr7" (UID: "c099894f-48eb-4431-acca-5464590389f8") : configmap "kube-rbac-proxy" not found Jul 06 19:16:04.192605 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:04.192318 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:04.192605 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:04.192366 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls podName:c099894f-48eb-4431-acca-5464590389f8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:06.192351059 +0000 UTC m=+267.407988569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6djr7" (UID: "c099894f-48eb-4431-acca-5464590389f8") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:04.240400 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:04.240378 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g6qxq_0e43ffe8-6267-4b64-94c9-8b6ce791a0b2/dns-node-resolver/0.log" Jul 06 19:16:04.293239 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:04.293213 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:16:04.293355 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:04.293342 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:16:04.293397 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:04.293357 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75b5544c69-jf4f5: secret "image-registry-tls" not found Jul 06 19:16:04.293397 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:04.293396 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls podName:2ae9e185-f263-472c-8ca9-03ed1f4f689c nodeName:}" failed. No retries permitted until 2026-07-06 19:16:12.29338425 +0000 UTC m=+273.509021762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls") pod "image-registry-75b5544c69-jf4f5" (UID: "2ae9e185-f263-472c-8ca9-03ed1f4f689c") : secret "image-registry-tls" not found Jul 06 19:16:04.393805 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:04.393778 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:16:04.393925 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:04.393849 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:16:04.393968 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:04.393921 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:16:04.393968 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:04.393955 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle podName:51a04452-ffcd-45fe-b844-519e1c90426b nodeName:}" failed. No retries permitted until 2026-07-06 19:16:12.393944159 +0000 UTC m=+273.609581667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle") pod "router-default-5df75489c9-898zs" (UID: "51a04452-ffcd-45fe-b844-519e1c90426b") : configmap references non-existent config key: service-ca.crt Jul 06 19:16:04.394060 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:04.393970 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs podName:51a04452-ffcd-45fe-b844-519e1c90426b nodeName:}" failed. No retries permitted until 2026-07-06 19:16:12.393964261 +0000 UTC m=+273.609601769 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs") pod "router-default-5df75489c9-898zs" (UID: "51a04452-ffcd-45fe-b844-519e1c90426b") : secret "router-metrics-certs-default" not found Jul 06 19:16:05.641865 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:05.641834 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d9l66_b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9/node-ca/0.log" Jul 06 19:16:06.208410 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:06.208371 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:06.208410 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:06.208413 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:06.208581 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:06.208485 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:16:06.208581 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:06.208524 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:06.208581 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:06.208544 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm podName:c099894f-48eb-4431-acca-5464590389f8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:10.208528301 +0000 UTC m=+271.424165808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6djr7" (UID: "c099894f-48eb-4431-acca-5464590389f8") : configmap "kube-rbac-proxy" not found Jul 06 19:16:06.208581 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:06.208567 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls podName:c099894f-48eb-4431-acca-5464590389f8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:10.208555927 +0000 UTC m=+271.424193435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6djr7" (UID: "c099894f-48eb-4431-acca-5464590389f8") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:06.241635 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:06.241609 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6xknf_218097a1-0eb0-4a54-af89-ba9b3d5c4586/serve-healthcheck-canary/0.log" Jul 06 19:16:07.065725 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:07.065692 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:16:07.066100 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:07.065746 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:16:07.066100 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:07.066046 2581 scope.go:117] "RemoveContainer" containerID="0f3a0ad5b0254668b6a1ff982182230648b2bce6c2e2847ca9b9261b47ce243e" Jul 06 19:16:07.066211 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:07.066191 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-ntfk2_openshift-console-operator(20ac7de1-b189-4c87-b6a0-9b45a480d3f0)\"" pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" podUID="20ac7de1-b189-4c87-b6a0-9b45a480d3f0" Jul 06 19:16:10.240255 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:10.240210 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:10.240255 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:10.240256 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:10.240723 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:10.240330 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:16:10.240723 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:10.240392 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm podName:c099894f-48eb-4431-acca-5464590389f8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:18.240377366 +0000 UTC m=+279.456014878 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6djr7" (UID: "c099894f-48eb-4431-acca-5464590389f8") : configmap "kube-rbac-proxy" not found Jul 06 19:16:10.240723 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:10.240396 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:10.240723 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:10.240435 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls podName:c099894f-48eb-4431-acca-5464590389f8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:18.240424338 +0000 UTC m=+279.456061846 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6djr7" (UID: "c099894f-48eb-4431-acca-5464590389f8") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:12.356664 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:12.356615 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:16:12.358963 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:12.358935 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls\") pod \"image-registry-75b5544c69-jf4f5\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:16:12.455933 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:12.455898 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:16:12.457096 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:12.457076 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:16:12.457190 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:12.457176 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:16:12.457264 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:12.457247 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle podName:51a04452-ffcd-45fe-b844-519e1c90426b nodeName:}" failed. No retries permitted until 2026-07-06 19:16:28.457229723 +0000 UTC m=+289.672867230 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle") pod "router-default-5df75489c9-898zs" (UID: "51a04452-ffcd-45fe-b844-519e1c90426b") : configmap references non-existent config key: service-ca.crt Jul 06 19:16:12.457333 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:12.457310 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 19:16:12.457387 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:12.457369 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs podName:51a04452-ffcd-45fe-b844-519e1c90426b nodeName:}" failed. No retries permitted until 2026-07-06 19:16:28.457353212 +0000 UTC m=+289.672990723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs") pod "router-default-5df75489c9-898zs" (UID: "51a04452-ffcd-45fe-b844-519e1c90426b") : secret "router-metrics-certs-default" not found Jul 06 19:16:12.577359 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:12.577322 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75b5544c69-jf4f5"] Jul 06 19:16:12.581634 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:16:12.581604 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae9e185_f263_472c_8ca9_03ed1f4f689c.slice/crio-30ee3f88f1fb1b16de81a0b6063958009a60b3c8c773d806169eae02e0ad4823 WatchSource:0}: Error finding container 30ee3f88f1fb1b16de81a0b6063958009a60b3c8c773d806169eae02e0ad4823: Status 404 returned error can't find the container with id 30ee3f88f1fb1b16de81a0b6063958009a60b3c8c773d806169eae02e0ad4823 Jul 06 19:16:12.944928 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:12.944893 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" event={"ID":"2ae9e185-f263-472c-8ca9-03ed1f4f689c","Type":"ContainerStarted","Data":"265b21d24ebf185a0e4bb60a3d11bc628dde2780cc539f3c2e40b5aecdf02742"} Jul 06 19:16:12.945095 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:12.944946 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" event={"ID":"2ae9e185-f263-472c-8ca9-03ed1f4f689c","Type":"ContainerStarted","Data":"30ee3f88f1fb1b16de81a0b6063958009a60b3c8c773d806169eae02e0ad4823"} Jul 06 19:16:12.945095 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:12.945038 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:16:12.965313 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:12.965263 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" podStartSLOduration=16.965245777 podStartE2EDuration="16.965245777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:12.964060263 +0000 UTC m=+274.179697793" watchObservedRunningTime="2026-07-06 19:16:12.965245777 +0000 UTC m=+274.180883305" Jul 06 19:16:18.302398 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:18.302359 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:18.302398 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:18.302393 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:18.302885 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:18.302480 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:18.302885 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:18.302536 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls podName:c099894f-48eb-4431-acca-5464590389f8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:34.3025178 +0000 UTC m=+295.518155308 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6djr7" (UID: "c099894f-48eb-4431-acca-5464590389f8") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:18.302977 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:18.302897 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c099894f-48eb-4431-acca-5464590389f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:20.144296 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:20.144264 2581 scope.go:117] "RemoveContainer" containerID="0f3a0ad5b0254668b6a1ff982182230648b2bce6c2e2847ca9b9261b47ce243e" Jul 06 19:16:20.648023 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:20.647983 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4rhqw" podUID="9c87bed4-66aa-420d-98e1-8071ede06a4b" Jul 06 19:16:20.970122 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:20.970098 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:16:20.970283 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:20.970211 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4rhqw" Jul 06 19:16:20.970283 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:20.970215 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" event={"ID":"20ac7de1-b189-4c87-b6a0-9b45a480d3f0","Type":"ContainerStarted","Data":"4d385f9661f53349f6f120152bf08f46b48b35717984ba156d74eb15613d05ce"} Jul 06 19:16:20.970483 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:20.970468 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:16:20.975085 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:20.975064 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" Jul 06 19:16:20.987894 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:20.987852 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-ntfk2" podStartSLOduration=21.543322899 podStartE2EDuration="24.987839791s" totalImagesPullingTime="3.444516892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:20.987281145 +0000 UTC m=+282.202918674" watchObservedRunningTime="2026-07-06 19:16:20.987839791 +0000 UTC m=+282.203477354" Jul 06 19:16:24.149047 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:24.149019 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:16:24.151275 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:24.151241 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c87bed4-66aa-420d-98e1-8071ede06a4b-metrics-tls\") pod \"dns-default-4rhqw\" (UID: \"9c87bed4-66aa-420d-98e1-8071ede06a4b\") " pod="openshift-dns/dns-default-4rhqw" Jul 06 19:16:24.273882 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:24.273847 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c8xcq\"" Jul 06 19:16:24.281949 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:24.281924 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4rhqw" Jul 06 19:16:24.432016 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:16:24.431983 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c87bed4_66aa_420d_98e1_8071ede06a4b.slice/crio-f0bff527f68865fd369ec4f01a03fdc951b462e8956c37bbe95f7ac467f3007c WatchSource:0}: Error finding container f0bff527f68865fd369ec4f01a03fdc951b462e8956c37bbe95f7ac467f3007c: Status 404 returned error can't find the container with id f0bff527f68865fd369ec4f01a03fdc951b462e8956c37bbe95f7ac467f3007c Jul 06 19:16:24.432306 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:24.432281 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4rhqw"] Jul 06 19:16:24.983615 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:24.983578 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4rhqw" event={"ID":"9c87bed4-66aa-420d-98e1-8071ede06a4b","Type":"ContainerStarted","Data":"f0bff527f68865fd369ec4f01a03fdc951b462e8956c37bbe95f7ac467f3007c"} Jul 06 19:16:27.980820 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:27.980782 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75b5544c69-jf4f5"] Jul 06 19:16:27.985261 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:27.985232 2581 patch_prober.go:28] interesting pod/image-registry-75b5544c69-jf4f5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 19:16:27.985397 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:27.985279 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" podUID="2ae9e185-f263-472c-8ca9-03ed1f4f689c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:16:27.997365 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:27.997331 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4rhqw" event={"ID":"9c87bed4-66aa-420d-98e1-8071ede06a4b","Type":"ContainerStarted","Data":"0ddc386ae01c926c8e2f86e8ce4086f7055d830c91ae55cf8504615733660393"} Jul 06 19:16:27.997542 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:27.997391 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4rhqw" event={"ID":"9c87bed4-66aa-420d-98e1-8071ede06a4b","Type":"ContainerStarted","Data":"d04183544f33f279a75f23b57c65751ad6d04cde13f5f51dbae8c822c544ad2f"} Jul 06 19:16:27.997542 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:27.997423 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4rhqw" Jul 06 19:16:28.034060 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.034009 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4rhqw" podStartSLOduration=251.287988046 podStartE2EDuration="4m14.033990956s" totalImagesPullingTime="2.74600291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:28.033201973 +0000 UTC m=+289.248839504" watchObservedRunningTime="2026-07-06 19:16:28.033990956 +0000 UTC m=+289.249628485" Jul 06 19:16:28.066131 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.066089 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74b5cb95d8-x7rkn"] Jul 06 19:16:28.068397 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.068363 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.087822 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.087790 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74b5cb95d8-x7rkn"] Jul 06 19:16:28.183477 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.183442 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63ea9f5d-efb4-4d37-9690-183d90437225-bound-sa-token\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.183477 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.183477 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63ea9f5d-efb4-4d37-9690-183d90437225-trusted-ca\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.183714 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.183507 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63ea9f5d-efb4-4d37-9690-183d90437225-registry-tls\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.183714 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.183575 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63ea9f5d-efb4-4d37-9690-183d90437225-ca-trust-extracted\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.183714 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.183608 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63ea9f5d-efb4-4d37-9690-183d90437225-image-registry-private-configuration\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.183714 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.183640 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm69x\" (UniqueName: \"kubernetes.io/projected/63ea9f5d-efb4-4d37-9690-183d90437225-kube-api-access-jm69x\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.183836 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.183724 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63ea9f5d-efb4-4d37-9690-183d90437225-installation-pull-secrets\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.183836 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.183755 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63ea9f5d-efb4-4d37-9690-183d90437225-registry-certificates\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.284574 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.284470 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63ea9f5d-efb4-4d37-9690-183d90437225-registry-certificates\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.284574 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.284561 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63ea9f5d-efb4-4d37-9690-183d90437225-bound-sa-token\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.284848 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.284588 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63ea9f5d-efb4-4d37-9690-183d90437225-trusted-ca\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.284848 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.284624 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63ea9f5d-efb4-4d37-9690-183d90437225-registry-tls\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.284848 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.284673 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63ea9f5d-efb4-4d37-9690-183d90437225-ca-trust-extracted\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.284848 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.284711 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63ea9f5d-efb4-4d37-9690-183d90437225-image-registry-private-configuration\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.284848 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.284750 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm69x\" (UniqueName: \"kubernetes.io/projected/63ea9f5d-efb4-4d37-9690-183d90437225-kube-api-access-jm69x\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.284848 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.284784 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63ea9f5d-efb4-4d37-9690-183d90437225-installation-pull-secrets\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.285454 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.285421 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/63ea9f5d-efb4-4d37-9690-183d90437225-ca-trust-extracted\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.285454 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.285443 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/63ea9f5d-efb4-4d37-9690-183d90437225-registry-certificates\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.285602 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.285546 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63ea9f5d-efb4-4d37-9690-183d90437225-trusted-ca\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.287450 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.287431 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/63ea9f5d-efb4-4d37-9690-183d90437225-image-registry-private-configuration\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.287837 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.287818 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/63ea9f5d-efb4-4d37-9690-183d90437225-installation-pull-secrets\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.287930 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.287911 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/63ea9f5d-efb4-4d37-9690-183d90437225-registry-tls\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.297780 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.297758 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm69x\" (UniqueName: \"kubernetes.io/projected/63ea9f5d-efb4-4d37-9690-183d90437225-kube-api-access-jm69x\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.297918 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.297897 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63ea9f5d-efb4-4d37-9690-183d90437225-bound-sa-token\") pod \"image-registry-74b5cb95d8-x7rkn\" (UID: \"63ea9f5d-efb4-4d37-9690-183d90437225\") " pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.378302 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.378270 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:28.487397 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.487365 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:16:28.487601 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.487435 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:16:28.488045 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.488022 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a04452-ffcd-45fe-b844-519e1c90426b-service-ca-bundle\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:16:28.489636 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.489613 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51a04452-ffcd-45fe-b844-519e1c90426b-metrics-certs\") pod \"router-default-5df75489c9-898zs\" (UID: \"51a04452-ffcd-45fe-b844-519e1c90426b\") " pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:16:28.515230 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.515198 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74b5cb95d8-x7rkn"] Jul 06 19:16:28.517986 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:16:28.517959 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ea9f5d_efb4_4d37_9690_183d90437225.slice/crio-1b0d71aaf685f0d95f41f0dd1e9db20c23be020a3d0018b13a961fdbcb48359b WatchSource:0}: Error finding container 1b0d71aaf685f0d95f41f0dd1e9db20c23be020a3d0018b13a961fdbcb48359b: Status 404 returned error can't find the container with id 1b0d71aaf685f0d95f41f0dd1e9db20c23be020a3d0018b13a961fdbcb48359b Jul 06 19:16:28.764745 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.764702 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:16:28.903669 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:28.899892 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5df75489c9-898zs"] Jul 06 19:16:28.906266 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:16:28.906240 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a04452_ffcd_45fe_b844_519e1c90426b.slice/crio-1cba8c9f237eb3ce511c821d5548f571f467ff0c16e6af92e724edd905a657aa WatchSource:0}: Error finding container 1cba8c9f237eb3ce511c821d5548f571f467ff0c16e6af92e724edd905a657aa: Status 404 returned error can't find the container with id 1cba8c9f237eb3ce511c821d5548f571f467ff0c16e6af92e724edd905a657aa Jul 06 19:16:29.002401 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:29.002367 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5df75489c9-898zs" event={"ID":"51a04452-ffcd-45fe-b844-519e1c90426b","Type":"ContainerStarted","Data":"bf34a0c42e8eddf41a615bdc2b53aa7a6af2975ebcdf226edd8c7e61699a75ed"} Jul 06 19:16:29.002862 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:29.002420 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5df75489c9-898zs" event={"ID":"51a04452-ffcd-45fe-b844-519e1c90426b","Type":"ContainerStarted","Data":"1cba8c9f237eb3ce511c821d5548f571f467ff0c16e6af92e724edd905a657aa"} Jul 06 19:16:29.003806 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:29.003782 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" event={"ID":"63ea9f5d-efb4-4d37-9690-183d90437225","Type":"ContainerStarted","Data":"7b16c73356ad112469a5ba61f8095126bd2516f09d9f51ae2dc3ed5ec31d5068"} Jul 06 19:16:29.003891 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:29.003821 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" event={"ID":"63ea9f5d-efb4-4d37-9690-183d90437225","Type":"ContainerStarted","Data":"1b0d71aaf685f0d95f41f0dd1e9db20c23be020a3d0018b13a961fdbcb48359b"} Jul 06 19:16:29.003891 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:29.003872 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:29.021793 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:29.021740 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5df75489c9-898zs" podStartSLOduration=33.021723985 podStartE2EDuration="33.021723985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:29.020713435 +0000 UTC m=+290.236350966" watchObservedRunningTime="2026-07-06 19:16:29.021723985 +0000 UTC m=+290.237361516" Jul 06 19:16:29.040575 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:29.040538 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" podStartSLOduration=1.040527593 podStartE2EDuration="1.040527593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:29.039721524 +0000 UTC m=+290.255359054" watchObservedRunningTime="2026-07-06 19:16:29.040527593 +0000 UTC m=+290.256165123" Jul 06 19:16:29.765770 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:29.765723 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:16:29.768243 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:29.768220 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:16:30.006971 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:30.006935 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:16:30.008204 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:30.008184 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5df75489c9-898zs" Jul 06 19:16:34.336533 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:34.336486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:34.338766 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:34.338742 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c099894f-48eb-4431-acca-5464590389f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6djr7\" (UID: \"c099894f-48eb-4431-acca-5464590389f8\") " pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:34.568027 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:34.567987 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6djr7" Jul 06 19:16:34.691288 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:34.691159 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6djr7"] Jul 06 19:16:34.693839 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:16:34.693808 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc099894f_48eb_4431_acca_5464590389f8.slice/crio-ca3d2e75089bc74aea8789c8f4fa396e9b5e7afc929fa83dc6a2689271684be5 WatchSource:0}: Error finding container ca3d2e75089bc74aea8789c8f4fa396e9b5e7afc929fa83dc6a2689271684be5: Status 404 returned error can't find the container with id ca3d2e75089bc74aea8789c8f4fa396e9b5e7afc929fa83dc6a2689271684be5 Jul 06 19:16:35.023517 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:35.023472 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6djr7" event={"ID":"c099894f-48eb-4431-acca-5464590389f8","Type":"ContainerStarted","Data":"ca3d2e75089bc74aea8789c8f4fa396e9b5e7afc929fa83dc6a2689271684be5"} Jul 06 19:16:35.143217 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:35.143178 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-sr7tr" podUID="8d718bbb-07dd-481b-81a1-19b7b28c34c1" Jul 06 19:16:37.986531 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:37.986499 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:16:38.006734 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:38.006703 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4rhqw" Jul 06 19:16:39.039893 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:39.039857 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6djr7" event={"ID":"c099894f-48eb-4431-acca-5464590389f8","Type":"ContainerStarted","Data":"3d19212e9977b6d89ac22a934c3ceb30af5e16b420c275aa913a7b3addbcc9b5"} Jul 06 19:16:39.165750 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:39.165728 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:16:39.167288 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:39.167262 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:16:39.181730 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:39.181683 2581 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 19:16:40.045830 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:40.045789 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6djr7" event={"ID":"c099894f-48eb-4431-acca-5464590389f8","Type":"ContainerStarted","Data":"c2056958fecfe830426cc7e398c93372b51ad32e97c8396929c3e54906be3605"} Jul 06 19:16:42.055244 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.055210 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6djr7" event={"ID":"c099894f-48eb-4431-acca-5464590389f8","Type":"ContainerStarted","Data":"3fc7e9fc2026351649ca6691b4ed01c9924c334d1a7b951c483e07a87cfefae4"} Jul 06 19:16:42.086271 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.086167 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6djr7" podStartSLOduration=33.069922075 podStartE2EDuration="40.086145043s" totalImagesPullingTime="7.016222968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:16:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:42.085788248 +0000 UTC m=+303.301425779" watchObservedRunningTime="2026-07-06 19:16:42.086145043 +0000 UTC m=+303.301782574" Jul 06 19:16:42.480208 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.480174 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9n79t"] Jul 06 19:16:42.483420 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.483403 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.486048 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.486026 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 19:16:42.486048 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.486039 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 19:16:42.487085 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.487057 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 19:16:42.487187 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.487098 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l478r\"" Jul 06 19:16:42.487187 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.487116 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 19:16:42.487187 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.487159 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 19:16:42.487187 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.487177 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 19:16:42.498017 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.497998 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f3e8e0b-9d86-462f-b5e5-0849053916a0-node-exporter-tls\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.498116 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.498029 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f3e8e0b-9d86-462f-b5e5-0849053916a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.498116 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.498067 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f3e8e0b-9d86-462f-b5e5-0849053916a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.498191 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.498159 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f3e8e0b-9d86-462f-b5e5-0849053916a0-root\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.498191 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.498184 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f3e8e0b-9d86-462f-b5e5-0849053916a0-node-exporter-textfile\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.498259 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.498201 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f3e8e0b-9d86-462f-b5e5-0849053916a0-sys\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.498259 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.498218 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f3e8e0b-9d86-462f-b5e5-0849053916a0-node-exporter-wtmp\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.498259 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.498239 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f3e8e0b-9d86-462f-b5e5-0849053916a0-metrics-client-ca\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.498345 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.498284 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfqbw\" (UniqueName: \"kubernetes.io/projected/0f3e8e0b-9d86-462f-b5e5-0849053916a0-kube-api-access-zfqbw\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.598965 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.598927 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f3e8e0b-9d86-462f-b5e5-0849053916a0-metrics-client-ca\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.598965 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.598969 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfqbw\" (UniqueName: \"kubernetes.io/projected/0f3e8e0b-9d86-462f-b5e5-0849053916a0-kube-api-access-zfqbw\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.599200 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.598989 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f3e8e0b-9d86-462f-b5e5-0849053916a0-node-exporter-tls\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.599200 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.599011 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f3e8e0b-9d86-462f-b5e5-0849053916a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.599200 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.599060 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f3e8e0b-9d86-462f-b5e5-0849053916a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.599200 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.599124 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f3e8e0b-9d86-462f-b5e5-0849053916a0-root\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.599200 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.599150 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f3e8e0b-9d86-462f-b5e5-0849053916a0-node-exporter-textfile\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.599200 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.599173 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f3e8e0b-9d86-462f-b5e5-0849053916a0-sys\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.599420 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.599215 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f3e8e0b-9d86-462f-b5e5-0849053916a0-node-exporter-wtmp\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.599420 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.599224 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f3e8e0b-9d86-462f-b5e5-0849053916a0-root\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.599713 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.599688 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f3e8e0b-9d86-462f-b5e5-0849053916a0-node-exporter-wtmp\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.600261 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.599946 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f3e8e0b-9d86-462f-b5e5-0849053916a0-node-exporter-textfile\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.600261 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.599282 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f3e8e0b-9d86-462f-b5e5-0849053916a0-sys\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.600261 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.600096 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f3e8e0b-9d86-462f-b5e5-0849053916a0-metrics-client-ca\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.600261 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.600177 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f3e8e0b-9d86-462f-b5e5-0849053916a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.605237 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.605213 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f3e8e0b-9d86-462f-b5e5-0849053916a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.605324 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.605268 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f3e8e0b-9d86-462f-b5e5-0849053916a0-node-exporter-tls\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.612106 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.612086 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfqbw\" (UniqueName: \"kubernetes.io/projected/0f3e8e0b-9d86-462f-b5e5-0849053916a0-kube-api-access-zfqbw\") pod \"node-exporter-9n79t\" (UID: \"0f3e8e0b-9d86-462f-b5e5-0849053916a0\") " pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.792534 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.792438 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9n79t" Jul 06 19:16:42.803863 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:16:42.803833 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3e8e0b_9d86_462f_b5e5_0849053916a0.slice/crio-9b67cb74dce32362aad58987dc95d62898e3507f77bd2da49f1d03f6c56046b0 WatchSource:0}: Error finding container 9b67cb74dce32362aad58987dc95d62898e3507f77bd2da49f1d03f6c56046b0: Status 404 returned error can't find the container with id 9b67cb74dce32362aad58987dc95d62898e3507f77bd2da49f1d03f6c56046b0 Jul 06 19:16:42.805543 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:42.805528 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:16:43.060137 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.060044 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9n79t" event={"ID":"0f3e8e0b-9d86-462f-b5e5-0849053916a0","Type":"ContainerStarted","Data":"9b67cb74dce32362aad58987dc95d62898e3507f77bd2da49f1d03f6c56046b0"} Jul 06 19:16:43.482357 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.482323 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:16:43.486979 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.486961 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.489439 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.489405 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 19:16:43.489570 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.489471 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wdxxq\"" Jul 06 19:16:43.489570 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.489410 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 19:16:43.489570 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.489416 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 19:16:43.489843 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.489828 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 19:16:43.489892 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.489864 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 19:16:43.490111 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.490092 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 19:16:43.490195 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.490092 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 19:16:43.490195 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.490095 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 19:16:43.490309 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.490190 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 19:16:43.500797 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.500775 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:16:43.507644 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.507618 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ce1a6a-18b6-4307-b001-fca1ce319f78-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.507644 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.507668 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54ce1a6a-18b6-4307-b001-fca1ce319f78-tls-assets\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.507860 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.507721 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-config-volume\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.507860 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.507762 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.507860 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.507788 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.507860 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.507804 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-web-config\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.507860 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.507829 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.507860 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.507846 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/54ce1a6a-18b6-4307-b001-fca1ce319f78-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.508162 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.507865 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb4qm\" (UniqueName: \"kubernetes.io/projected/54ce1a6a-18b6-4307-b001-fca1ce319f78-kube-api-access-lb4qm\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.508162 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.507883 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ce1a6a-18b6-4307-b001-fca1ce319f78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.508162 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.507926 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54ce1a6a-18b6-4307-b001-fca1ce319f78-config-out\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.508162 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.507989 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.508162 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.508011 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.608949 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.608914 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ce1a6a-18b6-4307-b001-fca1ce319f78-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.609147 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.608955 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54ce1a6a-18b6-4307-b001-fca1ce319f78-tls-assets\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.609147 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.608981 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-config-volume\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.609147 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.609002 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.609147 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.609026 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.609147 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.609049 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-web-config\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.609147 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.609077 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.609147 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.609100 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/54ce1a6a-18b6-4307-b001-fca1ce319f78-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.609147 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:43.609119 2581 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 06 19:16:43.609604 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:43.609202 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-main-tls podName:54ce1a6a-18b6-4307-b001-fca1ce319f78 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:44.109180842 +0000 UTC m=+305.324818365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "54ce1a6a-18b6-4307-b001-fca1ce319f78") : secret "alertmanager-main-tls" not found Jul 06 19:16:43.609604 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.609125 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb4qm\" (UniqueName: \"kubernetes.io/projected/54ce1a6a-18b6-4307-b001-fca1ce319f78-kube-api-access-lb4qm\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.609604 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.609330 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ce1a6a-18b6-4307-b001-fca1ce319f78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.609604 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.609366 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54ce1a6a-18b6-4307-b001-fca1ce319f78-config-out\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.609604 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.609423 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.609604 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.609455 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.609604 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:43.609477 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54ce1a6a-18b6-4307-b001-fca1ce319f78-alertmanager-trusted-ca-bundle podName:54ce1a6a-18b6-4307-b001-fca1ce319f78 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:44.109460714 +0000 UTC m=+305.325098227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/54ce1a6a-18b6-4307-b001-fca1ce319f78-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "54ce1a6a-18b6-4307-b001-fca1ce319f78") : configmap references non-existent config key: ca-bundle.crt Jul 06 19:16:43.609999 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.609984 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/54ce1a6a-18b6-4307-b001-fca1ce319f78-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.610150 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.610124 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ce1a6a-18b6-4307-b001-fca1ce319f78-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.612120 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.612081 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54ce1a6a-18b6-4307-b001-fca1ce319f78-tls-assets\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.612278 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.612130 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.612341 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.612310 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54ce1a6a-18b6-4307-b001-fca1ce319f78-config-out\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.612616 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.612585 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-config-volume\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.612734 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.612695 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.612734 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.612702 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.612734 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.612720 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.612906 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.612872 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-web-config\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:43.618234 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:43.618213 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb4qm\" (UniqueName: \"kubernetes.io/projected/54ce1a6a-18b6-4307-b001-fca1ce319f78-kube-api-access-lb4qm\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:44.113321 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:44.113286 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ce1a6a-18b6-4307-b001-fca1ce319f78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:44.113713 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:44.113396 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:44.114131 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:44.114108 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ce1a6a-18b6-4307-b001-fca1ce319f78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:44.115629 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:44.115610 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:44.396751 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:44.396728 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:16:44.553842 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:44.553750 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:16:44.555844 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:16:44.555821 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ce1a6a_18b6_4307_b001_fca1ce319f78.slice/crio-68b139f0825bae9bc0d2f6fc76a0c0d20e39f82bf1aaea20f3182f6f851e9681 WatchSource:0}: Error finding container 68b139f0825bae9bc0d2f6fc76a0c0d20e39f82bf1aaea20f3182f6f851e9681: Status 404 returned error can't find the container with id 68b139f0825bae9bc0d2f6fc76a0c0d20e39f82bf1aaea20f3182f6f851e9681 Jul 06 19:16:45.067715 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:45.067679 2581 generic.go:356] "Generic (PLEG): container finished" podID="0f3e8e0b-9d86-462f-b5e5-0849053916a0" containerID="a9adcce065cd6cd2308daeba9b9d755aff101c65d25dc95f08d680ad3f47ec70" exitCode=0 Jul 06 19:16:45.067898 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:45.067767 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9n79t" event={"ID":"0f3e8e0b-9d86-462f-b5e5-0849053916a0","Type":"ContainerDied","Data":"a9adcce065cd6cd2308daeba9b9d755aff101c65d25dc95f08d680ad3f47ec70"} Jul 06 19:16:45.068984 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:45.068963 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54ce1a6a-18b6-4307-b001-fca1ce319f78","Type":"ContainerStarted","Data":"68b139f0825bae9bc0d2f6fc76a0c0d20e39f82bf1aaea20f3182f6f851e9681"} Jul 06 19:16:46.073963 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:46.073930 2581 generic.go:356] "Generic (PLEG): container finished" podID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerID="f01d227c63ec69e155f77332d2df2915ce76eac6c18fdcaa974562e867c1f319" exitCode=0 Jul 06 19:16:46.074381 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:46.074015 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54ce1a6a-18b6-4307-b001-fca1ce319f78","Type":"ContainerDied","Data":"f01d227c63ec69e155f77332d2df2915ce76eac6c18fdcaa974562e867c1f319"} Jul 06 19:16:46.076397 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:46.076371 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9n79t" event={"ID":"0f3e8e0b-9d86-462f-b5e5-0849053916a0","Type":"ContainerStarted","Data":"b1e07a8654b23d2af3bd7c60ecece194551a0a94e393c73785f9a75c3f9ccb17"} Jul 06 19:16:46.076494 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:46.076425 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9n79t" event={"ID":"0f3e8e0b-9d86-462f-b5e5-0849053916a0","Type":"ContainerStarted","Data":"67d583904564a756eef478dcbf5fab2d2a3670ed683e2c332b1eebbc8c92cbe3"} Jul 06 19:16:46.128504 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:46.128456 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9n79t" podStartSLOduration=2.508948917 podStartE2EDuration="4.128441278s" totalImagesPullingTime="1.547509244s" totalInitContainerRuntime="71.983117ms" isStatefulPod=true podCreationTimestamp="2026-07-06 19:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:46.126893485 +0000 UTC m=+307.342531016" watchObservedRunningTime="2026-07-06 19:16:46.128441278 +0000 UTC m=+307.344078807" Jul 06 19:16:48.085483 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.085450 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54ce1a6a-18b6-4307-b001-fca1ce319f78","Type":"ContainerStarted","Data":"664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5"} Jul 06 19:16:48.085881 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.085543 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54ce1a6a-18b6-4307-b001-fca1ce319f78","Type":"ContainerStarted","Data":"6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270"} Jul 06 19:16:48.353372 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.353341 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56bf948d64-5hqw7"] Jul 06 19:16:48.356633 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.356608 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.359197 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.359178 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 19:16:48.359320 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.359258 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 19:16:48.359451 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.359432 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 19:16:48.360626 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.360607 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 19:16:48.360741 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.360613 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 19:16:48.360817 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.360765 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r8p8r\"" Jul 06 19:16:48.360872 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.360834 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 19:16:48.361080 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.361068 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 19:16:48.368160 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.368140 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 19:16:48.369683 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.369664 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56bf948d64-5hqw7"] Jul 06 19:16:48.455843 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.455764 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-trusted-ca-bundle\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.455843 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.455800 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hlrc\" (UniqueName: \"kubernetes.io/projected/4e63ce29-9031-41c5-80d3-aeba80c45a80-kube-api-access-8hlrc\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.455843 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.455835 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-oauth-serving-cert\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.456046 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.455921 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-service-ca\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.456046 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.456014 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-oauth-config\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.456046 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.456044 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-config\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.456141 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.456076 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-serving-cert\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.557318 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.557281 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-oauth-config\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.557510 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.557331 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-config\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.557510 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.557357 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-serving-cert\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.557510 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.557403 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-trusted-ca-bundle\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.557510 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.557427 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hlrc\" (UniqueName: \"kubernetes.io/projected/4e63ce29-9031-41c5-80d3-aeba80c45a80-kube-api-access-8hlrc\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.557510 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.557480 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-oauth-serving-cert\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.557800 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.557516 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-service-ca\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.558108 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.558084 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-config\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.558194 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.558170 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-service-ca\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.558291 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.558270 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-oauth-serving-cert\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.558350 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.558329 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-trusted-ca-bundle\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.559856 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.559833 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-oauth-config\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.559943 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.559926 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-serving-cert\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.565028 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.565010 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hlrc\" (UniqueName: \"kubernetes.io/projected/4e63ce29-9031-41c5-80d3-aeba80c45a80-kube-api-access-8hlrc\") pod \"console-56bf948d64-5hqw7\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.666469 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.666431 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:48.798060 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:48.798028 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56bf948d64-5hqw7"] Jul 06 19:16:48.800785 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:16:48.800762 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e63ce29_9031_41c5_80d3_aeba80c45a80.slice/crio-a63c26b5cde3097325cd5fc2aee5ccd13f74b4fe0d1ce4c2b117735e26171f3a WatchSource:0}: Error finding container a63c26b5cde3097325cd5fc2aee5ccd13f74b4fe0d1ce4c2b117735e26171f3a: Status 404 returned error can't find the container with id a63c26b5cde3097325cd5fc2aee5ccd13f74b4fe0d1ce4c2b117735e26171f3a Jul 06 19:16:49.091395 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:49.091315 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54ce1a6a-18b6-4307-b001-fca1ce319f78","Type":"ContainerStarted","Data":"5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558"} Jul 06 19:16:49.091802 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:49.091399 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54ce1a6a-18b6-4307-b001-fca1ce319f78","Type":"ContainerStarted","Data":"5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551"} Jul 06 19:16:49.091802 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:49.091435 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54ce1a6a-18b6-4307-b001-fca1ce319f78","Type":"ContainerStarted","Data":"d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a"} Jul 06 19:16:49.092476 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:49.092453 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56bf948d64-5hqw7" event={"ID":"4e63ce29-9031-41c5-80d3-aeba80c45a80","Type":"ContainerStarted","Data":"a63c26b5cde3097325cd5fc2aee5ccd13f74b4fe0d1ce4c2b117735e26171f3a"} Jul 06 19:16:49.141796 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:49.141749 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:16:50.011926 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:50.011899 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74b5cb95d8-x7rkn" Jul 06 19:16:50.100384 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:50.100349 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54ce1a6a-18b6-4307-b001-fca1ce319f78","Type":"ContainerStarted","Data":"04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad"} Jul 06 19:16:50.132619 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:50.132560 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.892754837 podStartE2EDuration="7.132543441s" totalImagesPullingTime="4.223817343s" totalInitContainerRuntime="15.971261ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:50.131224035 +0000 UTC m=+311.346861556" watchObservedRunningTime="2026-07-06 19:16:50.132543441 +0000 UTC m=+311.348180971" Jul 06 19:16:52.109196 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:52.109156 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56bf948d64-5hqw7" event={"ID":"4e63ce29-9031-41c5-80d3-aeba80c45a80","Type":"ContainerStarted","Data":"4591dede44507182e23f6fd9fb78927b0afbc394c4a299a06f7322f9aeacc8cb"} Jul 06 19:16:52.130472 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:52.130422 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56bf948d64-5hqw7" podStartSLOduration=1.298845195 podStartE2EDuration="4.130404437s" totalImagesPullingTime="2.831559242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:52.12948563 +0000 UTC m=+313.345123232" watchObservedRunningTime="2026-07-06 19:16:52.130404437 +0000 UTC m=+313.346041967" Jul 06 19:16:53.002671 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.002581 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" podUID="2ae9e185-f263-472c-8ca9-03ed1f4f689c" containerName="registry" containerID="cri-o://265b21d24ebf185a0e4bb60a3d11bc628dde2780cc539f3c2e40b5aecdf02742" gracePeriod=30 Jul 06 19:16:53.241497 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.241474 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:16:53.301859 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.301796 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ae9e185-f263-472c-8ca9-03ed1f4f689c-installation-pull-secrets\") pod \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " Jul 06 19:16:53.301859 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.301853 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ae9e185-f263-472c-8ca9-03ed1f4f689c-image-registry-private-configuration\") pod \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " Jul 06 19:16:53.302017 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.301881 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-certificates\") pod \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " Jul 06 19:16:53.302017 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.301898 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls\") pod \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " Jul 06 19:16:53.302017 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.301931 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-bound-sa-token\") pod \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " Jul 06 19:16:53.302017 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.301953 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7qnw\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-kube-api-access-z7qnw\") pod \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " Jul 06 19:16:53.302017 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.302003 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ae9e185-f263-472c-8ca9-03ed1f4f689c-ca-trust-extracted\") pod \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " Jul 06 19:16:53.302238 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.302050 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ae9e185-f263-472c-8ca9-03ed1f4f689c-trusted-ca\") pod \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\" (UID: \"2ae9e185-f263-472c-8ca9-03ed1f4f689c\") " Jul 06 19:16:53.302295 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.302225 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2ae9e185-f263-472c-8ca9-03ed1f4f689c" (UID: "2ae9e185-f263-472c-8ca9-03ed1f4f689c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:16:53.302408 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.302386 2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-certificates\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:16:53.302643 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.302607 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae9e185-f263-472c-8ca9-03ed1f4f689c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2ae9e185-f263-472c-8ca9-03ed1f4f689c" (UID: "2ae9e185-f263-472c-8ca9-03ed1f4f689c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:16:53.304103 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.304079 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae9e185-f263-472c-8ca9-03ed1f4f689c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2ae9e185-f263-472c-8ca9-03ed1f4f689c" (UID: "2ae9e185-f263-472c-8ca9-03ed1f4f689c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:16:53.304237 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.304214 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae9e185-f263-472c-8ca9-03ed1f4f689c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2ae9e185-f263-472c-8ca9-03ed1f4f689c" (UID: "2ae9e185-f263-472c-8ca9-03ed1f4f689c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:16:53.304463 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.304442 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2ae9e185-f263-472c-8ca9-03ed1f4f689c" (UID: "2ae9e185-f263-472c-8ca9-03ed1f4f689c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:16:53.304555 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.304469 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2ae9e185-f263-472c-8ca9-03ed1f4f689c" (UID: "2ae9e185-f263-472c-8ca9-03ed1f4f689c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:16:53.304713 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.304697 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-kube-api-access-z7qnw" (OuterVolumeSpecName: "kube-api-access-z7qnw") pod "2ae9e185-f263-472c-8ca9-03ed1f4f689c" (UID: "2ae9e185-f263-472c-8ca9-03ed1f4f689c"). InnerVolumeSpecName "kube-api-access-z7qnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:16:53.310792 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.310770 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae9e185-f263-472c-8ca9-03ed1f4f689c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2ae9e185-f263-472c-8ca9-03ed1f4f689c" (UID: "2ae9e185-f263-472c-8ca9-03ed1f4f689c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:16:53.403398 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.403349 2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-bound-sa-token\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:16:53.403398 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.403397 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7qnw\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-kube-api-access-z7qnw\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:16:53.403581 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.403411 2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2ae9e185-f263-472c-8ca9-03ed1f4f689c-ca-trust-extracted\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:16:53.403581 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.403424 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2ae9e185-f263-472c-8ca9-03ed1f4f689c-trusted-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:16:53.403581 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.403435 2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2ae9e185-f263-472c-8ca9-03ed1f4f689c-installation-pull-secrets\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:16:53.403581 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.403447 2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2ae9e185-f263-472c-8ca9-03ed1f4f689c-image-registry-private-configuration\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:16:53.403581 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:53.403459 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2ae9e185-f263-472c-8ca9-03ed1f4f689c-registry-tls\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:16:54.117432 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:54.117396 2581 generic.go:356] "Generic (PLEG): container finished" podID="2ae9e185-f263-472c-8ca9-03ed1f4f689c" containerID="265b21d24ebf185a0e4bb60a3d11bc628dde2780cc539f3c2e40b5aecdf02742" exitCode=0 Jul 06 19:16:54.117631 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:54.117445 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" Jul 06 19:16:54.117631 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:54.117475 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" event={"ID":"2ae9e185-f263-472c-8ca9-03ed1f4f689c","Type":"ContainerDied","Data":"265b21d24ebf185a0e4bb60a3d11bc628dde2780cc539f3c2e40b5aecdf02742"} Jul 06 19:16:54.117631 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:54.117548 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75b5544c69-jf4f5" event={"ID":"2ae9e185-f263-472c-8ca9-03ed1f4f689c","Type":"ContainerDied","Data":"30ee3f88f1fb1b16de81a0b6063958009a60b3c8c773d806169eae02e0ad4823"} Jul 06 19:16:54.117631 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:54.117569 2581 scope.go:117] "RemoveContainer" containerID="265b21d24ebf185a0e4bb60a3d11bc628dde2780cc539f3c2e40b5aecdf02742" Jul 06 19:16:54.125617 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:54.125601 2581 scope.go:117] "RemoveContainer" containerID="265b21d24ebf185a0e4bb60a3d11bc628dde2780cc539f3c2e40b5aecdf02742" Jul 06 19:16:54.125890 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:16:54.125870 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"265b21d24ebf185a0e4bb60a3d11bc628dde2780cc539f3c2e40b5aecdf02742\": container with ID starting with 265b21d24ebf185a0e4bb60a3d11bc628dde2780cc539f3c2e40b5aecdf02742 not found: ID does not exist" containerID="265b21d24ebf185a0e4bb60a3d11bc628dde2780cc539f3c2e40b5aecdf02742" Jul 06 19:16:54.125965 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:54.125902 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"265b21d24ebf185a0e4bb60a3d11bc628dde2780cc539f3c2e40b5aecdf02742"} err="failed to get container status \"265b21d24ebf185a0e4bb60a3d11bc628dde2780cc539f3c2e40b5aecdf02742\": rpc error: code = NotFound desc = could not find container \"265b21d24ebf185a0e4bb60a3d11bc628dde2780cc539f3c2e40b5aecdf02742\": container with ID starting with 265b21d24ebf185a0e4bb60a3d11bc628dde2780cc539f3c2e40b5aecdf02742 not found: ID does not exist" Jul 06 19:16:54.150092 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:54.150069 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75b5544c69-jf4f5"] Jul 06 19:16:54.172202 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:54.172183 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75b5544c69-jf4f5"] Jul 06 19:16:56.146477 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:56.146447 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ae9e185-f263-472c-8ca9-03ed1f4f689c" path="/var/lib/kubelet/pods/2ae9e185-f263-472c-8ca9-03ed1f4f689c/volumes" Jul 06 19:16:58.667504 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:58.667475 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:58.667910 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:58.667535 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:58.672938 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:58.672913 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:16:59.142221 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:16:59.142139 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:17:07.166354 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:17:07.166315 2581 generic.go:356] "Generic (PLEG): container finished" podID="d3ee22af-f46c-445f-97dd-e25e4206aa7c" containerID="3e9161bdc407a5aa416f420dae5988c1d1151551f16a28089d9dd42ac9d66ea7" exitCode=0 Jul 06 19:17:07.166733 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:17:07.166386 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg" event={"ID":"d3ee22af-f46c-445f-97dd-e25e4206aa7c","Type":"ContainerDied","Data":"3e9161bdc407a5aa416f420dae5988c1d1151551f16a28089d9dd42ac9d66ea7"} Jul 06 19:17:07.166733 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:17:07.166643 2581 scope.go:117] "RemoveContainer" containerID="3e9161bdc407a5aa416f420dae5988c1d1151551f16a28089d9dd42ac9d66ea7" Jul 06 19:17:08.171279 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:17:08.171250 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-wsgdg" event={"ID":"d3ee22af-f46c-445f-97dd-e25e4206aa7c","Type":"ContainerStarted","Data":"4dc08def05a39474a44286cc3b494bdec25936ed6dd850fa25fe973a2ce0007e"} Jul 06 19:17:22.216419 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:17:22.216385 2581 generic.go:356] "Generic (PLEG): container finished" podID="a65794b0-70cc-4b6f-bc1c-cd741bfd9de6" containerID="76e2b372c15073c9aa30bc784cd65c4687e0d740c2b3eb0a2821375768ed6efa" exitCode=0 Jul 06 19:17:22.216897 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:17:22.216457 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" event={"ID":"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6","Type":"ContainerDied","Data":"76e2b372c15073c9aa30bc784cd65c4687e0d740c2b3eb0a2821375768ed6efa"} Jul 06 19:17:22.216897 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:17:22.216769 2581 scope.go:117] "RemoveContainer" containerID="76e2b372c15073c9aa30bc784cd65c4687e0d740c2b3eb0a2821375768ed6efa" Jul 06 19:17:23.222230 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:17:23.222193 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-ftqzp" event={"ID":"a65794b0-70cc-4b6f-bc1c-cd741bfd9de6","Type":"ContainerStarted","Data":"379cfcb221b1e02ebdccfbfe943675a0444719ae14b47a5329c94b566265e0ec"} Jul 06 19:17:53.070610 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:17:53.070574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs\") pod \"network-metrics-daemon-sr7tr\" (UID: \"8d718bbb-07dd-481b-81a1-19b7b28c34c1\") " pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:17:53.072973 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:17:53.072950 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d718bbb-07dd-481b-81a1-19b7b28c34c1-metrics-certs\") pod \"network-metrics-daemon-sr7tr\" (UID: \"8d718bbb-07dd-481b-81a1-19b7b28c34c1\") " pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:17:53.344967 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:17:53.344894 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r97mf\"" Jul 06 19:17:53.353146 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:17:53.353127 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sr7tr" Jul 06 19:17:53.477339 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:17:53.477304 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sr7tr"] Jul 06 19:17:53.482719 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:17:53.482686 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d718bbb_07dd_481b_81a1_19b7b28c34c1.slice/crio-6d7d16b0639139986607d0944161e9932aa023a65af22a66d7e0f8b920222f85 WatchSource:0}: Error finding container 6d7d16b0639139986607d0944161e9932aa023a65af22a66d7e0f8b920222f85: Status 404 returned error can't find the container with id 6d7d16b0639139986607d0944161e9932aa023a65af22a66d7e0f8b920222f85 Jul 06 19:17:54.325722 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:17:54.325683 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sr7tr" event={"ID":"8d718bbb-07dd-481b-81a1-19b7b28c34c1","Type":"ContainerStarted","Data":"6d7d16b0639139986607d0944161e9932aa023a65af22a66d7e0f8b920222f85"} Jul 06 19:17:55.331473 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:17:55.331440 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sr7tr" event={"ID":"8d718bbb-07dd-481b-81a1-19b7b28c34c1","Type":"ContainerStarted","Data":"2a1811544a1d8c8119deb6c1796894a653a778bd67101968d4202bcd001160df"} Jul 06 19:17:55.331884 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:17:55.331499 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sr7tr" event={"ID":"8d718bbb-07dd-481b-81a1-19b7b28c34c1","Type":"ContainerStarted","Data":"6a4ef18b5c05654339871139abcb5c96b6abe33086832b947fb9e87675d81aae"} Jul 06 19:17:55.351210 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:17:55.351166 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sr7tr" podStartSLOduration=374.222343281 podStartE2EDuration="6m15.351151219s" totalImagesPullingTime="1.128807938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:17:55.350965838 +0000 UTC m=+376.566603373" watchObservedRunningTime="2026-07-06 19:17:55.351151219 +0000 UTC m=+376.566788748" Jul 06 19:18:02.810039 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:02.810003 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:18:02.810440 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:02.810310 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="alertmanager" containerID="cri-o://6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270" gracePeriod=120 Jul 06 19:18:02.810440 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:02.810368 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="kube-rbac-proxy-metric" containerID="cri-o://5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558" gracePeriod=120 Jul 06 19:18:02.810556 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:02.810412 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="kube-rbac-proxy-web" containerID="cri-o://d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a" gracePeriod=120 Jul 06 19:18:02.810556 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:02.810457 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="prom-label-proxy" containerID="cri-o://04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad" gracePeriod=120 Jul 06 19:18:02.810556 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:02.810473 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="kube-rbac-proxy" containerID="cri-o://5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551" gracePeriod=120 Jul 06 19:18:02.810556 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:02.810426 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="config-reloader" containerID="cri-o://664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5" gracePeriod=120 Jul 06 19:18:03.365045 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:03.365016 2581 generic.go:356] "Generic (PLEG): container finished" podID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerID="04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad" exitCode=0 Jul 06 19:18:03.365045 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:03.365038 2581 generic.go:356] "Generic (PLEG): container finished" podID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerID="5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551" exitCode=0 Jul 06 19:18:03.365045 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:03.365045 2581 generic.go:356] "Generic (PLEG): container finished" podID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerID="664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5" exitCode=0 Jul 06 19:18:03.365045 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:03.365051 2581 generic.go:356] "Generic (PLEG): container finished" podID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerID="6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270" exitCode=0 Jul 06 19:18:03.365306 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:03.365085 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54ce1a6a-18b6-4307-b001-fca1ce319f78","Type":"ContainerDied","Data":"04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad"} Jul 06 19:18:03.365306 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:03.365149 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54ce1a6a-18b6-4307-b001-fca1ce319f78","Type":"ContainerDied","Data":"5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551"} Jul 06 19:18:03.365306 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:03.365176 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54ce1a6a-18b6-4307-b001-fca1ce319f78","Type":"ContainerDied","Data":"664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5"} Jul 06 19:18:03.365306 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:03.365202 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54ce1a6a-18b6-4307-b001-fca1ce319f78","Type":"ContainerDied","Data":"6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270"} Jul 06 19:18:04.049945 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.049923 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.156793 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.156774 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54ce1a6a-18b6-4307-b001-fca1ce319f78-tls-assets\") pod \"54ce1a6a-18b6-4307-b001-fca1ce319f78\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " Jul 06 19:18:04.156922 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.156807 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-main-tls\") pod \"54ce1a6a-18b6-4307-b001-fca1ce319f78\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " Jul 06 19:18:04.156922 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.156851 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy\") pod \"54ce1a6a-18b6-4307-b001-fca1ce319f78\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " Jul 06 19:18:04.156922 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.156871 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/54ce1a6a-18b6-4307-b001-fca1ce319f78-alertmanager-main-db\") pod \"54ce1a6a-18b6-4307-b001-fca1ce319f78\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " Jul 06 19:18:04.156922 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.156908 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-web-config\") pod \"54ce1a6a-18b6-4307-b001-fca1ce319f78\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " Jul 06 19:18:04.157116 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.156927 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-config-volume\") pod \"54ce1a6a-18b6-4307-b001-fca1ce319f78\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " Jul 06 19:18:04.157116 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.156945 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb4qm\" (UniqueName: \"kubernetes.io/projected/54ce1a6a-18b6-4307-b001-fca1ce319f78-kube-api-access-lb4qm\") pod \"54ce1a6a-18b6-4307-b001-fca1ce319f78\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " Jul 06 19:18:04.157116 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.156972 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy-metric\") pod \"54ce1a6a-18b6-4307-b001-fca1ce319f78\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " Jul 06 19:18:04.157116 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.156997 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ce1a6a-18b6-4307-b001-fca1ce319f78-alertmanager-trusted-ca-bundle\") pod \"54ce1a6a-18b6-4307-b001-fca1ce319f78\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " Jul 06 19:18:04.157116 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.157026 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy-web\") pod \"54ce1a6a-18b6-4307-b001-fca1ce319f78\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " Jul 06 19:18:04.157116 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.157074 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ce1a6a-18b6-4307-b001-fca1ce319f78-metrics-client-ca\") pod \"54ce1a6a-18b6-4307-b001-fca1ce319f78\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " Jul 06 19:18:04.157116 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.157107 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54ce1a6a-18b6-4307-b001-fca1ce319f78-config-out\") pod \"54ce1a6a-18b6-4307-b001-fca1ce319f78\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " Jul 06 19:18:04.157465 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.157140 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-cluster-tls-config\") pod \"54ce1a6a-18b6-4307-b001-fca1ce319f78\" (UID: \"54ce1a6a-18b6-4307-b001-fca1ce319f78\") " Jul 06 19:18:04.157465 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.157241 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ce1a6a-18b6-4307-b001-fca1ce319f78-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "54ce1a6a-18b6-4307-b001-fca1ce319f78" (UID: "54ce1a6a-18b6-4307-b001-fca1ce319f78"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:18:04.157465 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.157409 2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/54ce1a6a-18b6-4307-b001-fca1ce319f78-alertmanager-main-db\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:04.157695 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.157558 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ce1a6a-18b6-4307-b001-fca1ce319f78-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "54ce1a6a-18b6-4307-b001-fca1ce319f78" (UID: "54ce1a6a-18b6-4307-b001-fca1ce319f78"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:18:04.158028 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.157999 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ce1a6a-18b6-4307-b001-fca1ce319f78-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "54ce1a6a-18b6-4307-b001-fca1ce319f78" (UID: "54ce1a6a-18b6-4307-b001-fca1ce319f78"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:18:04.160115 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.160072 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "54ce1a6a-18b6-4307-b001-fca1ce319f78" (UID: "54ce1a6a-18b6-4307-b001-fca1ce319f78"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:04.160481 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.160443 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ce1a6a-18b6-4307-b001-fca1ce319f78-kube-api-access-lb4qm" (OuterVolumeSpecName: "kube-api-access-lb4qm") pod "54ce1a6a-18b6-4307-b001-fca1ce319f78" (UID: "54ce1a6a-18b6-4307-b001-fca1ce319f78"). InnerVolumeSpecName "kube-api-access-lb4qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:18:04.160870 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.160833 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ce1a6a-18b6-4307-b001-fca1ce319f78-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "54ce1a6a-18b6-4307-b001-fca1ce319f78" (UID: "54ce1a6a-18b6-4307-b001-fca1ce319f78"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:18:04.160870 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.160844 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "54ce1a6a-18b6-4307-b001-fca1ce319f78" (UID: "54ce1a6a-18b6-4307-b001-fca1ce319f78"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:04.160870 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.160858 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-config-volume" (OuterVolumeSpecName: "config-volume") pod "54ce1a6a-18b6-4307-b001-fca1ce319f78" (UID: "54ce1a6a-18b6-4307-b001-fca1ce319f78"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:04.161081 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.160876 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "54ce1a6a-18b6-4307-b001-fca1ce319f78" (UID: "54ce1a6a-18b6-4307-b001-fca1ce319f78"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:04.161528 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.161496 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "54ce1a6a-18b6-4307-b001-fca1ce319f78" (UID: "54ce1a6a-18b6-4307-b001-fca1ce319f78"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:04.161624 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.161541 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ce1a6a-18b6-4307-b001-fca1ce319f78-config-out" (OuterVolumeSpecName: "config-out") pod "54ce1a6a-18b6-4307-b001-fca1ce319f78" (UID: "54ce1a6a-18b6-4307-b001-fca1ce319f78"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:18:04.164065 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.163926 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "54ce1a6a-18b6-4307-b001-fca1ce319f78" (UID: "54ce1a6a-18b6-4307-b001-fca1ce319f78"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:04.169501 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.169481 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-web-config" (OuterVolumeSpecName: "web-config") pod "54ce1a6a-18b6-4307-b001-fca1ce319f78" (UID: "54ce1a6a-18b6-4307-b001-fca1ce319f78"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:04.258020 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.257981 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54ce1a6a-18b6-4307-b001-fca1ce319f78-metrics-client-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:04.258020 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.258018 2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54ce1a6a-18b6-4307-b001-fca1ce319f78-config-out\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:04.258020 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.258028 2581 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-cluster-tls-config\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:04.258201 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.258037 2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54ce1a6a-18b6-4307-b001-fca1ce319f78-tls-assets\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:04.258201 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.258047 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-main-tls\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:04.258201 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.258056 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:04.258201 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.258086 2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-web-config\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:04.258201 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.258094 2581 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-config-volume\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:04.258201 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.258102 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb4qm\" (UniqueName: \"kubernetes.io/projected/54ce1a6a-18b6-4307-b001-fca1ce319f78-kube-api-access-lb4qm\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:04.258201 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.258111 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:04.258201 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.258120 2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ce1a6a-18b6-4307-b001-fca1ce319f78-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:04.258201 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.258128 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54ce1a6a-18b6-4307-b001-fca1ce319f78-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:04.372912 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.372888 2581 generic.go:356] "Generic (PLEG): container finished" podID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerID="5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558" exitCode=0 Jul 06 19:18:04.372912 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.372909 2581 generic.go:356] "Generic (PLEG): container finished" podID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerID="d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a" exitCode=0 Jul 06 19:18:04.373058 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.372964 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.373058 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.372960 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54ce1a6a-18b6-4307-b001-fca1ce319f78","Type":"ContainerDied","Data":"5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558"} Jul 06 19:18:04.373127 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.373097 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54ce1a6a-18b6-4307-b001-fca1ce319f78","Type":"ContainerDied","Data":"d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a"} Jul 06 19:18:04.373127 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.373124 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54ce1a6a-18b6-4307-b001-fca1ce319f78","Type":"ContainerDied","Data":"68b139f0825bae9bc0d2f6fc76a0c0d20e39f82bf1aaea20f3182f6f851e9681"} Jul 06 19:18:04.373188 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.373152 2581 scope.go:117] "RemoveContainer" containerID="04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad" Jul 06 19:18:04.382479 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.382461 2581 scope.go:117] "RemoveContainer" containerID="5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558" Jul 06 19:18:04.389288 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.389271 2581 scope.go:117] "RemoveContainer" containerID="5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551" Jul 06 19:18:04.395632 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.395613 2581 scope.go:117] "RemoveContainer" containerID="d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a" Jul 06 19:18:04.399801 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.399782 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:18:04.403218 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.403199 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:18:04.403715 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.403703 2581 scope.go:117] "RemoveContainer" containerID="664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5" Jul 06 19:18:04.410162 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.410147 2581 scope.go:117] "RemoveContainer" containerID="6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270" Jul 06 19:18:04.416642 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.416627 2581 scope.go:117] "RemoveContainer" containerID="f01d227c63ec69e155f77332d2df2915ce76eac6c18fdcaa974562e867c1f319" Jul 06 19:18:04.422905 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.422875 2581 scope.go:117] "RemoveContainer" containerID="04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad" Jul 06 19:18:04.423132 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:18:04.423115 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad\": container with ID starting with 04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad not found: ID does not exist" containerID="04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad" Jul 06 19:18:04.423194 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.423138 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad"} err="failed to get container status \"04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad\": rpc error: code = NotFound desc = could not find container \"04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad\": container with ID starting with 04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad not found: ID does not exist" Jul 06 19:18:04.423194 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.423156 2581 scope.go:117] "RemoveContainer" containerID="5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558" Jul 06 19:18:04.423367 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:18:04.423350 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558\": container with ID starting with 5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558 not found: ID does not exist" containerID="5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558" Jul 06 19:18:04.423407 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.423372 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558"} err="failed to get container status \"5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558\": rpc error: code = NotFound desc = could not find container \"5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558\": container with ID starting with 5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558 not found: ID does not exist" Jul 06 19:18:04.423407 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.423386 2581 scope.go:117] "RemoveContainer" containerID="5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551" Jul 06 19:18:04.423599 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:18:04.423583 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551\": container with ID starting with 5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551 not found: ID does not exist" containerID="5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551" Jul 06 19:18:04.423673 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.423605 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551"} err="failed to get container status \"5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551\": rpc error: code = NotFound desc = could not find container \"5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551\": container with ID starting with 5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551 not found: ID does not exist" Jul 06 19:18:04.423673 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.423625 2581 scope.go:117] "RemoveContainer" containerID="d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a" Jul 06 19:18:04.423933 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:18:04.423905 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a\": container with ID starting with d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a not found: ID does not exist" containerID="d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a" Jul 06 19:18:04.423993 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.423935 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a"} err="failed to get container status \"d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a\": rpc error: code = NotFound desc = could not find container \"d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a\": container with ID starting with d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a not found: ID does not exist" Jul 06 19:18:04.423993 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.423952 2581 scope.go:117] "RemoveContainer" containerID="664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5" Jul 06 19:18:04.424286 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:18:04.424272 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5\": container with ID starting with 664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5 not found: ID does not exist" containerID="664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5" Jul 06 19:18:04.424325 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.424293 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5"} err="failed to get container status \"664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5\": rpc error: code = NotFound desc = could not find container \"664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5\": container with ID starting with 664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5 not found: ID does not exist" Jul 06 19:18:04.424325 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.424307 2581 scope.go:117] "RemoveContainer" containerID="6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270" Jul 06 19:18:04.424543 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:18:04.424527 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270\": container with ID starting with 6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270 not found: ID does not exist" containerID="6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270" Jul 06 19:18:04.424582 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.424548 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270"} err="failed to get container status \"6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270\": rpc error: code = NotFound desc = could not find container \"6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270\": container with ID starting with 6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270 not found: ID does not exist" Jul 06 19:18:04.424582 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.424563 2581 scope.go:117] "RemoveContainer" containerID="f01d227c63ec69e155f77332d2df2915ce76eac6c18fdcaa974562e867c1f319" Jul 06 19:18:04.424780 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:18:04.424765 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01d227c63ec69e155f77332d2df2915ce76eac6c18fdcaa974562e867c1f319\": container with ID starting with f01d227c63ec69e155f77332d2df2915ce76eac6c18fdcaa974562e867c1f319 not found: ID does not exist" containerID="f01d227c63ec69e155f77332d2df2915ce76eac6c18fdcaa974562e867c1f319" Jul 06 19:18:04.424819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.424785 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01d227c63ec69e155f77332d2df2915ce76eac6c18fdcaa974562e867c1f319"} err="failed to get container status \"f01d227c63ec69e155f77332d2df2915ce76eac6c18fdcaa974562e867c1f319\": rpc error: code = NotFound desc = could not find container \"f01d227c63ec69e155f77332d2df2915ce76eac6c18fdcaa974562e867c1f319\": container with ID starting with f01d227c63ec69e155f77332d2df2915ce76eac6c18fdcaa974562e867c1f319 not found: ID does not exist" Jul 06 19:18:04.424819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.424799 2581 scope.go:117] "RemoveContainer" containerID="04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad" Jul 06 19:18:04.425002 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.424984 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad"} err="failed to get container status \"04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad\": rpc error: code = NotFound desc = could not find container \"04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad\": container with ID starting with 04b4f0c4bc674166f831809af409375cddabe6cd6696b94123090f3686cb0dad not found: ID does not exist" Jul 06 19:18:04.425106 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.425007 2581 scope.go:117] "RemoveContainer" containerID="5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558" Jul 06 19:18:04.425198 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.425183 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558"} err="failed to get container status \"5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558\": rpc error: code = NotFound desc = could not find container \"5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558\": container with ID starting with 5dcb760520db8aea0ad9b9cc4e21f18d5245eff7c5049e7f78a5094470ef1558 not found: ID does not exist" Jul 06 19:18:04.425236 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.425198 2581 scope.go:117] "RemoveContainer" containerID="5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551" Jul 06 19:18:04.425414 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.425398 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551"} err="failed to get container status \"5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551\": rpc error: code = NotFound desc = could not find container \"5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551\": container with ID starting with 5f9f8d7132e6b93825ab25f1f2b873d9fdfd2707fb9d178d0e236d789b28f551 not found: ID does not exist" Jul 06 19:18:04.425457 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.425416 2581 scope.go:117] "RemoveContainer" containerID="d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a" Jul 06 19:18:04.425615 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.425595 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a"} err="failed to get container status \"d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a\": rpc error: code = NotFound desc = could not find container \"d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a\": container with ID starting with d4e5fa901233f3b6b83b28019c66c20005f7ad9a6786dd093ef0f7e115d9e46a not found: ID does not exist" Jul 06 19:18:04.425680 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.425616 2581 scope.go:117] "RemoveContainer" containerID="664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5" Jul 06 19:18:04.425895 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.425877 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5"} err="failed to get container status \"664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5\": rpc error: code = NotFound desc = could not find container \"664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5\": container with ID starting with 664e261f60b5865be043018deeffb7c3e40b31527268d046cdccfd2b5440b0e5 not found: ID does not exist" Jul 06 19:18:04.425941 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.425896 2581 scope.go:117] "RemoveContainer" containerID="6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270" Jul 06 19:18:04.426133 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.426116 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270"} err="failed to get container status \"6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270\": rpc error: code = NotFound desc = could not find container \"6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270\": container with ID starting with 6425eb72dc10be996b6697eb4c0f6e7c1545678e5f1107850752cc3784204270 not found: ID does not exist" Jul 06 19:18:04.426203 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.426134 2581 scope.go:117] "RemoveContainer" containerID="f01d227c63ec69e155f77332d2df2915ce76eac6c18fdcaa974562e867c1f319" Jul 06 19:18:04.426348 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.426333 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01d227c63ec69e155f77332d2df2915ce76eac6c18fdcaa974562e867c1f319"} err="failed to get container status \"f01d227c63ec69e155f77332d2df2915ce76eac6c18fdcaa974562e867c1f319\": rpc error: code = NotFound desc = could not find container \"f01d227c63ec69e155f77332d2df2915ce76eac6c18fdcaa974562e867c1f319\": container with ID starting with f01d227c63ec69e155f77332d2df2915ce76eac6c18fdcaa974562e867c1f319 not found: ID does not exist" Jul 06 19:18:04.434936 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.434915 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:18:04.435163 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435151 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="prom-label-proxy" Jul 06 19:18:04.435209 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435167 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="prom-label-proxy" Jul 06 19:18:04.435209 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435178 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ae9e185-f263-472c-8ca9-03ed1f4f689c" containerName="registry" Jul 06 19:18:04.435209 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435183 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ae9e185-f263-472c-8ca9-03ed1f4f689c" containerName="registry" Jul 06 19:18:04.435209 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435194 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="kube-rbac-proxy-web" Jul 06 19:18:04.435209 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435200 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="kube-rbac-proxy-web" Jul 06 19:18:04.435209 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435208 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="init-config-reloader" Jul 06 19:18:04.435375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435213 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="init-config-reloader" Jul 06 19:18:04.435375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435222 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="alertmanager" Jul 06 19:18:04.435375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435228 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="alertmanager" Jul 06 19:18:04.435375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435236 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="config-reloader" Jul 06 19:18:04.435375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435241 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="config-reloader" Jul 06 19:18:04.435375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435247 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="kube-rbac-proxy" Jul 06 19:18:04.435375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435252 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="kube-rbac-proxy" Jul 06 19:18:04.435375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435257 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="kube-rbac-proxy-metric" Jul 06 19:18:04.435375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435262 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="kube-rbac-proxy-metric" Jul 06 19:18:04.435375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435299 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="alertmanager" Jul 06 19:18:04.435375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435306 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="config-reloader" Jul 06 19:18:04.435375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435311 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="kube-rbac-proxy-metric" Jul 06 19:18:04.435375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435317 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ae9e185-f263-472c-8ca9-03ed1f4f689c" containerName="registry" Jul 06 19:18:04.435375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435321 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="kube-rbac-proxy-web" Jul 06 19:18:04.435375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435326 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="prom-label-proxy" Jul 06 19:18:04.435375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.435331 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" containerName="kube-rbac-proxy" Jul 06 19:18:04.439837 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.439817 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.442434 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.442415 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wdxxq\"" Jul 06 19:18:04.442526 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.442435 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 19:18:04.442526 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.442444 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 19:18:04.442688 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.442676 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 19:18:04.442919 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.442903 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 19:18:04.443004 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.442909 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 19:18:04.443004 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.442933 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 19:18:04.443004 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.442985 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 19:18:04.443116 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.442935 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 19:18:04.448852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.448837 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 19:18:04.458094 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.458067 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:18:04.559959 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.559931 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.560116 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.559966 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.560116 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.559987 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-config-volume\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.560116 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.560006 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.560116 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.560076 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.560265 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.560123 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.560265 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.560158 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae4b624-b77e-45c1-9ae6-9fb8454777ea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.560265 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.560182 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68vvh\" (UniqueName: \"kubernetes.io/projected/eae4b624-b77e-45c1-9ae6-9fb8454777ea-kube-api-access-68vvh\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.560265 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.560211 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eae4b624-b77e-45c1-9ae6-9fb8454777ea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.560265 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.560248 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eae4b624-b77e-45c1-9ae6-9fb8454777ea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.560423 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.560271 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eae4b624-b77e-45c1-9ae6-9fb8454777ea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.560423 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.560291 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eae4b624-b77e-45c1-9ae6-9fb8454777ea-config-out\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.560423 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.560319 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-web-config\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.661754 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.661665 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae4b624-b77e-45c1-9ae6-9fb8454777ea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.661754 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.661717 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vvh\" (UniqueName: \"kubernetes.io/projected/eae4b624-b77e-45c1-9ae6-9fb8454777ea-kube-api-access-68vvh\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.661754 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.661746 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eae4b624-b77e-45c1-9ae6-9fb8454777ea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.662008 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.661763 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eae4b624-b77e-45c1-9ae6-9fb8454777ea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.662008 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.661787 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eae4b624-b77e-45c1-9ae6-9fb8454777ea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.662008 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.661818 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eae4b624-b77e-45c1-9ae6-9fb8454777ea-config-out\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.662008 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.661846 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-web-config\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.662008 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.661885 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.662008 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.661911 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.662008 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.661940 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-config-volume\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.662008 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.661966 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.662008 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.662006 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.662438 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.662040 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.662438 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.662149 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/eae4b624-b77e-45c1-9ae6-9fb8454777ea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.662615 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.662591 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae4b624-b77e-45c1-9ae6-9fb8454777ea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.663071 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.663048 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eae4b624-b77e-45c1-9ae6-9fb8454777ea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.664590 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.664561 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eae4b624-b77e-45c1-9ae6-9fb8454777ea-config-out\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.665105 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.665075 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-web-config\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.665202 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.665122 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.665202 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.665144 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.665311 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.665208 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eae4b624-b77e-45c1-9ae6-9fb8454777ea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.665447 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.665425 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.665482 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.665463 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-config-volume\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.665588 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.665573 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.666255 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.666241 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/eae4b624-b77e-45c1-9ae6-9fb8454777ea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.669699 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.669684 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68vvh\" (UniqueName: \"kubernetes.io/projected/eae4b624-b77e-45c1-9ae6-9fb8454777ea-kube-api-access-68vvh\") pod \"alertmanager-main-0\" (UID: \"eae4b624-b77e-45c1-9ae6-9fb8454777ea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.749713 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.749678 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:18:04.883303 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:04.883267 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:18:04.890189 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:18:04.890147 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae4b624_b77e_45c1_9ae6_9fb8454777ea.slice/crio-1ae5c92ac0e2921b9b7c9d457b8bba581344a1db0deb9a35afcc0a307b697293 WatchSource:0}: Error finding container 1ae5c92ac0e2921b9b7c9d457b8bba581344a1db0deb9a35afcc0a307b697293: Status 404 returned error can't find the container with id 1ae5c92ac0e2921b9b7c9d457b8bba581344a1db0deb9a35afcc0a307b697293 Jul 06 19:18:05.378600 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:05.378515 2581 generic.go:356] "Generic (PLEG): container finished" podID="eae4b624-b77e-45c1-9ae6-9fb8454777ea" containerID="d27ca13d12f55ed0a9595320067426d507ea8c3ed68e45da1dac4db57ed7ebff" exitCode=0 Jul 06 19:18:05.378600 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:05.378550 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eae4b624-b77e-45c1-9ae6-9fb8454777ea","Type":"ContainerDied","Data":"d27ca13d12f55ed0a9595320067426d507ea8c3ed68e45da1dac4db57ed7ebff"} Jul 06 19:18:05.379010 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:05.378614 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eae4b624-b77e-45c1-9ae6-9fb8454777ea","Type":"ContainerStarted","Data":"1ae5c92ac0e2921b9b7c9d457b8bba581344a1db0deb9a35afcc0a307b697293"} Jul 06 19:18:06.147836 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.147801 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54ce1a6a-18b6-4307-b001-fca1ce319f78" path="/var/lib/kubelet/pods/54ce1a6a-18b6-4307-b001-fca1ce319f78/volumes" Jul 06 19:18:06.385532 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.385498 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eae4b624-b77e-45c1-9ae6-9fb8454777ea","Type":"ContainerStarted","Data":"4ea96dd65e9ee9a48919e1af0fc6cb6bd2a95619a0d058ca8cfd2124295775e6"} Jul 06 19:18:06.385911 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.385574 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eae4b624-b77e-45c1-9ae6-9fb8454777ea","Type":"ContainerStarted","Data":"ecc19c36646bf93fafd837d50e5155f98b1f0d206deb8ed4ca77aa995b62267c"} Jul 06 19:18:06.385911 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.385617 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eae4b624-b77e-45c1-9ae6-9fb8454777ea","Type":"ContainerStarted","Data":"a9923c6d05a311a20bfa50fb85ae4098edffe938e932fce1a382e93545fbfe3f"} Jul 06 19:18:06.385911 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.385676 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eae4b624-b77e-45c1-9ae6-9fb8454777ea","Type":"ContainerStarted","Data":"1fe2055ba07167d870de9db7957e197d85095be152855b7cbff4b4b9813ac0b4"} Jul 06 19:18:06.385911 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.385716 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eae4b624-b77e-45c1-9ae6-9fb8454777ea","Type":"ContainerStarted","Data":"e1f7eb73dabb05eb95a9ec441a9e223cbd5e2672fb5480e5027074c4cde131f4"} Jul 06 19:18:06.385911 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.385751 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"eae4b624-b77e-45c1-9ae6-9fb8454777ea","Type":"ContainerStarted","Data":"3bf1887c746078963250d2c8ab0a041f293ac7954247ca4bf556f351a5bfb4c8"} Jul 06 19:18:06.415558 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.415508 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.399650555 podStartE2EDuration="2.415490279s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.839724ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:18:06.414080276 +0000 UTC m=+387.629717806" watchObservedRunningTime="2026-07-06 19:18:06.415490279 +0000 UTC m=+387.631127815" Jul 06 19:18:06.830198 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.830123 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-b988d8588-5lqd9"] Jul 06 19:18:06.833664 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.833632 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:06.836040 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.836017 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 19:18:06.836040 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.836025 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vpcr6\"" Jul 06 19:18:06.836232 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.836091 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 19:18:06.836316 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.836299 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 19:18:06.836358 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.836315 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 19:18:06.836522 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.836506 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 19:18:06.840995 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.840937 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 19:18:06.846241 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.846217 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b988d8588-5lqd9"] Jul 06 19:18:06.983984 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.983947 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9835372-f08c-4f0b-afcc-5a65a237e012-metrics-client-ca\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:06.983984 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.983987 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4rkg\" (UniqueName: \"kubernetes.io/projected/e9835372-f08c-4f0b-afcc-5a65a237e012-kube-api-access-l4rkg\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:06.984205 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.984016 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9835372-f08c-4f0b-afcc-5a65a237e012-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:06.984205 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.984084 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e9835372-f08c-4f0b-afcc-5a65a237e012-federate-client-tls\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:06.984205 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.984135 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9835372-f08c-4f0b-afcc-5a65a237e012-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:06.984205 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.984182 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9835372-f08c-4f0b-afcc-5a65a237e012-serving-certs-ca-bundle\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:06.984205 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.984203 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e9835372-f08c-4f0b-afcc-5a65a237e012-telemeter-client-tls\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:06.984373 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:06.984297 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e9835372-f08c-4f0b-afcc-5a65a237e012-secret-telemeter-client\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.084864 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.084778 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e9835372-f08c-4f0b-afcc-5a65a237e012-secret-telemeter-client\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.084864 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.084815 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9835372-f08c-4f0b-afcc-5a65a237e012-metrics-client-ca\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.084864 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.084839 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rkg\" (UniqueName: \"kubernetes.io/projected/e9835372-f08c-4f0b-afcc-5a65a237e012-kube-api-access-l4rkg\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.085132 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.084872 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9835372-f08c-4f0b-afcc-5a65a237e012-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.085132 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.084893 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e9835372-f08c-4f0b-afcc-5a65a237e012-federate-client-tls\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.085132 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.084911 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9835372-f08c-4f0b-afcc-5a65a237e012-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.085132 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.084954 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9835372-f08c-4f0b-afcc-5a65a237e012-serving-certs-ca-bundle\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.085132 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.084983 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e9835372-f08c-4f0b-afcc-5a65a237e012-telemeter-client-tls\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.085741 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.085712 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9835372-f08c-4f0b-afcc-5a65a237e012-metrics-client-ca\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.085876 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.085747 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9835372-f08c-4f0b-afcc-5a65a237e012-serving-certs-ca-bundle\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.085986 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.085962 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9835372-f08c-4f0b-afcc-5a65a237e012-telemeter-trusted-ca-bundle\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.087566 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.087546 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e9835372-f08c-4f0b-afcc-5a65a237e012-telemeter-client-tls\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.087690 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.087593 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e9835372-f08c-4f0b-afcc-5a65a237e012-federate-client-tls\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.087821 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.087803 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e9835372-f08c-4f0b-afcc-5a65a237e012-secret-telemeter-client\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.087880 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.087806 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9835372-f08c-4f0b-afcc-5a65a237e012-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.094282 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.094258 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rkg\" (UniqueName: \"kubernetes.io/projected/e9835372-f08c-4f0b-afcc-5a65a237e012-kube-api-access-l4rkg\") pod \"telemeter-client-b988d8588-5lqd9\" (UID: \"e9835372-f08c-4f0b-afcc-5a65a237e012\") " pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.143527 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.143484 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" Jul 06 19:18:07.285283 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.280978 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-b988d8588-5lqd9"] Jul 06 19:18:07.390593 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:07.390506 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" event={"ID":"e9835372-f08c-4f0b-afcc-5a65a237e012","Type":"ContainerStarted","Data":"220c5cecf827a24f65fd0d21c28bea2c1252932cb3f89b09c251fc26ace3eaa1"} Jul 06 19:18:10.403617 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:10.403583 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" event={"ID":"e9835372-f08c-4f0b-afcc-5a65a237e012","Type":"ContainerStarted","Data":"be6e6800c7c35065bcb56a4fc7fc09298c7d617ccb7097746b7553002f6acb56"} Jul 06 19:18:10.404110 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:10.403642 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" event={"ID":"e9835372-f08c-4f0b-afcc-5a65a237e012","Type":"ContainerStarted","Data":"355a689e0d308cb1f185e32d013b114a435e9a734e55bdcfce081f9b88cae3ef"} Jul 06 19:18:10.404110 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:10.403683 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" event={"ID":"e9835372-f08c-4f0b-afcc-5a65a237e012","Type":"ContainerStarted","Data":"63a9bc95be2ea6b49e1951c90db4408be02db2ff3c9f570b98ce6af2f997787b"} Jul 06 19:18:10.429516 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:10.429464 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-b988d8588-5lqd9" podStartSLOduration=1.828265006 podStartE2EDuration="4.429448426s" totalImagesPullingTime="2.60118342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:18:10.427810658 +0000 UTC m=+391.643448187" watchObservedRunningTime="2026-07-06 19:18:10.429448426 +0000 UTC m=+391.645085956" Jul 06 19:18:11.123981 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.123951 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65cdccb47-vhdxt"] Jul 06 19:18:11.127521 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.127495 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.139436 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.139415 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65cdccb47-vhdxt"] Jul 06 19:18:11.222400 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.222363 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/378cc72f-7a43-4126-9b9b-24d38109b247-console-serving-cert\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.222400 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.222406 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-service-ca\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.222598 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.222429 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-console-config\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.222598 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.222511 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-trusted-ca-bundle\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.222598 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.222557 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmnh4\" (UniqueName: \"kubernetes.io/projected/378cc72f-7a43-4126-9b9b-24d38109b247-kube-api-access-mmnh4\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.222723 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.222616 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/378cc72f-7a43-4126-9b9b-24d38109b247-console-oauth-config\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.222723 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.222636 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-oauth-serving-cert\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.323367 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.323331 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/378cc72f-7a43-4126-9b9b-24d38109b247-console-oauth-config\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.323367 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.323369 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-oauth-serving-cert\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.323588 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.323399 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/378cc72f-7a43-4126-9b9b-24d38109b247-console-serving-cert\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.323588 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.323433 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-service-ca\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.323588 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.323462 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-console-config\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.323588 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.323493 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-trusted-ca-bundle\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.323588 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.323536 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmnh4\" (UniqueName: \"kubernetes.io/projected/378cc72f-7a43-4126-9b9b-24d38109b247-kube-api-access-mmnh4\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.324245 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.324219 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-oauth-serving-cert\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.324369 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.324343 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-console-config\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.324415 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.324370 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-service-ca\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.324480 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.324464 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-trusted-ca-bundle\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.325870 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.325841 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/378cc72f-7a43-4126-9b9b-24d38109b247-console-oauth-config\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.325965 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.325895 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/378cc72f-7a43-4126-9b9b-24d38109b247-console-serving-cert\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.332338 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.332319 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmnh4\" (UniqueName: \"kubernetes.io/projected/378cc72f-7a43-4126-9b9b-24d38109b247-kube-api-access-mmnh4\") pod \"console-65cdccb47-vhdxt\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.438045 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.438012 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:11.572415 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:11.572390 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65cdccb47-vhdxt"] Jul 06 19:18:11.575096 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:18:11.575068 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378cc72f_7a43_4126_9b9b_24d38109b247.slice/crio-857354732c9c3326ad3a1cca1be73e727ea4cacbdd15f2d33240a15bc4c97416 WatchSource:0}: Error finding container 857354732c9c3326ad3a1cca1be73e727ea4cacbdd15f2d33240a15bc4c97416: Status 404 returned error can't find the container with id 857354732c9c3326ad3a1cca1be73e727ea4cacbdd15f2d33240a15bc4c97416 Jul 06 19:18:12.412638 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:12.412602 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cdccb47-vhdxt" event={"ID":"378cc72f-7a43-4126-9b9b-24d38109b247","Type":"ContainerStarted","Data":"55912172d2dce4f4889db9a7fbc1171ef3ca7d7d39535ec57fc1577ca6916719"} Jul 06 19:18:12.412864 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:12.412672 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cdccb47-vhdxt" event={"ID":"378cc72f-7a43-4126-9b9b-24d38109b247","Type":"ContainerStarted","Data":"857354732c9c3326ad3a1cca1be73e727ea4cacbdd15f2d33240a15bc4c97416"} Jul 06 19:18:12.430853 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:12.430795 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65cdccb47-vhdxt" podStartSLOduration=1.430779374 podStartE2EDuration="1.430779374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:18:12.429817554 +0000 UTC m=+393.645455085" watchObservedRunningTime="2026-07-06 19:18:12.430779374 +0000 UTC m=+393.646416903" Jul 06 19:18:21.438733 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:21.438692 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:21.439136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:21.438750 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:21.443383 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:21.443359 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:21.451383 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:21.451363 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:18:21.506648 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:21.506610 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56bf948d64-5hqw7"] Jul 06 19:18:46.533356 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:46.533307 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-56bf948d64-5hqw7" podUID="4e63ce29-9031-41c5-80d3-aeba80c45a80" containerName="console" containerID="cri-o://4591dede44507182e23f6fd9fb78927b0afbc394c4a299a06f7322f9aeacc8cb" gracePeriod=15 Jul 06 19:18:46.775229 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:46.775203 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:18:46.921596 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:46.921566 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-oauth-config\") pod \"4e63ce29-9031-41c5-80d3-aeba80c45a80\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " Jul 06 19:18:46.921805 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:46.921613 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-serving-cert\") pod \"4e63ce29-9031-41c5-80d3-aeba80c45a80\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " Jul 06 19:18:46.921805 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:46.921636 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-trusted-ca-bundle\") pod \"4e63ce29-9031-41c5-80d3-aeba80c45a80\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " Jul 06 19:18:46.921805 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:46.921714 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hlrc\" (UniqueName: \"kubernetes.io/projected/4e63ce29-9031-41c5-80d3-aeba80c45a80-kube-api-access-8hlrc\") pod \"4e63ce29-9031-41c5-80d3-aeba80c45a80\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " Jul 06 19:18:46.921805 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:46.921734 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-oauth-serving-cert\") pod \"4e63ce29-9031-41c5-80d3-aeba80c45a80\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " Jul 06 19:18:46.921805 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:46.921751 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-service-ca\") pod \"4e63ce29-9031-41c5-80d3-aeba80c45a80\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " Jul 06 19:18:46.922054 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:46.921865 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-config\") pod \"4e63ce29-9031-41c5-80d3-aeba80c45a80\" (UID: \"4e63ce29-9031-41c5-80d3-aeba80c45a80\") " Jul 06 19:18:46.922180 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:46.922153 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-service-ca" (OuterVolumeSpecName: "service-ca") pod "4e63ce29-9031-41c5-80d3-aeba80c45a80" (UID: "4e63ce29-9031-41c5-80d3-aeba80c45a80"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:18:46.922243 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:46.922196 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4e63ce29-9031-41c5-80d3-aeba80c45a80" (UID: "4e63ce29-9031-41c5-80d3-aeba80c45a80"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:18:46.922359 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:46.922338 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4e63ce29-9031-41c5-80d3-aeba80c45a80" (UID: "4e63ce29-9031-41c5-80d3-aeba80c45a80"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:18:46.922419 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:46.922389 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-config" (OuterVolumeSpecName: "console-config") pod "4e63ce29-9031-41c5-80d3-aeba80c45a80" (UID: "4e63ce29-9031-41c5-80d3-aeba80c45a80"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:18:46.924105 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:46.924083 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e63ce29-9031-41c5-80d3-aeba80c45a80-kube-api-access-8hlrc" (OuterVolumeSpecName: "kube-api-access-8hlrc") pod "4e63ce29-9031-41c5-80d3-aeba80c45a80" (UID: "4e63ce29-9031-41c5-80d3-aeba80c45a80"). InnerVolumeSpecName "kube-api-access-8hlrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:18:46.924105 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:46.924084 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4e63ce29-9031-41c5-80d3-aeba80c45a80" (UID: "4e63ce29-9031-41c5-80d3-aeba80c45a80"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:46.924105 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:46.924101 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4e63ce29-9031-41c5-80d3-aeba80c45a80" (UID: "4e63ce29-9031-41c5-80d3-aeba80c45a80"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:47.023084 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:47.023050 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-oauth-config\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:47.023084 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:47.023079 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-serving-cert\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:47.023084 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:47.023088 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-trusted-ca-bundle\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:47.023303 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:47.023097 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hlrc\" (UniqueName: \"kubernetes.io/projected/4e63ce29-9031-41c5-80d3-aeba80c45a80-kube-api-access-8hlrc\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:47.023303 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:47.023106 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-oauth-serving-cert\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:47.023303 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:47.023115 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-service-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:47.023303 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:47.023123 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e63ce29-9031-41c5-80d3-aeba80c45a80-console-config\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:18:47.545963 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:47.545927 2581 generic.go:356] "Generic (PLEG): container finished" podID="4e63ce29-9031-41c5-80d3-aeba80c45a80" containerID="4591dede44507182e23f6fd9fb78927b0afbc394c4a299a06f7322f9aeacc8cb" exitCode=0 Jul 06 19:18:47.546378 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:47.545978 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56bf948d64-5hqw7" event={"ID":"4e63ce29-9031-41c5-80d3-aeba80c45a80","Type":"ContainerDied","Data":"4591dede44507182e23f6fd9fb78927b0afbc394c4a299a06f7322f9aeacc8cb"} Jul 06 19:18:47.546378 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:47.545985 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56bf948d64-5hqw7" Jul 06 19:18:47.546378 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:47.546043 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56bf948d64-5hqw7" event={"ID":"4e63ce29-9031-41c5-80d3-aeba80c45a80","Type":"ContainerDied","Data":"a63c26b5cde3097325cd5fc2aee5ccd13f74b4fe0d1ce4c2b117735e26171f3a"} Jul 06 19:18:47.546378 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:47.546063 2581 scope.go:117] "RemoveContainer" containerID="4591dede44507182e23f6fd9fb78927b0afbc394c4a299a06f7322f9aeacc8cb" Jul 06 19:18:47.554900 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:47.554883 2581 scope.go:117] "RemoveContainer" containerID="4591dede44507182e23f6fd9fb78927b0afbc394c4a299a06f7322f9aeacc8cb" Jul 06 19:18:47.555138 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:18:47.555123 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4591dede44507182e23f6fd9fb78927b0afbc394c4a299a06f7322f9aeacc8cb\": container with ID starting with 4591dede44507182e23f6fd9fb78927b0afbc394c4a299a06f7322f9aeacc8cb not found: ID does not exist" containerID="4591dede44507182e23f6fd9fb78927b0afbc394c4a299a06f7322f9aeacc8cb" Jul 06 19:18:47.555186 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:47.555146 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4591dede44507182e23f6fd9fb78927b0afbc394c4a299a06f7322f9aeacc8cb"} err="failed to get container status \"4591dede44507182e23f6fd9fb78927b0afbc394c4a299a06f7322f9aeacc8cb\": rpc error: code = NotFound desc = could not find container \"4591dede44507182e23f6fd9fb78927b0afbc394c4a299a06f7322f9aeacc8cb\": container with ID starting with 4591dede44507182e23f6fd9fb78927b0afbc394c4a299a06f7322f9aeacc8cb not found: ID does not exist" Jul 06 19:18:47.566565 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:47.566535 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56bf948d64-5hqw7"] Jul 06 19:18:47.572229 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:47.572205 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56bf948d64-5hqw7"] Jul 06 19:18:48.146356 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:18:48.146324 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e63ce29-9031-41c5-80d3-aeba80c45a80" path="/var/lib/kubelet/pods/4e63ce29-9031-41c5-80d3-aeba80c45a80/volumes" Jul 06 19:19:19.265744 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.265708 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57b56cbf86-prbfx"] Jul 06 19:19:19.266769 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.266743 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e63ce29-9031-41c5-80d3-aeba80c45a80" containerName="console" Jul 06 19:19:19.266941 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.266929 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e63ce29-9031-41c5-80d3-aeba80c45a80" containerName="console" Jul 06 19:19:19.267136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.267125 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e63ce29-9031-41c5-80d3-aeba80c45a80" containerName="console" Jul 06 19:19:19.271563 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.271541 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.280762 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.280728 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e43b482d-c881-44e9-a963-96f447c64e7f-console-serving-cert\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.280902 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.280841 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-trusted-ca-bundle\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.280964 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.280914 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-oauth-serving-cert\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.281012 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.280976 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e43b482d-c881-44e9-a963-96f447c64e7f-console-oauth-config\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.281056 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.281038 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-service-ca\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.281109 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.281083 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz82d\" (UniqueName: \"kubernetes.io/projected/e43b482d-c881-44e9-a963-96f447c64e7f-kube-api-access-gz82d\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.281158 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.281115 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-console-config\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.281892 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.281866 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57b56cbf86-prbfx"] Jul 06 19:19:19.382037 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.382002 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e43b482d-c881-44e9-a963-96f447c64e7f-console-serving-cert\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.382227 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.382057 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-trusted-ca-bundle\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.382227 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.382093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-oauth-serving-cert\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.382227 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.382138 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e43b482d-c881-44e9-a963-96f447c64e7f-console-oauth-config\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.382227 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.382161 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-service-ca\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.382434 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.382298 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz82d\" (UniqueName: \"kubernetes.io/projected/e43b482d-c881-44e9-a963-96f447c64e7f-kube-api-access-gz82d\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.382434 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.382348 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-console-config\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.382964 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.382940 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-oauth-serving-cert\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.383084 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.383054 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-service-ca\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.383084 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.383068 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-console-config\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.383349 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.383327 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-trusted-ca-bundle\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.384617 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.384594 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e43b482d-c881-44e9-a963-96f447c64e7f-console-serving-cert\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.384756 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.384734 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e43b482d-c881-44e9-a963-96f447c64e7f-console-oauth-config\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.391513 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.391488 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz82d\" (UniqueName: \"kubernetes.io/projected/e43b482d-c881-44e9-a963-96f447c64e7f-kube-api-access-gz82d\") pod \"console-57b56cbf86-prbfx\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.582165 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.582069 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:19.725179 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:19.725141 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57b56cbf86-prbfx"] Jul 06 19:19:19.728132 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:19:19.728103 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43b482d_c881_44e9_a963_96f447c64e7f.slice/crio-9e4f65310a01c4738deb9732bb0edb3acd744bbead8a85a4d38bd65a6ec0a2e4 WatchSource:0}: Error finding container 9e4f65310a01c4738deb9732bb0edb3acd744bbead8a85a4d38bd65a6ec0a2e4: Status 404 returned error can't find the container with id 9e4f65310a01c4738deb9732bb0edb3acd744bbead8a85a4d38bd65a6ec0a2e4 Jul 06 19:19:20.670117 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:20.670080 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57b56cbf86-prbfx" event={"ID":"e43b482d-c881-44e9-a963-96f447c64e7f","Type":"ContainerStarted","Data":"7342735fe5346e73135993fb9e6c5bf02bfeee0e71b0710e612a5a55bb3ef61a"} Jul 06 19:19:20.670513 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:20.670158 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57b56cbf86-prbfx" event={"ID":"e43b482d-c881-44e9-a963-96f447c64e7f","Type":"ContainerStarted","Data":"9e4f65310a01c4738deb9732bb0edb3acd744bbead8a85a4d38bd65a6ec0a2e4"} Jul 06 19:19:20.690906 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:20.690857 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57b56cbf86-prbfx" podStartSLOduration=1.6908417249999999 podStartE2EDuration="1.690841725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:19:20.689886934 +0000 UTC m=+461.905524466" watchObservedRunningTime="2026-07-06 19:19:20.690841725 +0000 UTC m=+461.906479234" Jul 06 19:19:29.583232 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:29.583187 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:29.583696 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:29.583279 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:29.588124 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:29.588098 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:29.711542 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:29.711514 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:19:29.773332 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:29.773293 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65cdccb47-vhdxt"] Jul 06 19:19:54.796828 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:54.796773 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-65cdccb47-vhdxt" podUID="378cc72f-7a43-4126-9b9b-24d38109b247" containerName="console" containerID="cri-o://55912172d2dce4f4889db9a7fbc1171ef3ca7d7d39535ec57fc1577ca6916719" gracePeriod=15 Jul 06 19:19:55.062044 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.062017 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:19:55.232460 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.232401 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-service-ca\") pod \"378cc72f-7a43-4126-9b9b-24d38109b247\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " Jul 06 19:19:55.232460 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.232470 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/378cc72f-7a43-4126-9b9b-24d38109b247-console-serving-cert\") pod \"378cc72f-7a43-4126-9b9b-24d38109b247\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " Jul 06 19:19:55.232808 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.232495 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-trusted-ca-bundle\") pod \"378cc72f-7a43-4126-9b9b-24d38109b247\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " Jul 06 19:19:55.232808 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.232538 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/378cc72f-7a43-4126-9b9b-24d38109b247-console-oauth-config\") pod \"378cc72f-7a43-4126-9b9b-24d38109b247\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " Jul 06 19:19:55.232808 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.232562 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-oauth-serving-cert\") pod \"378cc72f-7a43-4126-9b9b-24d38109b247\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " Jul 06 19:19:55.232808 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.232595 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-console-config\") pod \"378cc72f-7a43-4126-9b9b-24d38109b247\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " Jul 06 19:19:55.232808 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.232644 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmnh4\" (UniqueName: \"kubernetes.io/projected/378cc72f-7a43-4126-9b9b-24d38109b247-kube-api-access-mmnh4\") pod \"378cc72f-7a43-4126-9b9b-24d38109b247\" (UID: \"378cc72f-7a43-4126-9b9b-24d38109b247\") " Jul 06 19:19:55.233193 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.233114 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-service-ca" (OuterVolumeSpecName: "service-ca") pod "378cc72f-7a43-4126-9b9b-24d38109b247" (UID: "378cc72f-7a43-4126-9b9b-24d38109b247"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:19:55.233193 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.233128 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "378cc72f-7a43-4126-9b9b-24d38109b247" (UID: "378cc72f-7a43-4126-9b9b-24d38109b247"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:19:55.233193 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.233141 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-console-config" (OuterVolumeSpecName: "console-config") pod "378cc72f-7a43-4126-9b9b-24d38109b247" (UID: "378cc72f-7a43-4126-9b9b-24d38109b247"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:19:55.233193 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.233146 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "378cc72f-7a43-4126-9b9b-24d38109b247" (UID: "378cc72f-7a43-4126-9b9b-24d38109b247"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:19:55.235276 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.235244 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378cc72f-7a43-4126-9b9b-24d38109b247-kube-api-access-mmnh4" (OuterVolumeSpecName: "kube-api-access-mmnh4") pod "378cc72f-7a43-4126-9b9b-24d38109b247" (UID: "378cc72f-7a43-4126-9b9b-24d38109b247"). InnerVolumeSpecName "kube-api-access-mmnh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:19:55.235385 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.235268 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/378cc72f-7a43-4126-9b9b-24d38109b247-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "378cc72f-7a43-4126-9b9b-24d38109b247" (UID: "378cc72f-7a43-4126-9b9b-24d38109b247"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:19:55.235385 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.235311 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/378cc72f-7a43-4126-9b9b-24d38109b247-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "378cc72f-7a43-4126-9b9b-24d38109b247" (UID: "378cc72f-7a43-4126-9b9b-24d38109b247"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:19:55.334047 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.333885 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-service-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:19:55.334047 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.333940 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/378cc72f-7a43-4126-9b9b-24d38109b247-console-serving-cert\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:19:55.334047 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.333951 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-trusted-ca-bundle\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:19:55.334047 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.333961 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/378cc72f-7a43-4126-9b9b-24d38109b247-console-oauth-config\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:19:55.334047 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.333972 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-oauth-serving-cert\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:19:55.334047 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.333984 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/378cc72f-7a43-4126-9b9b-24d38109b247-console-config\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:19:55.334047 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.333999 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmnh4\" (UniqueName: \"kubernetes.io/projected/378cc72f-7a43-4126-9b9b-24d38109b247-kube-api-access-mmnh4\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:19:55.811905 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.811865 2581 generic.go:356] "Generic (PLEG): container finished" podID="378cc72f-7a43-4126-9b9b-24d38109b247" containerID="55912172d2dce4f4889db9a7fbc1171ef3ca7d7d39535ec57fc1577ca6916719" exitCode=0 Jul 06 19:19:55.812439 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.811914 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cdccb47-vhdxt" Jul 06 19:19:55.812439 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.811953 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cdccb47-vhdxt" event={"ID":"378cc72f-7a43-4126-9b9b-24d38109b247","Type":"ContainerDied","Data":"55912172d2dce4f4889db9a7fbc1171ef3ca7d7d39535ec57fc1577ca6916719"} Jul 06 19:19:55.812439 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.812023 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cdccb47-vhdxt" event={"ID":"378cc72f-7a43-4126-9b9b-24d38109b247","Type":"ContainerDied","Data":"857354732c9c3326ad3a1cca1be73e727ea4cacbdd15f2d33240a15bc4c97416"} Jul 06 19:19:55.812439 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.812055 2581 scope.go:117] "RemoveContainer" containerID="55912172d2dce4f4889db9a7fbc1171ef3ca7d7d39535ec57fc1577ca6916719" Jul 06 19:19:55.822604 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.822584 2581 scope.go:117] "RemoveContainer" containerID="55912172d2dce4f4889db9a7fbc1171ef3ca7d7d39535ec57fc1577ca6916719" Jul 06 19:19:55.823004 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:19:55.822980 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55912172d2dce4f4889db9a7fbc1171ef3ca7d7d39535ec57fc1577ca6916719\": container with ID starting with 55912172d2dce4f4889db9a7fbc1171ef3ca7d7d39535ec57fc1577ca6916719 not found: ID does not exist" containerID="55912172d2dce4f4889db9a7fbc1171ef3ca7d7d39535ec57fc1577ca6916719" Jul 06 19:19:55.823069 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.823016 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55912172d2dce4f4889db9a7fbc1171ef3ca7d7d39535ec57fc1577ca6916719"} err="failed to get container status \"55912172d2dce4f4889db9a7fbc1171ef3ca7d7d39535ec57fc1577ca6916719\": rpc error: code = NotFound desc = could not find container \"55912172d2dce4f4889db9a7fbc1171ef3ca7d7d39535ec57fc1577ca6916719\": container with ID starting with 55912172d2dce4f4889db9a7fbc1171ef3ca7d7d39535ec57fc1577ca6916719 not found: ID does not exist" Jul 06 19:19:55.833206 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.833162 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65cdccb47-vhdxt"] Jul 06 19:19:55.835245 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:55.835218 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65cdccb47-vhdxt"] Jul 06 19:19:56.146946 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:19:56.146836 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="378cc72f-7a43-4126-9b9b-24d38109b247" path="/var/lib/kubelet/pods/378cc72f-7a43-4126-9b9b-24d38109b247/volumes" Jul 06 19:20:31.578971 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.578888 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n"] Jul 06 19:20:31.579418 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.579163 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="378cc72f-7a43-4126-9b9b-24d38109b247" containerName="console" Jul 06 19:20:31.579418 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.579173 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="378cc72f-7a43-4126-9b9b-24d38109b247" containerName="console" Jul 06 19:20:31.579418 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.579217 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="378cc72f-7a43-4126-9b9b-24d38109b247" containerName="console" Jul 06 19:20:31.582055 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.582038 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" Jul 06 19:20:31.584599 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.584561 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4wwgt\"" Jul 06 19:20:31.584748 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.584601 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 19:20:31.585639 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.585624 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 19:20:31.594717 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.594697 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n"] Jul 06 19:20:31.714962 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.714938 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4691fbca-628f-4310-8fd4-fde5275103d1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n\" (UID: \"4691fbca-628f-4310-8fd4-fde5275103d1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" Jul 06 19:20:31.715078 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.714969 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhd5x\" (UniqueName: \"kubernetes.io/projected/4691fbca-628f-4310-8fd4-fde5275103d1-kube-api-access-bhd5x\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n\" (UID: \"4691fbca-628f-4310-8fd4-fde5275103d1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" Jul 06 19:20:31.715078 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.714994 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4691fbca-628f-4310-8fd4-fde5275103d1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n\" (UID: \"4691fbca-628f-4310-8fd4-fde5275103d1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" Jul 06 19:20:31.815692 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.815665 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4691fbca-628f-4310-8fd4-fde5275103d1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n\" (UID: \"4691fbca-628f-4310-8fd4-fde5275103d1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" Jul 06 19:20:31.815817 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.815728 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4691fbca-628f-4310-8fd4-fde5275103d1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n\" (UID: \"4691fbca-628f-4310-8fd4-fde5275103d1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" Jul 06 19:20:31.815817 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.815752 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhd5x\" (UniqueName: \"kubernetes.io/projected/4691fbca-628f-4310-8fd4-fde5275103d1-kube-api-access-bhd5x\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n\" (UID: \"4691fbca-628f-4310-8fd4-fde5275103d1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" Jul 06 19:20:31.815951 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.815934 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4691fbca-628f-4310-8fd4-fde5275103d1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n\" (UID: \"4691fbca-628f-4310-8fd4-fde5275103d1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" Jul 06 19:20:31.816052 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.816036 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4691fbca-628f-4310-8fd4-fde5275103d1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n\" (UID: \"4691fbca-628f-4310-8fd4-fde5275103d1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" Jul 06 19:20:31.825228 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.825203 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhd5x\" (UniqueName: \"kubernetes.io/projected/4691fbca-628f-4310-8fd4-fde5275103d1-kube-api-access-bhd5x\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n\" (UID: \"4691fbca-628f-4310-8fd4-fde5275103d1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" Jul 06 19:20:31.891186 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:31.891135 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" Jul 06 19:20:32.018995 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:32.018832 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n"] Jul 06 19:20:32.021100 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:20:32.021065 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4691fbca_628f_4310_8fd4_fde5275103d1.slice/crio-6df763d66de80af28106572583e9bf98ea74a75e7d96b2b85b334e9d4530aa27 WatchSource:0}: Error finding container 6df763d66de80af28106572583e9bf98ea74a75e7d96b2b85b334e9d4530aa27: Status 404 returned error can't find the container with id 6df763d66de80af28106572583e9bf98ea74a75e7d96b2b85b334e9d4530aa27 Jul 06 19:20:32.954193 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:32.954156 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" event={"ID":"4691fbca-628f-4310-8fd4-fde5275103d1","Type":"ContainerStarted","Data":"6df763d66de80af28106572583e9bf98ea74a75e7d96b2b85b334e9d4530aa27"} Jul 06 19:20:41.991208 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:41.991175 2581 generic.go:356] "Generic (PLEG): container finished" podID="4691fbca-628f-4310-8fd4-fde5275103d1" containerID="1f2802a4bfbe87ccdf82bca7a11e470102314df5abf1232d7ac5f1907e898206" exitCode=0 Jul 06 19:20:41.991630 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:41.991265 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" event={"ID":"4691fbca-628f-4310-8fd4-fde5275103d1","Type":"ContainerDied","Data":"1f2802a4bfbe87ccdf82bca7a11e470102314df5abf1232d7ac5f1907e898206"} Jul 06 19:20:45.005228 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:45.005197 2581 generic.go:356] "Generic (PLEG): container finished" podID="4691fbca-628f-4310-8fd4-fde5275103d1" containerID="5f63b0c2817ff32dda9386749fbf29eb68a291b5c0fa3c10ebc00d5ffa813454" exitCode=0 Jul 06 19:20:45.005643 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:45.005276 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" event={"ID":"4691fbca-628f-4310-8fd4-fde5275103d1","Type":"ContainerDied","Data":"5f63b0c2817ff32dda9386749fbf29eb68a291b5c0fa3c10ebc00d5ffa813454"} Jul 06 19:20:56.052698 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:56.052641 2581 generic.go:356] "Generic (PLEG): container finished" podID="4691fbca-628f-4310-8fd4-fde5275103d1" containerID="d9867031f7b393e2d33f402955b71f0b597d43e99c6653b567a9ad21c8888f92" exitCode=0 Jul 06 19:20:56.053103 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:56.052702 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" event={"ID":"4691fbca-628f-4310-8fd4-fde5275103d1","Type":"ContainerDied","Data":"d9867031f7b393e2d33f402955b71f0b597d43e99c6653b567a9ad21c8888f92"} Jul 06 19:20:57.185445 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:57.185420 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" Jul 06 19:20:57.241361 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:57.241330 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhd5x\" (UniqueName: \"kubernetes.io/projected/4691fbca-628f-4310-8fd4-fde5275103d1-kube-api-access-bhd5x\") pod \"4691fbca-628f-4310-8fd4-fde5275103d1\" (UID: \"4691fbca-628f-4310-8fd4-fde5275103d1\") " Jul 06 19:20:57.241361 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:57.241365 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4691fbca-628f-4310-8fd4-fde5275103d1-bundle\") pod \"4691fbca-628f-4310-8fd4-fde5275103d1\" (UID: \"4691fbca-628f-4310-8fd4-fde5275103d1\") " Jul 06 19:20:57.241557 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:57.241385 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4691fbca-628f-4310-8fd4-fde5275103d1-util\") pod \"4691fbca-628f-4310-8fd4-fde5275103d1\" (UID: \"4691fbca-628f-4310-8fd4-fde5275103d1\") " Jul 06 19:20:57.242086 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:57.242052 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4691fbca-628f-4310-8fd4-fde5275103d1-bundle" (OuterVolumeSpecName: "bundle") pod "4691fbca-628f-4310-8fd4-fde5275103d1" (UID: "4691fbca-628f-4310-8fd4-fde5275103d1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:20:57.243606 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:57.243580 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4691fbca-628f-4310-8fd4-fde5275103d1-kube-api-access-bhd5x" (OuterVolumeSpecName: "kube-api-access-bhd5x") pod "4691fbca-628f-4310-8fd4-fde5275103d1" (UID: "4691fbca-628f-4310-8fd4-fde5275103d1"). InnerVolumeSpecName "kube-api-access-bhd5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:20:57.246261 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:57.246241 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4691fbca-628f-4310-8fd4-fde5275103d1-util" (OuterVolumeSpecName: "util") pod "4691fbca-628f-4310-8fd4-fde5275103d1" (UID: "4691fbca-628f-4310-8fd4-fde5275103d1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:20:57.342397 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:57.342316 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhd5x\" (UniqueName: \"kubernetes.io/projected/4691fbca-628f-4310-8fd4-fde5275103d1-kube-api-access-bhd5x\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:20:57.342397 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:57.342343 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4691fbca-628f-4310-8fd4-fde5275103d1-bundle\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:20:57.342397 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:57.342352 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4691fbca-628f-4310-8fd4-fde5275103d1-util\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:20:58.063089 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:58.063060 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" Jul 06 19:20:58.063298 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:58.063060 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409t86n" event={"ID":"4691fbca-628f-4310-8fd4-fde5275103d1","Type":"ContainerDied","Data":"6df763d66de80af28106572583e9bf98ea74a75e7d96b2b85b334e9d4530aa27"} Jul 06 19:20:58.063298 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:20:58.063181 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6df763d66de80af28106572583e9bf98ea74a75e7d96b2b85b334e9d4530aa27" Jul 06 19:21:16.998192 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:16.998153 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bmvc9"] Jul 06 19:21:16.998703 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:16.998459 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4691fbca-628f-4310-8fd4-fde5275103d1" containerName="extract" Jul 06 19:21:16.998703 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:16.998470 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4691fbca-628f-4310-8fd4-fde5275103d1" containerName="extract" Jul 06 19:21:16.998703 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:16.998483 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4691fbca-628f-4310-8fd4-fde5275103d1" containerName="pull" Jul 06 19:21:16.998703 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:16.998489 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4691fbca-628f-4310-8fd4-fde5275103d1" containerName="pull" Jul 06 19:21:16.998703 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:16.998498 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4691fbca-628f-4310-8fd4-fde5275103d1" containerName="util" Jul 06 19:21:16.998703 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:16.998504 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4691fbca-628f-4310-8fd4-fde5275103d1" containerName="util" Jul 06 19:21:16.998703 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:16.998566 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4691fbca-628f-4310-8fd4-fde5275103d1" containerName="extract" Jul 06 19:21:17.000596 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:17.000579 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bmvc9" Jul 06 19:21:17.003236 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:17.003209 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-p5tnt\"" Jul 06 19:21:17.004229 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:17.004211 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 19:21:17.004360 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:17.004290 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 19:21:17.018388 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:17.018343 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bmvc9"] Jul 06 19:21:17.105550 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:17.105511 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h7b8\" (UniqueName: \"kubernetes.io/projected/3678b9f8-f7c6-4b46-b5f3-87088ba8dad1-kube-api-access-4h7b8\") pod \"cert-manager-cainjector-559cdd8c87-bmvc9\" (UID: \"3678b9f8-f7c6-4b46-b5f3-87088ba8dad1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bmvc9" Jul 06 19:21:17.105764 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:17.105569 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3678b9f8-f7c6-4b46-b5f3-87088ba8dad1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bmvc9\" (UID: \"3678b9f8-f7c6-4b46-b5f3-87088ba8dad1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bmvc9" Jul 06 19:21:17.206307 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:17.206270 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3678b9f8-f7c6-4b46-b5f3-87088ba8dad1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bmvc9\" (UID: \"3678b9f8-f7c6-4b46-b5f3-87088ba8dad1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bmvc9" Jul 06 19:21:17.206520 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:17.206353 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h7b8\" (UniqueName: \"kubernetes.io/projected/3678b9f8-f7c6-4b46-b5f3-87088ba8dad1-kube-api-access-4h7b8\") pod \"cert-manager-cainjector-559cdd8c87-bmvc9\" (UID: \"3678b9f8-f7c6-4b46-b5f3-87088ba8dad1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bmvc9" Jul 06 19:21:17.216215 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:17.216179 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3678b9f8-f7c6-4b46-b5f3-87088ba8dad1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bmvc9\" (UID: \"3678b9f8-f7c6-4b46-b5f3-87088ba8dad1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bmvc9" Jul 06 19:21:17.216338 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:17.216233 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h7b8\" (UniqueName: \"kubernetes.io/projected/3678b9f8-f7c6-4b46-b5f3-87088ba8dad1-kube-api-access-4h7b8\") pod \"cert-manager-cainjector-559cdd8c87-bmvc9\" (UID: \"3678b9f8-f7c6-4b46-b5f3-87088ba8dad1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bmvc9" Jul 06 19:21:17.317902 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:17.317812 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bmvc9" Jul 06 19:21:17.447559 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:17.447523 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bmvc9"] Jul 06 19:21:17.451249 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:21:17.451220 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3678b9f8_f7c6_4b46_b5f3_87088ba8dad1.slice/crio-549f6c6020cd4a02a4e51201c852e0dd86f67e3abd7e1c4c0754057dd7e03c62 WatchSource:0}: Error finding container 549f6c6020cd4a02a4e51201c852e0dd86f67e3abd7e1c4c0754057dd7e03c62: Status 404 returned error can't find the container with id 549f6c6020cd4a02a4e51201c852e0dd86f67e3abd7e1c4c0754057dd7e03c62 Jul 06 19:21:18.137933 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:18.137890 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bmvc9" event={"ID":"3678b9f8-f7c6-4b46-b5f3-87088ba8dad1","Type":"ContainerStarted","Data":"549f6c6020cd4a02a4e51201c852e0dd86f67e3abd7e1c4c0754057dd7e03c62"} Jul 06 19:21:20.149050 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:20.148955 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bmvc9" event={"ID":"3678b9f8-f7c6-4b46-b5f3-87088ba8dad1","Type":"ContainerStarted","Data":"ebfc55c41134fe033d1ddc3e72fe66586b0c628598b9a5ff0686bb94927a47ed"} Jul 06 19:21:20.169672 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:20.169597 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bmvc9" podStartSLOduration=1.786746663 podStartE2EDuration="4.16957986s" totalImagesPullingTime="2.382833197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:20.16759952 +0000 UTC m=+581.383237050" watchObservedRunningTime="2026-07-06 19:21:20.16957986 +0000 UTC m=+581.385217391" Jul 06 19:21:39.196404 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:39.196378 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:21:39.196922 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:21:39.196378 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:22:09.398889 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:09.398855 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229"] Jul 06 19:22:09.401229 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:09.401212 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" Jul 06 19:22:09.404250 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:09.404219 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 19:22:09.404378 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:09.404361 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 19:22:09.404434 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:09.404406 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4wwgt\"" Jul 06 19:22:09.411974 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:09.411953 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229"] Jul 06 19:22:09.430046 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:09.430012 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229\" (UID: \"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" Jul 06 19:22:09.430258 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:09.430066 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt66z\" (UniqueName: \"kubernetes.io/projected/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-kube-api-access-gt66z\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229\" (UID: \"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" Jul 06 19:22:09.430258 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:09.430133 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229\" (UID: \"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" Jul 06 19:22:09.531038 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:09.530994 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229\" (UID: \"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" Jul 06 19:22:09.531038 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:09.531030 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt66z\" (UniqueName: \"kubernetes.io/projected/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-kube-api-access-gt66z\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229\" (UID: \"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" Jul 06 19:22:09.531310 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:09.531151 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229\" (UID: \"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" Jul 06 19:22:09.531394 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:09.531375 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229\" (UID: \"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" Jul 06 19:22:09.531491 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:09.531474 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229\" (UID: \"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" Jul 06 19:22:09.540637 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:09.540614 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt66z\" (UniqueName: \"kubernetes.io/projected/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-kube-api-access-gt66z\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229\" (UID: \"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" Jul 06 19:22:09.710519 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:09.710489 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" Jul 06 19:22:09.841487 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:09.841461 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229"] Jul 06 19:22:09.844251 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:22:09.844222 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbddd6f1_e8f1_4b6f_b57e_304d9a60b40b.slice/crio-0e34b37f2de8bca9e8d6dd6a9504a091552abcce73c7ae19c978414ec625a4ef WatchSource:0}: Error finding container 0e34b37f2de8bca9e8d6dd6a9504a091552abcce73c7ae19c978414ec625a4ef: Status 404 returned error can't find the container with id 0e34b37f2de8bca9e8d6dd6a9504a091552abcce73c7ae19c978414ec625a4ef Jul 06 19:22:10.344644 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:10.344609 2581 generic.go:356] "Generic (PLEG): container finished" podID="fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b" containerID="b1c1f6ea372dec65668ce18ffbefce58b2133e5d4945006249d6dcac3a5b67b5" exitCode=0 Jul 06 19:22:10.344914 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:10.344692 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" event={"ID":"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b","Type":"ContainerDied","Data":"b1c1f6ea372dec65668ce18ffbefce58b2133e5d4945006249d6dcac3a5b67b5"} Jul 06 19:22:10.344914 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:10.344747 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" event={"ID":"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b","Type":"ContainerStarted","Data":"0e34b37f2de8bca9e8d6dd6a9504a091552abcce73c7ae19c978414ec625a4ef"} Jul 06 19:22:10.345203 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:10.345189 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:22:11.349889 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:11.349854 2581 generic.go:356] "Generic (PLEG): container finished" podID="fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b" containerID="e37a93ad25695fac10fec8c3a971eda736ceef684a66727b81277aabb58f1f4d" exitCode=0 Jul 06 19:22:11.350300 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:11.349909 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" event={"ID":"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b","Type":"ContainerDied","Data":"e37a93ad25695fac10fec8c3a971eda736ceef684a66727b81277aabb58f1f4d"} Jul 06 19:22:12.355955 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:12.355917 2581 generic.go:356] "Generic (PLEG): container finished" podID="fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b" containerID="193a7c919819a33c47a2ae41606dbf71bad32813e786ef8c648790e9a5a7501b" exitCode=0 Jul 06 19:22:12.356374 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:12.356001 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" event={"ID":"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b","Type":"ContainerDied","Data":"193a7c919819a33c47a2ae41606dbf71bad32813e786ef8c648790e9a5a7501b"} Jul 06 19:22:13.487999 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:13.487970 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" Jul 06 19:22:13.564448 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:13.564411 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-util\") pod \"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b\" (UID: \"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b\") " Jul 06 19:22:13.564448 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:13.564454 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt66z\" (UniqueName: \"kubernetes.io/projected/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-kube-api-access-gt66z\") pod \"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b\" (UID: \"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b\") " Jul 06 19:22:13.564714 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:13.564487 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-bundle\") pod \"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b\" (UID: \"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b\") " Jul 06 19:22:13.565211 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:13.565183 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-bundle" (OuterVolumeSpecName: "bundle") pod "fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b" (UID: "fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:22:13.566698 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:13.566675 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-kube-api-access-gt66z" (OuterVolumeSpecName: "kube-api-access-gt66z") pod "fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b" (UID: "fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b"). InnerVolumeSpecName "kube-api-access-gt66z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:22:13.570452 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:13.570431 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-util" (OuterVolumeSpecName: "util") pod "fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b" (UID: "fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:22:13.665371 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:13.665343 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-util\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:22:13.665371 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:13.665370 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt66z\" (UniqueName: \"kubernetes.io/projected/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-kube-api-access-gt66z\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:22:13.665571 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:13.665379 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b-bundle\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:22:14.365702 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:14.365669 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" event={"ID":"fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b","Type":"ContainerDied","Data":"0e34b37f2de8bca9e8d6dd6a9504a091552abcce73c7ae19c978414ec625a4ef"} Jul 06 19:22:14.365881 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:14.365717 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e34b37f2de8bca9e8d6dd6a9504a091552abcce73c7ae19c978414ec625a4ef" Jul 06 19:22:14.365881 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:14.365716 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn9229" Jul 06 19:22:15.168570 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.168537 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-665d858957-529kg"] Jul 06 19:22:15.169044 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.168860 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b" containerName="util" Jul 06 19:22:15.169044 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.168874 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b" containerName="util" Jul 06 19:22:15.169044 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.168892 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b" containerName="extract" Jul 06 19:22:15.169044 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.168899 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b" containerName="extract" Jul 06 19:22:15.169044 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.168916 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b" containerName="pull" Jul 06 19:22:15.169044 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.168922 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b" containerName="pull" Jul 06 19:22:15.169044 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.168976 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbddd6f1-e8f1-4b6f-b57e-304d9a60b40b" containerName="extract" Jul 06 19:22:15.171287 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.171268 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" Jul 06 19:22:15.176478 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.174700 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 19:22:15.176478 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.175879 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 19:22:15.176478 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.176144 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:22:15.176478 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.176314 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 19:22:15.176478 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.176320 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4pdhd\"" Jul 06 19:22:15.176899 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.176556 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 19:22:15.183005 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.182980 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-665d858957-529kg"] Jul 06 19:22:15.280039 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.279994 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f59248b1-579e-4193-8fe7-e70da0907913-cert\") pod \"lws-controller-manager-665d858957-529kg\" (UID: \"f59248b1-579e-4193-8fe7-e70da0907913\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" Jul 06 19:22:15.280039 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.280039 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g8j6\" (UniqueName: \"kubernetes.io/projected/f59248b1-579e-4193-8fe7-e70da0907913-kube-api-access-6g8j6\") pod \"lws-controller-manager-665d858957-529kg\" (UID: \"f59248b1-579e-4193-8fe7-e70da0907913\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" Jul 06 19:22:15.280277 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.280071 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f59248b1-579e-4193-8fe7-e70da0907913-metrics-cert\") pod \"lws-controller-manager-665d858957-529kg\" (UID: \"f59248b1-579e-4193-8fe7-e70da0907913\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" Jul 06 19:22:15.280277 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.280181 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f59248b1-579e-4193-8fe7-e70da0907913-manager-config\") pod \"lws-controller-manager-665d858957-529kg\" (UID: \"f59248b1-579e-4193-8fe7-e70da0907913\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" Jul 06 19:22:15.381596 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.381555 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f59248b1-579e-4193-8fe7-e70da0907913-manager-config\") pod \"lws-controller-manager-665d858957-529kg\" (UID: \"f59248b1-579e-4193-8fe7-e70da0907913\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" Jul 06 19:22:15.381596 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.381605 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f59248b1-579e-4193-8fe7-e70da0907913-cert\") pod \"lws-controller-manager-665d858957-529kg\" (UID: \"f59248b1-579e-4193-8fe7-e70da0907913\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" Jul 06 19:22:15.381901 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.381733 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g8j6\" (UniqueName: \"kubernetes.io/projected/f59248b1-579e-4193-8fe7-e70da0907913-kube-api-access-6g8j6\") pod \"lws-controller-manager-665d858957-529kg\" (UID: \"f59248b1-579e-4193-8fe7-e70da0907913\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" Jul 06 19:22:15.381901 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.381786 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f59248b1-579e-4193-8fe7-e70da0907913-metrics-cert\") pod \"lws-controller-manager-665d858957-529kg\" (UID: \"f59248b1-579e-4193-8fe7-e70da0907913\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" Jul 06 19:22:15.382357 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.382332 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f59248b1-579e-4193-8fe7-e70da0907913-manager-config\") pod \"lws-controller-manager-665d858957-529kg\" (UID: \"f59248b1-579e-4193-8fe7-e70da0907913\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" Jul 06 19:22:15.384083 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.384064 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f59248b1-579e-4193-8fe7-e70da0907913-cert\") pod \"lws-controller-manager-665d858957-529kg\" (UID: \"f59248b1-579e-4193-8fe7-e70da0907913\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" Jul 06 19:22:15.384185 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.384108 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f59248b1-579e-4193-8fe7-e70da0907913-metrics-cert\") pod \"lws-controller-manager-665d858957-529kg\" (UID: \"f59248b1-579e-4193-8fe7-e70da0907913\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" Jul 06 19:22:15.390423 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.390403 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g8j6\" (UniqueName: \"kubernetes.io/projected/f59248b1-579e-4193-8fe7-e70da0907913-kube-api-access-6g8j6\") pod \"lws-controller-manager-665d858957-529kg\" (UID: \"f59248b1-579e-4193-8fe7-e70da0907913\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" Jul 06 19:22:15.487285 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.487187 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" Jul 06 19:22:15.619293 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:15.619260 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-665d858957-529kg"] Jul 06 19:22:15.623289 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:22:15.623253 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59248b1_579e_4193_8fe7_e70da0907913.slice/crio-8bec044b9779e9c94e1a3b8cfb3e3ba39c5975cfcc432ce45a1b7892a61baded WatchSource:0}: Error finding container 8bec044b9779e9c94e1a3b8cfb3e3ba39c5975cfcc432ce45a1b7892a61baded: Status 404 returned error can't find the container with id 8bec044b9779e9c94e1a3b8cfb3e3ba39c5975cfcc432ce45a1b7892a61baded Jul 06 19:22:16.377145 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:16.377101 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" event={"ID":"f59248b1-579e-4193-8fe7-e70da0907913","Type":"ContainerStarted","Data":"8bec044b9779e9c94e1a3b8cfb3e3ba39c5975cfcc432ce45a1b7892a61baded"} Jul 06 19:22:26.304218 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.304187 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj"] Jul 06 19:22:26.306526 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.306511 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:26.311126 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.311104 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 19:22:26.312566 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.312546 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jdd5t\"" Jul 06 19:22:26.316890 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.316874 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 19:22:26.317781 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.317763 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 19:22:26.318803 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.318786 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 19:22:26.337076 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.337054 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj"] Jul 06 19:22:26.371542 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.371514 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3a43c343-443c-442d-80b1-8b2dc50294ba-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-g64zj\" (UID: \"3a43c343-443c-442d-80b1-8b2dc50294ba\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:26.371704 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.371559 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3a43c343-443c-442d-80b1-8b2dc50294ba-manifests\") pod \"opendatahub-operator-controller-manager-56f978bf57-g64zj\" (UID: \"3a43c343-443c-442d-80b1-8b2dc50294ba\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:26.371704 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.371620 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52cxz\" (UniqueName: \"kubernetes.io/projected/3a43c343-443c-442d-80b1-8b2dc50294ba-kube-api-access-52cxz\") pod \"opendatahub-operator-controller-manager-56f978bf57-g64zj\" (UID: \"3a43c343-443c-442d-80b1-8b2dc50294ba\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:26.371795 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.371703 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a43c343-443c-442d-80b1-8b2dc50294ba-tmp\") pod \"opendatahub-operator-controller-manager-56f978bf57-g64zj\" (UID: \"3a43c343-443c-442d-80b1-8b2dc50294ba\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:26.371795 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.371749 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a43c343-443c-442d-80b1-8b2dc50294ba-webhook-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-g64zj\" (UID: \"3a43c343-443c-442d-80b1-8b2dc50294ba\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:26.472478 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.472442 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3a43c343-443c-442d-80b1-8b2dc50294ba-manifests\") pod \"opendatahub-operator-controller-manager-56f978bf57-g64zj\" (UID: \"3a43c343-443c-442d-80b1-8b2dc50294ba\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:26.472688 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.472503 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52cxz\" (UniqueName: \"kubernetes.io/projected/3a43c343-443c-442d-80b1-8b2dc50294ba-kube-api-access-52cxz\") pod \"opendatahub-operator-controller-manager-56f978bf57-g64zj\" (UID: \"3a43c343-443c-442d-80b1-8b2dc50294ba\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:26.472688 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.472549 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a43c343-443c-442d-80b1-8b2dc50294ba-tmp\") pod \"opendatahub-operator-controller-manager-56f978bf57-g64zj\" (UID: \"3a43c343-443c-442d-80b1-8b2dc50294ba\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:26.472688 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.472604 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a43c343-443c-442d-80b1-8b2dc50294ba-webhook-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-g64zj\" (UID: \"3a43c343-443c-442d-80b1-8b2dc50294ba\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:26.472864 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.472689 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3a43c343-443c-442d-80b1-8b2dc50294ba-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-g64zj\" (UID: \"3a43c343-443c-442d-80b1-8b2dc50294ba\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:26.473036 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.473008 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3a43c343-443c-442d-80b1-8b2dc50294ba-manifests\") pod \"opendatahub-operator-controller-manager-56f978bf57-g64zj\" (UID: \"3a43c343-443c-442d-80b1-8b2dc50294ba\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:26.475040 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.475017 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a43c343-443c-442d-80b1-8b2dc50294ba-tmp\") pod \"opendatahub-operator-controller-manager-56f978bf57-g64zj\" (UID: \"3a43c343-443c-442d-80b1-8b2dc50294ba\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:26.475247 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.475230 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a43c343-443c-442d-80b1-8b2dc50294ba-webhook-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-g64zj\" (UID: \"3a43c343-443c-442d-80b1-8b2dc50294ba\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:26.475396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.475376 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3a43c343-443c-442d-80b1-8b2dc50294ba-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-g64zj\" (UID: \"3a43c343-443c-442d-80b1-8b2dc50294ba\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:26.481206 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.481184 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52cxz\" (UniqueName: \"kubernetes.io/projected/3a43c343-443c-442d-80b1-8b2dc50294ba-kube-api-access-52cxz\") pod \"opendatahub-operator-controller-manager-56f978bf57-g64zj\" (UID: \"3a43c343-443c-442d-80b1-8b2dc50294ba\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:26.617230 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.617142 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:26.760128 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:26.755675 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj"] Jul 06 19:22:27.423366 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:27.423327 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" event={"ID":"3a43c343-443c-442d-80b1-8b2dc50294ba","Type":"ContainerStarted","Data":"993c3c983c00e2493b5d626d562cf0cab42611a8d189d572b8cd919f1c06b60f"} Jul 06 19:22:31.443923 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:31.443886 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" event={"ID":"3a43c343-443c-442d-80b1-8b2dc50294ba","Type":"ContainerStarted","Data":"b8b4a37bca6bf2ed1346df05a6eb34e461f717d3dc2522955600264ae8bedf46"} Jul 06 19:22:33.455385 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:33.455349 2581 generic.go:356] "Generic (PLEG): container finished" podID="3a43c343-443c-442d-80b1-8b2dc50294ba" containerID="b8b4a37bca6bf2ed1346df05a6eb34e461f717d3dc2522955600264ae8bedf46" exitCode=0 Jul 06 19:22:33.455844 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:33.455410 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" event={"ID":"3a43c343-443c-442d-80b1-8b2dc50294ba","Type":"ContainerDied","Data":"b8b4a37bca6bf2ed1346df05a6eb34e461f717d3dc2522955600264ae8bedf46"} Jul 06 19:22:34.461979 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:34.461945 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" event={"ID":"3a43c343-443c-442d-80b1-8b2dc50294ba","Type":"ContainerStarted","Data":"122926926b235ab4738b2e0227cb84ede3942a5141eef581dc3fbb22cd7d7ed4"} Jul 06 19:22:34.462354 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:34.462155 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:34.484855 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:34.484808 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" podStartSLOduration=2.356903263 podStartE2EDuration="8.484789289s" totalImagesPullingTime="4.967099138s" totalInitContainerRuntime="1.160786888s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:34.482486369 +0000 UTC m=+655.698123898" watchObservedRunningTime="2026-07-06 19:22:34.484789289 +0000 UTC m=+655.700426820" Jul 06 19:22:45.469187 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:45.469151 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-g64zj" Jul 06 19:22:49.526832 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:49.526789 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" event={"ID":"f59248b1-579e-4193-8fe7-e70da0907913","Type":"ContainerStarted","Data":"d234a3511f1a78e777d1115c5dcb1bdefccf1f11401acccd9b5184cc0ece72ba"} Jul 06 19:22:49.527335 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:49.526919 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" Jul 06 19:22:49.543879 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:49.543832 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" podStartSLOduration=1.456953871 podStartE2EDuration="34.543814632s" totalImagesPullingTime="33.086860761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:49.543038962 +0000 UTC m=+670.758676492" watchObservedRunningTime="2026-07-06 19:22:49.543814632 +0000 UTC m=+670.759452162" Jul 06 19:22:55.520038 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.519998 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w"] Jul 06 19:22:55.522297 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.522278 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.525558 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.525526 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 06 19:22:55.525726 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.525526 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 06 19:22:55.525726 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.525540 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 06 19:22:55.525726 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.525639 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-m5m87\"" Jul 06 19:22:55.525726 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.525539 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:22:55.530795 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.530775 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8cf813ee-2998-49c0-81b0-ba574c55e85f-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.530935 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.530800 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8cf813ee-2998-49c0-81b0-ba574c55e85f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.530935 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.530822 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf813ee-2998-49c0-81b0-ba574c55e85f-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.530935 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.530847 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf813ee-2998-49c0-81b0-ba574c55e85f-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.530935 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.530910 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8cf813ee-2998-49c0-81b0-ba574c55e85f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.531085 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.530940 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk85z\" (UniqueName: \"kubernetes.io/projected/8cf813ee-2998-49c0-81b0-ba574c55e85f-kube-api-access-sk85z\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.531085 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.530957 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8cf813ee-2998-49c0-81b0-ba574c55e85f-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.574058 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.574021 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w"] Jul 06 19:22:55.631606 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.631562 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8cf813ee-2998-49c0-81b0-ba574c55e85f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.631606 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.631613 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk85z\" (UniqueName: \"kubernetes.io/projected/8cf813ee-2998-49c0-81b0-ba574c55e85f-kube-api-access-sk85z\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.631888 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.631635 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8cf813ee-2998-49c0-81b0-ba574c55e85f-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.631888 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.631720 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8cf813ee-2998-49c0-81b0-ba574c55e85f-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.631888 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.631739 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8cf813ee-2998-49c0-81b0-ba574c55e85f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.631888 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.631759 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf813ee-2998-49c0-81b0-ba574c55e85f-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.631888 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.631774 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf813ee-2998-49c0-81b0-ba574c55e85f-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.632554 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.632526 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8cf813ee-2998-49c0-81b0-ba574c55e85f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.634131 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.634099 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf813ee-2998-49c0-81b0-ba574c55e85f-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.634357 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.634143 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8cf813ee-2998-49c0-81b0-ba574c55e85f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.634531 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.634507 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8cf813ee-2998-49c0-81b0-ba574c55e85f-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.634611 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.634582 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8cf813ee-2998-49c0-81b0-ba574c55e85f-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.649631 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.649589 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf813ee-2998-49c0-81b0-ba574c55e85f-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.650724 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.650703 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk85z\" (UniqueName: \"kubernetes.io/projected/8cf813ee-2998-49c0-81b0-ba574c55e85f-kube-api-access-sk85z\") pod \"istiod-openshift-gateway-94bb8fbfd-m2x4w\" (UID: \"8cf813ee-2998-49c0-81b0-ba574c55e85f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.831533 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.831438 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:55.977129 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:55.977096 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w"] Jul 06 19:22:55.980546 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:22:55.980519 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf813ee_2998_49c0_81b0_ba574c55e85f.slice/crio-7109b8fd108137c2bb1d1dac8f6489a2c6381adad884b2fcb466f4cfa62cc046 WatchSource:0}: Error finding container 7109b8fd108137c2bb1d1dac8f6489a2c6381adad884b2fcb466f4cfa62cc046: Status 404 returned error can't find the container with id 7109b8fd108137c2bb1d1dac8f6489a2c6381adad884b2fcb466f4cfa62cc046 Jul 06 19:22:56.559250 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:56.559201 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" event={"ID":"8cf813ee-2998-49c0-81b0-ba574c55e85f","Type":"ContainerStarted","Data":"7109b8fd108137c2bb1d1dac8f6489a2c6381adad884b2fcb466f4cfa62cc046"} Jul 06 19:22:58.665055 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:58.665008 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:22:58.665370 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:58.665088 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:22:59.575744 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:59.575708 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" event={"ID":"8cf813ee-2998-49c0-81b0-ba574c55e85f","Type":"ContainerStarted","Data":"3456bdbc8f6f3ca21cb002afe74863c4596a5717c8a800e467abd6fbc6a9ecde"} Jul 06 19:22:59.575928 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:59.575866 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:59.577574 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:59.577553 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" Jul 06 19:22:59.621837 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:22:59.621783 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m2x4w" podStartSLOduration=1.939569789 podStartE2EDuration="4.621761125s" totalImagesPullingTime="2.682191336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:59.619225064 +0000 UTC m=+680.834862595" watchObservedRunningTime="2026-07-06 19:22:59.621761125 +0000 UTC m=+680.837398658" Jul 06 19:23:00.534211 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:00.534181 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-665d858957-529kg" Jul 06 19:23:25.854600 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:25.854569 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57b56cbf86-prbfx"] Jul 06 19:23:50.878335 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:50.878274 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-57b56cbf86-prbfx" podUID="e43b482d-c881-44e9-a963-96f447c64e7f" containerName="console" containerID="cri-o://7342735fe5346e73135993fb9e6c5bf02bfeee0e71b0710e612a5a55bb3ef61a" gracePeriod=15 Jul 06 19:23:51.120760 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.120736 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:23:51.209748 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.209711 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-console-config\") pod \"e43b482d-c881-44e9-a963-96f447c64e7f\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " Jul 06 19:23:51.209904 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.209761 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e43b482d-c881-44e9-a963-96f447c64e7f-console-oauth-config\") pod \"e43b482d-c881-44e9-a963-96f447c64e7f\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " Jul 06 19:23:51.209904 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.209790 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e43b482d-c881-44e9-a963-96f447c64e7f-console-serving-cert\") pod \"e43b482d-c881-44e9-a963-96f447c64e7f\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " Jul 06 19:23:51.209904 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.209807 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-service-ca\") pod \"e43b482d-c881-44e9-a963-96f447c64e7f\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " Jul 06 19:23:51.209904 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.209843 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-trusted-ca-bundle\") pod \"e43b482d-c881-44e9-a963-96f447c64e7f\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " Jul 06 19:23:51.209904 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.209874 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-oauth-serving-cert\") pod \"e43b482d-c881-44e9-a963-96f447c64e7f\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " Jul 06 19:23:51.209904 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.209898 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz82d\" (UniqueName: \"kubernetes.io/projected/e43b482d-c881-44e9-a963-96f447c64e7f-kube-api-access-gz82d\") pod \"e43b482d-c881-44e9-a963-96f447c64e7f\" (UID: \"e43b482d-c881-44e9-a963-96f447c64e7f\") " Jul 06 19:23:51.210227 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.210190 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-console-config" (OuterVolumeSpecName: "console-config") pod "e43b482d-c881-44e9-a963-96f447c64e7f" (UID: "e43b482d-c881-44e9-a963-96f447c64e7f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:23:51.210337 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.210312 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-service-ca" (OuterVolumeSpecName: "service-ca") pod "e43b482d-c881-44e9-a963-96f447c64e7f" (UID: "e43b482d-c881-44e9-a963-96f447c64e7f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:23:51.210406 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.210358 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e43b482d-c881-44e9-a963-96f447c64e7f" (UID: "e43b482d-c881-44e9-a963-96f447c64e7f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:23:51.210459 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.210409 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e43b482d-c881-44e9-a963-96f447c64e7f" (UID: "e43b482d-c881-44e9-a963-96f447c64e7f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:23:51.212056 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.212030 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e43b482d-c881-44e9-a963-96f447c64e7f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e43b482d-c881-44e9-a963-96f447c64e7f" (UID: "e43b482d-c881-44e9-a963-96f447c64e7f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:23:51.212175 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.212056 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e43b482d-c881-44e9-a963-96f447c64e7f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e43b482d-c881-44e9-a963-96f447c64e7f" (UID: "e43b482d-c881-44e9-a963-96f447c64e7f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:23:51.212175 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.212119 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43b482d-c881-44e9-a963-96f447c64e7f-kube-api-access-gz82d" (OuterVolumeSpecName: "kube-api-access-gz82d") pod "e43b482d-c881-44e9-a963-96f447c64e7f" (UID: "e43b482d-c881-44e9-a963-96f447c64e7f"). InnerVolumeSpecName "kube-api-access-gz82d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:23:51.311476 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.311440 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-trusted-ca-bundle\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:23:51.311476 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.311469 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-oauth-serving-cert\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:23:51.311476 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.311480 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz82d\" (UniqueName: \"kubernetes.io/projected/e43b482d-c881-44e9-a963-96f447c64e7f-kube-api-access-gz82d\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:23:51.311728 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.311490 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-console-config\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:23:51.311728 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.311500 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e43b482d-c881-44e9-a963-96f447c64e7f-console-oauth-config\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:23:51.311728 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.311509 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e43b482d-c881-44e9-a963-96f447c64e7f-console-serving-cert\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:23:51.311728 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.311517 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b482d-c881-44e9-a963-96f447c64e7f-service-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:23:51.803223 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.803188 2581 generic.go:356] "Generic (PLEG): container finished" podID="e43b482d-c881-44e9-a963-96f447c64e7f" containerID="7342735fe5346e73135993fb9e6c5bf02bfeee0e71b0710e612a5a55bb3ef61a" exitCode=0 Jul 06 19:23:51.803399 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.803248 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57b56cbf86-prbfx" Jul 06 19:23:51.803399 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.803273 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57b56cbf86-prbfx" event={"ID":"e43b482d-c881-44e9-a963-96f447c64e7f","Type":"ContainerDied","Data":"7342735fe5346e73135993fb9e6c5bf02bfeee0e71b0710e612a5a55bb3ef61a"} Jul 06 19:23:51.803399 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.803347 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57b56cbf86-prbfx" event={"ID":"e43b482d-c881-44e9-a963-96f447c64e7f","Type":"ContainerDied","Data":"9e4f65310a01c4738deb9732bb0edb3acd744bbead8a85a4d38bd65a6ec0a2e4"} Jul 06 19:23:51.803399 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.803366 2581 scope.go:117] "RemoveContainer" containerID="7342735fe5346e73135993fb9e6c5bf02bfeee0e71b0710e612a5a55bb3ef61a" Jul 06 19:23:51.812538 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.812526 2581 scope.go:117] "RemoveContainer" containerID="7342735fe5346e73135993fb9e6c5bf02bfeee0e71b0710e612a5a55bb3ef61a" Jul 06 19:23:51.812830 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:23:51.812812 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7342735fe5346e73135993fb9e6c5bf02bfeee0e71b0710e612a5a55bb3ef61a\": container with ID starting with 7342735fe5346e73135993fb9e6c5bf02bfeee0e71b0710e612a5a55bb3ef61a not found: ID does not exist" containerID="7342735fe5346e73135993fb9e6c5bf02bfeee0e71b0710e612a5a55bb3ef61a" Jul 06 19:23:51.812893 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.812837 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7342735fe5346e73135993fb9e6c5bf02bfeee0e71b0710e612a5a55bb3ef61a"} err="failed to get container status \"7342735fe5346e73135993fb9e6c5bf02bfeee0e71b0710e612a5a55bb3ef61a\": rpc error: code = NotFound desc = could not find container \"7342735fe5346e73135993fb9e6c5bf02bfeee0e71b0710e612a5a55bb3ef61a\": container with ID starting with 7342735fe5346e73135993fb9e6c5bf02bfeee0e71b0710e612a5a55bb3ef61a not found: ID does not exist" Jul 06 19:23:51.826723 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.826702 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57b56cbf86-prbfx"] Jul 06 19:23:51.832664 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:51.832630 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57b56cbf86-prbfx"] Jul 06 19:23:52.146857 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:23:52.146756 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e43b482d-c881-44e9-a963-96f447c64e7f" path="/var/lib/kubelet/pods/e43b482d-c881-44e9-a963-96f447c64e7f/volumes" Jul 06 19:24:44.520826 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:44.520784 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw"] Jul 06 19:24:44.521213 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:44.521093 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e43b482d-c881-44e9-a963-96f447c64e7f" containerName="console" Jul 06 19:24:44.521213 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:44.521105 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e43b482d-c881-44e9-a963-96f447c64e7f" containerName="console" Jul 06 19:24:44.521213 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:44.521156 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e43b482d-c881-44e9-a963-96f447c64e7f" containerName="console" Jul 06 19:24:44.524275 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:44.524258 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" Jul 06 19:24:44.526531 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:44.526506 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82gqf\" (UniqueName: \"kubernetes.io/projected/fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9-kube-api-access-82gqf\") pod \"kuadrant-operator-controller-manager-86878c769f-l8mmw\" (UID: \"fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" Jul 06 19:24:44.526669 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:44.526551 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-l8mmw\" (UID: \"fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" Jul 06 19:24:44.528276 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:44.528258 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z2nsb\"" Jul 06 19:24:44.528490 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:44.528475 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:24:44.528797 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:44.528781 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:24:44.540353 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:44.540334 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw"] Jul 06 19:24:44.627317 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:44.627286 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-l8mmw\" (UID: \"fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" Jul 06 19:24:44.627470 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:44.627351 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82gqf\" (UniqueName: \"kubernetes.io/projected/fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9-kube-api-access-82gqf\") pod \"kuadrant-operator-controller-manager-86878c769f-l8mmw\" (UID: \"fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" Jul 06 19:24:44.627637 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:44.627617 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-l8mmw\" (UID: \"fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" Jul 06 19:24:44.636812 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:44.636793 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82gqf\" (UniqueName: \"kubernetes.io/projected/fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9-kube-api-access-82gqf\") pod \"kuadrant-operator-controller-manager-86878c769f-l8mmw\" (UID: \"fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" Jul 06 19:24:44.833359 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:44.833285 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" Jul 06 19:24:44.979915 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:44.979894 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw"] Jul 06 19:24:44.982737 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:24:44.982707 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1fbcf4_a90f_4fc4_bd1a_0a7ee60ff0c9.slice/crio-d611da6b1c27e674b20c640097bc485baaa62a40aa31fb2c1fe90ebef4057720 WatchSource:0}: Error finding container d611da6b1c27e674b20c640097bc485baaa62a40aa31fb2c1fe90ebef4057720: Status 404 returned error can't find the container with id d611da6b1c27e674b20c640097bc485baaa62a40aa31fb2c1fe90ebef4057720 Jul 06 19:24:45.035452 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:45.035425 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" event={"ID":"fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9","Type":"ContainerStarted","Data":"d611da6b1c27e674b20c640097bc485baaa62a40aa31fb2c1fe90ebef4057720"} Jul 06 19:24:51.069904 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:51.069819 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" event={"ID":"fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9","Type":"ContainerStarted","Data":"20947f390aa27992c3d427b9782f29e4594ceeecafd8509b3f00d564bf7ed37b"} Jul 06 19:24:51.070290 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:51.069919 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" Jul 06 19:24:51.089999 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:24:51.089953 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" podStartSLOduration=1.415490488 podStartE2EDuration="7.089940759s" totalImagesPullingTime="5.674450271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:51.088904034 +0000 UTC m=+792.304541574" watchObservedRunningTime="2026-07-06 19:24:51.089940759 +0000 UTC m=+792.305578287" Jul 06 19:25:02.076319 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:25:02.076243 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" Jul 06 19:26:39.222782 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:39.222756 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:26:39.223774 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:39.223746 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:26:41.861316 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:41.861284 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-dfb658bc5-kzrgj"] Jul 06 19:26:41.863526 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:41.863509 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-dfb658bc5-kzrgj" Jul 06 19:26:41.866696 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:41.866674 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 06 19:26:41.866840 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:41.866677 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-5zqhd\"" Jul 06 19:26:41.871751 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:41.871725 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d0828b-9eb2-4926-9fcb-31bd6e87f727-openshift-service-ca\") pod \"maas-api-dfb658bc5-kzrgj\" (UID: \"20d0828b-9eb2-4926-9fcb-31bd6e87f727\") " pod="opendatahub/maas-api-dfb658bc5-kzrgj" Jul 06 19:26:41.871843 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:41.871805 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20d0828b-9eb2-4926-9fcb-31bd6e87f727-maas-api-tls\") pod \"maas-api-dfb658bc5-kzrgj\" (UID: \"20d0828b-9eb2-4926-9fcb-31bd6e87f727\") " pod="opendatahub/maas-api-dfb658bc5-kzrgj" Jul 06 19:26:41.871889 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:41.871848 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t2lv\" (UniqueName: \"kubernetes.io/projected/20d0828b-9eb2-4926-9fcb-31bd6e87f727-kube-api-access-2t2lv\") pod \"maas-api-dfb658bc5-kzrgj\" (UID: \"20d0828b-9eb2-4926-9fcb-31bd6e87f727\") " pod="opendatahub/maas-api-dfb658bc5-kzrgj" Jul 06 19:26:41.891104 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:41.891085 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-dfb658bc5-kzrgj"] Jul 06 19:26:41.972388 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:41.972366 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d0828b-9eb2-4926-9fcb-31bd6e87f727-openshift-service-ca\") pod \"maas-api-dfb658bc5-kzrgj\" (UID: \"20d0828b-9eb2-4926-9fcb-31bd6e87f727\") " pod="opendatahub/maas-api-dfb658bc5-kzrgj" Jul 06 19:26:41.972475 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:41.972403 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20d0828b-9eb2-4926-9fcb-31bd6e87f727-maas-api-tls\") pod \"maas-api-dfb658bc5-kzrgj\" (UID: \"20d0828b-9eb2-4926-9fcb-31bd6e87f727\") " pod="opendatahub/maas-api-dfb658bc5-kzrgj" Jul 06 19:26:41.972475 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:41.972427 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t2lv\" (UniqueName: \"kubernetes.io/projected/20d0828b-9eb2-4926-9fcb-31bd6e87f727-kube-api-access-2t2lv\") pod \"maas-api-dfb658bc5-kzrgj\" (UID: \"20d0828b-9eb2-4926-9fcb-31bd6e87f727\") " pod="opendatahub/maas-api-dfb658bc5-kzrgj" Jul 06 19:26:41.972544 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:26:41.972504 2581 secret.go:189] Couldn't get secret opendatahub/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 06 19:26:41.972580 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:26:41.972564 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20d0828b-9eb2-4926-9fcb-31bd6e87f727-maas-api-tls podName:20d0828b-9eb2-4926-9fcb-31bd6e87f727 nodeName:}" failed. No retries permitted until 2026-07-06 19:26:42.47255081 +0000 UTC m=+903.688188321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20d0828b-9eb2-4926-9fcb-31bd6e87f727-maas-api-tls") pod "maas-api-dfb658bc5-kzrgj" (UID: "20d0828b-9eb2-4926-9fcb-31bd6e87f727") : secret "maas-api-serving-cert" not found Jul 06 19:26:41.972982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:41.972966 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20d0828b-9eb2-4926-9fcb-31bd6e87f727-openshift-service-ca\") pod \"maas-api-dfb658bc5-kzrgj\" (UID: \"20d0828b-9eb2-4926-9fcb-31bd6e87f727\") " pod="opendatahub/maas-api-dfb658bc5-kzrgj" Jul 06 19:26:41.984004 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:41.983976 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t2lv\" (UniqueName: \"kubernetes.io/projected/20d0828b-9eb2-4926-9fcb-31bd6e87f727-kube-api-access-2t2lv\") pod \"maas-api-dfb658bc5-kzrgj\" (UID: \"20d0828b-9eb2-4926-9fcb-31bd6e87f727\") " pod="opendatahub/maas-api-dfb658bc5-kzrgj" Jul 06 19:26:42.476013 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:42.475986 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20d0828b-9eb2-4926-9fcb-31bd6e87f727-maas-api-tls\") pod \"maas-api-dfb658bc5-kzrgj\" (UID: \"20d0828b-9eb2-4926-9fcb-31bd6e87f727\") " pod="opendatahub/maas-api-dfb658bc5-kzrgj" Jul 06 19:26:42.478207 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:42.478179 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20d0828b-9eb2-4926-9fcb-31bd6e87f727-maas-api-tls\") pod \"maas-api-dfb658bc5-kzrgj\" (UID: \"20d0828b-9eb2-4926-9fcb-31bd6e87f727\") " pod="opendatahub/maas-api-dfb658bc5-kzrgj" Jul 06 19:26:42.774359 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:42.774283 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-dfb658bc5-kzrgj" Jul 06 19:26:43.106839 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:43.106802 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-dfb658bc5-kzrgj"] Jul 06 19:26:43.109217 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:26:43.109191 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d0828b_9eb2_4926_9fcb_31bd6e87f727.slice/crio-a4b92bdb444600fc34407b41c05c48502d211830ec87d3c7ffcfe55eec11e672 WatchSource:0}: Error finding container a4b92bdb444600fc34407b41c05c48502d211830ec87d3c7ffcfe55eec11e672: Status 404 returned error can't find the container with id a4b92bdb444600fc34407b41c05c48502d211830ec87d3c7ffcfe55eec11e672 Jul 06 19:26:43.552898 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:43.552863 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-dfb658bc5-kzrgj" event={"ID":"20d0828b-9eb2-4926-9fcb-31bd6e87f727","Type":"ContainerStarted","Data":"a4b92bdb444600fc34407b41c05c48502d211830ec87d3c7ffcfe55eec11e672"} Jul 06 19:26:45.563688 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:45.563592 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-dfb658bc5-kzrgj" event={"ID":"20d0828b-9eb2-4926-9fcb-31bd6e87f727","Type":"ContainerStarted","Data":"bbb09604a55bfd90564d613269c24c5146af42e0e6ba437a19550913c2409915"} Jul 06 19:26:45.564087 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:45.563726 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-dfb658bc5-kzrgj" Jul 06 19:26:45.581937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:45.581892 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-dfb658bc5-kzrgj" podStartSLOduration=2.460341861 podStartE2EDuration="4.581877184s" totalImagesPullingTime="2.121535323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:26:45.580256892 +0000 UTC m=+906.795894422" watchObservedRunningTime="2026-07-06 19:26:45.581877184 +0000 UTC m=+906.797514713" Jul 06 19:26:51.574268 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:26:51.574236 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-dfb658bc5-kzrgj" Jul 06 19:27:24.230187 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.230150 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx"] Jul 06 19:27:24.232460 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.232441 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.240895 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.240872 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h6bns\"" Jul 06 19:27:24.241543 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.241524 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 19:27:24.241743 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.241729 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 19:27:24.241993 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.241976 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 06 19:27:24.250749 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.250726 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx"] Jul 06 19:27:24.258972 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.258946 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c7f816-51e2-4e00-b92a-738bf261e73a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.259077 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.258982 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c7f816-51e2-4e00-b92a-738bf261e73a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.259077 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.259011 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c7f816-51e2-4e00-b92a-738bf261e73a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.259161 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.259087 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c7f816-51e2-4e00-b92a-738bf261e73a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.259161 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.259125 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c7f816-51e2-4e00-b92a-738bf261e73a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.259273 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.259238 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hx5p\" (UniqueName: \"kubernetes.io/projected/10c7f816-51e2-4e00-b92a-738bf261e73a-kube-api-access-4hx5p\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.359976 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.359921 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c7f816-51e2-4e00-b92a-738bf261e73a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.359976 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.359976 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c7f816-51e2-4e00-b92a-738bf261e73a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.360169 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.360000 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c7f816-51e2-4e00-b92a-738bf261e73a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.360169 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.360055 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hx5p\" (UniqueName: \"kubernetes.io/projected/10c7f816-51e2-4e00-b92a-738bf261e73a-kube-api-access-4hx5p\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.360169 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.360102 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c7f816-51e2-4e00-b92a-738bf261e73a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.360169 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.360128 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c7f816-51e2-4e00-b92a-738bf261e73a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.360372 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.360333 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c7f816-51e2-4e00-b92a-738bf261e73a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.360434 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.360371 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c7f816-51e2-4e00-b92a-738bf261e73a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.360485 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.360469 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c7f816-51e2-4e00-b92a-738bf261e73a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.362451 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.362429 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c7f816-51e2-4e00-b92a-738bf261e73a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.362773 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.362754 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c7f816-51e2-4e00-b92a-738bf261e73a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.371979 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.371957 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hx5p\" (UniqueName: \"kubernetes.io/projected/10c7f816-51e2-4e00-b92a-738bf261e73a-kube-api-access-4hx5p\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z7qdx\" (UID: \"10c7f816-51e2-4e00-b92a-738bf261e73a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.543842 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.543792 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:24.685149 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.685123 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx"] Jul 06 19:27:24.686597 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:27:24.686574 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c7f816_51e2_4e00_b92a_738bf261e73a.slice/crio-d969666aaca089cf62d89779e0cc6d7e497dfa5f02fa558693fc446b8d2f9b6e WatchSource:0}: Error finding container d969666aaca089cf62d89779e0cc6d7e497dfa5f02fa558693fc446b8d2f9b6e: Status 404 returned error can't find the container with id d969666aaca089cf62d89779e0cc6d7e497dfa5f02fa558693fc446b8d2f9b6e Jul 06 19:27:24.687986 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.687972 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:27:24.741485 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:24.741457 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" event={"ID":"10c7f816-51e2-4e00-b92a-738bf261e73a","Type":"ContainerStarted","Data":"d969666aaca089cf62d89779e0cc6d7e497dfa5f02fa558693fc446b8d2f9b6e"} Jul 06 19:27:25.746956 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:25.746925 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" event={"ID":"10c7f816-51e2-4e00-b92a-738bf261e73a","Type":"ContainerStarted","Data":"4c80a230fffd40745aba4475aef4f4b117ac586e4876465f914038847410d1fa"} Jul 06 19:27:25.778829 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:25.778785 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" podStartSLOduration=0.813592102 podStartE2EDuration="1.778771957s" totalImagesPullingTime="965.179855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:25.777961768 +0000 UTC m=+946.993599296" watchObservedRunningTime="2026-07-06 19:27:25.778771957 +0000 UTC m=+946.994409485" Jul 06 19:27:34.544791 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:34.544752 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:34.545306 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:34.544813 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:34.556949 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:34.556919 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:27:34.800244 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:27:34.800166 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z7qdx" Jul 06 19:30:00.150905 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:00.150872 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722770-ft6x6"] Jul 06 19:30:00.154127 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:00.154099 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722770-ft6x6" Jul 06 19:30:00.193078 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:00.193052 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6jw6\" (UniqueName: \"kubernetes.io/projected/177b0d59-f9b3-42f5-b2b4-60bf17264d46-kube-api-access-m6jw6\") pod \"maas-api-key-cleanup-29722770-ft6x6\" (UID: \"177b0d59-f9b3-42f5-b2b4-60bf17264d46\") " pod="opendatahub/maas-api-key-cleanup-29722770-ft6x6" Jul 06 19:30:00.294318 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:00.294297 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6jw6\" (UniqueName: \"kubernetes.io/projected/177b0d59-f9b3-42f5-b2b4-60bf17264d46-kube-api-access-m6jw6\") pod \"maas-api-key-cleanup-29722770-ft6x6\" (UID: \"177b0d59-f9b3-42f5-b2b4-60bf17264d46\") " pod="opendatahub/maas-api-key-cleanup-29722770-ft6x6" Jul 06 19:30:00.305620 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:00.305584 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6jw6\" (UniqueName: \"kubernetes.io/projected/177b0d59-f9b3-42f5-b2b4-60bf17264d46-kube-api-access-m6jw6\") pod \"maas-api-key-cleanup-29722770-ft6x6\" (UID: \"177b0d59-f9b3-42f5-b2b4-60bf17264d46\") " pod="opendatahub/maas-api-key-cleanup-29722770-ft6x6" Jul 06 19:30:00.463465 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:00.463445 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722770-ft6x6" Jul 06 19:30:25.814554 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:25.814522 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722770-ft6x6"] Jul 06 19:30:25.992030 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:25.991830 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722770-ft6x6"] Jul 06 19:30:25.994506 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:30:25.994483 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177b0d59_f9b3_42f5_b2b4_60bf17264d46.slice/crio-99bf6257fc69956b7ec2006dc318b00b9183aa6f6123ff5611150f45d34f02dd WatchSource:0}: Error finding container 99bf6257fc69956b7ec2006dc318b00b9183aa6f6123ff5611150f45d34f02dd: Status 404 returned error can't find the container with id 99bf6257fc69956b7ec2006dc318b00b9183aa6f6123ff5611150f45d34f02dd Jul 06 19:30:26.544813 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:26.544774 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722770-ft6x6" event={"ID":"177b0d59-f9b3-42f5-b2b4-60bf17264d46","Type":"ContainerStarted","Data":"99bf6257fc69956b7ec2006dc318b00b9183aa6f6123ff5611150f45d34f02dd"} Jul 06 19:30:30.565106 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:30.565014 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722770-ft6x6" event={"ID":"177b0d59-f9b3-42f5-b2b4-60bf17264d46","Type":"ContainerStarted","Data":"bbedcd846986eda11a421294b3c3539b44455dfddcfac8307c46025e8482c375"} Jul 06 19:30:30.581155 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:30.581107 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722770-ft6x6" podStartSLOduration=26.663488519 podStartE2EDuration="30.581091336s" totalImagesPullingTime="3.917602817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:30:30.580368815 +0000 UTC m=+1131.796006355" watchObservedRunningTime="2026-07-06 19:30:30.581091336 +0000 UTC m=+1131.796728864" Jul 06 19:30:31.571215 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:31.571176 2581 generic.go:356] "Generic (PLEG): container finished" podID="177b0d59-f9b3-42f5-b2b4-60bf17264d46" containerID="bbedcd846986eda11a421294b3c3539b44455dfddcfac8307c46025e8482c375" exitCode=0 Jul 06 19:30:31.571711 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:31.571230 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722770-ft6x6" event={"ID":"177b0d59-f9b3-42f5-b2b4-60bf17264d46","Type":"ContainerDied","Data":"bbedcd846986eda11a421294b3c3539b44455dfddcfac8307c46025e8482c375"} Jul 06 19:30:32.700912 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:32.700887 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722770-ft6x6" Jul 06 19:30:32.845428 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:32.845352 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6jw6\" (UniqueName: \"kubernetes.io/projected/177b0d59-f9b3-42f5-b2b4-60bf17264d46-kube-api-access-m6jw6\") pod \"177b0d59-f9b3-42f5-b2b4-60bf17264d46\" (UID: \"177b0d59-f9b3-42f5-b2b4-60bf17264d46\") " Jul 06 19:30:32.847411 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:32.847388 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177b0d59-f9b3-42f5-b2b4-60bf17264d46-kube-api-access-m6jw6" (OuterVolumeSpecName: "kube-api-access-m6jw6") pod "177b0d59-f9b3-42f5-b2b4-60bf17264d46" (UID: "177b0d59-f9b3-42f5-b2b4-60bf17264d46"). InnerVolumeSpecName "kube-api-access-m6jw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:30:32.946106 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:32.946079 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6jw6\" (UniqueName: \"kubernetes.io/projected/177b0d59-f9b3-42f5-b2b4-60bf17264d46-kube-api-access-m6jw6\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:30:33.580301 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:33.580264 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722770-ft6x6" event={"ID":"177b0d59-f9b3-42f5-b2b4-60bf17264d46","Type":"ContainerDied","Data":"99bf6257fc69956b7ec2006dc318b00b9183aa6f6123ff5611150f45d34f02dd"} Jul 06 19:30:33.580458 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:33.580316 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99bf6257fc69956b7ec2006dc318b00b9183aa6f6123ff5611150f45d34f02dd" Jul 06 19:30:33.580458 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:30:33.580282 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722770-ft6x6" Jul 06 19:31:39.256358 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:31:39.256330 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:31:39.258624 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:31:39.258601 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:35:25.273752 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:25.273719 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw"] Jul 06 19:35:25.275410 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:25.273924 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" podUID="fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9" containerName="manager" containerID="cri-o://20947f390aa27992c3d427b9782f29e4594ceeecafd8509b3f00d564bf7ed37b" gracePeriod=10 Jul 06 19:35:27.220300 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:27.220279 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" Jul 06 19:35:27.339354 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:27.339281 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9-extensions-socket-volume\") pod \"fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9\" (UID: \"fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9\") " Jul 06 19:35:27.339354 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:27.339324 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82gqf\" (UniqueName: \"kubernetes.io/projected/fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9-kube-api-access-82gqf\") pod \"fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9\" (UID: \"fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9\") " Jul 06 19:35:27.339630 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:27.339608 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9" (UID: "fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:35:27.341155 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:27.341131 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9-kube-api-access-82gqf" (OuterVolumeSpecName: "kube-api-access-82gqf") pod "fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9" (UID: "fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9"). InnerVolumeSpecName "kube-api-access-82gqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:35:27.440173 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:27.440150 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9-extensions-socket-volume\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:35:27.440173 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:27.440173 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82gqf\" (UniqueName: \"kubernetes.io/projected/fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9-kube-api-access-82gqf\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:35:27.897273 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:27.897248 2581 generic.go:356] "Generic (PLEG): container finished" podID="fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9" containerID="20947f390aa27992c3d427b9782f29e4594ceeecafd8509b3f00d564bf7ed37b" exitCode=0 Jul 06 19:35:27.897386 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:27.897302 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" event={"ID":"fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9","Type":"ContainerDied","Data":"20947f390aa27992c3d427b9782f29e4594ceeecafd8509b3f00d564bf7ed37b"} Jul 06 19:35:27.897386 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:27.897374 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" event={"ID":"fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9","Type":"ContainerDied","Data":"d611da6b1c27e674b20c640097bc485baaa62a40aa31fb2c1fe90ebef4057720"} Jul 06 19:35:27.897386 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:27.897304 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw" Jul 06 19:35:27.897500 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:27.897400 2581 scope.go:117] "RemoveContainer" containerID="20947f390aa27992c3d427b9782f29e4594ceeecafd8509b3f00d564bf7ed37b" Jul 06 19:35:27.909897 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:27.909876 2581 scope.go:117] "RemoveContainer" containerID="20947f390aa27992c3d427b9782f29e4594ceeecafd8509b3f00d564bf7ed37b" Jul 06 19:35:27.910194 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:35:27.910171 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20947f390aa27992c3d427b9782f29e4594ceeecafd8509b3f00d564bf7ed37b\": container with ID starting with 20947f390aa27992c3d427b9782f29e4594ceeecafd8509b3f00d564bf7ed37b not found: ID does not exist" containerID="20947f390aa27992c3d427b9782f29e4594ceeecafd8509b3f00d564bf7ed37b" Jul 06 19:35:27.910286 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:27.910205 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20947f390aa27992c3d427b9782f29e4594ceeecafd8509b3f00d564bf7ed37b"} err="failed to get container status \"20947f390aa27992c3d427b9782f29e4594ceeecafd8509b3f00d564bf7ed37b\": rpc error: code = NotFound desc = could not find container \"20947f390aa27992c3d427b9782f29e4594ceeecafd8509b3f00d564bf7ed37b\": container with ID starting with 20947f390aa27992c3d427b9782f29e4594ceeecafd8509b3f00d564bf7ed37b not found: ID does not exist" Jul 06 19:35:27.923086 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:27.923063 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw"] Jul 06 19:35:27.927998 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:27.927976 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l8mmw"] Jul 06 19:35:28.147730 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:35:28.147645 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9" path="/var/lib/kubelet/pods/fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9/volumes" Jul 06 19:36:26.468943 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.468907 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p57v5"] Jul 06 19:36:26.469396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.469202 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="177b0d59-f9b3-42f5-b2b4-60bf17264d46" containerName="cleanup" Jul 06 19:36:26.469396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.469213 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="177b0d59-f9b3-42f5-b2b4-60bf17264d46" containerName="cleanup" Jul 06 19:36:26.469396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.469224 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9" containerName="manager" Jul 06 19:36:26.469396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.469229 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9" containerName="manager" Jul 06 19:36:26.469396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.469288 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc1fbcf4-a90f-4fc4-bd1a-0a7ee60ff0c9" containerName="manager" Jul 06 19:36:26.469396 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.469296 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="177b0d59-f9b3-42f5-b2b4-60bf17264d46" containerName="cleanup" Jul 06 19:36:26.471314 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.471298 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p57v5" Jul 06 19:36:26.474370 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.474348 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:36:26.474370 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.474355 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z2nsb\"" Jul 06 19:36:26.474553 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.474378 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:36:26.481252 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.481227 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52591f3b-7bed-4f00-a64e-ae8f96294957-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-p57v5\" (UID: \"52591f3b-7bed-4f00-a64e-ae8f96294957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p57v5" Jul 06 19:36:26.481368 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.481292 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsll2\" (UniqueName: \"kubernetes.io/projected/52591f3b-7bed-4f00-a64e-ae8f96294957-kube-api-access-wsll2\") pod \"kuadrant-operator-controller-manager-86878c769f-p57v5\" (UID: \"52591f3b-7bed-4f00-a64e-ae8f96294957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p57v5" Jul 06 19:36:26.483292 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.483273 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p57v5"] Jul 06 19:36:26.581896 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.581874 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52591f3b-7bed-4f00-a64e-ae8f96294957-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-p57v5\" (UID: \"52591f3b-7bed-4f00-a64e-ae8f96294957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p57v5" Jul 06 19:36:26.581985 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.581913 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsll2\" (UniqueName: \"kubernetes.io/projected/52591f3b-7bed-4f00-a64e-ae8f96294957-kube-api-access-wsll2\") pod \"kuadrant-operator-controller-manager-86878c769f-p57v5\" (UID: \"52591f3b-7bed-4f00-a64e-ae8f96294957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p57v5" Jul 06 19:36:26.582193 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.582177 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52591f3b-7bed-4f00-a64e-ae8f96294957-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-p57v5\" (UID: \"52591f3b-7bed-4f00-a64e-ae8f96294957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p57v5" Jul 06 19:36:26.590649 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.590622 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsll2\" (UniqueName: \"kubernetes.io/projected/52591f3b-7bed-4f00-a64e-ae8f96294957-kube-api-access-wsll2\") pod \"kuadrant-operator-controller-manager-86878c769f-p57v5\" (UID: \"52591f3b-7bed-4f00-a64e-ae8f96294957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p57v5" Jul 06 19:36:26.781508 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.781450 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p57v5" Jul 06 19:36:26.916597 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:26.916577 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p57v5"] Jul 06 19:36:26.919303 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:36:26.919276 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52591f3b_7bed_4f00_a64e_ae8f96294957.slice/crio-c864f7d0d4295a3d02f943fa5957f3adfa02946d907a31ce14643d77543cea5b WatchSource:0}: Error finding container c864f7d0d4295a3d02f943fa5957f3adfa02946d907a31ce14643d77543cea5b: Status 404 returned error can't find the container with id c864f7d0d4295a3d02f943fa5957f3adfa02946d907a31ce14643d77543cea5b Jul 06 19:36:27.181741 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:27.181718 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p57v5" event={"ID":"52591f3b-7bed-4f00-a64e-ae8f96294957","Type":"ContainerStarted","Data":"9ef08cdf266b50aa5b6eb26bb023a9054c6e4de778310cd3c99305f30a783d05"} Jul 06 19:36:27.181868 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:27.181761 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p57v5" event={"ID":"52591f3b-7bed-4f00-a64e-ae8f96294957","Type":"ContainerStarted","Data":"c864f7d0d4295a3d02f943fa5957f3adfa02946d907a31ce14643d77543cea5b"} Jul 06 19:36:27.181868 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:27.181792 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p57v5" Jul 06 19:36:27.202270 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:27.202229 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p57v5" podStartSLOduration=1.202215541 podStartE2EDuration="1.202215541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:36:27.200529675 +0000 UTC m=+1488.416167204" watchObservedRunningTime="2026-07-06 19:36:27.202215541 +0000 UTC m=+1488.417853087" Jul 06 19:36:38.189103 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:38.189073 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-p57v5" Jul 06 19:36:39.286001 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:39.285974 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:36:39.289765 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:36:39.289745 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:41:39.313069 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:41:39.313045 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:41:39.316757 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:41:39.316735 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:42:58.495095 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:42:58.495063 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7"] Jul 06 19:42:58.497177 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:42:58.497161 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" Jul 06 19:42:58.500327 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:42:58.500308 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-9fb5e44b-serving-cert\"" Jul 06 19:42:58.540613 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:42:58.540589 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7"] Jul 06 19:42:58.606920 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:42:58.606896 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-maas-api-tls\") pod \"maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7\" (UID: \"dc817208-cc2e-4a22-8ccc-6ad285c2b63a\") " pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" Jul 06 19:42:58.607029 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:42:58.606928 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-openshift-service-ca\") pod \"maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7\" (UID: \"dc817208-cc2e-4a22-8ccc-6ad285c2b63a\") " pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" Jul 06 19:42:58.607029 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:42:58.606978 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4tpk\" (UniqueName: \"kubernetes.io/projected/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-kube-api-access-b4tpk\") pod \"maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7\" (UID: \"dc817208-cc2e-4a22-8ccc-6ad285c2b63a\") " pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" Jul 06 19:42:58.707983 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:42:58.707957 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-maas-api-tls\") pod \"maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7\" (UID: \"dc817208-cc2e-4a22-8ccc-6ad285c2b63a\") " pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" Jul 06 19:42:58.708107 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:42:58.707989 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-openshift-service-ca\") pod \"maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7\" (UID: \"dc817208-cc2e-4a22-8ccc-6ad285c2b63a\") " pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" Jul 06 19:42:58.708107 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:42:58.708032 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4tpk\" (UniqueName: \"kubernetes.io/projected/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-kube-api-access-b4tpk\") pod \"maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7\" (UID: \"dc817208-cc2e-4a22-8ccc-6ad285c2b63a\") " pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" Jul 06 19:42:58.708682 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:42:58.708643 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-openshift-service-ca\") pod \"maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7\" (UID: \"dc817208-cc2e-4a22-8ccc-6ad285c2b63a\") " pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" Jul 06 19:42:58.710340 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:42:58.710319 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-maas-api-tls\") pod \"maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7\" (UID: \"dc817208-cc2e-4a22-8ccc-6ad285c2b63a\") " pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" Jul 06 19:42:58.726624 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:42:58.726604 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4tpk\" (UniqueName: \"kubernetes.io/projected/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-kube-api-access-b4tpk\") pod \"maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7\" (UID: \"dc817208-cc2e-4a22-8ccc-6ad285c2b63a\") " pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" Jul 06 19:42:58.806540 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:42:58.806485 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" Jul 06 19:42:58.963291 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:42:58.963261 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7"] Jul 06 19:42:58.964048 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:42:58.964023 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc817208_cc2e_4a22_8ccc_6ad285c2b63a.slice/crio-73f4747e8aa5d5a3e83695b9edaada89472f5b3769d827e9c2e79e82fc51deda WatchSource:0}: Error finding container 73f4747e8aa5d5a3e83695b9edaada89472f5b3769d827e9c2e79e82fc51deda: Status 404 returned error can't find the container with id 73f4747e8aa5d5a3e83695b9edaada89472f5b3769d827e9c2e79e82fc51deda Jul 06 19:42:58.965485 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:42:58.965464 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:42:59.911559 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:42:59.911524 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" event={"ID":"dc817208-cc2e-4a22-8ccc-6ad285c2b63a","Type":"ContainerStarted","Data":"73f4747e8aa5d5a3e83695b9edaada89472f5b3769d827e9c2e79e82fc51deda"} Jul 06 19:43:00.917494 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:00.917411 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" event={"ID":"dc817208-cc2e-4a22-8ccc-6ad285c2b63a","Type":"ContainerStarted","Data":"69b576480aaf460549e56fb7d6f11c09394c263541ec438d299cdf9cf0e0ceb8"} Jul 06 19:43:00.917494 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:00.917479 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" Jul 06 19:43:00.979082 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:00.979040 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" podStartSLOduration=2.091801154 podStartE2EDuration="2.979025535s" totalImagesPullingTime="887.224381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:43:00.977377624 +0000 UTC m=+1882.193015153" watchObservedRunningTime="2026-07-06 19:43:00.979025535 +0000 UTC m=+1882.194663063" Jul 06 19:43:04.153984 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.153954 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7"] Jul 06 19:43:04.154530 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.154145 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" podUID="dc817208-cc2e-4a22-8ccc-6ad285c2b63a" containerName="maas-api" containerID="cri-o://69b576480aaf460549e56fb7d6f11c09394c263541ec438d299cdf9cf0e0ceb8" gracePeriod=30 Jul 06 19:43:04.159676 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.159630 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" Jul 06 19:43:04.389847 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.389820 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" Jul 06 19:43:04.449267 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.449238 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-maas-api-tls\") pod \"dc817208-cc2e-4a22-8ccc-6ad285c2b63a\" (UID: \"dc817208-cc2e-4a22-8ccc-6ad285c2b63a\") " Jul 06 19:43:04.449402 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.449327 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4tpk\" (UniqueName: \"kubernetes.io/projected/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-kube-api-access-b4tpk\") pod \"dc817208-cc2e-4a22-8ccc-6ad285c2b63a\" (UID: \"dc817208-cc2e-4a22-8ccc-6ad285c2b63a\") " Jul 06 19:43:04.449465 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.449415 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-openshift-service-ca\") pod \"dc817208-cc2e-4a22-8ccc-6ad285c2b63a\" (UID: \"dc817208-cc2e-4a22-8ccc-6ad285c2b63a\") " Jul 06 19:43:04.449828 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.449804 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc817208-cc2e-4a22-8ccc-6ad285c2b63a" (UID: "dc817208-cc2e-4a22-8ccc-6ad285c2b63a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:43:04.451190 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.451171 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc817208-cc2e-4a22-8ccc-6ad285c2b63a" (UID: "dc817208-cc2e-4a22-8ccc-6ad285c2b63a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:43:04.451278 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.451258 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-kube-api-access-b4tpk" (OuterVolumeSpecName: "kube-api-access-b4tpk") pod "dc817208-cc2e-4a22-8ccc-6ad285c2b63a" (UID: "dc817208-cc2e-4a22-8ccc-6ad285c2b63a"). InnerVolumeSpecName "kube-api-access-b4tpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:43:04.550391 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.550366 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-openshift-service-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:43:04.550391 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.550387 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-maas-api-tls\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:43:04.550510 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.550396 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4tpk\" (UniqueName: \"kubernetes.io/projected/dc817208-cc2e-4a22-8ccc-6ad285c2b63a-kube-api-access-b4tpk\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:43:04.937916 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.937885 2581 generic.go:356] "Generic (PLEG): container finished" podID="dc817208-cc2e-4a22-8ccc-6ad285c2b63a" containerID="69b576480aaf460549e56fb7d6f11c09394c263541ec438d299cdf9cf0e0ceb8" exitCode=0 Jul 06 19:43:04.938063 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.937929 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" Jul 06 19:43:04.938063 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.937975 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" event={"ID":"dc817208-cc2e-4a22-8ccc-6ad285c2b63a","Type":"ContainerDied","Data":"69b576480aaf460549e56fb7d6f11c09394c263541ec438d299cdf9cf0e0ceb8"} Jul 06 19:43:04.938177 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.938062 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7" event={"ID":"dc817208-cc2e-4a22-8ccc-6ad285c2b63a","Type":"ContainerDied","Data":"73f4747e8aa5d5a3e83695b9edaada89472f5b3769d827e9c2e79e82fc51deda"} Jul 06 19:43:04.938177 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.938094 2581 scope.go:117] "RemoveContainer" containerID="69b576480aaf460549e56fb7d6f11c09394c263541ec438d299cdf9cf0e0ceb8" Jul 06 19:43:04.947113 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.947105 2581 scope.go:117] "RemoveContainer" containerID="69b576480aaf460549e56fb7d6f11c09394c263541ec438d299cdf9cf0e0ceb8" Jul 06 19:43:04.947346 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:43:04.947328 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b576480aaf460549e56fb7d6f11c09394c263541ec438d299cdf9cf0e0ceb8\": container with ID starting with 69b576480aaf460549e56fb7d6f11c09394c263541ec438d299cdf9cf0e0ceb8 not found: ID does not exist" containerID="69b576480aaf460549e56fb7d6f11c09394c263541ec438d299cdf9cf0e0ceb8" Jul 06 19:43:04.947389 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.947352 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b576480aaf460549e56fb7d6f11c09394c263541ec438d299cdf9cf0e0ceb8"} err="failed to get container status \"69b576480aaf460549e56fb7d6f11c09394c263541ec438d299cdf9cf0e0ceb8\": rpc error: code = NotFound desc = could not find container \"69b576480aaf460549e56fb7d6f11c09394c263541ec438d299cdf9cf0e0ceb8\": container with ID starting with 69b576480aaf460549e56fb7d6f11c09394c263541ec438d299cdf9cf0e0ceb8 not found: ID does not exist" Jul 06 19:43:04.961112 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.961089 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7"] Jul 06 19:43:04.966128 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:04.966110 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-9fb5e44b-6445b4fb8f-pk7m7"] Jul 06 19:43:06.147895 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:06.147862 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc817208-cc2e-4a22-8ccc-6ad285c2b63a" path="/var/lib/kubelet/pods/dc817208-cc2e-4a22-8ccc-6ad285c2b63a/volumes" Jul 06 19:43:12.005972 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.005943 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld"] Jul 06 19:43:12.006314 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.006234 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc817208-cc2e-4a22-8ccc-6ad285c2b63a" containerName="maas-api" Jul 06 19:43:12.006314 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.006243 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc817208-cc2e-4a22-8ccc-6ad285c2b63a" containerName="maas-api" Jul 06 19:43:12.006314 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.006307 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc817208-cc2e-4a22-8ccc-6ad285c2b63a" containerName="maas-api" Jul 06 19:43:12.008990 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.008975 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" Jul 06 19:43:12.012366 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.012343 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-557aa455-serving-cert\"" Jul 06 19:43:12.038431 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.038403 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld"] Jul 06 19:43:12.105772 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.105752 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f501cbe4-6096-47a7-a92a-cdcf81896662-openshift-service-ca\") pod \"maas-api-e2e-ait-557aa455-5c9967758f-769ld\" (UID: \"f501cbe4-6096-47a7-a92a-cdcf81896662\") " pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" Jul 06 19:43:12.105865 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.105785 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f501cbe4-6096-47a7-a92a-cdcf81896662-maas-api-tls\") pod \"maas-api-e2e-ait-557aa455-5c9967758f-769ld\" (UID: \"f501cbe4-6096-47a7-a92a-cdcf81896662\") " pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" Jul 06 19:43:12.105865 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.105847 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw2jt\" (UniqueName: \"kubernetes.io/projected/f501cbe4-6096-47a7-a92a-cdcf81896662-kube-api-access-sw2jt\") pod \"maas-api-e2e-ait-557aa455-5c9967758f-769ld\" (UID: \"f501cbe4-6096-47a7-a92a-cdcf81896662\") " pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" Jul 06 19:43:12.206510 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.206484 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw2jt\" (UniqueName: \"kubernetes.io/projected/f501cbe4-6096-47a7-a92a-cdcf81896662-kube-api-access-sw2jt\") pod \"maas-api-e2e-ait-557aa455-5c9967758f-769ld\" (UID: \"f501cbe4-6096-47a7-a92a-cdcf81896662\") " pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" Jul 06 19:43:12.206618 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.206529 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f501cbe4-6096-47a7-a92a-cdcf81896662-openshift-service-ca\") pod \"maas-api-e2e-ait-557aa455-5c9967758f-769ld\" (UID: \"f501cbe4-6096-47a7-a92a-cdcf81896662\") " pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" Jul 06 19:43:12.206618 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.206556 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f501cbe4-6096-47a7-a92a-cdcf81896662-maas-api-tls\") pod \"maas-api-e2e-ait-557aa455-5c9967758f-769ld\" (UID: \"f501cbe4-6096-47a7-a92a-cdcf81896662\") " pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" Jul 06 19:43:12.206750 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:43:12.206647 2581 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-557aa455-serving-cert: secret "maas-api-e2e-ait-557aa455-serving-cert" not found Jul 06 19:43:12.206750 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:43:12.206714 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f501cbe4-6096-47a7-a92a-cdcf81896662-maas-api-tls podName:f501cbe4-6096-47a7-a92a-cdcf81896662 nodeName:}" failed. No retries permitted until 2026-07-06 19:43:12.706700356 +0000 UTC m=+1893.922337864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f501cbe4-6096-47a7-a92a-cdcf81896662-maas-api-tls") pod "maas-api-e2e-ait-557aa455-5c9967758f-769ld" (UID: "f501cbe4-6096-47a7-a92a-cdcf81896662") : secret "maas-api-e2e-ait-557aa455-serving-cert" not found Jul 06 19:43:12.207158 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.207135 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f501cbe4-6096-47a7-a92a-cdcf81896662-openshift-service-ca\") pod \"maas-api-e2e-ait-557aa455-5c9967758f-769ld\" (UID: \"f501cbe4-6096-47a7-a92a-cdcf81896662\") " pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" Jul 06 19:43:12.249740 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.249718 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw2jt\" (UniqueName: \"kubernetes.io/projected/f501cbe4-6096-47a7-a92a-cdcf81896662-kube-api-access-sw2jt\") pod \"maas-api-e2e-ait-557aa455-5c9967758f-769ld\" (UID: \"f501cbe4-6096-47a7-a92a-cdcf81896662\") " pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" Jul 06 19:43:12.709350 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.709322 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f501cbe4-6096-47a7-a92a-cdcf81896662-maas-api-tls\") pod \"maas-api-e2e-ait-557aa455-5c9967758f-769ld\" (UID: \"f501cbe4-6096-47a7-a92a-cdcf81896662\") " pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" Jul 06 19:43:12.711685 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.711641 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f501cbe4-6096-47a7-a92a-cdcf81896662-maas-api-tls\") pod \"maas-api-e2e-ait-557aa455-5c9967758f-769ld\" (UID: \"f501cbe4-6096-47a7-a92a-cdcf81896662\") " pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" Jul 06 19:43:12.918500 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:12.918474 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" Jul 06 19:43:13.087216 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:13.087193 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld"] Jul 06 19:43:13.089628 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:43:13.089587 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf501cbe4_6096_47a7_a92a_cdcf81896662.slice/crio-6798459f9d7d8ddf27f34b6c5208867378890cbe42680adca0ff61bd339be003 WatchSource:0}: Error finding container 6798459f9d7d8ddf27f34b6c5208867378890cbe42680adca0ff61bd339be003: Status 404 returned error can't find the container with id 6798459f9d7d8ddf27f34b6c5208867378890cbe42680adca0ff61bd339be003 Jul 06 19:43:13.982908 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:13.982874 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" event={"ID":"f501cbe4-6096-47a7-a92a-cdcf81896662","Type":"ContainerStarted","Data":"7fe1d18f10b44672ba27318cf015e45a37726b054d7ea4f7f71501e16e13691c"} Jul 06 19:43:13.983066 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:13.982926 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" event={"ID":"f501cbe4-6096-47a7-a92a-cdcf81896662","Type":"ContainerStarted","Data":"6798459f9d7d8ddf27f34b6c5208867378890cbe42680adca0ff61bd339be003"} Jul 06 19:43:13.983066 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:13.982996 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" Jul 06 19:43:14.028484 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:14.028444 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" podStartSLOduration=2.68180431 podStartE2EDuration="3.028429546s" totalImagesPullingTime="346.625236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:43:14.025818347 +0000 UTC m=+1895.241455889" watchObservedRunningTime="2026-07-06 19:43:14.028429546 +0000 UTC m=+1895.244067075" Jul 06 19:43:17.534568 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:17.534539 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld"] Jul 06 19:43:17.535107 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:17.534791 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" podUID="f501cbe4-6096-47a7-a92a-cdcf81896662" containerName="maas-api" containerID="cri-o://7fe1d18f10b44672ba27318cf015e45a37726b054d7ea4f7f71501e16e13691c" gracePeriod=30 Jul 06 19:43:17.774018 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:17.773996 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" Jul 06 19:43:17.847365 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:17.847316 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f501cbe4-6096-47a7-a92a-cdcf81896662-maas-api-tls\") pod \"f501cbe4-6096-47a7-a92a-cdcf81896662\" (UID: \"f501cbe4-6096-47a7-a92a-cdcf81896662\") " Jul 06 19:43:17.847365 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:17.847350 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw2jt\" (UniqueName: \"kubernetes.io/projected/f501cbe4-6096-47a7-a92a-cdcf81896662-kube-api-access-sw2jt\") pod \"f501cbe4-6096-47a7-a92a-cdcf81896662\" (UID: \"f501cbe4-6096-47a7-a92a-cdcf81896662\") " Jul 06 19:43:17.847515 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:17.847380 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f501cbe4-6096-47a7-a92a-cdcf81896662-openshift-service-ca\") pod \"f501cbe4-6096-47a7-a92a-cdcf81896662\" (UID: \"f501cbe4-6096-47a7-a92a-cdcf81896662\") " Jul 06 19:43:17.847736 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:17.847712 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f501cbe4-6096-47a7-a92a-cdcf81896662-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f501cbe4-6096-47a7-a92a-cdcf81896662" (UID: "f501cbe4-6096-47a7-a92a-cdcf81896662"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:43:17.849363 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:17.849331 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f501cbe4-6096-47a7-a92a-cdcf81896662-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f501cbe4-6096-47a7-a92a-cdcf81896662" (UID: "f501cbe4-6096-47a7-a92a-cdcf81896662"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:43:17.849452 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:17.849384 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f501cbe4-6096-47a7-a92a-cdcf81896662-kube-api-access-sw2jt" (OuterVolumeSpecName: "kube-api-access-sw2jt") pod "f501cbe4-6096-47a7-a92a-cdcf81896662" (UID: "f501cbe4-6096-47a7-a92a-cdcf81896662"). InnerVolumeSpecName "kube-api-access-sw2jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:43:17.948314 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:17.948293 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f501cbe4-6096-47a7-a92a-cdcf81896662-maas-api-tls\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:43:17.948314 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:17.948315 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw2jt\" (UniqueName: \"kubernetes.io/projected/f501cbe4-6096-47a7-a92a-cdcf81896662-kube-api-access-sw2jt\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:43:17.948424 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:17.948324 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f501cbe4-6096-47a7-a92a-cdcf81896662-openshift-service-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:43:18.002485 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:18.002457 2581 generic.go:356] "Generic (PLEG): container finished" podID="f501cbe4-6096-47a7-a92a-cdcf81896662" containerID="7fe1d18f10b44672ba27318cf015e45a37726b054d7ea4f7f71501e16e13691c" exitCode=0 Jul 06 19:43:18.002573 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:18.002499 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" Jul 06 19:43:18.002573 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:18.002539 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" event={"ID":"f501cbe4-6096-47a7-a92a-cdcf81896662","Type":"ContainerDied","Data":"7fe1d18f10b44672ba27318cf015e45a37726b054d7ea4f7f71501e16e13691c"} Jul 06 19:43:18.002672 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:18.002584 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld" event={"ID":"f501cbe4-6096-47a7-a92a-cdcf81896662","Type":"ContainerDied","Data":"6798459f9d7d8ddf27f34b6c5208867378890cbe42680adca0ff61bd339be003"} Jul 06 19:43:18.002672 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:18.002607 2581 scope.go:117] "RemoveContainer" containerID="7fe1d18f10b44672ba27318cf015e45a37726b054d7ea4f7f71501e16e13691c" Jul 06 19:43:18.011156 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:18.011141 2581 scope.go:117] "RemoveContainer" containerID="7fe1d18f10b44672ba27318cf015e45a37726b054d7ea4f7f71501e16e13691c" Jul 06 19:43:18.011406 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:43:18.011391 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe1d18f10b44672ba27318cf015e45a37726b054d7ea4f7f71501e16e13691c\": container with ID starting with 7fe1d18f10b44672ba27318cf015e45a37726b054d7ea4f7f71501e16e13691c not found: ID does not exist" containerID="7fe1d18f10b44672ba27318cf015e45a37726b054d7ea4f7f71501e16e13691c" Jul 06 19:43:18.011443 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:18.011415 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe1d18f10b44672ba27318cf015e45a37726b054d7ea4f7f71501e16e13691c"} err="failed to get container status \"7fe1d18f10b44672ba27318cf015e45a37726b054d7ea4f7f71501e16e13691c\": rpc error: code = NotFound desc = could not find container \"7fe1d18f10b44672ba27318cf015e45a37726b054d7ea4f7f71501e16e13691c\": container with ID starting with 7fe1d18f10b44672ba27318cf015e45a37726b054d7ea4f7f71501e16e13691c not found: ID does not exist" Jul 06 19:43:18.052563 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:18.052541 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld"] Jul 06 19:43:18.063310 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:18.063289 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-557aa455-5c9967758f-769ld"] Jul 06 19:43:18.146934 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:18.146884 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f501cbe4-6096-47a7-a92a-cdcf81896662" path="/var/lib/kubelet/pods/f501cbe4-6096-47a7-a92a-cdcf81896662/volumes" Jul 06 19:43:31.433569 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.433537 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b"] Jul 06 19:43:31.433976 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.433855 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f501cbe4-6096-47a7-a92a-cdcf81896662" containerName="maas-api" Jul 06 19:43:31.433976 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.433865 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f501cbe4-6096-47a7-a92a-cdcf81896662" containerName="maas-api" Jul 06 19:43:31.433976 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.433918 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f501cbe4-6096-47a7-a92a-cdcf81896662" containerName="maas-api" Jul 06 19:43:31.437888 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.437871 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" Jul 06 19:43:31.440579 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.440560 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-cb88f886-serving-cert\"" Jul 06 19:43:31.466297 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.466276 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b"] Jul 06 19:43:31.555154 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.555126 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-maas-api-tls\") pod \"maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b\" (UID: \"b19b95d0-d92f-4c96-a1a8-3e36e70ee405\") " pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" Jul 06 19:43:31.555279 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.555197 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns5bq\" (UniqueName: \"kubernetes.io/projected/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-kube-api-access-ns5bq\") pod \"maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b\" (UID: \"b19b95d0-d92f-4c96-a1a8-3e36e70ee405\") " pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" Jul 06 19:43:31.555279 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.555250 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-openshift-service-ca\") pod \"maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b\" (UID: \"b19b95d0-d92f-4c96-a1a8-3e36e70ee405\") " pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" Jul 06 19:43:31.656134 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.656107 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns5bq\" (UniqueName: \"kubernetes.io/projected/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-kube-api-access-ns5bq\") pod \"maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b\" (UID: \"b19b95d0-d92f-4c96-a1a8-3e36e70ee405\") " pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" Jul 06 19:43:31.656247 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.656169 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-openshift-service-ca\") pod \"maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b\" (UID: \"b19b95d0-d92f-4c96-a1a8-3e36e70ee405\") " pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" Jul 06 19:43:31.656247 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.656227 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-maas-api-tls\") pod \"maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b\" (UID: \"b19b95d0-d92f-4c96-a1a8-3e36e70ee405\") " pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" Jul 06 19:43:31.656746 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.656725 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-openshift-service-ca\") pod \"maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b\" (UID: \"b19b95d0-d92f-4c96-a1a8-3e36e70ee405\") " pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" Jul 06 19:43:31.658473 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.658451 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-maas-api-tls\") pod \"maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b\" (UID: \"b19b95d0-d92f-4c96-a1a8-3e36e70ee405\") " pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" Jul 06 19:43:31.678900 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.678881 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns5bq\" (UniqueName: \"kubernetes.io/projected/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-kube-api-access-ns5bq\") pod \"maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b\" (UID: \"b19b95d0-d92f-4c96-a1a8-3e36e70ee405\") " pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" Jul 06 19:43:31.747368 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.747318 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" Jul 06 19:43:31.887274 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:43:31.887247 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19b95d0_d92f_4c96_a1a8_3e36e70ee405.slice/crio-4db5ad8ba2feaaab7f818d7731deb6c556605a69d63cfafc785cd1322c71eaf2 WatchSource:0}: Error finding container 4db5ad8ba2feaaab7f818d7731deb6c556605a69d63cfafc785cd1322c71eaf2: Status 404 returned error can't find the container with id 4db5ad8ba2feaaab7f818d7731deb6c556605a69d63cfafc785cd1322c71eaf2 Jul 06 19:43:31.887575 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:31.887554 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b"] Jul 06 19:43:32.066534 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:32.066464 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" event={"ID":"b19b95d0-d92f-4c96-a1a8-3e36e70ee405","Type":"ContainerStarted","Data":"4db5ad8ba2feaaab7f818d7731deb6c556605a69d63cfafc785cd1322c71eaf2"} Jul 06 19:43:33.072942 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:33.072907 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" event={"ID":"b19b95d0-d92f-4c96-a1a8-3e36e70ee405","Type":"ContainerStarted","Data":"4b41e6a271601d0ba5d9d488461c69ba6d5512f3d5e9cc8fb012b9b9ccc0b0e0"} Jul 06 19:43:33.073380 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:33.073022 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" Jul 06 19:43:33.107827 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:33.107777 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" podStartSLOduration=1.7300647009999999 podStartE2EDuration="2.107764099s" totalImagesPullingTime="377.699398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:43:33.105289994 +0000 UTC m=+1914.320927522" watchObservedRunningTime="2026-07-06 19:43:33.107764099 +0000 UTC m=+1914.323401627" Jul 06 19:43:36.199094 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:36.199063 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b"] Jul 06 19:43:36.199471 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:36.199248 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" podUID="b19b95d0-d92f-4c96-a1a8-3e36e70ee405" containerName="maas-api" containerID="cri-o://4b41e6a271601d0ba5d9d488461c69ba6d5512f3d5e9cc8fb012b9b9ccc0b0e0" gracePeriod=30 Jul 06 19:43:36.438360 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:36.438339 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" Jul 06 19:43:36.591550 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:36.591496 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-maas-api-tls\") pod \"b19b95d0-d92f-4c96-a1a8-3e36e70ee405\" (UID: \"b19b95d0-d92f-4c96-a1a8-3e36e70ee405\") " Jul 06 19:43:36.591550 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:36.591539 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns5bq\" (UniqueName: \"kubernetes.io/projected/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-kube-api-access-ns5bq\") pod \"b19b95d0-d92f-4c96-a1a8-3e36e70ee405\" (UID: \"b19b95d0-d92f-4c96-a1a8-3e36e70ee405\") " Jul 06 19:43:36.591758 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:36.591565 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-openshift-service-ca\") pod \"b19b95d0-d92f-4c96-a1a8-3e36e70ee405\" (UID: \"b19b95d0-d92f-4c96-a1a8-3e36e70ee405\") " Jul 06 19:43:36.591937 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:36.591914 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b19b95d0-d92f-4c96-a1a8-3e36e70ee405" (UID: "b19b95d0-d92f-4c96-a1a8-3e36e70ee405"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:43:36.593477 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:36.593454 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b19b95d0-d92f-4c96-a1a8-3e36e70ee405" (UID: "b19b95d0-d92f-4c96-a1a8-3e36e70ee405"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:43:36.593533 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:36.593498 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-kube-api-access-ns5bq" (OuterVolumeSpecName: "kube-api-access-ns5bq") pod "b19b95d0-d92f-4c96-a1a8-3e36e70ee405" (UID: "b19b95d0-d92f-4c96-a1a8-3e36e70ee405"). InnerVolumeSpecName "kube-api-access-ns5bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:43:36.692476 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:36.692452 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-openshift-service-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:43:36.692476 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:36.692473 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-maas-api-tls\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:43:36.692592 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:36.692482 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns5bq\" (UniqueName: \"kubernetes.io/projected/b19b95d0-d92f-4c96-a1a8-3e36e70ee405-kube-api-access-ns5bq\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:43:37.090906 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:37.090882 2581 generic.go:356] "Generic (PLEG): container finished" podID="b19b95d0-d92f-4c96-a1a8-3e36e70ee405" containerID="4b41e6a271601d0ba5d9d488461c69ba6d5512f3d5e9cc8fb012b9b9ccc0b0e0" exitCode=0 Jul 06 19:43:37.091025 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:37.090923 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" Jul 06 19:43:37.091025 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:37.090971 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" event={"ID":"b19b95d0-d92f-4c96-a1a8-3e36e70ee405","Type":"ContainerDied","Data":"4b41e6a271601d0ba5d9d488461c69ba6d5512f3d5e9cc8fb012b9b9ccc0b0e0"} Jul 06 19:43:37.091025 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:37.091018 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b" event={"ID":"b19b95d0-d92f-4c96-a1a8-3e36e70ee405","Type":"ContainerDied","Data":"4db5ad8ba2feaaab7f818d7731deb6c556605a69d63cfafc785cd1322c71eaf2"} Jul 06 19:43:37.091121 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:37.091040 2581 scope.go:117] "RemoveContainer" containerID="4b41e6a271601d0ba5d9d488461c69ba6d5512f3d5e9cc8fb012b9b9ccc0b0e0" Jul 06 19:43:37.099501 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:37.099486 2581 scope.go:117] "RemoveContainer" containerID="4b41e6a271601d0ba5d9d488461c69ba6d5512f3d5e9cc8fb012b9b9ccc0b0e0" Jul 06 19:43:37.099733 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:43:37.099716 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b41e6a271601d0ba5d9d488461c69ba6d5512f3d5e9cc8fb012b9b9ccc0b0e0\": container with ID starting with 4b41e6a271601d0ba5d9d488461c69ba6d5512f3d5e9cc8fb012b9b9ccc0b0e0 not found: ID does not exist" containerID="4b41e6a271601d0ba5d9d488461c69ba6d5512f3d5e9cc8fb012b9b9ccc0b0e0" Jul 06 19:43:37.099787 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:37.099740 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b41e6a271601d0ba5d9d488461c69ba6d5512f3d5e9cc8fb012b9b9ccc0b0e0"} err="failed to get container status \"4b41e6a271601d0ba5d9d488461c69ba6d5512f3d5e9cc8fb012b9b9ccc0b0e0\": rpc error: code = NotFound desc = could not find container \"4b41e6a271601d0ba5d9d488461c69ba6d5512f3d5e9cc8fb012b9b9ccc0b0e0\": container with ID starting with 4b41e6a271601d0ba5d9d488461c69ba6d5512f3d5e9cc8fb012b9b9ccc0b0e0 not found: ID does not exist" Jul 06 19:43:37.111097 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:37.111076 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b"] Jul 06 19:43:37.117286 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:37.117264 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-cb88f886-5c75dfcb54-2xj9b"] Jul 06 19:43:38.146928 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:43:38.146898 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b19b95d0-d92f-4c96-a1a8-3e36e70ee405" path="/var/lib/kubelet/pods/b19b95d0-d92f-4c96-a1a8-3e36e70ee405/volumes" Jul 06 19:45:00.179361 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:00.179327 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-7678z"] Jul 06 19:45:00.179837 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:00.179617 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b19b95d0-d92f-4c96-a1a8-3e36e70ee405" containerName="maas-api" Jul 06 19:45:00.179837 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:00.179629 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b19b95d0-d92f-4c96-a1a8-3e36e70ee405" containerName="maas-api" Jul 06 19:45:00.179837 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:00.179718 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b19b95d0-d92f-4c96-a1a8-3e36e70ee405" containerName="maas-api" Jul 06 19:45:00.182512 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:00.182496 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-7678z" Jul 06 19:45:00.197500 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:00.197470 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-7678z"] Jul 06 19:45:00.278213 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:00.278186 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwjg4\" (UniqueName: \"kubernetes.io/projected/fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e-kube-api-access-lwjg4\") pod \"maas-api-key-cleanup-29722785-7678z\" (UID: \"fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e\") " pod="opendatahub/maas-api-key-cleanup-29722785-7678z" Jul 06 19:45:00.379102 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:00.379079 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwjg4\" (UniqueName: \"kubernetes.io/projected/fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e-kube-api-access-lwjg4\") pod \"maas-api-key-cleanup-29722785-7678z\" (UID: \"fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e\") " pod="opendatahub/maas-api-key-cleanup-29722785-7678z" Jul 06 19:45:00.390477 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:00.390454 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwjg4\" (UniqueName: \"kubernetes.io/projected/fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e-kube-api-access-lwjg4\") pod \"maas-api-key-cleanup-29722785-7678z\" (UID: \"fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e\") " pod="opendatahub/maas-api-key-cleanup-29722785-7678z" Jul 06 19:45:00.492472 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:00.492406 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-7678z" Jul 06 19:45:00.620560 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:00.620539 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722785-7678z"] Jul 06 19:45:00.622378 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:45:00.622351 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4b4b4c_4e5d_410f_8f46_dd79db8f7e0e.slice/crio-c0edcf3cb369305bc4f68d373bc5cd1940ef39f09bf2ded1a6312f50785cd63a WatchSource:0}: Error finding container c0edcf3cb369305bc4f68d373bc5cd1940ef39f09bf2ded1a6312f50785cd63a: Status 404 returned error can't find the container with id c0edcf3cb369305bc4f68d373bc5cd1940ef39f09bf2ded1a6312f50785cd63a Jul 06 19:45:01.486641 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:01.486612 2581 generic.go:356] "Generic (PLEG): container finished" podID="fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e" containerID="a336e1f53fd8c2f51b82b021d289dd6dac81f482a852d26dbe7660c4293743d5" exitCode=0 Jul 06 19:45:01.486942 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:01.486685 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-7678z" event={"ID":"fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e","Type":"ContainerDied","Data":"a336e1f53fd8c2f51b82b021d289dd6dac81f482a852d26dbe7660c4293743d5"} Jul 06 19:45:01.486942 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:01.486746 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-7678z" event={"ID":"fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e","Type":"ContainerStarted","Data":"c0edcf3cb369305bc4f68d373bc5cd1940ef39f09bf2ded1a6312f50785cd63a"} Jul 06 19:45:02.615697 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:02.615673 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-7678z" Jul 06 19:45:02.694540 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:02.694513 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwjg4\" (UniqueName: \"kubernetes.io/projected/fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e-kube-api-access-lwjg4\") pod \"fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e\" (UID: \"fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e\") " Jul 06 19:45:02.696398 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:02.696379 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e-kube-api-access-lwjg4" (OuterVolumeSpecName: "kube-api-access-lwjg4") pod "fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e" (UID: "fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e"). InnerVolumeSpecName "kube-api-access-lwjg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:45:02.795691 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:02.795625 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwjg4\" (UniqueName: \"kubernetes.io/projected/fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e-kube-api-access-lwjg4\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:45:03.497766 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:03.497730 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722785-7678z" event={"ID":"fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e","Type":"ContainerDied","Data":"c0edcf3cb369305bc4f68d373bc5cd1940ef39f09bf2ded1a6312f50785cd63a"} Jul 06 19:45:03.497943 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:03.497780 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0edcf3cb369305bc4f68d373bc5cd1940ef39f09bf2ded1a6312f50785cd63a" Jul 06 19:45:03.497943 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:03.497741 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722785-7678z" Jul 06 19:45:28.955247 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:28.955215 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk"] Jul 06 19:45:28.955641 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:28.955537 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e" containerName="cleanup" Jul 06 19:45:28.955641 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:28.955547 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e" containerName="cleanup" Jul 06 19:45:28.955641 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:28.955602 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e" containerName="cleanup" Jul 06 19:45:28.958819 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:28.958792 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" Jul 06 19:45:28.965002 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:28.964978 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-b3c8385f-serving-cert\"" Jul 06 19:45:28.999703 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:28.999682 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk"] Jul 06 19:45:29.086006 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:29.085981 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/637fd180-1360-4c24-9268-3d91f2dc1edd-maas-api-tls\") pod \"maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk\" (UID: \"637fd180-1360-4c24-9268-3d91f2dc1edd\") " pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" Jul 06 19:45:29.086125 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:29.086010 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/637fd180-1360-4c24-9268-3d91f2dc1edd-openshift-service-ca\") pod \"maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk\" (UID: \"637fd180-1360-4c24-9268-3d91f2dc1edd\") " pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" Jul 06 19:45:29.086125 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:29.086051 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfdzj\" (UniqueName: \"kubernetes.io/projected/637fd180-1360-4c24-9268-3d91f2dc1edd-kube-api-access-hfdzj\") pod \"maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk\" (UID: \"637fd180-1360-4c24-9268-3d91f2dc1edd\") " pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" Jul 06 19:45:29.186849 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:29.186823 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfdzj\" (UniqueName: \"kubernetes.io/projected/637fd180-1360-4c24-9268-3d91f2dc1edd-kube-api-access-hfdzj\") pod \"maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk\" (UID: \"637fd180-1360-4c24-9268-3d91f2dc1edd\") " pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" Jul 06 19:45:29.186941 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:29.186888 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/637fd180-1360-4c24-9268-3d91f2dc1edd-maas-api-tls\") pod \"maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk\" (UID: \"637fd180-1360-4c24-9268-3d91f2dc1edd\") " pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" Jul 06 19:45:29.186988 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:45:29.186969 2581 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-b3c8385f-serving-cert: secret "maas-api-e2e-mt-b3c8385f-serving-cert" not found Jul 06 19:45:29.187026 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:29.186990 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/637fd180-1360-4c24-9268-3d91f2dc1edd-openshift-service-ca\") pod \"maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk\" (UID: \"637fd180-1360-4c24-9268-3d91f2dc1edd\") " pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" Jul 06 19:45:29.187063 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:45:29.187029 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/637fd180-1360-4c24-9268-3d91f2dc1edd-maas-api-tls podName:637fd180-1360-4c24-9268-3d91f2dc1edd nodeName:}" failed. No retries permitted until 2026-07-06 19:45:29.687013801 +0000 UTC m=+2030.902651312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/637fd180-1360-4c24-9268-3d91f2dc1edd-maas-api-tls") pod "maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" (UID: "637fd180-1360-4c24-9268-3d91f2dc1edd") : secret "maas-api-e2e-mt-b3c8385f-serving-cert" not found Jul 06 19:45:29.187534 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:29.187517 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/637fd180-1360-4c24-9268-3d91f2dc1edd-openshift-service-ca\") pod \"maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk\" (UID: \"637fd180-1360-4c24-9268-3d91f2dc1edd\") " pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" Jul 06 19:45:29.226723 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:29.226674 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfdzj\" (UniqueName: \"kubernetes.io/projected/637fd180-1360-4c24-9268-3d91f2dc1edd-kube-api-access-hfdzj\") pod \"maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk\" (UID: \"637fd180-1360-4c24-9268-3d91f2dc1edd\") " pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" Jul 06 19:45:29.690566 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:29.690541 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/637fd180-1360-4c24-9268-3d91f2dc1edd-maas-api-tls\") pod \"maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk\" (UID: \"637fd180-1360-4c24-9268-3d91f2dc1edd\") " pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" Jul 06 19:45:29.692793 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:29.692770 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/637fd180-1360-4c24-9268-3d91f2dc1edd-maas-api-tls\") pod \"maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk\" (UID: \"637fd180-1360-4c24-9268-3d91f2dc1edd\") " pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" Jul 06 19:45:29.867990 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:29.867961 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" Jul 06 19:45:30.003666 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:30.003624 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk"] Jul 06 19:45:30.006207 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:45:30.006181 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637fd180_1360_4c24_9268_3d91f2dc1edd.slice/crio-46e62b44ec42339b7e7dd17c9af1c9481570cf08bb4ae80b975aaf309369102b WatchSource:0}: Error finding container 46e62b44ec42339b7e7dd17c9af1c9481570cf08bb4ae80b975aaf309369102b: Status 404 returned error can't find the container with id 46e62b44ec42339b7e7dd17c9af1c9481570cf08bb4ae80b975aaf309369102b Jul 06 19:45:30.620144 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:30.620114 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" event={"ID":"637fd180-1360-4c24-9268-3d91f2dc1edd","Type":"ContainerStarted","Data":"83b42174071e51fafcf7d784eadd829686e725a95daaa1ec48c4b3c8ca72933c"} Jul 06 19:45:30.620259 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:30.620162 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" event={"ID":"637fd180-1360-4c24-9268-3d91f2dc1edd","Type":"ContainerStarted","Data":"46e62b44ec42339b7e7dd17c9af1c9481570cf08bb4ae80b975aaf309369102b"} Jul 06 19:45:30.620259 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:30.620204 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" Jul 06 19:45:30.671511 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:30.671468 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" podStartSLOduration=2.196701246 podStartE2EDuration="2.671453212s" totalImagesPullingTime="474.751966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:45:30.666931428 +0000 UTC m=+2031.882568956" watchObservedRunningTime="2026-07-06 19:45:30.671453212 +0000 UTC m=+2031.887090740" Jul 06 19:45:36.630372 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:36.630341 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" Jul 06 19:45:42.293025 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.292983 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk"] Jul 06 19:45:42.293503 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.293241 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" podUID="637fd180-1360-4c24-9268-3d91f2dc1edd" containerName="maas-api" containerID="cri-o://83b42174071e51fafcf7d784eadd829686e725a95daaa1ec48c4b3c8ca72933c" gracePeriod=30 Jul 06 19:45:42.529685 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.529645 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" Jul 06 19:45:42.673766 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.673742 2581 generic.go:356] "Generic (PLEG): container finished" podID="637fd180-1360-4c24-9268-3d91f2dc1edd" containerID="83b42174071e51fafcf7d784eadd829686e725a95daaa1ec48c4b3c8ca72933c" exitCode=0 Jul 06 19:45:42.673886 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.673782 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" Jul 06 19:45:42.673886 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.673829 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" event={"ID":"637fd180-1360-4c24-9268-3d91f2dc1edd","Type":"ContainerDied","Data":"83b42174071e51fafcf7d784eadd829686e725a95daaa1ec48c4b3c8ca72933c"} Jul 06 19:45:42.673886 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.673878 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk" event={"ID":"637fd180-1360-4c24-9268-3d91f2dc1edd","Type":"ContainerDied","Data":"46e62b44ec42339b7e7dd17c9af1c9481570cf08bb4ae80b975aaf309369102b"} Jul 06 19:45:42.673983 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.673900 2581 scope.go:117] "RemoveContainer" containerID="83b42174071e51fafcf7d784eadd829686e725a95daaa1ec48c4b3c8ca72933c" Jul 06 19:45:42.684909 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.684892 2581 scope.go:117] "RemoveContainer" containerID="83b42174071e51fafcf7d784eadd829686e725a95daaa1ec48c4b3c8ca72933c" Jul 06 19:45:42.685032 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.685017 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfdzj\" (UniqueName: \"kubernetes.io/projected/637fd180-1360-4c24-9268-3d91f2dc1edd-kube-api-access-hfdzj\") pod \"637fd180-1360-4c24-9268-3d91f2dc1edd\" (UID: \"637fd180-1360-4c24-9268-3d91f2dc1edd\") " Jul 06 19:45:42.685088 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.685060 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/637fd180-1360-4c24-9268-3d91f2dc1edd-maas-api-tls\") pod \"637fd180-1360-4c24-9268-3d91f2dc1edd\" (UID: \"637fd180-1360-4c24-9268-3d91f2dc1edd\") " Jul 06 19:45:42.685088 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.685079 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/637fd180-1360-4c24-9268-3d91f2dc1edd-openshift-service-ca\") pod \"637fd180-1360-4c24-9268-3d91f2dc1edd\" (UID: \"637fd180-1360-4c24-9268-3d91f2dc1edd\") " Jul 06 19:45:42.685196 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:45:42.685173 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b42174071e51fafcf7d784eadd829686e725a95daaa1ec48c4b3c8ca72933c\": container with ID starting with 83b42174071e51fafcf7d784eadd829686e725a95daaa1ec48c4b3c8ca72933c not found: ID does not exist" containerID="83b42174071e51fafcf7d784eadd829686e725a95daaa1ec48c4b3c8ca72933c" Jul 06 19:45:42.685246 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.685207 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b42174071e51fafcf7d784eadd829686e725a95daaa1ec48c4b3c8ca72933c"} err="failed to get container status \"83b42174071e51fafcf7d784eadd829686e725a95daaa1ec48c4b3c8ca72933c\": rpc error: code = NotFound desc = could not find container \"83b42174071e51fafcf7d784eadd829686e725a95daaa1ec48c4b3c8ca72933c\": container with ID starting with 83b42174071e51fafcf7d784eadd829686e725a95daaa1ec48c4b3c8ca72933c not found: ID does not exist" Jul 06 19:45:42.685460 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.685440 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/637fd180-1360-4c24-9268-3d91f2dc1edd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "637fd180-1360-4c24-9268-3d91f2dc1edd" (UID: "637fd180-1360-4c24-9268-3d91f2dc1edd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:45:42.686914 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.686893 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/637fd180-1360-4c24-9268-3d91f2dc1edd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "637fd180-1360-4c24-9268-3d91f2dc1edd" (UID: "637fd180-1360-4c24-9268-3d91f2dc1edd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:45:42.687023 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.687006 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/637fd180-1360-4c24-9268-3d91f2dc1edd-kube-api-access-hfdzj" (OuterVolumeSpecName: "kube-api-access-hfdzj") pod "637fd180-1360-4c24-9268-3d91f2dc1edd" (UID: "637fd180-1360-4c24-9268-3d91f2dc1edd"). InnerVolumeSpecName "kube-api-access-hfdzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:45:42.786721 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.786704 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfdzj\" (UniqueName: \"kubernetes.io/projected/637fd180-1360-4c24-9268-3d91f2dc1edd-kube-api-access-hfdzj\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:45:42.786721 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.786722 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/637fd180-1360-4c24-9268-3d91f2dc1edd-maas-api-tls\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:45:42.786852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:42.786730 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/637fd180-1360-4c24-9268-3d91f2dc1edd-openshift-service-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:45:43.003935 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:43.003911 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk"] Jul 06 19:45:43.009941 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:43.009914 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-b3c8385f-7c785bbb8c-fjzkk"] Jul 06 19:45:44.146973 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:45:44.146941 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="637fd180-1360-4c24-9268-3d91f2dc1edd" path="/var/lib/kubelet/pods/637fd180-1360-4c24-9268-3d91f2dc1edd/volumes" Jul 06 19:46:14.009809 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.009780 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb"] Jul 06 19:46:14.010209 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.010086 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="637fd180-1360-4c24-9268-3d91f2dc1edd" containerName="maas-api" Jul 06 19:46:14.010209 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.010100 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="637fd180-1360-4c24-9268-3d91f2dc1edd" containerName="maas-api" Jul 06 19:46:14.010209 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.010184 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="637fd180-1360-4c24-9268-3d91f2dc1edd" containerName="maas-api" Jul 06 19:46:14.013182 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.013167 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" Jul 06 19:46:14.017554 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.017534 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-9349783c-serving-cert\"" Jul 06 19:46:14.030982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.030963 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb"] Jul 06 19:46:14.111581 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.111546 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f4d2c64-f7d3-4011-8852-3d07ef090e67-maas-api-tls\") pod \"maas-api-e2e-mt-9349783c-547b7dd489-zngrb\" (UID: \"0f4d2c64-f7d3-4011-8852-3d07ef090e67\") " pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" Jul 06 19:46:14.111728 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.111605 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f4d2c64-f7d3-4011-8852-3d07ef090e67-openshift-service-ca\") pod \"maas-api-e2e-mt-9349783c-547b7dd489-zngrb\" (UID: \"0f4d2c64-f7d3-4011-8852-3d07ef090e67\") " pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" Jul 06 19:46:14.111728 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.111631 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cbkc\" (UniqueName: \"kubernetes.io/projected/0f4d2c64-f7d3-4011-8852-3d07ef090e67-kube-api-access-7cbkc\") pod \"maas-api-e2e-mt-9349783c-547b7dd489-zngrb\" (UID: \"0f4d2c64-f7d3-4011-8852-3d07ef090e67\") " pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" Jul 06 19:46:14.212799 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.212770 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f4d2c64-f7d3-4011-8852-3d07ef090e67-maas-api-tls\") pod \"maas-api-e2e-mt-9349783c-547b7dd489-zngrb\" (UID: \"0f4d2c64-f7d3-4011-8852-3d07ef090e67\") " pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" Jul 06 19:46:14.212932 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.212817 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f4d2c64-f7d3-4011-8852-3d07ef090e67-openshift-service-ca\") pod \"maas-api-e2e-mt-9349783c-547b7dd489-zngrb\" (UID: \"0f4d2c64-f7d3-4011-8852-3d07ef090e67\") " pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" Jul 06 19:46:14.212932 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.212845 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cbkc\" (UniqueName: \"kubernetes.io/projected/0f4d2c64-f7d3-4011-8852-3d07ef090e67-kube-api-access-7cbkc\") pod \"maas-api-e2e-mt-9349783c-547b7dd489-zngrb\" (UID: \"0f4d2c64-f7d3-4011-8852-3d07ef090e67\") " pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" Jul 06 19:46:14.213366 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.213343 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f4d2c64-f7d3-4011-8852-3d07ef090e67-openshift-service-ca\") pod \"maas-api-e2e-mt-9349783c-547b7dd489-zngrb\" (UID: \"0f4d2c64-f7d3-4011-8852-3d07ef090e67\") " pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" Jul 06 19:46:14.215264 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.215239 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f4d2c64-f7d3-4011-8852-3d07ef090e67-maas-api-tls\") pod \"maas-api-e2e-mt-9349783c-547b7dd489-zngrb\" (UID: \"0f4d2c64-f7d3-4011-8852-3d07ef090e67\") " pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" Jul 06 19:46:14.221821 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.221800 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cbkc\" (UniqueName: \"kubernetes.io/projected/0f4d2c64-f7d3-4011-8852-3d07ef090e67-kube-api-access-7cbkc\") pod \"maas-api-e2e-mt-9349783c-547b7dd489-zngrb\" (UID: \"0f4d2c64-f7d3-4011-8852-3d07ef090e67\") " pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" Jul 06 19:46:14.322915 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.322867 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" Jul 06 19:46:14.484720 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.484692 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb"] Jul 06 19:46:14.486024 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:46:14.486003 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4d2c64_f7d3_4011_8852_3d07ef090e67.slice/crio-47d311c7546a4e6fd224eae6f613ff9ec16aed3084cd1362039a8f94ac44f9f3 WatchSource:0}: Error finding container 47d311c7546a4e6fd224eae6f613ff9ec16aed3084cd1362039a8f94ac44f9f3: Status 404 returned error can't find the container with id 47d311c7546a4e6fd224eae6f613ff9ec16aed3084cd1362039a8f94ac44f9f3 Jul 06 19:46:14.829985 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:14.829959 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" event={"ID":"0f4d2c64-f7d3-4011-8852-3d07ef090e67","Type":"ContainerStarted","Data":"47d311c7546a4e6fd224eae6f613ff9ec16aed3084cd1362039a8f94ac44f9f3"} Jul 06 19:46:15.836508 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:15.836468 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" event={"ID":"0f4d2c64-f7d3-4011-8852-3d07ef090e67","Type":"ContainerStarted","Data":"ea65f5a2523034638a0bc2bab1abe4458f2c0382aa7ce99f62f460ba0f073556"} Jul 06 19:46:15.836925 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:15.836541 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" Jul 06 19:46:15.853411 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:15.853360 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" podStartSLOduration=2.529627253 podStartE2EDuration="2.853341693s" totalImagesPullingTime="323.71444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:46:15.85274457 +0000 UTC m=+2077.068382098" watchObservedRunningTime="2026-07-06 19:46:15.853341693 +0000 UTC m=+2077.068979224" Jul 06 19:46:21.847326 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:21.847299 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" Jul 06 19:46:24.906155 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:24.906117 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94"] Jul 06 19:46:24.909597 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:24.909581 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:24.912800 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:24.912775 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9349783c\"/\"kube-root-ca.crt\"" Jul 06 19:46:24.912917 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:24.912800 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9349783c\"/\"openshift-service-ca.crt\"" Jul 06 19:46:24.913837 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:24.913816 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9349783c\"/\"e2e-lifecycle-model-9349783c-kserve-self-signed-certs\"" Jul 06 19:46:24.913930 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:24.913865 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9349783c\"/\"default-dockercfg-k8h7j\"" Jul 06 19:46:24.925662 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:24.925633 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94"] Jul 06 19:46:24.992942 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:24.992919 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-tmp-dir\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:24.993034 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:24.992957 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/445a31ef-3be8-4bf5-adc6-76b064aa522f-tls-certs\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:24.993034 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:24.992981 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-home\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:24.993034 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:24.993009 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-model-cache\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:24.993136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:24.993044 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhc8m\" (UniqueName: \"kubernetes.io/projected/445a31ef-3be8-4bf5-adc6-76b064aa522f-kube-api-access-nhc8m\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:24.993136 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:24.993073 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-dshm\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:25.093852 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:25.093823 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-dshm\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:25.093944 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:25.093870 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-tmp-dir\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:25.093944 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:25.093906 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/445a31ef-3be8-4bf5-adc6-76b064aa522f-tls-certs\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:25.094014 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:25.093940 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-home\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:25.094014 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:25.093964 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-model-cache\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:25.094081 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:25.094015 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhc8m\" (UniqueName: \"kubernetes.io/projected/445a31ef-3be8-4bf5-adc6-76b064aa522f-kube-api-access-nhc8m\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:25.094313 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:25.094294 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-tmp-dir\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:25.094393 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:25.094354 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-home\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:25.094393 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:25.094379 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-model-cache\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:25.096095 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:25.096074 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-dshm\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:25.096243 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:25.096224 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/445a31ef-3be8-4bf5-adc6-76b064aa522f-tls-certs\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:25.102433 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:25.102409 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhc8m\" (UniqueName: \"kubernetes.io/projected/445a31ef-3be8-4bf5-adc6-76b064aa522f-kube-api-access-nhc8m\") pod \"e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:25.218293 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:25.218272 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:25.377204 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:25.377168 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94"] Jul 06 19:46:25.378031 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:46:25.378002 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod445a31ef_3be8_4bf5_adc6_76b064aa522f.slice/crio-b2953a248509dfe67bcc9e9760cfb4fe90ad2aa075beb33ae18318b0fcf54afc WatchSource:0}: Error finding container b2953a248509dfe67bcc9e9760cfb4fe90ad2aa075beb33ae18318b0fcf54afc: Status 404 returned error can't find the container with id b2953a248509dfe67bcc9e9760cfb4fe90ad2aa075beb33ae18318b0fcf54afc Jul 06 19:46:25.891472 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:25.891418 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" event={"ID":"445a31ef-3be8-4bf5-adc6-76b064aa522f","Type":"ContainerStarted","Data":"baae9da20b6899418dc7a2ec4b6ff6d85c33eb951ef4bdaafc4b1bcdd557fbe2"} Jul 06 19:46:25.891472 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:25.891478 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" event={"ID":"445a31ef-3be8-4bf5-adc6-76b064aa522f","Type":"ContainerStarted","Data":"b2953a248509dfe67bcc9e9760cfb4fe90ad2aa075beb33ae18318b0fcf54afc"} Jul 06 19:46:25.915944 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:25.915903 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" podStartSLOduration=1.9158847570000002 podStartE2EDuration="1.915884757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:46:25.913992474 +0000 UTC m=+2087.129630002" watchObservedRunningTime="2026-07-06 19:46:25.915884757 +0000 UTC m=+2087.131522282" Jul 06 19:46:35.218595 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:35.218565 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:35.218984 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:35.218609 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:35.230414 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:35.230383 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:35.949283 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:35.949251 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:46:39.342061 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:39.342024 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:46:39.348444 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:39.348419 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:46:46.537457 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.537424 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb"] Jul 06 19:46:46.538038 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.537682 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" podUID="0f4d2c64-f7d3-4011-8852-3d07ef090e67" containerName="maas-api" containerID="cri-o://ea65f5a2523034638a0bc2bab1abe4458f2c0382aa7ce99f62f460ba0f073556" gracePeriod=30 Jul 06 19:46:46.783596 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.783576 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" Jul 06 19:46:46.863343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.863292 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f4d2c64-f7d3-4011-8852-3d07ef090e67-openshift-service-ca\") pod \"0f4d2c64-f7d3-4011-8852-3d07ef090e67\" (UID: \"0f4d2c64-f7d3-4011-8852-3d07ef090e67\") " Jul 06 19:46:46.863343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.863324 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cbkc\" (UniqueName: \"kubernetes.io/projected/0f4d2c64-f7d3-4011-8852-3d07ef090e67-kube-api-access-7cbkc\") pod \"0f4d2c64-f7d3-4011-8852-3d07ef090e67\" (UID: \"0f4d2c64-f7d3-4011-8852-3d07ef090e67\") " Jul 06 19:46:46.863471 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.863394 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f4d2c64-f7d3-4011-8852-3d07ef090e67-maas-api-tls\") pod \"0f4d2c64-f7d3-4011-8852-3d07ef090e67\" (UID: \"0f4d2c64-f7d3-4011-8852-3d07ef090e67\") " Jul 06 19:46:46.863604 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.863583 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4d2c64-f7d3-4011-8852-3d07ef090e67-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f4d2c64-f7d3-4011-8852-3d07ef090e67" (UID: "0f4d2c64-f7d3-4011-8852-3d07ef090e67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:46:46.865223 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.865203 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4d2c64-f7d3-4011-8852-3d07ef090e67-kube-api-access-7cbkc" (OuterVolumeSpecName: "kube-api-access-7cbkc") pod "0f4d2c64-f7d3-4011-8852-3d07ef090e67" (UID: "0f4d2c64-f7d3-4011-8852-3d07ef090e67"). InnerVolumeSpecName "kube-api-access-7cbkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:46:46.865336 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.865317 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f4d2c64-f7d3-4011-8852-3d07ef090e67-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f4d2c64-f7d3-4011-8852-3d07ef090e67" (UID: "0f4d2c64-f7d3-4011-8852-3d07ef090e67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:46:46.964226 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.964201 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f4d2c64-f7d3-4011-8852-3d07ef090e67-maas-api-tls\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:46:46.964226 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.964225 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f4d2c64-f7d3-4011-8852-3d07ef090e67-openshift-service-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:46:46.964353 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.964238 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cbkc\" (UniqueName: \"kubernetes.io/projected/0f4d2c64-f7d3-4011-8852-3d07ef090e67-kube-api-access-7cbkc\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:46:46.987251 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.987228 2581 generic.go:356] "Generic (PLEG): container finished" podID="0f4d2c64-f7d3-4011-8852-3d07ef090e67" containerID="ea65f5a2523034638a0bc2bab1abe4458f2c0382aa7ce99f62f460ba0f073556" exitCode=0 Jul 06 19:46:46.987333 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.987271 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" Jul 06 19:46:46.987333 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.987293 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" event={"ID":"0f4d2c64-f7d3-4011-8852-3d07ef090e67","Type":"ContainerDied","Data":"ea65f5a2523034638a0bc2bab1abe4458f2c0382aa7ce99f62f460ba0f073556"} Jul 06 19:46:46.987402 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.987332 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb" event={"ID":"0f4d2c64-f7d3-4011-8852-3d07ef090e67","Type":"ContainerDied","Data":"47d311c7546a4e6fd224eae6f613ff9ec16aed3084cd1362039a8f94ac44f9f3"} Jul 06 19:46:46.987402 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.987352 2581 scope.go:117] "RemoveContainer" containerID="ea65f5a2523034638a0bc2bab1abe4458f2c0382aa7ce99f62f460ba0f073556" Jul 06 19:46:46.995477 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.995460 2581 scope.go:117] "RemoveContainer" containerID="ea65f5a2523034638a0bc2bab1abe4458f2c0382aa7ce99f62f460ba0f073556" Jul 06 19:46:46.995714 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:46:46.995697 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea65f5a2523034638a0bc2bab1abe4458f2c0382aa7ce99f62f460ba0f073556\": container with ID starting with ea65f5a2523034638a0bc2bab1abe4458f2c0382aa7ce99f62f460ba0f073556 not found: ID does not exist" containerID="ea65f5a2523034638a0bc2bab1abe4458f2c0382aa7ce99f62f460ba0f073556" Jul 06 19:46:46.995763 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:46.995722 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea65f5a2523034638a0bc2bab1abe4458f2c0382aa7ce99f62f460ba0f073556"} err="failed to get container status \"ea65f5a2523034638a0bc2bab1abe4458f2c0382aa7ce99f62f460ba0f073556\": rpc error: code = NotFound desc = could not find container \"ea65f5a2523034638a0bc2bab1abe4458f2c0382aa7ce99f62f460ba0f073556\": container with ID starting with ea65f5a2523034638a0bc2bab1abe4458f2c0382aa7ce99f62f460ba0f073556 not found: ID does not exist" Jul 06 19:46:47.065625 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:47.065599 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb"] Jul 06 19:46:47.067831 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:47.067810 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-9349783c-547b7dd489-zngrb"] Jul 06 19:46:48.148958 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:48.148925 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f4d2c64-f7d3-4011-8852-3d07ef090e67" path="/var/lib/kubelet/pods/0f4d2c64-f7d3-4011-8852-3d07ef090e67/volumes" Jul 06 19:46:57.129131 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:46:57.129098 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94"] Jul 06 19:47:12.149034 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.148987 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" podUID="445a31ef-3be8-4bf5-adc6-76b064aa522f" containerName="main" containerID="cri-o://baae9da20b6899418dc7a2ec4b6ff6d85c33eb951ef4bdaafc4b1bcdd557fbe2" gracePeriod=45 Jul 06 19:47:12.399552 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.399495 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:47:12.546321 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.546293 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/445a31ef-3be8-4bf5-adc6-76b064aa522f-tls-certs\") pod \"445a31ef-3be8-4bf5-adc6-76b064aa522f\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " Jul 06 19:47:12.546491 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.546330 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-model-cache\") pod \"445a31ef-3be8-4bf5-adc6-76b064aa522f\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " Jul 06 19:47:12.546491 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.546387 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-tmp-dir\") pod \"445a31ef-3be8-4bf5-adc6-76b064aa522f\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " Jul 06 19:47:12.546491 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.546450 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-dshm\") pod \"445a31ef-3be8-4bf5-adc6-76b064aa522f\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " Jul 06 19:47:12.546687 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.546523 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-home\") pod \"445a31ef-3be8-4bf5-adc6-76b064aa522f\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " Jul 06 19:47:12.546687 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.546555 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhc8m\" (UniqueName: \"kubernetes.io/projected/445a31ef-3be8-4bf5-adc6-76b064aa522f-kube-api-access-nhc8m\") pod \"445a31ef-3be8-4bf5-adc6-76b064aa522f\" (UID: \"445a31ef-3be8-4bf5-adc6-76b064aa522f\") " Jul 06 19:47:12.546687 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.546611 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-model-cache" (OuterVolumeSpecName: "model-cache") pod "445a31ef-3be8-4bf5-adc6-76b064aa522f" (UID: "445a31ef-3be8-4bf5-adc6-76b064aa522f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:47:12.546687 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.546675 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "445a31ef-3be8-4bf5-adc6-76b064aa522f" (UID: "445a31ef-3be8-4bf5-adc6-76b064aa522f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:47:12.546866 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.546818 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-home" (OuterVolumeSpecName: "home") pod "445a31ef-3be8-4bf5-adc6-76b064aa522f" (UID: "445a31ef-3be8-4bf5-adc6-76b064aa522f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:47:12.546923 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.546909 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-model-cache\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:47:12.546963 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.546928 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-tmp-dir\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:47:12.546963 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.546936 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-home\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:47:12.548550 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.548518 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-dshm" (OuterVolumeSpecName: "dshm") pod "445a31ef-3be8-4bf5-adc6-76b064aa522f" (UID: "445a31ef-3be8-4bf5-adc6-76b064aa522f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:47:12.548670 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.548614 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/445a31ef-3be8-4bf5-adc6-76b064aa522f-kube-api-access-nhc8m" (OuterVolumeSpecName: "kube-api-access-nhc8m") pod "445a31ef-3be8-4bf5-adc6-76b064aa522f" (UID: "445a31ef-3be8-4bf5-adc6-76b064aa522f"). InnerVolumeSpecName "kube-api-access-nhc8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:47:12.548670 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.548620 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/445a31ef-3be8-4bf5-adc6-76b064aa522f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "445a31ef-3be8-4bf5-adc6-76b064aa522f" (UID: "445a31ef-3be8-4bf5-adc6-76b064aa522f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:47:12.647964 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.647916 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/445a31ef-3be8-4bf5-adc6-76b064aa522f-dshm\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:47:12.647964 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.647962 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhc8m\" (UniqueName: \"kubernetes.io/projected/445a31ef-3be8-4bf5-adc6-76b064aa522f-kube-api-access-nhc8m\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:47:12.647964 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:12.647973 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/445a31ef-3be8-4bf5-adc6-76b064aa522f-tls-certs\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:47:13.108029 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:13.107998 2581 generic.go:356] "Generic (PLEG): container finished" podID="445a31ef-3be8-4bf5-adc6-76b064aa522f" containerID="baae9da20b6899418dc7a2ec4b6ff6d85c33eb951ef4bdaafc4b1bcdd557fbe2" exitCode=0 Jul 06 19:47:13.108167 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:13.108044 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" Jul 06 19:47:13.108167 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:13.108085 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" event={"ID":"445a31ef-3be8-4bf5-adc6-76b064aa522f","Type":"ContainerDied","Data":"baae9da20b6899418dc7a2ec4b6ff6d85c33eb951ef4bdaafc4b1bcdd557fbe2"} Jul 06 19:47:13.108167 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:13.108134 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94" event={"ID":"445a31ef-3be8-4bf5-adc6-76b064aa522f","Type":"ContainerDied","Data":"b2953a248509dfe67bcc9e9760cfb4fe90ad2aa075beb33ae18318b0fcf54afc"} Jul 06 19:47:13.108167 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:13.108153 2581 scope.go:117] "RemoveContainer" containerID="baae9da20b6899418dc7a2ec4b6ff6d85c33eb951ef4bdaafc4b1bcdd557fbe2" Jul 06 19:47:13.117648 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:13.117629 2581 scope.go:117] "RemoveContainer" containerID="baae9da20b6899418dc7a2ec4b6ff6d85c33eb951ef4bdaafc4b1bcdd557fbe2" Jul 06 19:47:13.117986 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:47:13.117967 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baae9da20b6899418dc7a2ec4b6ff6d85c33eb951ef4bdaafc4b1bcdd557fbe2\": container with ID starting with baae9da20b6899418dc7a2ec4b6ff6d85c33eb951ef4bdaafc4b1bcdd557fbe2 not found: ID does not exist" containerID="baae9da20b6899418dc7a2ec4b6ff6d85c33eb951ef4bdaafc4b1bcdd557fbe2" Jul 06 19:47:13.118062 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:13.117994 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baae9da20b6899418dc7a2ec4b6ff6d85c33eb951ef4bdaafc4b1bcdd557fbe2"} err="failed to get container status \"baae9da20b6899418dc7a2ec4b6ff6d85c33eb951ef4bdaafc4b1bcdd557fbe2\": rpc error: code = NotFound desc = could not find container \"baae9da20b6899418dc7a2ec4b6ff6d85c33eb951ef4bdaafc4b1bcdd557fbe2\": container with ID starting with baae9da20b6899418dc7a2ec4b6ff6d85c33eb951ef4bdaafc4b1bcdd557fbe2 not found: ID does not exist" Jul 06 19:47:13.131431 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:13.131405 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94"] Jul 06 19:47:13.135154 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:13.135133 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9349783c/e2e-lifecycle-model-9349783c-kserve-667cd649c5-8nk94"] Jul 06 19:47:14.147633 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:47:14.147598 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="445a31ef-3be8-4bf5-adc6-76b064aa522f" path="/var/lib/kubelet/pods/445a31ef-3be8-4bf5-adc6-76b064aa522f/volumes" Jul 06 19:49:00.208728 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.208700 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s"] Jul 06 19:49:00.209168 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.208983 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f4d2c64-f7d3-4011-8852-3d07ef090e67" containerName="maas-api" Jul 06 19:49:00.209168 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.208993 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f4d2c64-f7d3-4011-8852-3d07ef090e67" containerName="maas-api" Jul 06 19:49:00.209168 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.209011 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="445a31ef-3be8-4bf5-adc6-76b064aa522f" containerName="main" Jul 06 19:49:00.209168 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.209016 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="445a31ef-3be8-4bf5-adc6-76b064aa522f" containerName="main" Jul 06 19:49:00.209168 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.209061 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f4d2c64-f7d3-4011-8852-3d07ef090e67" containerName="maas-api" Jul 06 19:49:00.209168 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.209068 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="445a31ef-3be8-4bf5-adc6-76b064aa522f" containerName="main" Jul 06 19:49:00.211978 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.211962 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" Jul 06 19:49:00.215255 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.215214 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-d98408-serving-cert\"" Jul 06 19:49:00.226167 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.226142 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s"] Jul 06 19:49:00.320709 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.320681 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75f5108-fb30-4d89-8480-d9e97126ef33-maas-api-tls\") pod \"maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s\" (UID: \"b75f5108-fb30-4d89-8480-d9e97126ef33\") " pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" Jul 06 19:49:00.320838 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.320718 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvgj9\" (UniqueName: \"kubernetes.io/projected/b75f5108-fb30-4d89-8480-d9e97126ef33-kube-api-access-dvgj9\") pod \"maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s\" (UID: \"b75f5108-fb30-4d89-8480-d9e97126ef33\") " pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" Jul 06 19:49:00.320838 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.320753 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75f5108-fb30-4d89-8480-d9e97126ef33-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s\" (UID: \"b75f5108-fb30-4d89-8480-d9e97126ef33\") " pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" Jul 06 19:49:00.421582 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.421556 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvgj9\" (UniqueName: \"kubernetes.io/projected/b75f5108-fb30-4d89-8480-d9e97126ef33-kube-api-access-dvgj9\") pod \"maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s\" (UID: \"b75f5108-fb30-4d89-8480-d9e97126ef33\") " pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" Jul 06 19:49:00.421774 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.421595 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75f5108-fb30-4d89-8480-d9e97126ef33-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s\" (UID: \"b75f5108-fb30-4d89-8480-d9e97126ef33\") " pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" Jul 06 19:49:00.421774 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.421649 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75f5108-fb30-4d89-8480-d9e97126ef33-maas-api-tls\") pod \"maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s\" (UID: \"b75f5108-fb30-4d89-8480-d9e97126ef33\") " pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" Jul 06 19:49:00.422336 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.422319 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75f5108-fb30-4d89-8480-d9e97126ef33-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s\" (UID: \"b75f5108-fb30-4d89-8480-d9e97126ef33\") " pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" Jul 06 19:49:00.424086 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.424065 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75f5108-fb30-4d89-8480-d9e97126ef33-maas-api-tls\") pod \"maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s\" (UID: \"b75f5108-fb30-4d89-8480-d9e97126ef33\") " pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" Jul 06 19:49:00.468748 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.468699 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvgj9\" (UniqueName: \"kubernetes.io/projected/b75f5108-fb30-4d89-8480-d9e97126ef33-kube-api-access-dvgj9\") pod \"maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s\" (UID: \"b75f5108-fb30-4d89-8480-d9e97126ef33\") " pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" Jul 06 19:49:00.521466 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.521444 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" Jul 06 19:49:00.698853 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.698820 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s"] Jul 06 19:49:00.701802 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:49:00.701777 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75f5108_fb30_4d89_8480_d9e97126ef33.slice/crio-9cf169ed52c668f69ead62cff0fd75a491e68d0146bff1812f8c82d33dd49bb5 WatchSource:0}: Error finding container 9cf169ed52c668f69ead62cff0fd75a491e68d0146bff1812f8c82d33dd49bb5: Status 404 returned error can't find the container with id 9cf169ed52c668f69ead62cff0fd75a491e68d0146bff1812f8c82d33dd49bb5 Jul 06 19:49:00.703119 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:00.703102 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:49:01.613720 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:01.613679 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" event={"ID":"b75f5108-fb30-4d89-8480-d9e97126ef33","Type":"ContainerStarted","Data":"addf6abfb30367ca04e3a40cf5275f4d832e8ba4c5826cdd36de397177c1887c"} Jul 06 19:49:01.614117 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:01.613831 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" Jul 06 19:49:01.614117 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:01.613879 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" event={"ID":"b75f5108-fb30-4d89-8480-d9e97126ef33","Type":"ContainerStarted","Data":"9cf169ed52c668f69ead62cff0fd75a491e68d0146bff1812f8c82d33dd49bb5"} Jul 06 19:49:01.654353 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:01.654309 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" podStartSLOduration=1.250923204 podStartE2EDuration="1.654295387s" totalImagesPullingTime="403.372183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:49:01.653493945 +0000 UTC m=+2242.869131485" watchObservedRunningTime="2026-07-06 19:49:01.654295387 +0000 UTC m=+2242.869932915" Jul 06 19:49:07.623525 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:07.623490 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" Jul 06 19:49:09.132941 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:09.132909 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz"] Jul 06 19:49:09.136213 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:09.136198 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" Jul 06 19:49:09.139110 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:09.139089 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-f1ee26-serving-cert\"" Jul 06 19:49:09.191993 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:09.191973 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acdbafa9-823e-4f94-af55-e8abca8e0c8f-maas-api-tls\") pod \"maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz\" (UID: \"acdbafa9-823e-4f94-af55-e8abca8e0c8f\") " pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" Jul 06 19:49:09.192089 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:09.192007 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzxmv\" (UniqueName: \"kubernetes.io/projected/acdbafa9-823e-4f94-af55-e8abca8e0c8f-kube-api-access-kzxmv\") pod \"maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz\" (UID: \"acdbafa9-823e-4f94-af55-e8abca8e0c8f\") " pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" Jul 06 19:49:09.192089 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:09.192039 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdbafa9-823e-4f94-af55-e8abca8e0c8f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz\" (UID: \"acdbafa9-823e-4f94-af55-e8abca8e0c8f\") " pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" Jul 06 19:49:09.226935 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:09.226910 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz"] Jul 06 19:49:09.292570 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:09.292543 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acdbafa9-823e-4f94-af55-e8abca8e0c8f-maas-api-tls\") pod \"maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz\" (UID: \"acdbafa9-823e-4f94-af55-e8abca8e0c8f\") " pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" Jul 06 19:49:09.292725 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:09.292575 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzxmv\" (UniqueName: \"kubernetes.io/projected/acdbafa9-823e-4f94-af55-e8abca8e0c8f-kube-api-access-kzxmv\") pod \"maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz\" (UID: \"acdbafa9-823e-4f94-af55-e8abca8e0c8f\") " pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" Jul 06 19:49:09.292725 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:09.292600 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdbafa9-823e-4f94-af55-e8abca8e0c8f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz\" (UID: \"acdbafa9-823e-4f94-af55-e8abca8e0c8f\") " pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" Jul 06 19:49:09.292725 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:49:09.292698 2581 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-f1ee26-serving-cert: secret "maas-api-e2e-inf-b-f1ee26-serving-cert" not found Jul 06 19:49:09.292920 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:49:09.292768 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acdbafa9-823e-4f94-af55-e8abca8e0c8f-maas-api-tls podName:acdbafa9-823e-4f94-af55-e8abca8e0c8f nodeName:}" failed. No retries permitted until 2026-07-06 19:49:09.792751887 +0000 UTC m=+2251.008389399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/acdbafa9-823e-4f94-af55-e8abca8e0c8f-maas-api-tls") pod "maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" (UID: "acdbafa9-823e-4f94-af55-e8abca8e0c8f") : secret "maas-api-e2e-inf-b-f1ee26-serving-cert" not found Jul 06 19:49:09.293200 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:09.293181 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdbafa9-823e-4f94-af55-e8abca8e0c8f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz\" (UID: \"acdbafa9-823e-4f94-af55-e8abca8e0c8f\") " pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" Jul 06 19:49:09.308243 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:09.308221 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzxmv\" (UniqueName: \"kubernetes.io/projected/acdbafa9-823e-4f94-af55-e8abca8e0c8f-kube-api-access-kzxmv\") pod \"maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz\" (UID: \"acdbafa9-823e-4f94-af55-e8abca8e0c8f\") " pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" Jul 06 19:49:09.797803 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:09.797772 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acdbafa9-823e-4f94-af55-e8abca8e0c8f-maas-api-tls\") pod \"maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz\" (UID: \"acdbafa9-823e-4f94-af55-e8abca8e0c8f\") " pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" Jul 06 19:49:09.800001 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:09.799983 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acdbafa9-823e-4f94-af55-e8abca8e0c8f-maas-api-tls\") pod \"maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz\" (UID: \"acdbafa9-823e-4f94-af55-e8abca8e0c8f\") " pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" Jul 06 19:49:10.047414 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:10.047385 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" Jul 06 19:49:10.202239 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:10.202212 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz"] Jul 06 19:49:10.203134 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:49:10.203111 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacdbafa9_823e_4f94_af55_e8abca8e0c8f.slice/crio-4a23c89783c62a9d9b190d1f70bab89d5c03ee44aed322aa17e342d43f0b5317 WatchSource:0}: Error finding container 4a23c89783c62a9d9b190d1f70bab89d5c03ee44aed322aa17e342d43f0b5317: Status 404 returned error can't find the container with id 4a23c89783c62a9d9b190d1f70bab89d5c03ee44aed322aa17e342d43f0b5317 Jul 06 19:49:10.655126 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:10.655102 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" event={"ID":"acdbafa9-823e-4f94-af55-e8abca8e0c8f","Type":"ContainerStarted","Data":"4a23c89783c62a9d9b190d1f70bab89d5c03ee44aed322aa17e342d43f0b5317"} Jul 06 19:49:11.661043 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:11.661010 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" event={"ID":"acdbafa9-823e-4f94-af55-e8abca8e0c8f","Type":"ContainerStarted","Data":"133af0c98bf39a70e60dd986f0ce53c45cddaf6e07212dfd1e945573f1c20be2"} Jul 06 19:49:11.661393 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:11.661139 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" Jul 06 19:49:11.694128 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:11.694089 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" podStartSLOduration=2.251814287 podStartE2EDuration="2.694076142s" totalImagesPullingTime="442.261855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:49:11.693333039 +0000 UTC m=+2252.908970578" watchObservedRunningTime="2026-07-06 19:49:11.694076142 +0000 UTC m=+2252.909713670" Jul 06 19:49:17.669881 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:17.669848 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" Jul 06 19:49:25.451973 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.451939 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds"] Jul 06 19:49:25.455153 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.455129 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.457781 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.457760 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d98408\"/\"openshift-service-ca.crt\"" Jul 06 19:49:25.458696 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.458676 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d98408\"/\"kube-root-ca.crt\"" Jul 06 19:49:25.458805 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.458710 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d98408\"/\"test-model-d98408-kserve-self-signed-certs\"" Jul 06 19:49:25.458805 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.458751 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d98408\"/\"default-dockercfg-xhbwd\"" Jul 06 19:49:25.479139 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.479109 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds"] Jul 06 19:49:25.625167 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.625136 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-tmp-dir\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.625317 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.625198 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-model-cache\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.625317 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.625263 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-dshm\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.625317 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.625307 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-home\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.625438 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.625326 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-tls-certs\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.625438 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.625387 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx7sq\" (UniqueName: \"kubernetes.io/projected/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-kube-api-access-bx7sq\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.726289 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.726230 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-model-cache\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.726289 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.726270 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-dshm\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.726433 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.726292 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-home\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.726433 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.726308 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-tls-certs\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.726433 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.726333 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx7sq\" (UniqueName: \"kubernetes.io/projected/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-kube-api-access-bx7sq\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.726433 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.726363 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-tmp-dir\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.726764 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.726737 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-model-cache\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.726920 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.726794 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-tmp-dir\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.726920 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.726847 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-home\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.728773 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.728755 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-dshm\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.729020 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.729005 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-tls-certs\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.746254 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.746234 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx7sq\" (UniqueName: \"kubernetes.io/projected/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-kube-api-access-bx7sq\") pod \"test-model-d98408-kserve-7b49b5f66c-s6lds\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.764252 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.764234 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:25.904266 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:25.904209 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds"] Jul 06 19:49:25.906477 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:49:25.906436 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a9ac327_1c5d_49ce_962c_73ba3d30f53d.slice/crio-c71daeb40b15b2f3cd339aad0a976a2f8e583267a510c47ddbc2b937bd6dcf2e WatchSource:0}: Error finding container c71daeb40b15b2f3cd339aad0a976a2f8e583267a510c47ddbc2b937bd6dcf2e: Status 404 returned error can't find the container with id c71daeb40b15b2f3cd339aad0a976a2f8e583267a510c47ddbc2b937bd6dcf2e Jul 06 19:49:26.729794 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:26.729765 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" event={"ID":"1a9ac327-1c5d-49ce-962c-73ba3d30f53d","Type":"ContainerStarted","Data":"053a3222d5f651b472acfec11e45b5330c1460b3a7ce2cf9e455101712a2e8ef"} Jul 06 19:49:26.730125 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:26.729805 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" event={"ID":"1a9ac327-1c5d-49ce-962c-73ba3d30f53d","Type":"ContainerStarted","Data":"c71daeb40b15b2f3cd339aad0a976a2f8e583267a510c47ddbc2b937bd6dcf2e"} Jul 06 19:49:26.749781 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:26.749738 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" podStartSLOduration=1.7497238309999998 podStartE2EDuration="1.749723831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:49:26.748591418 +0000 UTC m=+2267.964228946" watchObservedRunningTime="2026-07-06 19:49:26.749723831 +0000 UTC m=+2267.965361360" Jul 06 19:49:35.764629 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:35.764601 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:35.765135 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:35.764644 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:35.776647 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:35.776625 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:49:35.787461 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:49:35.787436 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:50:12.227405 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:12.227376 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds"] Jul 06 19:50:14.068494 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.068462 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s"] Jul 06 19:50:14.069000 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.068693 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" podUID="b75f5108-fb30-4d89-8480-d9e97126ef33" containerName="maas-api" containerID="cri-o://addf6abfb30367ca04e3a40cf5275f4d832e8ba4c5826cdd36de397177c1887c" gracePeriod=30 Jul 06 19:50:14.313483 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.313461 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" Jul 06 19:50:14.391233 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.391173 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75f5108-fb30-4d89-8480-d9e97126ef33-openshift-service-ca\") pod \"b75f5108-fb30-4d89-8480-d9e97126ef33\" (UID: \"b75f5108-fb30-4d89-8480-d9e97126ef33\") " Jul 06 19:50:14.391233 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.391206 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvgj9\" (UniqueName: \"kubernetes.io/projected/b75f5108-fb30-4d89-8480-d9e97126ef33-kube-api-access-dvgj9\") pod \"b75f5108-fb30-4d89-8480-d9e97126ef33\" (UID: \"b75f5108-fb30-4d89-8480-d9e97126ef33\") " Jul 06 19:50:14.391426 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.391266 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75f5108-fb30-4d89-8480-d9e97126ef33-maas-api-tls\") pod \"b75f5108-fb30-4d89-8480-d9e97126ef33\" (UID: \"b75f5108-fb30-4d89-8480-d9e97126ef33\") " Jul 06 19:50:14.391564 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.391541 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75f5108-fb30-4d89-8480-d9e97126ef33-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b75f5108-fb30-4d89-8480-d9e97126ef33" (UID: "b75f5108-fb30-4d89-8480-d9e97126ef33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:50:14.393283 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.393250 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b75f5108-fb30-4d89-8480-d9e97126ef33-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b75f5108-fb30-4d89-8480-d9e97126ef33" (UID: "b75f5108-fb30-4d89-8480-d9e97126ef33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:50:14.393375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.393263 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75f5108-fb30-4d89-8480-d9e97126ef33-kube-api-access-dvgj9" (OuterVolumeSpecName: "kube-api-access-dvgj9") pod "b75f5108-fb30-4d89-8480-d9e97126ef33" (UID: "b75f5108-fb30-4d89-8480-d9e97126ef33"). InnerVolumeSpecName "kube-api-access-dvgj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:50:14.492790 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.492769 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75f5108-fb30-4d89-8480-d9e97126ef33-openshift-service-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:50:14.492790 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.492788 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvgj9\" (UniqueName: \"kubernetes.io/projected/b75f5108-fb30-4d89-8480-d9e97126ef33-kube-api-access-dvgj9\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:50:14.492917 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.492799 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75f5108-fb30-4d89-8480-d9e97126ef33-maas-api-tls\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:50:14.953918 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.953888 2581 generic.go:356] "Generic (PLEG): container finished" podID="b75f5108-fb30-4d89-8480-d9e97126ef33" containerID="addf6abfb30367ca04e3a40cf5275f4d832e8ba4c5826cdd36de397177c1887c" exitCode=0 Jul 06 19:50:14.954081 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.953931 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" Jul 06 19:50:14.954081 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.953968 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" event={"ID":"b75f5108-fb30-4d89-8480-d9e97126ef33","Type":"ContainerDied","Data":"addf6abfb30367ca04e3a40cf5275f4d832e8ba4c5826cdd36de397177c1887c"} Jul 06 19:50:14.954081 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.954017 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s" event={"ID":"b75f5108-fb30-4d89-8480-d9e97126ef33","Type":"ContainerDied","Data":"9cf169ed52c668f69ead62cff0fd75a491e68d0146bff1812f8c82d33dd49bb5"} Jul 06 19:50:14.954081 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.954038 2581 scope.go:117] "RemoveContainer" containerID="addf6abfb30367ca04e3a40cf5275f4d832e8ba4c5826cdd36de397177c1887c" Jul 06 19:50:14.963091 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.963077 2581 scope.go:117] "RemoveContainer" containerID="addf6abfb30367ca04e3a40cf5275f4d832e8ba4c5826cdd36de397177c1887c" Jul 06 19:50:14.963309 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:50:14.963291 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addf6abfb30367ca04e3a40cf5275f4d832e8ba4c5826cdd36de397177c1887c\": container with ID starting with addf6abfb30367ca04e3a40cf5275f4d832e8ba4c5826cdd36de397177c1887c not found: ID does not exist" containerID="addf6abfb30367ca04e3a40cf5275f4d832e8ba4c5826cdd36de397177c1887c" Jul 06 19:50:14.963354 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.963317 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addf6abfb30367ca04e3a40cf5275f4d832e8ba4c5826cdd36de397177c1887c"} err="failed to get container status \"addf6abfb30367ca04e3a40cf5275f4d832e8ba4c5826cdd36de397177c1887c\": rpc error: code = NotFound desc = could not find container \"addf6abfb30367ca04e3a40cf5275f4d832e8ba4c5826cdd36de397177c1887c\": container with ID starting with addf6abfb30367ca04e3a40cf5275f4d832e8ba4c5826cdd36de397177c1887c not found: ID does not exist" Jul 06 19:50:14.975058 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.975035 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s"] Jul 06 19:50:14.978895 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:14.978876 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-d98408-7c7bd48b8-qtn2s"] Jul 06 19:50:16.147581 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:16.147544 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b75f5108-fb30-4d89-8480-d9e97126ef33" path="/var/lib/kubelet/pods/b75f5108-fb30-4d89-8480-d9e97126ef33/volumes" Jul 06 19:50:27.247367 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.247331 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" podUID="1a9ac327-1c5d-49ce-962c-73ba3d30f53d" containerName="main" containerID="cri-o://053a3222d5f651b472acfec11e45b5330c1460b3a7ce2cf9e455101712a2e8ef" gracePeriod=45 Jul 06 19:50:27.486636 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.486614 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:50:27.582953 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.582898 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-tmp-dir\") pod \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " Jul 06 19:50:27.582953 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.582926 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx7sq\" (UniqueName: \"kubernetes.io/projected/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-kube-api-access-bx7sq\") pod \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " Jul 06 19:50:27.583133 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.582975 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-model-cache\") pod \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " Jul 06 19:50:27.583133 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.583045 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-tls-certs\") pod \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " Jul 06 19:50:27.583133 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.583077 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-dshm\") pod \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " Jul 06 19:50:27.583133 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.583116 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-home\") pod \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\" (UID: \"1a9ac327-1c5d-49ce-962c-73ba3d30f53d\") " Jul 06 19:50:27.583332 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.583163 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a9ac327-1c5d-49ce-962c-73ba3d30f53d" (UID: "1a9ac327-1c5d-49ce-962c-73ba3d30f53d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:50:27.583332 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.583275 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a9ac327-1c5d-49ce-962c-73ba3d30f53d" (UID: "1a9ac327-1c5d-49ce-962c-73ba3d30f53d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:50:27.583435 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.583400 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-home" (OuterVolumeSpecName: "home") pod "1a9ac327-1c5d-49ce-962c-73ba3d30f53d" (UID: "1a9ac327-1c5d-49ce-962c-73ba3d30f53d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:50:27.583492 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.583408 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-tmp-dir\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:50:27.583492 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.583458 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-model-cache\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:50:27.585018 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.584997 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-dshm" (OuterVolumeSpecName: "dshm") pod "1a9ac327-1c5d-49ce-962c-73ba3d30f53d" (UID: "1a9ac327-1c5d-49ce-962c-73ba3d30f53d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:50:27.585110 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.585092 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-kube-api-access-bx7sq" (OuterVolumeSpecName: "kube-api-access-bx7sq") pod "1a9ac327-1c5d-49ce-962c-73ba3d30f53d" (UID: "1a9ac327-1c5d-49ce-962c-73ba3d30f53d"). InnerVolumeSpecName "kube-api-access-bx7sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:50:27.585152 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.585129 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a9ac327-1c5d-49ce-962c-73ba3d30f53d" (UID: "1a9ac327-1c5d-49ce-962c-73ba3d30f53d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:50:27.684051 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.684030 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-home\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:50:27.684051 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.684051 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx7sq\" (UniqueName: \"kubernetes.io/projected/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-kube-api-access-bx7sq\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:50:27.684169 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.684061 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-tls-certs\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:50:27.684169 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:27.684069 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a9ac327-1c5d-49ce-962c-73ba3d30f53d-dshm\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:50:28.013700 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:28.013675 2581 generic.go:356] "Generic (PLEG): container finished" podID="1a9ac327-1c5d-49ce-962c-73ba3d30f53d" containerID="053a3222d5f651b472acfec11e45b5330c1460b3a7ce2cf9e455101712a2e8ef" exitCode=0 Jul 06 19:50:28.013795 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:28.013712 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" Jul 06 19:50:28.013795 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:28.013710 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" event={"ID":"1a9ac327-1c5d-49ce-962c-73ba3d30f53d","Type":"ContainerDied","Data":"053a3222d5f651b472acfec11e45b5330c1460b3a7ce2cf9e455101712a2e8ef"} Jul 06 19:50:28.013795 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:28.013760 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds" event={"ID":"1a9ac327-1c5d-49ce-962c-73ba3d30f53d","Type":"ContainerDied","Data":"c71daeb40b15b2f3cd339aad0a976a2f8e583267a510c47ddbc2b937bd6dcf2e"} Jul 06 19:50:28.013795 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:28.013781 2581 scope.go:117] "RemoveContainer" containerID="053a3222d5f651b472acfec11e45b5330c1460b3a7ce2cf9e455101712a2e8ef" Jul 06 19:50:28.023016 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:28.023001 2581 scope.go:117] "RemoveContainer" containerID="053a3222d5f651b472acfec11e45b5330c1460b3a7ce2cf9e455101712a2e8ef" Jul 06 19:50:28.023245 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:50:28.023227 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053a3222d5f651b472acfec11e45b5330c1460b3a7ce2cf9e455101712a2e8ef\": container with ID starting with 053a3222d5f651b472acfec11e45b5330c1460b3a7ce2cf9e455101712a2e8ef not found: ID does not exist" containerID="053a3222d5f651b472acfec11e45b5330c1460b3a7ce2cf9e455101712a2e8ef" Jul 06 19:50:28.023285 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:28.023255 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053a3222d5f651b472acfec11e45b5330c1460b3a7ce2cf9e455101712a2e8ef"} err="failed to get container status \"053a3222d5f651b472acfec11e45b5330c1460b3a7ce2cf9e455101712a2e8ef\": rpc error: code = NotFound desc = could not find container \"053a3222d5f651b472acfec11e45b5330c1460b3a7ce2cf9e455101712a2e8ef\": container with ID starting with 053a3222d5f651b472acfec11e45b5330c1460b3a7ce2cf9e455101712a2e8ef not found: ID does not exist" Jul 06 19:50:28.036726 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:28.036704 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds"] Jul 06 19:50:28.044703 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:28.044679 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d98408/test-model-d98408-kserve-7b49b5f66c-s6lds"] Jul 06 19:50:28.147120 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:28.147097 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a9ac327-1c5d-49ce-962c-73ba3d30f53d" path="/var/lib/kubelet/pods/1a9ac327-1c5d-49ce-962c-73ba3d30f53d/volumes" Jul 06 19:50:57.498388 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:57.498351 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz"] Jul 06 19:50:57.499014 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:57.498555 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" podUID="acdbafa9-823e-4f94-af55-e8abca8e0c8f" containerName="maas-api" containerID="cri-o://133af0c98bf39a70e60dd986f0ce53c45cddaf6e07212dfd1e945573f1c20be2" gracePeriod=30 Jul 06 19:50:57.736919 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:57.736897 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" Jul 06 19:50:57.912211 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:57.912186 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzxmv\" (UniqueName: \"kubernetes.io/projected/acdbafa9-823e-4f94-af55-e8abca8e0c8f-kube-api-access-kzxmv\") pod \"acdbafa9-823e-4f94-af55-e8abca8e0c8f\" (UID: \"acdbafa9-823e-4f94-af55-e8abca8e0c8f\") " Jul 06 19:50:57.912335 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:57.912227 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acdbafa9-823e-4f94-af55-e8abca8e0c8f-maas-api-tls\") pod \"acdbafa9-823e-4f94-af55-e8abca8e0c8f\" (UID: \"acdbafa9-823e-4f94-af55-e8abca8e0c8f\") " Jul 06 19:50:57.912335 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:57.912274 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdbafa9-823e-4f94-af55-e8abca8e0c8f-openshift-service-ca\") pod \"acdbafa9-823e-4f94-af55-e8abca8e0c8f\" (UID: \"acdbafa9-823e-4f94-af55-e8abca8e0c8f\") " Jul 06 19:50:57.912673 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:57.912616 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acdbafa9-823e-4f94-af55-e8abca8e0c8f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "acdbafa9-823e-4f94-af55-e8abca8e0c8f" (UID: "acdbafa9-823e-4f94-af55-e8abca8e0c8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:50:57.914156 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:57.914132 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acdbafa9-823e-4f94-af55-e8abca8e0c8f-kube-api-access-kzxmv" (OuterVolumeSpecName: "kube-api-access-kzxmv") pod "acdbafa9-823e-4f94-af55-e8abca8e0c8f" (UID: "acdbafa9-823e-4f94-af55-e8abca8e0c8f"). InnerVolumeSpecName "kube-api-access-kzxmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:50:57.914236 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:57.914157 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acdbafa9-823e-4f94-af55-e8abca8e0c8f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "acdbafa9-823e-4f94-af55-e8abca8e0c8f" (UID: "acdbafa9-823e-4f94-af55-e8abca8e0c8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:50:58.013608 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:58.013582 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzxmv\" (UniqueName: \"kubernetes.io/projected/acdbafa9-823e-4f94-af55-e8abca8e0c8f-kube-api-access-kzxmv\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:50:58.013608 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:58.013606 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acdbafa9-823e-4f94-af55-e8abca8e0c8f-maas-api-tls\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:50:58.013764 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:58.013619 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdbafa9-823e-4f94-af55-e8abca8e0c8f-openshift-service-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:50:58.162466 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:58.162401 2581 generic.go:356] "Generic (PLEG): container finished" podID="acdbafa9-823e-4f94-af55-e8abca8e0c8f" containerID="133af0c98bf39a70e60dd986f0ce53c45cddaf6e07212dfd1e945573f1c20be2" exitCode=0 Jul 06 19:50:58.162466 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:58.162446 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" Jul 06 19:50:58.162578 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:58.162486 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" event={"ID":"acdbafa9-823e-4f94-af55-e8abca8e0c8f","Type":"ContainerDied","Data":"133af0c98bf39a70e60dd986f0ce53c45cddaf6e07212dfd1e945573f1c20be2"} Jul 06 19:50:58.162578 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:58.162537 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" event={"ID":"acdbafa9-823e-4f94-af55-e8abca8e0c8f","Type":"ContainerDied","Data":"4a23c89783c62a9d9b190d1f70bab89d5c03ee44aed322aa17e342d43f0b5317"} Jul 06 19:50:58.162578 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:58.162558 2581 scope.go:117] "RemoveContainer" containerID="133af0c98bf39a70e60dd986f0ce53c45cddaf6e07212dfd1e945573f1c20be2" Jul 06 19:50:58.171106 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:58.171088 2581 scope.go:117] "RemoveContainer" containerID="133af0c98bf39a70e60dd986f0ce53c45cddaf6e07212dfd1e945573f1c20be2" Jul 06 19:50:58.171369 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:50:58.171346 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133af0c98bf39a70e60dd986f0ce53c45cddaf6e07212dfd1e945573f1c20be2\": container with ID starting with 133af0c98bf39a70e60dd986f0ce53c45cddaf6e07212dfd1e945573f1c20be2 not found: ID does not exist" containerID="133af0c98bf39a70e60dd986f0ce53c45cddaf6e07212dfd1e945573f1c20be2" Jul 06 19:50:58.171423 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:58.171380 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133af0c98bf39a70e60dd986f0ce53c45cddaf6e07212dfd1e945573f1c20be2"} err="failed to get container status \"133af0c98bf39a70e60dd986f0ce53c45cddaf6e07212dfd1e945573f1c20be2\": rpc error: code = NotFound desc = could not find container \"133af0c98bf39a70e60dd986f0ce53c45cddaf6e07212dfd1e945573f1c20be2\": container with ID starting with 133af0c98bf39a70e60dd986f0ce53c45cddaf6e07212dfd1e945573f1c20be2 not found: ID does not exist" Jul 06 19:50:58.182053 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:58.182031 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz"] Jul 06 19:50:58.184149 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:50:58.184131 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz"] Jul 06 19:51:00.147671 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:00.147633 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acdbafa9-823e-4f94-af55-e8abca8e0c8f" path="/var/lib/kubelet/pods/acdbafa9-823e-4f94-af55-e8abca8e0c8f/volumes" Jul 06 19:51:00.666780 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:00.666740 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-inf-b-f1ee26-7f899cbf-wxgcz" podUID="acdbafa9-823e-4f94-af55-e8abca8e0c8f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.41:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 06 19:51:04.674043 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.674010 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr"] Jul 06 19:51:04.674422 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.674398 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b75f5108-fb30-4d89-8480-d9e97126ef33" containerName="maas-api" Jul 06 19:51:04.674422 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.674413 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b75f5108-fb30-4d89-8480-d9e97126ef33" containerName="maas-api" Jul 06 19:51:04.674508 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.674427 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a9ac327-1c5d-49ce-962c-73ba3d30f53d" containerName="main" Jul 06 19:51:04.674508 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.674450 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a9ac327-1c5d-49ce-962c-73ba3d30f53d" containerName="main" Jul 06 19:51:04.674508 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.674470 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acdbafa9-823e-4f94-af55-e8abca8e0c8f" containerName="maas-api" Jul 06 19:51:04.674508 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.674477 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="acdbafa9-823e-4f94-af55-e8abca8e0c8f" containerName="maas-api" Jul 06 19:51:04.674647 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.674557 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a9ac327-1c5d-49ce-962c-73ba3d30f53d" containerName="main" Jul 06 19:51:04.674647 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.674568 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="acdbafa9-823e-4f94-af55-e8abca8e0c8f" containerName="maas-api" Jul 06 19:51:04.674647 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.674574 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b75f5108-fb30-4d89-8480-d9e97126ef33" containerName="maas-api" Jul 06 19:51:04.678846 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.678827 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.681171 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.681152 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-697818-openshift-default-dockercfg-bpknj\"" Jul 06 19:51:04.691862 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.691843 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr"] Jul 06 19:51:04.762741 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.762716 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f6bd97-2248-42c0-9249-ffd007256cbd-istiod-ca-cert\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.762870 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.762756 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx9kr\" (UniqueName: \"kubernetes.io/projected/b5f6bd97-2248-42c0-9249-ffd007256cbd-kube-api-access-lx9kr\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.762916 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.762871 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-workload-certs\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.762916 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.762909 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-envoy\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.763033 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.762929 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-podinfo\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.763033 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.762947 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-token\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.763033 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.762974 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-workload-socket\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.763129 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.763058 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-credential-socket\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.763129 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.763110 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-data\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.863796 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.863770 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f6bd97-2248-42c0-9249-ffd007256cbd-istiod-ca-cert\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.863914 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.863816 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx9kr\" (UniqueName: \"kubernetes.io/projected/b5f6bd97-2248-42c0-9249-ffd007256cbd-kube-api-access-lx9kr\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.863914 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.863857 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-workload-certs\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.863914 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.863881 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-envoy\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.863914 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.863900 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-podinfo\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.864074 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.863972 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-token\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.864074 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.864021 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-workload-socket\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.864074 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.864065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-credential-socket\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.864220 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.864092 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-data\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.864276 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.864246 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-workload-certs\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.864428 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.864404 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-data\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.864500 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.864451 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-workload-socket\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.864559 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.864524 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f6bd97-2248-42c0-9249-ffd007256cbd-istiod-ca-cert\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.864559 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.864546 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-credential-socket\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.866205 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.866183 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-envoy\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.866439 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.866411 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-podinfo\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.872145 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.872123 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-token\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.872600 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.872584 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx9kr\" (UniqueName: \"kubernetes.io/projected/b5f6bd97-2248-42c0-9249-ffd007256cbd-kube-api-access-lx9kr\") pod \"e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:04.989051 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:04.989001 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:05.122867 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:05.122835 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr"] Jul 06 19:51:05.125836 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:51:05.125810 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f6bd97_2248_42c0_9249_ffd007256cbd.slice/crio-ddd44c27d7f17fb9940388bcf641c917cc1f77f5bc460986fdb908473bae6b29 WatchSource:0}: Error finding container ddd44c27d7f17fb9940388bcf641c917cc1f77f5bc460986fdb908473bae6b29: Status 404 returned error can't find the container with id ddd44c27d7f17fb9940388bcf641c917cc1f77f5bc460986fdb908473bae6b29 Jul 06 19:51:05.194029 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:05.194002 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" event={"ID":"b5f6bd97-2248-42c0-9249-ffd007256cbd","Type":"ContainerStarted","Data":"ddd44c27d7f17fb9940388bcf641c917cc1f77f5bc460986fdb908473bae6b29"} Jul 06 19:51:07.513895 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:07.513861 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:51:07.514164 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:07.513928 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:51:07.514164 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:07.513959 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:51:08.209781 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:08.209749 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" event={"ID":"b5f6bd97-2248-42c0-9249-ffd007256cbd","Type":"ContainerStarted","Data":"eba4338182ea614a1add9e1cf868111f9dfd5d91bdb15890baf40a8da1df52b4"} Jul 06 19:51:08.232647 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:08.232600 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" podStartSLOduration=1.8467763430000002 podStartE2EDuration="4.232585704s" totalImagesPullingTime="2.385809361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:08.230959334 +0000 UTC m=+2369.446596864" watchObservedRunningTime="2026-07-06 19:51:08.232585704 +0000 UTC m=+2369.448223232" Jul 06 19:51:08.989959 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:08.989929 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:08.994353 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:08.994325 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:09.214510 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:09.214483 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:09.215241 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:09.215222 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:10.280768 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:10.280738 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g"] Jul 06 19:51:10.284330 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:10.284315 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" Jul 06 19:51:10.286999 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:10.286977 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-697818-serving-cert\"" Jul 06 19:51:10.305189 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:10.305165 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g"] Jul 06 19:51:10.312335 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:10.312317 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdmj4\" (UniqueName: \"kubernetes.io/projected/6f279dfb-9a0c-4472-9087-86bef91b5158-kube-api-access-bdmj4\") pod \"maas-api-e2e-api-a-697818-8446c4f6c-bmr7g\" (UID: \"6f279dfb-9a0c-4472-9087-86bef91b5158\") " pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" Jul 06 19:51:10.312448 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:10.312381 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f279dfb-9a0c-4472-9087-86bef91b5158-maas-api-tls\") pod \"maas-api-e2e-api-a-697818-8446c4f6c-bmr7g\" (UID: \"6f279dfb-9a0c-4472-9087-86bef91b5158\") " pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" Jul 06 19:51:10.312448 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:10.312434 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f279dfb-9a0c-4472-9087-86bef91b5158-openshift-service-ca\") pod \"maas-api-e2e-api-a-697818-8446c4f6c-bmr7g\" (UID: \"6f279dfb-9a0c-4472-9087-86bef91b5158\") " pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" Jul 06 19:51:10.412857 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:10.412826 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdmj4\" (UniqueName: \"kubernetes.io/projected/6f279dfb-9a0c-4472-9087-86bef91b5158-kube-api-access-bdmj4\") pod \"maas-api-e2e-api-a-697818-8446c4f6c-bmr7g\" (UID: \"6f279dfb-9a0c-4472-9087-86bef91b5158\") " pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" Jul 06 19:51:10.413018 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:10.412875 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f279dfb-9a0c-4472-9087-86bef91b5158-maas-api-tls\") pod \"maas-api-e2e-api-a-697818-8446c4f6c-bmr7g\" (UID: \"6f279dfb-9a0c-4472-9087-86bef91b5158\") " pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" Jul 06 19:51:10.413018 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:10.412920 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f279dfb-9a0c-4472-9087-86bef91b5158-openshift-service-ca\") pod \"maas-api-e2e-api-a-697818-8446c4f6c-bmr7g\" (UID: \"6f279dfb-9a0c-4472-9087-86bef91b5158\") " pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" Jul 06 19:51:10.413141 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:51:10.413042 2581 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-697818-serving-cert: secret "maas-api-e2e-api-a-697818-serving-cert" not found Jul 06 19:51:10.413141 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:51:10.413116 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f279dfb-9a0c-4472-9087-86bef91b5158-maas-api-tls podName:6f279dfb-9a0c-4472-9087-86bef91b5158 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:10.913093212 +0000 UTC m=+2372.128730720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f279dfb-9a0c-4472-9087-86bef91b5158-maas-api-tls") pod "maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" (UID: "6f279dfb-9a0c-4472-9087-86bef91b5158") : secret "maas-api-e2e-api-a-697818-serving-cert" not found Jul 06 19:51:10.413471 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:10.413452 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f279dfb-9a0c-4472-9087-86bef91b5158-openshift-service-ca\") pod \"maas-api-e2e-api-a-697818-8446c4f6c-bmr7g\" (UID: \"6f279dfb-9a0c-4472-9087-86bef91b5158\") " pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" Jul 06 19:51:10.430232 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:10.430204 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdmj4\" (UniqueName: \"kubernetes.io/projected/6f279dfb-9a0c-4472-9087-86bef91b5158-kube-api-access-bdmj4\") pod \"maas-api-e2e-api-a-697818-8446c4f6c-bmr7g\" (UID: \"6f279dfb-9a0c-4472-9087-86bef91b5158\") " pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" Jul 06 19:51:10.916375 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:10.916350 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f279dfb-9a0c-4472-9087-86bef91b5158-maas-api-tls\") pod \"maas-api-e2e-api-a-697818-8446c4f6c-bmr7g\" (UID: \"6f279dfb-9a0c-4472-9087-86bef91b5158\") " pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" Jul 06 19:51:10.918646 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:10.918625 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f279dfb-9a0c-4472-9087-86bef91b5158-maas-api-tls\") pod \"maas-api-e2e-api-a-697818-8446c4f6c-bmr7g\" (UID: \"6f279dfb-9a0c-4472-9087-86bef91b5158\") " pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" Jul 06 19:51:11.194886 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:11.194815 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" Jul 06 19:51:11.336013 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:11.335992 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g"] Jul 06 19:51:11.338568 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:51:11.338535 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f279dfb_9a0c_4472_9087_86bef91b5158.slice/crio-17722ac02f56a69923c7832f2cc25092d661f220e014e1cf86f164061632d313 WatchSource:0}: Error finding container 17722ac02f56a69923c7832f2cc25092d661f220e014e1cf86f164061632d313: Status 404 returned error can't find the container with id 17722ac02f56a69923c7832f2cc25092d661f220e014e1cf86f164061632d313 Jul 06 19:51:12.230037 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:12.230000 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" event={"ID":"6f279dfb-9a0c-4472-9087-86bef91b5158","Type":"ContainerStarted","Data":"d709cd79e16c2f2a41e7781e68e867aeeb4b88e5287b54214ffcdeae6f319f85"} Jul 06 19:51:12.230193 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:12.230067 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" event={"ID":"6f279dfb-9a0c-4472-9087-86bef91b5158","Type":"ContainerStarted","Data":"17722ac02f56a69923c7832f2cc25092d661f220e014e1cf86f164061632d313"} Jul 06 19:51:12.230193 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:12.230096 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" Jul 06 19:51:12.265258 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:12.265211 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" podStartSLOduration=1.7487903519999999 podStartE2EDuration="2.265196043s" totalImagesPullingTime="516.405691ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:12.263109619 +0000 UTC m=+2373.478747179" watchObservedRunningTime="2026-07-06 19:51:12.265196043 +0000 UTC m=+2373.480833632" Jul 06 19:51:18.247924 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:18.247895 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" Jul 06 19:51:18.808911 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:18.808881 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs"] Jul 06 19:51:18.812948 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:18.812912 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" Jul 06 19:51:18.816257 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:18.816236 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-d8b5cd-serving-cert\"" Jul 06 19:51:18.829593 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:18.829574 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs"] Jul 06 19:51:18.881306 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:18.881274 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6c28\" (UniqueName: \"kubernetes.io/projected/41a8834b-1e5c-4993-a728-a6c1f88a4828-kube-api-access-m6c28\") pod \"maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs\" (UID: \"41a8834b-1e5c-4993-a728-a6c1f88a4828\") " pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" Jul 06 19:51:18.881442 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:18.881398 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a8834b-1e5c-4993-a728-a6c1f88a4828-openshift-service-ca\") pod \"maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs\" (UID: \"41a8834b-1e5c-4993-a728-a6c1f88a4828\") " pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" Jul 06 19:51:18.881512 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:18.881446 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a8834b-1e5c-4993-a728-a6c1f88a4828-maas-api-tls\") pod \"maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs\" (UID: \"41a8834b-1e5c-4993-a728-a6c1f88a4828\") " pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" Jul 06 19:51:18.982753 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:18.982725 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a8834b-1e5c-4993-a728-a6c1f88a4828-openshift-service-ca\") pod \"maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs\" (UID: \"41a8834b-1e5c-4993-a728-a6c1f88a4828\") " pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" Jul 06 19:51:18.982918 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:18.982769 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a8834b-1e5c-4993-a728-a6c1f88a4828-maas-api-tls\") pod \"maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs\" (UID: \"41a8834b-1e5c-4993-a728-a6c1f88a4828\") " pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" Jul 06 19:51:18.982918 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:18.982793 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6c28\" (UniqueName: \"kubernetes.io/projected/41a8834b-1e5c-4993-a728-a6c1f88a4828-kube-api-access-m6c28\") pod \"maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs\" (UID: \"41a8834b-1e5c-4993-a728-a6c1f88a4828\") " pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" Jul 06 19:51:18.983408 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:18.983384 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a8834b-1e5c-4993-a728-a6c1f88a4828-openshift-service-ca\") pod \"maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs\" (UID: \"41a8834b-1e5c-4993-a728-a6c1f88a4828\") " pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" Jul 06 19:51:18.985048 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:18.985028 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a8834b-1e5c-4993-a728-a6c1f88a4828-maas-api-tls\") pod \"maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs\" (UID: \"41a8834b-1e5c-4993-a728-a6c1f88a4828\") " pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" Jul 06 19:51:18.993431 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:18.993406 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6c28\" (UniqueName: \"kubernetes.io/projected/41a8834b-1e5c-4993-a728-a6c1f88a4828-kube-api-access-m6c28\") pod \"maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs\" (UID: \"41a8834b-1e5c-4993-a728-a6c1f88a4828\") " pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" Jul 06 19:51:19.137694 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:19.137606 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" Jul 06 19:51:19.276035 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:19.276010 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs"] Jul 06 19:51:19.279368 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:51:19.279341 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a8834b_1e5c_4993_a728_a6c1f88a4828.slice/crio-a3049f74956db629d206a46361eb1b266fbe80bb80ece2bb47bf883b16e97204 WatchSource:0}: Error finding container a3049f74956db629d206a46361eb1b266fbe80bb80ece2bb47bf883b16e97204: Status 404 returned error can't find the container with id a3049f74956db629d206a46361eb1b266fbe80bb80ece2bb47bf883b16e97204 Jul 06 19:51:20.276300 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:20.276267 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" event={"ID":"41a8834b-1e5c-4993-a728-a6c1f88a4828","Type":"ContainerStarted","Data":"888afdc6483edd849416740ceebf2ceabac1683379aebad8ff1e68e64dce9489"} Jul 06 19:51:20.276710 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:20.276321 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" event={"ID":"41a8834b-1e5c-4993-a728-a6c1f88a4828","Type":"ContainerStarted","Data":"a3049f74956db629d206a46361eb1b266fbe80bb80ece2bb47bf883b16e97204"} Jul 06 19:51:20.276710 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:20.276430 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" Jul 06 19:51:20.305328 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:20.305285 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" podStartSLOduration=1.96789954 podStartE2EDuration="2.305273281s" totalImagesPullingTime="337.373741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:20.304346419 +0000 UTC m=+2381.519983946" watchObservedRunningTime="2026-07-06 19:51:20.305273281 +0000 UTC m=+2381.520910865" Jul 06 19:51:22.287928 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:22.287898 2581 generic.go:356] "Generic (PLEG): container finished" podID="b5f6bd97-2248-42c0-9249-ffd007256cbd" containerID="eba4338182ea614a1add9e1cf868111f9dfd5d91bdb15890baf40a8da1df52b4" exitCode=0 Jul 06 19:51:22.288352 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:22.287972 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" event={"ID":"b5f6bd97-2248-42c0-9249-ffd007256cbd","Type":"ContainerDied","Data":"eba4338182ea614a1add9e1cf868111f9dfd5d91bdb15890baf40a8da1df52b4"} Jul 06 19:51:22.288352 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:22.288321 2581 scope.go:117] "RemoveContainer" containerID="eba4338182ea614a1add9e1cf868111f9dfd5d91bdb15890baf40a8da1df52b4" Jul 06 19:51:22.289290 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:22.289265 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:51:22.289340 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:22.289332 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:51:22.289374 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:22.289362 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:51:23.294056 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:23.294026 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" event={"ID":"b5f6bd97-2248-42c0-9249-ffd007256cbd","Type":"ContainerStarted","Data":"f57a01182c14e84df6dc36560a6a6dedffb18ffc2c36d8c171d4b79d205b406f"} Jul 06 19:51:23.989559 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:23.989521 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:23.990619 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:23.990593 2581 patch_prober.go:28] interesting pod/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 06 19:51:23.990752 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:23.990663 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" podUID="b5f6bd97-2248-42c0-9249-ffd007256cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 06 19:51:24.214581 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:24.214554 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:24.989372 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:24.989330 2581 patch_prober.go:28] interesting pod/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 06 19:51:24.989863 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:24.989385 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" podUID="b5f6bd97-2248-42c0-9249-ffd007256cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 06 19:51:25.989707 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:25.989672 2581 patch_prober.go:28] interesting pod/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 06 19:51:25.990064 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:25.989729 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" podUID="b5f6bd97-2248-42c0-9249-ffd007256cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 06 19:51:26.287587 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:26.287521 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" Jul 06 19:51:26.994253 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:26.994222 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:26.995337 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:26.995318 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:32.262944 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:32.262915 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g"] Jul 06 19:51:32.263430 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:32.263096 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" podUID="6f279dfb-9a0c-4472-9087-86bef91b5158" containerName="maas-api" containerID="cri-o://d709cd79e16c2f2a41e7781e68e867aeeb4b88e5287b54214ffcdeae6f319f85" gracePeriod=30 Jul 06 19:51:32.507166 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:32.507144 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" Jul 06 19:51:32.603177 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:32.603119 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdmj4\" (UniqueName: \"kubernetes.io/projected/6f279dfb-9a0c-4472-9087-86bef91b5158-kube-api-access-bdmj4\") pod \"6f279dfb-9a0c-4472-9087-86bef91b5158\" (UID: \"6f279dfb-9a0c-4472-9087-86bef91b5158\") " Jul 06 19:51:32.603177 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:32.603146 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f279dfb-9a0c-4472-9087-86bef91b5158-maas-api-tls\") pod \"6f279dfb-9a0c-4472-9087-86bef91b5158\" (UID: \"6f279dfb-9a0c-4472-9087-86bef91b5158\") " Jul 06 19:51:32.603177 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:32.603173 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f279dfb-9a0c-4472-9087-86bef91b5158-openshift-service-ca\") pod \"6f279dfb-9a0c-4472-9087-86bef91b5158\" (UID: \"6f279dfb-9a0c-4472-9087-86bef91b5158\") " Jul 06 19:51:32.603556 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:32.603533 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f279dfb-9a0c-4472-9087-86bef91b5158-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f279dfb-9a0c-4472-9087-86bef91b5158" (UID: "6f279dfb-9a0c-4472-9087-86bef91b5158"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:51:32.605121 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:32.605094 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f279dfb-9a0c-4472-9087-86bef91b5158-kube-api-access-bdmj4" (OuterVolumeSpecName: "kube-api-access-bdmj4") pod "6f279dfb-9a0c-4472-9087-86bef91b5158" (UID: "6f279dfb-9a0c-4472-9087-86bef91b5158"). InnerVolumeSpecName "kube-api-access-bdmj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:51:32.605121 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:32.605115 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f279dfb-9a0c-4472-9087-86bef91b5158-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f279dfb-9a0c-4472-9087-86bef91b5158" (UID: "6f279dfb-9a0c-4472-9087-86bef91b5158"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:51:32.703727 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:32.703703 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdmj4\" (UniqueName: \"kubernetes.io/projected/6f279dfb-9a0c-4472-9087-86bef91b5158-kube-api-access-bdmj4\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:51:32.703727 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:32.703724 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f279dfb-9a0c-4472-9087-86bef91b5158-maas-api-tls\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:51:32.703837 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:32.703734 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f279dfb-9a0c-4472-9087-86bef91b5158-openshift-service-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:51:33.342883 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:33.342850 2581 generic.go:356] "Generic (PLEG): container finished" podID="6f279dfb-9a0c-4472-9087-86bef91b5158" containerID="d709cd79e16c2f2a41e7781e68e867aeeb4b88e5287b54214ffcdeae6f319f85" exitCode=0 Jul 06 19:51:33.343264 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:33.342911 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" Jul 06 19:51:33.343264 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:33.342936 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" event={"ID":"6f279dfb-9a0c-4472-9087-86bef91b5158","Type":"ContainerDied","Data":"d709cd79e16c2f2a41e7781e68e867aeeb4b88e5287b54214ffcdeae6f319f85"} Jul 06 19:51:33.343264 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:33.342989 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g" event={"ID":"6f279dfb-9a0c-4472-9087-86bef91b5158","Type":"ContainerDied","Data":"17722ac02f56a69923c7832f2cc25092d661f220e014e1cf86f164061632d313"} Jul 06 19:51:33.343264 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:33.343010 2581 scope.go:117] "RemoveContainer" containerID="d709cd79e16c2f2a41e7781e68e867aeeb4b88e5287b54214ffcdeae6f319f85" Jul 06 19:51:33.351712 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:33.351692 2581 scope.go:117] "RemoveContainer" containerID="d709cd79e16c2f2a41e7781e68e867aeeb4b88e5287b54214ffcdeae6f319f85" Jul 06 19:51:33.352010 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:51:33.351978 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d709cd79e16c2f2a41e7781e68e867aeeb4b88e5287b54214ffcdeae6f319f85\": container with ID starting with d709cd79e16c2f2a41e7781e68e867aeeb4b88e5287b54214ffcdeae6f319f85 not found: ID does not exist" containerID="d709cd79e16c2f2a41e7781e68e867aeeb4b88e5287b54214ffcdeae6f319f85" Jul 06 19:51:33.352116 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:33.352007 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d709cd79e16c2f2a41e7781e68e867aeeb4b88e5287b54214ffcdeae6f319f85"} err="failed to get container status \"d709cd79e16c2f2a41e7781e68e867aeeb4b88e5287b54214ffcdeae6f319f85\": rpc error: code = NotFound desc = could not find container \"d709cd79e16c2f2a41e7781e68e867aeeb4b88e5287b54214ffcdeae6f319f85\": container with ID starting with d709cd79e16c2f2a41e7781e68e867aeeb4b88e5287b54214ffcdeae6f319f85 not found: ID does not exist" Jul 06 19:51:33.366360 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:33.366334 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g"] Jul 06 19:51:33.370152 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:33.370133 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-697818-8446c4f6c-bmr7g"] Jul 06 19:51:34.147860 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:34.147827 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f279dfb-9a0c-4472-9087-86bef91b5158" path="/var/lib/kubelet/pods/6f279dfb-9a0c-4472-9087-86bef91b5158/volumes" Jul 06 19:51:38.715740 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:38.715709 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr"] Jul 06 19:51:38.716163 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:38.715915 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" podUID="b5f6bd97-2248-42c0-9249-ffd007256cbd" containerName="istio-proxy" containerID="cri-o://f57a01182c14e84df6dc36560a6a6dedffb18ffc2c36d8c171d4b79d205b406f" gracePeriod=30 Jul 06 19:51:39.376026 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:39.376000 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:51:39.382354 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:39.382334 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:51:40.492035 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:40.492008 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs"] Jul 06 19:51:40.492435 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:40.492185 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" podUID="41a8834b-1e5c-4993-a728-a6c1f88a4828" containerName="maas-api" containerID="cri-o://888afdc6483edd849416740ceebf2ceabac1683379aebad8ff1e68e64dce9489" gracePeriod=30 Jul 06 19:51:40.732213 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:40.732191 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" Jul 06 19:51:40.870616 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:40.870560 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a8834b-1e5c-4993-a728-a6c1f88a4828-maas-api-tls\") pod \"41a8834b-1e5c-4993-a728-a6c1f88a4828\" (UID: \"41a8834b-1e5c-4993-a728-a6c1f88a4828\") " Jul 06 19:51:40.870746 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:40.870629 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6c28\" (UniqueName: \"kubernetes.io/projected/41a8834b-1e5c-4993-a728-a6c1f88a4828-kube-api-access-m6c28\") pod \"41a8834b-1e5c-4993-a728-a6c1f88a4828\" (UID: \"41a8834b-1e5c-4993-a728-a6c1f88a4828\") " Jul 06 19:51:40.870746 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:40.870686 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a8834b-1e5c-4993-a728-a6c1f88a4828-openshift-service-ca\") pod \"41a8834b-1e5c-4993-a728-a6c1f88a4828\" (UID: \"41a8834b-1e5c-4993-a728-a6c1f88a4828\") " Jul 06 19:51:40.871057 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:40.871030 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a8834b-1e5c-4993-a728-a6c1f88a4828-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "41a8834b-1e5c-4993-a728-a6c1f88a4828" (UID: "41a8834b-1e5c-4993-a728-a6c1f88a4828"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:51:40.872573 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:40.872550 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41a8834b-1e5c-4993-a728-a6c1f88a4828-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "41a8834b-1e5c-4993-a728-a6c1f88a4828" (UID: "41a8834b-1e5c-4993-a728-a6c1f88a4828"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:51:40.872677 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:40.872611 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a8834b-1e5c-4993-a728-a6c1f88a4828-kube-api-access-m6c28" (OuterVolumeSpecName: "kube-api-access-m6c28") pod "41a8834b-1e5c-4993-a728-a6c1f88a4828" (UID: "41a8834b-1e5c-4993-a728-a6c1f88a4828"). InnerVolumeSpecName "kube-api-access-m6c28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:51:40.971296 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:40.971276 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6c28\" (UniqueName: \"kubernetes.io/projected/41a8834b-1e5c-4993-a728-a6c1f88a4828-kube-api-access-m6c28\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:51:40.971296 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:40.971297 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a8834b-1e5c-4993-a728-a6c1f88a4828-openshift-service-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:51:40.971420 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:40.971307 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a8834b-1e5c-4993-a728-a6c1f88a4828-maas-api-tls\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:51:41.383377 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:41.383351 2581 generic.go:356] "Generic (PLEG): container finished" podID="41a8834b-1e5c-4993-a728-a6c1f88a4828" containerID="888afdc6483edd849416740ceebf2ceabac1683379aebad8ff1e68e64dce9489" exitCode=0 Jul 06 19:51:41.383551 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:41.383392 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" Jul 06 19:51:41.383551 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:41.383416 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" event={"ID":"41a8834b-1e5c-4993-a728-a6c1f88a4828","Type":"ContainerDied","Data":"888afdc6483edd849416740ceebf2ceabac1683379aebad8ff1e68e64dce9489"} Jul 06 19:51:41.383551 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:41.383466 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs" event={"ID":"41a8834b-1e5c-4993-a728-a6c1f88a4828","Type":"ContainerDied","Data":"a3049f74956db629d206a46361eb1b266fbe80bb80ece2bb47bf883b16e97204"} Jul 06 19:51:41.383551 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:41.383496 2581 scope.go:117] "RemoveContainer" containerID="888afdc6483edd849416740ceebf2ceabac1683379aebad8ff1e68e64dce9489" Jul 06 19:51:41.396705 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:41.396686 2581 scope.go:117] "RemoveContainer" containerID="888afdc6483edd849416740ceebf2ceabac1683379aebad8ff1e68e64dce9489" Jul 06 19:51:41.396956 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:51:41.396938 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"888afdc6483edd849416740ceebf2ceabac1683379aebad8ff1e68e64dce9489\": container with ID starting with 888afdc6483edd849416740ceebf2ceabac1683379aebad8ff1e68e64dce9489 not found: ID does not exist" containerID="888afdc6483edd849416740ceebf2ceabac1683379aebad8ff1e68e64dce9489" Jul 06 19:51:41.397005 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:41.396963 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888afdc6483edd849416740ceebf2ceabac1683379aebad8ff1e68e64dce9489"} err="failed to get container status \"888afdc6483edd849416740ceebf2ceabac1683379aebad8ff1e68e64dce9489\": rpc error: code = NotFound desc = could not find container \"888afdc6483edd849416740ceebf2ceabac1683379aebad8ff1e68e64dce9489\": container with ID starting with 888afdc6483edd849416740ceebf2ceabac1683379aebad8ff1e68e64dce9489 not found: ID does not exist" Jul 06 19:51:41.421407 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:41.421383 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs"] Jul 06 19:51:41.427732 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:41.427712 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-d8b5cd-774b789745-gjmqs"] Jul 06 19:51:41.996356 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:41.996323 2581 patch_prober.go:28] interesting pod/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 19:51:41.996722 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:41.996373 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" podUID="b5f6bd97-2248-42c0-9249-ffd007256cbd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:51:42.148313 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:42.148281 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41a8834b-1e5c-4993-a728-a6c1f88a4828" path="/var/lib/kubelet/pods/41a8834b-1e5c-4993-a728-a6c1f88a4828/volumes" Jul 06 19:51:43.959691 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:43.959667 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:44.095648 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.095582 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-credential-socket\") pod \"b5f6bd97-2248-42c0-9249-ffd007256cbd\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " Jul 06 19:51:44.095648 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.095616 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-envoy\") pod \"b5f6bd97-2248-42c0-9249-ffd007256cbd\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " Jul 06 19:51:44.095648 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.095633 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-data\") pod \"b5f6bd97-2248-42c0-9249-ffd007256cbd\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " Jul 06 19:51:44.095934 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.095674 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-podinfo\") pod \"b5f6bd97-2248-42c0-9249-ffd007256cbd\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " Jul 06 19:51:44.095934 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.095717 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lx9kr\" (UniqueName: \"kubernetes.io/projected/b5f6bd97-2248-42c0-9249-ffd007256cbd-kube-api-access-lx9kr\") pod \"b5f6bd97-2248-42c0-9249-ffd007256cbd\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " Jul 06 19:51:44.095934 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.095748 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-token\") pod \"b5f6bd97-2248-42c0-9249-ffd007256cbd\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " Jul 06 19:51:44.095934 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.095786 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-workload-certs\") pod \"b5f6bd97-2248-42c0-9249-ffd007256cbd\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " Jul 06 19:51:44.095934 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.095833 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f6bd97-2248-42c0-9249-ffd007256cbd-istiod-ca-cert\") pod \"b5f6bd97-2248-42c0-9249-ffd007256cbd\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " Jul 06 19:51:44.095934 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.095862 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-workload-socket\") pod \"b5f6bd97-2248-42c0-9249-ffd007256cbd\" (UID: \"b5f6bd97-2248-42c0-9249-ffd007256cbd\") " Jul 06 19:51:44.095934 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.095914 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b5f6bd97-2248-42c0-9249-ffd007256cbd" (UID: "b5f6bd97-2248-42c0-9249-ffd007256cbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:44.096286 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.096081 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b5f6bd97-2248-42c0-9249-ffd007256cbd" (UID: "b5f6bd97-2248-42c0-9249-ffd007256cbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:44.096286 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.096178 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-workload-certs\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:51:44.096286 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.096195 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-credential-socket\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:51:44.096286 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.096278 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b5f6bd97-2248-42c0-9249-ffd007256cbd" (UID: "b5f6bd97-2248-42c0-9249-ffd007256cbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:44.096456 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.096310 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f6bd97-2248-42c0-9249-ffd007256cbd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b5f6bd97-2248-42c0-9249-ffd007256cbd" (UID: "b5f6bd97-2248-42c0-9249-ffd007256cbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:51:44.096522 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.096506 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-data" (OuterVolumeSpecName: "istio-data") pod "b5f6bd97-2248-42c0-9249-ffd007256cbd" (UID: "b5f6bd97-2248-42c0-9249-ffd007256cbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:44.097881 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.097846 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b5f6bd97-2248-42c0-9249-ffd007256cbd" (UID: "b5f6bd97-2248-42c0-9249-ffd007256cbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:51:44.098066 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.098042 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f6bd97-2248-42c0-9249-ffd007256cbd-kube-api-access-lx9kr" (OuterVolumeSpecName: "kube-api-access-lx9kr") pod "b5f6bd97-2248-42c0-9249-ffd007256cbd" (UID: "b5f6bd97-2248-42c0-9249-ffd007256cbd"). InnerVolumeSpecName "kube-api-access-lx9kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:51:44.098225 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.098188 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-token" (OuterVolumeSpecName: "istio-token") pod "b5f6bd97-2248-42c0-9249-ffd007256cbd" (UID: "b5f6bd97-2248-42c0-9249-ffd007256cbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:51:44.098299 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.098271 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b5f6bd97-2248-42c0-9249-ffd007256cbd" (UID: "b5f6bd97-2248-42c0-9249-ffd007256cbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:51:44.196608 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.196589 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-envoy\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:51:44.196608 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.196609 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-data\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:51:44.196746 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.196618 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-podinfo\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:51:44.196746 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.196626 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx9kr\" (UniqueName: \"kubernetes.io/projected/b5f6bd97-2248-42c0-9249-ffd007256cbd-kube-api-access-lx9kr\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:51:44.196746 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.196635 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f6bd97-2248-42c0-9249-ffd007256cbd-istio-token\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:51:44.196746 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.196673 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f6bd97-2248-42c0-9249-ffd007256cbd-istiod-ca-cert\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:51:44.196746 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.196681 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f6bd97-2248-42c0-9249-ffd007256cbd-workload-socket\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:51:44.399690 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.399611 2581 generic.go:356] "Generic (PLEG): container finished" podID="b5f6bd97-2248-42c0-9249-ffd007256cbd" containerID="f57a01182c14e84df6dc36560a6a6dedffb18ffc2c36d8c171d4b79d205b406f" exitCode=0 Jul 06 19:51:44.399690 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.399669 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" Jul 06 19:51:44.399829 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.399694 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" event={"ID":"b5f6bd97-2248-42c0-9249-ffd007256cbd","Type":"ContainerDied","Data":"f57a01182c14e84df6dc36560a6a6dedffb18ffc2c36d8c171d4b79d205b406f"} Jul 06 19:51:44.399829 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.399743 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr" event={"ID":"b5f6bd97-2248-42c0-9249-ffd007256cbd","Type":"ContainerDied","Data":"ddd44c27d7f17fb9940388bcf641c917cc1f77f5bc460986fdb908473bae6b29"} Jul 06 19:51:44.399829 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.399765 2581 scope.go:117] "RemoveContainer" containerID="f57a01182c14e84df6dc36560a6a6dedffb18ffc2c36d8c171d4b79d205b406f" Jul 06 19:51:44.408175 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.408160 2581 scope.go:117] "RemoveContainer" containerID="eba4338182ea614a1add9e1cf868111f9dfd5d91bdb15890baf40a8da1df52b4" Jul 06 19:51:44.415252 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.415206 2581 scope.go:117] "RemoveContainer" containerID="f57a01182c14e84df6dc36560a6a6dedffb18ffc2c36d8c171d4b79d205b406f" Jul 06 19:51:44.415487 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:51:44.415456 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57a01182c14e84df6dc36560a6a6dedffb18ffc2c36d8c171d4b79d205b406f\": container with ID starting with f57a01182c14e84df6dc36560a6a6dedffb18ffc2c36d8c171d4b79d205b406f not found: ID does not exist" containerID="f57a01182c14e84df6dc36560a6a6dedffb18ffc2c36d8c171d4b79d205b406f" Jul 06 19:51:44.415559 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.415494 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57a01182c14e84df6dc36560a6a6dedffb18ffc2c36d8c171d4b79d205b406f"} err="failed to get container status \"f57a01182c14e84df6dc36560a6a6dedffb18ffc2c36d8c171d4b79d205b406f\": rpc error: code = NotFound desc = could not find container \"f57a01182c14e84df6dc36560a6a6dedffb18ffc2c36d8c171d4b79d205b406f\": container with ID starting with f57a01182c14e84df6dc36560a6a6dedffb18ffc2c36d8c171d4b79d205b406f not found: ID does not exist" Jul 06 19:51:44.415559 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.415509 2581 scope.go:117] "RemoveContainer" containerID="eba4338182ea614a1add9e1cf868111f9dfd5d91bdb15890baf40a8da1df52b4" Jul 06 19:51:44.416002 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:51:44.415787 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eba4338182ea614a1add9e1cf868111f9dfd5d91bdb15890baf40a8da1df52b4\": container with ID starting with eba4338182ea614a1add9e1cf868111f9dfd5d91bdb15890baf40a8da1df52b4 not found: ID does not exist" containerID="eba4338182ea614a1add9e1cf868111f9dfd5d91bdb15890baf40a8da1df52b4" Jul 06 19:51:44.416002 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.415816 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eba4338182ea614a1add9e1cf868111f9dfd5d91bdb15890baf40a8da1df52b4"} err="failed to get container status \"eba4338182ea614a1add9e1cf868111f9dfd5d91bdb15890baf40a8da1df52b4\": rpc error: code = NotFound desc = could not find container \"eba4338182ea614a1add9e1cf868111f9dfd5d91bdb15890baf40a8da1df52b4\": container with ID starting with eba4338182ea614a1add9e1cf868111f9dfd5d91bdb15890baf40a8da1df52b4 not found: ID does not exist" Jul 06 19:51:44.417835 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.417816 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr"] Jul 06 19:51:44.421863 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:44.421844 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-697818-openshift-default-5c7fcd796f-96jlr"] Jul 06 19:51:45.739805 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:51:45.739769 2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a8834b_1e5c_4993_a728_a6c1f88a4828.slice/crio-a3049f74956db629d206a46361eb1b266fbe80bb80ece2bb47bf883b16e97204\": RecentStats: unable to find data in memory cache]" Jul 06 19:51:46.149120 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:46.149038 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5f6bd97-2248-42c0-9249-ffd007256cbd" path="/var/lib/kubelet/pods/b5f6bd97-2248-42c0-9249-ffd007256cbd/volumes" Jul 06 19:51:46.851960 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:46.851925 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n"] Jul 06 19:51:46.852309 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:46.852234 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f6bd97-2248-42c0-9249-ffd007256cbd" containerName="istio-proxy" Jul 06 19:51:46.852309 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:46.852244 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f6bd97-2248-42c0-9249-ffd007256cbd" containerName="istio-proxy" Jul 06 19:51:46.852309 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:46.852268 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41a8834b-1e5c-4993-a728-a6c1f88a4828" containerName="maas-api" Jul 06 19:51:46.852309 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:46.852273 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="41a8834b-1e5c-4993-a728-a6c1f88a4828" containerName="maas-api" Jul 06 19:51:46.852309 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:46.852280 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f279dfb-9a0c-4472-9087-86bef91b5158" containerName="maas-api" Jul 06 19:51:46.852309 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:46.852285 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f279dfb-9a0c-4472-9087-86bef91b5158" containerName="maas-api" Jul 06 19:51:46.852501 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:46.852332 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="41a8834b-1e5c-4993-a728-a6c1f88a4828" containerName="maas-api" Jul 06 19:51:46.852501 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:46.852339 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f279dfb-9a0c-4472-9087-86bef91b5158" containerName="maas-api" Jul 06 19:51:46.852501 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:46.852348 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f6bd97-2248-42c0-9249-ffd007256cbd" containerName="istio-proxy" Jul 06 19:51:46.852501 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:46.852355 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f6bd97-2248-42c0-9249-ffd007256cbd" containerName="istio-proxy" Jul 06 19:51:46.854410 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:46.854394 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:46.857566 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:46.857548 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-2240fd-openshift-default-dockercfg-4t4wm\"" Jul 06 19:51:46.905488 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:46.905460 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n"] Jul 06 19:51:47.015706 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.015684 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-token\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.015822 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.015720 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-envoy\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.015822 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.015740 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6b30664-5d84-49fb-a7bf-ff264cb30083-istiod-ca-cert\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.015822 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.015759 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-data\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.015822 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.015777 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjpcn\" (UniqueName: \"kubernetes.io/projected/c6b30664-5d84-49fb-a7bf-ff264cb30083-kube-api-access-sjpcn\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.015822 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.015795 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-credential-socket\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.015822 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.015810 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-workload-certs\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.016075 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.015844 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-workload-socket\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.016075 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.015929 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-podinfo\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.116308 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.116252 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-workload-socket\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.116308 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.116288 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-podinfo\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.116485 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.116318 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-token\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.116485 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.116353 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-envoy\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.116485 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.116378 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6b30664-5d84-49fb-a7bf-ff264cb30083-istiod-ca-cert\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.116485 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.116406 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-data\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.116485 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.116435 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjpcn\" (UniqueName: \"kubernetes.io/projected/c6b30664-5d84-49fb-a7bf-ff264cb30083-kube-api-access-sjpcn\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.116485 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.116462 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-credential-socket\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.116877 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.116487 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-workload-certs\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.116877 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.116709 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-workload-socket\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.116877 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.116774 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-data\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.116877 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.116859 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-workload-certs\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.117075 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.116991 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-credential-socket\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.117157 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.117138 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6b30664-5d84-49fb-a7bf-ff264cb30083-istiod-ca-cert\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.118589 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.118571 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-envoy\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.118751 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.118734 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-podinfo\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.132648 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.132622 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-token\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.133376 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.133355 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjpcn\" (UniqueName: \"kubernetes.io/projected/c6b30664-5d84-49fb-a7bf-ff264cb30083-kube-api-access-sjpcn\") pod \"e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.164316 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.164296 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:47.315993 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.315970 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n"] Jul 06 19:51:47.318406 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:51:47.318381 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b30664_5d84_49fb_a7bf_ff264cb30083.slice/crio-eb6ec81182deeabbaa0e42d891c8e264f9dbf49306c5ec4e646ed6daa5b28969 WatchSource:0}: Error finding container eb6ec81182deeabbaa0e42d891c8e264f9dbf49306c5ec4e646ed6daa5b28969: Status 404 returned error can't find the container with id eb6ec81182deeabbaa0e42d891c8e264f9dbf49306c5ec4e646ed6daa5b28969 Jul 06 19:51:47.320573 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.320532 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:51:47.320675 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.320606 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:51:47.320675 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.320635 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:51:47.417341 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.417301 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" event={"ID":"c6b30664-5d84-49fb-a7bf-ff264cb30083","Type":"ContainerStarted","Data":"a41d1043258f1704e5c1096007bcc38a19b488450a56794cff5bceb12e29aa8a"} Jul 06 19:51:47.417507 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.417371 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" event={"ID":"c6b30664-5d84-49fb-a7bf-ff264cb30083","Type":"ContainerStarted","Data":"eb6ec81182deeabbaa0e42d891c8e264f9dbf49306c5ec4e646ed6daa5b28969"} Jul 06 19:51:47.456526 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:47.456479 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" podStartSLOduration=1.456462448 podStartE2EDuration="1.456462448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:47.454227296 +0000 UTC m=+2408.669864827" watchObservedRunningTime="2026-07-06 19:51:47.456462448 +0000 UTC m=+2408.672099976" Jul 06 19:51:48.165037 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:48.165010 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:48.169360 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:48.169336 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:48.422486 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:48.422416 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:48.423174 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:48.423154 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:51:49.601867 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:49.601838 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq"] Jul 06 19:51:49.602317 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:49.602302 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f6bd97-2248-42c0-9249-ffd007256cbd" containerName="istio-proxy" Jul 06 19:51:49.602357 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:49.602320 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f6bd97-2248-42c0-9249-ffd007256cbd" containerName="istio-proxy" Jul 06 19:51:49.604514 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:49.604500 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" Jul 06 19:51:49.607237 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:49.607216 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-2240fd-serving-cert\"" Jul 06 19:51:49.616583 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:49.616561 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq"] Jul 06 19:51:49.736939 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:49.736912 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f550da-884b-4c83-a2dd-4e4d18296fbb-maas-api-tls\") pod \"maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq\" (UID: \"00f550da-884b-4c83-a2dd-4e4d18296fbb\") " pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" Jul 06 19:51:49.737052 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:49.736960 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f550da-884b-4c83-a2dd-4e4d18296fbb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq\" (UID: \"00f550da-884b-4c83-a2dd-4e4d18296fbb\") " pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" Jul 06 19:51:49.737096 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:49.737041 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmmpj\" (UniqueName: \"kubernetes.io/projected/00f550da-884b-4c83-a2dd-4e4d18296fbb-kube-api-access-wmmpj\") pod \"maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq\" (UID: \"00f550da-884b-4c83-a2dd-4e4d18296fbb\") " pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" Jul 06 19:51:49.837982 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:49.837953 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f550da-884b-4c83-a2dd-4e4d18296fbb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq\" (UID: \"00f550da-884b-4c83-a2dd-4e4d18296fbb\") " pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" Jul 06 19:51:49.838103 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:49.838004 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmmpj\" (UniqueName: \"kubernetes.io/projected/00f550da-884b-4c83-a2dd-4e4d18296fbb-kube-api-access-wmmpj\") pod \"maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq\" (UID: \"00f550da-884b-4c83-a2dd-4e4d18296fbb\") " pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" Jul 06 19:51:49.838103 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:49.838036 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f550da-884b-4c83-a2dd-4e4d18296fbb-maas-api-tls\") pod \"maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq\" (UID: \"00f550da-884b-4c83-a2dd-4e4d18296fbb\") " pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" Jul 06 19:51:49.838632 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:49.838610 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f550da-884b-4c83-a2dd-4e4d18296fbb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq\" (UID: \"00f550da-884b-4c83-a2dd-4e4d18296fbb\") " pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" Jul 06 19:51:49.840506 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:49.840482 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f550da-884b-4c83-a2dd-4e4d18296fbb-maas-api-tls\") pod \"maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq\" (UID: \"00f550da-884b-4c83-a2dd-4e4d18296fbb\") " pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" Jul 06 19:51:49.854489 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:49.854434 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmmpj\" (UniqueName: \"kubernetes.io/projected/00f550da-884b-4c83-a2dd-4e4d18296fbb-kube-api-access-wmmpj\") pod \"maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq\" (UID: \"00f550da-884b-4c83-a2dd-4e4d18296fbb\") " pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" Jul 06 19:51:49.918309 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:49.918285 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" Jul 06 19:51:50.059916 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:50.059892 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq"] Jul 06 19:51:50.062262 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:51:50.062228 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f550da_884b_4c83_a2dd_4e4d18296fbb.slice/crio-5716ec31285b04f01886a019cb463f76872f6d1eafadb731bab1cd815feef689 WatchSource:0}: Error finding container 5716ec31285b04f01886a019cb463f76872f6d1eafadb731bab1cd815feef689: Status 404 returned error can't find the container with id 5716ec31285b04f01886a019cb463f76872f6d1eafadb731bab1cd815feef689 Jul 06 19:51:50.243047 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:51:50.243024 2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a8834b_1e5c_4993_a728_a6c1f88a4828.slice/crio-a3049f74956db629d206a46361eb1b266fbe80bb80ece2bb47bf883b16e97204\": RecentStats: unable to find data in memory cache]" Jul 06 19:51:50.431483 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:50.431455 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" event={"ID":"00f550da-884b-4c83-a2dd-4e4d18296fbb","Type":"ContainerStarted","Data":"5716ec31285b04f01886a019cb463f76872f6d1eafadb731bab1cd815feef689"} Jul 06 19:51:51.438062 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:51.438018 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" event={"ID":"00f550da-884b-4c83-a2dd-4e4d18296fbb","Type":"ContainerStarted","Data":"62842ce52cafe8c3d695a46c1c0b0e4b17e1774b58c27b97fd68f465749973eb"} Jul 06 19:51:51.438436 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:51.438177 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" Jul 06 19:51:51.465142 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:51.465095 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" podStartSLOduration=2.070551896 podStartE2EDuration="2.465081967s" totalImagesPullingTime="394.530071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:51.463260129 +0000 UTC m=+2412.678897657" watchObservedRunningTime="2026-07-06 19:51:51.465081967 +0000 UTC m=+2412.680719496" Jul 06 19:51:55.789401 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:51:55.789369 2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a8834b_1e5c_4993_a728_a6c1f88a4828.slice/crio-a3049f74956db629d206a46361eb1b266fbe80bb80ece2bb47bf883b16e97204\": RecentStats: unable to find data in memory cache]" Jul 06 19:51:57.455405 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:57.455373 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" Jul 06 19:51:58.410607 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:58.410573 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px"] Jul 06 19:51:58.413015 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:58.412996 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" Jul 06 19:51:58.417638 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:58.417616 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-c6207b-serving-cert\"" Jul 06 19:51:58.508670 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:58.508629 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs6nm\" (UniqueName: \"kubernetes.io/projected/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-kube-api-access-xs6nm\") pod \"maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px\" (UID: \"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e\") " pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" Jul 06 19:51:58.508983 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:58.508684 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px\" (UID: \"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e\") " pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" Jul 06 19:51:58.508983 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:58.508734 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-maas-api-tls\") pod \"maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px\" (UID: \"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e\") " pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" Jul 06 19:51:58.609615 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:58.609583 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs6nm\" (UniqueName: \"kubernetes.io/projected/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-kube-api-access-xs6nm\") pod \"maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px\" (UID: \"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e\") " pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" Jul 06 19:51:58.609615 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:58.609617 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px\" (UID: \"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e\") " pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" Jul 06 19:51:58.609824 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:58.609641 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-maas-api-tls\") pod \"maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px\" (UID: \"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e\") " pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" Jul 06 19:51:58.610351 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:58.610331 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px\" (UID: \"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e\") " pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" Jul 06 19:51:58.612070 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:58.612050 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-maas-api-tls\") pod \"maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px\" (UID: \"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e\") " pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" Jul 06 19:51:58.622323 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:58.622303 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px"] Jul 06 19:51:58.671587 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:58.671524 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs6nm\" (UniqueName: \"kubernetes.io/projected/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-kube-api-access-xs6nm\") pod \"maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px\" (UID: \"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e\") " pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" Jul 06 19:51:58.723685 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:58.723645 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" Jul 06 19:51:58.959726 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:58.956122 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px"] Jul 06 19:51:58.972034 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:51:58.972002 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855eadad_c5d9_4ae0_b69a_5ee9410c1c1e.slice/crio-181604ebef948c1852c78e023cf489757a7798674cc32f6fde9c7526282a5a80 WatchSource:0}: Error finding container 181604ebef948c1852c78e023cf489757a7798674cc32f6fde9c7526282a5a80: Status 404 returned error can't find the container with id 181604ebef948c1852c78e023cf489757a7798674cc32f6fde9c7526282a5a80 Jul 06 19:51:59.484310 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:59.484275 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" event={"ID":"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e","Type":"ContainerStarted","Data":"63aafccf53013f4878d52c1a085ef5ae715151c9e81514d44d02d2771566b385"} Jul 06 19:51:59.484440 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:59.484332 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" event={"ID":"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e","Type":"ContainerStarted","Data":"181604ebef948c1852c78e023cf489757a7798674cc32f6fde9c7526282a5a80"} Jul 06 19:51:59.484440 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:59.484372 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" Jul 06 19:51:59.516939 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:51:59.516857 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" podStartSLOduration=1.237182038 podStartE2EDuration="1.516843261s" totalImagesPullingTime="279.661223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:59.515955615 +0000 UTC m=+2420.731593144" watchObservedRunningTime="2026-07-06 19:51:59.516843261 +0000 UTC m=+2420.732480842" Jul 06 19:52:02.500167 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:02.500142 2581 generic.go:356] "Generic (PLEG): container finished" podID="c6b30664-5d84-49fb-a7bf-ff264cb30083" containerID="a41d1043258f1704e5c1096007bcc38a19b488450a56794cff5bceb12e29aa8a" exitCode=0 Jul 06 19:52:02.500565 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:02.500204 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" event={"ID":"c6b30664-5d84-49fb-a7bf-ff264cb30083","Type":"ContainerDied","Data":"a41d1043258f1704e5c1096007bcc38a19b488450a56794cff5bceb12e29aa8a"} Jul 06 19:52:02.500565 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:02.500506 2581 scope.go:117] "RemoveContainer" containerID="a41d1043258f1704e5c1096007bcc38a19b488450a56794cff5bceb12e29aa8a" Jul 06 19:52:02.501414 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:02.501386 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:52:02.501476 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:02.501456 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:52:02.501513 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:02.501498 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 06 19:52:03.423078 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:03.423048 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:52:03.506185 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:03.506144 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" event={"ID":"c6b30664-5d84-49fb-a7bf-ff264cb30083","Type":"ContainerStarted","Data":"3a424f6a544a1320a692435379cc43c1a0e488902561cf196421547e6cb350fa"} Jul 06 19:52:04.166059 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:04.166026 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:52:04.166262 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:04.166149 2581 patch_prober.go:28] interesting pod/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 06 19:52:04.166262 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:04.166191 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" podUID="c6b30664-5d84-49fb-a7bf-ff264cb30083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 06 19:52:05.165900 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:05.165446 2581 patch_prober.go:28] interesting pod/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 06 19:52:05.165900 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:05.165511 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" podUID="c6b30664-5d84-49fb-a7bf-ff264cb30083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 06 19:52:05.291913 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:52:05.291881 2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a8834b_1e5c_4993_a728_a6c1f88a4828.slice/crio-a3049f74956db629d206a46361eb1b266fbe80bb80ece2bb47bf883b16e97204\": RecentStats: unable to find data in memory cache]" Jul 06 19:52:05.495269 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:05.495196 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" Jul 06 19:52:05.798177 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:52:05.798106 2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a8834b_1e5c_4993_a728_a6c1f88a4828.slice/crio-a3049f74956db629d206a46361eb1b266fbe80bb80ece2bb47bf883b16e97204\": RecentStats: unable to find data in memory cache]" Jul 06 19:52:06.165182 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:06.165128 2581 patch_prober.go:28] interesting pod/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 06 19:52:06.165371 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:06.165223 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" podUID="c6b30664-5d84-49fb-a7bf-ff264cb30083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 06 19:52:07.168487 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:07.168459 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:52:07.168911 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:07.168723 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:52:07.169435 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:52:07.169414 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:52:09.549347 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:52:09.549311 2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a8834b_1e5c_4993_a728_a6c1f88a4828.slice/crio-a3049f74956db629d206a46361eb1b266fbe80bb80ece2bb47bf883b16e97204\": RecentStats: unable to find data in memory cache]" Jul 06 19:52:09.549808 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:52:09.549570 2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a8834b_1e5c_4993_a728_a6c1f88a4828.slice/crio-a3049f74956db629d206a46361eb1b266fbe80bb80ece2bb47bf883b16e97204\": RecentStats: unable to find data in memory cache]" Jul 06 19:52:15.839509 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:52:15.839479 2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a8834b_1e5c_4993_a728_a6c1f88a4828.slice/crio-a3049f74956db629d206a46361eb1b266fbe80bb80ece2bb47bf883b16e97204\": RecentStats: unable to find data in memory cache]" Jul 06 19:52:20.239706 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:52:20.239639 2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a8834b_1e5c_4993_a728_a6c1f88a4828.slice/crio-a3049f74956db629d206a46361eb1b266fbe80bb80ece2bb47bf883b16e97204\": RecentStats: unable to find data in memory cache]" Jul 06 19:52:25.848067 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:52:25.848023 2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a8834b_1e5c_4993_a728_a6c1f88a4828.slice/crio-a3049f74956db629d206a46361eb1b266fbe80bb80ece2bb47bf883b16e97204\": RecentStats: unable to find data in memory cache]" Jul 06 19:52:35.282714 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:52:35.282681 2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a8834b_1e5c_4993_a728_a6c1f88a4828.slice/crio-a3049f74956db629d206a46361eb1b266fbe80bb80ece2bb47bf883b16e97204\": RecentStats: unable to find data in memory cache]" Jul 06 19:52:35.857010 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:52:35.856977 2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a8834b_1e5c_4993_a728_a6c1f88a4828.slice/crio-a3049f74956db629d206a46361eb1b266fbe80bb80ece2bb47bf883b16e97204\": RecentStats: unable to find data in memory cache]" Jul 06 19:56:39.404343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:56:39.404310 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:56:39.411879 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:56:39.411854 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:57:40.696950 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:40.696915 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq"] Jul 06 19:57:40.697481 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:40.697175 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" podUID="00f550da-884b-4c83-a2dd-4e4d18296fbb" containerName="maas-api" containerID="cri-o://62842ce52cafe8c3d695a46c1c0b0e4b17e1774b58c27b97fd68f465749973eb" gracePeriod=30 Jul 06 19:57:40.945636 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:40.945609 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" Jul 06 19:57:40.995456 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:40.995357 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmmpj\" (UniqueName: \"kubernetes.io/projected/00f550da-884b-4c83-a2dd-4e4d18296fbb-kube-api-access-wmmpj\") pod \"00f550da-884b-4c83-a2dd-4e4d18296fbb\" (UID: \"00f550da-884b-4c83-a2dd-4e4d18296fbb\") " Jul 06 19:57:40.995634 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:40.995549 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f550da-884b-4c83-a2dd-4e4d18296fbb-openshift-service-ca\") pod \"00f550da-884b-4c83-a2dd-4e4d18296fbb\" (UID: \"00f550da-884b-4c83-a2dd-4e4d18296fbb\") " Jul 06 19:57:40.995634 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:40.995615 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f550da-884b-4c83-a2dd-4e4d18296fbb-maas-api-tls\") pod \"00f550da-884b-4c83-a2dd-4e4d18296fbb\" (UID: \"00f550da-884b-4c83-a2dd-4e4d18296fbb\") " Jul 06 19:57:40.995885 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:40.995859 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f550da-884b-4c83-a2dd-4e4d18296fbb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "00f550da-884b-4c83-a2dd-4e4d18296fbb" (UID: "00f550da-884b-4c83-a2dd-4e4d18296fbb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:57:40.995959 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:40.995940 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f550da-884b-4c83-a2dd-4e4d18296fbb-openshift-service-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:57:40.997527 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:40.997496 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f550da-884b-4c83-a2dd-4e4d18296fbb-kube-api-access-wmmpj" (OuterVolumeSpecName: "kube-api-access-wmmpj") pod "00f550da-884b-4c83-a2dd-4e4d18296fbb" (UID: "00f550da-884b-4c83-a2dd-4e4d18296fbb"). InnerVolumeSpecName "kube-api-access-wmmpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:57:40.997645 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:40.997526 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f550da-884b-4c83-a2dd-4e4d18296fbb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "00f550da-884b-4c83-a2dd-4e4d18296fbb" (UID: "00f550da-884b-4c83-a2dd-4e4d18296fbb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:57:41.081853 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:41.081809 2581 generic.go:356] "Generic (PLEG): container finished" podID="00f550da-884b-4c83-a2dd-4e4d18296fbb" containerID="62842ce52cafe8c3d695a46c1c0b0e4b17e1774b58c27b97fd68f465749973eb" exitCode=0 Jul 06 19:57:41.081853 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:41.081858 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" Jul 06 19:57:41.082083 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:41.081853 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" event={"ID":"00f550da-884b-4c83-a2dd-4e4d18296fbb","Type":"ContainerDied","Data":"62842ce52cafe8c3d695a46c1c0b0e4b17e1774b58c27b97fd68f465749973eb"} Jul 06 19:57:41.082083 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:41.081974 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq" event={"ID":"00f550da-884b-4c83-a2dd-4e4d18296fbb","Type":"ContainerDied","Data":"5716ec31285b04f01886a019cb463f76872f6d1eafadb731bab1cd815feef689"} Jul 06 19:57:41.082083 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:41.081997 2581 scope.go:117] "RemoveContainer" containerID="62842ce52cafe8c3d695a46c1c0b0e4b17e1774b58c27b97fd68f465749973eb" Jul 06 19:57:41.091294 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:41.091277 2581 scope.go:117] "RemoveContainer" containerID="62842ce52cafe8c3d695a46c1c0b0e4b17e1774b58c27b97fd68f465749973eb" Jul 06 19:57:41.091553 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:57:41.091536 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62842ce52cafe8c3d695a46c1c0b0e4b17e1774b58c27b97fd68f465749973eb\": container with ID starting with 62842ce52cafe8c3d695a46c1c0b0e4b17e1774b58c27b97fd68f465749973eb not found: ID does not exist" containerID="62842ce52cafe8c3d695a46c1c0b0e4b17e1774b58c27b97fd68f465749973eb" Jul 06 19:57:41.091613 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:41.091567 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62842ce52cafe8c3d695a46c1c0b0e4b17e1774b58c27b97fd68f465749973eb"} err="failed to get container status \"62842ce52cafe8c3d695a46c1c0b0e4b17e1774b58c27b97fd68f465749973eb\": rpc error: code = NotFound desc = could not find container \"62842ce52cafe8c3d695a46c1c0b0e4b17e1774b58c27b97fd68f465749973eb\": container with ID starting with 62842ce52cafe8c3d695a46c1c0b0e4b17e1774b58c27b97fd68f465749973eb not found: ID does not exist" Jul 06 19:57:41.097023 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:41.097005 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00f550da-884b-4c83-a2dd-4e4d18296fbb-maas-api-tls\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:57:41.097085 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:41.097026 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmmpj\" (UniqueName: \"kubernetes.io/projected/00f550da-884b-4c83-a2dd-4e4d18296fbb-kube-api-access-wmmpj\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:57:41.127557 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:41.127526 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq"] Jul 06 19:57:41.139459 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:41.139434 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2240fd-94dfd9f4c-sq9rq"] Jul 06 19:57:42.147067 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:57:42.147036 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00f550da-884b-4c83-a2dd-4e4d18296fbb" path="/var/lib/kubelet/pods/00f550da-884b-4c83-a2dd-4e4d18296fbb/volumes" Jul 06 19:58:22.578404 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:22.578321 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n"] Jul 06 19:58:22.578954 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:22.578555 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" podUID="c6b30664-5d84-49fb-a7bf-ff264cb30083" containerName="istio-proxy" containerID="cri-o://3a424f6a544a1320a692435379cc43c1a0e488902561cf196421547e6cb350fa" gracePeriod=30 Jul 06 19:58:23.746071 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:23.746036 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px"] Jul 06 19:58:23.746499 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:23.746290 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" podUID="855eadad-c5d9-4ae0-b69a-5ee9410c1c1e" containerName="maas-api" containerID="cri-o://63aafccf53013f4878d52c1a085ef5ae715151c9e81514d44d02d2771566b385" gracePeriod=30 Jul 06 19:58:23.996027 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:23.996000 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" Jul 06 19:58:24.083143 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.083080 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs6nm\" (UniqueName: \"kubernetes.io/projected/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-kube-api-access-xs6nm\") pod \"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e\" (UID: \"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e\") " Jul 06 19:58:24.083256 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.083151 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-maas-api-tls\") pod \"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e\" (UID: \"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e\") " Jul 06 19:58:24.083256 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.083200 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-openshift-service-ca\") pod \"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e\" (UID: \"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e\") " Jul 06 19:58:24.083589 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.083561 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "855eadad-c5d9-4ae0-b69a-5ee9410c1c1e" (UID: "855eadad-c5d9-4ae0-b69a-5ee9410c1c1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:58:24.085061 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.085029 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-kube-api-access-xs6nm" (OuterVolumeSpecName: "kube-api-access-xs6nm") pod "855eadad-c5d9-4ae0-b69a-5ee9410c1c1e" (UID: "855eadad-c5d9-4ae0-b69a-5ee9410c1c1e"). InnerVolumeSpecName "kube-api-access-xs6nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:58:24.085146 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.085094 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "855eadad-c5d9-4ae0-b69a-5ee9410c1c1e" (UID: "855eadad-c5d9-4ae0-b69a-5ee9410c1c1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:58:24.183923 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.183895 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-maas-api-tls\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:58:24.184058 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.183941 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-openshift-service-ca\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:58:24.184058 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.183951 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs6nm\" (UniqueName: \"kubernetes.io/projected/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e-kube-api-access-xs6nm\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:58:24.275382 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.275346 2581 generic.go:356] "Generic (PLEG): container finished" podID="855eadad-c5d9-4ae0-b69a-5ee9410c1c1e" containerID="63aafccf53013f4878d52c1a085ef5ae715151c9e81514d44d02d2771566b385" exitCode=0 Jul 06 19:58:24.275552 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.275410 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" Jul 06 19:58:24.275552 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.275435 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" event={"ID":"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e","Type":"ContainerDied","Data":"63aafccf53013f4878d52c1a085ef5ae715151c9e81514d44d02d2771566b385"} Jul 06 19:58:24.275552 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.275512 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px" event={"ID":"855eadad-c5d9-4ae0-b69a-5ee9410c1c1e","Type":"ContainerDied","Data":"181604ebef948c1852c78e023cf489757a7798674cc32f6fde9c7526282a5a80"} Jul 06 19:58:24.275552 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.275534 2581 scope.go:117] "RemoveContainer" containerID="63aafccf53013f4878d52c1a085ef5ae715151c9e81514d44d02d2771566b385" Jul 06 19:58:24.284308 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.284291 2581 scope.go:117] "RemoveContainer" containerID="63aafccf53013f4878d52c1a085ef5ae715151c9e81514d44d02d2771566b385" Jul 06 19:58:24.284616 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:58:24.284598 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63aafccf53013f4878d52c1a085ef5ae715151c9e81514d44d02d2771566b385\": container with ID starting with 63aafccf53013f4878d52c1a085ef5ae715151c9e81514d44d02d2771566b385 not found: ID does not exist" containerID="63aafccf53013f4878d52c1a085ef5ae715151c9e81514d44d02d2771566b385" Jul 06 19:58:24.284690 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.284625 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63aafccf53013f4878d52c1a085ef5ae715151c9e81514d44d02d2771566b385"} err="failed to get container status \"63aafccf53013f4878d52c1a085ef5ae715151c9e81514d44d02d2771566b385\": rpc error: code = NotFound desc = could not find container \"63aafccf53013f4878d52c1a085ef5ae715151c9e81514d44d02d2771566b385\": container with ID starting with 63aafccf53013f4878d52c1a085ef5ae715151c9e81514d44d02d2771566b385 not found: ID does not exist" Jul 06 19:58:24.292281 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.292256 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px"] Jul 06 19:58:24.298213 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:24.298194 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c6207b-7cdbd477d6-kp7px"] Jul 06 19:58:26.148500 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:26.148467 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="855eadad-c5d9-4ae0-b69a-5ee9410c1c1e" path="/var/lib/kubelet/pods/855eadad-c5d9-4ae0-b69a-5ee9410c1c1e/volumes" Jul 06 19:58:27.823841 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.823813 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:58:27.914581 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.914559 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-credential-socket\") pod \"c6b30664-5d84-49fb-a7bf-ff264cb30083\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " Jul 06 19:58:27.914746 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.914596 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-token\") pod \"c6b30664-5d84-49fb-a7bf-ff264cb30083\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " Jul 06 19:58:27.914746 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.914615 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjpcn\" (UniqueName: \"kubernetes.io/projected/c6b30664-5d84-49fb-a7bf-ff264cb30083-kube-api-access-sjpcn\") pod \"c6b30664-5d84-49fb-a7bf-ff264cb30083\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " Jul 06 19:58:27.914746 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.914707 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-data\") pod \"c6b30664-5d84-49fb-a7bf-ff264cb30083\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " Jul 06 19:58:27.914933 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.914761 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-envoy\") pod \"c6b30664-5d84-49fb-a7bf-ff264cb30083\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " Jul 06 19:58:27.914933 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.914787 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6b30664-5d84-49fb-a7bf-ff264cb30083-istiod-ca-cert\") pod \"c6b30664-5d84-49fb-a7bf-ff264cb30083\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " Jul 06 19:58:27.914933 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.914822 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-podinfo\") pod \"c6b30664-5d84-49fb-a7bf-ff264cb30083\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " Jul 06 19:58:27.914933 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.914877 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-workload-socket\") pod \"c6b30664-5d84-49fb-a7bf-ff264cb30083\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " Jul 06 19:58:27.914933 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.914909 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c6b30664-5d84-49fb-a7bf-ff264cb30083" (UID: "c6b30664-5d84-49fb-a7bf-ff264cb30083"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:58:27.915244 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.914992 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-workload-certs\") pod \"c6b30664-5d84-49fb-a7bf-ff264cb30083\" (UID: \"c6b30664-5d84-49fb-a7bf-ff264cb30083\") " Jul 06 19:58:27.915244 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.915118 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c6b30664-5d84-49fb-a7bf-ff264cb30083" (UID: "c6b30664-5d84-49fb-a7bf-ff264cb30083"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:58:27.915353 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.915287 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-workload-socket\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:58:27.915353 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.915294 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c6b30664-5d84-49fb-a7bf-ff264cb30083" (UID: "c6b30664-5d84-49fb-a7bf-ff264cb30083"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:58:27.915353 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.915306 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-credential-socket\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:58:27.915597 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.915571 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b30664-5d84-49fb-a7bf-ff264cb30083-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c6b30664-5d84-49fb-a7bf-ff264cb30083" (UID: "c6b30664-5d84-49fb-a7bf-ff264cb30083"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:58:27.915641 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.915586 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-data" (OuterVolumeSpecName: "istio-data") pod "c6b30664-5d84-49fb-a7bf-ff264cb30083" (UID: "c6b30664-5d84-49fb-a7bf-ff264cb30083"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:58:27.916837 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.916812 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-token" (OuterVolumeSpecName: "istio-token") pod "c6b30664-5d84-49fb-a7bf-ff264cb30083" (UID: "c6b30664-5d84-49fb-a7bf-ff264cb30083"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:58:27.916837 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.916821 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b30664-5d84-49fb-a7bf-ff264cb30083-kube-api-access-sjpcn" (OuterVolumeSpecName: "kube-api-access-sjpcn") pod "c6b30664-5d84-49fb-a7bf-ff264cb30083" (UID: "c6b30664-5d84-49fb-a7bf-ff264cb30083"). InnerVolumeSpecName "kube-api-access-sjpcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:58:27.917153 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.917125 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c6b30664-5d84-49fb-a7bf-ff264cb30083" (UID: "c6b30664-5d84-49fb-a7bf-ff264cb30083"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 19:58:27.917238 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:27.917160 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c6b30664-5d84-49fb-a7bf-ff264cb30083" (UID: "c6b30664-5d84-49fb-a7bf-ff264cb30083"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:58:28.016698 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.016648 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-envoy\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:58:28.016698 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.016693 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6b30664-5d84-49fb-a7bf-ff264cb30083-istiod-ca-cert\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:58:28.016698 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.016701 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-podinfo\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:58:28.016907 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.016711 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-workload-certs\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:58:28.016907 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.016720 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-token\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:58:28.016907 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.016728 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjpcn\" (UniqueName: \"kubernetes.io/projected/c6b30664-5d84-49fb-a7bf-ff264cb30083-kube-api-access-sjpcn\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:58:28.016907 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.016738 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6b30664-5d84-49fb-a7bf-ff264cb30083-istio-data\") on node \"ip-10-0-143-192.ec2.internal\" DevicePath \"\"" Jul 06 19:58:28.296140 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.296053 2581 generic.go:356] "Generic (PLEG): container finished" podID="c6b30664-5d84-49fb-a7bf-ff264cb30083" containerID="3a424f6a544a1320a692435379cc43c1a0e488902561cf196421547e6cb350fa" exitCode=0 Jul 06 19:58:28.296140 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.296108 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" Jul 06 19:58:28.296362 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.296137 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" event={"ID":"c6b30664-5d84-49fb-a7bf-ff264cb30083","Type":"ContainerDied","Data":"3a424f6a544a1320a692435379cc43c1a0e488902561cf196421547e6cb350fa"} Jul 06 19:58:28.296362 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.296190 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n" event={"ID":"c6b30664-5d84-49fb-a7bf-ff264cb30083","Type":"ContainerDied","Data":"eb6ec81182deeabbaa0e42d891c8e264f9dbf49306c5ec4e646ed6daa5b28969"} Jul 06 19:58:28.296362 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.296213 2581 scope.go:117] "RemoveContainer" containerID="3a424f6a544a1320a692435379cc43c1a0e488902561cf196421547e6cb350fa" Jul 06 19:58:28.304915 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.304896 2581 scope.go:117] "RemoveContainer" containerID="a41d1043258f1704e5c1096007bcc38a19b488450a56794cff5bceb12e29aa8a" Jul 06 19:58:28.312433 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.312408 2581 scope.go:117] "RemoveContainer" containerID="3a424f6a544a1320a692435379cc43c1a0e488902561cf196421547e6cb350fa" Jul 06 19:58:28.312705 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:58:28.312685 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a424f6a544a1320a692435379cc43c1a0e488902561cf196421547e6cb350fa\": container with ID starting with 3a424f6a544a1320a692435379cc43c1a0e488902561cf196421547e6cb350fa not found: ID does not exist" containerID="3a424f6a544a1320a692435379cc43c1a0e488902561cf196421547e6cb350fa" Jul 06 19:58:28.312775 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.312712 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a424f6a544a1320a692435379cc43c1a0e488902561cf196421547e6cb350fa"} err="failed to get container status \"3a424f6a544a1320a692435379cc43c1a0e488902561cf196421547e6cb350fa\": rpc error: code = NotFound desc = could not find container \"3a424f6a544a1320a692435379cc43c1a0e488902561cf196421547e6cb350fa\": container with ID starting with 3a424f6a544a1320a692435379cc43c1a0e488902561cf196421547e6cb350fa not found: ID does not exist" Jul 06 19:58:28.312775 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.312727 2581 scope.go:117] "RemoveContainer" containerID="a41d1043258f1704e5c1096007bcc38a19b488450a56794cff5bceb12e29aa8a" Jul 06 19:58:28.312985 ip-10-0-143-192 kubenswrapper[2581]: E0706 19:58:28.312963 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41d1043258f1704e5c1096007bcc38a19b488450a56794cff5bceb12e29aa8a\": container with ID starting with a41d1043258f1704e5c1096007bcc38a19b488450a56794cff5bceb12e29aa8a not found: ID does not exist" containerID="a41d1043258f1704e5c1096007bcc38a19b488450a56794cff5bceb12e29aa8a" Jul 06 19:58:28.313029 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.312995 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41d1043258f1704e5c1096007bcc38a19b488450a56794cff5bceb12e29aa8a"} err="failed to get container status \"a41d1043258f1704e5c1096007bcc38a19b488450a56794cff5bceb12e29aa8a\": rpc error: code = NotFound desc = could not find container \"a41d1043258f1704e5c1096007bcc38a19b488450a56794cff5bceb12e29aa8a\": container with ID starting with a41d1043258f1704e5c1096007bcc38a19b488450a56794cff5bceb12e29aa8a not found: ID does not exist" Jul 06 19:58:28.333526 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.333498 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n"] Jul 06 19:58:28.335312 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:28.335292 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-2240fd-openshift-default-9546bf67-q8b9n"] Jul 06 19:58:30.147542 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:30.147500 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6b30664-5d84-49fb-a7bf-ff264cb30083" path="/var/lib/kubelet/pods/c6b30664-5d84-49fb-a7bf-ff264cb30083/volumes" Jul 06 19:58:36.127397 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:36.127347 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-dfb658bc5-kzrgj_20d0828b-9eb2-4926-9fcb-31bd6e87f727/maas-api/0.log" Jul 06 19:58:36.247012 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:36.246978 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722770-ft6x6_177b0d59-f9b3-42f5-b2b4-60bf17264d46/cleanup/0.log" Jul 06 19:58:36.362949 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:36.362916 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722785-7678z_fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e/cleanup/0.log" Jul 06 19:58:36.859619 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:36.859593 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f978bf57-g64zj_3a43c343-443c-442d-80b1-8b2dc50294ba/copy-manifests/0.log" Jul 06 19:58:36.866058 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:36.866023 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f978bf57-g64zj_3a43c343-443c-442d-80b1-8b2dc50294ba/manager/0.log" Jul 06 19:58:39.269219 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:39.269173 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-p57v5_52591f3b-7bed-4f00-a64e-ae8f96294957/manager/0.log" Jul 06 19:58:40.028866 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:40.028836 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-m2x4w_8cf813ee-2998-49c0-81b0-ba574c55e85f/discovery/0.log" Jul 06 19:58:40.795574 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:40.795545 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5df75489c9-898zs_51a04452-ffcd-45fe-b844-519e1c90426b/router/0.log" Jul 06 19:58:41.397052 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:41.397019 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-z7qdx_10c7f816-51e2-4e00-b92a-738bf261e73a/main/0.log" Jul 06 19:58:45.861302 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.861268 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bq5kf/must-gather-2tpbq"] Jul 06 19:58:45.861714 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.861585 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6b30664-5d84-49fb-a7bf-ff264cb30083" containerName="istio-proxy" Jul 06 19:58:45.861714 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.861596 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6b30664-5d84-49fb-a7bf-ff264cb30083" containerName="istio-proxy" Jul 06 19:58:45.861714 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.861607 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6b30664-5d84-49fb-a7bf-ff264cb30083" containerName="istio-proxy" Jul 06 19:58:45.861714 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.861613 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6b30664-5d84-49fb-a7bf-ff264cb30083" containerName="istio-proxy" Jul 06 19:58:45.861714 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.861631 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00f550da-884b-4c83-a2dd-4e4d18296fbb" containerName="maas-api" Jul 06 19:58:45.861714 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.861636 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="00f550da-884b-4c83-a2dd-4e4d18296fbb" containerName="maas-api" Jul 06 19:58:45.861714 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.861645 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="855eadad-c5d9-4ae0-b69a-5ee9410c1c1e" containerName="maas-api" Jul 06 19:58:45.861714 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.861665 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="855eadad-c5d9-4ae0-b69a-5ee9410c1c1e" containerName="maas-api" Jul 06 19:58:45.861714 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.861712 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="855eadad-c5d9-4ae0-b69a-5ee9410c1c1e" containerName="maas-api" Jul 06 19:58:45.862003 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.861722 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6b30664-5d84-49fb-a7bf-ff264cb30083" containerName="istio-proxy" Jul 06 19:58:45.862003 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.861727 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="00f550da-884b-4c83-a2dd-4e4d18296fbb" containerName="maas-api" Jul 06 19:58:45.862003 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.861732 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6b30664-5d84-49fb-a7bf-ff264cb30083" containerName="istio-proxy" Jul 06 19:58:45.864490 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.864472 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq5kf/must-gather-2tpbq" Jul 06 19:58:45.867463 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.867443 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bq5kf\"/\"default-dockercfg-xrlgm\"" Jul 06 19:58:45.868143 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.868124 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bq5kf\"/\"openshift-service-ca.crt\"" Jul 06 19:58:45.868208 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.868150 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bq5kf\"/\"kube-root-ca.crt\"" Jul 06 19:58:45.879099 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.879077 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq5kf/must-gather-2tpbq"] Jul 06 19:58:45.970609 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.970578 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6922156e-4184-41ba-98cd-e242365f5e63-must-gather-output\") pod \"must-gather-2tpbq\" (UID: \"6922156e-4184-41ba-98cd-e242365f5e63\") " pod="openshift-must-gather-bq5kf/must-gather-2tpbq" Jul 06 19:58:45.970609 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:45.970615 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwxgv\" (UniqueName: \"kubernetes.io/projected/6922156e-4184-41ba-98cd-e242365f5e63-kube-api-access-gwxgv\") pod \"must-gather-2tpbq\" (UID: \"6922156e-4184-41ba-98cd-e242365f5e63\") " pod="openshift-must-gather-bq5kf/must-gather-2tpbq" Jul 06 19:58:46.071951 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:46.071915 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6922156e-4184-41ba-98cd-e242365f5e63-must-gather-output\") pod \"must-gather-2tpbq\" (UID: \"6922156e-4184-41ba-98cd-e242365f5e63\") " pod="openshift-must-gather-bq5kf/must-gather-2tpbq" Jul 06 19:58:46.071951 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:46.071959 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwxgv\" (UniqueName: \"kubernetes.io/projected/6922156e-4184-41ba-98cd-e242365f5e63-kube-api-access-gwxgv\") pod \"must-gather-2tpbq\" (UID: \"6922156e-4184-41ba-98cd-e242365f5e63\") " pod="openshift-must-gather-bq5kf/must-gather-2tpbq" Jul 06 19:58:46.072269 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:46.072246 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6922156e-4184-41ba-98cd-e242365f5e63-must-gather-output\") pod \"must-gather-2tpbq\" (UID: \"6922156e-4184-41ba-98cd-e242365f5e63\") " pod="openshift-must-gather-bq5kf/must-gather-2tpbq" Jul 06 19:58:46.081128 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:46.081104 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwxgv\" (UniqueName: \"kubernetes.io/projected/6922156e-4184-41ba-98cd-e242365f5e63-kube-api-access-gwxgv\") pod \"must-gather-2tpbq\" (UID: \"6922156e-4184-41ba-98cd-e242365f5e63\") " pod="openshift-must-gather-bq5kf/must-gather-2tpbq" Jul 06 19:58:46.172966 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:46.172940 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq5kf/must-gather-2tpbq" Jul 06 19:58:46.318154 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:46.318132 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq5kf/must-gather-2tpbq"] Jul 06 19:58:46.320529 ip-10-0-143-192 kubenswrapper[2581]: W0706 19:58:46.320499 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6922156e_4184_41ba_98cd_e242365f5e63.slice/crio-631a19ba124ef1fc6d920858ea84c918f2332f4cd8d627a1b54bdecbdea641c7 WatchSource:0}: Error finding container 631a19ba124ef1fc6d920858ea84c918f2332f4cd8d627a1b54bdecbdea641c7: Status 404 returned error can't find the container with id 631a19ba124ef1fc6d920858ea84c918f2332f4cd8d627a1b54bdecbdea641c7 Jul 06 19:58:46.322227 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:46.322211 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:58:46.376318 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:46.376293 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq5kf/must-gather-2tpbq" event={"ID":"6922156e-4184-41ba-98cd-e242365f5e63","Type":"ContainerStarted","Data":"631a19ba124ef1fc6d920858ea84c918f2332f4cd8d627a1b54bdecbdea641c7"} Jul 06 19:58:48.388534 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:48.388501 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq5kf/must-gather-2tpbq" event={"ID":"6922156e-4184-41ba-98cd-e242365f5e63","Type":"ContainerStarted","Data":"4a4c1b6757650e71f5353c07b6d1980422d1d028381ea2afe15f559bb71cadb8"} Jul 06 19:58:48.388974 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:48.388543 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq5kf/must-gather-2tpbq" event={"ID":"6922156e-4184-41ba-98cd-e242365f5e63","Type":"ContainerStarted","Data":"495c5cc44a60ba79acffe34b8d72ad54549052d21838a3d00c8c5da8322608d1"} Jul 06 19:58:48.409065 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:48.409014 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bq5kf/must-gather-2tpbq" podStartSLOduration=2.093780552 podStartE2EDuration="3.408997425s" totalImagesPullingTime="1.315216873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:58:48.407336368 +0000 UTC m=+2829.622973899" watchObservedRunningTime="2026-07-06 19:58:48.408997425 +0000 UTC m=+2829.624634953" Jul 06 19:58:49.187713 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:49.187670 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sk86p_fee777de-2d7a-4470-8a1d-b64c9fad8390/global-pull-secret-syncer/0.log" Jul 06 19:58:49.330587 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:49.330546 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zlhst_3d885e22-9b77-4736-a3a4-a5652effa44e/konnectivity-agent/0.log" Jul 06 19:58:49.388071 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:49.388044 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-192.ec2.internal_1b77e81dd26018ebe7401261e83c3a4d/haproxy/0.log" Jul 06 19:58:54.618908 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:54.618766 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-p57v5_52591f3b-7bed-4f00-a64e-ae8f96294957/manager/0.log" Jul 06 19:58:56.360124 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:56.360048 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eae4b624-b77e-45c1-9ae6-9fb8454777ea/alertmanager/0.log" Jul 06 19:58:56.384364 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:56.384330 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eae4b624-b77e-45c1-9ae6-9fb8454777ea/config-reloader/0.log" Jul 06 19:58:56.416787 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:56.416705 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eae4b624-b77e-45c1-9ae6-9fb8454777ea/kube-rbac-proxy-web/0.log" Jul 06 19:58:56.443244 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:56.443212 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eae4b624-b77e-45c1-9ae6-9fb8454777ea/kube-rbac-proxy/0.log" Jul 06 19:58:56.469455 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:56.469417 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eae4b624-b77e-45c1-9ae6-9fb8454777ea/kube-rbac-proxy-metric/0.log" Jul 06 19:58:56.496715 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:56.496679 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eae4b624-b77e-45c1-9ae6-9fb8454777ea/prom-label-proxy/0.log" Jul 06 19:58:56.522603 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:56.522559 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_eae4b624-b77e-45c1-9ae6-9fb8454777ea/init-config-reloader/0.log" Jul 06 19:58:56.825552 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:56.825521 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9n79t_0f3e8e0b-9d86-462f-b5e5-0849053916a0/node-exporter/0.log" Jul 06 19:58:56.854964 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:56.854936 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9n79t_0f3e8e0b-9d86-462f-b5e5-0849053916a0/kube-rbac-proxy/0.log" Jul 06 19:58:56.885790 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:56.885756 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9n79t_0f3e8e0b-9d86-462f-b5e5-0849053916a0/init-textfile/0.log" Jul 06 19:58:57.419343 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:57.419299 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b988d8588-5lqd9_e9835372-f08c-4f0b-afcc-5a65a237e012/telemeter-client/0.log" Jul 06 19:58:57.449342 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:57.449302 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b988d8588-5lqd9_e9835372-f08c-4f0b-afcc-5a65a237e012/reload/0.log" Jul 06 19:58:57.474647 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:57.474614 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-b988d8588-5lqd9_e9835372-f08c-4f0b-afcc-5a65a237e012/kube-rbac-proxy/0.log" Jul 06 19:58:59.110103 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.110064 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722"] Jul 06 19:58:59.115749 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.115719 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:58:59.123573 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.123542 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722"] Jul 06 19:58:59.209746 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.209704 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8be44b3a-b140-43dd-b7b9-dbc4214c0f6f-sys\") pod \"perf-node-gather-daemonset-f2722\" (UID: \"8be44b3a-b140-43dd-b7b9-dbc4214c0f6f\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:58:59.209945 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.209780 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8be44b3a-b140-43dd-b7b9-dbc4214c0f6f-lib-modules\") pod \"perf-node-gather-daemonset-f2722\" (UID: \"8be44b3a-b140-43dd-b7b9-dbc4214c0f6f\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:58:59.209945 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.209823 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8be44b3a-b140-43dd-b7b9-dbc4214c0f6f-podres\") pod \"perf-node-gather-daemonset-f2722\" (UID: \"8be44b3a-b140-43dd-b7b9-dbc4214c0f6f\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:58:59.209945 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.209865 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwbb4\" (UniqueName: \"kubernetes.io/projected/8be44b3a-b140-43dd-b7b9-dbc4214c0f6f-kube-api-access-bwbb4\") pod \"perf-node-gather-daemonset-f2722\" (UID: \"8be44b3a-b140-43dd-b7b9-dbc4214c0f6f\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:58:59.209945 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.209918 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8be44b3a-b140-43dd-b7b9-dbc4214c0f6f-proc\") pod \"perf-node-gather-daemonset-f2722\" (UID: \"8be44b3a-b140-43dd-b7b9-dbc4214c0f6f\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:58:59.310745 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.310704 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8be44b3a-b140-43dd-b7b9-dbc4214c0f6f-lib-modules\") pod \"perf-node-gather-daemonset-f2722\" (UID: \"8be44b3a-b140-43dd-b7b9-dbc4214c0f6f\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:58:59.310954 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.310773 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8be44b3a-b140-43dd-b7b9-dbc4214c0f6f-podres\") pod \"perf-node-gather-daemonset-f2722\" (UID: \"8be44b3a-b140-43dd-b7b9-dbc4214c0f6f\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:58:59.310954 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.310833 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwbb4\" (UniqueName: \"kubernetes.io/projected/8be44b3a-b140-43dd-b7b9-dbc4214c0f6f-kube-api-access-bwbb4\") pod \"perf-node-gather-daemonset-f2722\" (UID: \"8be44b3a-b140-43dd-b7b9-dbc4214c0f6f\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:58:59.310954 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.310865 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8be44b3a-b140-43dd-b7b9-dbc4214c0f6f-proc\") pod \"perf-node-gather-daemonset-f2722\" (UID: \"8be44b3a-b140-43dd-b7b9-dbc4214c0f6f\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:58:59.310954 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.310937 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8be44b3a-b140-43dd-b7b9-dbc4214c0f6f-lib-modules\") pod \"perf-node-gather-daemonset-f2722\" (UID: \"8be44b3a-b140-43dd-b7b9-dbc4214c0f6f\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:58:59.311198 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.310954 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8be44b3a-b140-43dd-b7b9-dbc4214c0f6f-sys\") pod \"perf-node-gather-daemonset-f2722\" (UID: \"8be44b3a-b140-43dd-b7b9-dbc4214c0f6f\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:58:59.311198 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.311097 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8be44b3a-b140-43dd-b7b9-dbc4214c0f6f-proc\") pod \"perf-node-gather-daemonset-f2722\" (UID: \"8be44b3a-b140-43dd-b7b9-dbc4214c0f6f\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:58:59.311198 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.311152 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8be44b3a-b140-43dd-b7b9-dbc4214c0f6f-sys\") pod \"perf-node-gather-daemonset-f2722\" (UID: \"8be44b3a-b140-43dd-b7b9-dbc4214c0f6f\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:58:59.311394 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.311261 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8be44b3a-b140-43dd-b7b9-dbc4214c0f6f-podres\") pod \"perf-node-gather-daemonset-f2722\" (UID: \"8be44b3a-b140-43dd-b7b9-dbc4214c0f6f\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:58:59.320711 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.320680 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwbb4\" (UniqueName: \"kubernetes.io/projected/8be44b3a-b140-43dd-b7b9-dbc4214c0f6f-kube-api-access-bwbb4\") pod \"perf-node-gather-daemonset-f2722\" (UID: \"8be44b3a-b140-43dd-b7b9-dbc4214c0f6f\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:58:59.435571 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.435287 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:58:59.597135 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.597057 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/1.log" Jul 06 19:58:59.606345 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.606300 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ntfk2_20ac7de1-b189-4c87-b6a0-9b45a480d3f0/console-operator/2.log" Jul 06 19:58:59.633161 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:58:59.632294 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722"] Jul 06 19:59:00.482914 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:00.482867 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" event={"ID":"8be44b3a-b140-43dd-b7b9-dbc4214c0f6f","Type":"ContainerStarted","Data":"2337b8d6b7df99dc97e9879a0e1462dc9bc270fc9829f87a586a4d011bd1a177"} Jul 06 19:59:00.483326 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:00.482927 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" event={"ID":"8be44b3a-b140-43dd-b7b9-dbc4214c0f6f","Type":"ContainerStarted","Data":"31bf2d6c1f53d2c41a08529cb5c1281a9d55aa2f379a4eadb9355aed163306c2"} Jul 06 19:59:00.483326 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:00.482958 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:59:00.502536 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:00.502478 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" podStartSLOduration=1.502462418 podStartE2EDuration="1.502462418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:00.500417199 +0000 UTC m=+2841.716054756" watchObservedRunningTime="2026-07-06 19:59:00.502462418 +0000 UTC m=+2841.718099948" Jul 06 19:59:01.753416 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:01.753392 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4rhqw_9c87bed4-66aa-420d-98e1-8071ede06a4b/dns/0.log" Jul 06 19:59:01.785452 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:01.785419 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4rhqw_9c87bed4-66aa-420d-98e1-8071ede06a4b/kube-rbac-proxy/0.log" Jul 06 19:59:01.926771 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:01.926743 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g6qxq_0e43ffe8-6267-4b64-94c9-8b6ce791a0b2/dns-node-resolver/0.log" Jul 06 19:59:02.622320 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:02.622271 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-74b5cb95d8-x7rkn_63ea9f5d-efb4-4d37-9690-183d90437225/registry/0.log" Jul 06 19:59:02.705316 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:02.705279 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d9l66_b9cd29e9-e9ac-4a2e-90f2-8e410ed0ede9/node-ca/0.log" Jul 06 19:59:03.776644 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:03.776611 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-m2x4w_8cf813ee-2998-49c0-81b0-ba574c55e85f/discovery/0.log" Jul 06 19:59:03.993455 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:03.993418 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5df75489c9-898zs_51a04452-ffcd-45fe-b844-519e1c90426b/router/0.log" Jul 06 19:59:04.669213 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:04.669183 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6xknf_218097a1-0eb0-4a54-af89-ba9b3d5c4586/serve-healthcheck-canary/0.log" Jul 06 19:59:05.171737 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:05.171664 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-ftqzp_a65794b0-70cc-4b6f-bc1c-cd741bfd9de6/insights-operator/0.log" Jul 06 19:59:05.173378 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:05.173355 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-ftqzp_a65794b0-70cc-4b6f-bc1c-cd741bfd9de6/insights-operator/1.log" Jul 06 19:59:05.194006 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:05.193981 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6djr7_c099894f-48eb-4431-acca-5464590389f8/kube-rbac-proxy/0.log" Jul 06 19:59:05.219586 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:05.219547 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6djr7_c099894f-48eb-4431-acca-5464590389f8/exporter/0.log" Jul 06 19:59:05.243320 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:05.243294 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6djr7_c099894f-48eb-4431-acca-5464590389f8/extractor/0.log" Jul 06 19:59:06.498164 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:06.498133 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-f2722" Jul 06 19:59:08.048346 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:08.048307 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-dfb658bc5-kzrgj_20d0828b-9eb2-4926-9fcb-31bd6e87f727/maas-api/0.log" Jul 06 19:59:08.070358 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:08.070283 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722770-ft6x6_177b0d59-f9b3-42f5-b2b4-60bf17264d46/cleanup/0.log" Jul 06 19:59:08.101072 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:08.100994 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722785-7678z_fd4b4b4c-4e5d-410f-8f46-dd79db8f7e0e/cleanup/0.log" Jul 06 19:59:08.325102 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:08.325038 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f978bf57-g64zj_3a43c343-443c-442d-80b1-8b2dc50294ba/manager/0.log" Jul 06 19:59:08.361084 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:08.361057 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f978bf57-g64zj_3a43c343-443c-442d-80b1-8b2dc50294ba/copy-manifests/0.log" Jul 06 19:59:09.564788 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:09.564758 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-m2x4w_8cf813ee-2998-49c0-81b0-ba574c55e85f/discovery/0.log" Jul 06 19:59:09.799314 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:09.799279 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5df75489c9-898zs_51a04452-ffcd-45fe-b844-519e1c90426b/router/0.log" Jul 06 19:59:10.034708 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:10.034680 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-665d858957-529kg_f59248b1-579e-4193-8fe7-e70da0907913/manager/0.log" Jul 06 19:59:11.117676 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:11.117609 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-m2x4w_8cf813ee-2998-49c0-81b0-ba574c55e85f/discovery/0.log" Jul 06 19:59:11.285016 ip-10-0-143-192 kubenswrapper[2581]: I0706 19:59:11.284986 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5df75489c9-898zs_51a04452-ffcd-45fe-b844-519e1c90426b/router/0.log"